luisy.tasks package

Submodules

luisy.tasks.base module

class luisy.tasks.base.ConcatenationTask(*args, **kwargs)

Bases: Task

preprocess(df)

run()

class luisy.tasks.base.ExternalTask(*args, **kwargs)

Bases: Task, ExternalTask

class luisy.tasks.base.SparkTask(*args, **kwargs)

Bases: Task

read_input()

Helper to read atomic inputs

property spark

write(df)

class luisy.tasks.base.Task(*args, **kwargs)

Bases: Task

Base task that provides interfaces for all decorators.

clean()

download()

Downloads the output of the task from the cloud.

get_file_name()

get_outdir()

get_outfile()

classmethod get_related_instances(regex_placeholder='[\\S]*')

Find executed instances of the task class on your local machine.

Parameters:

regex_placeholder (str) – Regex placeholder that replaces value strings with a regex to find all matching instances on filesystem.

Returns:

list of task instances related to given class

Return type:

list[luisy.Task]

Raises:

Exception – If problems in finding instances on the system occur, Exceptions are caught in general and the user is informed to look at his get_file_name() method.

get_sub_dir()

Default implementation of a subdir. By overriding this method one can specify an additional subdir under the current working dir.

output()

read()

Reads the output of the task and return is. If cloud synchronisation is activated, a file may be downloaded from the cloud

read_input()

Helper to read atomic inputs

target_cls

alias of PickleTarget

target_kwargs = {}

upload(overwrite=False)

Uploads the output of the task to the cloud.abs

Parameters:

overwrite (bool) – Whether an existing file in the cloud should be overwritten. Defaults to False.

visualize(ax=None, unique_children=True, parameters_to_exclude=())

Visualizes the dependencies tree of the task.

Parameters:

• ax (matplotlib.axis.Axis) – A matplotlib axis to draw in

• unique_children (bool) – If true, add only subset of unique types of children

• parameters_to_exclude (list) – List of names of py:mod:luigi-parameters to exclude in visualization.

write(obj)

class luisy.tasks.base.WrapperTask(*args, **kwargs)

Bases: Task, WrapperTask

download()

Task cannot be downloaded, because no file exists.

read()

Reads all the output of the required wrapped tasks. The type of the return value depends on the type of the return value of the WrapperTask.requires() method.

upload(overwrite=None)

Task cannot be uploaded, because no file exists.

Module contents