kernel.task

Module: kernel.task

Inheritance diagram for IPython.kernel.task:

Task farming representation of the ControllerService.

Classes

BaseTask

class IPython.kernel.task.BaseTask(clear_before=False, clear_after=False, retries=0, recovery_task=None, depend=None)

Bases: object

Common fuctionality for all objects implementing ITask.

__init__(clear_before=False, clear_after=False, retries=0, recovery_task=None, depend=None)

Make a generic task.

Parameters:

clear_before : boolean

Should the engines namespace be cleared before the task is run

clear_after : boolean

Should the engines namespace be clear after the task is run

retries : int

The number of times a task should be retries upon failure

recovery_task : any task object

If a task fails and it has a recovery_task, that is run upon a retry

depend : FunctionType

A function that is called to test for properties. This function must take one argument, the properties dict and return a boolean

can_task()

check_depend(properties)

Calls self.depend(properties) to see if a task should be run.

post_task(d, queued_engine)

Clear the engine after running the task if clear_after is set.

pre_task(d, queued_engine)

Clear the engine before running the task if clear_before is set.

process_result(result, engine_id)

Process a task result.

This is the default process_result that just returns the raw result or a Failure.

start_time(result)

Start the basic timers.

stop_time(result)

Stop the basic timers.

submit_task(d, queued_engine)

uncan_task()

FIFOScheduler

class IPython.kernel.task.FIFOScheduler

Bases: object

A basic First-In-First-Out (Queue) Scheduler.

This is the default Scheduler for the TaskController. See the docstrings for IScheduler for interface details.

__init__()

add_task(task, **flags)

add_worker(worker, **flags)

ntasks

nworkers

pop_task(id=None)

pop_worker(id=None)

schedule()

taskids

workerids

IScheduler

class IPython.kernel.task.IScheduler(name, bases=(), attrs=None, __doc__=None, __module__=None)

Bases: zope.interface.Interface

The interface for a Scheduler.

static __init__(name, bases=(), attrs=None, __doc__=None, __module__=None)

ITask

class IPython.kernel.task.ITask(name, bases=(), attrs=None, __doc__=None, __module__=None)

Bases: zope.interface.Interface

This interface provides a generic definition of what constitutes a task.

There are two sides to a task. First a task needs to take input from a user to determine what work is performed by the task. Second, the task needs to have the logic that knows how to turn that information info specific calls to a worker, through the IQueuedEngine interface.

Many method in this class get two things passed to them: a Deferred and an IQueuedEngine implementer. Such methods should register callbacks on the Deferred that use the IQueuedEngine to accomplish something. See the existing task objects for examples.

static __init__(name, bases=(), attrs=None, __doc__=None, __module__=None)

ITaskController

class IPython.kernel.task.ITaskController(name, bases=(), attrs=None, __doc__=None, __module__=None)

Bases: IPython.kernel.controllerservice.IControllerBase

The Task based interface to a ControllerService object

This adapts a ControllerService to the ITaskController interface.

static __init__(name, bases=(), attrs=None, __doc__=None, __module__=None)

IWorker

class IPython.kernel.task.IWorker(name, bases=(), attrs=None, __doc__=None, __module__=None)

Bases: zope.interface.Interface

The Basic Worker Interface.

A worked is a representation of an Engine that is ready to run tasks.

static __init__(name, bases=(), attrs=None, __doc__=None, __module__=None)

LIFOScheduler

class IPython.kernel.task.LIFOScheduler

Bases: IPython.kernel.task.FIFOScheduler

A Last-In-First-Out (Stack) Scheduler.

This scheduler should naively reward fast engines by giving them more jobs. This risks starvation, but only in cases with low load, where starvation does not really matter.

__init__()

add_task(task, **flags)

add_worker(worker, **flags)

MapTask

class IPython.kernel.task.MapTask(function, args=None, kwargs=None, clear_before=False, clear_after=False, retries=0, recovery_task=None, depend=None)

Bases: IPython.kernel.task.BaseTask

A task that consists of a function and arguments.

__init__(function, args=None, kwargs=None, clear_before=False, clear_after=False, retries=0, recovery_task=None, depend=None)

Create a task based on a function, args and kwargs.

This is a simple type of task that consists of calling: function(*args, **kwargs) and wrapping the result in a TaskResult.

The return value of the function, or a Failure wrapping an exception is the task result for this type of task.

can_task()

submit_task(d, queued_engine)

uncan_task()

ResultNS

class IPython.kernel.task.ResultNS(dikt)

Bases: object

A dict like object for holding the results of a task.

The result namespace object for use in TaskResult objects as tr.ns. It builds an object from a dictionary, such that it has attributes according to the key,value pairs of the dictionary.

This works by calling setattr on ALL key,value pairs in the dict. If a user chooses to overwrite the __repr__ or __getattr__ attributes, they can. This can be a bad idea, as it may corrupt standard behavior of the ns object.

Examples

>>> ns = ResultNS({'a':17,'foo':range(3)})
>>> print ns
NS{'a': 17, 'foo': [0, 1, 2]}
>>> ns.a
17
>>> ns['foo']
[0, 1, 2]

__init__(dikt)

StringTask

class IPython.kernel.task.StringTask(expression, pull=None, push=None, clear_before=False, clear_after=False, retries=0, recovery_task=None, depend=None)

Bases: IPython.kernel.task.BaseTask

A task that consists of a string of Python code to run.

__init__(expression, pull=None, push=None, clear_before=False, clear_after=False, retries=0, recovery_task=None, depend=None)

Create a task based on a Python expression and variables

This type of task lets you push a set of variables to the engines namespace, run a Python string in that namespace and then bring back a different set of Python variables as the result.

Because this type of task can return many results (through the pull keyword argument) it returns a special TaskResult object that wraps the pulled variables, statistics about the run and any exceptions raised.

process_result(result, engine_id)

submit_task(d, queued_engine)

TaskController

class IPython.kernel.task.TaskController(controller)

Bases: IPython.kernel.controllerservice.ControllerAdapterBase

The Task based interface to a Controller object.

If you want to use a different scheduler, just subclass this and set the SchedulerClass member to the class of your chosen scheduler.

__init__(controller)

SchedulerClass()

A basic First-In-First-Out (Queue) Scheduler.

This is the default Scheduler for the TaskController. See the docstrings for IScheduler for interface details.

abort(taskid)

Remove a task from the queue if it has not been run already.

barrier(taskids)

checkIdle()

clear()

Clear all previously run tasks from the task controller.

This is needed because the task controller keep all task results in memory. This can be a problem is there are many completed tasks. Users should call this periodically to clean out these cached task results.

distributeTasks()

Distribute tasks while self.scheduler has things to do.

failIdle()

get_task_result(taskid, block=False)

Returns a Deferred to the task result, or None.

queue_status(verbose=False)

readmitWorker(workerid)

Readmit a worker to the scheduler.

This is outside taskCompleted because of the failurePenalty being implemented through reactor.callLater.

registerWorker(id)

Called by controller.register_engine.

run(task)

Run a task and return Deferred to its taskid.

spin()

taskCompleted(success_and_result, taskid, workerid)

This is the err/callback for a completed task.

unregisterWorker(id)

Called by controller.unregister_engine

TaskResult

class IPython.kernel.task.TaskResult(results, engineid)

Bases: object

An object for returning task results for certain types of tasks.

This object encapsulates the results of a task. On task success it will have a keys attribute that will have a list of the variables that have been pulled back. These variables are accessible as attributes of this class as well. On success the failure attribute will be None.

In task failure, keys will be empty, but failure will contain the failure object that encapsulates the remote exception. One can also simply call the raise_exception method of this class to re-raise any remote exception in the local session.

The TaskResult has a .ns member, which is a property for access to the results. If the Task had pull=[‘a’, ‘b’], then the Task Result will have attributes tr.ns.a, tr.ns.b for those values. Accessing tr.ns will raise the remote failure if the task failed.

The engineid attribute should have the engineid of the engine that ran the task. But, because engines can come and go, the engineid may not continue to be valid or accurate.

The taskid attribute simply gives the taskid that the task is tracked under.

__init__(results, engineid)

ns

raise_exception()

Re-raise any remote exceptions in the local python session.

WorkerFromQueuedEngine

class IPython.kernel.task.WorkerFromQueuedEngine(qe)

Bases: object

Adapt an IQueuedEngine to an IWorker object

__init__(qe)

properties

run(task)

Run task in worker’s namespace.

This takes a task and calls methods on the task that actually cause self.queuedEngine to do the task. See the methods of ITask for more information about how these methods are called.

Parameters:task : a Task object Returns:Deferred to a tuple of (success, result) where success if a boolean that signifies success or failure and result is the task result.