The IPython task interface

The task interface to the controller presents the engines as a fault tolerant, dynamic load-balanced system or workers. Unlike the multiengine interface, in the task interface, the user have no direct access to individual engines. In some ways, this interface is simpler, but in other ways it is more powerful.

Best of all the user can use both of these interfaces running at the same time to take advantage or both of their strengths. When the user can break up the user’s work into segments that do not depend on previous execution, the task interface is ideal. But it also has more power and flexibility, allowing the user to guide the distribution of jobs, without having to assign tasks to engines explicitly.

Starting the IPython controller and engines

To follow along with this tutorial, you will need to start the IPython controller and four IPython engines. The simplest way of doing this is to use the ipcluster command:

$ ipcluster local -n 4

For more detailed information about starting the controller and engines, see our introduction to using IPython for parallel computing.

Creating a TaskClient instance

The first step is to import the IPython IPython.kernel.client module and then create a TaskClient instance:

In [1]: from IPython.kernel import client

In [2]: tc = client.TaskClient()

This form assumes that the ipcontroller-tc.furl is in the ~./ipython/security directory on the client’s host. If not, the location of the FURL file must be given as an argument to the constructor:

In [2]: mec = client.TaskClient('/path/to/my/ipcontroller-tc.furl')

Quick and easy parallelism

In many cases, you simply want to apply a Python function to a sequence of objects, but in parallel. Like the multiengine interface, the task interface provides two simple ways of accomplishing this: a parallel version of map() and @parallel function decorator. However, the verions in the task interface have one important difference: they are dynamically load balanced. Thus, if the execution time per item varies significantly, you should use the versions in the task interface.

Parallel map

The parallel map() in the task interface is similar to that in the multiengine interface:

In [63]: serial_result = map(lambda x:x**10, range(32))

In [64]: parallel_result = tc.map(lambda x:x**10, range(32))

In [65]: serial_result==parallel_result
Out[65]: True

Parallel function decorator

Parallel functions are just like normal function, but they can be called on sequences and in parallel. The multiengine interface provides a decorator that turns any Python function into a parallel function:

In [10]: @tc.parallel()
   ....: def f(x):
   ....:     return 10.0*x**4
   ....:

In [11]: f(range(32))    # this is done in parallel
Out[11]:
[0.0,10.0,160.0,...]

More details

The TaskClient has many more powerful features that allow quite a bit of flexibility in how tasks are defined and run. The next places to look are in the following classes:

• IPython.kernel.client.TaskClient
• IPython.kernel.client.StringTask
• IPython.kernel.client.MapTask

The following is an overview of how to use these classes together:

1. Create a TaskClient.
2. Create one or more instances of StringTask or MapTask to define your tasks.
3. Submit your tasks to using the run() method of your TaskClient instance.
4. Use TaskClient.get_task_result() to get the results of the tasks.

We are in the process of developing more detailed information about the task interface. For now, the docstrings of the TaskClient, StringTask and MapTask classes should be consulted.