The task interface to the controller presents the engines as a fault tolerant, dynamic load-balanced system or workers. Unlike the multiengine interface, in the task interface, the user have no direct access to individual engines. In some ways, this interface is simpler, but in other ways it is more powerful.
Best of all the user can use both of these interfaces running at the same time to take advantage or both of their strengths. When the user can break up the user’s work into segments that do not depend on previous execution, the task interface is ideal. But it also has more power and flexibility, allowing the user to guide the distribution of jobs, without having to assign tasks to engines explicitly.
To follow along with this tutorial, you will need to start the IPython controller and four IPython engines. The simplest way of doing this is to use the ipcluster command:
$ ipcluster -n 4
For more detailed information about starting the controller and engines, see our introduction to using IPython for parallel computing.
The first step is to import the IPython IPython.kernel.client module and then create a TaskClient instance:
In : from IPython.kernel import client In : tc = client.TaskClient()
This form assumes that the ipcontroller-tc.furl is in the ~./ipython/security directory on the client’s host. If not, the location of the .furl file must be given as an argument to the constructor:
In: mec = client.TaskClient('/path/to/my/ipcontroller-tc.furl')
In many cases, you simply want to apply a Python function to a sequence of objects, but in parallel. Like the multiengine interface, the task interface provides two simple ways of accomplishing this: a parallel version of map() and @parallel function decorator. However, the verions in the task interface have one important difference: they are dynamically load balanced. Thus, if the execution time per item varies significantly, you should use the versions in the task interface.
The parallel map() in the task interface is similar to that in the multiengine interface:
In : serial_result = map(lambda x:x**10, range(32)) In : parallel_result = tc.map(lambda x:x**10, range(32)) In : serial_result==parallel_result Out: True
Parallel functions are just like normal function, but they can be called on sequences and in parallel. The multiengine interface provides a decorator that turns any Python function into a parallel function:
In : @tc.parallel() ....: def f(x): ....: return 10.0*x**4 ....: In : f(range(32)) # this is done in parallel Out: [0.0,10.0,160.0,...]
The TaskClient has many more powerful features that allow quite a bit of flexibility in how tasks are defined and run. The next places to look are in the following classes:
The following is an overview of how to use these classes together:
We are in the process of developing more detailed information about the task interface. For now, the docstrings of the TaskClient, StringTask and MapTask classes should be consulted.