This file gives an overview of IPython’s sophisticated and powerful architecture for parallel and distributed computing. This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism including:
Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython. The following are some example usage cases for IPython:
The IPython architecture consists of three components:
These components live in the IPython.kernel package and are installed with IPython. They do, however, have additional dependencies that must be installed. For more information, see our installation documentation.
The IPython engine is a Python instance that takes Python commands over a network connection. Eventually, the IPython engine will be a full IPython interpreter, but for now, it is a regular Python interpreter. The engine can also handle incoming and outgoing Python objects sent over a network connection. When multiple engines are started, parallel and distributed computing becomes possible. An important feature of an IPython engine is that it blocks while user code is being executed. Read on for how the IPython controller solves this problem to expose a clean asynchronous API to the user.
The IPython controller provides an interface for working with a set of engines. At an general level, the controller is a process to which IPython engines can connect. For each connected engine, the controller manages a queue. All actions that can be performed on the engine go through this queue. While the engines themselves block when user code is run, the controller hides that from the user to provide a fully asynchronous interface to a set of engines.
Note
Because the controller listens on a network port for engines to connect to it, it must be started before any engines are started.
The controller also provides a single point of contact for users who wish to utilize the engines connected to the controller. There are different ways of working with a controller. In IPython these ways correspond to different interfaces that the controller is adapted to. Currently we have two default interfaces to the controller:
Advanced users can easily add new custom interfaces to enable other styles of parallelism.
Note
A single controller and set of engines can be accessed through multiple interfaces simultaneously. This opens the door for lots of interesting things.
For each controller interface, there is a corresponding client. These clients allow users to interact with a set of engines through the interface. Here are the two default clients:
By default (as long as pyOpenSSL is installed) all network connections between the controller and engines and the controller and clients are secure. What does this mean? First of all, all of the connections will be encrypted using SSL. Second, the connections are authenticated. We handle authentication in a capabilities based security model. In this model, a “capability (known in some systems as a key) is a communicable, unforgeable token of authority”. Put simply, a capability is like a key to your house. If you have the key to your house, you can get in. If not, you can’t.
In our architecture, the controller is the only process that listens on network ports, and is thus responsible to creating these keys. In IPython, these keys are known as Foolscap URLs, or FURLs, because of the underlying network protocol we are using. As a user, you don’t need to know anything about the details of these FURLs, other than that when the controller starts, it saves a set of FURLs to files named something.furl. The default location of these files is the ~./ipython/security directory.
To connect and authenticate to the controller an engine or client simply needs to present an appropriate furl (that was originally created by the controller) to the controller. Thus, the .furl files need to be copied to a location where the clients and engines can find them. Typically, this is the ~./ipython/security directory on the host where the client/engine is running (which could be a different host than the controller). Once the .furl files are copied over, everything should work fine.
Currently, there are three .furl files that the controller creates:
More details of how these .furl files are used are given below.
To use IPython for parallel computing, you need to start one instance of the controller and one or more instances of the engine. The controller and each engine can run on different machines or on the same machine. Because of this, there are many different possibilities for setting up the IP addresses and ports used by the various processes.
This is the simplest configuration that can be used and is useful for testing the system and on machines that have multiple cores and/or multple CPUs. The easiest way of getting started is to use the ipcluster command:
$ ipcluster -n 4
This will start an IPython controller and then 4 engines that connect to the controller. Lastly, the script will print out the Python commands that you can use to connect to the controller. It is that easy.
Warning
The ipcluster does not currently work on Windows. We are working on it though.
Underneath the hood, the controller creates .furl files in the ~./ipython/security directory. Because the engines are on the same host, they automatically find the needed ipcontroller-engine.furl there and use it to connect to the controller.
The ipcluster script uses two other top-level scripts that you can also use yourself. These scripts are ipcontroller, which starts the controller and ipengine which starts one engine. To use these scripts to start things on your local machine, do the following.
First start the controller:
$ ipcontroller
Next, start however many instances of the engine you want using (repeatedly) the command:
$ ipengine
The engines should start and automatically connect to the controller using the .furl files in ~./ipython/security. You are now ready to use the controller and engines from IPython.
Warning
The order of the above operations is very important. You must start the controller before the engines, since the engines connect to the controller as they get started.
Note
On some platforms (OS X), to put the controller and engine into the background you may need to give these commands in the form (ipcontroller &) and (ipengine &) (with the parentheses) for them to work properly.
When the controller and engines are running on different hosts, things are slightly more complicated, but the underlying ideas are the same:
The only thing you have to be careful of is to tell ipengine where the ipcontroller-engine.furl file is located. There are two ways you can do this:
The --furl-file flag works like this:
$ ipengine --furl-file=/path/to/my/ipcontroller-engine.furl
Note
If the controller’s and engine’s hosts all have a shared file system (~./ipython/security is the same on all of them), then things will just work!
At fist glance it may seem that that managing the .furl files is a bit annoying. Going back to the house and key analogy, copying the .furl around each time you start the controller is like having to make a new key everytime you want to unlock the door and enter your house. As with your house, you want to be able to create the key (or .furl file) once, and then simply use it at any point in the future.
This is possible. The only thing you have to do is decide what ports the controller will listen on for the engines and clients. This is done as follows:
$ ipcontroller --client-port=10101 --engine-port=10102
Then, just copy the furl files over the first time and you are set. You can start and stop the controller and engines any many times as you want in the future, just make sure to tell the controller to use the same ports.
Note
You may ask the question: what ports does the controller listen on if you don’t tell is to use specific ones? The default is to use high random port numbers. We do this for two reasons: i) to increase security through obcurity and ii) to multiple controllers on a given host to start and automatically use different ports.
The IPython engines can be started using mpirun/mpiexec, even if the engines don’t call MPI_Init() or use the MPI API in any way. This is supported on modern MPI implementations like Open MPI.. This provides an really nice way of starting a bunch of engine. On a system with MPI installed you can do:
mpirun -n 4 ipengine
to start 4 engine on a cluster. This works even if you don’t have any Python-MPI bindings installed.
More details on using MPI with IPython can be found here.
All of the components of IPython have log files associated with them. These log files can be extremely useful in debugging problems with IPython and can be found in the directory ~/.ipython/log. Sending the log files to us will often help us to debug any problems.
Once you have started the IPython controller and one or more engines, you are ready to use the engines to do something useful. To make sure everything is working correctly, try the following commands:
In [1]: from IPython.kernel import client
In [2]: mec = client.MultiEngineClient()
In [4]: mec.get_ids()
Out[4]: [0, 1, 2, 3]
In [5]: mec.execute('print "Hello World"')
Out[5]:
<Results List>
[0] In [1]: print "Hello World"
[0] Out[1]: Hello World
[1] In [1]: print "Hello World"
[1] Out[1]: Hello World
[2] In [1]: print "Hello World"
[2] Out[1]: Hello World
[3] In [1]: print "Hello World"
[3] Out[1]: Hello World
Remember, a client also needs to present a .furl file to the controller. How does this happen? When a multiengine client is created with no arguments, the client tries to find the corresponding .furl file in the local ~./ipython/security directory. If it finds it, you are set. If you have put the .furl file in a different location or it has a different name, create the client like this:
mec = client.MultiEngineClient('/path/to/my/ipcontroller-mec.furl')
Same thing hold true of creating a task client:
tc = client.TaskClient('/path/to/my/ipcontroller-tc.furl')
You are now ready to learn more about the MultiEngine and Task interfaces to the controller.
Note
Don’t forget that the engine, multiengine client and task client all have different furl files. You must move each of these around to an appropriate location so that the engines and clients can use them to connect to the controller.