To use IPython for parallel computing, you need to start one instance of the controller and one or more instances of the engine. The controller and each engine can run on different machines or on the same machine. Because of this, there are many different possibilities.
Broadly speaking, there are two ways of going about starting a controller and engines:
This document describes both of these methods. We recommend that new users start with the ipcluster command as it simplifies many common usage cases.
Before delving into the details about how you can start a controller and engines using the various methods, we outline some of the general issues that come up when starting the controller and engines. These things come up no matter which method you use to start your IPython cluster.
Let’s say that you want to start the controller on host0 and engines on hosts host1-hostn. The following steps are then required:
At this point, the controller and engines will be connected. By default, the FURL files created by the controller are put into the ~/.ipython/security directory. If the engines share a filesystem with the controller, step 2 can be skipped as the engines will automatically look at that location.
The final step required required to actually use the running controller from a client is to move the FURL files ipcontroller-mec.furl and ipcontroller-tc.furl from host0 to the host where the clients will be run. If these file are put into the ~/.ipython/security directory of the client’s host, they will be found automatically. Otherwise, the full path to them has to be passed to the client’s constructor.
The ipcluster command provides a simple way of starting a controller and engines in the following situations:
Note
It is also possible for advanced users to add support to ipcluster for starting controllers and engines using other methods (like Sun’s Grid Engine for example).
Note
Currently ipcluster requires that the ~/.ipython/security directory live on a shared filesystem that is seen by both the controller and engines. If you don’t have a shared file system you will need to use ipcontroller and ipengine directly. This constraint can be relaxed if you are using the ssh method to start the cluster.
Underneath the hood, ipcluster just uses ipcontroller and ipengine to perform the steps described above.
To start one controller and 4 engines on localhost, just do:
$ ipcluster local -n 4
To see other command line options for the local mode, do:
$ ipcluster local -h
The mpiexec/mpirun mode is useful if you:
Note
The preferred command to use is mpiexec. However, we also support mpirun for backwards compatibility. The underlying logic used is exactly the same, the only difference being the name of the command line program that is called.
If these are satisfied, you can start an IPython cluster using:
$ ipcluster mpiexec -n 4
This does the following:
On newer MPI implementations (such as OpenMPI), this will work even if you don’t make any calls to MPI or call MPI_Init(). However, older MPI implementations actually require each process to call MPI_Init() upon starting. The easiest way of having this done is to install the mpi4py [mpi4py] package and then call ipcluster with the --mpi option:
$ ipcluster mpiexec -n 4 --mpi=mpi4py
Unfortunately, even this won’t work for some MPI implementations. If you are having problems with this, you will likely have to use a custom Python executable that itself calls MPI_Init() at the appropriate time. Fortunately, mpi4py comes with such a custom Python executable that is easy to install and use. However, this custom Python executable approach will not work with ipcluster currently.
Additional command line options for this mode can be found by doing:
$ ipcluster mpiexec -h
More details on using MPI with IPython can be found here.
The PBS mode uses the Portable Batch System [PBS] to start the engines.
To start an ipcluster using the Portable Batch System:
$ ipcluster pbs -n 12
The above command will launch a PBS job array with 12 tasks using the default queue. If you would like to submit the job to a different queue use the -q option:
$ ipcluster pbs -n 12 -q hpcqueue
By default, ipcluster will generate and submit a job script to launch the engines. However, if you need to use your own job script use the -s option:
$ ipcluster pbs -n 12 -q hpcqueue -s mypbscript.sh
For example the default autogenerated script looks like:
#PBS -q hpcqueue
#!/bin/sh
#PBS -V
#PBS -t 1-12
#PBS -N ipengine
eid=$(($PBS_ARRAYID - 1))
ipengine --logfile=ipengine${eid}.log
Note
ipcluster relies on using PBS job arrays to start the engines. If you specify your own job script without specifying the job array settings ipcluster will automatically add the job array settings (#PBS -t 1-N) to your script.
Additional command line options for this mode can be found by doing:
$ ipcluster pbs -h
The SGE mode uses the Sun Grid Engine [SGE] to start the engines.
To start an ipcluster using Sun Grid Engine:
$ ipcluster sge -n 12
The above command will launch an SGE job array with 12 tasks using the default queue. If you would like to submit the job to a different queue use the -q option:
$ ipcluster sge -n 12 -q hpcqueue
By default, ipcluster will generate and submit a job script to launch the engines. However, if you need to use your own job script use the -s option:
$ ipcluster sge -n 12 -q hpcqueue -s mysgescript.sh
For example the default autogenerated script looks like:
#$ -q hpcqueue
#$ -V
#$ -S /bin/sh
#$ -t 1-12
#$ -N ipengine
eid=$(($SGE_TASK_ID - 1))
ipengine --logfile=ipengine${eid}.log #$ -V
Note
ipcluster relies on using SGE job arrays to start the engines. If you specify your own job script without specifying the job array settings ipcluster will automatically add the job array settings (#$ -t 1-N) to your script.
Additional command line options for this mode can be found by doing:
$ ipcluster sge -h
The LSF mode uses the Load Sharing Facility [LSF] to start the engines.
To start an ipcluster using the Load Sharing Facility:
$ ipcluster lsf -n 12
The above command will launch an LSF job array with 12 tasks using the default queue. If you would like to submit the job to a different queue use the -q option:
$ ipcluster lsf -n 12 -q hpcqueue
By default, ipcluster will generate and submit a job script to launch the engines. However, if you need to use your own job script use the -s option:
$ ipcluster lsf -n 12 -q hpcqueue -s mylsfscript.sh
For example the default autogenerated script looks like:
#BSUB -q hpcqueue
#!/bin/sh
#BSUB -J ipengine[1-12]
eid=$(($LSB_JOBINDEX - 1))
ipengine --logfile=ipengine${eid}.log
Note
ipcluster relies on using LSF job arrays to start the engines. If you specify your own job script without specifying the job array settings ipcluster will automatically add the job array settings (#BSUB -J ipengine[1-N]) to your script.
Additional command line options for this mode can be found by doing:
$ ipcluster lsf -h
The SSH mode uses ssh to execute ipengine on remote nodes and the ipcontroller on localhost.
When using using this mode it highly recommended that you have set up SSH keys and are using ssh-agent [SSH] for password-less logins.
To use this mode you need a python file describing the cluster, here is an example of such a “clusterfile”:
send_furl = True
engines = { 'host1.example.com' : 2,
'host2.example.com' : 5,
'host3.example.com' : 1,
'host4.example.com' : 8 }
Since this is a regular python file usual python syntax applies. Things to note:
The --clusterfile command line option lets you specify the file to use for the cluster definition. Once you have your cluster file and you can ssh into the remote hosts with out an password you are ready to start your cluster like so:
$ ipcluster ssh --clusterfile /path/to/my/clusterfile.py
Two helper shell scripts are used to start and stop ipengine on remote hosts:
Defaults for both of these are contained in the source code for ipcluster. The default scripts are written to a local file in a tmep directory and then copied to a temp directory on the remote host and executed from there. On most Unix, Linux and OS X systems this is /tmp.
The default sshx.sh is the following:
#!/bin/sh
"$@" &> /dev/null &
echo $!
If you want to use a custom sshx.sh script you need to use the --sshx option and specify the file to use. Using a custom sshx.sh file could be helpful when you need to setup the environment on the remote host before executing ipengine.
For a detailed options list:
$ ipcluster ssh -h
Current limitations of the SSH mode of ipcluster are:
It is also possible to use the ipcontroller and ipengine commands to start your controller and engines. This approach gives you full control over all aspects of the startup process.
To use ipcontroller and ipengine to start things on your local machine, do the following.
First start the controller:
$ ipcontroller
Next, start however many instances of the engine you want using (repeatedly) the command:
$ ipengine
The engines should start and automatically connect to the controller using the FURL files in ~./ipython/security. You are now ready to use the controller and engines from IPython.
Warning
The order of the above operations is very important. You must start the controller before the engines, since the engines connect to the controller as they get started.
Note
On some platforms (OS X), to put the controller and engine into the background you may need to give these commands in the form (ipcontroller &) and (ipengine &) (with the parentheses) for them to work properly.
When the controller and engines are running on different hosts, things are slightly more complicated, but the underlying ideas are the same:
The only thing you have to be careful of is to tell ipengine where the ipcontroller-engine.furl file is located. There are two ways you can do this:
The --furl-file flag works like this:
$ ipengine --furl-file=/path/to/my/ipcontroller-engine.furl
Note
If the controller’s and engine’s hosts all have a shared file system (~./ipython/security is the same on all of them), then things will just work!
At fist glance it may seem that that managing the FURL files is a bit annoying. Going back to the house and key analogy, copying the FURL around each time you start the controller is like having to make a new key every time you want to unlock the door and enter your house. As with your house, you want to be able to create the key (or FURL file) once, and then simply use it at any point in the future.
This is possible, but before you do this, you must remove any old FURL files in the ~/.ipython/security directory.
Warning
You must remove old FURL files before using persistent FURL files.
Then, The only thing you have to do is decide what ports the controller will listen on for the engines and clients. This is done as follows:
$ ipcontroller -r --client-port=10101 --engine-port=10102
These options also work with all of the various modes of ipcluster:
$ ipcluster local -n 2 -r --client-port=10101 --engine-port=10102
Then, just copy the furl files over the first time and you are set. You can start and stop the controller and engines any many times as you want in the future, just make sure to tell the controller to use the same ports.
Note
You may ask the question: what ports does the controller listen on if you don’t tell is to use specific ones? The default is to use high random port numbers. We do this for two reasons: i) to increase security through obscurity and ii) to multiple controllers on a given host to start and automatically use different ports.
All of the components of IPython have log files associated with them. These log files can be extremely useful in debugging problems with IPython and can be found in the directory ~/.ipython/log. Sending the log files to us will often help us to debug any problems.
| [PBS] | (1, 2) Portable Batch System. http://www.openpbs.org/ |
| [SGE] | (1, 2) Sun Grid Engine. http://www.sun.com/software/sge/ |
| [LSF] | (1, 2) Load Sharing Facility. http://www.platform.com/ |
| [SSH] | SSH-Agent http://en.wikipedia.org/wiki/Ssh-agent |