IPython is the next generation of IPython. It is named such for two reasons:
There are two, no three, main goals of the IPython effort:
While the third goal may seem a bit unrelated to the main focus of IPython, it turns out that the technologies required for this goal are nearly identical with those required for goal two. This is the main reason the interactive parallel computing capabilities are being put into IPython proper. Currently the third of these goals is furthest along.
This document describes IPython from the perspective of developers.
IPython is organized into semi self-contained subpackages. Each of the subpackages will have its own:
IPython will not use setuptools for installation. Instead, we will use standard setup.py scripts that use distutils. While there are a number a extremely nice features that setuptools has (like namespace packages), the current implementation of setuptools has performance problems, particularly on shared file systems. In particular, when Python packages are installed on NSF file systems, import times become much too long (up towards 10 seconds).
Because IPython is being used extensively in the context of high performance computing, where performance is critical but shared file systems are common, we feel these performance hits are not acceptable. Thus, until the performance problems associated with setuptools are addressed, we will stick with plain distutils. We are hopeful that these problems will be addressed and that we will eventually begin using setuptools. Because of this, we are trying to organize IPython in a way that will make the eventual transition to setuptools as painless as possible.
Because we will be using distutils, there will be no method for automatically installing dependencies. Instead, we are following the approach of Matplotlib which can be summarized as follows:
It is absolutely critical that each subpackage of IPython has a clearly specified set of dependencies and that dependencies are not carelessly inherited from other IPython subpackages. Furthermore, tests that have certain dependencies should not fail if those dependencies are not present. Instead they should be skipped and print a message.
In the past, IPython development has been done using Subversion. Recently, we made the transition to using Bazaar and Launchpad. This makes it much easier for people to contribute code to IPython. Here is a sketch of how to use Bazaar for IPython development. First, you should install Bazaar. After you have done that, make sure that it is working by getting the latest main branch of IPython:
$ bzr branch lp:ipython
Now you can create a new branch for you to do your work in:
$ bzr branch ipython ipython-mybranch
The typical work cycle in this branch will be to make changes in ipython-mybranch and then commit those changes using the commit command:
$ ...do work in ipython-mybranch... $ bzr ci -m "the commit message goes here"
Please note that since we now don’t use an old-style linear ChangeLog (that tends to cause problems with distributed version control systems), you should ensure that your log messages are reasonably detailed. Use a docstring-like approach in the commit messages (including the second line being left blank):
Single line summary of changes being committed. - more details when warranted ... - including crediting outside contributors if they sent the code/bug/idea!
If we couple this with a policy of making single commits for each reasonably atomic change, the bzr log should give an excellent view of the project, and the –short log option becomes a nice summary.
While working with this branch, it is a good idea to merge in changes that have been made upstream in the parent branch. This can be done by doing:
$ bzr pull
If this command shows that the branches have diverged, then you should do a merge instead:
$ bzr merge lp:ipython
If you want others to be able to see your branch, you can create an account with launchpad and push the branch to your own workspace:
$ bzr push bzr+ssh://<me>@bazaar.launchpad.net/~<me>/+junk/ipython-mybranch
Finally, once the work in your branch is done, you can merge your changes back into the ipython branch by using merge:
$ cd ipython $ merge ../ipython-mybranch [resolve any conflicts] $ bzr ci -m "Fixing that bug" $ bzr push
But this will require you to have write permissions to the ipython branch. It you don’t you can tell one of the IPython devs about your branch and they can do the merge for you.
More information about Bazaar workflows can be found here.
All standalone documentation should be written in plain text (.txt) files using reStructuredText for markup and formatting. All such documentation should be placed in the top level directory docs of the IPython source tree. Or, when appropriate, a suitably named subdirectory should be used. The documentation in this location will serve as the main source for IPython documentation and all existing documentation should be converted to this format.
In the future, the text files in the docs directory will be used to generate all forms of documentation for IPython. This include documentation on the IPython website as well as pdf documentation.
Good docstrings are very important. All new code will use Epydoc for generating API docs, so we will follow the Epydoc conventions. More specifically, we will use reStructuredText for markup and formatting, since it is understood by a wide variety of tools. This means that if in the future we have any reason to change from Epydoc to something else, we’ll have fewer transition pains.
Details about using reStructuredText for docstrings can be found here.
Additional PEPs of interest regarding documentation of code:
In general, we’ll try to follow the standard Python style conventions as described here:
Other comments:
In terms of naming conventions, we’ll follow the guidelines from the Style Guide for Python Code.
For all new IPython code (and much existing code is being refactored), we’ll use:
This may be confusing as most of the existing IPython codebase uses a different convention (lowerCamelCase for methods and attributes). Slowly, we will move IPython over to the new convention, providing shadow names for backward compatibility in public interfaces.
There are, however, some important exceptions to these rules. In some cases, IPython code will interface with packages (Twisted, Wx, Qt) that use other conventions. At some level this makes it impossible to adhere to our own standards at all times. In particular, when subclassing classes that use other naming conventions, you must follow their naming conventions. To deal with cases like this, we propose the following policy:
Implementation-specific private methods will use _single_underscore_prefix. Names with a leading double underscore will only be used in special cases, as they makes subclassing difficult (such names are not easily seen by child classes).
Occasionally some run-in lowercase names are used, but mostly for very short names or where we are implementing methods very similar to existing ones in a base class (like runlines() where runsource() and runcode() had established precedent).
The old IPython codebase has a big mix of classes and modules prefixed with an explicit IP. In Python this is mostly unnecessary, redundant and frowned upon, as namespaces offer cleaner prefixing. The only case where this approach is justified is for classes which are expected to be imported into external namespaces and a very generic name (like Shell) is too likely to clash with something else. We’ll need to revisit this issue as we clean up and refactor the code, but in general we should remove as many unnecessary IP/ip prefixes as possible. However, if a prefix seems absolutely necessary the more specific IPY or ipy are preferred.
It is extremely important that all code contributed to IPython has tests. Tests should be written as unittests, doctests or as entities that the Nose testing package will find. Regardless of how the tests are written, we will use Nose for discovering and running the tests. Nose will be required to run the IPython test suite, but will not be required to simply use IPython.
Tests of Twisted using code should be written by subclassing the TestCase class that comes with twisted.trial.unittest. When this is done, Nose will be able to run the tests and the twisted reactor will be handled correctly.
Each subpackage in IPython should have its own tests directory that contains all of the tests for that subpackage. This allows each subpackage to be self-contained. If a subpackage has any dependencies beyond the Python standard library, the tests for that subpackage should be skipped if the dependencies are not found. This is very important so users don’t get tests failing simply because they don’t have dependencies.
We also need to look into use Noses ability to tag tests to allow a more modular approach of running tests.
IPython uses .ini files for configuration purposes. This represents a huge improvement over the configuration system used in IPython. IPython works with these files using the ConfigObj package, which IPython includes as ipython1/external/configobj.py.
Currently, we are using raw ConfigObj objects themselves. Each subpackage of IPython should contain a config subdirectory that contains all of the configuration information for the subpackage. To see how configuration information is defined (along with defaults) see at the examples in ipython1/kernel/config and ipython1/core/config. Likewise, to see how the configuration information is used, see examples in ipython1/kernel/scripts/ipengine.py.
Eventually, we will add a new layer on top of the raw ConfigObj objects. We are calling this new layer, tconfig, as it will use a Traits-like validation model. We won’t actually use Traits, but will implement something similar in pure Python. But, even in this new system, we will still use ConfigObj and .ini files underneath the hood. Talk to Fernando if you are interested in working on this part of IPython. The current prototype of tconfig is located in the IPython sandbox.
This section outlines the various scenarios that we need to test before we release an IPython version. These scenarios represent different ways of installing IPython and its dependencies.
Install from tarball using python setup.py install. a. With only readline+nose dependencies installed. b. With all dependencies installed (readline, zope.interface, Twisted, foolscap, Sphinx, nose, pyOpenSSL).
Install using easy_install.
- a. With only readline+nose dependencies installed.
i. Default dependencies: easy_install ipython-0.9.beta3-py2.5.egg ii. Optional dependency sets: easy_install -f ipython-0.9.beta3-py2.5.egg IPython[kernel,doc,test,security]
b. With all dependencies already installed.
- Install everything from .exe installers
- easy_install?
- Run the full test suite.
- Start a controller and engines and try a few things by hand. a. Using ipcluster. b. Using ipcontroller/ipengine by hand.
- Run a few of the parallel examples.
- Try the kernel with and without security with and without PyOpenSSL installed.
- Beat on the IPython terminal a bunch.
- Make sure that furl files are being put in proper locations.
Most of the release process is automated by the release script in the tools directory. This is just a handy reminder for the release manager.