This document describes IPython from the perspective of developers. Most importantly, it gives information for people who want to contribute to the development of IPython. So if you want to help out, read on!
IPython development is done using Bazaar [Bazaar] and Launchpad [Launchpad]. This makes it easy for people to contribute to the development of IPython. There are several ways in which you can join in.
If you have a small change that you want to send to the team, you can edit your bazaar checkout of IPython (see below) in-place, and ask bazaar for the differences:
$ cd /path/to/your/copy/of/ipython
$ bzr diff > my_fixes.diff
This produces a patch file with your fixes, which we can apply to the source tree. This file should then be attached to a ticket in our bug tracker, indicating what it does.
This model of creating small, self-contained patches works very well and there are open source projects that do their entire development this way. However, in IPython we have found that for tracking larger changes, making use of bazaar’s full capabilities in conjunction with Launchpad’s code hosting services makes for a much better experience.
Making your own branch of IPython allows you to refine your changes over time, track the development of the main team, and propose your own full version of the code for others to use and review, with a minimum amount of fuss. The next parts of this document will explain how to do this.
First make sure you have installed Bazaar (see their website). To see that Bazaar is installed and knows about you, try the following:
$ bzr whoami
Joe Coder <jcoder@gmail.com>
This should display your name and email. Next, you will want to create an account on the Launchpad website and setup your ssh keys. For more information of setting up your ssh keys, see this link.
Now, you can get a copy of the main IPython development branch (we call this the “trunk”):
$ bzr branch lp:ipython
When working on IPython, you won’t actually make edits directly to the lp:ipython branch. Instead, you will create a separate branch for your changes. For now, let’s assume you want to do your work in a branch named “ipython-mybranch”. Create this branch by doing:
$ bzr branch ipython ipython-mybranch
When you actually create a branch, you will want to give it a name that reflects the nature of the work that you will be doing in it, like “install-docs-update”.
Now you are ready to actually make edits in your ipython-mybranch branch. Before doing this, it is helpful to install this branch so you can test your changes as you work. This is easiest if you have setuptools installed. Then, just do:
$ cd ipython-mybranch
$ python setupegg.py develop
Now, make some changes. After a while, you will want to commit your changes. This let’s Bazaar know that you like the changes you have made and gives you an opportunity to keep a nice record of what you have done. This looks like this:
$ ...do work in ipython-mybranch...
$ bzr commit -m "the commit message goes here"
Please note that since we now don’t use an old-style linear ChangeLog (that tends to cause problems with distributed version control systems), you should ensure that your log messages are reasonably detailed. Use a docstring-like approach in the commit messages (including the second line being left blank):
Single line summary of changes being committed.
* more details when warranted ...
* including crediting outside contributors if they sent the
code/bug/idea!
As you work, you will repeat this edit/commit cycle many times. If you work on your branch for a long time, you will also want to get the latest changes from the lp:ipython branch. This can be done with the following sequence of commands:
$ ls
ipython
ipython-mybranch
$ cd ipython
$ bzr pull
$ cd ../ipython-mybranch
$ bzr merge ../ipython
$ bzr commit -m "Merging changes from trunk"
Along the way, you should also run the IPython test suite. You can do this using the iptest command (which is basically a customized version of nosetests):
$ cd
$ iptest
The iptest command will also pick up and run any tests you have written. See _devel_testing for further details on the testing system.
Once you are done with your edits, you should post your branch on Launchpad so that other IPython developers can review the changes and help you merge your changes into the main development branch. To post your branch on Launchpad, do:
$ cd ipython-mybranch
$ bzr push lp:~yourusername/ipython/ipython-mybranch
Then, go to the IPython Launchpad site, and you should see your branch under the “Code” tab. If you click on your branch, you can provide a short description of the branch as well as mark its status. Most importantly, you should click the link that reads “Propose for merging into another branch”. What does this do?
This let’s the other IPython developers know that your branch is ready to be reviewed and merged into the main development branch. During this review process, other developers will give you feedback and help you get your code ready to be merged. What types of things will we be looking for:
Once your changes have been reviewed and approved, someone will merge them into the main development branch.
Core developers, who ultimately merge any approved branch (from themselves, another developer, or any third-party contribution) will typically use bzr merge to merge the branch into the trunk and push it to the main Launcphad site. This is a short list of things to keep in mind when doing this process, so that the project history is easy to understand in the long run, and that generating release notes is as painless and accurate as possible.
When you merge any non-trivial functionality (from one small bug fix to a big feature branch), please remember to always edit the changes_ file accordingly. This file has one main section for each release, and if you edit it as you go, noting what new features, bug fixes or API changes you have made, the release notes will be almost finished when they are needed later. This is much easier if done when you merge the work, rather than weeks or months later by re-reading a massive Bazaar log.
When big merges are done, the practice of putting a summary commit message in the merge is extremely useful. It makes this kind of job much nicer, because that summary log message can be almost copy/pasted without changes, if it was well written, rather than dissecting the next-level messages from the individual commits.
It’s important that we remember to always credit who gave us something if it’s not the committer. In general, we have been fairly good on this front, this is just a reminder to keep things up. As a note, if you are ever committing something that is completely (or almost so) a third-party contribution, do the commit as:
$ bzr commit --author="Someone Else"
This way it will show that name separately in the log, which makes it even easier to spot. Obviously we often rework third party contributions extensively, but this is still good to keep in mind for cases when we don’t touch the code too much.
All standalone documentation should be written in plain text (.txt) files using reStructuredText [reStructuredText] for markup and formatting. All such documentation should be placed in directory docs/source of the IPython source tree. The documentation in this location will serve as the main source for IPython documentation and all existing documentation should be converted to this format.
To build the final documentation, we use Sphinx [Sphinx]. Once you have Sphinx installed, you can build the html docs yourself by doing:
$ cd ipython-mybranch/docs
$ make html
Good docstrings are very important. All new code should have docstrings that are formatted using reStructuredText for markup and formatting, since it is understood by a wide variety of tools. Details about using reStructuredText for docstrings can be found here.
Additional PEPs of interest regarding documentation of code:
In general, we’ll try to follow the standard Python style conventions as described here:
Other comments:
In terms of naming conventions, we’ll follow the guidelines from the Style Guide for Python Code.
For all new IPython code (and much existing code is being refactored), we’ll use:
There are, however, some important exceptions to these rules. In some cases, IPython code will interface with packages (Twisted, Wx, Qt) that use other conventions. At some level this makes it impossible to adhere to our own standards at all times. In particular, when subclassing classes that use other naming conventions, you must follow their naming conventions. To deal with cases like this, we propose the following policy:
Implementation-specific private methods will use _single_underscore_prefix. Names with a leading double underscore will only be used in special cases, as they makes subclassing difficult (such names are not easily seen by child classes).
Occasionally some run-in lowercase names are used, but mostly for very short names or where we are implementing methods very similar to existing ones in a base class (like runlines() where runsource() and runcode() had established precedent).
The old IPython codebase has a big mix of classes and modules prefixed with an explicit IP. In Python this is mostly unnecessary, redundant and frowned upon, as namespaces offer cleaner prefixing. The only case where this approach is justified is for classes which are expected to be imported into external namespaces and a very generic name (like Shell) is too likely to clash with something else. We’ll need to revisit this issue as we clean up and refactor the code, but in general we should remove as many unnecessary IP/ip prefixes as possible. However, if a prefix seems absolutely necessary the more specific IPY or ipy are preferred.
It is extremely important that all code contributed to IPython has tests. Tests should be written as unittests, doctests or as entities that the Nose [Nose] testing package will find. Regardless of how the tests are written, we will use Nose for discovering and running the tests. Nose will be required to run the IPython test suite, but will not be required to simply use IPython.
Tests of Twisted using code need to follow two additional guidelines:
When these two things are done, Nose will be able to run the tests and the twisted reactor will be handled correctly.
Each subpackage in IPython should have its own tests directory that contains all of the tests for that subpackage. This allows each subpackage to be self-contained. A good convention to follow is to have a file named test_foo.py for each module foo.py in the package. This makes it easy to organize the tests, though like most conventions, it’s OK to break it if logic and common sense dictate otherwise.
If a subpackage has any dependencies beyond the Python standard library, the tests for that subpackage should be skipped if the dependencies are not found. This is very important so users don’t get tests failing simply because they don’t have dependencies. We ship a set of decorators in the IPython.testing package to tag tests that may be platform-specific or otherwise may have restrictions; if the existing ones don’t fit your needs, add a new decorator in that location so other tests can reuse it.
To run the IPython test suite, use the iptest command that is installed with IPython (if you are using IPython in-place, without installing it, you can find this script in the scripts directory):
$ iptest
This command colects all IPython tests into separate groups, and then calls either Nose with the proper options and extensions, or Twisted’s trial. This ensures that tests that need the Twisted reactor management facilities execute separate of Nose. If any individual test group fails, iptest will print what you need to type so you can rerun that particular test group alone for debugging.
By default, iptest runs the entire IPython test suite (skipping tests that may be platform-specific or which depend on tools you may not have). But you can also use it to run only one specific test file, or a specific test function. For example, this will run only the test_magic file from the test suite:
$ iptest IPython.tests.test_magic
----------------------------------------------------------------------
Ran 10 tests in 0.348s
OK (SKIP=3)
Deleting object: second_pass
while the path:function syntax allows you to select a specific function in that file to run:
$ iptest IPython.tests.test_magic:test_obj_del
----------------------------------------------------------------------
Ran 1 test in 0.204s
OK
Since iptest is based on nosetests, you can pass it any regular nosetests option. For example, you can use --pdb or --pdb-failures to automatically activate the interactive Pdb debugger on errors or failures. See the nosetests documentation for further details.
You can write tests either as normal test files, using all the conventions that Nose recognizes, or as doctests. Note that all IPython functions should have at least one example that serves as a doctest, whenever technically feasible. However, example doctests should only be in the main docstring if they are a good example, i.e. if they convey useful information about the function. If you simply would like to write a test as a doctest, put it in a separate test file and write a no-op function whose only purpose is its docstring.
Note, however, that in a file named test_X, functions whose only test is their docstring (as a doctest) and which have no test functionality of their own, should be called doctest_foo instead of test_foo, otherwise they get double-counted (the empty function call is counted as a test, which just inflates tests numbers artificially). This restriction does not apply to functions in files with other names, due to how Nose discovers tests.
You can use IPython examples in your docstrings. Those can make full use of IPython functionality (magics, variable substitution, etc), but be careful to keep them generic enough that they run identically on all Operating Systems.
The prompts in your doctests can be either of the plain Python >>> variety or In [1]: IPython style. Since this is the IPython system, after all, we encourage you to use IPython prompts throughout, unless you are illustrating a specific aspect of the normal prompts (such as the %doctest_mode magic).
If a test isn’t safe to run inside the main nose process (e.g. because it loads a GUI toolkit), consider running it in a subprocess and capturing its output for evaluation and test decision later. Here is an example of how to do it, by relying on the builtin _ip object that contains the public IPython api as defined in IPython.ipapi:
def test_obj_del():
"""Test that object's __del__ methods are called on exit."""
test_dir = os.path.dirname(__file__)
del_file = os.path.join(test_dir,'obj_del.py')
out = _ip.IP.getoutput('ipython %s' % del_file)
nt.assert_equals(out,'object A deleted')
If a doctest contains input whose output you don’t want to verify identically via doctest (random output, an object id, etc), you can mark a docstring with #random. All of these test will have their code executed but no output checking will be done:
>>> 1+3
junk goes here... # random
>>> 1+2
again, anything goes #random
if multiline, the random mark is only needed once.
>>> 1+2
You can also put the random marker at the end:
# random
>>> 1+2
# random
.. or at the beginning.
In a case where you want an entire docstring to be executed but not verified (this only serves to check that the code runs without crashing, so it should be used very sparingly), you can put # all-random in the docstring.
Most of the release process is automated by the release script in the tools directory. This is just a handy reminder for the release manager.
There are no definite plans for porting of IPython to python 3. The major issue is the dependency on twisted framework for the networking/threading stuff. It is possible that it the traditional IPython interactive console could be ported more easily since it has no such dependency. Here are a few things that will need to be considered when doing such a port especially if we want to have a codebase that works directly on both 2.x and 3.x.
- The syntax for exceptions changed (PEP 3110). The old except exc, var changed to except exc as var. At last count there was 78 occurences of this usage in the codebase. This is a particularly problematic issue, because it’s not easy to implement it in a 2.5-compatible way.
Because it is quite difficult to support simultaneously Python 2.5 and 3.x, we will likely at some point put out a release that requires strictly 2.6 and abandons 2.5 compatibility. This will then allow us to port the code to using print() as a function, except exc as var syntax, etc. But as of version 0.11 at least, we will retain Python 2.5 compatibility.
| [Bazaar] | Bazaar. http://bazaar-vcs.org/ |
| [Launchpad] | Launchpad. http://www.launchpad.net/ipython |
| [reStructuredText] | reStructuredText. http://docutils.sourceforge.net/rst.html |
| [Sphinx] | Sphinx. http://sphinx.pocoo.org/ |
| [Nose] | Nose: a discovery based unittest extension. http://code.google.com/p/python-nose/ |