Posted on by & filed under Content - Highlights and Reviews, Programming & Development.

codeA guest post by Anton I. Sipos, who has been programming computers since they were 8 bits old. He is a happy participant in the open source community, including contributions to IPython itself. His open source contributions can be found at, short quips at @aisipos, and longer musings at Email him at

As programmers, we often find ourselves needing to use multiple tools to get the job done. For many programming jobs, keeping a shell open for command line work is essential. As Python programmers, we also usually have a Python interpreter open. Switching between the two of these tools is a very common workflow. Using the IPython interpreter instead of the standard Python interpreter, however, allows us to rather seamlessly blend the two of these tools into one environment for many (but not all) cases. IPython’s own documentation makes the case for using it as a shell. In this post I’ll take you on a brief tour of IPython’s shell features and how to use them effectively. We’ll also see where these features aren’t appropriate, and thus why we can’t replace our shell entirely.

The Basics

The Python language itself provides methods of calling out to shell commands. These include the os.system function provides and the newer os.subprocess. These methods are appropriate for calling out to shell commands from within your own Python programs. For use at the interactive prompt, IPython provides an easier way.

The easiest way to call shell from IPython is to prepend an exclamation point to a shell command, like this:

IPython executes commands using the default system shell, in my case zsh. By default, IPython just prints the output of the command. If you need to capture the output of the command as well, you can do this by assigning the output to variable, like so:

Note how the result looks to be a list of the separate lines from stdout. This is actually an IPython string list (SList).

SList methods

SList instances can be used like a regular list, but they provide several methods that are useful when working with shell output. The main properties available in an SList instance are:

  • .s returns the elements joined together by spaces. This is useful for building command lines that take many arguments in a single invocation.
  • .n returns the elements joined together by a newline. Use this when you need
    the original output unmodified.
  • .p returns the elements as path objects, if they are filenames.

In addition, SList instances support grep() and fields() methods. Let’s use the fields() method on our df invocation above:

You can see that the fields method splits the output into each individual whitespace separated field. You can pass indices to fields() to pull out specific fields, and can pass in negative indices in the same way you can for Python lists.

Let’s use what we’ve learned so far for a slightly more involved example. Let’s use the dig command to find the local DNS address records for, and use the IPython SList functionality to pull out the IP addresses to a single string.

Using Python expressions in shell command lines

We’ve already seen how to capture shell output into Python. But how do you go the other way, placing Python output into shell commands? The IPython syntax for doing this is by placing {} blocks that contain python expressions into a shell command. Let’s try a simple example:

You can put any Python expression inside the {} block. Be aware though that if any exception is raised in this block, IPython won’t raise the exception and the entire {} block will just be passed unchanged to the shell. We can demonstrate this with a simple example:

Let’s use the {} ‘python escape’ mechanism in a slightly more complicated example, to do a reverse DNS lookup using the host command of all local servers that we found above in the variable google_a:

The IPython magic system

The exclamation point syntax is part of IPython magic
commands. These are special functions that provide helpful abilities to the interpreter. IPython ships many of these by default, and you can write your own if you need custom functionality. You can see a list of the ones available in your interpreter by running the %lsmagic command. Note that depending on which IPython interpreter you are running (console, qtconsole, notebook), not all magics are available. In particular, using cell level magics requires running the IPython notebook. For instance, on my system running the IPython notebook I have these magics available to me:

Shell-like features

Although IPython isn’t strictly a shell, it does have some shell like features. Many of these are implemented as magic commands as shown above. Let’s examine a few of these.

Much like most shells, you can customize the IPython prompt to show information of your own liking, including for instance the current working directory. See IPython’s own docs on prompt customization for how to do this.

Although you can use os.chdir in a regular Python interpreter to change directories, IPython provides the %cd magic to do this more easily. Note that IPython provides tab completion for filenames, which works not just for the %cd command, but filenames inside strings as well. There
are several other directory related magic functions: %dhist, %pushd,
%popd, %dirs. See IPython’s docs on how to use these.

Similarly, the plain Python interpreter can set or examine environment variables using os.environ. In IPython you can use the more convenient %env magic to both set and examine environment variables. For instance, you could set your AWS EC2 region for shell commands by issuing: %env EC2_REGION=us-west-1.

If you need help on a particular shell command, you can use the %man magic to format a shell man page to a the help region at the bottom of the page. This is handy for getting documentation right within IPython, removing the need to switch to the shell for one more use case. Note however than %man is only available in the IPython notebook and qtconsole, it is not available in the regular console version.

IPython shell limitations

IPython shell integration is effective for shell commands that produce unformatted, non-interactive output. Any command that requires user input, or produces continuously updating output (for instance, top), isn’t suitable for this method. For this reason, we still will have to keep our shell open, and thus keep more than one tool in our toolbox.


Sometimes using shell commands is the best fit for a particular problem. In other cases Python code is a better fit. You should always use the best tool for the job. But by using IPython, you can often use both tools conveniently from one single environment, and even intermix the two in one single invocation. I hope this overview helps you better use the power of shell from your IPython sessions.

Look below for some great Python and IPython books from Safari Books Online.

Not a subscriber? Sign up for a free trial.

Safari Books Online has the content you need

Python for Data Analysis is packed with practical cases studies that show you how to effectively solve a broad set of data analysis problems, using several Python libraries.
Mining the Social Web, 2nd Edition combines popular and useful social web data with analysis techniques and visualization to help you find the needles in the social haystack that you’ve been looking for—as well as many you probably didn’t even know existed.
Think Python takes you through the Python language one step at a time, beginning with basic programming concepts before moving on to functions, recursion, data structures, and object-oriented design.
Python Cookbook, 3rd Edition is your ticket if you need help writing programs in Python 3, or want to update older Python 2 code. Packed with practical recipes written and tested with Python 3.3, this unique cookbook is for experienced Python programmers who want to focus on modern tools and idioms.

Tags: Anton I. Sipos, IPython, os.subprocess, os.system, Shell commands, string list,

Comments are closed.