You are previewing Python for Unix and Linux System Administration.

Python for Unix and Linux System Administration

Cover of Python for Unix and Linux System Administration by Noah Gift... Published by O'Reilly Media, Inc.
  1. Copyright
  2. Foreword
  3. Preface
    1. P2.1. Conventions Used in This Book
    2. P2.2. Using Code Examples
    3. P2.3. Safari® Books Online
    4. P2.4. How to Contact Us
    5. P2.5. Acknowledgments
      1. P2.5.1. Noah’s Acknowledgments
      2. P2.5.2. Jeremy’s Acknowledgments
  4. 1. Introduction
    1. 1.1. Why Python?
    2. 1.2. Motivation
    3. 1.3. The Basics
    4. 1.4. Executing Statements in Python
      1. 1.4.1. Summary
    5. 1.5. Using Functions in Python
    6. 1.6. Reusing Code with the Import Statement
  5. 2. IPython
    1. 2.1. Installing IPython
    2. 2.2. Basic Concepts
      1. 2.2.1. Interacting with IPython
      2. 2.2.2. Tab Completion
      3. 2.2.3. Magic Edit
      4. 2.2.4. Configuring IPython
    3. 2.3. Help with Magic Functions
    4. 2.4. Unix Shell
      1. 2.4.1. alias
      2. 2.4.2. Shell Execute
      3. 2.4.3. rehash
      4. 2.4.4. rehashx
      5. 2.4.5. cd
      6. 2.4.6. bookmark
      7. 2.4.7. dhist
      8. 2.4.8. pwd
      9. 2.4.9. Variable Expansion
      10. 2.4.10. String Processing
      11. 2.4.11. sh Profile
    5. 2.5. Information Gathering
      1. 2.5.1. page
      2. 2.5.2. pdef
      3. 2.5.3. pdoc
      4. 2.5.4. pfile
      5. 2.5.5. pinfo
      6. 2.5.6. psource
      7. 2.5.7. psearch
      8. 2.5.8. who
      9. 2.5.9. who_ls
      10. 2.5.10. whos
      11. 2.5.11. History
    6. 2.6. Automation and Shortcuts
      1. 2.6.1. alias
      2. 2.6.2. macro
      3. 2.6.3. store
      4. 2.6.4. reset
      5. 2.6.5. run
      6. 2.6.6. save
      7. 2.6.7. rep
    7. 2.7. Summary
  6. 3. Text
    1. 3.1. Python Built-ins and Modules
      1. 3.1.1. str
      2. 3.1.2. re
      3. 3.1.3. Apache Config File Hacking
      4. 3.1.4. Working with Files
      5. 3.1.5. Standard Input and Output
      6. 3.1.6. StringIO
      7. 3.1.7. urllib
    2. 3.2. Log Parsing
    3. 3.3. ElementTree
    4. 3.4. Summary
  7. 4. Documentation and Reporting
    1. 4.1. Automated Information Gathering
      1. 4.1.1. Receiving Email
    2. 4.2. Manual Information Gathering
    3. 4.3. Information Formatting
      1. 4.3.1. Graphical Images
      2. 4.3.2. PDFs
    4. 4.4. Information Distribution
      1. 4.4.1. Sending Email
      2. 4.4.2. Trac
    5. 4.5. Summary
  8. 5. Networking
    1. 5.1. Network Clients
      1. 5.1.1. socket
      2. 5.1.2. httplib
      3. 5.1.3. ftplib
      4. 5.1.4. urllib
      5. 5.1.5. urllib2
    2. 5.2. Remote Procedure Call Facilities
      1. 5.2.1. XML-RPC
      2. 5.2.2. Pyro
    3. 5.3. SSH
    4. 5.4. Twisted
    5. 5.5. Scapy
    6. 5.6. Creating Scripts with Scapy
  9. 6. Data
    1. 6.1. Introduction
    2. 6.2. Using the OS Module to Interact with Data
    3. 6.3. Copying, Moving, Renaming, and Deleting Data
    4. 6.4. Working with Paths, Directories, and Files
    5. 6.5. Comparing Data
      1. 6.5.1. Using the filecmp Module
    6. 6.6. Merging Data
      1. 6.6.1. MD5 Checksum Comparisons
    7. 6.7. Pattern Matching Files and Directories
    8. 6.8. Wrapping Up rsync
    9. 6.9. Metadata: Data About Data
    10. 6.10. Archiving, Compressing, Imaging, and Restoring
    11. 6.11. Using tarfile Module to Create TAR Archives
    12. 6.12. Using a tarfile Module to Examine the Contents of TAR Files
  10. 7. SNMP
    1. 7.1. Introduction
    2. 7.2. Brief Introduction to SNMP
      1. 7.2.1. SNMP Overview
      2. 7.2.2. SNMP Installation and Configuration
    3. 7.3. IPython and Net-SNMP
    4. 7.4. Discovering a Data Center
    5. 7.5. Retrieving Multiple-Values with Net-SNMP
      1. 7.5.1. Finding Memory
    6. 7.6. Creating Hybrid SNMP Tools
    7. 7.7. Extending Net-SNMP
    8. 7.8. SNMP Device Control
    9. 7.9. Enterprise SNMP Integration with Zenoss
      1. 7.9.1. Zenoss API
  11. 8. OS Soup
    1. 8.1. Introduction
    2. 8.2. Cross-Platform Unix Programming in Python
      1. 8.2.1. Using SSH Keys, NFS-Mounted Source Directory, and Cross-Platform Python to Manage Systems
      2. 8.2.2. Creating a Cross-Platform, Systems Management Tool
      3. 8.2.3. Creating a Cross-Platform Build Network
    3. 8.3. PyInotify
    4. 8.4. OS X
      1. 8.4.1. Scripting DSCL or Directory Services Utility
      2. 8.4.2. OS X Scripting APIs
      3. 8.4.3. Automatically Re-Imaging Machines
      4. 8.4.4. Managing Plist Files from Python
    5. 8.5. Red Hat Linux Systems Administration
    6. 8.6. Ubuntu Administration
    7. 8.7. Solaris Systems Administration
    8. 8.8. Virtualization
      1. 8.8.1. VMware
    9. 8.9. Cloud Computing
      1. 8.9.1. Amazon Web Services with Boto
      2. 8.9.2. Google App Engine
    10. 8.10. Using Zenoss to Manage Windows Servers from Linux
  12. 9. Package Management
    1. 9.1. Introduction
    2. 9.2. Setuptools and Python Eggs
    3. 9.3. Using easy_install
    4. 9.4. easy_install Advanced Features
      1. 9.4.1. Search for Packages on a Web Page
      2. 9.4.2. Install Source Distribution from URL
      3. 9.4.3. Install Egg Located on Local or Network Filesystem
      4. 9.4.4. Upgrading Packages
      5. 9.4.5. Install an Unpacked Source Distribution in Current Working Directory
      6. 9.4.6. Extract Source Distribution to Specified Directory
      7. 9.4.7. Change Active Version of Package
      8. 9.4.8. Changing Standalone .py File into egg
      9. 9.4.9. Authenticating to a Password Protected Site
      10. 9.4.10. Using Configuration Files
      11. 9.4.11. Easy Install Advanced Features Summary
    5. 9.5. Creating Eggs
    6. 9.6. Entry Points and Console Scripts
    7. 9.7. Registering a Package with the Python Package Index
      1. 9.7.1. Where Can I Learn More About …
    8. 9.8. Distutils
    9. 9.9. Buildout
    10. 9.10. Using Buildout
    11. 9.11. Developing with Buildout
    12. 9.12. virtualenv
      1. 9.12.1. Creating a Custom Bootstrapped Virtual Environment
    13. 9.13. EPM Package Manager
      1. 9.13.1. EPM Package Manager Requirements and Installation
      2. 9.13.2. Creating a Hello World Command-Line Tool to Distribute
      3. 9.13.3. Creating Platform-Specific Packages with EPM
      4. 9.13.4. Making the Package
      5. 9.13.5. EPM Summary: It Really Is That Easy
  13. 10. Processes and Concurrency
    1. 10.1. Introduction
    2. 10.2. Subprocess
      1. 10.2.1. Using Return Codes with Subprocess
    3. 10.3. Using Supervisor to Manage Processes
    4. 10.4. Using Screen to Manage Processes
    5. 10.5. Threads in Python
      1. 10.5.1. Timed Delay of Threads with threading.Timer
      2. 10.5.2. Threaded Event Handler
    6. 10.6. Processes
    7. 10.7. Processing Module
    8. 10.8. Scheduling Python Processes
    9. 10.9. daemonizer
    10. 10.10. Summary
  14. 11. Building GUIs
    1. 11.1. GUI Building Theory
    2. 11.2. Building a Simple PyGTK App
    3. 11.3. Building an Apache Log Viewer Using PyGTK
    4. 11.4. Building an Apache Log Viewer Using Curses
    5. 11.5. Web Applications
    6. 11.6. Django
      1. 11.6.1. Apache Log Viewer Application
      2. 11.6.2. Simple Database Application
    7. 11.7. Conclusion
  15. 12. Data Persistence
    1. 12.1. Simple Serialization
      1. 12.1.1. Pickle
      2. 12.1.2. cPickle
      3. 12.1.3. shelve
      4. 12.1.4. YAML
      5. 12.1.5. ZODB
    2. 12.2. Relational Serialization
      1. 12.2.1. SQLite
      2. 12.2.2. Storm ORM
      3. 12.2.3. SQLAlchemy ORM
    3. 12.3. Summary
  16. 13. Command Line
    1. 13.1. Introduction
    2. 13.2. Basic Standard Input Usage
    3. 13.3. Introduction to Optparse
    4. 13.4. Simple Optparse Usage Patterns
      1. 13.4.1. No Options Usage Pattern
      2. 13.4.2. True/False Usage Pattern
      3. 13.4.3. Counting Options Usage Pattern
      4. 13.4.4. Choices Usage Pattern
      5. 13.4.5. Option with Multiple Arguments Usage Pattern
    5. 13.5. Unix Mashups: Integrating Shell Commands into Python Command-Line Tools
      1. 13.5.1. Kudzu Usage Pattern: Wrapping a Tool in Python
      2. 13.5.2. Hybrid Kudzu Design Pattern: Wrapping a Tool in Python, and Then Changing the Behavior
      3. 13.5.3. Hybrid Kudzu Design Pattern: Wrapping a Unix Tool in Python to Spawn Processes
    6. 13.6. Integrating Configuration Files
    7. 13.7. Summary
  17. 14. Pragmatic Examples
    1. 14.1. Managing DNS with Python
    2. 14.2. Using LDAP with OpenLDAP, Active Directory, and More with Python
      1. 14.2.1. Importing an LDIF File
    3. 14.3. Apache Log Reporting
    4. 14.4. FTP Mirror
  18. A. Callbacks
  19. Index
  20. B. Colophon

Chapter 1. Introduction

Why Python?

If you are a system administrator, it is likely that you have encountered Perl, Bash, ksh, or some other scripting language. You may have even used one or more yourself. Scripting languages are often used to do repetitive, tedious work at a rate and with an accuracy that far surpass what you could accomplish without them. All languages are tools. They are simply a means to get work done. They have value only insofar as they help you get your job done better. We believe that Python is a valuable tool, specifically because it enables you to get your work done efficiently.

So is Python better than Perl, Bash, Ruby, or any other language? It’s really difficult to put that sort of qualitative label on a programming language, since the tool is so closely tied to the thought process of the programmer who is using it. Programming is a subjective, deeply personal activity. For the language to be excellent, it must fit the person using it. So we’re not going to argue that Python is better, but we will explain the reasons that we believe Python can be an excellent choice. We’ll also explain why it is a great fit for performing sysadmin tasks.

The first reason that we think that Python is excellent is that it is easy to learn. If a language can’t help you become productive pretty quickly, the lure of that language is severely diminished. Why would you want to spend weeks or months studying a language before you are able to write a program that does something useful? This is especially the case for sysadmins. If you are a sysadmin, your work can pile up faster than you can unpile it. With Python, you can start writing useful scripts literally in hours rather than in days or weeks. If you can’t learn a language quickly enough to start writing scripts with it almost immediately, you should strongly question whether you should be learning it.

However, a language that is easy to learn but doesn’t allow you to do fairly complex tasks isn’t worth much either. So the second reason that we consider Python to be an excellent programming language is that, while it lets you start simply, it also allows you to perform tasks that are as complex as you can imagine. Do you need to read through a logfile line by line and pull out some pretty basic information? Python can handle that. Or do you need to parse through a logfile, extract every piece of information that it provides, compare usage from each IP address in this logfile to usage in each logfile (which are stored in a relational database, by the way) from the past three months, and then store the results to a relational database? Sure, Python can do that as well. Python is being used on some pretty complex problems, such as analysis of genomic sequences, multithreaded web servers, and heavy duty statistical analysis. You may never have to work on anything like that, but it’s nice to know that when you need to do complex things, the language is able to work with you.

Additionally, if you are able to perform complex operations, but the maintainability of your code suffers along the way, that isn’t a good thing. Python doesn’t prevent code maintenance problems, but it does allow you to express complex ideas with simple language constructs. Simplicity is a huge factor in writing code that is easy to maintain later. Python has made it pretty simple for us to go back over our own code and work on it after we haven’t touched it in months. It has also been pretty simple for us to work on code that we haven’t seen before. So the language, that is the language’s syntax and common idioms, are clear and concise and easy to work with over long periods of time.

The next reason we consider Python to be an excellent language is its readability. Python relies on whitespace to determine where code blocks begin and end. The indentation helps your eyes quickly follow the flow of a program. Python also tends to be “word-based.” By that we mean that while Python uses its share of special characters, features are often implemented as keywords or with libraries. The emphasis on words rather than special characters helps the reading and comprehension of code.

Now that we’ve outlined a few of Python’s benefits, we’ll show some comparisons of code examples in Python, Perl, and Bash. Along the way, we’ll also look at a few more of Python’s benefits. Here is a simple example, in Bash, of showing all the combinations of 1, 2 and a, b:


for a in 1 2; do
    for b in a b; do
        echo "$a $b"

And here is a comparable piece of Perl:


foreach $a ('1', '2') {
    foreach $b ('a', 'b') {
        print "$a $b\n";

This is a pretty simple nested loop. Let’s compare these looping mechanisms with a for loop in Python:

#!/usr/bin/env python

for a in [1, 2]:
    for b in ['a', 'b']:
        print a, b

Next, we’ll demonstrate using conditionals in Bash, Perl, and Python. We have a simple if/else condition check here. We’re just checking to see whether a certain file path is a directory:


if [ -d "/tmp" ] ; then
    echo "/tmp is a directory"
    echo "/tmp is not a directory"

Here is the Perl equivalent of the same script:


if (-d "/tmp") {
    print "/tmp is a directory\n";
else {
    print "/tmp is not a directory\n";

And here is the Python equivalent of the script:

#!/usr/bin/env python

import os

if os.path.isdir("/tmp"):
    print "/tmp is a directory"
    print "/tmp is not a directory"

Another point in favor of Python’s excellence is its simple support for object-oriented programming (OOP). And, actually, the converse of that is that you don’t have to do OOP if you don’t want to. But if you do, it’s dead simple in Python. OOP allows you to easily and cleanly break problems apart and bundle pieces of functionality together into single “things” or “objects.” Bash doesn’t support OOP, but both Perl and Python do. Here is a module in Perl that defines a class:

package Server;
use strict;

sub new {
    my $class = shift;
    my $self  = {};
    $self->{IP} = shift;
    $self->{HOSTNAME} = shift;
    return $self;

sub set_ip {
    my $self = shift;
    $self->{IP} = shift;
    return $self->{IP};

sub set_hostname {
    my $self = shift;
    $self->{HOSTNAME} = shift;
    return $self->{HOSTNAME};

sub ping {
    my $self = shift;
    my $external_ip = shift;
    my $self_ip = $self->{IP};
    my $self_host = $self->{HOSTNAME};
    print "Pinging $external_ip from $self_ip ($self_host)\n";
    return 0;


And here is a piece of code that uses it:


use Server;

$server = Server->new('', 'grumbly');

The code that makes use of the OO module is straightforward and simple. The OO module may take a bit more mental parsing if you’re not familiar with OOP or with the way that Perl tackles OOP.

A comparable Python class and use of the class looks something like this:

#!/usr/bin/env python

class Server(object):
    def __init__(self, ip, hostname):
        self.ip = ip
        self.hostname = hostname
    def set_ip(self, ip):
        self.ip = ip
    def set_hostname(self, hostname):
        self.hostname = hostname
    def ping(self, ip_addr):
        print "Pinging %s from %s (%s)" % (ip_addr, self.ip, self.hostname)

if __name__ == '__main__':
    server = Server('', 'bumbly')'')

Both the Perl and Python examples demonstrate some of the fundamental pieces of OOP. The two examples together display the different flavors that each respective language provides while reaching toward its respective goals. They both do the same thing, but are different from one another. So, if you want to use OOP, Python supports it. And it’s quite simple and clear to incorporate it into your programming.

Another element of Python’s excellence comes not from the language itself, but from the community. In the Python community, there is much consensus about the way to accomplish certain tasks and the idioms that you should (and should not) use. While the language itself may support certain phrasings for accomplishing something, the consensus of the community may steer you away from that phrasing. For example, from module import * at the top of a module is valid Python. However, the community frowns upon this and recommends that you use either: import module or: from module import resource. Importing all the contents of a module into another module’s namespace can cause serious annoyance when you try to figure out how a module works, what functions it is calling, and where those functions come from. This particular convention will help you write code that is clearer and will allow people who work on your code after you to have a more pleasant maintenance experience. Following common conventions for writing your code will put you on the path of best practices. We consider this a good thing.

The Python Standard Library is another excellent attribute of Python. If you ever hear the phrase “batteries included” in reference to Python, it simply means that the standard library allows you to perform all sorts of tasks without having to go elsewhere for modules to help you get it done. For example, though it isn’t built-in to the language directly, Python includes regular expression functionality; sockets; threads; date/time functionality; XML parsers; config file parser; file and directory functionality; data persistence; unit test capabilities; and http, ftp, imap, smpt, and nntp client libraries; and much more. So once Python is installed, modules to support all of these functions will be imported by your scripts as they are needed. You have all the functionality we just listed here. It is impressive that all of this comes with Python without requiring anything else. All of this functionality will help you out immensely as you write Python programs to do work for you.

Easy access to numerous third-party packages is another real advantage of Python. In addition to the many libraries in the Python Standard Library, there are a number of libraries and utilities that are easily accessible on the internet that you can install with a single shell command. The Python Package Index, PyPI (, is a place where anyone who has written a Python package can upload it for others to use. At the time we are writing this book, there are over 3,800 packages available for download and use. Packages include IPython, which we cover in the following chapter; Storm (an object-relational mapper, which we cover in Chapter 12, Data Persistence); and Twisted, a network framework, which we cover in Chapter 5, Networking—just to name 3 of the over 3,800 packages. Once you start using PyPI, you’ll find it nearly indispensible for finding and installing useful packages.

Many of the benefits that we see in Python stem from the central philosophy of Python. When you type import this at a Python prompt, you will see The Zen of Python by Tim Peters. Here it is:

In [1]: import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

This statement isn’t a dogmatic imperative that is strictly enforced at all levels of development of the language, but the spirit of it seems to permeate much of what happens in and with the language. And we have found this spirit to be a beautiful thing. This is perhaps the essence of why we choose to use Python day after day. This philosophy resonates within us as what we want and expect from a language. And if this resonates with you, then Python is probably a good choice for you as well.


If you justpicked up this book in a bookstore or are reading an introduction online somewhere, you may be asking yourself, how hard it is going to be to learn Python and if it is even worth it. Although Python is catching on like wildfire, there are many sysadmins who have been exposed to Bash and Perl only. If you find yourself in this category, you should take comfort in knowing that Python is very easy to learn. In fact, although it is a matter of opinion, Python is considered by many to be the easiest language to learn and teach, period!

If you already know Python, or are a programming guru in another language, you will probably be able to jump right into any of the following chapters without reading this intro and immediately start being productive using our examples. We made a concerted effort to create examples that will actually help you get your job done. There are examples of ways to discover and monitor subnets automatically with SNMP, to convert to an interactive Python shell called IPython, to build data processing pipelines, to write custom metadata management tools with object-relational mappers, to perform network programming, to write command-line tools, and much more.

If you are coming from a shell programming/scripting background, though, don’t worry at all. You, too, can learn Python quite easily. You need only motivation, curiosity, and determination, the same factors that led you to pick up this book and look at the introduction in the first place.

We sense there are still a few skeptics out there. Maybe some of the things you have heard about programming have scared you. One common, and horribly false, misconception is that only some people can learn to program, and they are a mysterious and elite few. The frank truth is that anyone can learn how to program. A second, equally false, misconception is that earning a computer science degree is the only way a person can truly become a software engineer. But some of the most prolific software developers do not have engineering degrees. There are people with philosophy, journalism, nutritional science, and English degrees who are competent Python programmers. Having a degree in computer science is not a requirement to learn Python, although it certainly doesn’t hurt.

Another funny, and false, misconception is that you must have started to program in your teenage years, or you will never learn to program. While this makes people who were lucky enough to have someone in their life that encouraged them to program at a young age feel good, it is another myth. It is very helpful to have started learning programming at a young age, but age is not a requirement to learn Python. Learning Python is most certainly not a “young person’s game,” as we have heard some people say. There are countless cases of developers who learned to program in their late 20s, 30s, 40s, and onward.

If you have gotten this far, we should point out that you, the reader, have an advantage many people do not. If you decided to pick up a book on Python for Unix and Linux system administration, then you most likely know something about how to execute commands from a shell. This is a tremendous advantage to learning to become a Python programmer. Having an understanding of the way to execute commands from a terminal is all that is required for this introduction to Python. If you truly believe you will learn how to program with Python, then read the next section immediately. If you don’t believe it yet, then reread this section again, and convince yourself it really is just a matter of getting your mind to understand you do have the power to learn how to program in Python. It is really that simple; if you make this decision, it will change your life.

The Basics

This introduction to Python is going to be very different from any other one we’ve seen, as it will use an interactive shell called IPython and a regular Bash shell. You will need to open two terminal windows, one with IPython and one with Bash. In every example, we will compare what we do in Python with a Bash example. The first steps are to download the correct version of IPython for your platform and install it. You can get a copy at If for some reason, you can’t get IPython to install you can also just use a regular Python shell. You can also download a copy of the virtual machine that includes all of the software for the book, as we have a copy of IPython preconfigured and ready to run. You just need to type in ipython, and you will get a prompt.

Once you have installed IPython and have an IPython shell prompt, it should look something like this:

[ngift@Macintosh-7][H:10679][J:0]# ipython
Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) 
Type "copyright", "credits" or "license" for more information.
IPython 0.8.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.
In [1]:

In your Python terminal, type in the following:

In [1]: print "I can program in Python"
I can program in Python

Executing Statements in Python

If you spend a lot of your day typing commands into a terminal, then you are used to executing statements and, perhaps, redirecting the output to a file or to another Unix command. Let’s look at the way we would execute a command in Bash and then compare that to the way it works in Python. In the Bash terminal, type the following:

[ngift@Macintosh-7][H:10701][J:0]# ls -l /tmp/
total 0
-rw-r--r--  1 ngift  wheel  0 Apr  7 00:26 file.txt

Example 1-1. Python wrapper for ls command

#!/usr/bin/env python
#Python wrapper for the ls command
import subprocess["ls","-l"])

Now if you run this script, you will get the exact same output that you would get if you ran ls -ls from the command line:

[ngift@Macintosh-7][H:10746][J:0]# ./ 
total 8
-rwxr-xr-x  1 ngift  staff  115 Apr  7 12:57

Example 1-2. System information script—Python

#!/usr/bin/env python
#A System Information Gathering Script
import subprocess
#Command 1
uname = “uname”
uname_arg = “-a”
print "Gathering system information with %s command:\n" % uname[uname, uname_arg])
#Command 2
diskspace = "df"
diskspace_arg = "-h"
print "Gathering diskspace information %s command:\n" % diskspace[diskspace, diskspace_arg])

Example 1-3. System information script—Bash

#!/usr/bin/env bash
#A System Information Gathering Script
#Command 1
UNAME="uname -a"
printf “Gathering system information with the $UNAME command: \n\n"
#Command 2
printf "Gathering diskspace information with the $DISKSPACE command: \n\n"

If we look at both of the scripts, we see that they look a lot a like. And if we run them, we see that the output of each is identical. One quick note though: splitting the command from the argument is completely optional using You can also use this syntax:"df -h", shell=True)

As we mentioned earlier, importing a module like subprocess is just importing a file that contains code you can use. You can create your own module or file and reuse code you have written in the same way you import subprocess. Importing is not magic at all, it is just a file with some code in it. One of the nice things about the IPython shell that you have open is its ability to inspect inside modules and files, and see the attributes that are available inside them. In Unix terms, this is a lot like running the ls command inside of /usr/bin. If you happen to be on a new system such as Ubuntu or Solaris, and you are used to Red Hat, you might do an ls of /usr/bin to see if tools such as wget, curl, or lynx are available. If you want to use a tool you find inside /usr/bin, you would simply type /usr/bin/wget, for example.

Modules such as subprocess are very similar. With IPython you can use tab complete to look at the tools that are available inside a module. Let’s walk through subprocess using tab complete to look at the attributes available inside of it. Remember, a module is just a file with some code in it. Here is what a tab complete looks like with the subprocess module in IPython:

In [12]: subprocess.
subprocess.CalledProcessError  subprocess.__hash__  
subprocess.MAXFD               subprocess.__init__            subprocess.check_call
subprocess.PIPE                subprocess.__name__            subprocess.errno
subprocess.Popen               subprocess.__new__             subprocess.fcntl
subprocess.STDOUT              subprocess.__reduce__          subprocess.list2cmdline
subprocess.__all__             subprocess.__reduce_ex__       subprocess.mswindows
subprocess.__builtins__        subprocess.__repr__            subprocess.os
subprocess.__class__           subprocess.__setattr__         subprocess.pickle
subprocess.__delattr__         subprocess.__str__   
subprocess.__dict__            subprocess._active             subprocess.sys
subprocess.__doc__             subprocess._cleanup            subprocess.traceback
subprocess.__file__            subprocess._demo_posix         subprocess.types
subprocess.__getattribute__    subprocess._demo_windows

Think of the special question mark syntax as a manpage query. If you want to know how a tool works in Unix, simply type:

man name_of_tool

When we look at this documentation, “Docstring” is the official term, we see an example of the way to use and a description of what it does.


You now have enough information to call yourself a Python programmer. You know how to write a simple Python script, how to translate simple scripts from Bash and call them with Python, and, finally, how to find documentation about new modules and attributes. In the next section, you’ll see how to better organize these flat sequences of commands into functions.

Using Functions in Python

In the previous section we went through executing statements one after another, which is pretty useful, because it means we were able to automate something that we would normally have to do manually. The next step to automating our code execution is to create functions. If you are not already familiar with writing functions in Bash or another language, then one way to think about functions is as miniscripts. A function allows you to create blocks of statements that get called in groups that live inside of the function. This is quite a bit like the Bash script we wrote in which there were two commands enclosed in a script. One of the differences between a Bash script and a function is that you can include many function scripts. Ultimately, you can have multiple functions that group statements together in a script, and then that group of statements can be called to run a miniprogram at the proper time in your script.

At this point, we need to talk about the topic of whitespace. In Python, a uniform level of indentation must be maintained in nesting code. In another language, like Bash, when you define a function you put brackets around the code inside of a function. With Python, you must indent the code inside of the bracket. This can trip up newcomers to the language, at first, but after a while it will grow on you, and you will realize that this encourages readable code. If you have trouble getting any of these examples to work interactively, make sure you refer to the actual source code to see the proper indentation level. The most common practice is to set a tab to indent exactly four spaces.

Let’s take a look at how this works in Python and Bash. If you still have the IPython shell open, you don’t need to create a Python script file, although you can if you like. Just type the following into the interactive IPython prompt:

In [1]: def pyfunc():
...:     print "Hello function"
In [2]: pyfunc
Out[2]: <function pyfunc at 0x2d5070>
In [3]: pyfunc()
Hello function
In [4]: for i in range(5):
...:     pyfunc()
Hello function
Hello function
Hello function
Hello function
Hello function

We can do the same thing in a live Bash shell as well. Here is one way:

bash-3.2$ function shfunc()
> {
>     printf "Hello function\n"
> }
bash-3.2$ for (( i=0 ; i < 5 ; i++))
> do
>     shfunc
> done
Hello function
Hello function
Hello function
Hello function
Hello function

In the Bash example, we created a simple function shfunc, and then called it five times, just like we did with the Python function earlier. One thing to notice is that the Bash example requires more “baggage” to do the same thing that Python does. Notice the difference between the Bash for loop and the Python for loop. If this is your first exposure to a function in Bash or Python, you should make some other functions in your IPython window before you continue.

Functions are not magic, and writing multiple functions interactively is a great way to take away the mystery if this is your first experience with them. Here are a couple of examples of simple functions:

In [1]: def print_many():
...:     print "Hello function"
...:     print "Hi again function"
...:     print "Sick of me yet"
In [2]: print_many()
Hello function
Hi again function
Sick of me yet

Now we have a few silly examples under our belt, in addition to the silly examples that you tried out on your own as well, right? So we can go back to the script we wrote that prints system information and convert those statements into functions. See Example 1-4, “Converted Python system info script:”.

Example 1-4. Converted Python system info script:

#!/usr/bin/env python
#A System Information Gathering Script
import subprocess

#Command 1
def uname_func():

    uname = "uname"
    uname_arg = "-a"
    print "Gathering system information with %s command:\n" % uname[uname, uname_arg])

#Command 2
def disk_func():

    diskspace = "df"
    diskspace_arg = "-h"
    print "Gathering diskspace information %s command:\n" % diskspace[diskspace, diskspace_arg])

#Main function that call other functions
def main():


Given our experiments with functions, this converted example of our previous script that we simply placed these statements inside functions and then used the main function to call them all at once. If you are not familiar with this style, you might not have known that it is common to create several functions inside a script and then call them all with one main function. One of many reasons for this is that if you decide to reuse this script for another program, you can either call the functions independently or together with the main method. The key is that you decide after the module is imported.

When there is no control flow, or main function, then all of the code gets executed immediately when it is imported. This may be OK for a one-off script, but if you plan to create reusable tools, and you should, then it is a good practice to create functions that encapsulate specific actions, and then have a main function that executes the whole program.

For comparison’s sake, let’s convert our previous Bash system information script to use functions as well. See Example 1-5, “Converted Bash system info script:”.

Example 1-5. Converted Bash system info script:

#!/usr/bin/env bash
#A System Information Gathering Script

#Command 1
function uname_func ()
    UNAME="uname -a"
    printf "Gathering system information with the $UNAME command: \n\n"
#Command 2
function disk_func ()
    DISKSPACE="df -h"
    printf "Gathering diskspace information with the $DISKSPACE command: \n\n"

function main ()


Looking at our Bash example, you can see it has quite a bit in common with its Python cousin. We created two functions and then called those two functions by calling the main function. If this is your first experience with functions, then we would highly recommend that you comment out the main method by placing a pound sign in front of both the Bash and the Python scripts and running them again. You should get absolutely nothing when you run both scripts, because the program should execute, but won’t call the two functions inside.

At this point, you are now a programmer capable of writing simple functions in both Bash and Python. Programmers learn by doing, though, so at this point we highly recommend that you change the system calls in these two Bash and Python programs and make them your own. Give yourself some bonus points if you add several new functions to the script and call them from a main function.

Reusing Code with the Import Statement

One problem with learning something new is that, if it is abstract, like calculus, for example, it is hard to justify caring about it. When was the last time you used the math you learned in high school at the grocery store? In our previous examples, we showed you how to create functions as an alternative to executing shell commands one after another in a script. We also told you that a module is really just a script, or some lines of code in a file. It isn’t anything tricky, but it does need to be arranged in a particular way so that it can be reused in another future program. Here is the point where we show you why you should care. Let’s import the previous system information scripts in both Bash and Python and execute.

Open the IPython and Bash windows if you closed them so that we can demonstrate very quickly why functions are important for code reuse. One of the first scripts we created in Python was a sequence of commands in a file named In Python because a file is a module and vice versa, we can import this script file into IPython. Keep in mind that you never need to specify the .py portion of the file you are importing. In fact if you do this, the import will not work. Here is what it looks like when we do that on Noah’s Macbook Pro laptop:

In [1]: import pysysinfo
Gathering system information with uname command:
Darwin Macintosh-8.local 9.2.2 Darwin Kernel Version 9.2.2: /
  Tue Mar  4 21:17:34 PST 2008; root:xnu-1228.4.31~1/RELEASE_I386 i386
Gathering diskspace information df command:
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2    93Gi   88Gi  4.2Gi    96%    /
devfs          110Ki  110Ki    0Bi   100%    /dev
fdesc          1.0Ki  1.0Ki    0Bi   100%    /dev
map -hosts       0Bi    0Bi    0Bi   100%    /net
map auto_home    0Bi    0Bi    0Bi   100%    /home
/dev/disk1s2   298Gi  105Gi  193Gi    36%    /Volumes/Backup
/dev/disk2s3   466Gi  240Gi  225Gi    52%    /Volumes/EditingDrive

Here is the output from the IPython terminal:

In [3]: import pysysinfo_func
Gathering system information with uname command:
Darwin Macintosh-8.local 9.2.2 Darwin Kernel Version 9.2.2: 
  Tue Mar  4 21:17:34 PST 2008; root:xnu-1228.4.31~1/RELEASE_I386 i386
Gathering diskspace information df command:
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2    93Gi   88Gi  4.1Gi    96%    /
devfs          110Ki  110Ki    0Bi   100%    /dev
fdesc          1.0Ki  1.0Ki    0Bi   100%    /dev
map -hosts       0Bi    0Bi    0Bi   100%    /net
map auto_home    0Bi    0Bi    0Bi   100%    /home
/dev/disk1s2   298Gi  105Gi  193Gi    36%    /Volumes/Backup
/dev/disk2s3   466Gi  240Gi  225Gi    52%    /Volumes/EditingDrive

Now, if we go back to our IPython interpreter and import this new script, we should see this:

In [1]: import pysysinfo_func_2
In [2]: pysysinfo_func_2.
pysysinfo_func_2.__builtins__      pysysinfo_func_2.disk_func
pysysinfo_func_2.__class__         pysysinfo_func_2.main
pysysinfo_func_2.__dict__          pysysinfo_func_2.pyc
pysysinfo_func_2.__doc__           pysysinfo_func_2.subprocess
pysysinfo_func_2.__file__          pysysinfo_func_2.uname_func

In this example, we can ignore anything with double underscores, because these are special methods that are beyond the scope of this introduction. Because IPython is also a regular shell, it picks up the filename and the byte-compiled Python file with the .pyc extension. Once we filter past all of those names, we can see that there is a pysysinfo_func_2.disk_func. Let’s go ahead and call that function:

In [2]: pysysinfo_func_2.disk_func()
Gathering diskspace information df command:
Filesystem      Size   Used  Avail Capacity  Mounted on
/dev/disk0s2    93Gi   89Gi  4.1Gi    96%    /
devfs          111Ki  111Ki    0Bi   100%    /dev
fdesc          1.0Ki  1.0Ki    0Bi   100%    /dev
map -hosts       0Bi    0Bi    0Bi   100%    /net
map auto_home    0Bi    0Bi    0Bi   100%    /home
/dev/disk1s2   298Gi  105Gi  193Gi    36%    /Volumes/Backup
/dev/disk2s3   466Gi  240Gi  225Gi    52%    /Volumes/EditingDrive

Often, the point of writing a reusable module is so that you can take some of the code and use it over and over again in a new script. So practice that by writing another script that uses one of the functions. See Example 1-6, “Reusing code with import: new_pysysinfo”.

Example 1-6. Reusing code with import: new_pysysinfo

#Very short script that reuses pysysinfo_func_2 code
from pysysinfo_func_2 import disk_func
import subprocess

def tmp_space():
    tmp_usage = "du"
    tmp_arg = "-h"
    path = "/tmp"
    print "Space used in /tmp directory"[tmp_usage, tmp_arg, path])

def main():

if __name__ == "__main__":

In this example, not only do we reuse the code we wrote earlier, but we use a special Python syntax that allows us to import the exact function we need. What’s fun about reusing code is that it is possible to make a completely different program just by importing the function from our previous program. Notice that in the main method we mix the function from the other module we created, disk_func(), and the new one we just created in this file.

In this section, we learned the power of code reuse and how simple it really is. In a nutshell, you put a function or two in a file and then, if you also want it to run as script, place that special if__name__ == "__main__": syntax. Later you can either import those functions into IPython or simply reuse them in another script. With the information you have just learned, you are truly dangerous. You could write some pretty sophisticated Python modules and reuse them over and over again to create new tools.

The best content for your career. Discover unlimited learning on demand for around $1/day.