You are previewing Programming Python, 4th Edition.

Programming Python, 4th Edition

Cover of Programming Python, 4th Edition by Mark Lutz Published by O'Reilly Media, Inc.
  1. Programming Python
  2. A Note Regarding Supplemental Files
  3. Preface
    1. “And Now for Something Completely Different…”
    2. About This Book
      1. This Book’s Ecosystem
      2. What This Book Is Not
    3. About This Fourth Edition
      1. Specific Changes in This Edition
    4. What’s Left, Then?
    5. Python 3.X Impacts on This Book
      1. Specific 3.X Changes
      2. Language Versus Library: Unicode
      3. Python 3.1 Limitations: Email, CGI
    6. Using Book Examples
      1. Where to Look for Examples and Updates
      2. Example Portability
      3. Demo Launchers
      4. Code Reuse Policies
    7. Contacting O’Reilly
    8. Conventions Used in This Book
    9. Acknowledgments
  4. I. The Beginning
    1. 1. A Sneak Preview
      1. “Programming Python: The Short Story”
      2. The Task
      3. Step 1: Representing Records
      4. Step 2: Storing Records Persistently
      5. Step 3: Stepping Up to OOP
      6. Step 4: Adding Console Interaction
      7. Step 5: Adding a GUI
      8. Step 6: Adding a Web Interface
      9. The End of the Demo
  5. II. System Programming
    1. 2. System Tools
      1. “The os.path to Knowledge”
      2. System Scripting Overview
      3. Introducing the sys Module
      4. Introducing the os Module
    2. 3. Script Execution Context
      1. “I’d Like to Have an Argument, Please”
      2. Current Working Directory
      3. Command-Line Arguments
      4. Shell Environment Variables
      5. Standard Streams
    3. 4. File and Directory Tools
      1. “Erase Your Hard Drive in Five Easy Steps!”
      2. File Tools
      3. Directory Tools
    4. 5. Parallel System Tools
      1. “Telling the Monkeys What to Do”
      2. Forking Processes
      3. Threads
      4. Program Exits
      5. Interprocess Communication
      6. The multiprocessing Module
      7. Other Ways to Start Programs
      8. A Portable Program-Launch Framework
      9. Other System Tools Coverage
    5. 6. Complete System Programs
      1. “The Greps of Wrath”
      2. A Quick Game of “Find the Biggest Python File”
      3. Splitting and Joining Files
      4. Generating Redirection Web Pages
      5. A Regression Test Script
      6. Copying Directory Trees
      7. Comparing Directory Trees
      8. Searching Directory Trees
      9. Visitor: Walking Directories “++”
      10. Playing Media Files
      11. Automated Program Launchers (External)
  6. III. GUI Programming
    1. 7. Graphical User Interfaces
      1. “Here’s Looking at You, Kid”
      2. Python GUI Development Options
      3. tkinter Overview
      4. Climbing the GUI Learning Curve
      5. tkinter Coding Alternatives
      6. Adding Buttons and Callbacks
      7. Adding User-Defined Callback Handlers
      8. Adding Multiple Widgets
      9. Customizing Widgets with Classes
      10. Reusable GUI Components with Classes
      11. The End of the Tutorial
      12. Python/tkinter for Tcl/Tk Converts
    2. 8. A tkinter Tour, Part 1
      1. “Widgets and Gadgets and GUIs, Oh My!”
      2. Configuring Widget Appearance
      3. Top-Level Windows
      4. Dialogs
      5. Binding Events
      6. Message and Entry
      7. Checkbutton, Radiobutton, and Scale
      8. Running GUI Code Three Ways
      9. Images
      10. Viewing and Processing Images with PIL
    3. 9. A tkinter Tour, Part 2
      1. “On Today’s Menu: Spam, Spam, and Spam”
      2. Menus
      3. Listboxes and Scrollbars
      4. Text
      5. Canvas
      6. Grids
      7. Time Tools, Threads, and Animation
      8. The End of the Tour
    4. 10. GUI Coding Techniques
      1. “Building a Better Mousetrap”
      2. GuiMixin: Common Tool Mixin Classes
      3. GuiMaker: Automating Menus and Toolbars
      4. ShellGui: GUIs for Command-Line Tools
      5. GuiStreams: Redirecting Streams to Widgets
      6. Reloading Callback Handlers Dynamically
      7. Wrapping Up Top-Level Window Interfaces
      8. GUIs, Threads, and Queues
      9. More Ways to Add GUIs to Non-GUI Code
      10. The PyDemos and PyGadgets Launchers
    5. 11. Complete GUI Programs
      1. “Python, Open Source, and Camaros”
      2. PyEdit: A Text Editor Program/Object
      3. PyPhoto: An Image Viewer and Resizer
      4. PyView: An Image and Notes Slideshow
      5. PyDraw: Painting and Moving Graphics
      6. PyClock: An Analog/Digital Clock Widget
      7. PyToe: A Tic-Tac-Toe Game Widget
      8. Where to Go from Here
  7. IV. Internet Programming
    1. 12. Network Scripting
      1. “Tune In, Log On, and Drop Out”
      2. Python Internet Development Options
      3. Plumbing the Internet
      4. Socket Programming
      5. Handling Multiple Clients
      6. Making Sockets Look Like Files and Streams
      7. A Simple Python File Server
    2. 13. Client-Side Scripting
      1. “Socket to Me!”
      2. FTP: Transferring Files over the Net
      3. Transferring Files with ftplib
      4. Transferring Directories with ftplib
      5. Transferring Directory Trees with ftplib
      6. Processing Internet Email
      7. POP: Fetching Email
      8. SMTP: Sending Email
      9. email: Parsing and Composing Mail Content
      10. A Console-Based Email Client
      11. The mailtools Utility Package
      12. NNTP: Accessing Newsgroups
      13. HTTP: Accessing Websites
      14. The urllib Package Revisited
      15. Other Client-Side Scripting Options
    3. 14. The PyMailGUI Client
      1. “Use the Source, Luke”
      2. Major PyMailGUI Changes
      3. A PyMailGUI Demo
      4. PyMailGUI Implementation
      5. Ideas for Improvement
    4. 15. Server-Side Scripting
      1. “Oh, What a Tangled Web We Weave”
      2. What’s a Server-Side CGI Script?
      3. Running Server-Side Examples
      4. Climbing the CGI Learning Curve
      5. Saving State Information in CGI Scripts
      6. The Hello World Selector
      7. Refactoring Code for Maintainability
      8. More on HTML and URL Escapes
      9. Transferring Files to Clients and Servers
    5. 16. The PyMailCGI Server
      1. “Things to Do When Visiting Chicago”
      2. The PyMailCGI Website
      3. The Root Page
      4. Sending Mail by SMTP
      5. Reading POP Email
      6. Processing Fetched Mail
      7. Utility Modules
      8. Web Scripting Trade-Offs
  8. V. Tools and Techniques
    1. 17. Databases and Persistence
      1. “Give Me an Order of Persistence, but Hold the Pickles”
      2. Persistence Options in Python
      3. DBM Files
      4. Pickled Objects
      5. Shelve Files
      6. The ZODB Object-Oriented Database
      7. SQL Database Interfaces
      8. ORMs: Object Relational Mappers
      9. PyForm: A Persistent Object Viewer (External)
    2. 18. Data Structures
      1. “Roses Are Red, Violets Are Blue; Lists Are Mutable, and So Is Set Foo”
      2. Implementing Stacks
      3. Implementing Sets
      4. Subclassing Built-in Types
      5. Binary Search Trees
      6. Graph Searching
      7. Permuting Sequences
      8. Reversing and Sorting Sequences
      9. PyTree: A Generic Tree Object Viewer
    3. 19. Text and Language
      1. “See Jack Hack. Hack, Jack, Hack”
      2. Strategies for Processing Text in Python
      3. String Method Utilities
      4. Regular Expression Pattern Matching
      5. XML and HTML Parsing
      6. Advanced Language Tools
      7. Custom Language Parsers
      8. PyCalc: A Calculator Program/Object
    4. 20. Python/C Integration
      1. “I Am Lost at C”
      2. Extending Python in C: Overview
      3. A Simple C Extension Module
      4. The SWIG Integration Code Generator
      5. Wrapping C Environment Calls
      6. Wrapping C++ Classes with SWIG
      7. Other Extending Tools
      8. Embedding Python in C: Overview
      9. Basic Embedding Techniques
      10. Registering Callback Handler Objects
      11. Using Python Classes in C
      12. Other Integration Topics
  9. VI. The End
    1. 21. Conclusion: Python and the Development Cycle
      1. “That’s the End of the Book, Now Here’s the Meaning of Life”
      2. “Something’s Wrong with the Way We Program Computers”
      3. The “Gilligan Factor”
      4. Doing the Right Thing
      5. Enter Python
      6. But What About That Bottleneck?
      7. On Sinking the Titanic
      8. “So What’s Python?”: The Sequel
      9. In the Final Analysis…
  10. Index
  11. About the Author
  12. Colophon
  13. Copyright
O'Reilly logo

Step 6: Adding a Web Interface

GUI interfaces are easier to use than command lines and are often all we need to simplify access to data. By making our database available on the Web, though, we can open it up to even wider use. Anyone with Internet access and a web browser can access the data, regardless of where they are located and which machine they are using. Anything from workstations to cell phones will suffice. Moreover, web-based interfaces require only a web browser; there is no need to install Python to access the data except on the single-server machine. Although traditional web-based approaches may sacrifice some of the utility and speed of in-process GUI toolkits, their portability gain can be compelling.

As we’ll also see later in this book, there are a variety of ways to go about scripting interactive web pages of the sort we’ll need in order to access our data. Basic server-side CGI scripting is more than adequate for simple tasks like ours. Because it’s perhaps the simplest approach, and embodies the foundations of more advanced techniques, CGI scripting is also well-suited to getting started on the Web.

For more advanced applications, a wealth of toolkits and frameworks for Python—including Django, TurboGears, Google’s App Engine, pylons, web2py, Zope, Plone, Twisted, CherryPy, Webware, mod_python, PSP, and Quixote—can simplify common tasks and provide tools that we might otherwise need to code from scratch in the CGI world. Though they pose a new set of tradeoffs, emerging technologies such as Flex, Silverlight, and pyjamas (an AJAX-based port of the Google Web Toolkit to Python, and Python-to-JavaScript compiler) offer additional paths to achieving interactive or dynamic user-interfaces in web pages on clients, and open the door to using Python in Rich Internet Applications (RIAs).

I’ll say more about these tools later. For now, let’s keep things simple and code a CGI script.

CGI Basics

CGI scripting in Python is easy as long as you already have a handle on things like HTML forms, URLs, and the client/server model of the Web (all topics we’ll address in detail later in this book). Whether you’re aware of all the underlying details or not, the basic interaction model is probably familiar.

In a nutshell, a user visits a website and receives a form, coded in HTML, to be filled out in his or her browser. After submitting the form, a script, identified within either the form or the address used to contact the server, is run on the server and produces another HTML page as a reply. Along the way, data typically passes through three programs: from the client browser, to the web server, to the CGI script, and back again to the browser. This is a natural model for the database access interaction we’re after—users can submit a database key to the server and receive the corresponding record as a reply page.

We’ll go into CGI basics in depth later in this book, but as a first example, let’s start out with a simple interactive example that requests and then echoes back a user’s name in a web browser. The first page in this interaction is just an input form produced by the HTML file shown in Example 1-30. This HTML file is stored on the web server machine, and it is transferred to the web browser running on the client machine upon request.

Example 1-30. PP4E\Preview\cgi101.html

<title>Interactive Page</title>
<form method=POST action="cgi-bin/">
    <P><B>Enter your name:</B>
    <P><input type=text name=user>
    <P><input type=submit>

Notice how this HTML form names the script that will process its input on the server in its action attribute. This page is requested by submitting its URL (web address). When received by the web browser on the client, the input form that this code produces is shown in Figure 1-10 (in Internet Explorer here).

cgi101.html input form page

Figure 1-10. cgi101.html input form page

When this input form is submitted, a web server intercepts the request (more on the web server in a moment) and runs the Python CGI script in Example 1-31. Like the HTML file, this Python script resides on the same machine as the web server; it’s run on the server machine to handle the inputs and generate a reply to the browser on the client. It uses the cgi module to parse the form’s input and insert it into the HTML reply stream, properly escaped. The cgi module gives us a dictionary-like interface to form inputs sent by the browser, and the HTML code that this script prints winds up rendering the next page on the client’s browser. In the CGI world, the standard output stream is connected to the client through a socket.

Example 1-31. PP4E\Preview\cgi-bin\

import cgi
form = cgi.FieldStorage()                 # parse form data
print('Content-type: text/html\n')        # hdr plus blank line
print('<title>Reply Page</title>')        # html reply page
if not 'user' in form:
    print('<h1>Who are you?</h1>')
    print('<h1>Hello <i>%s</i>!</h1>' % cgi.escape(form['user'].value))

And if all goes well, we receive the reply page shown in Figure 1-11—essentially, just an echo of the data we entered in the input page. The page in this figure is produced by the HTML printed by the Python CGI script running on the server. Along the way, the user’s name was transferred from a client to a server and back again—potentially across networks and miles. This isn’t much of a website, of course, but the basic principles here apply, whether you’re just echoing inputs or doing full-blown e-whatever. script reply page for input form

Figure 1-11. script reply page for input form

If you have trouble getting this interaction to run on Unix-like systems, you may need to modify the path to your Python in the #! line at the top of the script file and make it executable with a chmod command, but this is dependent on your web server (again, more on the missing server piece in a moment).

Also note that the CGI script in Example 1-31 isn’t printing complete HTML: the <html> and <body> tags of the static HTML file in Example 1-30 are missing. Strictly speaking, such tags should be printed, but web browsers don’t mind the omissions, and this book’s goal is not to teach legalistic HTML; see other resources for more on HTML.

GUIs versus the Web

Before moving on, it’s worth taking a moment to compare this basic CGI example with the simple GUI of Example 1-28 and Figure 1-6. Here, we’re running scripts on a server to generate HTML that is rendered in a web browser. In the GUI, we make calls to build the display and respond to events within a single process and on a single machine. The GUI runs multiple layers of software, but not multiple programs. By contrast, the CGI approach is much more distributed—the server, the browser, and possibly the CGI script itself run as separate programs that usually communicate over a network.

Because of such differences, the standalone GUI model may be simpler and more direct: there is no intermediate server, replies do not require invoking a new program, no HTML needs to be generated, and the full power of a GUI toolkit is at our disposal. On the other hand, a web-based interface can be viewed in any browser on any computer and only requires Python on the server machine.

And just to muddle the waters further, a GUI can also employ Python’s standard library networking tools to fetch and display data from a remote server (that’s how web browsers do their work internally), and some newer frameworks such as Flex, Silverlight, and pyjamas provide toolkits that support more full-featured user interfaces within web pages on the client (the RIAs I mentioned earlier), albeit at some added cost in code complexity and software stack depth. We’ll revisit the trade-offs of the GUI and CGI schemes later in this book, because it’s a major design choice today. First, let’s preview a handful of pragmatic issues related to CGI work before we apply it to our people database.

Running a Web Server

Of course, to run CGI scripts at all, we need a web server that will serve up our HTML and launch our Python scripts on request. The server is a required mediator between the browser and the CGI script. If you don’t have an account on a machine that has such a server available, you’ll want to run one of your own. We could configure and run a full production-level web server such as the open source Apache system (which, by the way, can be tailored with Python-specific support by the mod_python extension). For this chapter, however, I instead wrote a simple web server in Python using the code in Example 1-32.

We’ll revisit the tools used in this example later in this book. In short, because Python provides precoded support for various types of network servers, we can build a CGI-capable and portable HTTP web server in just 8 lines of code (and a whopping 16 if we include comments and blank lines).

As we’ll see later in this book, it’s also easy to build proprietary network servers with low-level socket calls in Python, but the standard library provides canned implementations for many common server types, web based or otherwise. The socketserver module, for instance, supports threaded and forking versions of TCP and UDP servers. Third-party systems such as Twisted provide even more implementations. For serving up web content, the standard library modules used in Example 1-32 provide what we need.

Example 1-32. PP4E\Preview\

Implement an HTTP web server in Python that knows how to run server-side
CGI scripts coded in Python;  serves files and scripts from current working
dir;  Python scripts must be stored in webdir\cgi-bin or webdir\htbin;

import os, sys
from http.server import HTTPServer, CGIHTTPRequestHandler

webdir = '.'   # where your html files and cgi-bin script directory live
port   = 80    # default http://localhost/, else use http://localhost:xxxx/

os.chdir(webdir)                                       # run in HTML root dir
srvraddr = ("", port)                                  # my hostname, portnumber
srvrobj  = HTTPServer(srvraddr, CGIHTTPRequestHandler)
srvrobj.serve_forever()                                # run as perpetual daemon

The classes this script uses assume that the HTML files to be served up reside in the current working directory and that the CGI scripts to be run live in a cgi-bin or htbin subdirectory there. We’re using a cgi-bin subdirectory for scripts, as suggested by the filename of Example 1-31. Some web servers look at filename extensions to detect CGI scripts; our script uses this subdirectory-based scheme instead.

To launch the server, simply run this script (in a console window, by an icon click, or otherwise); it runs perpetually, waiting for requests to be submitted from browsers and other clients. The server listens for requests on the machine on which it runs and on the standard HTTP port number 80. To use this script to serve up other websites, either launch it from the directory that contains your HTML files and a cgi-bin subdirectory that contains your CGI scripts, or change its webdir variable to reflect the site’s root directory (it will automatically change to that directory and serve files located there).

But where in cyberspace do you actually run the server script? If you look closely enough, you’ll notice that the server name in the addresses of the prior section’s examples (near the top right of the browser after the http://) is always localhost. To keep this simple, I am running the web server on the same machine as the web browser; that’s what the server name “localhost” (and the equivalent IP address “”) means. That is, the client and server machines are the same: the client (web browser) and server (web server) are just different processes running at the same time on the same computer.

Though not meant for enterprise-level work, this turns out to be a great way to test CGI scripts—you can develop them on the same machine without having to transfer code back to a remote server machine after each change. Simply run this script from the directory that contains both your HTML files and a cgi-bin subdirectory for scripts and then use http://localhost/ in your browser to access your HTML and script files. Here is the trace output the web server script produces in a Windows console window that is running on the same machine as the web browser and launched from the directory where the HTML files reside:

...\PP4E\Preview> python
mark-VAIO - - [28/Jan/2010 18:34:01] "GET /cgi101.html HTTP/1.1" 200 -
mark-VAIO - - [28/Jan/2010 18:34:12] "POST /cgi-bin/ HTTP/1.1" 200 -
mark-VAIO - - [28/Jan/2010 18:34:12] command: C:\Python31\python.exe -u C:\Users
\mark\Stuff\Books\4E\PP4E\dev\Examples\PP4E\Preview\cgi-bin\ ""
mark-VAIO - - [28/Jan/2010 18:34:13] CGI script exited OK
mark-VAIO - - [28/Jan/2010 18:35:25] "GET /cgi-bin/ HTTP
/1.1" 200 -
mark-VAIO - - [28/Jan/2010 18:35:25] command: C:\Python31\python.exe -u C:\Users
mark-VAIO - - [28/Jan/2010 18:35:26] CGI script exited OK

One pragmatic note here: you may need administrator privileges in order to run a server on the script’s default port 80 on some platforms: either find out how to run this way or try running on a different port. To run this server on a different port, change the port number in the script and name it explicitly in the URL (e.g., http://localhost:8888/). We’ll learn more about this convention later in this book.

And to run this server on a remote computer, upload the HTML files and CGI scripts subdirectory to the remote computer, launch the server script on that machine, and replace “localhost” in the URLs with the domain name or IP address of your server machine (e.g., When running the server remotely, all the interaction will be as shown here, but inputs and replies will be automatically shipped across network connections, not routed between programs running on the same computer.

To delve further into the server classes our web server script employs, see their implementation in Python’s standard library (C:\Python31\Lib for Python 3.1); one of the major advantages of open source system like Python is that we can always look under the hood this way. In Chapter 15, we’ll expand Example 1-32 to allow the directory name and port numbers to be passed in on the command line.

Using Query Strings and urllib

In the basic CGI example shown earlier, we ran the Python script by filling out and submitting a form that contained the name of the script. Really, server-side CGI scripts can be invoked in a variety of ways—either by submitting an input form as shown so far or by sending the server an explicit URL (Internet address) string that contains inputs at the end. Such an explicit URL can be sent to a server either inside or outside of a browser; in a sense, it bypasses the traditional input form page.

For instance, Figure 1-12 shows the reply generated by the server after typing a URL of the following form in the address field at the top of the web browser (+ means a space here):

http://localhost/cgi-bin/ reply to GET-style query parameters

Figure 1-12. reply to GET-style query parameters

The inputs here, known as query parameters, show up at the end of the URL after the ?; they are not entered into a form’s input fields. Adding inputs to URLs is sometimes called a GET request. Our original input form uses the POST method, which instead ships inputs in a separate step. Luckily, Python CGI scripts don’t have to distinguish between the two; the cgi module’s input parser handles any data submission method differences for us.

It’s even possible, and often useful, to submit URLs with inputs appended as query parameters completely outside any web browser. The Python urllib module package, for instance, allows us to read the reply generated by a server for any valid URL. In effect, it allows us to visit a web page or invoke a CGI script from within another script; your Python code, instead of a browser, acts as the web client. Here is this module in action, run from the interactive command line:

>>> from urllib.request import urlopen
>>> conn = urlopen('http://localhost/cgi-bin/')
>>> reply =
>>> reply
b'<title>Reply Page</title>\n<h1>Hello <i>Sue Smith</i>!</h1>\n'

>>> urlopen('http://localhost/cgi-bin/').read()
b'<title>Reply Page</title>\n<h1>Who are you?</h1>\n'

>>> urlopen('http://localhost/cgi-bin/').read()
b'<title>Reply Page</title>\n<h1>Hello <i>Bob</i>!</h1>\n'

The urllib module package gives us a file-like interface to the server’s reply for a URL. Notice that the output we read from the server is raw HTML code (normally rendered by a browser). We can process this text with any of Python’s text-processing tools, including:

  • String methods to search and split

  • The re regular expression pattern-matching module

  • Full-blown HTML and XML parsing support in the standard library, including html.parser, as well as SAX-, DOM-, and ElementTree–style XML parsing tools.

When combined with such tools, the urllib package is a natural for a variety of techniques—ad-hoc interactive testing of websites, custom client-side GUIs, “screen scraping” of web page content, and automated regression testing systems for remote server-side CGI scripts.

Formatting Reply Text

One last fine point: because CGI scripts use text to communicate with clients, they need to format their replies according to a set of rules. For instance, notice how Example 1-31 adds a blank line between the reply’s header and its HTML by printing an explicit newline (\n) in addition to the one print adds automatically; this is a required separator.

Also note how the text inserted into the HTML reply is run through the cgi.escape (a.k.a. html.escape in Python 3.2; see the note under Python HTML and URL Escape Tools) call, just in case the input includes a character that is special in HTML. For example, Figure 1-13 shows the reply we receive for form input Bob </i> Smith—the </i> in the middle becomes &lt;/i&gt; in the reply, and so doesn’t interfere with real HTML code (use your browser’s view source option to see this for yourself); if not escaped, the rest of the name would not be italicized.

Escaping HTML characters

Figure 1-13. Escaping HTML characters

Escaping text like this isn’t always required, but it is a good rule of thumb when its content isn’t known; scripts that generate HTML have to respect its rules. As we’ll see later in this book, a related call, urllib.parse.quote, applies URL escaping rules to text. As we’ll also see, larger frameworks often handle text formatting tasks for us.

A Web-Based Shelve Interface

Now, to use the CGI techniques of the prior sections for our database application, we basically just need a bigger input and reply form. Figure 1-14 shows the form we’ll implement for accessing our database in a web browser.

peoplecgi.html input page

Figure 1-14. peoplecgi.html input page

Coding the website

To implement the interaction, we’ll code an initial HTML input form, as well as a Python CGI script for displaying fetch results and processing update requests. Example 1-33 shows the input form’s HTML code that builds the page in Figure 1-14.

Example 1-33. PP4E\Preview\peoplecgi.html

<title>People Input Form</title>
<form method=POST action="cgi-bin/">
    <tr><th>Key <td><input type=text name=key>
    <tr><th>Name<td><input type=text name=name>
    <tr><th>Age <td><input type=text name=age>
    <tr><th>Job <td><input type=text name=job>
    <tr><th>Pay <td><input type=text name=pay>
    <input type=submit value="Fetch",  name=action>
    <input type=submit value="Update", name=action>

To handle form (and other) requests, Example 1-34 implements a Python CGI script that fetches and updates our shelve’s records. It echoes back a page similar to that produced by Example 1-33, but with the form fields filled in from the attributes of actual class objects in the shelve database.

As in the GUI, the same web page is used for both displaying results and inputting updates. Unlike the GUI, this script is run anew for each step of user interaction, and it reopens the database each time (the reply page’s action field provides a link back to the script for the next request). The basic CGI model provides no automatic memory from page to page, so we have to start from scratch each time.

Example 1-34. PP4E\Preview\cgi-bin\

Implement a web-based interface for viewing and updating class instances
stored in a shelve; the shelve lives on server (same machine if localhost)

import cgi, shelve, sys, os                   # cgi.test() dumps inputs
shelvename = 'class-shelve'                   # shelve files are in cwd
fieldnames = ('name', 'age', 'job', 'pay')

form = cgi.FieldStorage()                     # parse form data
print('Content-type: text/html')              # hdr, blank line is in replyhtml
sys.path.insert(0, os.getcwd())               # so this and pickler find person

# main html template
replyhtml = """
<title>People Input Form</title>
<form method=POST action="">
    <tr><th>key<td><input type=text name=key value="%(key)s">
    <input type=submit value="Fetch",  name=action>
    <input type=submit value="Update", name=action>

# insert html for data rows at $ROWS$
rowhtml  = '<tr><th>%s<td><input type=text name=%s value="%%(%s)s">\n'
rowshtml = ''
for fieldname in fieldnames:
    rowshtml += (rowhtml % ((fieldname,) * 3))
replyhtml = replyhtml.replace('$ROWS$', rowshtml)

def htmlize(adict):
    new = adict.copy()
    for field in fieldnames:                       # values may have &, >, etc.
        value = new[field]                         # display as code: quoted
        new[field] = cgi.escape(repr(value))       # html-escape special chars
    return new

def fetchRecord(db, form):
        key = form['key'].value
        record = db[key]
        fields = record.__dict__                   # use attribute dict
        fields['key'] = key                        # to fill reply string
        fields = dict.fromkeys(fieldnames, '?')
        fields['key'] = 'Missing or invalid key!'
    return fields

def updateRecord(db, form):
    if not 'key' in form:
        fields = dict.fromkeys(fieldnames, '?')
        fields['key'] = 'Missing key input!'
        key = form['key'].value
        if key in db:
            record = db[key]                       # update existing record
            from person import Person              # make/store new one for key
            record = Person(name='?', age='?')     # eval: strings must be quoted
        for field in fieldnames:
            setattr(record, field, eval(form[field].value))
        db[key] = record
        fields = record.__dict__
        fields['key'] = key
    return fields

db =
action = form['action'].value if 'action' in form else None
if action == 'Fetch':
    fields = fetchRecord(db, form)
elif action == 'Update':
    fields = updateRecord(db, form)
    fields = dict.fromkeys(fieldnames, '?')        # bad submit button value
    fields['key'] = 'Missing or invalid action!'
print(replyhtml % htmlize(fields))                 # fill reply from dict

This is a fairly large script, because it has to handle user inputs, interface with the database, and generate HTML for the reply page. Its behavior is fairly straightforward, though, and similar to the GUI of the prior section.

Directories, string formatting, and security

A few fine points before we move on. First of all, make sure the web server script we wrote earlier in Example 1-32 is running before you proceed; it’s going to catch our requests and route them to our script.

Also notice how this script adds the current working directory (os.getcwd) to the sys.path module search path when it first starts. Barring a PYTHONPATH change, this is required to allow both the pickler and this script itself to import the person module one level up from the script. Because of the new way the web server runs CGI scripts in Python 3, the current working directory isn’t added to sys.path, even though the shelve’s files are located there correctly when opened. Such details can vary per server.

The only other feat of semi-magic the CGI script relies on is using a record’s attribute dictionary (__dict__) as the source of values when applying HTML escapes to field values and string formatting to the HTML reply template string in the last line of the script. Recall that a %(key)code replacement target fetches a value by key from a dictionary:

>>> D = {'say': 5, 'get': 'shrubbery'}
>>> D['say']
>>> S = '%(say)s => %(get)s' % D
>>> S
'5 => shrubbery'

By using an object’s attribute dictionary, we can refer to attributes by name in the format string. In fact, part of the reply template is generated by code. If its structure is confusing, simply insert statements to print replyhtml and to call sys.exit, and run from a simple command line. This is how the table’s HTML in the middle of the reply is generated (slightly formatted here for readability):

   <tr><th>key<td><input type=text name=key value="%(key)s">
   <tr><th>name<td><input type=text name=name value="%(name)s">
   <tr><th>age<td><input type=text name=age value="%(age)s">
   <tr><th>job<td><input type=text name=job value="%(job)s">
   <tr><th>pay<td><input type=text name=pay value="%(pay)s">

This text is then filled in with key values from the record’s attribute dictionary by string formatting at the end of the script. This is done after running the dictionary through a utility to convert its values to code text with repr and escape that text per HTML conventions with cgi.escape (again, the last step isn’t always required, but it’s generally a good practice).

These HTML reply lines could have been hardcoded in the script, but generating them from a tuple of field names is a more general approach—we can add new fields in the future without having to update the HTML template each time. Python’s string processing tools make this a snap.

In the interest of fairness, I should point out that Python’s newer str.format method could achieve much the same effect as the traditional % format expression used by this script, and it provides specific syntax for referencing object attributes which to some might seem more explicit than using __dict__ keys:

>>> D = {'say': 5, 'get': 'shrubbery'}

>>> '%(say)s => %(get)s' % D                    # expression: key reference
'5 => shrubbery'
>>> '{say} => {get}'.format(**D)                # method: key reference
'5 => shrubbery'

>>> from person import Person
>>> bob = Person('Bob', 35)

>>> '%(name)s, %(age)s' % bob.__dict__          # expression: __dict__ keys
'Bob, 35'
>>> '{} => {0.age}'.format(bob)           # method: attribute syntax
'Bob => 35'

Because we need to escape attribute values first, though, the format method call’s attribute syntax can’t be used directly this way; the choice is really between both technique’s key reference syntax above. (At this writing, it’s not clear which formatting technique may come to dominate, so we take liberties with using either in this book; if one replaces the other altogether someday, you’ll want to go with the winner.)

In the interest of security, I also need to remind you one last time that the eval call used in this script to convert inputs to Python objects is powerful, but not secure—it happily runs any Python code, which can perform any system modifications that the script’s process has permission to make. If you care, you’ll need to trust the input source, run in a restricted environment, or use more focused input converters like int and float. This is generally a larger concern in the Web world, where request strings might arrive from arbitrary sources. Since we’re all friends here, though, we’ll ignore the threat.

Using the website

Despite the extra complexities of servers, directories, and strings, using the web interface is as simple as using the GUI, and it has the added advantage of running on any machine with a browser and Web connection. To fetch a record, fill in the Key field and click Fetch; the script populates the page with field data grabbed from the corresponding class instance in the shelve, as illustrated in Figure 1-15 for key bob. reply page

Figure 1-15. reply page

Figure 1-15 shows what happens when the key comes from the posted form. As usual, you can also invoke the CGI script by instead passing inputs on a query string at the end of the URL; Figure 1-16 shows the reply we get when accessing a URL of the following form:

http://localhost/cgi-bin/ reply for query parameters

Figure 1-16. reply for query parameters

As we’ve seen, such a URL can be submitted either within your browser or by scripts that use tools such as the urllib package. Again, replace “localhost” with your server’s domain name if you are running the script on a remote machine.

To update a record, fetch it by key, enter new values in the field inputs, and click Update; the script will take the input fields and store them in the attributes of the class instance in the shelve. Figure 1-17 shows the reply we get after updating sue.

Finally, adding a record works the same as in the GUI: fill in a new key and field values and click Update; the CGI script creates a new class instance, fills out its attributes, and stores it in the shelve under the new key. There really is a class object behind the web page here, but we don’t have to deal with the logic used to generate it. Figure 1-18 shows a record added to the database in this way. update reply

Figure 1-17. update reply after adding a new record

Figure 1-18. after adding a new record

In principle, we could also update and add records by submitting a URL—either from a browser or from a script—such as:

http://localhost/cgi-bin/ ...more...

Except for automated tools, though, typing such a long URL will be noticeably more difficult than filling out the input page. Here is part of the reply page generated for the “guido” record’s display of Figure 1-18 (use your browser’s “view page source” option to see this for yourself). Note how the < and > characters are translated to HTML escapes with cgi.escape before being inserted into the reply:

<tr><th>key<td><input type=text name=key value="guido">
<tr><th>name<td><input type=text name=name value="'GvR'">
<tr><th>age<td><input type=text name=age value="None">
<tr><th>job<td><input type=text name=job value="'BDFL'">
<tr><th>pay<td><input type=text name=pay value="'&lt;shrubbery&gt;'">

As usual, the standard library urllib module package comes in handy for testing our CGI script; the output we get back is raw HTML, but we can parse it with other standard library tools and use it as the basis of a server-side script regression testing system run on any Internet-capable machine. We might even parse the server’s reply fetched this way and display its data in a client-side GUI coded with tkinter; GUIs and web pages are not mutually exclusive techniques. The last test in the following interaction shows a portion of the error message page’s HTML that is produced when the action is missing or invalid in the inputs, with line breaks added for readability:

>>> from urllib.request import urlopen
>>> url = 'http://localhost/cgi-bin/'
>>> urlopen(url).read()
b'<html>\n<title>People Input Form</title>\n<body>\n
<form method=POST action="">\n    <table>\n
<tr><th>key<td><input type=text name=key value="sue">\n
<tr><th>name<td><input type=text name=name value="\'Sue Smith\'">\n
<tr><t ...more deleted...

>>> urlopen('http://localhost/cgi-bin/').read()
b'<html>\n<title>People Input Form</title>\n<body>\n
<form method=POST action="">\n    <table>\n
<tr><th>key<td><input type=text name=key value="Missing or invalid action!">\n
    <tr><th>name<td><input type=text name=name value="\'?\'">\n
<tr><th>age<td><input type=text name=age value="\'?\'">\n<tr> ...more deleted...

In fact, if you’re running this CGI script on “localhost,” you can use both the last section’s GUI and this section’s web interface to view the same physical shelve file—these are just alternative interfaces to the same persistent Python objects. For comparison, Figure 1-19 shows what the record we saw in Figure 1-18 looks like in the GUI; it’s the same object, but we are not contacting an intermediate server, starting other scripts, or generating HTML to view it.

Same object displayed in the GUI

Figure 1-19. Same object displayed in the GUI

And as before, we can always check our work on the server machine either interactively or by running scripts. We may be viewing a database through web browsers and GUIs, but, ultimately, it is just Python objects in a Python shelve file:

>>> import shelve
>>> db ='class-shelve')
>>> db['sue'].name
'Sue Smith'
>>> db['guido'].job
>>> list(db['guido'].name)
['G', 'v', 'R']
>>> list(db.keys())
['sue', 'bill', 'nobody', 'tomtom', 'tom', 'bob', 'peg', 'guido']

Here in action again is the original database script we wrote in Example 1-19 before we moved on to GUIs and the web; there are many ways to view Python data:

sue =>
   Sue Smith 60000
bill =>
   bill 9999
nobody =>
   John Doh None
tomtom =>
   Tom Tom 40000
tom =>
   Tom Doe 90000
bob =>
   Bob Smith 30000
peg =>
   1 4
guido =>
   GvR <shrubbery>

Future directions

Naturally, there are plenty of improvements we could make here, too:

  • The HTML code of the initial input page in Example 1-33, for instance, is somewhat redundant with the script in Example 1-34, and it could be automatically generated by another script that shares common information.

  • In fact, we might avoid hardcoding HTML in our script completely if we use one of the HTML generator tools we’ll meet later, including HTMLgen (a system for creating HTML from document object trees) and PSP (Python Server Pages, a server-side HTML templating system for Python similar to PHP and ASP).

  • For ease of maintenance, it might also be better to split the CGI script’s HTML code off to a separate file in order to better divide display from logic (different parties with possibly different skill sets could work on the different files).

  • Moreover, if this website might be accessed by many people simultaneously, we would have to add file locking or move to a database such as ZODB or MySQL to support concurrent updates. ZODB and other full-blown database systems would also provide transaction rollbacks in the event of failures. For basic file locking, the call and its flags provide the tools we need.

  • ORMs (object relational mappers) for Python such as SQLObject and SQLAlchemy mentioned earlier might also allow us to gain concurrent update support of an underlying relational database system, but retain our Python class view of the data.

  • In the end, if our site grows much beyond a few interactive pages, we might also migrate from basic CGI scripting to a more complete web framework such as one of those mentioned at the start of this section— Django, TurboGears, pyjamas, and others. If we must retain information across pages, tools such as cookies, hidden inputs, mod_python session data, and FastCGI may help too.

  • If our site eventually includes content produced by its own users, we might transition to Plone, a popular open source Python- and Zope-based site builder that, using a workflow model, delegates control of site content to its producers.

  • And if wireless or cloud interfaces are on our agenda, we might eventually migrate our system to cell phones using a Python port such as those available for scripting Nokia platforms and Google’s Android, or to a cloud-computing platform such as Google’s Python-friendly App Engine. Python tends to go wherever technology trends lead.

For now, though, both the GUI and web-based interfaces we’ve coded get the job done.

The best content for your career. Discover unlimited learning on demand for around $1/day.