You are previewing Programming Python, 4th Edition.

Programming Python, 4th Edition

Cover of Programming Python, 4th Edition by Mark Lutz Published by O'Reilly Media, Inc.
  1. Programming Python
  2. A Note Regarding Supplemental Files
  3. Preface
    1. “And Now for Something Completely Different…”
    2. About This Book
      1. This Book’s Ecosystem
      2. What This Book Is Not
    3. About This Fourth Edition
      1. Specific Changes in This Edition
    4. What’s Left, Then?
    5. Python 3.X Impacts on This Book
      1. Specific 3.X Changes
      2. Language Versus Library: Unicode
      3. Python 3.1 Limitations: Email, CGI
    6. Using Book Examples
      1. Where to Look for Examples and Updates
      2. Example Portability
      3. Demo Launchers
      4. Code Reuse Policies
    7. Contacting O’Reilly
    8. Conventions Used in This Book
    9. Acknowledgments
  4. I. The Beginning
    1. 1. A Sneak Preview
      1. “Programming Python: The Short Story”
      2. The Task
      3. Step 1: Representing Records
      4. Step 2: Storing Records Persistently
      5. Step 3: Stepping Up to OOP
      6. Step 4: Adding Console Interaction
      7. Step 5: Adding a GUI
      8. Step 6: Adding a Web Interface
      9. The End of the Demo
  5. II. System Programming
    1. 2. System Tools
      1. “The os.path to Knowledge”
      2. System Scripting Overview
      3. Introducing the sys Module
      4. Introducing the os Module
    2. 3. Script Execution Context
      1. “I’d Like to Have an Argument, Please”
      2. Current Working Directory
      3. Command-Line Arguments
      4. Shell Environment Variables
      5. Standard Streams
    3. 4. File and Directory Tools
      1. “Erase Your Hard Drive in Five Easy Steps!”
      2. File Tools
      3. Directory Tools
    4. 5. Parallel System Tools
      1. “Telling the Monkeys What to Do”
      2. Forking Processes
      3. Threads
      4. Program Exits
      5. Interprocess Communication
      6. The multiprocessing Module
      7. Other Ways to Start Programs
      8. A Portable Program-Launch Framework
      9. Other System Tools Coverage
    5. 6. Complete System Programs
      1. “The Greps of Wrath”
      2. A Quick Game of “Find the Biggest Python File”
      3. Splitting and Joining Files
      4. Generating Redirection Web Pages
      5. A Regression Test Script
      6. Copying Directory Trees
      7. Comparing Directory Trees
      8. Searching Directory Trees
      9. Visitor: Walking Directories “++”
      10. Playing Media Files
      11. Automated Program Launchers (External)
  6. III. GUI Programming
    1. 7. Graphical User Interfaces
      1. “Here’s Looking at You, Kid”
      2. Python GUI Development Options
      3. tkinter Overview
      4. Climbing the GUI Learning Curve
      5. tkinter Coding Alternatives
      6. Adding Buttons and Callbacks
      7. Adding User-Defined Callback Handlers
      8. Adding Multiple Widgets
      9. Customizing Widgets with Classes
      10. Reusable GUI Components with Classes
      11. The End of the Tutorial
      12. Python/tkinter for Tcl/Tk Converts
    2. 8. A tkinter Tour, Part 1
      1. “Widgets and Gadgets and GUIs, Oh My!”
      2. Configuring Widget Appearance
      3. Top-Level Windows
      4. Dialogs
      5. Binding Events
      6. Message and Entry
      7. Checkbutton, Radiobutton, and Scale
      8. Running GUI Code Three Ways
      9. Images
      10. Viewing and Processing Images with PIL
    3. 9. A tkinter Tour, Part 2
      1. “On Today’s Menu: Spam, Spam, and Spam”
      2. Menus
      3. Listboxes and Scrollbars
      4. Text
      5. Canvas
      6. Grids
      7. Time Tools, Threads, and Animation
      8. The End of the Tour
    4. 10. GUI Coding Techniques
      1. “Building a Better Mousetrap”
      2. GuiMixin: Common Tool Mixin Classes
      3. GuiMaker: Automating Menus and Toolbars
      4. ShellGui: GUIs for Command-Line Tools
      5. GuiStreams: Redirecting Streams to Widgets
      6. Reloading Callback Handlers Dynamically
      7. Wrapping Up Top-Level Window Interfaces
      8. GUIs, Threads, and Queues
      9. More Ways to Add GUIs to Non-GUI Code
      10. The PyDemos and PyGadgets Launchers
    5. 11. Complete GUI Programs
      1. “Python, Open Source, and Camaros”
      2. PyEdit: A Text Editor Program/Object
      3. PyPhoto: An Image Viewer and Resizer
      4. PyView: An Image and Notes Slideshow
      5. PyDraw: Painting and Moving Graphics
      6. PyClock: An Analog/Digital Clock Widget
      7. PyToe: A Tic-Tac-Toe Game Widget
      8. Where to Go from Here
  7. IV. Internet Programming
    1. 12. Network Scripting
      1. “Tune In, Log On, and Drop Out”
      2. Python Internet Development Options
      3. Plumbing the Internet
      4. Socket Programming
      5. Handling Multiple Clients
      6. Making Sockets Look Like Files and Streams
      7. A Simple Python File Server
    2. 13. Client-Side Scripting
      1. “Socket to Me!”
      2. FTP: Transferring Files over the Net
      3. Transferring Files with ftplib
      4. Transferring Directories with ftplib
      5. Transferring Directory Trees with ftplib
      6. Processing Internet Email
      7. POP: Fetching Email
      8. SMTP: Sending Email
      9. email: Parsing and Composing Mail Content
      10. A Console-Based Email Client
      11. The mailtools Utility Package
      12. NNTP: Accessing Newsgroups
      13. HTTP: Accessing Websites
      14. The urllib Package Revisited
      15. Other Client-Side Scripting Options
    3. 14. The PyMailGUI Client
      1. “Use the Source, Luke”
      2. Major PyMailGUI Changes
      3. A PyMailGUI Demo
      4. PyMailGUI Implementation
      5. Ideas for Improvement
    4. 15. Server-Side Scripting
      1. “Oh, What a Tangled Web We Weave”
      2. What’s a Server-Side CGI Script?
      3. Running Server-Side Examples
      4. Climbing the CGI Learning Curve
      5. Saving State Information in CGI Scripts
      6. The Hello World Selector
      7. Refactoring Code for Maintainability
      8. More on HTML and URL Escapes
      9. Transferring Files to Clients and Servers
    5. 16. The PyMailCGI Server
      1. “Things to Do When Visiting Chicago”
      2. The PyMailCGI Website
      3. The Root Page
      4. Sending Mail by SMTP
      5. Reading POP Email
      6. Processing Fetched Mail
      7. Utility Modules
      8. Web Scripting Trade-Offs
  8. V. Tools and Techniques
    1. 17. Databases and Persistence
      1. “Give Me an Order of Persistence, but Hold the Pickles”
      2. Persistence Options in Python
      3. DBM Files
      4. Pickled Objects
      5. Shelve Files
      6. The ZODB Object-Oriented Database
      7. SQL Database Interfaces
      8. ORMs: Object Relational Mappers
      9. PyForm: A Persistent Object Viewer (External)
    2. 18. Data Structures
      1. “Roses Are Red, Violets Are Blue; Lists Are Mutable, and So Is Set Foo”
      2. Implementing Stacks
      3. Implementing Sets
      4. Subclassing Built-in Types
      5. Binary Search Trees
      6. Graph Searching
      7. Permuting Sequences
      8. Reversing and Sorting Sequences
      9. PyTree: A Generic Tree Object Viewer
    3. 19. Text and Language
      1. “See Jack Hack. Hack, Jack, Hack”
      2. Strategies for Processing Text in Python
      3. String Method Utilities
      4. Regular Expression Pattern Matching
      5. XML and HTML Parsing
      6. Advanced Language Tools
      7. Custom Language Parsers
      8. PyCalc: A Calculator Program/Object
    4. 20. Python/C Integration
      1. “I Am Lost at C”
      2. Extending Python in C: Overview
      3. A Simple C Extension Module
      4. The SWIG Integration Code Generator
      5. Wrapping C Environment Calls
      6. Wrapping C++ Classes with SWIG
      7. Other Extending Tools
      8. Embedding Python in C: Overview
      9. Basic Embedding Techniques
      10. Registering Callback Handler Objects
      11. Using Python Classes in C
      12. Other Integration Topics
  9. VI. The End
    1. 21. Conclusion: Python and the Development Cycle
      1. “That’s the End of the Book, Now Here’s the Meaning of Life”
      2. “Something’s Wrong with the Way We Program Computers”
      3. The “Gilligan Factor”
      4. Doing the Right Thing
      5. Enter Python
      6. But What About That Bottleneck?
      7. On Sinking the Titanic
      8. “So What’s Python?”: The Sequel
      9. In the Final Analysis…
  10. Index
  11. About the Author
  12. Colophon
  13. Copyright
O'Reilly logo

Python 3.X Impacts on This Book

As mentioned, this edition now covers Python 3.X only. Python 3.X is an incompatible version of the language. The 3.X core language itself is very similar to Python 2.X, but there are substantial changes in both the language and its many standard libraries. Although some readers with no prior background in 2.X may be able to bypass the differences, the changes had a big impact on the content of this edition. For the still very large existing Python 2.X user base, this section documents the most noteworthy changes in this category.

If you’re interested in 2.X differences, I also suggest finding a copy of the Fourth Edition of the book Python Pocket Reference described earlier. That book gives both 2.X and 3.X versions of core language structures, built-in functions and exceptions, and many of the standard library modules and tools used in this book. Though not designed to be a reference or version translator per se, the Fourth Edition of Learning Python similarly covers both 2.X and 3.X, and as stated, is prerequisite material to this book. The goal of this 3.X-only Programming Python is not to abandon the current vast 2.X user base in favor of a still imaginary one for 3.X; it is to help readers with the migration, and avoid doubling the size of an already massive book.

Specific 3.X Changes

Luckily, many of the 2.X/3.X differences that impact this book’s presentation are trivial. For instance, the tkinter GUI toolkit, used extensively in this book, is shown under its 3.X tkinter name and package structure only; its 2.X Tkinter module incarnation is not described. This mostly boils down to different import statements, but only their Python 3 versions are given here. Similarly, to satisfy 3.X module naming conventions, 2.X’s anydbm, Queue, thread, StringIO.StringIO, and urllib.open become dbm, queue, _thread, io.StringIO, and urllib.request.urlopen, respectively, in both Python 3.X and this edition. Other tools are similarly renamed.

On the other hand, 3.X implies broader idiomatic changes which are, of course, more radical. For example, Python 3.X’s new Unicode awareness has inspired fully Internationalized versions of the PyEdit text editor and the PyMailGUI email client examples in this edition (more on this in a moment). Furthermore: the replacement of os.popen2 with the subprocess module required new examples; the demise of os.path.walk in favor of os.walk allowed some examples to be trimmed; the new Unicode and binary dichotomy of files and strings impacted a host of additional existing examples and material; and new modules such as multiprocessing offer new options covered in this edition.

Beyond such library changes, core language changes in Python 3 are also reflected in this book’s example code. For instance, changes to 2.X’s print, raw_input, keys, has_key, map, and apply all required changes here. In addition, 3.X’s new package-relative import model impacted a few examples including mailtools and expression parsers, and its different flavor of division forced some minor math updates in canvas-based GUI examples such as PyClock, PyDraw, and PyPhoto.

Of note here, I did not change all % string formatting expressions to use the new str.format, since both forms are supported in Python 3.1, and it now appears that they will be either indefinitely or forever. In fact, per a “grep” we’ll build and run in Chapter 11’s PyEdit example, it seems that this expression still appears over 3,000 times in Python 3.1’s own library code. Since I cannot predict Python evolution completely, see the first chapter for more on this if it ever requires updates in an unexpected future.

Also because of the 3.X scope, this edition is unable to use some third-party packages that are still in 2.X form only, as described earlier. This includes the leading MySQL interface, ZODB, PyCrypto, and others; as also mentioned, PIL was ported to 3.1 for use in this book, but this required a special patch and an official 3.X release is still presently pending. Many of these may be available in 3.X form by the time you read these words, assuming the Python world can either break some of the current cross dependencies in 2.X packages or adopt new 3.X-only tools.

Language Versus Library: Unicode

As a book focused on applications instead of core language fundamentals, language changes are not always obtrusive here. Indeed, in retrospect the book Learning Python may have been affected by 3.X core language changes more than this book. In most cases here, more example changes were probably made in the name of clarity or functionality than in support of 3.X itself.

On the other hand, Python 3.X does impact much code, and the impacts can be subtle at times. Readers with Python 2.X backgrounds will find that while 3.X core language changes are often simple to apply, updates required for changes in the 3.X standard library are sometimes more far reaching.

Chief among these, Python 3.X’s Unicode strings have had broad ramifications. Let’s be honest: to people who have spent their lives in an ASCII world, the impacts of the 3.X Unicode model can be downright aggravating at times! As we’ll see in this book, it affects file content; file names; pipe descriptors; sockets; text in GUIs; Internet protocols such as FTP and email; CGI scripts; and even some persistence tools. For better or worse, once we reach the world of applications programming as covered in this book, Unicode is no longer an optional topic for many or most Python 3.X programmers.

Of course, Unicode arguably never should have been entirely optional for many programmers in the first place. Indeed, we’ll find that things that may have appeared to work in 2.X never really did—treating text as raw byte strings can mask issues such as comparison results across encodings (see the grep utility of Chapter 11’s PyEdit for a prime example of code that should fail in the face of Unicode mismatches). Python 3.X elevates such issues to potentially every programmer’s panorama.

Still, porting nontrivial code to 3.X is not at all an insurmountable task. Moreover, many readers of this edition have the luxury of approaching Python 3.X as their first Python and need not deal with existing 2.X code. If this is your case, you’ll find Python 3.X to be a robust and widely applicable scripting and programming language, which addresses head-on many issues that once lurked in the shadows in 2.X.

Python 3.1 Limitations: Email, CGI

There’s one exception that I should call out here because of its impact on major book examples. In order to make its code relevant to the widest possible audience, this book’s major examples are related to Internet email and have much new support in this edition for Internationalization and Unicode in this domain. Chapter 14’s PyMailGUI and Chapter 16’s PyMailCGI, and all the prior examples they reuse, fall into this category. This includes the PyEdit text editor—now Unicode-aware for files, display, and greps.

On this front, there is both proverbial good news and bad. The good news is that in the end, we will be able to develop the feature-rich and fully Internationalized PyMailGUI email client in this book, using the email package as it currently exists. This will include support for arbitrary encodings in both text content and message headers, for both viewing and composing messages. The less happy news is that this will come at some cost in workaround complexity in Python 3.1.

Unfortunately, as we’ll learn in Chapter 13, the email package in Python 3.1 has a number of issues related to str/bytes combinations in Python 3.X. For example, there’s no simple way to guess the encoding needed to convert mail bytes returned by the poplib module to the str expected by the email parser. Moreover, the email package is currently broken altogether for some types of messages, and it has uneven or type-specific support for some others.

This situation appears to be temporary. Some of the issues encountered in this book are already scheduled to be repaired (in fact, one such fix in 3.2 required a last-minute patch to one of this book’s 3.1 workarounds in Chapter 13). Furthermore, a new version of email is being developed to accommodate the 3.X Unicode/bytes dichotomy more accurately, but it won’t materialize until long after this book is published, and it might be backward-incompatible with the current package’s API, much like Python 3.X itself. Because of that, this book both codes workarounds and makes some assumption along the way, but please watch its website (described ahead) for required updates in future Pythons. One upside here is that the dilemmas posed neatly reflect those common in realistic programming—an underlying theme of this text.

These issues in the email package are also inherited by the cgi module for CGI file uploads, which are in large measure broken in 3.1. CGI scripts are a basic technique eclipsed by many web frameworks today, but they still serve as an entry-level way to learn Web fundamentals and are still at the heart of many larger toolkits. A future fix seems likely for this 3.1 flaw as well, but we have to make do with nonbinary CGI file uploads for this edition in Chapters 15 and 16, and limited email attachments in PyMailCGI. This seems less than ideal nearly two years after 3.0’s release, but such is life in the dynamic worlds of both software development at large and books that aim to lead the curve instead of following it.

The best content for your career. Discover unlimited learning on demand for around $1/day.