Twisted Network Programming Essentials, 2nd Edition

Preface

This book is about Twisted, an open source, event-driven networking engine written in Python.

Twisted supports many common transport and application layer protocols, including TCP, UDP, SSL/TLS, HTTP, IMAP, SSH, IRC, and FTP. Like the language in which it is written, it is “batteries-included”; Twisted comes with client and server implementations for all of its protocols, as well as utilities that make it easy to configure and deploy production-grade Twisted applications from the command line.

Twisted includes both high- and low-level tools for building performant, cross-platform applications. You can deploy a web or mail server with just a few lines of code, or you can write your own protocol from scratch. At every level, Twisted provides a tested, RFC-conforming, extensible API that makes it possible to rapidly develop powerful network software.

In keeping with the spirit of the O’Reilly Essentials series, this book is not about torturing you with the nitty-gritty details of specific networking protocols, or with exhaustively documenting Twisted’s APIs. For a comprehensive treatment of how to use Twisted to build applications for a particular protocol, please get your footing with this book and then consult the official documentation.

Instead, the goal of this book is to give you fluency in the primitives Twisted provides and the idioms that it uses to build network clients and servers. It starts with an introduction to the underlying event-driven model and a big-picture view of Twisted as a framework, focusing on simple, self-contained examples that you can play with and extend as you explore Twisted’s APIs. Where possible, the client and server examples are written so they can be run together.

After reading this book, you will have the tools and conceptual background to build any event-driven client or server application you need, not just for the protocols that are a part of Twisted and covered in this book, but also for new protocols that you build using Twisted’s primitives.

Why Use Twisted?

Why should you use Twisted instead of some other networking library or framework? Here are a few compelling reasons. Twisted is:

Python-powered

Twisted is written in Python, a powerful, object-oriented, interpreted language. Python is a language that inspires great enthusiasm among its fans, and for good reason. It’s a joy to program in Python, which is easy to write, easy to read, and easy to run. And because Python is cross-platform, you can run the same Twisted application on Linux, Windows, Unix, and Mac OS X.

Asynchronous and event-based

Synchronous network libraries leave developers with a painful choice: either allow the application to become unresponsive during network operations, or introduce the additional complexity of threading. Twisted’s event-based, asynchronous framework makes it possible to write applications that stay responsive while processing events from multiple network connections, without using threads.

Full-featured

Twisted includes an amazing amount of functionality. Mail, web, news, chat, DNS, SSH, Telnet, RPC, database access, and more—it’s all there, ready for you to use.

Flexible and extensible

Twisted provides high-level classes to let you get started quickly. But you’ll never find yourself limited by the way things work out of the box. If you need advanced functionality, or if you need to customize the way a protocol works, you can. You can also write your own protocol implementation, to control every byte sent over the wire.

Open source

Twisted is free, both as in beer and as in freedom. It includes full source code and is released under a liberal license. Want to distribute all or part of Twisted with your application? You’re welcome to do so, with no obligations to release your own code or pay any licensing fees. Want to get a better understanding of how an object in Twisted works? Take a look at the source. And when you get to the point where you’re developing your own improvements and extensions to Twisted, you can contribute them to the community for the benefit of others.

Community-backed

Twisted has an active community of developers and users. If you run into a problem, you’ll find many fellow developers ready to help on one of the Twisted mailing lists (see Finding Answers to Your Questions, in Chapter 1). Or you can drop into the #twisted IRC channel, where the chances are good you’ll be able to start a live conversation with the very person who wrote the code you’re having trouble with.

An integration-friendly platform

A Twisted application can share data between several different services within the same process. This makes integration tasks a snap. You can write an SMTP-to-XMLRPC proxy, an SSH server that lets you update a website, or a web discussion board that includes an NNTP interface. If you need to transfer data between systems that don’t speak the same protocol, Twisted will make your job a whole lot easier.

What This Book Covers

This book does not attempt to exhaustively document every module and class available for the Twisted framework. Instead, it focuses on presenting practical examples of the most common tasks that developers building network applications face. This book will also help you to understand the key concepts and design patterns used in Twisted applications.

This book has three parts:

Learning Twisted basics through building basic clients and servers

This part covers installing Twisted, an architectural overview of the framework, and building basic TCP clients and servers. We then apply the primitives and idioms from the chapters on basic applications to a variety of client and server examples for a particular protocol, HTTP.

Building production-grade servers

At this point, well-practiced with basic clients and servers, we focus on deploying these applications in a robust and standardized fashion using the Twisted application infrastructure. This part also adds to our repertoire common components of production-grade servers: logging, database access, authentication, using threads and processes in a Twisted-safe way, and testing.

More practice through examples from other protocols

For more practice, to give a sense of Twisted’s breadth, and to cover many popular uses of Twisted, the final part of the book explores clients and servers for IRC, various mail protocols, and SSH.

Conventions Used in This Book

This book uses standard typographical conventions to highlight different types of text. You’ll see the following font styles used:

Italic

Used for emphasis, to highlight technical terms the first time they appear, and for commands, packages, filenames, directories, and URLs

Constant width

Used for code samples, and for the names of variables, classes, objects, and functions when they are used within the main text of the book

Constant width bold

Shows user input at the command line and interactive prompts

Constant width bold italic

Shows placeholder user input that you should replace with something that makes sense for you

Tip

This icon signifies a tip, suggestion, or general note.

Warning

This icon indicates a warning or caution.

What You’ll Need

This book assumes a familiarity with programming in Python. If you’re looking for a good introduction to Python, check out Learning Python, by Mark Lutz (O’Reilly), or Dive Into Python, by Mark Pilgrim (Apress). You should have a Linux, Mac OS X, or Windows computer with Python version 2.6 or 2.7 installed. Python 2.6 is included in Mac OS X 10.6 (“Snow Leopard”) and higher and in many Linux distributions. If you don’t already have Python installed, you can download it for free from the Python home page.

Changes Since the Previous Edition

The first edition of Twisted Networking Essentials was released in 2005. Since then, networking protocols have come in and out of fashion, and Twisted’s APIs have evolved and matured. This second edition builds upon the excellent foundation first edition author Abe Fettig crafted by trimming off aged protocols and Twisted APIs and covering more Twisted subprojects and features.

In particular, this edition removes the chapter on NNTP and adds chapters on building IRC clients and servers and testing your Twisted applications using the Trial framework. The sections on deploying production-grade services using the Twisted application infrastructure have been significantly expanded. In addition to a discussion and examples of Twisted applications and Twisted plugins, logging, working with databases, and using threads and processes all now get more coverage in their own chapters.

The focus of this book has also been sharpened to give you fluency in Twisted’s primitives and idioms with minimal distraction from the nitty-gritty details of specific protocols. Almost all of the examples have been streamlined, and where reasonable, reworked so that you have client and server pairs that can be run together to maximize experimentation value. Also, as part of building a solid conceptual foundation, the section on Deferreds, a frequent source of confusion and frustration for developers new to event-driven programming, has been expanded into its own chapter with many more examples.

Since the structure and many of the examples have changed, it is hard to give a short and complete enumeration of the differences between this edition and the last. I hope this has given you some idea, though, and I welcome your thoughts and feedback.

Portions of Chapters 2, 3, and 6 were adapted from the author’s chapter on Twisted for The Architecture of Open Source Applications, Volume II under a Creative Commons Attribution 3.0 Unported license. You can find out more about this book at The Architecture of Open Source Applications home page and about this license at the Creative Commons website.

Using Code Examples

This book is here to help you get your job done. In general, if this book includes code examples, you may use the code in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Twisted Network Programming Essentials, Second Edition, by Jessica McKellar and Abe Fettig (O’Reilly). Copyright 2013 Jessica McKellar, 978-1-4493-2611-1.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

Safari® Books Online

Note

Safari Books Online (www.safaribooksonline.com) is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business.

Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.

Safari Books Online offers a range of product mixes and pricing programs for organizations, government agencies, and individuals. Subscribers have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and dozens more. For more information about Safari Books Online, please visit us online.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://oreil.ly/twisted-network-2e.

To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

Twisted was my first-ever experience with open source contribution. I am so grateful that as a naive and clueless intern way back when, Glyph, JP, Itamar, and others patiently guided me through the contribution process and invested their time in making me a core developer for the project. What I’ve learned from this wonderful community continues to influence my open source and software engineering sensibilities and discipline today, and I strive to give back half as much as they’ve given me.

Thank you Christopher Armstrong, Andrew Bennetts, Jean-Paul Calderone, Thomas Herve, Kevin Horn, Laurens Van Houtven, James Knight, Jonathan Lange, Glyph Lefkowitz, Ying Li, Duncan McGreggor, Ashwini Oruganti, David Reid, Allen Short, David Sturgis, Kevin Turner, and the many other contributors who have helped me and who steward Twisted, support new contributors, help users, write code, write documentation, write tests, and maintain the infrastructure for Twisted. It is truly a group effort.

Thank you Adam Fletcher and Laurens Van Houtven for providing technical reviews for this edition. I appreciate your tolerance for my propensity for deadline-driven development. Your feedback has made this book much stronger. Thank you to my editor Meghan Blanchette, whose stuck with and pushed me patiently as at each deadline I tried to creep in one...last...tweak...I promise.