Building the Realtime User Experience

Chapter 5. Taming the Firehose with Tornado

In the previous chapter, we built an application that gets the data to users as quickly as possible. However, that application is limited by how fast the authors can post new content. This chapter is about speed and about handling huge amounts of data. We’re going to build an application that shows off just how fast the realtime web can work. We’ll be combining one of the best sources of big data that is publicly available with a truly great piece of software designed for creating realtime web experiences.

Using Twitter’s streaming APIs, we’re going to build an application to make sense of the firehose of information they make publicly available. Instead of Cometd, this example will use a web server and framework called Tornado. Tornado is a set of Python libraries that were originally created by FriendFeed and open sourced shortly after they were acquired by Facebook.

Tornado

Tornado is a really fantastic piece of software that is both a web framework and a nonblocking web server written entirely in Python. The nonblocking architecture of the web server allows it to scale up to handle thousands of simultaneous connections. On top of being able to handle so many connections, it can also keep them open for long periods of time, making it perfectly suited for realtime web applications. The server was built specifically for the realtime aspects of FriendFeed, and it enables each user to maintain an active connection to the server. ...