Rapid Web Applications with TurboGears: Using Python to Create Ajax-Powered Sites

Chapter 11.4.1. Don’t list() a Large Result Set. Use Slices.

Listing a large result set will bring it all to memory (or virtual memory). This is often not what you want: maybe you don’t have enough memory, it takes longer to respond, and finally by the time you actually process some Nth row, it might already be stale. A much better approach is to use slices and process large result sets in manageable chunks. The following code demonstrates this:

class Bigwell(SQLObject): robot = StringCol() Bigwell.dropTable(ifExists=True) Bigwell.createTable() for i in range (1, 101): Bigwell(robot='Robot #%d' % i) # Getting the lazy SearchResults iterator (no DB access yet) sr = Bigwell.select() # List()ing the entire result set robots = list(sr) for r ...