Rapid Web Applications with TurboGears: Using Python to Create Ajax-Powered Sites

Chapter 11.4. Working with Large Result Sets

If you work with databases, you will end up working with large result sets, and you will need to think about performance. How large is large? It depends. Some applications need to access the DB frequently and touch every row or perform complex queries while inserting and deleting rows, so a few thousand rows can be considered a large dataset. Other applications access only a small part of the data in well-known patterns, so the DB can be optimized and even millions of rows will not present a problem.

There are several common performance gotchas:

1. Loading data you don’t need

2. Loading too much data at a time into memory

3. Loading the same data multiple times

SQLObject is an abstraction layer that normally ...