Programming Google App Engine, 2nd Edition

Chapter 12. Large Data and the Blobstore

App Engine limits the sizes of chunks of data that can be passed between the clients, application instances, and services. Incoming requests and outgoing responses cannot exceed 32 megabytes, datastore entities and memcache values cannot contain more than 1 megabyte, and services limit API calls and responses to sizes ranging between 1 and 32 megabytes, depending on the service. These limits help App Engine maintain a responsive infrastructure at a large scale. Google has managed to increase some of these limits by improving and tuning its infrastructure, but the role of limits remains.

1 megabyte data entities and 32 megabyte requests and responses are generous for many purposes, but insufficient for others. Images, audio, video, and large data applications such as scientific computing often involve transmitting and manipulating larger units. Even if most of your user-oriented transactions are small, administrative tasks such as backups and data aggregation can get hefty.

The App Engine Blobstore is a specialized service for creating, manipulating, and serving very large amounts of data. The size of a single Blobstore value is unlimited. (Or rather, it’s limited only by your budget.) Your app can accept a large file as a user upload, serve a Blobstore value in response to a request, create a new Blobstore value, or append to or delete an existing one. An app can also read from a Blobstore value, a portion at a time.

For the same reasons ...