Why?

There are a number of reasons why one might opt to process records individually or at least in smaller chunks. Three of the most compelling reasons are:

  • Limits of computing resources
  • Network latency
  • Pareto's principle

From the perspective of efficiency, any one of these reasons is good enough to warrant retrieval of smaller amounts of data. We look at each in greater detail in the following section.

Computing resources

Despite Moore's law holding true for many years now, it is coming under increasing pressure as a trend and cannot be presumed to hold true indefinitely. While one may have a budget with numbers so big that it looks like a phone directory, one's resources are only faster or more powerful on a comparative scale.

Note

Moore's law states ...

Get MySQL for Python now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.