The implementation of HTTP caching is simple, yet complex at the same time. The HttpCacheMiddleware provided by Scrapy has a plethora of configuration options based upon your needs. Ultimately, it comes down to storing each URL and its content in a store along with an associated duration for cache expiration. If a second request is made for a URL within the expiration interval, then the local copy will be retrieved instead of making a remote request. If the time has expired, then the contents are fetched from the web server, stored in the cache, and a new expiration time set.
How it works
Get Python Web Scraping Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
