There are many options for configuration scrapy caching, including means of storing content (file system, DBM, or LevelDB), cache policies, and how Http Cache-Control directives from the server are handled. To explore these options, check out the following URL: https://doc.scrapy.org/en/latest/topics/downloader-middleware.html?_ga=2.50242598.1404351387.1507758575-507079265.1505263737#dummy-policy-default.
There's more...
Get Python Web Scraping Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.