Scrapy will cache requests using a middleware module named HttpCacheMiddleware. Enabling it is as simple as configuring the HTTPCACHE_ENABLED setting to True:
process = CrawlerProcess({ 'AUTOTHROTTLE_TARGET_CONCURRENCY': 3})process.crawl(Spider)process.start()