The standard performance model holds true when Scrapy is functioning properly and the downloader is the performance bottleneck. In this case, you will see some requests in the scheduler, and the maximum number of concurrent requests in the downloader. The scraper (spider and pipelines) will be lightly loaded and the number of
Responses in progress will not be constantly increasing.
There are three main settings that control the downloader's capacity:
CONCURRENT_REQUESTS_PER_IP. The first one ...