Failed page requests can be easily handled by Scrapy using retry middleware. When installed, Scrapy will attempt retries when receiving the following HTTP error codes:
[500, 502, 503, 504, 408]
The process can be further configured using the following parameters:
- RETRY_ENABLED (True/False - default is True)
- RETRY_TIMES (# of times to retry on any errors - default is 2)
- RETRY_HTTP_CODES (a list of HTTP error codes which should be retried - default is [500, 502, 503, 504, 408])
