Use of scrapy-proxies is done by configuration. It starts by configuring DOWNLOADER_MIDDLEWARES, and making sure they have RetryMiddleware, RandomProxy, and HttpProxyMiddleware installed. The following would be a typical configuration:
# Retry many times since proxies often failRETRY_TIMES = 10# Retry on most error codes since proxies fail for different reasonsRETRY_HTTP_CODES = [500, 503, 504, 400, 403, 404, 408]DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90, 'scrapy_proxies.RandomProxy': 100, 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,}
The the PROXY_LIST setting is configured to point to a file containing a list of proxies:
PROXY_LIST = '/path/to/proxy/list.txt'
