Python Web Scraping Cookbook

How to do it

We proceed with the recipe as follows:

1. scrapy-fake-useragent is available on GitHub at https://github.com/alecxe/scrapy-fake-useragent, and scrapy-random-useragent is available at https://github.com/cnu/scrapy-random-useragent.  You can include them using pip install scrapy-fake-agent and/or  pip install scrapy-random-useragent.
2. scrapy-random-useragent will select a random user agent for each of your requests from a file. It is configured in two settings:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'random_useragent.RandomUserAgentMiddleware': 400
}

3. This disables the existing UserAgentMiddleware, and replaces it with the implementation provided in RandomUserAgentMiddleware ...