Python Web Scraping Cookbook

There's more...

The script also defines code for when executing the script directly with Python:

if __name__ == "__main__":    process = CrawlerProcess({        'DOWNLOAD_DELAY': 5,        'RANDOMIZED_DOWNLOAD_DELAY': False,        'LOG_LEVEL': 'DEBUG'    })    process.crawl(Spider)    process.start()

This begins by creating a CrawlerProcess object. This object can be passed a dictionary representing the settings and values to configure the crawl with. This defaults to a five-second delay, without randomization, and an output level of DEBUG.