Getting ready...

There are several means of creating a scraper with Scrapy.  One is a programmatic pattern where we create the crawler and spider in our code.  It is also possible to configure a Scrapy project from templates or generators and then run the scraper from the command line using the scrapy command.  This book will follow the programmatic pattern as it contains the code in a single file more effectively.  This will help when we are putting together specific, targeted, recipes with Scrapy. 

This isn't necessarily a better way of running a Scrapy scraper than using the command line execution, just one that is a design decision for this book.  Ultimately this book is not about Scrapy (there are other books on just Scrapy), but more ...

Get Python Web Scraping Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.