How it works

We will get into some details about Scrapy in later chapters, but let's just go through this code quick to get a feel how it is accomplishing this scrape.  Everything in Scrapy revolves around creating a spider.  Spiders crawl through pages on the Internet based upon rules that we provide.  This spider only processes one single page, so it's not really much of a spider.  But it shows the pattern we will use through later Scrapy examples.

The spider is created with a class definition that derives from one of the Scrapy spider classes.  Ours derives from the scrapy.Spider class.

class PythonEventsSpider(scrapy.Spider):    name = 'pythoneventsspider'    start_urls = ['https://www.python.org/events/python-events/',]

Every spider is given ...

Get Python Web Scraping Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.