O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Overview of Scrapy architecture

The following diagram summarizes Scrapy's architecture:

Overview of Scrapy architecture

Scrapy's architecture

You may notice three familiar types of objects upon which this architecture operates; Requests, Responses, and Items. Our spiders lie right at the core of the architecture. They create Requests, process Responses, and generate Items and more Requests.

Each Item generated by a spider is postprocessed by a sequence of Item Pipelines using their process_item() method. Typically, process_item() modifies Items and passes them to the subsequent pipelines by returning them. Occasionally (for example, in the case of a duplicate or invalid data), we ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required