O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Essential settings

Scrapy has so many settings that categorizing them becomes an urgent necessity. We start with the most essential settings that we summarize in the following diagram. They give you awareness of the important system features and you may find yourself adjusting them somewhat frequently.

Essential settings

Essential Scrapy settings

Analysis

By using those settings you can configure the way Scrapy provides performance and debugging information via logs, statistics, and the telnet facility.

Logging

Scrapy has different levels of logs based on their severity: DEBUG (lowest level), INFO, WARNING, ERROR, and CRITICAL (highest level). Beyond that, there's a

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required