O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 10. Understanding Scrapy's Performance

Generally, it's easy to get performance wrong. With Scrapy, it's not just easy—it's almost certain because there are quite a few counterintuitive behaviors. Unless you have a good understanding of Scrapy's internals, you will find yourself working hard, optimizing performance while getting zero gains. That is part of the complexity of working with high-performance, low-latency, and highly-concurrent environments. Amdahl's law still holds true while optimizing bottleneck performance, but unless you identify the real bottleneck, optimizations on any other part of the system will not increase the number of items you scrape per second (throughput). More intuition can be gained by reading the classic ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required