O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

A 30-times faster property spider

There is a tendency when you start with a framework to use, maybe, the most sophisticated and, typically, the most complex way for anything you do. You will likely find yourself doing that with Scrapy too. Just before you go crazy with XPath and technology, it is worth to pause for a moment and wonder; is the way I chose the easiest way to extract data from this website?

You can have orders-of-magnitude savings if you avoid scraping every single listing page if you can extract about the same information from index pages.

Tip

Please keep in mind that many websites offer a different number of items on their index pages. For example, a website might be able to give you 10, 50 or 100 listings per index page by tuning ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required