O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

UR2IM – the fundamental scraping process

Every website is different, and you will certainly need to do some extra study, or ask some questions on the Scrapy mailing list if something is unusual. However, what is important in order to know where and how to search is to have an overview of the process, and know the related terminology. While working with Scrapy, the general process that you most often follow is the UR2IM process.

UR2IM – the fundamental scraping process

The UR2IM process

The URL

It all starts with a URL. You will need a few example URLs from the site you want to scrape. I'm going to demonstrate this using the Gumtree classifieds site (https://www.gumtree.com/) as an example. ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required