O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

A Scrapy project

Up to now, we were "playing" with Scrapy shell. Now we have all the necessary ingredients to start our first Scrapy project, and we can quit Scrapy shell by hitting Ctrl + D. Notice that everything you might have typed up to now gets lost. Obviously, we don't want to type the code each time we want to crawl something, so it's important to remember that the Scrapy shell is just a utility to help us play with pages, XPath expressions, and Scrapy objects. Don't invest much time in writing complicated code there, because it's bound to get lost as soon as you exit. In order to write real Scrapy code, we use projects. Let's create a Scrapy project and name it "properties", since we are scraping real estate properties.

$ scrapy startproject ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required