O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 5. Quick Spider Recipes

In Chapter 3, Basic Crawling, we focused on how to extract information from pages and store them into Items. What we learned covers the most common Scrapy use cases, and it should be enough to get you up and running. In this chapter, we will examine more specialized cases in order to become more familiar with the two most important Scrapy classes—Request and Response—the two R's on the UR2IM scraping model we presented in Chapter 3, Basic Crawling.

A spider that logs in

Quite often, you may find yourself wanting to extract data from websites that have a login mechanism. In the most common case, a website will require you to provide a username and a password in order to log in. We are going to use the example that ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required