O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

A spider that uses JSON APIs and AJAX pages

Sometimes, you will find yourself exploring pages with data that you'll be unable to find on the HTML of the page. For example, if you visit http://localhost:9312/static/ and right-click inspect element (1, 2) somewhere in the page you will see the DOM tree with all the usual HTML elements. On the other hand, if you use scrapy shell or right-click on View Page Source (3, 4) in Chrome, you will see that the HTML code for this page doesn't contain any information relevant to properties. Where does this data come from?

A spider that uses JSON APIs and AJAX pages

Request and Response on pages that load JSON objects dynamically

In these cases your next ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required