We start by importing Selector from scrapy, and also requests so that we can retrieve the page:
In [1]: from scrapy.selector import Selector ...: import requests ...:
Next we load the page. For this example we are going to retrieve the most recent questions on StackOverflow and extract their titles. We can make this query with the the following:
In [2]: response = requests.get("http://stackoverflow.com/questions")
Now create a Selector and pass it the response object:
In [3]: selector = Selector(response) ...: selector ...:Out[3]: <Selector xpath=None data='<html>\r\n\r\n <head>\r\n\r\n <title>N'>
Examining the content of this page we can see that questions have the following structure to their HTML:
The HTML of a StackOverflow ...
