O'Reilly logo

Python Penetration Testing Cookbook by Rejah Rehim

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

How to do it...

Let's create a simple spider with the Scrapy:

  1. For creating a new spider project, open the Terminal and go to the folder for our spider:
$ mkdir new-spider
$ cd new-spider  
  1. Then run the following command to create a new spider project with scrapy:
$ scrapy startproject books  

This will create a project with the name books and some useful files for creating the crawler. Now you have a folder structure, as shown in the following screenshot:

  1. Now we can create a crawler with the following command:
$ scrapy genspider home books.toscrape.com  

This will generate the code for the spider with the name home, as we are planning to ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required