O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Interfacing databases with standard Python clients

There are many important databases, including MySQL, PostgreSQL, Oracle, Microsoft SQL Server, and SQLite, that adhere to the Python Database API Specification 2.0. Their drivers are often complex and very well tested, and it would be a big waste if they had to be reimplemented for Twisted. One can use these database clients in Twisted applications, such as Scrapy using the twisted.enterprise.adbapi library. We will use MySQL as an example to demonstrate its usage, but the same principles apply to any other compliant database.

A pipeline that writes to MySQL

MySQL is a great and very popular database. We will write a pipeline that writes items to it. We already have a MySQL instance running on our ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required