O'Reilly logo

Learning Scrapy by Dimitrios Kouzis-Loukas

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Interfacing services using Twisted-specific clients

Until now, we saw how to use REST-like APIs using treq. Scrapy can interface with many other services using Twisted-specific clients. For example, if we want to interface MongoDB, and we search for "MongoDB Python", we will get PyMongo, which is blocking/synchronous and shouldn't be used with Twisted unless we use threads as described in the pipeline that handle blocking operations in a later section. If we search for "MongoDB Twisted Python", we get txmongo, which is perfectly fine to use with Twisted and Scrapy. Usually, the communities behind Twisted clients are smaller, but this is still a better option than writing our own client. We will use such a Twisted-specific client to interface with ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required