We have barely touched many of the capabilities of Docker and Docker Compose, and we have not even yet got into looking at using services such as Docker swarm. While docker Compose is convenient, it only runs the containers on a single host, which ultimately has scalability limitations. Docker swarm will perform similar things to Docker Compose, but work that magic across multiple systems within a cluster, allowing much greater scalability. But hopefully this has given you a feel for the value of Docker and Docker Compose, and how they can be of value when creating a flexible scraping service.
There's more...
Get Python Web Scraping Cookbook now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
