Storing configuration in the environment 

This recipe points out a change made in the code of the API in the previous recipe to support one of the factors of a 12-Factor application.  A 12-Factor app is defined as an app that is designed to be run as a software as a service.  We have been moving our scraper in this direction for a while now, breaking it into components that can be run independently, as scripts, or in containers, and as we will see soon, in the cloud.  You can learn all about 12-Factor apps at https://12factor.net/.

Factor-3 states that we should pass in configuration to our application through environment variables.  While we definitely don't want to hardcode things, such as URLs, to external services, it also isn't best ...

Get Python Web Scraping Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.