Python Web Scraping Cookbook

There's more...

The decision to perform the search from within the API implementation is one that should be considered as a system evolves.  It is my opinion, and just mine (but I think others would agree), that like how the API calls a microservice for the actual scraping, it should also call a microservice that handles the search (and that microservice would then interface with Elasticsearch).  This would also be the case for storing the document returned from the scraping microservice, as well as accessing Elasticsearch to check for a cached document.  But for our purposes here, we'll try and keep it simple.