Getting ready

We will crawl a page in the containers web site at the following URL: http://localhost:5001/home/secured.  On this page, and links from that page, there is content we would like to scrape. However, this page is blocked by a login. When opening the page in a browser, we are presented with the following login form, where we can enter darkhelmet as the user name and vespa as the password:

Username and password credentials are entered

Upon pressing Enter we are authenticated and taken to our originally desired page.

There's not a great deal of content there, but the message is enough to verify that we have logged in, and our scraper ...

Get Python Web Scraping Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.