Python Web Scraping Cookbook

How to do it

Let's go and get the job description from this page.  We will simply retrieve the contents in this recipe.  We will clean it up in the next recipe.

The full code for this example is in the 07/12_scrape_job_stackoverflow.py file.  Let's walk through it:

1. First we read the file:

with open("spacex-job-listing.txt", "r") as file:    content = file.read()

2. Then we load the content into a BeautifulSoup object, and retrieve the <script type="application/ld+json"> tag:

bs = BeautifulSoup(content, "lxml")script_tag = bs.find("script", {"type": "application/ld+json"})

3. Now that we have that tag, we can load its contents into a Python dictionary using the json library:  

job_listing_contents = json.loads(script_tag.contents[0])print(job_listing_contents ...