There's more...

While it is possible to set any content you want in the User-Agent header, some web servers will inspect the User-Agent header and make decisions on how to respond based upon the content. A common example of this is using the header to identify mobile devices to provide a mobile presentation.

But some sites also only allow access to content to specific User-Agent values. Setting your own value could have the effect of having the web server not respond or return other errors, such as unauthorized. So when you use this technique, make sure to check it will work.

Get Python Web Scraping Cookbook now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.