Setting the user agent differs depending upon what tools you use. Ultimately, it is just ensuring that the User-Agent header is set to a string that you specify. When using a browser, this is normally set by the browser to identity the browser and the operating system. But you can put anything you want into this header. When using requests, it is very straightforward:
url = 'https://api.github.com/some/endpoint' headers = {'user-agent': 'MyCompany-MyCrawler (mybot@mycompany.com)'} r = requests.get(url, headers=headers)
When using Scrapy, it is as simple as configuring a setting:
process = CrawlerProcess({ 'USER_AGENT': 'MyCompany-MyCrawler (mybot@mycompany.com)'})process.crawl(Spider)process.start()
