O'Reilly logo

Python Network Programming Cookbook by Dr. M. O. Faruque Sarker

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Crawling links present in a web page

At times you would like to find a specific keyword present in a web page. In a web browser, you can use the browser's in-page search facility to locate the terms. Some browsers can highlight it. In a complex situation, you would like to dig deep and follow every URL present in a web page and find that specific term. This recipe will automate that task for you.

How to do it...

Let us write a search_links() function that will take three arguments: the search URL, the depth of the recursive search, and the search key/term, since every URL may have links present in the content and that content may have more URLs to crawl. To limit the recursive search, we define a depth. Upon reaching that depth, no more recursive ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required