What the Search-Ranking Webbot Does
This webbot (actually a specialized spider) submits a search term to a search web page and looks for the subject web page in the search results. If the webbot finds the subject web page within the organic search results, it reports the web page’s ranking. If, however, the webbot doesn’t find the subject in the organics on that page, it downloads the next page of search results and tries again. The webbot continues searching deeper into the pages of search results until it finds a link to the subject web page. If the webbot can’t find the subject web page within a specified number of pages, it will stop looking and report that it could not find the web page within the number of result pages searched.
Get Webbots, Spiders, and Screen Scrapers, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
