References

D. Boswell "Distributed High-performance Web Crawlers: A Survey of the State of the Art." 2003. http://www.cs.ucsd.edu/~dboswell/PastWork/WebCrawlingSurvey.pdf.

D. Gomes and M. Silva. "The Viuva Negra crawler: An experience report." Software—Practice & Experience, Volume 38 (2), pp. 161–188, 2006.

A. Heydon and M. Najork. "Mercator: A Scalable, Extensible Web Crawler." Compaq Systems Research Center, 1999.

S. Chakrabarti M. Berg, and B. Dom. "Focused crawling: a new approach to topic-specific Web resource discovery." WWW8 International World Wide Web Conference, vol. 31, pp. 1623–1640. Toronto, 1999. http://www8.org/w8-papers/5a-search-query/crawling/index.html.

V. Shkapenyuk and T. Suel. "Design and Implementation of a High-Performance ...

Get Algorithms of the Intelligent Web now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial