For More Information
For more information on web clients, refer to:
- http://www.robotstxt.org/wc/robots.html
The Web Robots Pages—resources for robot developers, including the registry of Internet Robots.
- http://www.searchengineworld.com
Search Engine World—resources for search engines and robots.
- http://www.searchtools.com
Search Tools for Web Sites and Intranets—resources for search tools and robots.
- http://search.cpan.org/doc/ILYAZ/perl_ste/WWW/RobotRules.pm
RobotRules Perl source.
- http://www.conman.org/people/spc/robots2.html
An Extended Standard for Robot Exclusion.
- Managing Gigabytes: Compressing and Indexing Documents and Images
Witten, I., Moffat, A., and Bell, T., Morgan Kaufmann.
Get HTTP: The Definitive Guide now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
