Webbots, Spiders, and Screen Scrapers, 2nd Edition

Chapter 31. Keeping Webbots out of Trouble

By this point, you know how to access, download, parse, and process nearly any of the 386 million websites on the Internet.^[88] Knowing how to do something, however, does not give you the right to do it. While I have cast warnings throughout the book, I haven’t, until now, focused on the consequences of designing webbots or spiders that act selfishly and without regard to the rights of website owners or related infrastructure.^[89]

Since many businesses rely on the performance of their websites to conduct business, you should consider interfering with a corporate website equivalent to interfering with a physical store or factory. When deploying a webbot or spider, remember that someone else is paying for ...