Webbots, Spiders, and Screen Scrapers, 2nd Edition

Scaling and Denial-of-Service Attacks

An easy and inadvertent mistake many beginning webbot developers make is to launch a denial-of-service attack on websites, where the webbot consumes too much of the target website’s resources and other people are not able to use the website for its intended purposes.^[70]

Even Simple Webbots Can Generate a Lot of Traffic

Writing webbots that interfere with a target website’s performance can happen even without an intentional effort to increase the capacity of a webbot. For example, a single webbot script—without the appropriate delays and other mechanisms that simulate human behavior—is capable of causing much more network activity than most imagine. Even old, outdated equipment can easily consume all the bandwidth ...