Many-to-Many Environment
The many-to-many environment, as shown in Figure 25-3, is the classic multiwebbot configuration that all search engines use.
Figure 25-3. A many-to-many geometry
In the many-to-many environment, a team of webbots harvests data from several target websites. The team is comprised of multiple instances of the webbot script working in a managed environment to achieve a common goal. This is essentially how to scale a one-to-many environment. You simply apply additional webbot resources as the number of targets grows. Since you’re gathering information from multiple targets, issues relating to overusing a single source don’t apply. ...
Get Webbots, Spiders, and Screen Scrapers, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
