Internet Forensics

Web Server Statistics

Thus far I have discussed HTTP headers as a means to learn about an individual server. By taking a broader view and looking at them across many sites, it is possible to derive interesting statistics on the Internet at large.

Netcraft, a U.K.-based Internet services company, has built much of its reputation by looking at HTTP headers. Since 1995 they have been archiving the headers and other information from web servers around the world. This has allowed them to track a number of important statistics about the Internet such as the relative popularity of different web servers, operating systems, ISPs, and so forth.

Their survey of the market share of each major type of web server has long been used to highlight the impact that the Apache server and, by association, open source software (OSS), have had on the development of the Internet. The current version of this survey is available at http://news.netcraft.com/archives/web_server_survey.html. Figure 6-2, taken from that survey, shows how Apache dominates the field and continues to grow in importance.

Netcraft provides an excellent Frequently Asked Questions (FAQ) page that details how they capture their data (http://uptime.netcraft.com/up/accuracy.html).

As well as their impressive summary statistics, Netcraft also makes available information on individual sites. You can see what they know about any specific site by visiting http://uptime.netcraft.com/up/graph and entering its URL. With most small sites, the information ...