System administrators deal with logs on a daily basis. However, it’s only recently that administrators have been realizing the true value of the enormous amount of logs that automatically flow into their systems. Traditionally administrators have looked at logs as something they reviewed during troubleshooting exercises. Now, log files are being increasingly seen as continuous streams of events or information that can yield valuable insights based on underlying data patterns.
Let’s take the case of the ubiquitous HTTP server access logs. These logs contain both error messages (404, page not found) as well as successful transactions. In addition you have data such as client IPs and response time. You can use some of this data to dig deeper into the business processes, for example by getting the geographical locations of your users based on their originating IPs. Logs can be quite useful not only for troubleshooting, but also for understanding user behavior. Organizations use the log data for various purposes such as automating security scanning, and scaling their web services.
System administrators can use log data for debugging, root case analysis and understand customer behavior in terms of their usage or buying patterns. Machine data allows an organization to ask questions that they couldn’t envision earlier to find out answers to fundamental questions about their IT infrastructure.
In this chapter, ...