Cover by Eric Sammer

Safari, the world’s most comprehensive technology and business learning platform.

Find the exact information you need to solve a problem on the fly, or go deeper to master the technologies and skills you need to succeed

Start Free Trial

No credit card required

O'Reilly logo

Chapter 10. Monitoring

An Overview

It’s hard to talk about building large shared, mission-critical systems without having a way to know their operational state and performance metrics. Most organizations (I hope) have some form of monitoring system that keeps track of various systems that occupy the data center. No one runs a large Hadoop cluster by itself, and while a lot of time is spent on data integration, monitoring integration sometimes falls by the wayside.

Most monitoring systems can be divided into two major components: metric collection and consumption of the resultant data. Hadoop is another source from which metrics should be collected. Consumption of the data can mean presenting aggregate metrics as dashboards, raw metrics as time series data for diagnoses and analysis, and very commonly, rule evaluation for alerting. In fact, many monitoring systems provide more than one of these features. It helps to further divide monitoring into two distinct types: health monitoring, where the goal is to determine that a service is in an expected operational state, and performance monitoring, where the goal is to use regular samples of performance metrics, over time, to gain a better understanding of how the system functions. Performance monitoring tends to be harder to cover outside of the context of a specific environment and set of workloads, so instead we’ll focus primarily on health monitoring.

Hadoop, like most distributed systems, constitutes a monitoring challenge because the ...

Find the exact information you need to solve a problem on the fly, or go deeper to master the technologies and skills you need to succeed

Start Free Trial

No credit card required