Moving Hadoop to the Cloud

Appendix C. Monitoring Cloud Clusters with Nagios

Nagios is a popular system for monitoring infrastructure. It can monitor networks, hardware, and applications using its built-in capabilities along with a plug-in architecture.

Nagios is more than capable of monitoring cloud clusters for you, and a great choice if you are already familiar with the tool. It can be configured with custom checks that work with cloud provider infrastructures, Hadoop components, or anything else you can think of.

Tip

Code is available at this book’s code repository.

Where Nagios Should Run

As discussed in “Hadoop Daemon Status”, there are benefits and drawbacks for running a monitoring system within a cloud provider or outside it. These considerations apply to where Nagios runs as well. If you opt to run Nagios outside the cloud provider and outside a VPN with privileged access to the network where the Hadoop cluster runs, you must then loosen security rules to permit Nagios to reach all of the ports necessary for effective cluster monitoring. Also, unless you assign static public IP addresses to your instances, you will need to edit the Nagios configuration as those addresses change over time.

Instance Existence Through Ping

Nagios normally checks if instances exist by attempting to ping them. The security rules set up in your cloud provider may block ping traffic, especially if it is running outside the cloud provider. You have the option of redefining the standard existence check, but you ...