Sink groups

In order to remove single points of failure in your data processing pipeline, Flume has the ability to send events to different sinks using either load balancing or failover. In order to do this we need to introduce a new concept called a sink group. A sink group is used to create a logical grouping of sinks. The behavior of this grouping is dictated by something called the sink processor, which determines how events are routed.

There is a default sink processor that contains a single sink that is used whenever you have a sink that isn't part of any sink group. Our Hello World example in Chapter 2, Flume Quick Start, used the default sink processor. No special configuration is necessary for single sinks.

In order for Flume to know about ...

Get Apache Flume: Distributed Log Collection for Hadoop now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.