Hadoop Essentials

Examples of configuring Flume

Flume can be configured as a Single Agent or Multi Agent; we will see the respective examples in the upcoming sections.

The Single agent example

We will look at an example of the logger example and save it in HDFS and a memory channel, using the following code:

# Source of an Agent with tail agent.source = source_log-tail agent.sources.source_log-tail.type = exec agent.sources.source_log-tail.command = tail -F /log/logger.log agent.sources.source_log-tail.channels = memoryChannel # Sink of an Agent to save in HDFS agent.sinks = log-hdfs agent.sinks.log-hdfs.channel = memoryChannel agent.sinks.log-hdfs.type = hdfs agent.sinks.log-hdfs.hdfs.path = /log/logger.log # Channel of an Agent to store in memory agent.channels ...