Flume setup

Flume agent configuration is stored in a local text file. Please refer to the sample Flume agent configuration file in the code repository of this book. Flume 1.7.0 supports various sources and sinks. Widely used Flume sources (a summary) are as follows:

Source Description
Avro source Listens on Avro port and receives events from external Avro client streams
Exec source Runs a given Unix command and expects that process to continuously produce data on standard out
Spooling directory source Ingests data from files on disk
Taildir source Tails files in near real-time after new lines are detected in the files
Kafka source Reads messages from Kafka topics
Syslog source Reads syslog data (supports syslog-TCP and syslog-UDP) ...

Get Modern Big Data Processing with Hadoop now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.