In day-to-day business scenarios, we always find the need to get log files and make sense out of them. For example, we always find the need to get logs from different applications and servers and merge them together to find trends and patterns. Let me extend this example further. Let's assume that we have five web servers deployed on five different servers. We want to get all five web server logs and merge/aggregate them together to analyze them further by storing one copy on HDFS and another copy to be shipped on to a Kafka topic for real-time analytics. The question is how we design Flume-based log aggregation architecture. The following is the Flume architecture for our web server log aggregation scenario:
Let ...
