Hadoop: Data Processing and Modelling

Time for action – multi level Flume networks

Let's put together a few pieces we touched on earlier and see how one Flume agent can use another as its sink.

1. Create the following file as agent6.conf:

   agent6.sources = avrosource agent6.sinks = avrosink agent6.channels = memorychannel agent6.sources.avrosource.type = avro agent6.sources.avrosource.bind = localhost agent6.sources.avrosource.port = 2000 agent6.sources.avrosource.threads = 5 agent6.sinks.avrosink.type = avro agent6.sinks.avrosink.hostname = localhost agent6.sinks.avrosink.port = 4000 agent6.channels.memorychannel.type = memory agent6.channels.memorychannel.capacity = 1000 agent6.channels.memorychannel.transactionCapacity = 100 agent6.sources.avrosource.channels = memorychannel agent6.sinks.avrosink.channel ...