O'Reilly logo

Learning Storm by Anand Nalya, Ankit Jain

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Producing the Apache log in Kafka

As explained in Chapter 4, Storm and Kafka Integration, Kafka is a distributed messaging queue and can integrate with Storm very well. In this section, you'll see how to write a Kafka producer that will read the server logfile and produce the log in Kafka.

As we all know, Storm provides guaranteed message processing, which means every message that enters the Storm topology will be processed at least once. In Storm, data loss is possible only at the spout. This happens if the processing capacity of the Storm spout is less than the producing capacity of the data publisher. Hence, to avoid data loss at the Storm spout, we will generally publish the data into a messaging queue, and the Storm spout will use that messaging ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required