O'Reilly logo

Storm Real-time Processing Cookbook by Quinton Anderson

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Indexing and persisting the log data

Log data needs to be stored for some defined period of time in order to be useful; it also needs to be searchable. In order to achieve this, the recipe integrates with an open source product call Elastic Search, which is a general-use, clustered search engine with a RESTful API (http://www.elasticsearch.org/).

How to do it…

  1. Create a new BaseRichBolt class called IndexerBolt and declare the org.elasticsearch.client.Client client as a private member variable. You must initialize it as follows within the prepare method:
    if((Boolean)stormConf.get(backtype.storm.Config.TOPOLOGY_DEBUG) == true){ node = NodeBuilder.nodeBuilder().local(true).node(); } else { String clusterName = (String) stormConf.get(Conf.ELASTIC_CLUSTER_NAME); ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required