Storm Blueprints: Patterns for Distributed Real-time Computation

Summary

In this chapter, we gained a deeper appreciation for the Trident State API. We created a direct implementation of the State and StateUpdater interfaces instead of relying on default implementations. Specifically, we implemented these interfaces to bridge the gap between a transactional spout and a non-transactional system, namely Druid. Although it is impossible to establish exactly-once semantics into a non-transactional store, we put mechanisms in place to alert when the system encounters issues. Ostensibly, upon failure we could then use a batch processing mechanism to reconstruct any suspect aggregation segments.

For future investigation, it would be beneficial to establish an idempotent interface between Storm and Druid. To do this, ...