Fast Data Architectures for Streaming Applications

Chapter 7. Where to Go from Here

Fast data is the natural evolution of big data to be stream oriented and quickly processed, while still enabling classic batch-mode analytics, data warehousing, and interactive queries.

Long-running streaming jobs raise the bar for a fast data architecture’s ability to stay resilient, scale up and down on demand, remain responsive, and be adaptable as tools and techniques evolve.

There are many tools with various levels of support for sophisticated stream processing semantics, other features, and deployment scenarios. I didn’t discuss all the possible engines. I omitted those that appear to be declining in popularity, such as Storm and Samza, as well as newer but still obscure options. There are also many commercial tools that are worth considering. However, I chose to focus on the current open source choices that seem most important, along with their strengths and weaknesses.

I encourage you to explore the links to additional information throughout this report. Form your own opinions and let me know what you discover.¹ At Lightbend, we’ve been working hard to build tools, techniques, and expertise to help our customers succeed with fast data. Please take a look.

Additional References

Besides the links throughout this report, the following references are very good for further exploration:

1. Justin Sheehy, “There Is No Now,” ACM Queue, Vol. 13, Issue 3, March 10, 2015, https://queue.acm.org/detail.cfm?id=2745385.

2. Jay Kreps, I Heart Logs, September ...