It almost goes without saying that for many users, any risk of dropped records or data loss in their data-processing pipelines unacceptable. Even so, historically many general-purpose streaming systems made no guarantees about record processing - all processing was “best effort” only. Other systems provided at-least-once guarantees, ensuring that records were always processed at least once, but records might be duplicated (and thus result in inaccurate aggregations); in practice, many such at-least-once systems performed aggregations in memory, and thus their aggregations could still be lost when machines crashed. These systems were used for low-latency, speculative results, but generally could guarantee nothing about the ...
Share this highlighthttp://learning.oreilly.com/a/streaming-systems/17258317/