Big data patterns

It may seem confusing that lambda can refer to both AWS Lambda functions as well as a pattern in and of itself. The lambda pattern was born from the need to analyze large amounts of data in real time. Before this, the big data movement, where large batch jobs would run to calculate and recalculate things, was in full swing. The problem faced by this movement was that these batch jobs, in order to get the latest results, would need to spend the majority of their computing resources recalculating metrics on data that hadn't changed.

The lambda pattern, which we will discuss in Chapter 7, Data Processing Using the Lambda Pattern, creates two parallel planes of computation, a batch layer, and a speed layer. The naming of these ...

Get Serverless Design Patterns and Best Practices now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.