Data storage

Since there are two distinct layers in this pattern, storage choices will likely be different since the two layers are drastically different in their data requirements. The batch layer requires extreme scalability and should perform well for a high number of concurrent reads during batch processing. The speed layer, on the other hand, doesn't need to store as much data but should be extremely fast for both reads and writes.

In many examples of this pattern, you'll see references to Hadoop Filesystem (HDFS) for storing historical data and NoSQL databases for real-time data. While it's near impossible to say what you should pick, it is possible to speak to some of your options.

Cloud storage systems such as AWS S3 or Google Cloud ...

Get Serverless Design Patterns and Best Practices now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.