Our batch layer gets a bit more interesting. Some of the details are AWS-centric, but you can carry the general idea across cloud providers. AWS Kinesis Firehose is another version of Kinesis that is designed to transport data in batches to various locations. In this architecture, we'll set up the Kinesis Firehose stream to ingest data from the primary Kinesis stream. I'll also configure the Firehose stream to deliver batches of messages to S3 every minute.
If you have heard about AWS Kinesis but the term Kinesis Firehose is new to you, don't worry. Kinesis Firehose's specialty is loading data into various services, such as S3 and Redshift. A plain Kinesis stream captures data and makes it available for consumption, but that ...
