You can think of a queue as a broker of messages. Some data producer will place messages onto the queue, and some data consumer will read those messages. The queue simply brokers that exchange of message passing. Stream processing systems provide similar functionality, but with much different behavior, features, and applications. I'll present a brief discussion of the differences between queues and streams for the sake of clarity.
Apache Kafka is a very popular stream processing system in widespread use, of which you can have heard. Today, cloud providers have come out with hosted stream processing systems:
- Azure: Event Hubs
- AWS: Kinesis
- Google Compute Cloud: Cloud Dataflow
So, what exactly is a stream processing system as ...
