We now have a complete pipeline of streaming data that is caught, transformed, and stored. Once you have collected a few thousand tweets, you are ready to analyze your data. There are a couple of things that remain to be done before we can get the answers we set out to find at the beginning of this chapter:
- What are the most popular vegetables on twitter?
- How does TextBlob compare to our Amazon ML classification model?
Our simple producer does not attempt to handle duplicate tweets. However, in the end, our dataset has many duplicate tweets. Broccoli and carrots are less frequent Tweet subjects than one could expect them to be. So, as we collect about a hundred tweets every 10 minutes, many tweets end up being collected ...
