The system architecture for our image analysis example is quite simple. Ingestion of tweets will begin using a stream listener for the Twitter API via the Python tweepy library, https://github.com/tweepy/tweepy. This listener will filter out only specific tweets on our behalf. From there, the listener will place messages onto an SQS queue. Once it delivers messages to the queue, the job of our stream listener is complete. With this type of design, we realize a real separation of concerns. If we enumerated the things our stream listener cares about, the list would be quite short:
- Twitter access
- Some business logic as to what types of data to extract from tweets
- Which queue to place the extracted tweet information in
