The preceding Logstash configuration will allow us to ingest network logs and create the index on Elasticsearch. What would happen if the text format we intend on putting into ELK is not a standard log format? This is where Python can help. In the next example, we will perform the following:
- Use the Python script to retrieve a list of IPs that the Spamhaus project considers to be a drop list (https://www.spamhaus.org/drop/drop.txt)
- Use the Python logging module to format the information in such a way that Logstash can ingest it
- Modify the Logstash configuration file so any new input could be sent to the AWS Elasticsearch Service
The chapter8_logstash_1.py script contains the code we will use. ...
