The data lake is a valuable source of information. Elasticsearch is uniquely suited for indexing this coarse-grained time series information. Kibana is the data visualization plugin for Elasticsearch. Kibana is a great tool for creating dashboards containing statistics about the events in the data lake and to perform ad hoc searches to troubleshoot system problems based on the contents of the events.
In this recipe, we are using Kinesis Firehose because it performs the heavy lifting of writing the events to Elasticsearch. It provides buffering based on time and size, hides the complexity of the Elasticsearch bulk index API, provides error handling, and supports index rotation. In the custom elasticsearch Serverless plugin, ...
