Using a message broker
The traffic on production servers is supposed to be high at times. When this happens, log entries and statistics data become very critical and the amount of such data in total is also high. All of the Beats will be doing their work and will be sending respective data to Elasticsearch, but it is possible that some of the packets/data are lost while processing. It may happen because of a network failure, very high peaks of data, or any other possible reason. The point is, the data being indexed must not be lost in any case.
To address this problem, using a message broker or buffer is a good choice. There are many tools that can be evaluated for your choice of the message broker. For Open Source, there are two good tools available: ...
Get Mastering Elastic Stack now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
