Logstash

Logstash is a lightweight, open source data processing pipeline. It allows collecting data from a wide variety of sources, transforming it on the fly, and sending it to any desired destination.

It is most often used as a data pipeline for Elasticsearch, a popular analytics and search engine. Logstash is a popular choice for loading data into Elasticsearch because of its tight integration, powerful log processing capabilities, and over 200 prebuilt open source plugins that can help you get your data indexed the way you want it.

The following is a structure of Logstash.conf:

input {...}filter {...}output {..}

Installation of Logstash:

$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.1.2.rpm$ sudo rpm --install logstash-6.1.2.rpm ...

Get Modern Big Data Processing with Hadoop now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.