You are previewing Learning ELK Stack.
O'Reilly logo
Learning ELK Stack

Book Description

Build mesmerizing visualizations, analytics, and logs from your data using Elasticsearch, Logstash, and Kibana

About This Book

  • Solve all your data analytics problems with the ELK stack

  • Explore the power of Kibana4 search and visualizations built over Elasticsearch queries and learn about the features and plugins of Logstash

  • Develop a complete data pipeline using the ELK stack

  • Who This Book Is For

    If you are a developer or DevOps engineer interested in building a system that provides amazing insights and business metrics out of data sources, of various formats and types, using the open source technology stack that ELK provides, then this book is for you. Basic knowledge of Unix or any programming language will be helpful to make the most out of this book.

    What You Will Learn

  • Install, configure, and run Elasticsearch, Logstash, and Kibana

  • Understand the need for log analytics and the current challenges in log analysis

  • Build your own data pipeline using the ELK stack

  • Familiarize yourself with the key features of Logstash and the variety of input, filter, and output plugins it provides

  • Build your own custom Logstash plugin

  • Create actionable insights using charts, histograms, and quick search features in Kibana4

  • Understand the role of Elasticsearch in the ELK stack

  • In Detail

    The ELK stack—Elasticsearch, Logstash, and Kibana, is a powerful combination of open source tools. Elasticsearch is for deep search and data analytics. Logstash is for centralized logging, log enrichment, and parsing. Kibana is for powerful and beautiful data visualizations. In short, the Elasticsearch ELK stack makes searching and analyzing data easier than ever before.

    This book will introduce you to the ELK (Elasticsearch, Logstash, and Kibana) stack, starting by showing you how to set up the stack by installing the tools, and basic configuration. You’ll move on to building a basic data pipeline using the ELK stack.

    Next, you’ll explore the key features of Logstash and its role in the ELK stack, including creating Logstash plugins, which will enable you to use your own customized plugins. The importance of Elasticsearch and Kibana in the ELK stack is also covered, along with various types of advanced data analysis, and a variety of charts, tables ,and maps.

    Finally, by the end of the book you will be able to develop full-fledged data pipeline using the ELK stack and have a solid understanding of the role of each of the components.

    Style and approach

    This book is a step-by-step guide, complete with various examples to solve your data analytics problems by using the ELK stack to explore and visualize data.

    Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

    Table of Contents

    1. Learning ELK Stack
      1. Table of Contents
      2. Learning ELK Stack
      3. Credits
      4. About the Author
      5. About the Reviewers
      6. www.PacktPub.com
        1. Support files, eBooks, discount offers, and more
          1. Why subscribe?
          2. Free access for Packt account holders
      7. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Conventions
        5. Reader feedback
        6. Customer support
          1. Downloading the example code
          2. Downloading the color images of this book
          3. Errata
          4. Piracy
          5. Questions
      8. 1. Introduction to ELK Stack
        1. The need for log analysis
          1. Issue debugging
          2. Performance analysis
          3. Security analysis
          4. Predictive analysis
          5. Internet of things and logging
        2. Challenges in log analysis
          1. Non-consistent log format
            1. Tomcat logs
            2. Apache access logs – combined log format
            3. IIS logs
          2. Variety of time formats
            1. Decentralized logs
          3. Expert knowledge requirement
        3. The ELK Stack
          1. Elasticsearch
          2. Logstash
          3. Kibana
        4. ELK data pipeline
        5. ELK Stack installation
          1. Installing Elasticsearch
          2. Running Elasticsearch
          3. Elasticsearch configuration
            1. Network Address
            2. Paths
            3. The cluster name
            4. The node name
          4. Elasticsearch plugins
          5. Installing Logstash
          6. Running Logstash
          7. Logstash with file input
          8. Logstash with Elasticsearch output
          9. Configuring Logstash
          10. Installing Logstash forwarder
          11. Logstash plugins
            1. Input plugin
            2. Filters plugin
            3. Output plugin
          12. Installing Kibana
          13. Configuring Kibana
          14. Running Kibana
          15. Kibana interface
            1. Discover
            2. Visualize
            3. Dashboard
            4. Settings
        6. Summary
      9. 2. Building Your First Data Pipeline with ELK
        1. Input dataset
          1. Data format for input dataset
        2. Configuring Logstash input
        3. Filtering and processing input
        4. Putting data to Elasticsearch
        5. Visualizing with Kibana
          1. Running Kibana
          2. Kibana visualizations
          3. Building a line chart
          4. Building a bar chart
          5. Building a Metric
          6. Building a data table
        6. Summary
      10. 3. Collect, Parse and Transform Data with Logstash
        1. Configuring Logstash
        2. Logstash plugins
          1. Listing all plugins in Logstash
          2. Data types for plugin properties
            1. Array
            2. Boolean
            3. Codec
            4. Hash
            5. String
            6. Comments
            7. Field references
          3. Logstash conditionals
          4. Types of Logstash plugins
            1. Input plugins
              1. file
                1. Configuration options
                  1. add_field
                  2. codec
                  3. delimiter
                  4. exclude
                  5. path
                  6. sincedb_path
                  7. sincedb_write_interval
                  8. start_position
                  9. tags
                  10. type
              2. stdin
                1. Configuration options
                  1. add_field
                  2. codec
                  3. tags
                  4. type
              3. twitter
                1. Configuration options
                  1. add_field
                  2. codec
                  3. consumer_key
                  4. consumer_secret
                  5. full_tweet
                  6. keywords
                  7. oauth_token
                  8. oauth_token_secret
                  9. tags
                  10. type
              4. lumberjack
                1. Configuration options
                  1. add_field
                  2. codec
                  3. host
                  4. port
                  5. ssl_certificate
                  6. ssl_key
                  7. ssl_key_passphrase
                  8. tags
                  9. type
              5. redis
                1. Configuration options
                  1. add_field
                  2. codec
                  3. data_type
                  4. host
                  5. key
                  6. password
                  7. port
            2. Output plugins
              1. csv
                1. Configuration options
                  1. codec
                  2. csv_options
                  3. fields
                  4. gzip
                  5. path
              2. file
                1. Configuration options
              3. email
                1. Configuration options
                  1. attachments
                  2. body
                  3. cc
                  4. from
                  5. to
                  6. htmlbody
                  7. replyto
                  8. subject
              4. elasticsearch
                1. Configuration options
              5. ganglia
                1. Configuration options
                  1. metric
                  2. unit
                  3. value
              6. jira
                1. Configuration options
              7. kafka
                1. Configuration options
                  1. topic_id
              8. lumberjack
                1. Configuration options
                  1. hosts
                  2. port
                  3. ssl_certificate
              9. redis
                1. Configuration options
              10. rabbitmq
              11. stdout
              12. mongodb
                1. Configuration options
                  1. collection
                  2. database
                  3. uri
            3. Filter plugins
              1. csv
                1. Configuration options
              2. date
                1. Configuration options
              3. drop
                1. Configuration options
              4. geoip
                1. Configuration options
                  1. source
              5. grok
                1. Custom grok patterns
              6. mutate
                1. Configuration options
              7. sleep
            4. Codec plugins
              1. json
              2. line
              3. multiline
              4. plain
              5. rubydebug
        3. Summary
      11. 4. Creating Custom Logstash Plugins
        1. Logstash plugin management
        2. Plugin lifecycle management
          1. Installing a plugin
          2. Updating a plugin
          3. Uninstalling a plugin
        3. Structure of a Logstash plugin
          1. Required dependencies
          2. Class declaration
          3. Configuration name
          4. Configuration options setting
          5. Plugin methods
            1. Input plugin
            2. Filter plugin
            3. Output plugin
            4. Codec plugin
          6. Writing a Logstash filter plugin
          7. Building the plugin
        4. Summary
      12. 5. Why Do We Need Elasticsearch in ELK?
        1. Why Elasticsearch?
        2. Elasticsearch basic concepts
          1. Index
          2. Document
          3. Field
          4. Type
          5. Mapping
          6. Shard
          7. Primary shard and replica shard
          8. Cluster
          9. Node
        3. Exploring the Elasticsearch API
          1. Listing all available indices
          2. Listing all nodes in a cluster
          3. Checking the health of the cluster
            1. Health status of the cluster
          4. Creating an index
          5. Retrieving the document
          6. Deleting documents
          7. Deleting an index
        4. Elasticsearch Query DSL
        5. Elasticsearch plugins
          1. Bigdesk plugin
          2. Elastic-Hammer plugin
          3. Head plugin
        6. Summary
      13. 6. Finding Insights with Kibana
        1. Kibana 4 features
          1. Search highlights
          2. Elasticsearch aggregations
          3. Scripted fields
          4. Dynamic dashboards
        2. Kibana interface
          1. Discover page
            1. Time filter
              1. Quick time filter
              2. Relative time filter
              3. Absolute time filter
              4. Kibana Auto-refresh setting
          2. Querying and searching data
            1. Freetext search
              1. AND
              2. OR
              3. NOT
              4. Groupings
              5. Wildcard searches
            2. Field searches
            3. Range searches
            4. Special characters escaping
            5. New search
            6. Saving the search
            7. Loading a search
            8. Field searches using field list
        3. Summary
      14. 7. Kibana – Visualization and Dashboard
        1. Visualize page
          1. Creating a visualization
          2. Visualization types
          3. Metrics and buckets aggregations
            1. Buckets
              1. Date Histogram
              2. Histogram
              3. Range
              4. Date Range
              5. Terms
            2. Metrics
              1. Count
              2. Average, Sum, Min, and Max
              3. Unique Count
            3. Advanced options
          4. Visualizations
            1. Area chart
            2. Data table
            3. Line chart
            4. Markdown widget
            5. Metric
            6. Pie chart
            7. Tile map
            8. Vertical bar chart
        2. Dashboard page
          1. Building a new dashboard
          2. Saving and loading a dashboard
          3. Sharing a dashboard
        3. Summary
      15. 8. Putting It All Together
        1. Input dataset
        2. Configuring Logstash input
          1. Grok pattern for access logs
        3. Visualizing with Kibana
          1. Running Kibana
          2. Searching on the Discover page
          3. Visualizations – charts
          4. Building a Line chart
          5. Building an Area chart
          6. Building a Bar chart
          7. Building a Markdown
          8. Dashboard page
        4. Summary
      16. 9. ELK Stack in Production
        1. Prevention of data loss
        2. Data protection
        3. System scalability
        4. Data retention
        5. ELK Stack implementations
          1. ELK Stack at LinkedIn
            1. Problem statement
            2. Criteria for solution
            3. Solution
            4. Kafka at LinkedIn
            5. Operational challenges
            6. Logging using Kafka at LinkedIn
        6. ELK at SCA
          1. How is ELK used in SCA?
          2. How is it helping in analytics?
          3. ELK for monitoring at SCA
        7. ELK at Cliffhanger Solutions
        8. Kibana demo – Packetbeat dashboard
        9. Summary
      17. 10. Expanding Horizons with ELK
        1. Elasticsearch plugins and utilities
          1. Curator for index management
            1. Curator commands
            2. Curator installation
          2. Shield for security
            1. Shield installation
            2. Adding users and roles
            3. Using Kibana4 on shield protected Elasticsearch
          3. Marvel to monitor
            1. Marvel installation
            2. Marvel dashboards
        2. ELK roadmap
          1. Elasticsearch roadmap
          2. Logstash roadmap
            1. Event persistence capability
            2. End-to-end message acknowledgement
            3. Logstash monitoring and management API
          3. Kibana roadmap
        3. Summary
      18. Index