You are previewing Splunk Operational Intelligence Cookbook - Second Edition.
O'Reilly logo
Splunk Operational Intelligence Cookbook - Second Edition

Book Description

Over 70 practical recipes to gain operational data intelligence with Splunk Enterprise

About This Book

  • This is the most up-to-date book on Splunk 6.3 and teaches you how to tackle real-world operational intelligence scenarios efficiently

  • Get business insights using machine data using this easy-to-follow guide

  • Search, monitor, and analyze your operational data skillfully using this recipe-based, practical guide

  • Who This Book Is For

    This book is intended for users of all levels who are looking to leverage the Splunk Enterprise platform as a valuable operational intelligence tool. The recipes provided in this book will appeal to individuals from all facets of business, IT, security, product, marketing, and many more!

    Also, existing users of Splunk who want to upgrade and get up and running with Splunk 6.3 will find this book invaluable.

    What You Will Learn

  • Use Splunk to gather, analyze, and report on data

  • Create dashboards and visualizations that make data meaningful

  • Build an operational intelligence application with extensive features and functionality

  • Enrich operational data with lookups and workflows

  • Model and accelerate data and perform pivot-based reporting

  • Build real-time, scripted, and other intelligence-driven alerts

  • Summarize data for longer term trending, reporting, and analysis

  • Integrate advanced JavaScript charts and leverage Splunk's API

  • In Detail

    Splunk makes it easy for you to take control of your data, and with Splunk Operational Cookbook, you can be confident that you are taking advantage of the Big Data revolution and driving your business with the cutting edge of operational intelligence and business analytics.

    With more than 70 recipes that demonstrate all of Splunk’s features, not only will you find quick solutions to common problems, but you’ll also learn a wide range of strategies and uncover new ideas that will make you rethink what operational intelligence means to you and your organization.

    You’ll discover recipes on data processing, searching and reporting, dashboards, and visualizations to make data shareable, communicable, and most importantly meaningful. You’ll also find step-by-step demonstrations that walk you through building an operational intelligence application containing vital features essential to understanding data and to help you successfully integrate a data-driven way of thinking in your organization.

    Throughout the book, you’ll dive deeper into Splunk, explore data models and pivots to extend your intelligence capabilities, and perform advanced searching to explore your data in even more sophisticated ways. Splunk is changing the business landscape, so make sure you’re taking advantage of it.

    Style and approach

    Splunk is an excellent platform that allows you to make sense of machine data with ease. The adoption of Splunk has been huge and everyone who has gone beyond installing Splunk wants to know how to make most of it. This book will not only teach you how to use Splunk in real-world scenarios to get business insights, but will also get existing Splunk users up to date with the latest Splunk 6.3 release.

    Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

    Table of Contents

    1. Splunk Operational Intelligence Cookbook Second Edition
      1. Table of Contents
      2. Splunk Operational Intelligence Cookbook Second Edition
      3. Credits
      4. About the Authors
      5. About the Reviewer
      6. www.PacktPub.com
        1. eBooks, discount offers, and more
          1. Why subscribe?
          2. Instant updates on new Packt books
      7. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Sections
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        5. Conventions
        6. Reader feedback
        7. Customer support
          1. Downloading the example code
          2. Downloading the color images of this book
          3. Errata
          4. Piracy
          5. Questions
      8. 1. Play Time – Getting Data In
        1. Introduction
        2. Indexing files and directories
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Adding a file or directory data input via the CLI
            2. Adding a file or directory input via inputs.conf
            3. One-time indexing of data files via the Splunk CLI
            4. Indexing the Windows event logs
          5. See also
        3. Getting data through network ports
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Adding a network input via the CLI
            2. Adding a network input via inputs.conf
          5. See also
        4. Using scripted inputs
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        5. Using modular inputs
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        6. Using the Universal Forwarder to gather data
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Add the receiving indexer via outputs.conf
        7. Loading the sample data for this book
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        8. Defining field extractions
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        9. Defining event types and tags
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Adding event types and tags via eventtypes.conf and tags.conf
          5. See also
      9. 2. Diving into Data – Search and Report
        1. Introduction
        2. Making raw event data readable
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Tabulating every field
            2. Removing fields, then tabulating everything else
        3. Finding the most accessed web pages
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Searching for the top 10 accessed web pages
            2. Searching for the most accessed pages by user
          5. See also
        4. Finding the most used web browsers
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Searching for the web browser data for the most used OS types
          5. See also
        5. Identifying the top-referring websites
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Searching for the top 10 using stats instead of top
          5. See also
        6. Charting web page response codes
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Totaling success and error web page response codes
          5. See also
        7. Displaying web page response time statistics
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Displaying web page response time by action
          5. See also
        8. Listing the top viewed products
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Searching for the percentage of cart additions from product views
          5. See also
        9. Charting the application's functional performance
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        10. Charting the application's memory usage
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        11. Counting the total number of database connections
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
      10. 3. Dashboards and Visualizations – Making Data Shine
        1. Introduction
        2. Creating an Operational Intelligence dashboard
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Changing dashboard permissions
        3. Using a pie chart to show the most accessed web pages
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Searching for the top 10 accessed web pages
          5. See also
        4. Displaying the unique number of visitors
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Coloring the value based on ranges
            2. Adding trends and sparklines to the values
          5. See also
        5. Using a gauge to display the number of errors
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        6. Charting the number of method requests by type and host
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        7. Creating a timechart of method requests, views, and response times
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Method requests, views, and response times by host
          5. See also
        8. Using a scatter chart to identify discrete requests by size and response time
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Using time series data points with a scatter chart
          5. See also
        9. Creating an area chart of the application's functional statistics
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        10. Using a bar chart to show the average amount spent by category
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        11. Creating a line chart of item views and purchases over time
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
      11. 4. Building an Operational Intelligence Application
        1. Introduction
        2. Creating an Operational Intelligence application
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Creating an application from another application
            2. Downloading and installing a Splunk app
          5. See also
        3. Adding dashboards and reports
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Changing permissions of saved reports
          5. See also
        4. Organizing the dashboards more efficiently
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Modifying the Simple XML directly
          5. See also
        5. Dynamically drilling down on activity reports
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Disabling the drilldown feature in tables and charts
          5. See also
        6. Creating a form for searching web activity
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Adding a Submit button to your form
          5. See also
        7. Linking web page activity reports to the form
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Adding an overlay to the Sessions Over Time chart
          5. See also
        8. Displaying a geographical map of visitors
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Adding a map panel using Simple XML
            2. Mapping different distributions by area
          5. See also
        9. Scheduling PDF delivery of a dashboard
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
      12. 5. Extending Intelligence – Data Models and Pivoting
        1. Introduction
        2. Creating a data model for web access logs
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Searching data models using the search interface
          5. See also
        3. Creating a data model for application logs
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        4. Accelerating data models
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Viewing data model and acceleration summary information
            2. Advanced configuration of data model acceleration
          5. See also
        5. Pivoting total sales transactions
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Pivot searching using the pivot command and search interface
          5. See also
        6. Pivoting purchases by geographic location
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        7. Pivoting slowest responding web pages
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        8. Pivot charting top error codes
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
      13. 6. Diving Deeper – Advanced Searching
        1. Introduction
        2. Calculating the average session time on a website
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Starts with a website visit, ends with a checkout
            2. Defining maximum pause, span, and events in a transaction
          5. See also
        3. Calculating the average execution time for multi-tier web requests
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Calculating the average execution time without using a join
          5. See also
        4. Displaying the maximum concurrent checkouts
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. See also
        5. Analyzing the relationship of web requests
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Analyzing relationships of DB actions to memory utilization
          5. See also
        6. Predicting website traffic volumes
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Predicting the total number of items purchased
            2. Predicting the average response time of function calls
          5. See also
        7. Finding abnormally-sized web requests
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. The anomalies command
            2. The anomalousvalues command
            3. The anomalydetection command
            4. The cluster command
          5. See also
        8. Identifying potential session spoofing
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Creating logic for urgency
          5. See also
      14. 7. Enriching Data – Lookups and Workflows
        1. Introduction
        2. Looking up product code descriptions
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Manually adding the lookup to Splunk
          5. See also
        3. Flagging suspect IP addresses
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Modifying an existing saved search to populate a lookup table
          5. See also
        4. Creating a session state table
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Use the Splunk KV store to maintain the session state table
          5. See also
        5. Adding hostnames to IP addresses
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Enabling automatic external field lookups
          5. See also
        6. Searching ARIN for a given IP address
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Limiting workflow actions by event types
          5. See also
        7. Triggering a Google search for a given error
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Triggering a Google search from the chart drilldown options
          5. See also
        8. Creating a ticket for application errors
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Adding a workflow action manually in Splunk
          5. See also
        9. Looking up inventory from an external database
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Use DB Connect for direct external DB lookups
          5. See also
      15. 8. Being Proactive – Creating Alerts
        1. Introduction
        2. Alerting on abnormal web page response times
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Viewing triggered alerts in Splunk's Alert manager
          5. See also
        3. Alerting on errors during checkout in real time
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Building alerts via a configuration file
            2. Editing alert configuration attributes using Advanced edit
            3. Identify the real-time searches that are running
          5. See also
        4. Alerting on abnormal user behavior
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Alerting on abnormal user purchases without checkouts
          5. See also
        5. Alerting on failure and triggering a scripted response
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
          5. See also
        6. Alerting when predicted sales exceed inventory
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Adding an RSS feed notification action to an alert
          5. See also
      16. 9. Speeding Up Intelligence – Data Summarization
        1. Introduction
        2. Calculating an hourly count of sessions versus completed transactions
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Generating the summary more frequently
            2. Avoiding summary index overlaps and gaps
          5. See also
        3. Backfilling the number of purchases by city
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Backfilling a summary index from within a search directly
          5. See also
        4. Displaying the maximum number of concurrent sessions over time
          1. Getting ready
          2. How to do it…
          3. How it works…
          4. There's more…
            1. Viewing the status of an accelerated report
          5. See also
      17. 10. Above and Beyond – Customization, Web Framework, REST API, HTTP Event Collector, and SDKs
        1. Introduction
        2. Customizing the application navigation
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more…
        3. Adding a force-directed graph of web hits
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more…
            1. Changing the time range on the search manager
          5. See also
        4. Adding a calendar heatmap of product purchases
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        5. Adding cell highlighting of average product price
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more…
          5. See also
        6. Remotely querying Splunk's REST API for unique page views
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more…
            1. Authenticating with a session token
          5. See also
        7. Creating a Python application to return unique IP addresses
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. There's more...
            1. Paginating the results of your search
          5. See also
        8. Creating a custom search command to format product names
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
        9. Collecting data from remote scanning devices
          1. Getting ready
          2. How to do it...
          3. How it works...
          4. See also
      18. Index