O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Splunk Operational Intelligence Cookbook - Third Edition

Book Description

Leverage Splunk's operational intelligence capabilities to unlock new hidden business insights and drive success

About This Book
  • Tackle any problems related to searching and analyzing your data with Splunk
  • Get the latest information and business insights on Splunk 7.x
  • Explore the all new machine learning toolkit in Splunk 7.x
Who This Book Is For

This book is intended for data professionals who are looking to leverage the Splunk Enterprise platform as a valuable operational intelligence tool. The recipes provided in this book will appeal to individuals from all facets of business, IT, security, product, marketing, and many more! Even the existing users of Splunk who want to upgrade and get up and running with Splunk 7.x will find this book to be of great value.

What You Will Learn
  • Learn how to use Splunk to gather, analyze, and report on data
  • Create dashboards and visualizations that make data meaningful
  • Build an intelligent application with extensive functionalities
  • Enrich operational data with lookups and workflows
  • Model and accelerate data and perform pivot-based reporting
  • Apply ML algorithms for forecasting and anomaly detection
  • Summarize data for long term trending, reporting, and analysis
  • Integrate advanced JavaScript charts and leverage Splunk's API
In Detail

Splunk makes it easy for you to take control of your data, and with Splunk Operational Cookbook, you can be confident that you are taking advantage of the Big Data revolution and driving your business with the cutting edge of operational intelligence and business analytics.

With more than 70 recipes that demonstrate all of Splunk's features, not only will you find quick solutions to common problems, but you'll also learn a wide range of strategies and uncover new ideas that will make you rethink what operational intelligence means to you and your organization.

You'll discover recipes on data processing, searching and reporting, dashboards, and visualizations to make data shareable, communicable, and most importantly meaningful. You'll also find step-by-step demonstrations that walk you through building an operational intelligence application containing vital features essential to understanding data and to help you successfully integrate a data-driven way of thinking in your organization.

Throughout the book, you'll dive deeper into Splunk, explore data models and pivots to extend your intelligence capabilities, and perform advanced searching with machine learning to explore your data in even more sophisticated ways. Splunk is changing the business landscape, so make sure you're taking advantage of it.

Style and approach

With more than 70 recipes that demonstrate all of Splunk's features, not only will you find quick solutions to common problems, but you'll also learn a wide range of strategies and uncover new ideas that will make you rethink what operational intelligence means to you and your organization.

Downloading the example code for this book You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.

Table of Contents

  1. Title Page
  2. Copyright and Credits
    1. Splunk Operational Intelligence Cookbook Third Edition
  3. Packt Upsell
    1. Why subscribe?
    2. PacktPub.com
  4. Contributors
    1. About the authors
    2. About the reviewer
    3. Packt is searching for authors like you
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Conventions used
    4. Sections
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    5. Get in touch
      1. Reviews
  6. Play Time – Getting Data In
    1. Introduction
    2. Indexing files and directories
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding a file or directory data input using the CLI
        2. Adding a file or directory input using inputs.conf
        3. One-time indexing of data files using the Splunk CLI
        4. Indexing the Windows event logs
      5. See also
    3. Getting data through network ports
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding a network input using the CLI
        2. Adding a network input using inputs.conf
      5. See also
    4. Using scripted inputs
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    5. Using modular inputs
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    6. Using the Universal Forwarder to gather data
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding the receiving indexer via outputs.conf
    7. Receiving data using the HTTP Event Collector
      1. Getting ready
      2. How to do it...
      3. How it works...
    8. Getting data from databases using DB Connect
      1. Getting ready
      2. How to do it...
      3. How it works...
    9. Loading the sample data for this book
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    10. Data onboarding – defining field extractions
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    11. Data onboarding - defining event types and tags
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding event types and tags using eventtypes.conf and tags.conf
      5. See also
    12. Installing the Machine Learning Toolkit
      1. Getting ready
      2. How to do it...
      3. How it works...
  7. Diving into Data – Search and Report
    1. Introduction
      1. The Search Processing Language 
      2. Searching in Splunk
      3. Boolean operators
      4. Common commands
      5. Time modifiers
      6. Working with fields
      7. Saving searches in Splunk
    2. Making raw event data readable
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Tabulating every field
        2. Removing fields, then tabulating everything else
    3. Finding the most accessed web pages
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Searching for the top 10 accessed web pages
        2. Searching for the most accessed pages by user
      5. See also
    4. Finding the most used web browsers
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Searching for the web browser data for the most used OS types
      5. See also
    5. Identifying the top-referring websites
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Searching for the top 10 using stats instead of top
      5. See also
    6. Charting web page response codes
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Totaling success and error web page response codes
      5. See also
    7. Displaying web page response time statistics
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Displaying web page response time by action
      5. See also
    8. Listing the top-viewed products
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Searching for the percentage of cart additions from product views
      5. See also
    9. Charting the application's functional performance
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    10. Charting the application's memory usage
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    11. Counting the total number of database connections
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
  8. Dashboards and Visualizations - Make Data Shine
    1. Introduction
      1. About Splunk dashboards
      2. Using dashboards for Operational Intelligence
      3. Enriching data with visualizations
      4. Available visualizations
      5. Trellis layout
      6. Best practices for visualizations
    2. Creating an Operational Intelligence dashboard
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Changing dashboard permissions
    3. Using a pie chart to show the most accessed web pages
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Searching for the top ten accessed web pages
      5. See also
    4. Displaying the unique number of visitors
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding labels to a single value panel
        2. Coloring the value based on ranges
        3. Adding trends and sparklines to the values
      5. See also
    5. Using a gauge to display the number of errors
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    6. Charting the number of method requests by type and host
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    7. Creating a timechart of method requests, views, and response times
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Method requests, views, and response times by host
      5. See also
    8. Using a scatter chart to identify discrete requests by size and response time
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Using time series data points with a scatter chart
      5. See also
    9. Creating an area chart of the application's functional statistics
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    10. Using metrics data and a trellis layout to monitor physical environment operating conditions
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    11. Using a bar chart to show the average amount spent by category
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    12. Creating a line chart of item views and purchases over time
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
  9. Building an Operational Intelligence Application
    1. Introduction
    2. Creating an Operational Intelligence application
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Creating an application from another application
        2. Downloading and installing a Splunk app
      5. See also
    3. Adding dashboards and reports
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Changing permissions of saved reports
      5. See also
    4. Organizing the dashboards more efficiently
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Modifying the Simple XML directly
      5. See also
    5. Dynamically drilling down on activity reports
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Disabling the drilldown feature in tables and charts
      5. See also
    6. Creating a form for searching web activity
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding a Submit button to your form
      5. See also
    7. Linking web page activity reports to the form
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding an overlay to the Sessions Over Time chart
      5. See also
    8. Displaying a geographical map of visitors
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding a map panel using Simple XML
        2. Mapping different distributions by area
      5. See also
    9. Highlighting average product price
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    10. Scheduling the PDF delivery of a dashboard
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
  10. Extending Intelligence – Datasets, Modeling and Pivoting
    1. Introduction
    2. Creating a data model for web access logs
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Viewing datasets using the dataset listing page
        2. Searching datasets using the search interface
      5. See also
    3. Creating a data model for application logs
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    4. Accelerating data models
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Viewing data model and acceleration summary information
        2. Advanced configuration of data model acceleration
      5. See also
    5. Pivoting total sales transactions
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Searching datasets using the pivot command
        2. Searching accelerated datasets using the tstats command
      5. See also
    6. Pivoting purchases by geographic location
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    7. Pivoting slowest responding web pages
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    8. Pivot charting top error codes
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
  11. Diving Deeper – Advanced Searching, Machine Learning and Predictive Analytics
    1. Introduction
      1. Identifying and grouping transactions
      2. Converging data sources
      3. Identifying relationships between fields
      4. Predicting future values
      5. Discovering anomalous values
      6. Leveraging machine learning
    2. Calculating the average session time on a website
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Starts with a website visit, ends with a checkout
        2. Defining maximum pause, span, and events in a transaction
      5. See also
    3. Calculating the average execution time for multi-tier web requests
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Calculating the average execution time without using a join
      5. See also
    4. Displaying the maximum concurrent checkouts
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    5. Analyzing the relationship of web requests
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Analyzing relationships of DB actions to memory utilization
      5. See also
    6. Predicting website traffic volumes
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Create and apply a machine learning model of traffic over time
        2. Predicting the total number of items purchased
        3. Predicting the average response time of function calls
      5. See also
    7. Finding abnormally-sized web requests
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. The anomalies command
        2. The anomalousvalue command
        3. The anomalydetection command
        4. The cluster command
      5. See also
    8. Identifying potential session spoofing
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Creating logic for urgency
      5. See also
    9. Detecting outliers in server response times
      1. Getting ready
      2. How to do it...
      3. How it works...
    10. Forecasting weekly sales
      1. Getting ready
      2. How to do it...
      3. How it works...
    11. Summary
  12. Enriching Data – Lookups and Workflows
    1. Introduction
      1. Lookups
      2. Workflows
      3. DB Connect
    2. Looking up product code descriptions
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Manually adding the lookup to Splunk
      5. See also
    3. Flagging suspect IP addresses
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Modifying an existing saved search to populate a lookup table
      5. See also
    4. Creating a session state table
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Use the Splunk KV store to maintain the session state table
      5. See also
    5. Adding hostnames to IP addresses
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Enabling automatic external field lookups
      5. See also
    6. Searching ARIN for a given IP address
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Limiting workflow actions by event types
      5. See also
    7. Triggering a Google search for a given error
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Triggering a Google search from the chart drilldown options
      5. See also
    8. Generating a chat notification for application errors
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Adding a workflow action manually in Splunk
      5. See also
    9. Looking up inventory from an external database
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Using DB Connect for direct external DB lookups
      5. See also
  13. Being Proactive – Creating Alerts
    1. Introduction
      1. About Splunk alerts
      2. Types of alert
      3. Alert Trigger Conditions
      4. Alert Trigger Actions
    2. Alerting on abnormal web page response times
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Viewing alerts in Splunk's Triggered Alert view
      5. See also
    3. Alerting on errors during checkout in real time
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Building alerts via a configuration file
        2. Editing alert configuration attributes using Advanced edit
        3. Identify the real-time searches that are running
      5. See also
    4. Alerting on abnormal user behavior
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Alerting on abnormal user purchases without checkouts
      5. See also
    5. Alerting on failure and triggering a chat notification
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    6. Alerting when predicted sales exceed inventory
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    7. Generating alert events for high sensor readings
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
  14. Speeding Up Intelligence – Data Summarization
    1. Introduction
      1. Data summarization
      2. Data summarization methods
        1. About summary indexing
          1. How summary indexing helps
        2. About report acceleration
          1. The simplicity of report acceleration
    2. Calculating an hourly count of sessions versus completed transactions
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Generating the summary more frequently
        2. Avoiding summary index overlaps and gaps
      5. See also
    3. Backfilling the number of purchases by city
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Backfilling a summary index from within a search directly
      5. See also
    4. Displaying the maximum number of concurrent sessions over time
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Viewing the status of an accelerated report and how 
      5. See also
  15. Above and Beyond – Customization, Web Framework, HTTP Event Collector, REST API, and SDKs
    1. Introduction
      1. Web framework
      2. REST API
      3. Software development kits (SDKs)
      4. HTTP Event Collector (HEC)
    2. Customizing the application navigation
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
    3. Adding a Sankey diagram of web hits
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Changing the Sankey diagram options
      5. See also
    4. Developing a tag cloud of purchases by country
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's More...
      5. See also
    5. Adding Cell Icons to Highlight Average Product Price
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    6. Remotely querying Splunk's REST API for unique page views
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Authenticating with a session token
      5. See also
    7. Creating a Python application to return unique IP addresses
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Paginating the results of your search
      5. See also
    8. Creating a custom search command to format product names
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
    9. Collecting data from remote scanning devices
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. See also
  16. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think