You are previewing Implementing Splunk: Big Data Reporting and Development for Operational Intelligence.
O'Reilly logo
Implementing Splunk: Big Data Reporting and Development for Operational Intelligence

Book Description

Learn to transform your machine data into valuable IT and business insights all using this comprehensive and practical tutorial

  • Learn to search, dashboard, configure, and deploy Splunk on one machine or thousands

  • Start working with Splunk fast, with a tested set of practical examples and useful advice

  • Step-by-step instructions and examples with a comprehensive coverage for Splunk veterans and newbies alike

  • In Detail

    Splunk is a data collection, indexing, and visualization engine for operational intelligence. It's a powerful and versatile search and analysis engine that lets you investigate, troubleshoot, monitor, alert, and report on everything that's happening in your entire IT infrastructure from one location in real time. Splunk collects, indexes, and harnesses all the fast moving machine data generated by our applications, servers, and devices - physical, virtual, and in the cloud.

    Given a mountain of machine data, this book shows you exactly how to learn to use Splunk to make something useful from it. Depending on your needs, you can learn to search, transform, and display data, or learn to administer your Splunk installation, large or small.

    "Implementing Splunk: Big Data Reporting and Development for Operational Intelligence" will help you get your job done faster, whether you read from the beginning or jump to what you need to know today. New and experienced users alike will find nuggets of wisdom throughout.

    This book provides you with valuable examples and step-by-step instructions, showing you how to take advantage of everything Splunk has to offer you, to make the most out of your machine data.

    "Implementing Splunk: Big Data Reporting and Development for Operational Intelligence" takes you on a journey right from inception to a fully functioning implementation of Splunk. Using a real-world data walkthrough, you’ll be shown how to search effectively, create fields, build dashboards, reports, and package apps, manage your indexes, integrate into the enterprise, and extend Splunk. This practical implementation guide equips you with high-level knowledge for configuring, deploying, extending, and integrating Splunk. Depending on the goal and skills of the reader, enough topics are covered to get you on your way to dashboard guru, app developer, or enterprise administrator. This book uses examples curates reference, and sage advice to help you make the most of this incredibly powerful tool.

    Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at If you purchased this book elsewhere, you can visit and register to have the files e-mailed directly to you.

    Table of Contents

    1. Implementing Splunk: Big Data Reporting and Development for Operational Intelligence
      1. Table of Contents
      2. Implementing Splunk: Big Data Reporting and Development for Operational Intelligence
      3. Credits
      4. About the Author
      5. About the Reviewers
        1. Support files, eBooks, discount offers and more
          1. Why Subscribe?
          2. Free Access for Packt account holders
      7. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Conventions
        5. Reader feedback
        6. Customer support
          1. Downloading the example code
          2. Errata
          3. Piracy
          4. Questions
      8. 1. The Splunk Interface
        1. Logging in to Splunk
        2. The Home app
        3. The top bar
        4. Search app
          1. Data generator
          2. The Summary view
          3. Search
          4. Actions
          5. Timeline
          6. The field picker
            1. Fields
          7. Search results
            1. Options
            2. Events viewer
        5. Using the time picker
        6. Using the field picker
        7. Using Manager
        8. Summary
      9. 2. Understanding Search
        1. Using search terms effectively
        2. Boolean and grouping operators
        3. Clicking to modify your search
          1. Event segmentation
          2. Field widgets
          3. Time
        4. Using fields to search
          1. Using the field picker
        5. Using wildcards efficiently
          1. Only trailing wildcards are efficient
          2. Wildcards are tested last
          3. Supplementing wildcards in fields
        6. All about time
          1. How Splunk parses time
          2. How Splunk stores time
          3. How Splunk displays time
          4. How time zones are determined and why it matters
          5. Different ways to search against time
          6. Specifying time in-line in your search
          7. _indextime versus _time
        7. Making searches faster
        8. Sharing results with others
        9. Saving searches for reuse
        10. Creating alerts from searches
          1. Schedule
          2. Actions
        11. Summary
      10. 3. Tables, Charts, and Fields
        1. About the pipe symbol
        2. Using top to show common field values
          1. Controlling the output of top
        3. Using stats to aggregate values
        4. Using chart to turn data
        5. Using timechart to show values over time
          1. timechart options
        6. Working with fields
          1. A regular expression primer
          2. Commands that create fields
            1. eval
            2. rex
          3. Extracting loglevel
            1. Using the Extract Fields interface
            2. Using rex to prototype a field
            3. Using the admin interface to build a field
            4. Indexed fields versus extracted fields
              1. Indexed field case 1 – rare instances of a common term
              2. Indexed field case 2 – splitting words
              3. Indexed field case 3 – application from source
              4. Indexed field case 4 – slow requests
              5. Indexed field case 5 – unneeded work
        7. Summary
      11. 4. Simple XML Dashboards
        1. The purpose of dashboards
        2. Using wizards to build dashboards
        3. Scheduling the generation of dashboards
        4. Editing the XML directly
        5. UI Examples app
        6. Building forms
          1. Creating a form from a dashboard
          2. Driving multiple panels from one form
          3. Post-processing search results
          4. Post-processing limitations
            1. Panel 1
            2. Panel 2
            3. Panel 3
            4. Final XML
        7. Summary
      12. 5. Advanced Search Examples
        1. Using subsearches to find loosely related events
          1. Subsearch
          2. Subsearch caveats
          3. Nested subsearches
        2. Using transaction
          1. Using transaction to determine the session length
          2. Calculating the aggregate of transaction statistics
          3. Combining subsearches with transaction
        3. Determining concurrency
          1. Using transaction with concurrency
          2. Using concurrency to estimate server load
          3. Calculating concurrency with a by clause
        4. Calculating events per slice of time
          1. Using timechart
          2. Calculating average requests per minute
          3. Calculating average events per minute, per hour
        5. Rebuilding top
        6. Summary
      13. 6. Extending Search
        1. Using tags to simplify search
        2. Using event types to categorize results
        3. Using lookups to enrich data
          1. Defining a lookup table file
          2. Defining a lookup definition
          3. Defining an automatic lookup
          4. Troubleshooting lookups
        4. Using macros to reuse logic
          1. Creating a simple macro
          2. Creating a macro with arguments
          3. Using eval to build a macro
        5. Creating workflow actions
          1. Running a new search using values from an event
          2. Linking to an external site
          3. Building a workflow action to show field context
            1. Building the context workflow action
            2. Building the context macro
        6. Using external commands
          1. Extracting values from XML
            1. xmlkv
            2. XPath
          2. Using Google to generate results
        7. Summary
      14. 7. Working with Apps
        1. Defining an app
        2. Included apps
        3. Installing apps
          1. Installing apps from Splunkbase
            1. Using Geo Location Lookup Script
            2. Using Google Maps
          2. Installing apps from a file
        4. Building your first app
        5. Editing navigation
        6. Customizing the appearance of your app
          1. Customizing the launcher icon
          2. Using custom CSS
          3. Using custom HTML
            1. Custom HTML in a simple dashboard
            2. Using ServerSideInclude in a complex dashboard
        7. Object permissions
          1. How permissions affect navigation
          2. How permissions affect other objects
          3. Correcting permission problems
        8. App directory structure
        9. Adding your app to Splunkbase
          1. Preparing your app
            1. Confirming sharing settings
            2. Cleaning up our directories
          2. Packaging your app
          3. Uploading your app
        10. Summary
      15. 8. Building Advanced Dashboards
        1. Reasons for working with advanced XML
        2. Reasons for not working with advanced XML
        3. Development process
        4. Advanced XML structure
        5. Converting simple XML to advanced XML
        6. Module logic flow
        7. Understanding layoutPanel
          1. Panel placement
        8. Reusing a query
        9. Using intentions
          1. stringreplace
          2. addterm
        10. Creating a custom drilldown
          1. Building a drilldown to a custom query
          2. Building a drilldown to another panel
          3. Building a drilldown to multiple panels using HiddenPostProcess
        11. Third-party add-ons
          1. Google Maps
          2. Sideview Utils
            1. The Sideview Search module
            2. Linking views with Sideview
            3. Sideview URLLoader
            4. Sideview forms
        12. Summary
      16. 9. Summary Indexes and CSV Files
        1. Understanding summary indexes
          1. Creating a summary index
        2. When to use a summary index
        3. When to not use a summary index
        4. Populating summary indexes with saved searches
        5. Using summary index events in a query
        6. Using sistats, sitop, and sitimechart
        7. How latency affects summary queries
        8. How and when to backfill summary data
          1. Using to backfill
          2. Using collect to produce custom summary indexes
        9. Reducing summary index size
          1. Using eval and rex to define grouping fields
          2. Using a lookup with wildcards
          3. Using event types to group results
        10. Calculating top for a large time frame
        11. Storing raw events in a summary index
        12. Using CSV files to store transient data
          1. Pre-populating a dropdown
          2. Creating a running calculation for a day
        13. Summary
      17. 10. Configuring Splunk
        1. Locating Splunk configuration files
        2. The structure of a Splunk configuration file
        3. Configuration merging logic
          1. Merging order
            1. Merging order outside of search
            2. Merging order when searching
          2. Configuration merging logic
            1. Configuration merging example 1
            2. Configuration merging example 2
            3. Configuration merging example 3
            4. Configuration merging example 4 (search)
          3. Using btool
        4. An overview of Splunk .conf files
          1. props.conf
            1. Common attributes
              1. Search-time attributes
              2. Index-time attributes
              3. Parse-time attributes
              4. Input time attributes
            2. Stanza types
            3. Priorities inside a type
            4. Attributes with class
          2. inputs.conf
            1. Common input attributes
            2. Files as inputs
              1. Using patterns to select rolled logs
              2. Using blacklist and whitelist
              3. Selecting files recursively
              4. Following symbolic links
              5. Setting the value of host from source
              6. Ignoring old data at installation
              7. When to use crcSalt
              8. Destructively indexing files
            3. Network inputs
            4. Native Windows inputs
            5. Scripts as inputs
          3. transforms.conf
            1. Creating indexed fields
              1. Creating a loglevel field
              2. Creating a session field from source
              3. Creating a "tag" field
              4. Creating host categorization fields
            2. Modifying metadata fields
              1. Overriding host
              2. Overriding source
              3. Overriding sourcetype
              4. Routing events to a different index
            3. Lookup definitions
              1. Wildcard lookups
              2. CIDR wildcard lookups
              3. Using time in lookups
            4. Using REPORT
              1. Creating multivalue fields
              2. Creating dynamic fields
            5. Chaining transforms
            6. Dropping events
          4. fields.conf
          5. outputs.conf
          6. indexes.conf
          7. authorize.conf
          8. savedsearches.conf
          9. times.conf
          10. commands.conf
          11. web.conf
        5. User interface resources
          1. Views and navigation
          2. Appserver resources
          3. Metadata
        6. Summary
      18. 11. Advanced Deployments
        1. Planning your installation
        2. Splunk instance types
          1. Splunk forwarders
          2. Splunk indexer
          3. Splunk search
        3. Common data sources
          1. Monitoring logs on servers
          2. Monitoring logs on a shared drive
          3. Consuming logs in batch
          4. Receiving syslog events
            1. Receiving events directly on the Splunk indexer
            2. Using a native syslog receiver
            3. Receiving syslog with a Splunk forwarder
          5. Consuming logs from a database
          6. Using scripts to gather data
        4. Sizing indexers
        5. Planning redundancy
          1. Indexer load balancing
          2. Understanding typical outages
        6. Working with multiple indexes
          1. Directory structure of an index
          2. When to create more indexes
            1. Testing data
            2. Differing longevity
            3. Differing permissions
            4. Using more indexes to increase performance
          3. The lifecycle of a bucket
          4. Sizing an index
          5. Using volumes to manage multiple indexes
        7. Deploying the Splunk binary
          1. Deploying from a tar file
          2. Deploying using msiexec
          3. Adding a base configuration
          4. Configuring Splunk to launch at boot
        8. Using apps to organize configuration
          1. Separate configurations by purpose
        9. Configuration distribution
          1. Using your own deployment system
          2. Using Splunk deployment server
            1. Step 1 – Deciding where your deployment server will run
            2. Step 2 – Defining your deploymentclient.conf configuration
            3. Step 3 – Defining our machine types and locations
            4. Step 4 – Normalizing our configurations into apps appropriately
            5. Step 5 – Mapping these apps to deployment clients in serverclass.conf
            6. Step 6 – Restarting the deployment server
            7. Step 7 – Installing deploymentclient.conf
        10. Using LDAP for authentication
        11. Using Single Sign On
        12. Load balancers and Splunk
          1. web
          2. splunktcp
          3. deployment server
        13. Multiple search heads
        14. Summary
      19. 12. Extending Splunk
        1. Writing a scripted input to gather data
          1. Capturing script output with no date
          2. Capturing script output as a single event
          3. Making a long-running scripted input
        2. Using Splunk from the command line
        3. Querying Splunk via REST
        4. Writing commands
          1. When not to write a command
          2. When to write a command
          3. Configuring commands
          4. Adding fields
          5. Manipulating data
          6. Transforming data
          7. Generating data
        5. Writing a scripted lookup to enrich data
        6. Writing an event renderer
          1. Using specific fields
          2. Table of fields based on field value
          3. Pretty print XML
        7. Writing a scripted alert action to process results
        8. Summary
      20. Index