You are previewing Implementing Splunk - Second Edition.
O'Reilly logo
Implementing Splunk - Second Edition

Book Description

A comprehensive guide to help you transform Big Data into valuable business insights with Splunk 6.2

In Detail

Splunk is a type of analysis and reporting software for analyzing machine-generated Big Data. It captures, indexes, and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations. It aims to make machine data accessible across an organization for a variety of purposes.

Implementing Splunk Second Edition is a learning guide that introduces you to all the latest features and improvements of Splunk 6.2. The book starts by introducing you to various concepts such as charting, reporting, clustering, and visualization. Every chapter is dedicated to enhancing your knowledge of a specific concept, including data models and pivots, speeding up your queries, backfilling, data replication, and so on. By the end of the book, you'll have a very good understanding of Splunk and be able to perform efficient data analysis.

What You Will Learn

  • Enrich your data with lookups and commands

  • Transform your data into useful and beautiful reports

  • Build professional-looking, informative dashboards

  • Get to know what Splunk data models and pivots are

  • Learn about pivot editor, pivot elements, filters, Sparklines, and more

  • Manage configurations from one to thousands of instances

  • Extend Splunk with scripts and advanced configuration

  • Create fields from your unstructured data

  • Write searches that are fast and lean

  • Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.

    Table of Contents

    1. Implementing Splunk Second Edition
      1. Table of Contents
      2. Implementing Splunk Second Edition
      3. Credits
      4. About the Authors
      5. About the Reviewers
      6. www.PacktPub.com
        1. Support files, eBooks, discount offers, and more
          1. Why subscribe?
          2. Free access for Packt account holders
          3. Instant updates on new Packt books
      7. Preface
        1. What this book covers
        2. What you need for this book
        3. Who this book is for
        4. Conventions
        5. Reader feedback
        6. Customer support
          1. Downloading the example code
          2. Errata
          3. Piracy
          4. Questions
      8. 1. The Splunk Interface
        1. Logging into Splunk
        2. The home app
        3. The top bar
        4. The search & reporting app
          1. The data generator
          2. The summary view
          3. Search
          4. Actions
          5. Timeline
          6. The field picker
            1. Fields
          7. Search results
            1. Options
            2. The events viewer
        5. Using the time picker
        6. Using the field picker
        7. The settings section
        8. Summary
      9. 2. Understanding Search
        1. Using search terms effectively
        2. Boolean and grouping operators
        3. Clicking to modify your search
          1. Event segmentation
          2. Field widgets
          3. Time
        4. Using fields to search
          1. Using the field picker
        5. Using wildcards efficiently
          1. Supplementing wildcards in fields
        6. All about time
          1. How Splunk parses time
          2. How Splunk stores time
          3. How Splunk displays time
          4. How time zones are determined and why it matters
          5. Different ways to search against time
            1. Presets
            2. Relative
            3. Real-time
              1. Windowed real-time versus all-time real-time searches
            4. Date range
            5. Date and time range
            6. Advanced
          6. Specifying time in-line in your search
            1. _indextime versus _time
        7. Making searches faster
        8. Sharing results with others
          1. The URL
          2. Save as report
          3. Save as dashboard panel
          4. Save as alert
          5. Save as event type
        9. Search job settings
        10. Saving searches for reuse
        11. Creating alerts from searches
          1. Enable actions
          2. Action options
          3. Sharing
        12. Summary
      10. 3. Tables, Charts, and Fields
        1. About the pipe symbol
        2. Using top to show common field values
          1. Controlling the output of top
        3. Using stats to aggregate values
        4. Using chart to turn data
        5. Using timechart to show values over time
          1. Timechart options
        6. Working with fields
          1. A regular expression primer
          2. Commands that create fields
            1. eval
            2. rex
          3. Extracting loglevel
            1. Using the extract fields interface
            2. Using rex to prototype a field
            3. Using the admin interface to build a field
            4. Indexed fields versus extracted fields
              1. Indexed field case 1 – rare instances of a common term
              2. Indexed field case 2 – splitting words
              3. Indexed field case 3 – application from source
              4. Indexed field case 4 – slow requests
              5. Indexed field case 5 – unneeded work
        7. Summary
      11. 4. Data Models and Pivots
        1. What is a data model?
        2. What does a data model search?
          1. Data model objects
            1. Object constraining
            2. Attributes
        3. Creating a data model
          1. Filling in the new data model dialog
          2. Editing attributes
        4. Lookup attributes
          1. Children
        5. What is a pivot?
          1. The pivot editor
          2. Working with pivot elements
            1. Filtering your pivots
          3. Split (row or column)
            1. Column values
          4. Pivot table formatting
        6. A quick example
        7. Sparklines
        8. Summary
      12. 5. Simple XML Dashboards
        1. The purpose of dashboards
        2. Using wizards to build dashboards
          1. Adding another panel
            1. A cool trick
        3. Converting the panel to a report
          1. More options
        4. Back to the dashboard
          1. Add input
          2. Edit source
        5. Editing XML directly
        6. UI examples app
        7. Building forms
          1. Creating a form from a dashboard
          2. Driving multiple panels from one form
          3. Post-processing search results
          4. Post-processing limitations
        8. Features replaced
        9. Autorun dashboard
        10. Scheduling the generation of dashboards
        11. Summary
      13. 6. Advanced Search Examples
        1. Using subsearches to find loosely related events
          1. Subsearch
          2. Subsearch caveats
          3. Nested subsearches
        2. Using transaction
          1. Using transaction to determine the session's length
          2. Calculating the aggregate of transaction statistics
          3. Combining subsearches with transaction
        3. Determining concurrency
          1. Using transaction with concurrency
          2. Using concurrency to estimate server load
          3. Calculating concurrency with a by clause
        4. Calculating events per slice of time
          1. Using timechart
          2. Calculating average requests per minute
          3. Calculating average events per minute, per hour
        5. Rebuilding top
        6. Acceleration
          1. Big data - summary strategy
          2. Report acceleration
          3. Report acceleration availability
        7. Summary
      14. 7. Extending Search
        1. Using tags to simplify search
        2. Using event types to categorize results
        3. Using lookups to enrich data
          1. Defining a lookup table file
          2. Defining a lookup definition
          3. Defining an automatic lookup
          4. Troubleshooting lookups
        4. Using macros to reuse logic
          1. Creating a simple macro
          2. Creating a macro with arguments
        5. Creating workflow actions
          1. Running a new search using values from an event
          2. Linking to an external site
          3. Building a workflow action to show field context
            1. Building the context workflow action
            2. Building the context macro
        6. Using external commands
          1. Extracting values from XML
            1. xmlkv
            2. XPath
          2. Using Google to generate results
        7. Summary
      15. 8. Working with Apps
        1. Defining an app
        2. Included apps
        3. Installing apps
          1. Installing apps from Splunkbase
            1. Using Geo Location Lookup Script
            2. Using Google Maps
          2. Installing apps from a file
        4. Building your first app
        5. Editing navigation
        6. Customizing the appearance of your app
          1. Customizing the launcher icon
          2. Using custom CSS
          3. Using custom HTML
            1. Custom HTML in a simple dashboard
            2. Using server-side include in a complex dashboard
        7. Object permissions
          1. How permissions affect navigation
          2. How permissions affect other objects
          3. Correcting permission problems
        8. The app directory structure
          1. Adding your app to Splunkbase
            1. Preparing your app
            2. Confirming sharing settings
            3. Cleaning up our directories
          2. Packaging your app
          3. Uploading your app
        9. Summary
      16. 9. Building Advanced Dashboards
        1. Reasons for working with advanced XML
        2. Reasons for not working with advanced XML
        3. The development process
        4. The advanced XML structure
        5. Converting simple XML to advanced XML
        6. Module logic flow
        7. Understanding layoutPanel
          1. Panel placement
        8. Reusing a query
        9. Using intentions
          1. stringreplace
          2. addterm
        10. Creating a custom drilldown
          1. Building a drilldown to a custom query
          2. Building a drilldown to another panel
          3. Building a drilldown to multiple panels using HiddenPostProcess
        11. Third-party add-ons
          1. Google Maps
          2. Sideview Utils
          3. The Sideview search module
            1. Linking views with Sideview
            2. Sideview URLLoader
            3. Sideview forms
          4. Summary
      17. 10. Summary Indexes and CSV Files
        1. Understanding summary indexes
          1. Creating a summary index
        2. When to use a summary index
        3. When not to use a summary index
        4. Populating summary indexes with saved searches
        5. Using summary index events in a query
        6. Using sistats, sitop, and sitimechart
        7. How latency affects summary queries
        8. How and when to backfill summary data
          1. Using fill_summary_index.py to backfill
          2. Using collect to produce custom summary indexes
        9. Reducing summary index size
          1. Using eval and rex to define grouping fields
          2. Using a lookup with wildcards
          3. Using event types to group results
        10. Calculating top for a large time frame
          1. Summary index searches
        11. Using CSV files to store transient data
          1. Pre-populating a dropdown
          2. Creating a running calculation for a day
        12. Summary
      18. 11. Configuring Splunk
        1. Locating Splunk configuration files
        2. The structure of a Splunk configuration file
        3. The configuration merging logic
          1. The merging order
            1. The merging order outside of search
            2. The merging order when searching
          2. The configuration merging logic
            1. Configuration merging – example 1
            2. Configuration merging – example 2
            3. Configuration merging – example 3
            4. Configuration merging – example 4
          3. Using btool
        4. An overview of Splunk .conf files
          1. props.conf
            1. Common attributes
              1. Search-time attributes
              2. Index-time attributes
              3. Parse-time attributes
              4. Input-time attributes
            2. Stanza types
            3. Priorities inside a type
            4. Attributes with class
          2. inputs.conf
            1. Common input attributes
            2. Files as inputs
              1. Using patterns to select rolled logs
              2. Using blacklist and whitelist
              3. Selecting files recursively
              4. Following symbolic links
              5. Setting the value of the host from the source
              6. Ignoring old data at installation
              7. When to use crcSalt
              8. Destructively indexing files
            3. Network inputs
            4. Native Windows inputs
            5. Scripts as inputs
          3. transforms.conf
            1. Creating indexed fields
              1. Creating a loglevel field
              2. Creating a session field from the source
              3. Creating a tag field
              4. Creating host categorization fields
            2. Modifying metadata fields
              1. Overriding the host
              2. Overriding the source
              3. Overriding sourcetype
              4. Routing events to a different index
            3. Lookup definitions
              1. Wildcard lookups
              2. CIDR wildcard lookups
              3. Using time in lookups
            4. Using REPORT
              1. Creating multivalue fields
              2. Creating dynamic fields
            5. Chaining transforms
            6. Dropping events
          4. fields.conf
          5. outputs.conf
          6. indexes.conf
          7. authorize.conf
          8. savedsearches.conf
          9. times.conf
          10. commands.conf
          11. web.conf
        5. User interface resources
          1. Views and navigation
          2. Appserver resources
          3. Metadata
        6. Summary
      19. 12. Advanced Deployments
        1. Planning your installation
        2. Splunk instance types
          1. Splunk forwarders
          2. Splunk indexer
          3. Splunk search
        3. Common data sources
          1. Monitoring logs on servers
          2. Monitoring logs on a shared drive
          3. Consuming logs in batch
          4. Receiving syslog events
            1. Receiving events directly on the Splunk indexer
            2. Using a native syslog receiver
            3. Receiving syslog with a Splunk forwarder
          5. Consuming logs from a database
          6. Using scripts to gather data
        4. Sizing indexers
        5. Planning redundancy
          1. The replication factor
            1. Configuring your replication factors
              1. Syntax
          2. Indexer load balancing
          3. Understanding typical outages
        6. Working with multiple indexes
          1. The directory structure of an index
          2. When to create more indexes
            1. Testing data
            2. Differing longevity
            3. Differing permissions
            4. Using more indexes to increase performance
          3. The lifecycle of a bucket
          4. Sizing an index
          5. Using volumes to manage multiple indexes
        7. Deploying the Splunk binary
          1. Deploying from a tar file
          2. Deploying using msiexec
          3. Adding a base configuration
          4. Configuring Splunk to launch at boot
        8. Using apps to organize configuration
          1. Separate configurations by purpose
        9. Configuration distribution
          1. Using your own deployment system
          2. Using the Splunk deployment server
            1. Step 1 – deciding where your deployment server will run from
            2. Step 2 – defining your deploymentclient.conf configuration
            3. Step 3 – defining our machine types and locations
            4. Step 4 – normalizing our configurations into apps appropriately
            5. Step 5 – mapping these apps to deployment clients in serverclass.conf
            6. Step 6 – restarting the deployment server
            7. Step 7 – installing deploymentclient.conf
        10. Using LDAP for authentication
        11. Using Single Sign On
        12. Load balancers and Splunk
          1. web
          2. splunktcp
          3. The deployment server
        13. Multiple search heads
        14. Summary
      20. 13. Extending Splunk
        1. Writing a scripted input to gather data
          1. Capturing script output with no date
          2. Capturing script output as a single event
          3. Making a long-running scripted input
        2. Using Splunk from the command line
        3. Querying Splunk via REST
        4. Writing commands
          1. When not to write a command
          2. When to write a command
          3. Configuring commands
          4. Adding fields
          5. Manipulating data
          6. Transforming data
          7. Generating data
        5. Writing a scripted lookup to enrich data
        6. Writing an event renderer
          1. Using specific fields
          2. A table of fields based on field value
          3. Pretty print XML
        7. Writing a scripted alert action to process results
        8. Hunk
        9. Summary
      21. Index