You are previewing Logging and Log Management.
O'Reilly logo
Logging and Log Management

Book Description

Effectively analyzing large volumes of diverse logs can pose many challenges. Logging and Log Management helps to simplify this complex process using practical guidance and real-world examples. Packed with information you need to know for system, network and security logging. Log management and log analysis methods are covered in detail, including approaches to creating useful logs on systems and applications, log searching and log review.



  • Comprehensive coverage of log management including analysis, visualization, reporting and more
  • Includes information on different uses for logs -- from system operations to regulatory compliance
  • Features case Studies on syslog-ng and actual real-world situations where logs came in handy in incident response
  • Provides practical guidance in the areas of report, log analysis system selection, planning a log analysis system and log data normalization and correlation

Table of Contents

  1. Cover image
  2. Title page
  3. Table of Contents
  4. Copyright
  5. Acknowledgments
    1. Dr. Anton A. Chuvakin
    2. Kevin J. Schmidt
    3. Christopher Phillips
  6. About the Authors
  7. About the Technical Editor
  8. Foreword
  9. Preface
    1. Intended Audience
    2. Prerequisites
    3. Organization of the Book
    4. Chapter 5: Case Study: syslog-ng
    5. Chapter 6: Covert logging
    6. Chapter 7: Analysis Goals, Planning and Preparation: What Are We Looking for?
    7. Chapter 8: Simple Analysis Techniques
    8. Chapter 9: Filtering, Matching and Correlation
    9. Chapter 10: Statistical Analysis
    10. Chapter 11: Log Data Mining
    11. Chapter 12: Reporting and Summarization
    12. Chapter 13: Visualizing Log Data
    13. Chapter 14: Logging Laws and Logging Mistakes
    14. Chapter 15: Tools for Log Analysis and Collection
    15. Chapter 16: Log Management Procedures: Escalation, Response
    16. Chapter 17: Attacks Against Logging Systems
    17. Chapter 18: Logging for Programmers
    18. Chapter 19: Logs and Compliance
    19. Chapter 20: Planning Your Own Log Analysis System
    20. Chapter 21: Cloud Logging
    21. Chapter 22: Log Standard and Future Trends
  10. Chapter 1. Logs, Trees, Forest: The Big Picture
    1. Introduction
    2. Log Data Basics
    3. A Look at Things to Come
    4. Logs Are Underrated
    5. Logs Can Be Useful
    6. People, Process, Technology
    7. Security Information and Event Management (SIEM)
    8. Summary
    9. References
  11. Chapter 2. What is a Log?
    1. Introduction
    2. Logs? What logs?
    3. Criteria of Good Logging
    4. Summary
    5. References
  12. Chapter 3. Log Data Sources
    1. Introduction
    2. Logging Sources
    3. Log Source Classification
    4. Summary
  13. Chapter 4. Log Storage Technologies
    1. Introduction
    2. Log Retention Policy
    3. Log Storage Formats
    4. Database Storage of Log Data
    5. Hadoop Log Storage
    6. The Cloud and Hadoop
    7. Log Data Retrieval and Archiving
    8. Summary
    9. References
  14. Chapter 5. syslog-ng Case Study
    1. Introduction
    2. Obtaining syslog-ng
    3. What Is syslog-ngsyslog-ng?
    4. Example Deployment
    5. Troubleshooting syslog-ng
    6. Summary
    7. References
  15. Chapter 6. Covert Logging
    1. Introduction
    2. Complete Stealthy Log Setup
    3. Logging in Honeypots
    4. Covert Channels for Logging Brief
    5. Summary
    6. References
  16. Chapter 7. Analysis Goals, Planning, and Preparation: What Are We Looking for?
    1. Introduction
    2. Goals
    3. Planning
    4. Preparation
    5. Summary
  17. Chapter 8. Simple Analysis Techniques
    1. Introduction
    2. Line by Line: Road to Despair
    3. Simple Log Viewers
    4. Limitations of Manual Log Review
    5. Responding to the Results of Analysis
    6. Examples
    7. Summary
    8. References
  18. Chapter 9. Filtering, Normalization, and Correlation
    1. Introduction
    2. Filtering
    3. Normalization
    4. Correlation
    5. Common Patterns to Look For
    6. The Future
    7. Summary
    8. Reference
  19. Chapter 10. Statistical Analysis
    1. Introduction
    2. Frequency
    3. Baseline
    4. Machine Learning
    5. Combining Statistical Analysis with Rules-based Correlation
    6. Summary
    7. References
  20. Chapter 11. Log Data Mining
    1. Introduction
    2. Data Mining Intro
    3. Log Mining Intro
    4. Log Mining Requirements
    5. What We Mine For?
    6. Deeper into Interesting
    7. Summary
    8. References
  21. Chapter 12. Reporting and Summarization
    1. Introduction
    2. Defining the Best Reports
    3. Network Activity Reports
    4. Resource Access Reports
    5. Malware Activity Reports
    6. Critical Errors and Failures Reports
    7. Summary
  22. Chapter 13. Visualizing Log Data
    1. Introduction
    2. Visual Correlation
    3. Real-time Visualization
    4. Treemaps
    5. Log Data Constellations
    6. Traditional Log Data Graphing
    7. Summary
    8. References
  23. Chapter 14. Logging Laws and Logging Mistakes
    1. Introduction
    2. Logging Laws
    3. Logging Mistakes
    4. Summary
    5. Reference
  24. Chapter 15. Tools for Log Analysis and Collection
    1. Introduction
    2. Outsource, Build, or Buy
    3. Basic Tools for Log Analysis
    4. Utilities for Centralizing Log Information
    5. Log Analysis Tools—Beyond the Basics
    6. Commercial Vendors
    7. Summary
    8. References
  25. Chapter 16. Log Management Procedures: Log Review, Response, and Escalation
    1. Introduction
    2. Assumptions, Requirements, and Precautions
    3. Common Roles and Responsibilities
    4. PCI and Log Data
    5. Logging Policy
    6. Review, Response, and Escalation Procedures and Workflows
    7. Validation of Log Review
    8. Logbook—Evidence of Exception of Investigations
    9. PCI Compliance Evidence Package
    10. Management Reporting
    11. Periodic Operational Tasks
    12. Additional Resources
    13. Summary
    14. References
  26. Chapter 17. Attacks Against Logging Systems
    1. Introduction
    2. Attacks
    3. Summary
    4. References
  27. Chapter 18. Logging for Programmers
    1. Introduction
    2. Roles and Responsibilities
    3. Logging for Programmers
    4. Security Considerations
    5. Performance Considerations
    6. Summary
    7. References
  28. Chapter 19. Logs and Compliance
    1. Introduction
    2. PCI DSS
    3. ISO2700x Series
    4. HIPAA
    5. FISMA
    6. Summary
  29. Chapter 20. Planning Your Own Log Analysis System
    1. Introduction
    2. Planning
    3. Software Selection
    4. Policy Definition
    5. Architecture
    6. Scaling
    7. Summary
  30. Chapter 21. Cloud Logging
    1. Introduction
    2. Cloud Computing
    3. Cloud Logging
    4. Regulatory, Compliance, and Security Issues
    5. Big Data in the Cloud
    6. SIEM in the Cloud
    7. Pros and Cons of Cloud Logging
    8. Cloud Logging Provider Inventory
    9. Additional Resources
    10. Summary
    11. References
  31. Chapter 22. Log Standards and Future Trends
    1. Introduction
    2. Extrapolations of Today to the Future
    3. Log Future and Standards
    4. Desired Future
    5. Summary
  32. Index