You are previewing Resilience and Reliability on AWS.

Resilience and Reliability on AWS

Cover of Resilience and Reliability on AWS by Jasper Geurtsen... Published by O'Reilly Media, Inc.
  1. Resilience & Reliability on AWS
  2. Foreword
  3. Preface
    1. Audience
    2. Conventions Used in This Book
    3. Using Code Examples
    4. Safari® Books Online
    5. How to Contact Us
    6. Acknowledgments
  4. 1. Introduction
  5. 2. The Road to Resilience and Reliability
    1. Once Upon a Time, There Was a Mason
    2. Rip. Mix. Burn.
    3. Cradle to Cradle
    4. In Short
  6. 3. Crash Course in AWS
    1. Regions and Availability Zones
      1. Route 53: Domain Name System Service
      2. IAM (Identity and Access Management)
    2. The Basics: EC2, RDS, ElastiCache, S3, CloudFront, SES, and CloudWatch
      1. CloudWatch
      2. EC2 (et al.)
      3. RDS
      4. ElastiCache
      5. S3/CloudFront
      6. SES
    3. Growing Up: ELB, Auto Scaling
      1. ELB (Elastic Load Balancer)
      2. Auto Scaling
    4. Decoupling: SQS, SimpleDB & DynamoDB, SNS, SWF
      1. SQS (Simple Queue Service)
      2. SimpleDB
      3. SNS (Simple Notification Service)
      4. SWF (Simple Workflow Service)
  7. 4. Top 10 Survival Tips
    1. Make a Choice
    2. Embrace Change
    3. Everything Will Break
    4. Know Your Enemy
    5. Know Yourself
    6. Engineer for Today
    7. Question Everything
    8. Don’t Waste
    9. Learn from Others
    10. You Are Not Alone
  8. 5. elasticsearch
    1. Introduction
    2. EC2 Plug-in
    3. Missing Features
    4. Conclusion
  9. 6. Postgres
    1. Pragmatism First
    2. The Challenge
      1. Tablespaces
    3. Building Blocks
      1. Configuration with userdata
      2. IAM Policies (Identity and Access Management)
      3. Postgres Persistence (backup/restore)
      4. Self Reliance
    4. Monitoring
    5. Conclusion
  10. 7. MongoDB
    1. How It Works
      1. Replica Set
      2. Backups
    2. Auto Scaling
    3. Monitoring
    4. Conclusion
  11. 8. Redis
    1. The Problem
    2. Our Approach
    3. Implementation
      1. userdata
      2. Redis
      3. Chaining (Replication)
    4. In Practice
  12. 9. Logstash
    1. Build
    2. Shipper
      1. Output Plug-in
    3. Reader
      1. Input Plug-in
      2. Grok
    4. Kibana
  13. 10. Global (Content) Delivery
    1. CloudFront
      1. (Live) Streaming
      2. CloudFormation
      3. Orchestration
    2. Route 53
      1. Global Database
  14. 11. Conclusion
  15. Copyright
O'Reilly logo

Chapter 9. Logstash

Managing logs is difficult. It gets even more complicated when your infrastructure is bigger. And, making it dynamic (changing instances, all the time) doesn’t help at all.

You have commercial services like Splunk and Loggly, but they can get very expensive, very quickly. We would prefer to run it ourselves, providing that the following are true:

  • We can design this to be reliable.
  • It has a small footprint for log shipping.
  • It comes with out of the box interface for reading (analyzing).
  • It is scalable.

Logstash calls itself a log management solution. You can collect events, parse them (add meaning), search, and store them. Logstash can be easily decoupled, and run in a setup (see Figure 9-1).

Logstash distributed logging

Figure 9-1. Logstash distributed logging

The base setup should be able to handle several hundred events per second. The shippers have a small footprint. The reader (and interface) run on high CPU medium. For the elasticsearch (powering the interface) we’ll use two high memory medium instances.


Logstash works with input, filter, and output definitions. Most of the time shippers read from file (input), do not filter very much, and write out to a middleware message bus (output). Logstash comes with many plug-ins.

The default isolated message bus is RabbitMQ; you can also use Redis, but we want to use SQS, of course.

The latest logstash.jar comes with the AWS SDK. This means ...

The best content for your career. Discover unlimited learning on demand for around $1/day.