You are previewing Programming Amazon Web Services.

Programming Amazon Web Services

Cover of Programming Amazon Web Services by James Murty Published by O'Reilly Media, Inc.
  1. Programming Amazon Web Services
    1. SPECIAL OFFER: Upgrade this ebook with O’Reilly
    2. A Note Regarding Supplemental Files
    3. Preface
      1. What’s in This Book?
      2. Ruby and Interactive Examples
      3. Conventions Used in This Book
      4. Using Code Examples
      5. Safari® Enabled
      6. How to Contact Us
      7. Acknowledgments
    4. 1. Infrastructure in the Cloud
      1. Amazon Web Services for Infrastructure
      2. Thinking Like Amazon
      3. Reality Check
      4. Interfaces: REST and Query Versus SOAP
    5. 2. Interacting with Amazon Web Services
      1. REST-Based APIs
      2. User Authentication
      3. Performing AWS Requests
    6. 3. S3: Simple Storage Service
      1. S3 Overview
      2. Interacting with S3
      3. Buckets
      4. Objects
      5. Alternative Hostnames
      6. Access Control Lists
      7. Server Access Logging (Beta)
      8. Signed URIs
      9. Distributing Objects with BitTorrent
    7. 4. S3 Applications
      1. Share Large Files
      2. Online Backup with AWS::S3
      3. S3 Filesystem with ElasticDrive
      4. Mediated Access to S3 with JetS3t
    8. 5. EC2: Elastic Compute Cloud (Beta)
      1. EC2 Overview
      2. Interacting with EC2
      3. Keypairs
      4. Network Security by IP
      5. Finding Amazon Machine Images
      6. Controlling Instances
      7. Log In to an Instance
      8. Security Groups
      9. Managing and Sharing AMIs
      10. Console Output and Instance Reboot
    9. 6. Using EC2 Instances and Images
      1. EC2 Instances in Detail
      2. Data Management in EC2
      3. Modifying an AMI
      4. Registering an AMI
      5. Create an AMI from Scratch
    10. 7. EC2 Applications
      1. Dynamic DNS
      2. On-Demand VPN Server with OpenVPN
      3. Web Photo Album with Gallery 2
    11. 8. SQS: Simple Queue Service
      1. SQS Overview
      2. Interacting with SQS
      3. Queues
      4. Messages
      5. Queue Attributes
      6. Queue Access Control
    12. 9. SQS Applications
      1. Messaging Simulator
      2. Distributed Application Services with BOTO
      3. Automated Management of EC2 Instance Pools with Lifeguard
    13. 10. FPS: Flexible Payments Service (Beta)
      1. FPS Overview
      2. Interacting with FPS
      3. Managing Your Tokens
      4. Acquiring Third-Party Tokens
      5. Pay Now Widgets
    14. 11. FPS Transactions and Accounts
      1. Performing FPS Transactions
      2. Account Management and Information
    15. 12. FPS Advanced Topics
      1. Gatekeeper Language Guide
      2. Micropayments with FPS
      3. Building a Marketplace Application
      4. Subscribing to FPS Event Notifications
    16. 13. SimpleDB (Beta)
      1. SimpleDB Overview
      2. Interacting with SimpleDB
      3. Domains
      4. Items and Attributes
      5. Representing Data in SimpleDB
      6. Performing Queries
      7. Stock Price Database: A Mini SimpleDB Application
    17. A. AWS Resources
      1. AWS Online Resources
      2. Client Tools
      3. API Libraries
      4. Third-Party AWS Solutions
    18. B. AWS API Error Codes
      1. S3: Simple Storage Service
      2. EC2: Elastic Compute Cloud
      3. SQS: Simple Queue Service
      4. FPS: Flexible Payments Service
      5. SimpleDB
    19. Index
    20. About the Author
    21. Colophon
    22. SPECIAL OFFER: Upgrade this ebook with O’Reilly

Distributed Application Services with BOTO

The full value of Amazon’s SQS messaging service is most evident when it is used to connect the components of a distributed application, bringing together a set of computing resources to cooperate on a task. In this application, we will apply everything we have learned about the Amazon’s Web Services (AWS) so far to create a distributed application that can run arbitrary Linux applications to process data files any way you wish. This application will use S3 for storage, EC2 instances for computer processing power, and SQS messaging to tie the whole system together.

In this application, we will use the open source boto library (version 0.9d) as a starting point. This Python library includes support for the three Amazon services we will use in this example, but most importantly it also includes a framework to create and run service components that process SQS messages. The boto library will allow us to concentrate on the functionality of our application without having to build the underlying plumbing from scratch.

The Boto Services Framework

Mitch Garnaat, the creator of boto, has written an excellent article explaining how his library can be used as the basis for a distributed application that converts video files from one format to another. In the article “Monster Muck Mashup: Mass Video Conversion Using AWS,” he describes the architecture, message format, and processing steps used in the application, as well as offering a prepackaged public ...

The best content for your career. Discover unlimited learning on demand for around $1/day.