You are previewing Programming Amazon EC2.

Programming Amazon EC2

Cover of Programming Amazon EC2 by Flavia Paganelli... Published by O'Reilly Media, Inc.


Both SQS and SimpleDB are kind of passive, or static. You can add things, and if you need something from it, you have to pull. This is OK for many services, but sometimes you need something more disruptive—you need to push instead of pull. This is what Amazon SNS gives us. You can push information to any component that is listening, and the messages are delivered right away. Table 4-2 shows a comparison of SNS and SQS.

Table 4-2. Comparing SNS to SQS

Doesn’t require receivers to subscribeReceivers subscribe and confirm
Each message is normally handled by one receiver, then deletedNotifications are “broadcasted” to all subscribers
Pull model: messages are read by receivers when they wantPush model: notifications are automatically pushed to all subscribers

SNS is not an easy service, but it is incredibly versatile. Luckily, we are all living in “the network society,” so the essence should be familiar to most of us. It is basically the same concept as a mailing list or LinkedIn group—there is something you are interested in (a topic, in SNS-speak), and you show that interest by subscribing. Once the topic verifies that you exist by confirming the subscription, you become part of the group receiving messages on that topic.

Of course, notifications are not new—we have been using them for a while now. In software systems, one of the more successful models of publish/subscribe communication is the event model. Events are part of most high-level programming languages like Java and Objective-C. In a software system that spans multiple components, there is the notion of an event. In the field of web development, events are abundant, though mostly implicit. Every PHP script or Rails Action Controller method (public, of course) is an event handler; it reacts to a user requesting a page or another piece of software talking to an API.

There is one important difference between SNS and the events in programming languages as we know them: permissions are implemented differently. In Java, for example, you don’t need explicit permissions to subscribe to events from a certain component. This is left to the environment the application operates in. In the Android mobile platform, on the other hand, you have to request access to, say, the compass or the Internet in your application’s manifest file. If you don’t specify that you need those privileges, you get a compile-time error. Because SNS is much more public, not just confined to one user on one machine, permissions are part of the events themselves. You need permission from the topic owner to receive information, and you need permission from the recipient to send information.

SNS can be seen as an event system, but how does it work? In SNS, you create topics. Topics are the conduits for sending (publishing) and receiving messages, or events. Anyone with an AWS account can subscribe to a topic, though that doesn’t mean they will be automatically permitted to receive messages. And the topic owner can subscribe non-AWS users on their behalf. Every subscriber has to explicitly opt in, though that term is usually related to mailing lists and spam. But it is the logical consequence in an open system like the Web (you can see this as the equivalent of border control in a country).

The most interesting thing about SNS has to do with the subscriber, the recipient of the messages. A subscriber can configure an endpoint, specifying how and where the message will be delivered. Currently SNS supports three types of endpoints: HTTP/HTTPS, email, and SQS. And this is exactly the reason we feel it is more than a notification system. You can integrate an API using SNS, enabling totally different ways of execution.

But, even as just a notification system, SNS rocks. (And Amazon promises to support SMS endpoints, as well.)


Happily, SNS is now included in the AWS Console, so you can take a look for yourself and experiment with topics, subscriptions, and messages.

Example 1: Implementing Contest Rules for Kulitzer (Ruby)

A contest is the central component of Kulitzer. Suppose our company, 9Apps, needs a new identity. With our (still) limited budget, we can’t ask five high-profile marketing agencies to come to us and pitch for this project—we just don’t have that much money. What we can do, however, is ask a number of designers in our community to participate in a contest organized on Kulitzer. And because we use Kulitzer, we can easily invite our partners and customers to vote on the best design.

In our quest for flexibility, we realize that this problem would benefit from using SNS. The lifecycle of a contest is littered with events. We start a contest, we invite participants and a jury, and admission is opened and closed, as is the voting. And during the different periods of the contest, we want to notify different interested parties of progress. Later, we can add all sorts of components that “listen to a contest” and take action when certain events occur, for example, leaving messages on Facebook walls or sending tweets.

We could implement this with SNS by having a few central topics, like “registration” for indicating that a contest is open for participants, “admission” when the admission process starts, and “judging” when the voting starts. We could then have subscribers to each of these topics, but they would receive notifications related to all existing contests. It would move the burden of implementing security to the components themselves.

Instead, we choose to implement this differently. The idea is that each contest has its own topics for the kind of events we mentioned. The contest will handle granting/revoking permissions with SNS so that listeners are simpler to implement. It will give us much more flexibility to add features to Kulitzer later on.


The current limits for SNS —which is in beta— state that you can have a maximum of 100 topics per account. This would clearly limit us when choosing to have several topics per contest. But we contacted Amazon about this issue, and its reply expressed interest in our use case, so it will not be a problem.

This is not the first time we needed to surpass the limits imposed by AWS. When this happens, Amazon requests that you explain your case. Amazon will then analyze it and decide if these limitations can be lifted for you.

Preparing tools for SNS in Ruby

The RightAWS library doesn’t officially support SNS yet, but we found a branch by Brendon Murphy that does. The branch is called add_sns, and you can download it from GitHub and create the gem yourself.

You will need to install some dependencies:

sudo gem install hoe
sudo gem install rcov

Then download the sources, extract them, and run the following to generate the gem in the pkg directory:

rake gem

To install it, run the following from the pkg directory:

sudo gem install right_aws-1.11.0.gem


If you installed the official version of RightAWS before, you can uninstall it before (or after) installing this one, to make sure you use the right one. To do that, execute the following:

sudo gem uninstall right_aws

Topics per contest

Here, we show how to create the topics for a specific contest:

require "right_aws"

contest_id = 12

# create the topics

registration = sns.create_topic("#{contest_id}_registration")
# set a human readable name for the topic
registration.display_name = "Registration process begins"

admission = sns.create_topic("#{contest_id}_admission")
admission.display_name = "Admission process begins"

judging = sns.create_topic("#{contest_id}_judging")
judging.display_name = "Judging process begins"

Subscribing to registration updates

There are two steps to subscribing to a topic:

  1. Call the subscribe method, providing the type of endpoint (email, email-json, http, https, sqs).

  2. Confirm the subscription by calling confirm_subscription.

For the second step, SNS will send the endpoint a request for confirmation in the form of an email message or an HTTP request, depending on the type of endpoint. In all cases, a string token is provided, which needs to be sent back in the confirm_subscription call. If the endpoint is email, the user just clicks on a link to confirm the subscription, and the token is sent as a query string parameter of the URL (shown in Figure 4-7).

If the endpoint is http, https, or email-json, the token is sent as part of a JSON object.

Step one is subscribing some email addresses to a topic:

require "right_aws"

contest_id = 12

# get the judging topic
topic = sns.create_topic("#{contest_id}_judging")

# Subscribe a few email addresses to the topic.
# This will return "Pending Confirmation" and send a confirmation
# email to the target email address which contains a "Confirm Subscription"
# link back to Amazon Web Services.  Other possible subscription protocols
# include http, https, email, email-json and sqs

%w[].each do |email|
  topic.subscribe("email", email)

# Let's subscribe this address as email-json, imagining it
# is an email box feeding to a script parsing the content.  Note
# that the confirmation email this address receives will contain
# content as JSON
topic.subscribe("email-json", "")

# Another kind of endpoint is http
Subscription confirmation via email

Figure 4-7. Subscription confirmation via email

In step two, the endpoint confirms the subscription by sending back the token:

require "right_aws"

# Now, imagine you subscribe an internal resource to the topic and instruct
# it to use the http protocol and post to a url you provide.  It will receive
# a post including the following bit of JSON data:
 "Type" : "SubscriptionConfirmation",
 "MessageId" : "b00e1384-6a4f-4bc5-abd5-9b7f82e3cff4",
 "Token" : "51b2ff3edb4487553c7dd2f29566c2aecada20b9...",
 "TopicArn" : "arn:aws:sns:us-east-1:235698110812:12_judging",
 "Message" : "You have chosen to subscribe to the topic
    To confirm the subscription, visit the SubscribeURL included in this
 "SubscribeURL" : 
 "Timestamp" : "2011-01-07T11:41:02.417Z",
 "SignatureVersion" : "1",
 "Signature" : "UHWoZfMkpH/FrhICs6An0cTtjjcj5nBEweVbWgrARD5B..."

# You can confirm the subscription with a call to confirm_subscription.
# Note that in order to receive messages published to a topic, the subscriber
# must be confirmed.


arn = "arn:aws:sns:us-east-1:724187402011:12_judging"
topic = sns.topic(arn)

token = "51b2ff3edb4487553c7dd2f29566c2aecada20b9..."

Publishing messages to a topic

Publish messages to a topic as follows:

# Sending a message allows for an optional subject to be included along with
# the required message itself.

# Find the sns topic using the arn.  Note that you can call
# RightAws::SNS#create_topic if you only know the topic name
# and the current AWS api will fetch it for you.  Otherwise,
# you need to store the arn for future lookups.  It's probably
# advisable to store the arn's anyway as they serve as a reliable
# uid.


arn = "arn:aws:sns:us-east-1:235698110812:12_judging"
topic = sns.topic(arn)

message = "Dear Judges, admission is closed, you can now start evaluating 
optional_subject = "Judging begins"
topic.send_message(message, optional_subject)

Deleting a topic

Of course, when a topic is no longer valid, you can delete it using the following:


Publishing to Facebook

The use of SNS is not limited to sending messages to subscribed users, of course. In Kulitzer, when the status of a contest changes, we might want to update a particular Facebook account to tell the world about it. In Rails, we can implement this by having a separate FacebookController that is subscribed to the different topics and publishes messages to Facebook when notified by SNS of an event.

Example 2: PDF Processing Status (Monitoring) for Marvia (PHP)

The core of Marvia is the PDF engine, for lack of a better term. The PDF engine is becoming a standalone service. We are slowly extracting features from the app, moving them to the PDF engine. We introduced priority queues to distinguish between important and less important jobs. We extracted accounts so we can share information and act on that information, for example keeping track of the number of PDFs generated.

One of the missing parts of the PDF engine is notification. We send jobs to the engine, and it helps us keep track of what is created. But we do not know the status of a job. The only way to learn the status is to check whether the PDF has been generated. It would be great if the process that creates the job could register an event handler to be called with status updates. It could be something simple like a message, or something more elaborate like a service that moves the PDF to a Dropbox account. We could even tell the PDF engine to add a message to a particular queue for printing, for example.

But we’ll start with what we need right now: notifying the customer with an email message when the PDF is ready. The emails SNS sends are not really customizable, and we don’t want to expose our end users to notification messages with opt-out pointing to Amazon AWS. We want to hide the implementation as much as possible. Therefore, we have notifications sent to our app, which sends the actual email message.

Creating a topic for an account

Creating a topic is not much different from creating a queue in SQS or a domain in SimpleDB. Topic names must be made up of only uppercase and lowercase ASCII letters, numbers, and hyphens, and must be between 1 and 256 characters long. We want a simple mechanism for creating unique topic names, but we still want to be flexible. An easy way is to create a hash of what we would have used if we could. We will create a topic for each Marvia customer account, so we will use the customer’s provided email address. In the following, we create a topic for the customer 9apps:


    define('AWS_SECRET_KEY', 'w2Y3dx82vcY1YSKbJY51GmfFQn3705ftW4uSBrHn');

    $sns = new AmazonSNS();

    $topic = md5('');

    # if topic already exists, create_topic returns it
    $response = $sns->create_topic($topic);
    $response->isOK() or
        die('could not create topic ' . $topic);


    function pr($var) { print '<pre>'; print_r($var); print '</pre>'; }

Subscription and confirmation

The endpoint for notification is http; we are not sending any sensitive information, so HTTPS is not necessary. But we do want to prepare a nicely formatted email message, so we won’t use the standard email endpoint. We can then generate the content of the email message ourselves.

Below, we call subscribe to add an http endpoint to the topic:

    require_once( '/usr/share/php/AWSSDKforPHP/sdk.class.php');

    define('AWS_SECRET_KEY', 'w2Y3dx82vcY1YSKbJY51GmfFQn3705ftW4uSBrHn');

    $sns = new AmazonSNS();

    $topic = md5('');

    $response = $sns->create_topic( $topic);
    $response->isOK() or
        die( 'could not create topic ' . $topic);

    # subscribe to the topic by
    # passing topic arn (topic identifier),
    # type of endpoint 'http',
    # and the given url as endpoint
    $response = $sns->subscribe(


    function pr($var) { print '<pre>'; print_r($var); print '</pre>'; }

Before messages are delivered to this endpoint, it has to be confirmed by calling confirm_subscription. In the code below, you can see that the endpoint is responsible for both confirming the subscription and handling messages. A JSON object is posted, and this indicates if the HTTP POST is a notification or a confirmation. The rest of the information, such as the message (if it is a notification), topic identifier (ARN), and token (if it is a subscription confirmation), is also part of the JSON object:

    require_once( '/usr/share/php/AWSSDKforPHP/sdk.class.php');

    define('AWS_SECRET_KEY', 'w2Y3dx82vcY1YSKbJY51GmfFQn3705ftW4uSBrHn');

    $sns = new AmazonSNS();

    # we only accept a POST, before we get the raw input (as json)
    if ( $_SERVER['REQUEST_METHOD'] === 'POST' ) {
        $json = trim(file_get_contents('php://input'));

        $notification = json_decode($json);
        # do we have a message? or do we need to confirm?
        if( $notification->{'Type'} === 'Notification') {

            # a message is text, in our case json so we decode and proceed...

            $message = json_decode($notification->{'Message'});

            # and now we can act on the message
            # ...

        } else if($notification->{'Type'} === 'SubscriptionConfirmation') {

            # we received a request to confirm subscription, let's confirm...

            $response = $sns->confirm_subscription(

            $response->isOK() or
                die("could not confirm subscription for topic 'status'");

Publishing and receiving status updates

Publishing a message is not difficult. If you are the owner, you can send a text message to the topic. We need to pass something with a bit more structure, so we use JSON to deliver an object:

    require_once( '/usr/share/php/AWSSDKforPHP/sdk.class.php');

    define('AWS_SECRET_KEY', 'w2Y3dx82vcY1YSKbJY51GmfFQn3705ftW4uSBrHn');

    $sns = new AmazonSNS();

    $topic = md5( '');
    $job = "457964863276";

    $response = $sns->create_topic( $topic);
    $response->isOK() or
        die( 'could not create topic ' . $topic);

    $response = $sns->publish(
        json_encode( array( "job" => $job,
                        "result" => "200",
                        "message" => "Job $job finished"))

    pr( $response->body);

    function pr($var) { print '<pre>'; print_r($var); print '</pre>'; }

Example 3: SNS in Decaf (Java)

An interesting idea is to use the phone as a beacon, sending notifications to the world if an unwanted situation occurs. We could do that easily with Decaf on Android. We already monitor our EC2 instances with Decaf, and we could use SNS to notify others. Though interesting, the only useful application of this is to check the health of an app in a much more distributed way. Central monitoring services can only implement a limited number of network origins. In the case of Layar, for example, we can add this behavior to the mobile clients.

For now, we’ll implement something else. Just as with SimpleDB and SQS, we are curious about the state of SNS in our application. For queues, we showed how to monitor the number of items in the queue, and for SimpleDB, we did something similar. With SNS, we want to know if the application is operational.

To determine if a particular topic is still working, we only have to listen in for a while by subscribing to the topic. If we see messages, the SNS part of the app works. What we need for this is simple browsing/searching for topics, subscribing to topics with an email address, and sending simple messages.

Listing topics

Getting a list of available topics is very simple. The number of results are limited and paginated as in SimpleDB. The code below lists all topics in the given account for the region us-east-1:

import com.amazonaws.auth.BasicAWSCredentials;

// ...

// prepare the credentials
String accessKey = "AKIAIGKECZXA7AEIJLMQ";
String secretKey = "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";

// create the SNS service
AmazonSNS snsService = new AmazonSNSClient(
    new BasicAWSCredentials(accessKey, secretKey));

// set the endpoint for us-east-1 region

List<Topic> topics = new ArrayList<Topic>();
String nextToken = null;

do {

    // create the request, with nextToken if not empty
    ListTopicsRequest request = new ListTopicsRequest();
    if (nextToken != null) request = request.withNextToken(nextToken);

        // call the web service
        ListTopicsResult result = snsService.listTopics(request);

        nextToken = result.getNextToken();

        // get that list of topics

// go on if there are more elements    
} while (nextToken != null);

// show the list of topics...

Subscribing to a topic via email

Given the list of topics, we can give the user the possibility of subscribing to some of these with his email address. For that, we just need to use the ARN and the email address of choice:


// ...

// get sns service
AmazonSNS snsService = ...

// obtain the email of the user who wants to subscribe
String address = ...

// obtain the topic chosen by the user (from the list obtained above)
Topic topic = ...
String topicARN = topic.getTopicArn(); 

// subscribe the user to the topic with protocol = "email"
snsService.subscribe(new SubscribeRequest(topicARN, "email", address));

The user will then receive an email message requesting confirmation of this subscription.

Sending notifications on a topic

It’s also very easy to send notifications about a given topic. Just use the topic ARN and provide a String for a message and an optional subject:


// ...

AmazonSNS snsService = ...

// obtain the topic chosen by the user (from the list obtained above)
Topic topic = ...
String topicARN = topic.getTopicArn(); 

snsService.publish(new PublishRequest(topicARN,
    "A server is in trouble!", "server alarm"));

The best content for your career. Discover unlimited learning on demand for around $1/day.