Getting Started with Mule Cloud Connect

Chapter 1. Getting Started

It all starts with a simple API that publishes someone’s status to Facebook, sends a Tweet, or updates a contact in Salesforce. As you start to integrate more and more of these external services with your applications, trying to identify the tasks that one might want to perform when you’re surrounded by SOAP, REST, JSON, XML, GETs, PUTs, POSTs, and DELETEs, can be a real challenge.

Open APIs are all about endpoints. Most services follow the current trend of providing a RESTful endpoint, others use older RPC-based protocols such as SOAP or XML-RPC, some use newer “real-time”, push-focused endpoints like WebSockets or HTTP Streaming, others may offer a number of different endpoints to meet different requirements, and some just use what seems to be best for a specific job, which might mean not strictly following protocol rules. This is one of the biggest challenges with open APIs: inconsistency. Figure 1-1 shows the estimated popularity of different styles of APIs.

Each API is different, with different data formats and authorization mechanisms. One API’s interpretation of REST may even differ from another. One reason for this is the nature of REST itself. The RESTful principles come from a paper published by Roy Fielding in 2000 and since then RESTful services have dominated SOAP-based services on the web year after year. Although REST services have many advantages over SOAP-based services, the original paper only included a set of constraints and provides no specification about how to define a RESTful API and handle things like URI schemes, authentication, error handling, and more.

By observing the vastly different opinions out there, there is no one right way to define a RESTful API, which has resulted in many inconsistencies, even between APIs from the same service provider. Top that off with the remaining SOAP services and newer technologies such as HTTP Streaming and you’re left with a lot of different API styles and protocols to learn. Working with all these APIs can just be too damn hard, and this is where Mule Cloud Connect comes in. Mule Cloud Connect is a powerful, lightweight toolset providing a consistent interface to a large number of cloud, SaaS, social media, and Web 2.0 APIs.

Figure 1-1. Distribution of API protocols

Cloud Connectors versus the REST of the World

There are many different levels of working with APIs. To put Cloud Connectors into context, let’s first look at some other approaches to integrating APIs.

To demonstrate, we will use the GeoNames API as our external service. I tend to use GeoNames as the API equivalent of the Northwind database, because it’s easy to consume (providing both XML and JSON formats) and does not require any account setup for demo purposes.

GeoNames is a worldwide geographical database that contains over 10 million geographical names and consists of 7.5 million unique features, of which 2.8 million are populated places and 5.5 million are alternate names. All features are categorized into one out of nine feature classes and further subcategorized into one out of 645 feature codes. In addition to listing names of places in various languages, data stored by GeoNames includes latitude, longitude, elevation, population, administrative subdivision, and postal codes. GeoNames features include direct and reverse geocoding, finding places through postal codes, finding places next to a given place, and finding Wikipedia articles about neighboring places.

Transport-Specific Clients

Transport-specific clients deal directly with APIs over the wire. These clients deal with the actual bytes that pass between your application and the external API. For a RESTful service, it requires you to build a URL and associate it with the correct URI parameters and HTTP headers. For a SOAP-based service, it requires you to build the contents of the HTTP POST yourself, including the SOAP:Envelope and any WS-* content. Example 1-1 shows a very simple Java snippet for constructing a simple client for a RESTful service using Java’s HTTP packages.

Example 1-1. RESTful Java client with java.net URL

URL url = new URL("http://api.geonames.org/findNearbyJSON" +
                  "?lat=37.51&amp;lng=-122.18&amp;username=demo");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");

if (conn.getResponseCode() != 200) { throw new RuntimeException("Failed :
                                     HTTP error code : " + conn.getResponseCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
                    (conn.getInputStream())));

String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) { System.out.println(output);
}

conn.disconnect();

This is the most abstract way of working with APIs. The semantics of HTTP libraries match the HTTP protocol and not REST or SOAP APIs specifically. This leaves it up to you to construct URLs, build up request structures, and write them to and from input and output streams, requiring you to know the API very well.

When you start working with more complex APIs that require connection or state management, you’re left to do this manually, which is error prone and requires far more effort to handle reliably.

Language-Specific Clients

Language-specific libraries, such as Jersey clients for Rest APIs or Apache CXF for SOAP APIs, wrap the underlying protocols in methods that are more familiar and comfortable for programmers in that language. For example, Example 1-2 shows a very simple code snippet for using Jersey to invoke RESTful service.

Example 1-2. Jersey REST client

WebResource webResource = client.resource("http://api.geonames.org/findNearbyJSON");
MultivaluedMap queryParams = new MultivaluedMapImpl();
queryParams.add("lat", "lat");
queryParams.add("lng", "-122.18");
queryParams.add("username", "demo");
String s = webResource.queryParams(queryParams).get(String.class);

Using this example, the Jersey client libraries abstract away a lot of the HTTP specifics and make API clients a lot clearer by providing short code that helps express the semantics of the particular API protocol. This is one advantage over using transports, but you’re still left importing WSDLs for SOAP services and object binding to and from request structures. If you’re using multiple protocols, you may have to learn and maintain multiple libraries. Because they are generic and not specific to any particular API, you will still have to write custom code to work with each API’s little idiosyncrasies or custom features such as session-based authentication and OAuth.

Service-Specific Client Libraries

A client library specifically developed for a particular API, such as Twitter4j for the Twitter APIs, makes things easier by extracting away a lot of the protocol and transport specifics. Example 1-3 shows an example of working with GeoNames’ Java library.

Example 1-3. Service-specific client library

WebService.setUserName("demo");
ToponymSearchCriteria searchCriteria = new ToponymSearchCriteria();
searchCriteria.setQ("zurich");
ToponymSearchResult searchResult = WebService.search(searchCriteria);
for (Toponym toponym : searchResult.getToponyms()) {
   System.out.println(toponym.getName()+" "+ toponym.getCountryName());
}

Convenient as these are, because they fit the semantics of the service closely, they are typically developed by the individual service providers or developer communities. Therefore, there is no consistency between implementations.

Cloud Connectors

Mule Cloud Connect offers a more maintainable way to work with APIs. Built on top of the Mule and CloudHub integration platforms, Cloud Connectors are service-specific clients that abstract away the complexities of transports and protocols. Many complex but common processes such as authorization and session management work without you having to write a single line of code. Although service-specific, Cloud Connectors all share a common and consistent interface to configure typical API tasks such as OAuth, WebHooks, and connection management. They remove the pain from working with multiple, individual client libraries. Example 1-4 shows a really basic example of configuring a Cloud Connector to access the GeoNames API, which will be covered in more detail shortly.

Example 1-4. Cloud Connector configuration

<geonames:config username="demo" />

<geonames:find-nearby-pois-osm latitude="37.451"
            longitude="-127" />

Cloud Connectors are essentially plain old Java objects (POJOs) developed by Mule and the community using the Cloud Connect SDK called the DevKit. The DevKit is the successor to the original Cloud Connect SDK, which was developed with just external APIs in mind but has since been opened up to create any manner of Mule extension such as transformers or pretty much anything. The DevKit uses annotations that mimic typical integration tasks to simplify development, and when processed, are converted into fully featured components for the Mule ESB and CloudHub integration platforms.

Mule Cloud Connect supports many of the most widely-used open APIs from SaaS to social media, with more being developed every day. Current Connectors include Twitter, Facebook, LinkedIn, Salesforce, Amazon WebServices, Twillio, and many more. A full categorized list of available connectors and what they offer can be found here.

Mule: A Primer

Before diving straight into configuring Cloud Connectors, it’s important to understand some basic concepts. After this short overview, you’ll be ready to build your first application and start taking advantage of Mule Cloud Connectors. To begin, we will first build a simple Mule application that we can use as the base of our examples and introduce some core concepts for those unfamiliar with Mule.

As mentioned previously, Mule is an integration platform that allows developers to connect applications together quickly and easily, enabling them to exchange data regardless of the different technologies that the applications use. It is also at the core of CloudHub, an Integration Platform as a Service (IPaaS). CloudHub allows you to integrate cross-cloud services, create new APIs on top of existing data sources, and integrate on-premise applications with cloud services.

Later in the book we will look at specific connectors, but to start let’s take a look at a simple API proxy that can be used to mediate an external service and introduce some transformation and some routing between the two. This application will expose a simple RESTful interface that can be invoked through a browser or HTTP client, contact an external service, and transform the returned response to the browser.

Mule Configuration

XML is the format for the files that control Mule, and it uses schemas and namespaces to provide a dynamic schema language (DSL) authoring environment. Example 1-5 shows the finished application.

Example 1-5. Simple Mule API proxy application

<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:spring="http://www.springframework.org/schema/beans"
    xmlns:http="http://www.mulesoft.org/schema/mule/http"
    xsi:schemaLocation="
        http://www.mulesoft.org/schema/mule/core
        http://www.mulesoft.org/schema/mule/core/current/mule.xsd
        http://www.mulesoft.org/schema/mule/http
        http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">

    <flow name="main">
        <http:inbound-endpoint host="localhost" port="8080" path="geonamesproxy" 
            exchange-pattern="request-response" />

        <!--
            TODO add your service component here. This can also be a Spring bean
            using <spring-object bean="name"/>
        -->
        
        <echo-component />

        <http:outbound-endpoint
           address="http://api.geonames.org/findNearbyPOIsOSM?lat=37.451
           &amp;lng=-122.18&amp;username=demo" method="GET" />
    </flow>
</mule>

Inspecting this configuration, we can see that it is an XML document with a root element of mule. This element is the key element and must always be included. It is this element that contains references to specific Mule modules, via schema and namespace declarations, to provide the DSL authoring environment. The most important of these is the core namespace, xmlns="http://www.mulesoft.org/schema/mule/core", which allows you to use all the Mule core components such as flows, routers, transformers, and filters. The core namespace is then followed by subsequent namespace declarations that represent individual Mule modules, such as the HTTP module represented by xmlns:http="http://www.mulesoft.org/schema/mule/http" and the Spring module represented by xmlns:spring="http://www.springframework.org/schema/beans".

Flows

Within the mule root element is a critical child element: flow. Flows are underlying configurations for your Mule or CloudHub integration and are the default constructs for orchestrating message processing. Each flow has a name attribute, which must be a unique identifier within your configuration. The flow then consists of a message source followed by a sequence of message processors. Flows are executed from top to bottom, just like any imperative programming language. Example 1-6 shows the flow we have created with the unique ID: main.

Example 1-6. A Mule flow

<flow name="main">
    <http:inbound-endpoint host="localhost" port="8080"
        path="geonamesproxy" exchange-pattern="request-response" />

    <!--
        TODO add your service component here. This can also be a Spring bean
        using <spring-object bean="name"/>
    -->
    
    <echo-component />

    <http:outbound-endpoint
        address="http://api.geonames.org/findNearbyPOIsOSM?lat=37.451
        &amp;lng=-122.18&amp;username=demo" method="GET" />
</flow>

Message Sources

A message source appears at the beginning of a flow. It receives or generates messages and forwards them on to a set of message processors to start working with the message. The message source is typically an inbound endpoint, such as HTTP or JMS, which can listen or poll on a certain address. The flow in the previous example has an HTTP message source for listening on a specific HTTP port, as shown in Example 1-7.

Example 1-7. HTTP message source

<http:inbound-endpoint host="localhost" port="8080"
    path="geonamesproxy" exchange-pattern="request-response" />

In this case, we have added a host attribute with the value localhost, a port attribute with the value 8080, and a path attribute with the value geonamesproxy. This flow, when run, will create a web server that will listen on http://localhost:8080/geonamesproxy.

Message Processors

With the message source in place, we now need some message processors to actually do something with the received message. A message processor is used by Mule to process any messages received by a message source. Each processor can be a transformer, a Java component, or an outbound endpoint to forward on the message to an external system or to another flow.

In this case, we want to forward the message on to the GeoNames API. The GeoNames API is a simple HTTP RESTful API, so we can create an HTTP outbound endpoint similar to that of our message source to forward on the message:

<http:outbound-endpoint
        address="http://api.geonames.org/findNearbyPOIsOSM?lat=37.451
        &amp;lng=-122.18&amp;username=demo" method="GET" />

As you can see, this is very similar to the message source, with the most noticeable difference being that we have changed the element name from -inbound-endpoint to -outbound-endpoint. In this element we have then specified an address attribute with the value of one of the GeoNames APIs and some hard-coded query parameters:

http://api.geonames.org/findNearbyPOIsOSM?lat=37.451&amp;lng=-122.18&amp;username=demo

The GeoNames API also requires the GET HTTP method, so we have included the method attribute on the endpoint and set its value to GET.

Variables and Expressions

To support the work of message processors, Mule provides the Mule Expression Language (MEL) to access, manipulate, and consume information from the message and its environment. Mule makes this data available via the following four contexts:

Server: The operating system on which the message processor is running
Mule: The Mule instance on which the application is running
Application: The user application within which the current flow is deployed
Message: The package (payload, attachments, properties) that the message processor is processing

These contexts are at the heart of most MEL expressions. A typical MEL expression combines one of these contexts with one or more operands and zero or more operators in a Java-like syntax and returns the resulting value. For example, to access the payload of the message, we can use the expression #[message.payload], where message represents the message context and payload represents the payload property within the specified context. The syntax consists of a preceding #[ followed by the expression to execute and a terminating ] character.

In most cases, MEL expressions work within message processors to modify the way those processors do their main jobs, such as routing and filtering based on the message content. The following sections will focus on using the message context and cover some of the main use-cases that will be used throughout the book.

Message properties

Aside from the payload of the message, which is typically the main body of a message, message processors such as inbound and outbound endpoints add additional headers to a message called message properties. Message properties are defined within the following two scopes:

Inbound properties: Inbound properties are placed on a message receiving a request on an inbound endpoint or a response from an outbound endpoint. For example, if a message to an inbound endpoint is called via HTTP with a Content-Type header, this property will be placed as a property within the inbound scope.
Outbound properties: Outbound properties are set on a message to be sent via an outbound endpoint. For example, if a message with an outbound property Content-Type is sent via HTTP, the Content-Type property will be placed as an HTTP header on the outbound message.

MEL expressions allow you to refer to these message properties via a java.util.Map interface. For each property scope, Mule associates a map containing each property with the current message. You can refer to these maps using the following syntax:

#[message.inboundProperties['someProperty']]
#[message.outboundProperties['someProperty']]

where inboundProperties and outboundProperties are the maps within the message context and someProperty is they key of the property you want to retrieve from the map. Example 1-8 amends our GeoNames example to extract the latitude query parameter from the incoming request to use as an input to the original GeoNames request URL.

Example 1-8. Using message properties

<flow name="main">
    <http:inbound-endpoint host="localhost" port="8080"
    path="geonamesproxy" exchange-pattern="request-response" />

    <!--
        TODO add your service component here. This can also be a Spring bean
        using <spring-object bean="name"/>
    -->
    
    <echo-component />

    <http:outbound-endpoint
        address="http://api.geonames.org/findNearbyPOIsOSM
        ?lat=#[message.inboundProperties['latitude']]
        &amp;lng=-122.18&amp;username=demo" method="GET" />
</flow>

With the amended configuration in place, if you execute the flow with your browser using the URL http://localhost:8080/geonamesproxy?latitude=37.451, Mule will now propagate the latitude parameter to the lat argument in the GeoNames request URL.

Additional variables

Typically, message properties should be reserved for the Mule message for things such as HTTP headers or JMS headers. To store additional information during the execution of a flow, like variables in Java, Mule provides two more types of scoped variables:

Flow variables: Flow variables are global to the current flow. They retain their values as control passes from one message processor to another. Thus, you can set them in one message processor and use them in another.
Session variables: Session variables are essentially the same as flow variables, but in addition, when one flow calls another one via a Mule endpoint, they are propagated and are available in the subsequent flow.

As with message properties, flow and session variables are available via a java.util.Map interface. This map data can be referenced using the following syntax:

#[flowVars['someProperty']]
#[sessionVars['someProperty']]

Storing variable data

In order to store variable data, Mule provides a set of message processors to simplify working with each property or variable scope.

Setting properties

To set a message property, Mule provides the set-property message processor. This message property works only with outbound scoped properties as the inbound scoped properties are immutable. The following example shows how to set the Content-Type property on a message using this message processor:

<set-property propertyName="Content-Type" value="text/plain"/>

This message processor takes two mandatory arguments: propertyName and value. propertyName is the name of the property to set and value is the value of the property. Either of these arguments’ values can also be expressions themselves. For example, to copy the Content-Type property from the inbound scope to the outbound scope, you could use the following example:

<set-property propertyName="Content-Type"
          value="#[message.inboundProperties['Content-Type']]"/>

Setting variables

As with properties, similar message processors are available for both flow and session variables. set-variable sets a flow variable and set-session-variable sets a session variable. The syntax for these message processors are very similar as the previous set-property message processor, with variableName being the name of the variable to set and value being the value of the variable. The following example demonstrates setting both flow and session variables:

<set-variable variableName="myFlowVariable" value="some data"/>

<set-session-variable variableName="mySessionVariable" value="some data"/>

Enrichment

Another way of setting message properties or variables is via enrichment. Mule provides an enricher element to enrich the current message with extra information. It allows you to call out to another resource and set extra information on the message without overriding the current payload of the message. For example, you can call out to another endpoint or message processor and store its return value in a message property or variable. The following example demonstrates this effect, using the enricher to call the GeoNames service and store the response in a message property:

<flow name="main">
    <http:inbound-endpoint host="localhost" port="8080"
        path="geonamesproxy" exchange-pattern="request-response" />


    <enricher target="#[message.outboundProperties['response']]">
        <http:outbound-endpoint
            address="http://api.geonames.org/findNearbyPOIsOSM
            ?lat=#[message.inboundProperties['latitude']]
            &amp;lng=-122.18&amp;username=demo" method="GET" />
    </enricher>
</flow>

The target attribute defines how the current message is enriched by using expressions to define where the value is stored on the message. Here we are using standard MEL syntax to refer to an outbound property using #[message.outboundProperties['response']. This will add or overwrite the specified message property with the result of the outbound endpoint. The main difference between using the enricher and the set-property message processor is that the enricher supports setting the value of the property via a nested message processor such as an outbound endpoint, whereas the set-property and other associated message processors only support setting the value’s value attribute. This just demonstrates the broad strokes of the procedure. More information on enrichment can be found here.

Functions

In addition to getting or setting information within a specific context, Mule also provides an expression syntax for executing certain functions. Functions provide a way of extracting information that doesn’t already exist as a single value within a particular context. For example, if you have an XML document and care only about a particular node or value within that document, you can use the xpath function to extract that particular value. Or if you want extract a specific part of a string, you can use the regex function, and so on.

Note

Xpath is a closely related sister specification of the XML document specification and provides a declarative query language for addressing parts of an XML document.

Our current configuration will return an XML-formatted document representing the GeoNames response. Example 1-9 demonstrates using a simple xpath expression to log the name of the root element.

Example 1-9. Using functions

<flow name="main">
    <http:inbound-endpoint host="localhost" port="8080"
       path="geonamesproxy" exchange-pattern="request-response" />

    <!--
        TODO add your service component here. This can also be a Spring bean
        using <spring-object bean="name"/>
    -->
    
    <echo-component />

    <http:outbound-endpoint
        address="http://api.geonames.org/findNearbyPOIsOSM
        ?lat=#[message.inboundProperties['latitude']]
        &amp;lng=-122.18&amp;username=demo" method="GET" />

    <logger level="INFO" message="#[xpath('local-name(/*)')]" />
</flow>

Routing

Mule has always had support for many routing options. Routers in Mule implement the Enterprise Integration Patterns (EIP). They are message processors that determine how messages are directed within a flow. Some of the most common routers are:

all: Sends the message to each endpoint
choice: Sends the message to the first endpoint that matches
recipient-list: Sends the message to all endpoints in the expression evaluated with the given evaluator
round-robin: Each message received by the router is sent to alternating endpoints.
wire-tap: Sends a copy of the message to the supplied endpoint, then passes the original message to the next processor in the chain
first-successful: Sends the message to the first endpoint that doesn’t throw an exception
splitter: Splits the current message into parts using a MEL expression, or just splits elements of a list
aggregator: Combines related messages into a message collection

Alongside MEL, routers can decide on a course of action based on the contents, properties, or context of a message. Example 1-10 demonstrates using the choice router. It builds upon Example 1-8 to call the GeoNames API only if the latitude property is sent in the request.

Example 1-10. Choice router with expressions

<flow name="main">
    <http:inbound-endpoint host="localhost" port="8080"
    path="geonamesproxy" exchange-pattern="request-response" />

    <!--
        TODO add your service component here. This can also be a Spring bean
        using <spring-object bean="name"/>
    -->
    
    <echo-component />

    <choice>
        <when expression="#[message.inboundProperties['latitude']] != null]">
            <http:outbound-endpoint
                address="http://api.geonames.org/findNearbyPOIsOSM
                ?lat=#[message.inboundProperties['latitude']]
                &amp;lng=-122.18&amp;username=demo" method="GET" />
        </when>
    </choice>
</flow>

Summary

This chapter has offered a primer on Mule. You have been introduced to some its core features, started your first working Mule application, and connected your first API. But you have merely scratched the surface of Mule, and there are many more features for you to explore. But you’re now ready to delve into Mule Cloud Connect.

Get Getting Started with Mule Cloud Connect now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Getting Started with Mule Cloud Connect by Ryan Carter