The Data Pipeline

For all the complexity we attribute to computers, they can really only do three things:

  • Read data from somewhere (e.g., a device, a file, a relational database, or a network socket)

  • Transform data (e.g., add 2 to the variable n, XOR all the bits in a buffer, or append the string “world” to the end of the string “hello”)

  • Write data to somewhere (e.g., a device, a file, or a computer screen)

You can view data processing as a pipeline. Data comes into the pipeline, passes through various filters that change it in various ways (some simple, others complex) until the data spills out the other end.

This model best describes the Ajax design pattern presented here. The pipeline includes databases, web services, proxies, data filters such as XSLT, and it all eventually spills out onto the user’s computer screen.

It seems simple because some details have been left out. Ajax applications are client-server applications, but the decision as to which tasks happen in the browser and which tasks happen on the server is not set in concrete.

Figure 1 illustrates the data pipeline used by one Ajax application. A web service extracts data from a relational database (e.g., Oracle or mySQL), transforming that data into XML. A proxy forwards the XML to the browser. There, a JavaScript library uses an XSL stylesheet to translate the XML into HTML. That HTML is then inserted into the web page using DHTML.

Figure 1. A data pipeline model for one Ajax application

Get Ajax and Web Services now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.