WE LIVE IN HARD TIMES. THE SOCIAL MARKET ECONOMY IS BEING REPLACED BY A GLOBAL MARKET economy, and the marketing guys rule the world. As a consequence, you have to be fast and flexible to survive. It's a renaissance of Darwinism:
It is not the strongest of the species that survive, nor the most intelligent, but the ones most responsive to change.
The key is flexibility. For all major companies and large distributed systems, information technology (IT) flexibility is paramount. In fact, IT has become a key business value enabler.
At the same time, processes and systems are also becoming more and more complex. We have left the stage where automation was primarily a matter of individual systems, and are fast moving toward a world where all those individual systems will become one distributed system. The challenge is maintainability.
It turns out that the old ways of dealing with the problems of scalability and distribution don't work anymore. We can no longer harmonize or maintain control. Centralization, the precondition for harmonization and control, does not scale, and we have reached its limits. For this reason, we need a new approach—an approach that accepts heterogeneity and leads to decentralization.
In addition, we have to solve the problem of the business/IT gap. This gap is primarily one of semantics—business people and IT people appear to speak and think in entirely different languages. The new approach must bring business and IT much closer than ever before.
Service-oriented architecture (SOA) is exactly what's needed. It's an approach that helps systems remain scalable and flexible while growing, and that also helps bridge the business/IT gap. The approach consists of three major elements:
Services, which on the one hand represent self-contained business functionalities that can be part of one or more processes, and on the other hand, can be implemented by any technology on any platform.
A specific infrastructure, called the enterprise service bus (ESB), that allows us to combine these services in an easy and flexible manner.
SOA accepts that the only way to maintain flexibility in large distributed systems is to support heterogeneity, decentralization, and fault tolerance.
Sounds like a dream, doesn't it?
The problem is that you can't just buy SOA; you have to understand it and live it. SOA is a paradigm. SOA is a way of thinking. SOA is a value system for architecture and design.
This book will explain the paradigm and value system of SOA, drawing on real experiences. SOA is often explained with brief statements and prototypes, which leads to a problem illustrated by the infamous "hockey stick function" (see Figure 1-1). Up to a certain level of complexity, the amount of effort required is low, and things look fine. But when this level of complexity is exceeded, the amount of effort required suddenly begins to rise faster than the benefit you gain, and finally, things collapse.
Too often, SOA is only partly explained and installed. Just introducing an infrastructure like Web Services might help up to a certain level of complexity, but this is not enough to guarantee scalability. The whole architecture, dealing with services, infrastructure, policies, and processes, must match. Once you understand how to implement SOA, it's not hard, but it takes time and courage. (OK, so it is hard.) And a lot of effort is required to help people understand (beginning with yourself). If you're not willing to put in the effort, you will fail.
Before we get into the details, I'd like to provide a foundation for the rest of this book by talking about the context and history of SOA. The following sections will present some of the "tales and mystery" of SOA to help you get familiar with SOA.
First, large systems must deal with legacies. You can't introduce SOA by designing everything from scratch. You have to deal with the fact that most of the systems that are in use will remain in use. This also means that establishing SOA is not a project like designing a new system. It involves changing the structure of an existing system, which means you have to deal with old platforms and backward-compatibility issues. In fact, SOA is an approach for the maintenance of large system landscapes.
By nature, all large systems are also heterogeneous. These systems have different purposes, times of implementation, and ages, and you will find that the system landscapes are accretions of different platforms, programming languages, programming paradigms, and even middleware. In the past, there have been many attempts to solve the problems of scalability by harmonization. And, yes, harmonization helps. Withdrawing old platforms or systems that are no longer maintainable is an important improvement. But chances are that your systems will never be fully harmonized. Right before you remove the last piece of heterogeneity, a company merger, or some other change will open Pandora's box again.
One reason for the heterogeneity is that large systems and their data have an incredibly long lifetime. During this lifetime, new functionality that promotes the business is developed by adding new systems and processes. Removing existing systems and data may seem to have no business value, but such changes are investments in the maintainability of your system. Often, these investments come too late, and become incredibly expensive because the systems are out of control, and all the knowledge about them is gone.
By nature, large systems are complex. For this reason, finding out the right places for and determining the effects of modifications can be tough. As [Spanyi03] states:
There is no such thing as a "quick fix . . . ". Organizations are complex business systems, within which a change in any one component is likely to have an impact on other components.
Large distributed systems also have an important additional property: different owners. Different teams, departments, divisions, or companies may maintain the systems, and that means different budgets, schedules, views, and interests must be taken into account. Independent from formal structures, you are usually not in a situation where you have enough power and control to command the overall system design and behavior. Negotiation and collaboration are required, which can be problematic due to political intrigue.
Another key characteristic of large systems is imperfection. Perfectionism is just too expensive. Or, as Winston Churchill once said:
Perfectionism spells P-A-R-A-L-Y-S-I-S.
Working systems usually behave a bit sloppily. They may do 99 percent of their work well, but run into trouble with the remaining 1 percent, which usually results in additional manual effort, the need for problem management, or angry customers. Note that the amount of imperfection differs (vitally important systems usually have a higher ratio of perfection, but even for them, there is always a point at which eliminating a risk is not worth the effort).
Similarly, large systems always have a certain amount of redundancy. While some redundancy might be accidental, there will also be a significant amount of intentional and "managed" redundancy, because in practice, it is just not possible to have all data normalized so that it is stored in only one place. Eliminating redundancy is difficult, and incurs fundamental runtime penalties. In a simple form of redundancy, at least the master of the data is clear (all redundant data is derived from it). In complex scenarios, there are multiple masters, and/or the master is not clearly defined. Maintaining consistency can thus become very complicated in real-world scenarios.
Finally, for large systems, bottlenecks are suicide. That does not mean that they do not exist, but in general, it is a goal to avoid bottlenecks, and to be able to scale. Note that I don't only mean technical bottlenecks. In large systems, bottlenecks also hinder scalability when they are part of a process or the organizational structure.
 Thanks to Gregor Hohpe, who told me about the "hockey stick function" at OOP 2006.