Something old, something new
As we take deeper dives into ecosystem architectural styles, we can see that we already solved some problems using Event-Driven Architecture and Broker Architecture, but more challenges come with scale. As we keep introducing new complexity and dependencies with new business systems, those architectures start to become over cumbersome, with a large number of repetitions. Now we are facing a challenge that could be named "Dedicated vs Composable integration flows” which would be an architectural equivalent of the “DRY code vs WET code” in development.
Let’s apply some architectural thinking and analyze the trade-offs for each option:
Dedicated integration flow:
In dedicated integrations, the level of reusability is fairly limited, mostly to reusable libraries, specific snippets of code. The integration flows themselves are closely knitted to the business needs and are a reflection of the underlying business process. This means that quite often access logic to specific systems is duplicated, with small variations to the query parameters or scope of data. This brings a benefit of less abstract code and being able to precisely fit to the business requirements. This comes at the cost of having a bigger codebase to maintain and operate, which can lead to development bottlenecks as the focus will shift from development to operations and maintenance.
Composable integration flow:
In composable integrations, the integration flow is split into microservices, facilitating a specific part of the communication. In that case we’re not only reusing code libraries, but whole integration services. This means that while the logic does not get duplicated as much, it usually has to be designed either in a generic way, to cover a broader spectrum of possible needs or on an iterative basis to extend the services as needed to reach that broad spectrum. While this comes at the benefit of having a smaller code base, that is easier to scale and manage as separate services, the cost is that it becomes a bit more abstract and harder to understand. This requires proper governance and additional tools to govern all available services and make them discoverable and searchable, otherwise the duplication will not happen at code level, but rather on the level of whole services.
Ecosystem Architecture Styles Scope
How does that relate to Ecosystem Architectural Styles?? If we take a look at EDA and Broker Architecture, there is a large number of repetitions within those architectural styles. With EDA it is due to the effort on communication being distributed among various producers and consumers and encapsulated within those systems. With integration brokers, the balance shifts a little bit closer to removing duplication by providing a finite level of abstraction. This is due to the workflow-like nature of integration flows, where, while there is reusability on the code level, it makes the flows a bit repetitive as they might be required to be implemented over and over again for different systems with some variance. Both of those architectural styles can be placed on a spectrum between dedicated and composable approaches, yet closer to dedicated.
Something borrowed
Going into the direction of the composable approach we can find API-Led Architecture, which is an architectural style that is a descendant from Service-Oriented Architecture’s (SOA) Enterprise Service Bus (ESB). While there are many similarities, the core concept moved so far away from the original that it deserved a name of its own. This architectural approach provides a three-layered separation of concerns within an integration platform that was present in the industry at the beginning of 2010s. Mulesoft refined the concept further and coined the term “API-Led Connectivity” around 2016. A large part of what API-Led Architecture is, was refined and coined as a standard by that technology vendor, despite the fact that you can implement this architectural style in a number of different technologies.
What is API-Led Architecture?
API-Led Architecture is an ecosystem architectural style that builds atop the previous architectural styles expanding their capabilities and adding a more standardized and reusable approach to interoperability. The key aspects are:
- Separation of concerns - the communication between business systems is separated by three layers of integration applications, each with different responsibilities.
- Separation of access - each business system has a dedicated set of integration applications,
- adapter for traffic towards a system - exposing standardized data services (CRUD-based) to the rest of the ecosystem,
- channel for outbound traffic - exposing dedicated services to systems,
Otherwise this architectural style reinforces the integration platform aspects inherited from Broker Architecture:
- Protocol agnostic approach to communication,
- Mediating between various contract data models and formats,
- Provide orchestration logic,
- Provide observability, abstraction and extensibility capabilities,
Three Abstraction layers
One of the most important aspects of API-Led Architecture is the capability to provide abstraction layers separating the business systems from each other. Those layers have distinct responsibilities that they deal with as they abstract different aspects of the communication:
- Channel Layer - an abstraction layer composed of integration apps and their APIs exposed directly to the consumers of the integration platform, with each one of them represented by its own channel application is responsible for:
- Exposing to the consumer dedicated services that they need,
- Translating from the integration contract to the internal platform service contract,
- Limiting the service response if the payload scope of data is too extensive, narrowing it down only to the needed set,
- Provide additional communication metadata, that could be based on the payload, to enable proper message routing in case of publishing events,
- Caching responses to frequent consumer requests (if applicable),
- Creating a secure dedicated channel, that will only be used by a single business system,
- Providing observability metadata on service usage of the consumer system,
- Abstract the service or data service contract to mitigate the impact of some service changes on the customer,
- Composition Layer - an abstraction layer composed of integration apps and their standardized APIs responsible for composing complex services usually grouped around a specific set of business processes or capabilities (e.g. customer, or order related),
- Exposing standardized reusable services to the Channel layer,
- Encapsulating the complexity of service and data orchestration between various data services,
- Caching responses to frequent service requests for all consumers (if applicable),
- Providing observability metadata on the integration platform service usage,
- Adapter Layer - an abstraction layer composed of integration apps and their standardized APIs exposed for the internal use of integration platform. Applications within this layer hold data services of all provider systems, with each one of them represented by its own adapter application, which is responsible for:
- Exposing a standardized reusable data services to the Channel and Composition layers,
- Encapsulating the complexity of data access logic,
- Translating between the integration platform internal service contract and the provider system contract model for the messages passing between the integration platform and the system,
- Limiting event-based communication payload scope if the data is too extensive, narrowing it down only to the needed set,
- Caching responses to frequent data service requests for all consumers (if applicable),
- Creating a secure dedicated adapter to funnel all traffic to the system,
- Providing observability metadata on data usage of the provider system,
Qualitative Analysis
As we did with previous architectural styles, let’s explore the qualities of API-Led Architecture and look into a few pitfalls that can be crucial to consider when trying to apply this architectural style. For that we will be using a comparison table that was produced through a qualitative analysis of architectural styles taking several architectural characteristics into account. If you would like to learn more about this analysis or read how we define those characteristics, you can do so by reading this article, where we explain how this comparison was created.
API-Led Architecture is always a significant investment. At least this is the case in the first few years, when this new concept of a mediator will still be debated, challenged and its governance will be in a state of flux till it stabilizes. The development cost ( ) at first encompasses the creation of all of the base infrastructure for each business system, that is: their respective integration apps, as well as those for processing, figuring out the logic on how to expose and standardize data, etc.. Over time this cost lowers as a lot of data services and composed services are already there, which means they can be leveraged to compose new services and expose them as needed to new business applications, gradually lowering the development effort, time to market and cost. The cost score here is an average of the initial cost and the later lowered value.
Operational cost ( ) in API-Led architecture is usually high. This is due to several factors:
- Licensing cost and model - technologies tailored to facilitate this architectural style are expensive and have various licensing models that monetize their use differently on premise and in cloud,
- Number of components - the number of deployable components and their size may be large in the later stages of the platform’s life, so that will be a factor increasing the cost with growing ecosystem complexity,
- Standardized logging, monitoring and analytics - API-Led based integration platforms can produce a significant amount of logs that need to be processed, stored and made searchable, this means additional costs, especially in cloud environments, where data transfer is the costly part,
The distributed nature of the platform means that, while it is a lot more readable, without support from proper tooling it will likely take a lot of effort and knowledge to perform a proper Root Cause Analysis (RCA) for any incidents, so the initial operational investment into the right tools is crucial.
The real value of API-Led Architecture lies in the architectural change cost ( ) as it is one of the super powers of this architectural style. As a result of architecting for the right level of abstractions, whole systems can be replaced with limited impact on the ecosystem, as most of it will be encapsulated within the integration platform and never propagated to the business systems, unless there is a reason for that.
Architectural and design time analysis
From the perspective of the qualitative analysis it may seem like API-Led Architecture is always the best choice for an ecosystem architectural style. So before we jump into the great qualities of it, let’s discuss the less beneficial scores. Starting off with simplicity ( ), which is scored similarly to Broker Architecture, but for a completely different reason. If we look at integration brokers, this is a collection of easy workflow-like integration flows, usually highly custom and tightly knit with the business process, which gives it that particular score. With API-Led Architecture, we have a single integration flow distributed across several integration apps starting at at least two. The complexity of this style seems a lot higher by that alone. Fortunately due to the way abstraction layers are organized and the fact that integration applications are assigned in every layer, the overall architecture is fairly more readable, than a collection of integration flows built in a broker. Because of that the score cannot really be lower, and at the same time the sheer scale of this kind of platform does not allow for it to be higher.
Moving on to qualities that were scored higher in the qualitative analysis, let’s start with the capability to provide abstraction ( ), as it is the key benefit of this platform that to an extent sets the score higher for other architectural characteristics as well. As mentioned above API-Led Architecture has three distinct abstraction layers, which enables encapsulation of logic and a clear division of responsibility within the code. This highly impacts contract resilience ( ) and composability ( ), as any impactful changes (e.g. a change of a contract with the system) can be contained as close to that system as possible, limiting the bubbling upstream and forcing changes on other business systems. With this encapsulation of data access logic and service orchestration, API-Led Architecture enables architects to mix and match, to create new services to facilitate business needs. Which fosters a reusable mindset. Lastly, the encapsulation of logic in abstraction layers enables a certain ease in adding new consumers to the platform, boosting its extensibility ( ), making each future change somewhat easier.
Operational analysis
Moving on to the operational qualities of this architectural style, we can see that the scores are a little bit more polarized. Looking at the superpowers of API-Led, we can see that it is a highly scalable approach with scalability scoring at . This is another aspect deriving from the separation of concerns and the abstraction layers, where due to the microservice nature of the integration platform every single element is separately scalable. This enables a great degree of fine tuning flexibility, giving a lot of control over resource consumption, especially with high reuse of all services. However the fact that we have so many layers and applications composing the integration platform negatively impacts testability ( ) and performance ( ). There simply are too many moving parts, which bars us from performing tests really fast, especially if end-to-end testing reveals some bugs - analysis may take some time. Performance wise, the chaining of microservices introduces more latency than it was with the Broker Architecture, making this architectural style slightly slower than the competition.
Looking at a different aspect of observability ( ), the abstraction layers and dedicated integration apps may provide a significant amount of data and metadata about the health and operations of the whole ecosystem, often leading to situations that the integration platform operations may spot problems in connected business systems, before their operations teams see them themselves. This level of granularity and fine details, combined with proper metadata, supports auditability ( ) in a fantastic way, enabling gathering of significant data that can be later used for audit trails, tracing or simply RCA.
Lastly, as a result of the abstraction capabilities and integration apps granularity, the security ( ) aspects of the whole ecosystem gain a great benefit, as not only metadata of all communication is available for anomaly analysis, there is more room for swift reactions as well. If a system is compromised for some reason, the abstraction layers halt the progress of the intrusion to a degree. If needed, all communications with that breached application can be cut by disabling the whole integration applications responsible for facilitating communication or if the danger is lesser only specific operations tied to the compromised parts.
Conclusions
The evolution of integration architectures mirrors the growing complexity of modern IT ecosystems. API-Led Architecture emerges as a pivotal advancement, addressing the shortcomings of earlier approaches like Event-Driven Architecture and Broker Architecture. By embracing a three-layered abstraction (Channel, Processing, and Adapter), this style not only enhances efficiency and agility within integration processes but also empowers organizations to gain deeper insights into data flows and system dependencies. This improved understanding can serve as a catalyst for strategic business decisions and drive innovation across the enterprise. While the initial investment in implementing API-Led Architecture is significant, the long-term benefits in terms of scalability, observability, and security, coupled with the potential to reduce architectural change costs, position it as a compelling option for organizations seeking to modernize their integration landscapes.