A spark for discussion
Callbacks are the asynchronous analogs of the on-demand call. It is as simple and as complex as that. They are used when those synchronous capabilities to retrieve or process data cannot adhere to the requirements of responsiveness and performance. Structurally they are a sequential composition of two basic EDA patterns, e.g. broadcast and point-to-point or two point-to-point, etc.
Callbacks are one of those patterns that might be a bit troublesome to form a philosophical point of view. They are fairly straightforward technically speaking, but it is debatable whether or not they fall under the category of event-driven. The key question to explore here is that if we are sending a message and we are expecting a response, perhaps not an instantaneous one, rather over time or in an undetermined time, is that still event-driven? If we consider the callback alone, without the triggering subscription or event, then it seems to always be an event of sorts, marking the completion of a processing task, an occurrence in a system, etc. Let us consider this when exploring this pattern.
Pattern nameplate
Name: Callback
Communication mode: Asynchronous
Architectural style: Event-Driven Architecture
Common use cases:
- Data quality assurance based on API contracts,
- Real-time subscribed data delivery as a result of an earlier data subscription,
- Confirmation of event processing,
- Decoupling processing with long or non deterministic processing time,
Architectural coupling:
- Contract coupling - the provider and consumer of the event are locked by an agreed data model, it is wanted within the bounds of a p2p communication, might not be, if it is extended into a broadcast or multicast pattern with additional consumers that only use a subset of the data model. Furthermore, there are at least two contract couplings present in this pattern, depending on the variant (e.g. if dedicated callback queues are used)
- Data type and format coupling - the provider and consumer must have the same understanding of the data model types and format (e.g. JSON, XML, CSV)
- Conversation coupling - depending on the broker implementation, the consumer and provider may be locked by the protocol of the event broker
Operational coupling:
- Semantic coupling - unavoidable with any data exchange
- There are two or more distinct architectural quanta, one for each event producer, one for each event consumer, both overlapping on the event broker structure used (topic or queue)
- Temporal coupling - for a callback to occur successfully, there must be first a successful event/command/message that results in said callback
Diagrams
Pattern diagrams
Simple callback
Callback to a broadcast with a single contract queue
Callback to a broadcast with dedicated queues
Behavioral diagram
Pattern analysis
Callback patterns at times seem like not the most common patterns in interoperability on the ecosystem level, and sometimes their limited or common use can be considered industry specific. In either case, they may prove extremely useful, both from the real-time data delivery and real-time operations points of view. They are the asynchronous equivalent of an on-demand call in some use cases, but can also serve several operational functions, e.g. delivery confirmation, statuses. It is also important to note that while using synchronous communication it is implied that there will be a response, issuing a callback is optional and left to the discretion of the callback producer and based on the business process.
Simple callback
The simplest form of a callback is an exchange between two systems, where the callback producer first receives a message, state, event or a command from the callback consumer that triggers processing and the creation of a callback event. This is a fairly easy way to decouple two systems, by providing a separation in the form of a message broker between them.
Architectural considerations
While any messaging structure can be used to implement this pattern, let’s consider implementing it over queues only. This will be a dedicated flow, tailored to the specific exchange need. It has limited extensibility as it may be turned into a service fronted with queues as its API implementation, changing the topology from one-to-one-to-one to many-to-one-to-one. This means that there may be several callback consumers that produced a trigger for the producer to create a callback. In that case additional configuration needs to happen over the callback queue to enable proper filtering based on metadata (e.g. message headers, correlators, topic/queue taxonomy) so that the right consumer receives the right response. It is similar to a one-way point-to-point over a queue, because this is this exact pattern used twice in sequence.
Operational Considerations
Utilizing this pattern will provide easy scalability if needed for each party consuming messages from them, especially if there might be multiple threads producing callbacks to various triggers coming in from the callback consumer. This pattern, while may have varied performance due to the communication characteristics, processing time of the callback producer, which involves domain complexity, it is useful for supporting responsiveness of the callback consumer as the process is not confined to a single thread that becomes blocked awaiting the callback.
Callback to a broadcast with a single contract queue
A different callback pattern, following one-to-many-to-one topology, that has more of an operational use, chaining a broadcast/multicast and an one-way point-to-point patterns in sequence. One of the key use cases for this pattern is mitigating the operational lack of delivery or processing confirmation, which makes error handling and IT operations a bit more difficult. This might at times be crucial for specific business areas, e.g. product configuration, where the business needs to be sure that all sales channels and order processing have the same product definitions (e.g. bundles, promotions). In that case a lack of callback to the data master confirming successful processing or a callback signalling processing errors will give more operational agility. This unfortunately comes at a trade-off that the data master must be aware of all consumers that need to send an operational callback. These callback messages could be considered to be events, but might as well be state.
Architectural considerations
Given the broadcast nature of a callback trigger, this pattern will support extensibility. With a callback originating from multiple producers to a single queue the key consideration is that all of those producers are bound to a single contract managed by the callback consumer. This needs to be considered when designing, as this contract coupling may be a problem with high volatility of requirements, as any change to the scope of data will impact all of the communication participants.
Operational Considerations
From an operational perspective this kind of callback using the queue topology of many-to-one might be troublesome to monitor, as there are many applications writing to a single queue. That would require more effort from the operations teams in terms of monitoring the message broker for which producers actually wrote to the queue.
Callback to a broadcast with dedicated queues
Callback to a broadcast with dedicated queues is probably the rarest of the callback patterns. Not only does it require the most effort to set up, but also facilitates more refined and deliberate processes. This key use case is that all the callback producers react to an event published and that triggers their processing resulting with a callback that is specific to what the functionality of each of those event processors is. If each processor delivers different data as a callback to a trigger in this scenario, this means that each one will have a different schema for said data, to properly separate objects and avoid contract brittleness. This also means that ownership of the schema and each callback queue lies within the scope of the respective callback producer as they are governing the data transferred.
Architectural considerations
First thing that comes to mind is that this setup is complex, and while it may seem simple for each callback producer, it is a lot more complicated for the consumer of all those callbacks as now that application needs to implement processing logic to consume multiple endpoints as well as various contracts. Additionally if we’d consider replacing the dedicated queue with a topic or topic bridged to a queue (changing the topology from one-to-many-to-one, to one-to-many-to-many), we can create extensibility and reusability of the callback produced. This later can be used in additional contexts for whichever application this data might bear relevance. This in turn contributes to even better real-time communication within the ecosystem.
Operational Considerations
Turning again to the operational perspective, this variation of the callback pattern will be easier to maintain and operate as there is a clear separation for each callback producer, so it will be easier to spot if for some reason there is a communication breakdown. Otherwise, operationally they do not differ from one-way point-to-point or broadcast/multicast patterns, which are the basis for this communication.
Conclusions
Callback is a useful pattern that enables the organization and the data transfers between applications to be more agile and run in real-time, with the business applications acting on events as they happen rather than over time polled with a delay. While it may present some additional complexity and operational challenges, the benefit outweighs the cost. Additionally, here we can see that it is not straightforward whether all messages we send are actual events. It is worth considering what is the process as a whole as in some specific cases we will be dealing with commands and states that will be triggering the event-driven behaviour.