As we dive deeper into Event-Driven Architecture (EDA), there is a specific scenario that deserves attention on its own. As we already established that EDA patterns can be used to loosen the coupling between different applications and systems so they can operate in a more independent manner. The waiter pattern builds on top of that capability allowing the applications that produce data over a long span of time to deliver them in an organized and standardized way.
Pattern nameplate
Name: Waiter pattern
Communication mode: Asynchronous
Architectural style: Event-Driven Architecture
Common use cases:
- Real-time requested data delivery in chunks as they are produced as a result of an earlier command or event,
- Delivery of larger amount of data in parts,
- Parallel complex processing chunked based on a single event/command.
Architectural coupling:
- Contract coupling - the provider and consumer of the event are locked by an agreed data model, it is tolerable within the bounds of a p2p communication, might not be, if it is extended into a broadcast or multicast pattern with additional consumers that only use a subset of the data model. Furthermore, there are at least two contract couplings present in this pattern, depending on the number of callback producers
- Data type and format coupling - the provider and consumer must have the same understanding of the data model types and format (e.g. JSON, XML, CSV)
- Conversation coupling - depending on the broker implementation, the consumer and provider may be locked by the protocol of the event broker
Operational coupling:
- Semantic coupling - unavoidable with any data exchange
- There are two or more distinct architectural quanta, one for each event producer, one for each event consumer, both overlapping on the event broker structure used (topic or queue)
- Temporal coupling - for a subscription callback to occur successfully, there must be first a successful event/command/message that results in said callback
Diagram(s)
Behavioral Diagrams
Waiter pattern behaviour
Pattern analysis
The core function of the waiter pattern is to provide value with produced data as soon as possible. Depending on the business scenario, and in this case quite a specific one, this data will be delivered at least once, meaning that this pattern behaves like a callback in its simplest form. The real value of it is in the more advanced scenario, where the callback producer’s processing time is very long - minutes, up to hours, but the callback data can be partitioned into usable chunks and sent as separate callbacks. This might seem like a subscription to events like webhooks, but there is one key difference - the processing and publication of callbacks is temporally coupled to the requesting party. Without the initial trigger message (command or event), no data will be produced.
An example of such a mechanism would be a callback function for a web crawler, which is charged with crawling through news sites in search of specific keywords, like news about companies. Usually these news are copied from site to site with very little addition, so the job of such crawlers is to identify a theme of an article and aggregate all news on the same theme, then deliver a summary and sources. If you span that across a time frame of for example two years, that is a lot of data to cover and it will take a lot of time. Packaging each theme into a single deliverable callback brings value faster to the business as their actions are not blocked entirely by waiting for the processing to finish. And sometimes partial results might be sufficient to finish work, such as risk assessment.
Architectural considerations
Given the scope of use cases for the Waiter pattern, essentially all callback patterns are viable, as they largely depend on the number of event processors producing callbacks and whether or not they all produce data in the same data model. In that sense architectural considerations are the same as described for callbacks .
What can be an additional consideration to be had, is that since the number of callbacks will be usually more than one per event triggering the processing, this requires mindful handling of persistence and errors, especially when running parallel jobs. As there might be a connectivity problem (as per the first fallacy of distributed computing), the number of messages pending being sent from the callback producer or delivered and temporarily stored on the message broker, will gradually increase. This in turn might pose a risk of overwhelming the message broker and disabling the event based communication for all involved, making it important to be managed on an architectural level first.
Operational Considerations
Turning to operational aspects of the Waiter pattern, other than the aforementioned inheritance from callbacks, communication metadata might play a more important role. Correlation identifiers may be especially useful, not only to track and correlate log data between various instances of callbacks from a single processing job, but also to provide robust statistics and reporting on services and data usage. This is especially crucial if the callbacks are turned into broadcasts enabling other consumers, than the original requestor, to subscribe to that data, as this may provide an easy way to track all consumption.
Conclusions
Waiter pattern as a specific case of a callback pattern is a very useful tool in an integration architects toolbelt as it provides additional capabilities to balance long processing times clashing with low time to market, that seems to be a very common business driver of the information age. It enhances the agility of data consumers to provide value to the end user in a timely manner.
However it is important to consider the error handling, persistence and operational aspects of this pattern, especially in high volume scenarios, as, if not addressed, they may lead to critical failures, severely impacting communication. This needs to be defined and implemented in cohesion with the business process, as these things will be case specific. Luckily if managed properly, this pattern can provide significant value to any organization.