Loosely Coupled - Exploring Publish-Subscribe (EDA)

Good morning, good afternoon, good evening, everybody.

Hope you all have a time of day appropriate beverage.

And I'm very sorry for today's delay.

We were supposed to start at 7 p.m. Central European summertime.

But, due to unforeseen consequences of various weather problems in North America where Aaron is currently residing, and, well, his flight from Chicago to Ottawa was cancelled, so he was, what, delayed, I think, over, definitely over 15 hours in his arrival to Ottawa, so we're a little bit late.

Luckily, Aaron is with us today, and, today, we're diving into a very lovely topic being event-driven architecture, one of my favourites in terms of interoperability, and, with Aaron, we will be diving into publish, subscribe.

For those who don't know, Aaron is a solutions engineer with Solace, a very, very, very good event broker, I think one of the best on the market at the moment.

He used to be a developer advocate for Solace, so a lot of talking with a lot of people, and also, Aaron is running office hours, Solace office hours with, well, with Aaron, obviously, and if you haven't seen those, they're available on the Solace channel on YouTube, and, without further ado, hello!

Hey, everybody, hey, Carol, how's it going, man?

I was quite surprised that you wouldn't make it!

I, it was touch and go there for a while.

The last 24 hours has been, you know, planes, trains, and automobiles kind of excitement, so I'm glad I literally just landed at the airport about half an hour ago, and I just drove up home, and I jumped out of the car, I ran upstairs, and I booted up my laptop, and that's it, so I'm, I'm fresh off the plane.

Yeah, there was a bunch of, I was in Chicago for Monday and Tuesday for some training with one of our prospects, and then, yeah, a bunch of storms came through, so all these flights got cancelled and messed up Tuesday evening.

I'm sure that was probably quite a few places across the eastern part of the Canada and US.

Anyway, rebooked this morning via Montreal, so anyway, I'm warm and happy to be here, so yeah, thanks for having me.

So, if you mind me asking, what's the weather like now in Ottawa, in North America in general?

The car said it was 31 degrees on the way home.

It's quite warm, quite humid out there.

That's typical summer, kind of August weather, usually anywhere from kind of like upper 20s to low 30s, and yeah, humid, humid is quite standard for this part of the world, so yeah, summer times.

So, pretty much like right here in the Netherlands, humid and warm.

I've been sweating all day.

It's been just a sunny all day and horrendously hot, so with this humidity, it's, oof.

Yeah, yeah.

Do you have aircon in your house there?

Okay, yeah, yeah, yeah.

I'm on the top floor, and it doesn't really reach up here, so I got a fan I can put on if I need to.

Yeah, I have a fan as well.

I just put it in the window, and it's just venting air all the time, and it kind of works, but otherwise, oof.

It's difficult to work at times because of the temperature, but global warming.

We'll do what we can.

That's all right.

We do what we can.

All right, so we have a lovely evening today with Event Driven Architecture, but first maybe, because some of the audience might not know you, so tell us something about what you do.

I see you already have the appropriate t-shirt on.

I changed.

I changed.

I have to do a little bit of branding for this.

Obviously.

There we go.

Nicely done, yeah.

So, Carol's written this whole implementation agnostic course, and I guess I'm on the other end of that, right?

I'm very biassed about how we think implementations should go, within reason.

Yeah, so I'm Aaron.

Aaron Lee.

I work for a company called Solace.

We're based in Ottawa, Canada, headquartered here.

That's where I'm from, but yeah, for the last eight years, I've been living in Singapore.

I was doing a role of a developer advocate, where we kind of do a lot of developer events, workshops, tutorial samples, hackathons.

I make a lot of YouTube videos.

Perhaps if you've ever seen the Solace YouTube channel, I have a few on there.

I do a monthly office hours, most months, not every month.

And I have another, my one in two weeks is going to be all about our event portal and all the new features that we've been building on that.

So, we're overdue for an update on Solace event portal.

And yeah, so as of now, I'm a solutions engineer, pre-sales role, where as part of the sales team, I'm the technical person who might go in and do a proof of concept, kind of do the hardcore technical presentations that sales people don't want to do.

So, that's my current role.

And yeah, been at Solace 14 years.

Check us out, Solace.com.

We are a publish, subscribe, event-driven broker.

And that's why I guess why I'm here today, to talk a little bit about how we do things and maybe contrast it to some other viewpoints.

I mean, Carol, from our very first time we had a conversation, I think it was via the talk to your dev advocate kind of like feedback link there.

We've had like hour, hour and a half, two hour long conversations that were like, this could easily just be a podcast.

Like our conversations just turn into this anyway.

So, now we're making it official, I guess.

Yeah, that's very true.

That was the idea in one of our conversations.

We've been chatting on and off, what, over a year now.

Every once in a while, just popped on a call.

You eating your late night dinner in the office in Singapore.

And yeah, me just sitting midday or late afternoon, just having a conversation about event-driven architecture, different patterns, use cases for different architectural styles or well, ecosystem architectural styles to be precise.

Let's hope other people find it as interesting as we do talking about this stuff.

I mean, there are nerdy people on LinkedIn and YouTube that enjoy these kinds of conversations.

So, I hope they do.

But speaking about architectural styles, one thing that I really find interesting in terms of, and that's not only marketing-wise, but positioning-wise in terms of Solace.

Solace at this point, I think is the only event broker on the market that actually positions itself as a product.

The company positions the product as a piece of technology that can solve ecosystem problems.

Because if people think message broker, event broker, a lot of people think that it's in a context of a singular system.

So, internal messaging within a system.

Sometimes they think about it, maybe hooked up to a integration platform like Boomi, MuleSoft, Tibco BusinessWorks, something like that, to facilitate that asynchronous communication or a little bit of a fake load balancing through queues.

Whereas those message brokers typically use with those, they do not have that kind of philosophy there that they're actually solving ecosystem-wide problems.

Whereas time and time again, when I see any presentation from Solace, that's a large distinction there that anybody who I watch from Solace, especially from the CTO office or field CTO office, they always talk about ecosystem problems.

They always talk about the scale that is way beyond the singular system, addressing different

domains, addressing different business processes, addressing organisational issues to a certain

extent, and addressing the way of thinking in terms of thinking about your business in

a manner that it's not just request reply processes and we're just going to put that

in a system, put that in an ERP, and we're going to be great.

Or CRM, right?

No, it's addressing that way of thinking about the business itself.

And that's the differentiator there, for me, at least.

It's a paradigm shift.

Solace came from capital markets, trading platforms.

They've been dealing with this on-demand, publish-subscribe, event-driven style of data movement because you have to be as real-time as possible.

When a price changes, that tick needs to get sent right to my trading platform or my trader or my dashboard or whatever.

I want to see that real-time.

You're not pulling these things.

And so they've kind of come from that.

They've existed in this world for a long time.

And now, as all these other industries are kind of slowly becoming IT companies, like I mean car manufacturers and manufacturing, and everybody's an IT shop now.

And their kind of legacy way of integration, database at the centre of everything, kind of everything goes into a database, everything comes out, it has to modernise.

We need more real-time data access to our inventory.

Customers want more real-time access to their information.

And I guess, again, I'm a bit biassed, but Solace views itself as just an extension of your network.

So whenever you have to send anything over the network, you could make it better by going on Solace in some way or another.

Maybe not every single thing, but that's the way I do think about it.

We enhance your TCP communication with more stateful connections, broadcast, unicast.

You can do fan out.

I mean, anytime you want to send a file from one server to another, you could send it over a Solace event broker or a message broker event process.

It could go to other places, right?

It could be, if your downstream system is not there, we could persist it temporarily until that downstream system comes online.

It's unlocking a lot of patterns and interesting things that are fundamental to how a business operates.

And so it's not just one little piece of integration or one little message bus between apps.

It needs to be, I think, a shift in how you think about your business operating and the data that's flowing around, getting out of these silos, moving data everywhere with guardrails, of course, ACLs, things like that.

So people can't subscribe to anything.

Anyway, that's...

Remind me, because I'm not sure if I remember this correctly.

Did Solace actually originate from more of a hardware broker?

Yeah.

Yeah.

So a bit of history.

I mean, yeah.

So Solace was a hardware company for many, many years.

We built a purpose-built appliance, FPGAs, network processors.

There was a Linux OS on there, but only to run the control plane with our monitoring and watchdogs.

And the entire data path was all hardware.

So we got very high-performance numbers.

And this is, I think, why the investment banks like this so much, because we do microseconds of latency from your API, say, send a message to go through the network into the router, the Solace broker, get routed to who it's going to go to, and back into the subscriber to the API receive.

That we do in about 18 to 20 microseconds.

So that's fast, right?

Not everybody needs that level of performance.

Investment banks like it.

You know, the average person, a millisecond is probably real time, right?

So for them...

I think that's the core aspect of that background, the trading markets, the financial world.

That's half of it.

That's half of it.

A lot of the founders of Solace came from IP networking backgrounds.

So they viewed this as a kind of a network-level-up solution rather than an application-to-application integration.

You know, they didn't view messaging or brokered communication as an application thing.

They viewed it as a networking thing.

So our DNA, a lot of our original founders and people that worked early on came from IP switching, quite a few patents around TCP and stuff like that.

So yeah, it's in the DNA of Solace, I guess.

And now it clicks for me.

Now I understand why certain aspects of the rhetoric and certain terms used in different presentations from Solace do not exactly match my understanding of interoperability from a perspective of application integration, because if you take that from a perspective of networking, yeah, that's completely different.

But that's a very narrow aspect of what we do in IT.

The network infrastructure, it's a very, very, very narrow element, whereas...

It is.

But everybody uses it, right?

Obviously, but it's the one thing that I always say when discussing that narrative.

The misuse of the word decoupled.

I was thinking about this on the plane.

I was like, what are we going to talk about today?

And I hope we do.

I'm sure we're going to get around to talking about decoupling, because you and I have somewhat similar and somewhat different ideas on this word.

So yeah, I do want to jump into that in a little bit.

Let's jump straight into that.

So I had a lot of discussions about coupling and decoupling.

Can I just say that we're not a hardware company only, by the way.

Some people, some of our old customers don't actually know we have a software product.

So when cloud finally started to become a real thing, we virtualized our appliance into a software broker.

So it's a Docker container.

You can run it on prem.

You can run it on your laptop.

I got one right here.

You can run it in whatever cloud you want.

We have Kubernetes, Helm charts, and all sorts of stuff.

And we have a fully managed SaaS offering as well, where basically, rather than having to run our software broker yourself, you can just go into Solace Cloud and say deploy.

So we have three different kind of flavours, I guess, of running a Solace broker, right?

So we're not only hardware, just for the record.

See, I wouldn't ever know about that hardware part, because I know Solace only from the actual application part, the software part.

Hi, Stefan.

Nice to see you again.

Hello.

Hello.

So yeah, let's jump straight into that coupling, because when I was researching for the architecture course that I was designing back a year ago, a very important topic was how to cover coupling in such a complex environment as application integration and integration platforms.

Because that's difficult.

And the biggest struggle was how to actually define coupling properly.

And at one hand, I had the narrative of something coupled-decoupled, which was very common, to which I didn't really subscribe.

And I still don't like the word decoupled, if we're talking about two systems that are communicating.

And on the other hand, I have read the book from Vlad.

Now, I'm going to blank out and forget what was Vlad's last name.

I'll look it up in a moment.

Sure.

He's known for DVD books and balancing coupling in software design, I believe.

Kononov?

Quick check.

Yes, Kononov.

So he wrote the book on coupling.

That took him about five years to do research and write that book.

All right.

And I get it, because coupling is such a complex topic.

Extremely complex.

And on various levels, in a singular piece of software, on an ecosystem level, in communication, in terms of understanding network and beyond the network, etc.

So it's very complex.

But yet, there is this notion in a lot of architects that they say, well, we're decoupling things.

Sure.

And that's problematic for me.

And I know that Solace uses this rhetoric extensively.

And I talk to a lot of architects from Solace.

And we agree and disagree at the same time.

And here's the thing that I figured about coupling.

It has a dual nature, just like light.

OK.

Did you ever hear me talk about this analogue there?

This one specifically?

I don't think so.

I don't think.

OK.

Yeah.

So let's see if you agree.

OK.

My take on coupling, it can be binary.

So something being either coupled or decoupled, which basically means two pieces of software either communicate or not.

OK.

And then it's on a spectrum, can be tightly coupled or loosely coupled if they communicate.

OK.

And if I look at an event broker, for me, it's a tool as an architect to provide a loosening of a coupling in communication.

OK.

What's your take then?

So in general, I agree.

But I think you're going to like this.

I've kind of come to the realisation that the term coupling, it doesn't refer to just a singular variable or dimension.

It's multidimensional.

It's in terms of direct communication.

It's in terms of time.

It's in terms of data formats.

It's in terms of knock-on effects that maybe you actually uncouple something, which produces more coupling somewhere else.

So it's all about, if we're talking patterns, there's always balances and constraints and things that trade off.

So yeah, I have two slides I want to flash up real quick.

Can I do that?

Is that allowed?

Absolutely.

If people are only listening, I just I'll have to talk to them.

But let me click share my screen here.

And hopefully this will come up there.

Let me know if you can.

Yes.

Because I'm going to talk briefly about the publish subscribe pattern.

But before I do, I just want to mention this one here, the observer pattern.

I learned this in second year university when I was doing a C++ course.

And I got some nice UML.

Who does UML anymore?

I don't know.

But that's, oh, really?

Sequence diagrams, maybe.

Sequence diagrams all the way.

I wouldn't be able to describe what I do without sequence diagrams.

That's a must.

All right.

So the observer pattern is basically you have a subject and an observer, right?

The subject is the thing you want to watch.

Think of a button on a GUI, right?

Typical example.

And you want to know when that thing has changed state, right?

You want an event when it's changed state.

So you will register yourself as an observer to that subject.

So I go to the button.

I say add me as an event handler.

Run this method whenever you change state, right?

So that's what this register unregister is.

And then when the subject changes state, he looks through his collection of observers and notifies them that something has changed.

So, you know, your callback gets fired saying, hey, the button has been clicked.

Because imagine designing a GUI where you have to pull every single element in that GUI to find out if it's changed state.

Like in a huge loop, you have hundreds of elements.

That's an insane way to design a GUI, right?

So GUIs are by nature event driven in the sense that they notify you when you scroll it, when you move the window, when you click something, right?

That's all being event driven.

Yeah.

A lot of the functions are on something.

On click, on hover, on...

Yeah.

So that's your event, right?

So that the button has been clicked.

So publish subscribe is a kind of a decoupled version of this.

So if I just go forward two slides, like we...

So think of publish subscribe very similar to the observer pattern in the sense that I need to notify when something has changed, right?

So the observer is the subscriber of this data.

You're the consumer of the event.

The subject, the thing that's notifying people downstream is that it's going to be...

It's the producer.

It's the publisher of this data.

One, in terms of coupling, like if you look at the observer pattern, the subject has this collection of observers, right?

So it's the responsibility of the subject to have to go and tell all these observers.

It needs to keep a list of them.

It needs to maintain that.

It's the one that's doing the calling of this method, this update method.

So there's a coupling there, right?

Because the subject now has to keep track of these guys.

So if I'm a pitch, that would be the equivalent of an application that needs to notify something without having it with some sort of an async communication.

Basically, the subject would be doing a request reply to each one of them.

Something like that.

Yeah.

It could be a UDP.

It doesn't have to be a request reply.

I mean, maybe if you don't get the update, it's not the end of the world, right?

But if you want to make sure that everybody got it, it would have to be some kind of acknowledgement coming back.

And so as this scales out, what if this is like a thousand things?

Now the subject has to do all this work every time it clicks its button to notify everybody.

So there is this coupling there in that sense of responsibility.

Whereas with the brokered, the published subscribe pattern, as implemented with a broker, an event broker like Solace, is that they are decoupled in the sense that they all connect into this broker and the broker is doing the responsibility of messaging, of notification to the different people, right?

And the individual producers don't have to know how many subscribers there are out there.

It just sends one message into the broker.

The broker is keeping track of all that.

And the consumers don't have to know where is this producer?

Where is this button that's being clicked?

As long as I get the event, that's all that I care about.

And so that's a decoupling, right?

That my consumers and producers don't have to know about each other.

They can be lots of them or very few of them and it doesn't really change anything.

So that's one.

And the second one that I have there you can see is temporally.

If your broker has persistent storage, obviously, if one of your consumers is offline, maybe it's a database and it's getting rebooted because you did an upgrade.

Your broker can store that stuff and the producer does not get impacted.

Like if I'm trying to write to this database and it goes away, now I'm stuck trying to write to this database.

But if you put it into an event broker in the middle and the database goes away, the broker is like, all right, I'll just buffer this temporarily.

Bless you.

And then it'll go out.

So in this specific way, not talking about data, not talking about schemas, event brokers can decouple quite architecturally and temporally.

On the two characteristics of the spectrum, basically.

Yeah, but there's more to that.

So I took the learnings from Vlad Kononov that I found.

By the way, if you would want to read the book, here's the QR for the book at O'Reilly right there.

So check it out.

That's a really, really good book on coupling.

If you want to understand coupling, I think that's the easiest explanation ever in a context of software design and development.

Does it talk about decoupling as well, or is it only about coupling?

It talks about loosening the coupling, not decoupling.

Okay, all right, all right, cool.

We can take a slide down there, Carol.

I just wanted to flash those two.

All right, all right, all right.

Oh, nope, not this one.

Other way, other way.

Nope, nope.

See, this is my- Too many buttons, there we go.

Too many buttons and not yet that familiar with the studio.

But I just had to turn off your screen, and that's fine.

So if we're considering coupling on the spectrum, and I absolutely agree that if we're looking at message brokers in general, or Solace as an event broker, message broker, yes.

If we add to that guaranteed delivery and the temporary persistence of in JMS, this would be queues, or in topics, it would be the durable subscriber.

Yeah, and then also the same kind of configuration for AMQP and other relevant.

We're working here on the level of temporal coupling due to that persistence.

And then that's a good question.

I think that would be the interaction coupling in terms of who's interacting with who.

Because now we enable, because we have that temporary persistence in middle and that mediator in terms of communication.

Now we can have one-to-one, one-to-many, many-to-one, and many-to-many.

And those systems on both sides, they have no idea who's talking to who.

That's right.

We're obfuscating that knowledge through that mediation over a message or event broker.

That's right.

But there are other types of coupling.

And for the purpose of ecosystem architecture, we divided them into architectural or operational.

So there's many.

And some of them are solvable on the level of an event broker.

Some of them are completely not solvable on the level of an event broker.

So if we're talking, for example, a pure JMS broker, like let's say Tibco EMS, a very old-school message broker, it implemented only JMS. Right?

So at this point, we already had this kind of a, not conversation coupling, but that was more of a technology coupling, protocol coupling, right?

Because both systems communicating, let's say, only two are communicating through that broker at this time, both are forced to be protocol-specific.

So they're already coupled by technology and by the protocol.

Okay.

In comparison, that's not a problem for the Solace event broker, because Solace implements multiple different protocols.

So JMS, AMQP, you can access the topics via REST, etcétera.

MQTT, yeah.

Yeah, MQTT, that makes it another degree of coupling that's dismantled by the way the broker is implemented, right?

This makes it easier.

But we still have the problem, and this is a typical event-driven problem, that we have contract coupling in the terms of what is the scope of data.

And we still have a data model and data format coupling, well, data type and data format coupling, because contract coupling covers model.

Which is basically, if we take just an event broker, it's unsolvable by an event broker, because event broker would have to parse the payload to be able to resolve that coupling.

Yeah, it's outside of our scope.

It's above our layer.

So it might be.

Coming from this networking thing, right?

Like we don't want to be messing around with payloads.

The hot path, we're just the transport among applications.

Absolutely.

And then last of all, you always have semantic coupling, which is something we want.

This is the type of coupling we need to have.

If we don't have semantic coupling, that means we're doing something very wrong, because that means we don't understand what data we're moving from point A to point B.

Define what you mean by semantic coupling.

The generalised understanding and meaning of transfer data, both business and metadata, on the level of users, business process, purpose of use, etc.

That's the semantic understanding of the instantiated piece of a payload.

We all agree what this thing is about, right?

Exactly.

We want to understand what it's for and what it's about.

Otherwise, we're doing something very wrong in our IT ecosystems, that we're moving data, but we're using it outside of the scope of what it means.

This happens quite a lot, actually, and it's a horrendous thing to resolve, because that leads to a lot of problems, and not in IT, but in business in general.

But this is the type of coupling we actually want to have, and the rest we need to either mitigate or handle.

Be aware of, at least, at the very least.

It's about choices.

And thinking about from the data perspective, I don't have as many strong opinions as you do.

I know some installers have considered this more.

As we've slowly started to move up the stack, we released our micro-integrations.

We do not want to be a full-blown integration platform.

We have partners, and there's really good products out there that we don't want to do that.

But for tiny, little, simple flows, maybe you don't need a whole-blown platform.

We have these micro-integrations just to do a little bit of...

It's like a connector that has some transform capabilities.

So people that have been working on that might have more things to say about data, data formats.

I just always thought it should be like Star Wars, where you can speak whatever...

As long as you can understand, you can speak your own language.

You don't have to all speak one common language.

So it's more about the...

Would that be Star Wars or more like Star Trek?

No, Star Wars, man.

So Han Solo is speaking English, and Greedo is speaking whatever language.

But they can each understand each other, right?

So everybody else is speaking...

Yeah, everybody speaks their own language.

And as long as you can understand, if everybody can understand every language, nobody needs to speak more than one.

All right, but you mentioned a very interesting feature provided by Solus being the micro-integrations.

Because we talked about micro-integrations quite a bit in our sessions and also where we met some time ago.

I'll put them a little bit in the context because that's an interesting...

In general, an interesting feature in solving some very important problems in terms of EDA problems, which are, well, not native to EDA.

So if you ever consider publish-subscribe

in a context of one-to-many,

and this is something I, at first,

I have had troubles absolutely understanding

because me coming from service-orientated world,

where you have three layers of abstractions,

you have an integration platform,

you have the message broker in there,

and you have those abstraction layers

that are responsible for things,

and I did not realise that they actually mitigate

the EDA problem of publish-subscribe,

the problem being security.

Okay.

If we do broadcasts, publish-subscribe, one-to-many, the mechanism itself, it's secure because every message broker, Solus included, has ACLs, only those that are on the access list, only those can access the structures and actually read the messages from topics.

Perfectly fine.

Sure, yeah.

The problem with using this pattern, publish-subscribe in general, the security problem doesn't come from the transport mechanism itself, it comes from what is the result of that.

And the typical even driven architecture result of that is that every subscriber receives exactly the same payload.

Sure, yeah.

Now, imagine you're sending GDPR-restricted data or whatever is the equivalent in North America, I don't really know the names for those privacy laws.

Privacy stuff, yeah, I don't know.

Privacy-restricted data.

So let's say employee data, healthcare-related data, etcétera.

Sure.

And you're doing this via publish-subscribe.

Yep.

So that means that every subscriber gets exactly the same set of data.

Yep.

Where they actually probably, for their business purposes, they actually probably need a subset.

So while they, let's say, persist and save in their databases that subset, well, that data still arrived there.

Arrived, meaning that it was probably at least logged somewhere in the log.

It could have been looked at, yeah.

If you send something, it could have been looked at, right?

By any kind of, yeah.

Exactly, and now imagine one of those systems is compromised by whatever means.

For example, a disgruntled employee just copied all the logs from that system and that system never was sufficiently updated to anonymise logs.

Yep.

And now we have a humongous leak of personal or healthcare-related data somewhere.

And this is the problem that is a consequence of using publish-subscribe.

Now, the interesting part to put the micro-integrations into context is that while all the message brokers will have that problem with publish-subscribe, Solace is actually actively solving that problem by introducing micro-integrations.

Because if you have micro-integrations, you can trim down that scope to the exact need to know.

Now, the absolutely funny thing, and analogous in my head from the perspective of service-orientated architecture, that is exactly the responsibility of the adapter layer in API-led architecture.

This is what adapters do in front of systems when those adapters subscribe to topics and then translate that to those particular systems.

So if you put micro-integrations in the context of an integration platform in API-led architecture, micro-integrations serve as the channel or adapter layer.

In this architecture, right?

Sure.

So contextually, they take that responsibility and they resolve certain pain points of pure event-driven architecture.

And that's the very interesting part which none of the event and message brokers address at all, because they never look at it from a perspective of an ecosystem and the ecosystem problems.

They look at it from a perspective of communication and protocol or transport.

Yeah, well, that's what I was thinking.

Like, I've never thought of it as from a security perspective, but you're absolutely right, where if you are sending something, the full payload, maybe some of these consumers don't need that full payload.

The typical way to do that would have been like a kind of a double publish, right?

You kind of read that message coming in, you might format the payload.

But again, this is outside the scope of the broker itself, right?

You need to kind of go, yeah.

But I will say, unlike some other brokers that are out there that people use, we do have a very, I wouldn't say complex.

We have a very smart topic system design for doing routing and filtering in the broker itself, right?

Going back to your comment on ACLs, we have topics that are very granular.

They're multi-level, they're hierarchical, you know, with a slash between each level where you can specify on a level-by-level basis, certain identifiers, certain keywords to describe what's actually inside the payload of the message.

And so as a subscriber, you can be very specific or very coarse-grained or fine-grained.

You can use wildcards on a particular level.

I want all data from all regions or I only want UK data, right?

I'm subscribing to slash UK slash da-da-da-da-da or this particular company or this particular person.

I mean, depending on how you've built your topic structure using your ACLs and you're subscribing to your topics, you can be very selective about what data you want.

And the broker does that filtering for you.

It's not a big, fat, coarse-grained topic, which is like a log file of some other brokers that are out there where you actually have to read the whole topic in and throw away stuff that you don't want, right?

That's how they do filtering, is in client space.

Now I see you're very, very, very careful about mentioning specific names.

No names, no names, of course, no.

But it's just a different way of doing things, right?

So for certain use cases, you want that full feed most of the time.

So that's probably great.

And this is a very EDA way of doing it, very operational way of doing it.

And most message brokers are operational message brokers.

Yes, there are those log brokers, which shall not be named.

But yeah, but see, this is a very good distinction there in terms of security.

In JMS, you could also apply that kind of filtering.

In AMQP as well, you can apply that kind of filtering.

I'm not familiar with AMQP-1.

AMQP doesn't, yeah.

Well, so Rabbit, Rabbit, AMQP-091, which is Rabbit.

Rabbit MQ.

Only, Rabbit only.

Yeah, yeah.

So you can use a key exchange and...

Routing key.

Routing key, that's it, yeah.

Yeah, and in JMS, you could have used selectors on subscribing to a topic or on bridging, bridges between topics and topics, topics and queues, whatever that may be.

EMS has hierarchical topics as well.

It's just people don't use it quite...

The way they've implemented it is a little bit different than Solace, but you can do similar.

Just the closest analogue, if people know MQTT, it's very similar to that, right?

In terms of the hierarchical topic structure, very dynamic topics.

You know, publishers specify on each message what that topic for that message is going to be.

It's metadata.

It's not a thing.

It's not an object that I've created on the broker.

But the problem with that kind of URI approach is that, of course, we're tending a little bit to security because now the subscriber does not get the full funnel of all the data published into that format, whatever the variables may be in that format.

It's securing the subscriber on a level of volume of that data, not on the level of payload.

Well, yeah, yeah.

I mean, the broker can't know payload, right?

Exactly.

It would have to be governed some other way that like, yeah, you should not be publishing secure payloads on these topics over here or something like that.

Yeah, and that's exactly why either we need an integration platform in place or we need the micro integrations.

If somebody doesn't want to invest in an integration platform because that's not a sufficient use case, then micro integrations are the answer there.

And this resembles the very old style.

There is a transitional architectural style between broker architecture and API led architecture, where basically you had those brokers that were handling that orchestration and everything.

But that was all centralised.

And somebody figured at a certain point, okay, let's add adapters that are deployed with those systems to do that small translation from a unified data model within the broker integration platform to the data model of that specific system and abstract the logic of that system in terms of data access.

And this very much resembles what now Solace is doing with micro integrations in that sense.

So there are plenty of very old school analogues to that approach.

And these are all well proven analogues so the approach here with micro integrations solving that problem is also a very well proven approach for that smaller kind of uses where that kind of huge deployment of all those interoperability tools is required.

And another analogous way to do it, and Solace is a very good broker to do that due to the implementation of REST, but also possible with other message brokers is to, instead of doing micro integrations, use an API gateway.

Some API gateways enable a protocol mediation.

Protocol mediation being that the gateway is the subscriber to a topic, pulling the data and then turning that communication into a REST call to a system on the other side.

And this is exactly the same pattern.

And now we have a race between different product vendors.

Who can deliver that better?

So we have the API management vendors that, oh, we can do protocol mediation.

You can use any message broker and that's going to be fine.

We have Solace with micro integrations.

Hey, we delivered the same thing.

Go with us.

We have integration platforms and they say, yeah, use any message broker you want.

We're going to be great with that.

It's all a matter of balancing what are the fine grained requirements for interoperability.

And that's what maybe someone like a solutions engineer like myself might work with a customer to help identify those requirements and kind of figure out the best path forward.

There are times when Solace is not the best fit, you know, and I try to be honest.

I know the sales guys always want to sell lots of things, but I'll try to be like, all right, you might not need this particular thing or feature.

I do want to say micro integrations, people are like, why do you call them that?

It came from like microservices, right?

Like kind of a couple, like little small pieces of functionality.

We wanted to build some kind of way and more than just a connector, a lot of people still call them connectors.

It's the fact that there are connectors plus a transform in there, like you can configure within there.

You don't have to, it could just be a straight through connector, but you know, you want to, you know, our brokers, if you stick enough of them together, we call, you know, we call it an event mesh where data can just kind of flow around between different brokers on-prem or different clouds.

And we want to move that integration to the edge.

And that's why we're at it.

So micro integrations, just a little, it's a connector plus a little bit more smarts.

And yeah, that's microservice, micro integration.

It kind of makes sense.

I don't know.

I tried to get our PLM to call them Mintz, like M-I-N-T-Z, Mintz.

And then we could have like Mentos as like, you know, as our swag at conferences and stuff, but he didn't go for it.

I thought it was a great name.

So we call them M-I, we call them M-I-N-T-Z internally.

But Andrew, Mintz, I think Mintz is a great name.

So micro integration.

Oh man.

We got some questions popping up there.

Oh yeah, this is a good one.

I think this is very relatable to micro integrations, but I think that's a very different aspect.

Micro gateway.

Yes, yes we do.

Except nobody uses that, I think.

We built it for a customer for a use case that went away.

And so Solace, yeah.

So Solace is a multi-protocol broker.

It has our native protocol, which is the fully featured SMF.

You know, that's what we had for many, many, many years.

As AMQP and MQTT, these standards.

MQTT coming from IoT is made by guys at IBM.

MQTT, AMQP as well.

These standards got made and we implemented them in our broker.

But we also have, our broker can talk REST. Now we typically, I don't wanna say bash REST, but REST is not the most performant of protocols, right?

It's asynchronous, quite, it's not very compact.

It's point to point, it's a blocking call on the network.

So it's easy.

It's easy and simple.

Everything understands REST. And sometimes you just wanna like post somewhere, post one thing.

You know, you're not streaming hundreds of thousands of messages.

So we have this ability to- I'll only argue on one thing.

What's that?

It still can be very performant.

It's not very responsive.

Responsiveness is the characteristic.

And this is coming from hours of discussions with Mark Richards and listening also to his partner in crime, Neil Ford, rant about it on different trainings.

There is a distinction between performance and responsiveness.

The distinction is, again, exactly about blocking and non-blocking communication, which you mentioned then.

The communication can be still very performant and very low latency with a blocking communication.

Sure, yep.

But it's gonna be a lot better.

The system is gonna be a lot better responsive when the communication is not blocking, which means that if we use asynchronous, the communication is, let's say, 25 milliseconds and we're done.

If we use sync, we're always dependent on the response time of the system or the API in the back, right?

So that accounts to, well, getting the call out, waiting for the response, and then processing that response, which is the whole thing can amount to various times, right?

Not just 25 milliseconds needed to send out that message, but also the wait time and then to process the response.

So we're literally blocking a thread in our programme to wait for that response.

So the programme, if we would consider that programme having a single thread, becomes unresponsive.

Yes.

So we're now, if we're doing async, now we're boosting the responsiveness because we're freeing up the resources.

So it's not performance-wise.

It's very close to performance because obviously it will somewhat impact that performance, but it's the characteristic is responsiveness, which is a very fun distinction.

Not a lot of people actually go about to look at.

That's an interesting way of phrasing it.

But this is why some of these kind of micro-architecture, the microservice architectures that are built on REST, when they get into scale, they have to scale out their web server because every single call is a blocking, it's a socket that's being held open, right?

And if you have a chain of these things bouncing around between different microservices, you have all these web servers basically sitting around holding all these sockets open while the communication is kind of chained through a bunch of systems.

So if you can, I mean, there's also blocking at a programming level, right?

Like in my thread, blocking programming is often easier because if there's an exception or an error, I can deal with it right there because my thread stops and it either succeeds or I throw an exception and there's an error.

Asynchronous programming is more complicated because your error response might come on a different thread over there later, and now you have to somehow correlate it, right?

To build good high performance, high throughput systems is probably much more complicated than just writing simple rest posts here and there, right?

Let's not get into the error handling in EDA.

That's a topic we can talk more than one evening about because that's a very, very complex problem with the correlation aspect, error handling aspect, handling metadata to get all that to work.

It is.

I'll just say that at least log if you get an exception anywhere in your code, please.

Because I've done a lot of developer workshops.

I've done code reviews and people just code the happy path and here's some error handlers that you have to implement so at least you know if the broker or the API complains and some people just don't have any commented out.

So if you're a developer out there, just please log exceptions even if you're not acting on them just so you can debug them later, please.

And if you wanna go better in terms of what you're building in terms of communication, always use some sort of a metadata use a message ID, correlation ID for your messages whether or not they are rest or asynchronous or asynchronous, it doesn't matter.

Use the metadata because that's gonna help you then later with traceability, root cause analysis, finding that error, understanding what the hell happened in my system or my ecosystem.

That's a very important thing that a lot of people forget about.

Well, there's standards for this stuff now, right?

There's open telemetry.

You know, it's I think the second biggest CNCF project after Kubernetes.

Like they've standardised with Zipkin and a bunch of other kind of like ways to do tracing and obviously in a restful architecture.

The nice thing about having a broker like an event broker like Solace is that all your communication goes through there, right?

So it's easy to monitor it, to like log things, to like watch communication as it goes.

When you have a bunch of distributed non-brokered communication like a restful architecture where everybody's just doing individual calls to unless you have something like Istio like a service mesh that can kind of monitoring and managing that.

How do you track data flying around between your architect, like your apps?

There's, it's very difficult, right?

So open telemetry allows you to insert these trace IDs, these span IDs, so that you can watch the hops of messages going around.

And this is a very interesting use case going back to the original question that's micro gateways.

And I'm now trying to see if I have this book somewhere.

Yeah, I actually have it somewhere in my bookshelf here.

There we go.

This is very, very old book called Enterprise Integration Patterns.

Oh, classic, yeah.

Classic from what?

2006 written by Gregor Hohpe.

That's a classic.

If you're an integrator, that's a must have just to understand even driven architecture well.

And I'm trying to find- All sorts of integration patterns, yeah.

Yeah, all sorts of, well, most of them are related to even driven design on a- Messaging, yeah.

Yeah, messaging and even driven design on a system level.

Since the creation of this book, a lot of things have happened in the industry that changed interoperability drastically.

So that book doesn't cover a lot of the topics, which is somewhat of a problem.

It's 20 years old, so- 20 years old, yeah, exactly.

So the aforementioned micro gateways relate to this lovely chapter, which nobody can see in the camera.

So I'll just say it's a wiretap pattern.

Yeah, wiretapping, all right.

Wiretapping, this is a very lovely pattern from this book.

I never heard of it before I read this book, to be honest, because again, my background service-orientated architecture integration platforms.

So if we're doing integration platforms, we don't really care about wiretapping because we're already parsing the goddamn payload.

But wiretapping in event-driven architecture is something absolutely brilliant for use cases other than just transferring that data somewhere.

And micro gateways is equivalent of a wiretap on a synchronous communication to turn the request and the response into an event for whatever use you have.

And I found that the micro gateway is absolutely brilliant if you want to have proper observability on your RESTful communication, on the medium of the communication, on the protocol level, not on the system level.

Yeah, if I could just explain it for 30 seconds so people actually know what we're talking about.

So Solace, as a multi-protocol broker, we have a REST API, an HTTP, it's not REST, it's an HTTP API where you can post into the broker as a publisher, and that message can then, you know, that becomes a message and you can listen to it with a JMS subscriber or a MQTT connection.

And the broker can also webhook out.

Like if you put a message on a queue, we can say, send this, post this message to somewhere else.

So that's our messaging mode.

Then our gateway mode, the broker acts as a proxy.

And it basically, you publish into the broker and then it just goes straight through the broker and out the other side to your designated endpoint.

And it sits in the middle.

And you might say, well, that sounds like extra hops and a waste of time, why would I want to do that?

And for this one, one of these is exactly the right use cases.

I want to wiretap.

Publish subscribe by default gives you a wiretapping ability.

It's like, if your communication is over publish subscribe and you want to listen, you want to tap into that flow of information, you just add another subscriber.

It's called subscribing to that topic or topics, right?

It's nothing much.

And so when that REST post or REST get or REST whatever it is comes into the broker, it gets translated into a message and then shot out the other end.

But while it's in the broker, it's a message and you can subscribe to it.

And you can listen to the response as it comes back through.

And you could use that to do triggers or logs or all sorts of other stuff.

I actually recently used the micro gateway mode.

I pointed it back to the broker's admin port.

And so I logged into the PubSubPlus manager GUI via the micro gateway port.

And then I could listen to all of the HTML and GUI and all the calls going back and forth via messages.

It's kind of nerdy, but yeah, I enjoyed it.

You can wiretap, point it to like amazon.com, point your Solace broker's backend to amazon.com.

And then when you point your browser to Solace, it will actually, all of the responses, all that webpage will fly through the broker as messages.

I don't know what that's useful for, but it's kind of neat.

So yeah, but the reason why we make a distinction between a PubSub and wiretap is primarily that PubSub, if we think about integration patterns, we're thinking about moving the business data.

While if we're thinking about wiretapping, we're not thinking about the business use case.

We're thinking about more of a metadata operational use case.

So if we're thinking, okay, we have some sort of a communication, let's say, Salesforce and NetSuite, they're related systems.

They often cooperate and they need to communicate.

So now we're trying to figure out in terms of temporary analytics, what's going on.

So if we proxy all that communication through a micro gateway and wiretap it, we can source that even for, let's say, temporarily for the purposes of our analysis.

Let's say we're trying to analyse anomalies in traffic.

Anomalies being the traffic originating from somewhere else in the actual Salesforce system.

Somebody is representing themselves as Salesforce, but they're actually a malicious party that's trying to do something in our network, trying to communicate with NetSuite to, let's say, source some information from NetSuite.

And if we are wiretapping that, and that's a temporary situation, we could source a sample of data over, let's say, next two hours, the next day, and then analyse that in terms of the metadata of the communication.

Because if we're wiretapping, all the HTTP headers will be available.

All that metadata that we've put into that will be available.

So we can see IPs.

We can see how the headers are composed.

If there are discrepancies in the build of the headers, we know that Salesforce is, for example, sending a very standardised set of HTTP headers because we define those headers and the calls that these have to pay.

Okay.

And if we see that there are discrepancies and there is communication towards NetSuite that doesn't have all the headers or has excessive headers or has headers with, let's say, correlation IDs formatted in a different way.

Say we use, in Salesforce, we use UUID or GUID.

And this one is, for example, sending just some sort of a string that, let's say, a millisecond timestamp.

Now we can weed those out in analysis thanks to the micro gateway and figure out, oh, we have our anomaly.

This is the source of our anomaly.

This is the source of our discrepancies in data.

Okay.

Let's analyse further where that comes from.

Let's track it down.

And this kind of behaviour enables us to do so.

If all of your REST communication was going through an API gateway, like none of these things are making direct service calls to each other, then you could probably do it there as well, right?

It's essentially, you want to have a broker.

You want to have something in the middle that you can kind of observe the traffic going through it.

But yeah, like any kind of audit type of use case or any kind of analytics type of use case, you can easily just tune in, wiretap to that feed that you want.

And it's a great...

And it makes it a lot more...

The thing I try to impress upon people when I'm doing these workshops, you know, around Publish, Subscribe, is that it kind of future-proofs you to some degree.

It allows you to be flexible in the future with your architecture in a sense that you're not locked into a point-to-point style of communication where you might say, all right, now I want to send some of this data over there, or some other consumer is interested in the same data.

If I'm locked into a point-to-point architecture, now I have to do extra work to like double publish or put it somewhere else.

And so I had a...

Just as an analogy, I had a demo written where you could interact with it.

Cars were driving around and like, you know, you could break down and you could fix your tyre.

And we wanted to gamify it.

My boss at the time wanted to gamify it for user group.

And I couldn't find the source code.

I didn't know where it was, but all of the communication was going over Solace.

And I just thought, you know, I could just subscribe to these topics that people are like interacting with, like when you break down or when you fix your flat tyre.

And I just wrote a little subscriber that listened to people playing with the game.

And then I just built a dashboard off that data without having to like touch the original code.

The publishers, the producers, the game just kept working the way it was, but I could now subscribe to it and build additional value, my little, you know, leaderboard without actually touching it.

So it's great in the sense that, yeah, you can like wire tap that data for other use cases without having to touch your original producers and subscribers.

So yeah.

Well, we got to, let's see.

What does Stefan say?

Is there a person setting this up?

Hey, yeah, I guess so.

Yes, yes, they can.

Gotta be very careful about that.

All software needs to be a subject to audit and reviews and checking the configs secured from a network perspective and all that.

There's so many layers to that what needs to be checked to secure your IT ecosystem.

Absolutely.

But you can set this kind of wire tap anywhere and that's kind of backdoor.

You're using an API management in your API management policies and security policies.

You can create a wire tap.

You can create a backdoor then there, right?

A lot of the API gateways right now are SaaS solutions.

So they're all still accessible.

You can still figure out how to access them, right?

If you're using micro integrations and you can still have network access to some of the systems that are not properly secured or somebody didn't deactivate your VPN access, there you go.

You have a backdoor.

There's always a way.

There's always this human error element that will be that problematic aspect where we're actually having problems because of that, right?

Yeah, yeah, yeah.

You gotta watch out for those bad apples but you can hopefully detect them.

So I mean, I still remember use cases from back in the day when I was working for T-Mobile and we had to secure the SMS gateway because SMSs have to be tested, right?

All the notifications that we have around specific products and then integrations into the SMS gateway, it has to be tested in low level environments because that's part of the functionality that was introduced with certain releases of software.

And people realised, testers realised and integration specialists realised, especially from our dev vendors, that it's actually the production SMS gateway hooked up to all the non-prod environments because otherwise people wouldn't get those test SMS on their phones to test them.

Yeah, and those enabled to actually put something else than your number.

So whatever string you wanted, so you could put whoever's number in and people started playing around that.

It's just a matter of where you sit, where you are in the ecosystem to spot these kinds of opportunities either to enhance your security or to use the whole to your advantage, whatever that advantage may be.

Maybe some harmless fun, maybe something a little bit more cringey or straight up illegal, right?

Yep, yep.

Where there's always a whole.

So yeah, exactly.

Humans are fallible, right?

I mean, you just got to make sure we kind of watch out for these guys, girls, guys and girls.

Any gender can be duplicitous.

So I was just going to say that in Solace world, we do have a kind of a separation between the control plane and the data plane.

And that as an administrator of the broker, I can't actually connect to listen to the messages on the broker.

I can't look at the queues and see what's in there.

You actually have to have client credentials.

So there's a bit of separation of church and state in that.

Now, obviously as an admin, you can go and create yourself a client username to use for your nefarious purposes.

But obviously the logging would see you doing that.

So there's ways around it that basically on our side, yeah, we try to make sure that people administrating the brokers can actually look at the payload of the data so that, yeah, it's sensitive or secure.

And we have a lovely comment from YouTube here from Shin, I guess.

Yes, we could encrypt everything, but then we jump into the problem of trade-offs again.

So the common contradicting pair in trade-offs is performance versus security.

If we want something to be highly performant, security will definitely hinder that performance.

If we want something to be highly secure, it's not going to be highly performant.

These two don't go well together.

So if we're dealing with...

Now we're having a lot of money.

If we're dealing with somebody who's got a lot of money, like the trade markets or financial institutions, yes, we could try and do both, but it will still be contradicting each other to a certain way.

So there's a matter of having that discussion.

What is more important and to what degree?

And that's always going to be the same problem, right?

We either want it very fast or very secure.

There is a middle ground, but it's neither going to be fast nor entirely secure.

So you have to ship that security aspect somewhere else to mitigate that security risk in a different area of your ecosystem so that that transport layer is not impacted.

It's all trade-offs, man.

It's always all trade-offs.

It's either money or effort or time or security or loss.

This training that I was doing in Chicago there, we were talking about...

We talked about coupling for a little while and I invoked you in my head, actually.

It was kind of funny.

But we were talking about DR. And in Solace world, DR is when we go between data centres.

So not like HA, high availability within one data centre.

You have two data centres.

In case you lose a whole data centre, you can switch to a different one.

And we have...

You can do it either sync or async, right?

Sync means it's going to be in both data centres before the publisher is acknowledged his message is guaranteed.

But obviously throughput is going to be a problem because if the distance between data centres is quite far, now you have...

Your overall throughput is going to be less.

Versus asynchronous, we'll just enqueue it to be delivered to the other data centre before when the publisher gets an act, much more performant.

But if that data centre crashes right when that buffer hasn't gone out the door, then you might lose some data, right?

So everything is about...

Trade-offs.

Well, trade-offs.

It's all...

And that's what patterns is all about, right?

Yeah.

And understanding the context in which we're doing those trade-offs because we're doing those trade-offs either way.

But if we don't get that context right, we might misjudge the trade-offs.

Yeah, yeah, yeah, yeah.

And that's problematic.

Yeah.

What Stefan said there is pretty...

That's pretty accurate there.

IT and business.

Yeah.

Yeah.

Trade-offs.

Trade-offs.

Yeah.

Fast or secure?

Fast or secure?

No way around it.

That's always going to be a discussion.

It's a very difficult topic.

I find it that one of the best explanations I've seen so far how to work with trade-offs is doing the trade-off analysis at the way that Mark Richards and Neil Ford portray it with a discussion around which characteristic is actually the driving characteristic for the business.

Okay.

So you're putting priority on those characteristics.

So we prioritise them in a way that, well, okay, if we're talking about that, what is the most important?

What is now the thing that is mandatory?

Because if we ask a business, do you want performance, security, availability?

We want it all.

Always.

There's no way around it.

They're always going to say, yes, we want everything.

But it's impossible because as architects, we cannot ever design the best solution.

We always design the least worst.

Ultimately, the technology should enable you to do what the business wants to do.

At the end of it, it's almost always a business decision to some degree.

And our synchronous replication, we built for a specific customer that they do horse racing.

And they were like, it has to be like, it's mandated that this, they can never lose a ticket, right?

Every time you purchase a ticket, we send it to both data centres and it's perfectly guaranteed.

It won't get lost.

And this was all fine and great until they had a network outage between their two data centres.

Everything was fine, but this one link that they used to communicate between them was down.

And basically that meant all of their ticket purchasing had to stop because they couldn't replicate it to both sides.

And it was like that for about half an hour.

And then the business came along and said, all right, you know what?

We're going to take the risk that we're not going to lose this data on the primary side.

We're going to allow it to these tickets to be purchased, even though we can't replicate it and have.

So sometimes you think, you know, all best intentions and everything.

And then when actually things hit the fan, you're like, I guess we don't know.

We don't need fully asynchronous in the worst case scenario.

So yeah, it's always, you never know how things are going to go sometimes.

And yeah, it's a business.

All right.

I see we have a question popping about Gen AI and the agents.

We'll dive into that somewhere around the end of the stream.

Let's dive back into publish, subscribe.

So sadly, we're going to go off the hook on Gen AI and agents by the end.

Hope you stick with us and we'll touch upon that later.

But publish, subscribe.

Adam, you have a game for us.

Oh, I was going to show the demo.

I can do it.

We can do the game.

Okay.

Do the demo first and then we're going to do the game.

You ready with the demo?

No, no.

Give me a second here.

So this is, yeah, just let it load.

This is, I like to write demos.

I write a lot of demos.

I'm a Java programmer, but I dabble in JavaScript because why not?

All right.

Go ahead and share it there.

I hope this is big enough.

So there we go.

So this is actually, I mean, this is not real data, but this is representative of Solace's very first IoT use case.

This is Singapore.

This is the island of Singapore, pretty much a hundred miles north of the equator, right down at the Malaysian Peninsula at the tip.

And these are buses.

These are buses driving around.

If you zoom in here, you can actually see their buses.

If you keep zooming in, I got a nice satellite map.

These aren't, again, this is simulated data.

This is not real, but this is meant to kind of show and visualise what publish-subscribe is.

So this is a bit of a, just a kind of a learning kind of tool.

Down at the bottom, you can, so every time one of these buses moves, what we're looking at here is a Google Maps API, JavaScript, and the Solace JavaScript API.

So it's kind of a mashup.

And my browser here is a client connecting into Solace Broker that's running in Singapore and is subscribed to a bunch of bus topics, all right?

And so as these buses move, again, they're not real buses.

It's a Java publisher just simulating this data.

Every time a bus moves, it publishes a message to say, hey, my GPS position has been updated.

Here's my new position.

And my map here is subscribed to these bus topics.

So as these buses move, the map gets this message data and I just move the icon, right?

That's essentially all I'm doing here.

So this is exactly publish, subscribe visually.

So buses are updating.

My map is not polling for these locations, right?

I'm not querying some database saying, where are you now?

It's basically getting jammed into me through an event handler.

Like here's your message.

All right, what is the message?

Which bus is this?

Okay, what are the coordinates?

You can see at the very, very, very, very bottom, this is the subscription that I'm actually listening on right now.

So as I mentioned earlier, Solace has wildcards on different levels.

So you can be very coarse grained.

In this case, I'm subscribed to the whole feed.

Stars everywhere, right?

All stars.

But you can see the, what we call the topic hierarchy just on the top of it there.

Route number is level four.

Then a bus ID, level five.

Latitude and longitude I actually have is level six and seven.

A direction indicator of some sort.

And then a status is the last level.

So I've built this rather complex, fairly complex, I don't know, topic hierarchy, multi-level.

But what that allows me to do is I can now subscribe to parts of this feed, right?

As we talked earlier about the different kind of filtering and capabilities that are built in.

If I wanted to listen to say, just route number seven, I could go up here.

I could update my subscription.

You see my subscription here at the very bottom is now changed to 007 with a star because we support prefixes for our star.

So 07A, B, C, all the other buses go away because I'm not getting them anymore.

I'm not getting updates for these things anymore.

The broker is doing the filtering.

Notice the message rate here.

The rate that I'm receiving messages from my broker is dropped way, way down.

So the broker is filtering this stuff, right?

I'm not listening to the full thousand buses or 500 buses and throwing away all this stuff I don't want.

The broker is doing it.

So it saves me network bandwidth, CPU.

It's secure.

There's a lot of good reasons why you want to do filtering.

Let me put a bit of a context because you showed the topic taxonomy.

It's easy for people using REST to understand because that basically resembles a REST URI path, which is absolutely brilliant for people to learn because it makes life a lot easier for them.

But basically if we're contextualised to that to other protocols, this data that we have as variables in the topic name, so route, bus ID, latitude, longitude, direction, and status, this would be the equivalent of, let's say, JMS headers in JMS as metadata.

This would be the equivalent of routing keys in RabbitMQ implementation of AMQP, etcétera, etcétera.

So this is your metadata.

Of course, you can have also that, the same information in your payload.

It would be.

It would be in your payload because you want to read that payload.

And well, if you zoom in a little because it's a little bit small.

So you see that's already there somewhere.

But basically when you're publishing that message from a perspective of the producer of the event, you duplicate that data.

Part of it goes into all of this.

It goes into the payload.

And then some of it goes as, is treated as metadata and goes into the topic name, which creates that lovely taxonomy that we can filter by.

The same would be with JMS. You would have to then publish that with the message as JMS headers properly describing that data.

Then we could do the same level of filtering by JMS headers and select this.

But it would be a lot more difficult to do because it's just tricky.

Of course, JMS also enabled us to do dynamic queues, etcétera, which would also enable that kind of wildcard subscription.

Again, a little bit harder to achieve than as it is implemented in Solace.

But it's not the same capability there.

It's just that people didn't use that that often.

Yep.

I mean, you can do that with, I mean, Solace is a full-blown JMS broker as well, right?

So, I mean, you can do that kind of stuff.

It's just our topic matching is infinitely faster than using selectors.

Selectors are because you have to go into the message.

I mean, it's part of the message.

It's not maybe it's metadata, but you don't have to look at the whole payload.

But yeah, you have to go and look at this key value pair and analyse it and stuff.

Whereas this is just if you can build...

Selectors also, you know, it supports SQL 92 type syntax.

So you can do more complicated.

You can do greater than, you can do less than, you know, you can do ranges.

You know, this is just basic string matching.

Topics are simple, fast, maybe not exactly what you, you know, maybe if you need something more complicated, like if you do XPath stuff, then no, that's not...

Topics aren't going to work, right?

But I do want to say, like people might look at this and say, why is the latitude longitude in the topic string?

That seems unnecessary, right?

I might want to search by route.

I might want to filter for a specific bus or maybe the buses that are broken down.

You know, some of them are, you can see they have little red Xs on them down here.

So like those are like, this guy has a fault.

Nope, not that one, this guy.

Yeah, anyway.

But yeah, I came up with this idea of like, can we use topics to do kind of geo filtering or numerical filtering?

And if I watch this, if I add a square here, add a polygon, you'll notice that the bottom, my subscriptions now change to like 1.3 star, 103.8 star.

All the buses that match that means that their latitude and longitude are in this area.

And so, I mean, obviously it'll match ABC as well, right?

I mean, you have to kind of make sure that you're publishing actual floats and they look nice like numbers, but I can now filter by location just with the topic.

I wrote this as a paper, I presented it at a conference and I don't think any of our customers ever use this.

I mean, it's just the topic, right?

It's a topic.

I was hoping UPS or someone would be like, hey, that's cool.

I can now filter based on area.

Every one of these little orange rectangles, squares is a topic in Solace.

It's a separate subscription.

It's a separate subscription.

So yeah, so right now there you can see I have there's 300, I capped it at 300.

So to approximate this shape, I can use topics and subscriptions to approximately filter to this area.

So if you just want to know when a boat comes into a particular part of the harbour, you could do that with topics, right?

You don't have to use ArcGIS or something from Esri.

Yeah, so basically this kind of level of filtering on an operational basis, real time basis, enables you to really build robust observability, really specific observability tools for whatever.

It doesn't have to be observability in the sense of IT ecosystems and what's happening in your systems.

But for example, for logistics, for traffic, yeah, it's amazing.

If I look at the apps I ever used on my phone, this kind of filtering would be something that, let's say Uber or any other taxi app to just find taxis in my area.

Tell them to call me.

I'll give it to, I'll tell them how to do it.

But also if you look at, for example, dating apps, a lot of them have, like in what range from your current location you're looking for potential dates, right?

Yeah.

Tinder had that definitely.

I remember that kind of setting somewhere back in the day when I was using Tinder.

There was that, in the range of 10 kilometres from me.

So that's basically, that would basically be a subscription to that kind of a topic.

Kind of, but the one thing I'll make it just like this is streaming data, right?

These buses are continually moving.

So they're continually publishing their location.

And so I'm just doing it with this topic.

If you are not constantly publishing your location, you're going to need a database to store that so that we can come along and query for that area later, right?

So for this particular use case, it works fine because they're constantly publishing where they are.

But for something like Tinder, you're probably like, I'm in this area now and then tomorrow I might update it.

But you know, it's, you're going to want it.

It's not that fast changing data.

It's more of the use case that Stefan is commenting here on.

So for example, live tracking of our phones, our service providers for mobile phones, they basically can track SIM and email because those are registered metadata in the network.

Or Bluetooth.

Or Bluetooth.

Or your Wi-Fi, MAC address.

I mean, that's, that's happening.

Yes.

But basically if we're looking at our phones, they have to register with a base transmission station, right?

So old school wise, you had triangulation, which means that you had those triangulate your position versus the strength of the signal between three different transmission stations.

While this would be like tracking real time because you subscribe to the data from the transmission stations and you can really triangulate that position of that phone in real time.

So that's actually another really great use case for leveraging that data in telco and real time leveraging, which could be, for example, used by police.

So one of my colleagues, one of my colleagues in Italy actually just did a POC with the Rome airport and it was around trying to like track people how they move through the airport, like in case there's any bottlenecks and they were using their Wi-Fi, they're using their Wi-Fi as they went from base station to base station to get an idea.

Like it's not perfect, you know, it's like, but you know, plus or minus, you get an idea of where people are and like the flows and they observe how people flow through the airport in order to do efficiency stuff.

That could be useful for retail people, maybe, you know, like it could be there's, yeah, I've heard of retail stores that will like mine this location data to lay out their store in a particular way that causes people to spend the most money, right?

Like they set it up so that when you browse around, I don't know, I'm not a retail person, but there's all sorts of tracky.

Yeah, but using that location data, Google Maps, the most basic use case of real-time location data.

There was a, even a episode in Germany, I'm not going to find it right now because I don't even know the keywords for that, but there was a artist, like an event artist in Germany that just put on 10, 100 active phones in a basket and moved around the street with them on foot.

Basically what happened, he was simulating traffic walking with 100 phones at a pace of a pedestrian.

So the streets were...

So Google Maps is like showing this red block wherever he was.

Yeah, so the streets around him were literally empty, devoid of traffic because he emulated the nav system that there's actually heavy traffic there and there's a traffic jam.

So everybody routed around it.

And that's basically real-time use case of such data and filtering of such data.

But then again, if you provide proper anomaly detection to such data, you would then discover that this is not distinct data, but then it depends on the triangulation, the actual distinction of where that phone is.

If we don't have that distinction in place, then obviously this kind of tricks like the one here I told you here is possible.

But it's again, security of our data and the misuse of the software in essence.

Anyway, so that's my bus demo.

If people want to check it out, that's the URL at the top there.

It's running all the time.

You can send it to me.

I'll put it in a QR in a second.

Just drop it in the private chat.

I'll drop it in.

Do we want to talk about something else or should we do the interactive one now?

Oh yeah, let's do the interactive one.

So I'm putting the bus demo in a QR.

So for those of you who want to look at the bus demo done by Adam just now, you can play around with it.

It's available online.

There you go.

There's a QR for that.

So scan it, play around, have fun.

That's a very fun demo.

I played around with it myself.

It's a very interesting use case.

It's visual.

I mean, people like maps, right?

I mean, explaining a networking technology like Solace, like a backend thing is not the most easy.

Hey, look, it's a message and a payload going across the network.

How do you visualise that?

But people like maps.

And so I thought, yeah, let's make a cool...

These were originally taxis.

This started off as the London taxi demo because that's where I first came up with this geo algorithm.

And now it's the Singapore bus demo.

So quite cool.

All right.

So let me boot this up here.

Let's see.

All right.

Nobody look for a second.

A loosely coupled race.

There we go.

That's all right.

I'm just like, is there an...

I can't...

Loosely.

Is there an E in loosely?

How do you spell it?

L-O-O-S-E-L-Y.

There we go.

All right.

So run that.

There we go.

All right.

Let's turn it into a little bit of a bigger screen.

So for all of you watching, let's say grab a phone.

I'm grabbing mine right now.

I'll grab mine too.

Sure, why not?

Go into...

Big prizes to be won.

So whatever scanning app you have for QR codes.

There we go.

This is another lovely example of doing a PubSub or public subscribe.

Little PubSub, yeah.

Yep.

Okay.

I'm on the pink team.

I wonder if it's just going to be the two of us or if we'll have some other...

We'll see.

Maybe at least Stefan is still watching and maybe he'll scan it.

Yeah.

So we'll give it a second.

So...

Oh, we have a yellow team.

Yeah, there's somebody else.

I'm on the green.

Oh, we got four.

Oh, nice.

All right.

Woo.

Okay.

We need at least four.

We need at least four because there are four teams.

So this is good.

We'll see if a few...

We'll give it a few more seconds.

I'll explain what you're going to see here in a second.

Basically, there's going to be a little countdown.

I'm going to start the race.

There's going to be a little countdown.

When it says go, just tap your phone.

Just tap anywhere on the background, like anywhere on the screen.

And every time you tap, you row your boat, right?

Publishing messages.

And the winner is going to win a big prize from Carol.

Yeah, right.

Like I have the budget.

We can agree on a private sparring session.

Sure.

All right.

Well, if there's more people out there that are not participating, at least you can watch this amazing race.

Again, this is one of my very first JavaScript demos.

And so it's not that good.

I should probably tidy it up a bit.

All right.

Are we ready?

We're going to do this race.

Some event-driven racing.

All right.

Here, let's do this thing.

Ooh.

All right.

Ready, set, go.

Go, go, go, go, go.

Holy hell.

Carol.

What the heck?

Oh, six fingers.

Six fingers.

I think you've done this before.

Have we done this?

We must have done this before.

Yeah, yeah, we've done.

Yeah, see.

You have to play like someone who doesn't know all the tricks.

Anyway, thanks.

Should we restart?

I'll give you a chance.

That's right.

So this is a silly little demo that we use in some of our user groups to get people.

I use it in my training, actually, to kind of get people excited a little bit.

It actually uses Solace messaging underneath.

Actually, it's not.

It's MQTT.

It's connecting to a Solace broker.

But every time you tap, it's publishing a message, right?

It's a row message.

And because this is a bidirectional WebSocket communication, it's very asynchronous.

It's very fast.

You can stream lots and lots, lots of clicks, lots of rows, you know, without having to, unlike REST, the whole point of this demo is to kind of say, how would you implement this using REST?

And if you had any amount of latency, you can see down in the bottom left corner there.

Actually, you can't because my name's in the way.

There's a latency counter.

You know, if you had a hundred, if you have 50 milliseconds of latency between you and your broker or your server, you could at most do 20, you know, events or round trips per second.

So you have to use something streaming there.

So do you want to do one more?

I don't know.

People might be bummed out and they got beat so bad.

But I promise I'll use one hand this time.

All right.

All right.

All right.

I'll just hit refresh here then.

We'll do it one more time.

There we go.

Now it's going to be a dev jam.

All right.

There we go.

You have to reload.

You have to refresh your app.

The URL hasn't changed.

Yeah, I already closed the browser.

So I just need to go.

OK, I'm on the pink team again.

Again, the pink.

I'm on the green team one more time.

So, yeah.

All right.

All right.

We'll see if our other two competitors are still out there.

Or if they're chicken now.

So I'll give it a second while they join.

But yeah, so using, because it's bidirectional, you know, I mean, the server, me, the racer, the controller can send the start message to all the players.

The players, when they want to join the race, you know, they can send, I'm listening on a well, what's called a well-known topic, you know, just a topic that says, hey, I'm offering races and you publish on race on that topic and I can hear it.

So even if there are lots of other races playing right now, you publish one message, like as a service discovery message, say, who are the races out there?

All of the races that are listening on that can then reply.

It's like, it's like request reply, but multi request reply.

You know what I mean?

So using public subscribe, you can really build these complex kind of cool.

It's basically request reply built over a asynchronous protocol.

Yeah, but also one to many.

It's also one to many, right?

It's like, it's like I publish one message saying who's out there.

That goes to like five different races.

They all respond saying here I am.

And then you can then choose which one you want to actually join.

And now we're joining this one.

Wow, we got six players.

Unless someone's refreshing their screen because it doesn't.

It's not smart enough to do that, but when the race starts, I do get rid of.

I promise I'm going to be using one hand only.

My everybody remain in the air.

As many fingers, as many taps as you can.

Are you ready?

Seven, seven.

Let's go.

All right.

All right.

Let's see if there's going to be eight.

Let's see if there's going to be eight.

And I do count like if one team has two people, you have to do twice as many clicks.

So it is fair.

All right, here we go.

Are you ready?

It's the only force that they're ready.

All right.

I'm going to go full speed this time.

Go, go, go, go, go.

Oh, yes.

That's pretty close.

I thought orange was whoever orange is.

All right.

Orange, I think, should actually win that prize because it's my game.

Let's give the prize to orange, whoever orange is out there.

Anyway, that's my silly demo.

But yeah, this is like, it's meant to show when you start thinking about the patterns, right?

Like, I want to join a race.

I want to subscribe to a race.

All right.

Get ready.

The race is starting.

You know, if this was using REST, which is client server, it's not server to client.

Like when the race is ready to start, how does the server tell all the clients that the race is starting?

You can't.

You have to pull.

Like the clients are just constantly pulling, saying, hey, has it started?

Not very efficient.

So yeah.

And that is actually, that has been studied.

I believe, who was it?

I think Zapier studied that back in 2013 or something.

That's quite an old study.

They studied polling mechanisms and the results of polling.

And they discovered that over 98% of polling results in no data.

I heard a similar metric.

I heard it.

I heard somewhere.

Yeah.

It's in your, in Solace Dex.

I seen Sean McAllister actually present that, exactly that metric and that study on a Solace user group.

But it's a study.

Okay.

So, okay.

All right.

That's, if it's something referenceable, then I'm down with it.

You know, anecdotal, I don't know.

I believe that as far as I did my research in that, that's a study done in 2013.

So this is a fairly old one, but they studied that.

I don't think anybody did these kinds of studies anywhere later, which is a shame.

I think this would be a topic worth studying, but it is what it is.

Put this on, put this on the screen for a second.

This is, this is the Simpsons equivalent of it.

It's just like, are we there yet?

Has something changed?

Has something changed?

No.

Has something changed?

No.

Has something changed?

No.

No.

How about I just tell you when something has changed?

You know, like this is polling.

This is polling when, you know, if you want it to be responsive, you have to pull fast, right?

You could pull every hour and then there's probably going to be some change, but now you're an hour out of date.

So you pull really fast and now you're just burning your network and just backend database saying, nothing, nothing, nothing, nothing's changed.

Yeah.

And see, that was, that could probably be a feasible mechanism 20 years ago.

Yeah.

If we're considering our data centres where companies have their own data centres at 20 years back, they didn't have cloud available.

They were hosting everything on premise.

That was a scenario where storing data was horrendously expensive.

Yeah.

But moving data was absolutely, they're cheap.

Well, now we have the other way around.

Networks were slow.

Disc was faster, right?

And so, yeah, the database as essential of your IT as the storage is the golden source.

That was the way almost all of these things were architected.

And now the network is fast.

Now the network goes like faster.

Like now the state is in the network and the database is just a hang.

It hangs off the side and it keeps that.

But the real truth is the network, not the database anymore.

Yeah.

So you want to tap into that.

But see now, if you consider cost-wise from a perspective of infrastructure, where most companies are now investing into that infrastructure being in cloud, they drop their own on-premise in favour of cloud.

Now storing data in cloud is dirt cheap.

Moving data in cloud is horrendously expensive.

Well, some of them make it cheaper than others, but yeah.

But in comparison, 20 years ago and now, moving data became a lot more expensive and storing data became a lot cheaper.

So if we're considering a polling mechanism and we want that polling to be near real time.

So we set the polling intervals, for example, every one minute.

And we poll every one minute.

Real time.

No, it's not real time.

That's near real time.

Real time is never achievable with polling.

Sure.

Ever.

And so if we want to be near real time, we set that polling interval to one minute and then we start polling.

But then we result in a lot of calls that are completely unnecessary that burn our budget because we're making those calls.

We're getting a response.

That response is empty, but we're already burning the money on getting that call in response, which means it's a problem.

It's a problem from infrastructure perspective.

It's a problem from a budgeting perspective.

It's just not feasible to do polling mechanisms anymore.

Polling mechanisms right now should be treated basically as a last resort mechanisms where we have them for legacy systems that cannot release events.

Otherwise, it's just a bad idea in cloud environments.

I'm going to flip that around just a little bit to say, also, you're now up to a minute late for whatever action you're trying to action on.

If you're polling every minute, even if a lot of things change that minute, maybe it's fine to pull every minute.

You want to pull at a rate that you're not getting no ops all the time because that's dumb.

Even if you're down to five seconds, now you're still five seconds possibly late.

The start of my race, there was only a five-second countdown, so you have to pull at least every two or three seconds to actually make sure you start the race on time.

So speaking of polling, what would you consider the best method to implement event-driven PubSub on relational DB containing the source data?

What implementations do you use?

And this is exactly the problem we just put in, the near real-time polling, which is like, why would we do that?

If we're having relational databases, that must be a very old relational database that we have to resort to polling.

As far as I remember, in telco environment, we resorted to polling with databases.

At that time, we used Oracle.

Yeah.

With Oracle in a version lower than 10.

Okay.

I don't know Oracle stuff, so yeah, that doesn't mean anything to me, but all right.

They didn't have the mechanisms for the database to make a proper call.

Yeah.

Right?

So if you had database-based systems and we communicated directly with those databases, fair enough that they were built properly so there was a proper segmentation of the database.

From the integration perspective, we didn't actually touch the production tables and whatnot.

There was always somewhere in staging before that data was actually released to the live production tables.

But basically, those systems, those very old school systems, before they were modernised to higher versions of Oracle databases, like 11, 12, they were incapacitated in their capability to communicate, to initiate communication.

So we had to resort to polling mechanisms.

But the moment that those databases gained proper libraries to actually initiate communication, we had database triggers that resulted emulated events based on whatever the data type was in the table or the procedure.

And basically, if we're talking about polling and turning into PubSub, well, replace your polling mechanism by using triggers in a database.

The answer, like if you're an Oracle user, the answer is GoldenGate, right?

Or some other kind of, it's change data capture.

If your database supports

change data capture,

there's like,

MSSQL does,

I mean, a number of them do,

where, and even if not that,

the log, logging database logs

as events are,

as rows are updated,

as tables are updated,

these are emitted as change data,

like they're emitted as events,

triggers,

I don't know the proper terminology,

but Oracle GoldenGate

was meant to allow you

to keep multiple databases in sync

by as changes are made over here,

these are emitted as events,

as triggers, as updates

to another database,

which can run over GMS.

I mean, that's actually

a really old use case

that saw us like 12 years ago.

When I first got hired, it was a GoldenGate replication use case using a Solace event broker to replicate customer data to like five different data centres, right?

So it was like, again, PubSub, one change in, and it went to five, four other data centres.

So yeah, change data capture, checkout, I mean, yeah, this kind of replication, on that, yeah, that kind of replication to multiple data centres to have that copy in all data centres wouldn't be possible by DB links.

It's not even, yeah, it's not even replication, but there's just, if you want to be event driven with your data and your database is your golden source, you want to initiate some kind of change to the capture mechanism to emit those as events that you can now tap into.

You can listen to say, hey, this has been updated.

Boom.

And you get that message right away.

Yeah.

So either database logging or changing the capture would be my answer on that.

So absolutely.

And Solace has some things around this.

We have a Debezium, like as part of our integration, micro integrations, we do integrate with Debezium.

So some databases are supported through that.

I don't know.

I'm not a micro integration expert, but if you're looking for a Solace answer, maybe check out our integration hub.

But I'm sure there's lots of other stuff out there to do.

There's plenty, plenty of solutions for change data capture and plenty of databases now they support these mechanisms.

Anyway, Kafka has all this stuff as well.

You know, I mean, it's not uncommon.

So I'm sure you'll find some.

It's just a matter of understanding the use case is that actually an operational event or is that something you need for like analytics?

Yeah, right.

Because these are operational events are very different than events for analytics or for audit security, etcétera.

And you have to consider the temporal coupling aspect of that.

How is that data going to be utilised in time?

And then you can figure out which kind of mechanism you need to actually liberate that data from the database, because sometimes that would be actually putting that in a log broker and just being able to replay that from that log instead of using operational broker with or without persistence doesn't matter, but with that robust filtering capability like Solace provides, right?

So it's all about understanding the context and then doing proper trade-off analysis.

What would be the least worst fit?

Yeah, I've been looped into, I don't know why me, but looped into a potential POC proof of concept with the customer of ours that they're doing some mainframe stuff.

They have mainframe databases that they want to get data out of these.

And so their existing solution is JDBC.

They pull their database.

I don't know if it's a rational database.

I don't know anything about mainframe, but it's some type of database and they have JDBC apps that are pulling this database to get change and stuff out of it.

And they're trying to figure out a better way to do it, right?

And so change data capture is not really an option, but they can enable database logging for some of these tables as they get updated.

However, the logs don't contain all of the data.

So we're thinking maybe we can use the logs as a trigger to then kick off some polling, right?

So you don't, you pull only when something happens rather than pulling by time.

There's maybe an interesting hybrid type of solution.

That's actually an EDA pattern.

Oh, good.

That would basically land within the pattern of anaemic events.

Oh yeah, thin events.

Sure.

Yeah, yeah, yeah.

Something's happened.

I'm not going to tell you exactly what's happened.

Yeah, yeah, yeah.

Of course you're right.

Yeah, yeah, yeah.

That's anaemic events versus thin events.

Yeah, yeah, exactly.

So you don't have the context of all the data that happened within whatever happened, but it's a notification.

So if you have that anaemic event that at least points you to the right data, then you can make that smart polling of data to pull that exact record instead of having to pull all records or by flag or whatever every whatever the interval might be.

And that's another viable solution for these kind of legacy systems that if you can find a trigger, some sort of a data like a log that could be utilised as an event beginning the communication, then you can utilise a polling mechanism that is based on those anaemic events.

That is a fair use of that pattern.

While a lot of people consider anaemic events to be an anti-pattern, it's a very, very useful pattern in these kind of niche situations.

Things are always applicable somewhere, right?

I mean, in general, anti-pattern, sure, but you can always find one thing where it's like, actually, that's a good idea to do it like that.

It's a difficult, in general, the discussion around patterns and anti-patterns is quite a difficult topic on its own.

Don't have time for that today, that's for sure.

But if you're not going to jump into the distinctions between patterns and anti-patterns, but...

I thought we were going to talk about AI.

We will, we will.

But if anybody is interested in looking into patterns and anti-patterns, I actually wrote an article about the distinguishing and trying to define what is actually a pattern or an anti-pattern.

Because in my research, I scoured through several books that mentioned anti-patterns and there was no cohesive definition of what an anti-pattern is.

So if somebody's interested in my take on what an anti-pattern is or how to address...

It's your article.

It's my article, exactly.

Yes, I wrote it.

So this is a piece I wrote some time ago while defining EDA patterns for the research I was doing, because I stumbled into, okay, how do I work with anaemic events or with content-based routing in EDA where people would consider that anti-patterns not to do ever.

But that problem is not that kind of a binary problem, which means that there's a lot more to it.

So it's always worth to jump into it, figure it out, try to understand the context.

And the context is always the most difficult aspect there is.

So if that helps, jump in, read it, ping me on LinkedIn later.

What do you guys think about that article?

Do you agree with my definition of an anti-pattern or a pattern in general?

Let's have a discussion about that because that's always an interesting topic to spar on.

I'm happy to jump into that outside of this live stream.

How does it know that someone has scanned the QR code?

I don't see any tags or anything.

It knows because the streaming service we're using to chat today is proxying the QR code.

So they're actually wiretapping on the QR code.

So they're actually gathering metadata that somebody accessed and then redirecting to the right place.

So we know you scan.

I will read that later.

So yeah, let's jump into, hopefully Sadi is still with us.

If not, I hope you'll watch the recording and the answer to his question.

So exception handling and identifying patterns, fantastic.

What, oh, the whole comment doesn't really fit into...

But luckily that's not a, it's just one word that's been cut out.

So thoughts about AI and generative agentic in the world of integration.

I have my own thoughts and I know you from the perspective of Solace, you have your own thoughts because you actually productize that to the open source project.

So shoot, let's have the EDA perspective and Solace perspective on...

Solace perspective, yeah.

Shoot, man.

So I guess I'll just, I could summarise it in two kind of phrases is, one, agentic AI, agentic AI just means you have a bunch of specialist agents, right?

Like not one agent is going to be the Uber agent that does everything, like building graphs and talking to the weather and looking at files and doing all this stuff, right?

So you have specialist agents, they need to talk to each other.

And so like probably in these types of systems, 20% of it is actually the AI, maybe the LLMs and all that stuff.

80% of it is the integration.

It's the plumbing.

It's the plumbing between these things and how to make them work together.

And we've had a number of calls of with various people in this space and they all kind of agree.

That's some say 90%, 10%, and 90% Solace's answer is 20 and 80.

But you know, integration in these agentic systems is significant and important.

And the way that some of these companies are building these things, it's so reminiscent of kind of microservice distributed architectures where you have a bunch of specialist microservices that are doing their one little thing and they're doing point to point communications to all their other little guys.

And as a Solace person or as a Solace as a company, we've seen this before and we've been trying to fix this with publish, subscribe, event driven, asynchronous communication and it's the exact same thing.

So yeah.

So that would resemble in that sense if we're talking about that microservice, pure microservice communication in agentic world that would be using the A2A protocol.

We've wrapped A2A with messaging.

So it's still at the payload is A2A but we're now using Solace and Topics to move this data around between various agents very like orchestrators and like you can now you can wiretap it.

We have all sorts of observability tools now because it's going through the broker.

You just subscribe to those A2A messages as they're flying through the broker and you can have a real time display.

Like we have demos of this stuff where you can see the different agents talking to each other and what they're doing and like where your query is.

It's anyway.

So Solace is building an agentic AI framework that you can plug in any MCP server to it and it's, you know, using Solace as its communication framework because we feel that that's where things are going.

It is you need to be scalable.

Oh, thank you.

To the liberty of popping the main page for that that's called the Solace Agent Bash.

So if you're interested in the topic how that works, it's exactly that and it's an open source.

It's not a product.

I don't think we're open sourcing the whole thing.

There is some like secret sauces in there, but it is significant.

It's all on GitHub.

So if you want to look it up, that's the QR code for the GitHub.

Play around with it.

See if you can use it.

That's definitely something to use for agentic environments, but that doesn't really fully answer the question.

No, no, that's not, yeah.

Pose there.

And I need to look it up in my Google Drive, but because I was actually interviewing with Solace a little bit ago, I had to prepare.

Didn't take it, you know.

Didn't take it.

Didn't take it.

It was a very, very fine offer and I would definitely, if I was 10 years younger and didn't have kids, that would be definitely something amazing to do.

I just need to find the right deck.

I actually had to prepare the deck on that because I was asked to present a Solace agent badge and what would that be.

So I tried to approach that from a perspective a little bit more agnostic.

While you're digging up your presentation there, I will give a shout out to my colleague, Tamimi, who actually made a nice little video series on our Solace YouTube channel, kind of explaining the, and they're nice little five minute videos, little bite size.

And he explains the kind of the Solace agent mesh, the different components of it and why we feel that kind of agentic AI needs to look at, you know, EDA and publish subscribe as a- Yeah, so actually, I actually used those to prepare this deck.

They're great videos.

I watched them all.

They really explain the Solace agent mesh and the principles about meshing agents great.

So watch them.

And now in terms of the shift in terms of interoperability and integration, so we have our traditional IT ecosystem set up.

We have our systems, our data and databases.

We have integration platforms and message brokers aligned with those.

We have API management.

That's the traditional IT setup, right?

In terms of interoperability.

That's that box, right?

So traditional IT ecosystem.

And the way I basically see it, then on top of that, we have all sorts of LLMs, generative AI.

We have all sorts of agents doing various things with or without the access to those LLMs. And this is basically where we come into this agentic ecosystem.

Now, these ecosystems are next to each other, but one relies on the other one.

The agentic ecosystem is basically useless without, right?

I don't really remember what's there.

So we have model context protocol and A2I.

A2I is between agents.

MCP is the boundary between the traditional IT ecosystem and the agents.

So if we're looking at the traditional integration or a traditional application integration, we still very much need it because that's our lifeline to actual data.

That's our lifeline to actual functions.

That's our lifeline to actual governance over what that agent can do in our IT ecosystems and in our systems.

And this is where the MCP really comes to play, where now a lot of integration platforms enabled to just add MCP on top of their APIs.

So instead of exposing a REST endpoint, we expose an MCP endpoint, MCP server as an endpoint.

And this is the shift.

It's still the same integration, just a different protocol.

And the MCP server then basically becomes that different of a protocol in the sense that it actually requires that state.

So that's the differentiator.

So we're still gonna have plenty of work as integrators.

That's exactly that 80% is that plumbing, thought through, understood so that we know which agent has access to what data.

While we can set up- Authorisation, authentication.

We can set up this as MCP servers solely because if you look at MCP server, what it is, it's a micro integration platform, a broker type integration platform.

But it's something that I'm working on in terms of properly describing.

But basically the analogue is, if you look at MCP from a perspective of the API led architecture, so that layered architecture where you have the channel layer that is the representation of what that system or what that agent has access to, then basically the MCP server becomes one application and channel layer.

And this is interesting because then you can apply a lot of governance over the mains, over the access to proper services and data within your ecosystem by governing what that one channel has access to.

And then you can host however much MCP servers you want, which with all of them having precisely fine tuned access to services that you already have.

So if a company already has a API led architecture platform in place, easy peasy.

That's just exposing that as another endpoint as an MCP.

And already those technologies are catching up and starting exposing, adding MCP as just another protocol in the portfolio of protocols you can expose from their applications.

And that makes it all interesting, very interesting, because that's a very large impact on the architecture of integration platforms.

And on top of that, if you add the communication, the agentic communication, which is A2A or A2A wrapped with async, like done by Solace.

Well, the most of the use cases we're seeing these days from customers that are interested is kind of like this talk to your data type use cases, right?

Where you might have agents for Salesforce for your problem, your JIRAs, maybe your trouble ticket, your orders.

And so like if you're a customer facing person, you like get a support call.

You could talk to your, you could basically say, hey, chatbot, tell me all the, tell me all the things that you need to know and give me an update on this particular customer.

And it'll pull all this data from disparate places and then format it into a nice HTML report or some PowerPoint slides and present it to you.

Here's boom, just like that.

Having that kind of, I mean, that's from the usability perspective, like in terms of how do you actually go and architect that and like kind of build all this.

There's a lot of integration.

There's, I mean, it's, yeah.

Yes, the agent basically has to access a lot of sources of data to compose, but this access needs to be use case specific.

Otherwise it's going to be just hallucination on top of hallucination because it's already been, Antropic already tested that in terms of the extent of MCP and how you can build MCP is that the more sources you add to the MCP server, the more the agent will hallucinate, the more the model will hallucinate because there's too many sources in that.

I think some of our AI, so Solace kind of made a dedicated AI team about two years ago to kind of start down this path.

So we've been thinking about this for a while, even though we're a message broker, event-driven kind of company, we do have people that were very keen to get on this and had been dabbling in AI for probably like five years, our chief AI guy.

However, the one thing I was going to say that you just key me to, they, I remember some of our slides talk about in order to prevent hallucinations, it's about providing the right amount of context to certain servers.

Like you don't want to overload them and you have to give them enough.

And so as part of that, I think a lot of stuff has gone into, in our implementation, we have this orchestrator kind of mega agent that basically you can scale it out.

So it's not really a bottleneck, but it is the one who is taking down your complex human query and figuring out how to break that down for the different agents based on their capabilities, which are advertised by these agent cards, you know, and providing that agent with the right amount of context to not hallucinate.

So I'm guessing they've probably gone down this path of, I want access.

Yeah, yeah.

Sometimes it's pretty, pretty funny.

Not when you're trying to get like, you know, writing documentation and it's like just making up crazy stuff.

You're like, I almost can't do that.

Like you're crazy chatbot, but.

And the same applies to just using generative AI.

I have a licence for Google workspace.

So I work on a daily basis with Gemini within my workspace.

So I often test the extent how those hallucinations occur.

So I figured one day I'm going to experiment with a Gemini based agent and just hook up all the data I downloaded from LinkedIn.

That's available as my data for my activity, my profile, etcétera.

And I just hooked up willy nilly all the data to that chatbot agent.

And I also hooked up my CD that I made myself and started asking that agent questions about me.

The results were astounding.

I was like, where do you get that information from?

There's nothing that would indicate that I studied marketing.

All right.

So work in progress, eh?

Yeah, it's probably going to be changing over the years.

Those models will be handling multiple data sources better.

But the more you hook up now, it's the more hallucinations.

So if we're talking integration, those integrations will have to be very much fine grained integrations with a very specific purpose per agent.

So the agents will also have to be very purpose related.

So they cannot work as very generalistic software.

So yeah, customer support agents, absolutely fine for somebody in customer support to use and pull that data and understand that data in that small specific context of a use case.

For example, reconciling problems with an order in a retail environment or something like that.

Or understanding the account of a customer in telco.

So what kind of services they have?

What's the usage of those services?

What are the boundaries or whatever else?

Or what are what incidents they have posted over the last half a year to see if that's maybe a reoccurring problem?

We are yet to see.

We already seen some companies backing out of AI, probably due to poor implementation.

So there were...

I recently read a case study of a company that replaced their call centres with AI.

And the number of complaints over very poor quality of service skyrocketed.

So after half a year, they basically rehired the majority of the call centre and switched back to humans handling the calls.

The speed of processing the calls was great with AI.

The quality of the processing, everybody hated it.

It was horrendous.

So yeah.

Hallucinogens, yes.

I'm happy to get those that AI is doing.

Looking at the...

Very creative.

Very creative.

Looking at the image processing with generative AI back like five years done by Google, the test where you have those wonky images of cats like merged into together.

Yeah, yeah, yeah.

Some hardcore drugs there for AI.

But now it's better.

But obviously it's still not to the level that it's going to be good.

So proper handling of data, proper handling of integrations is now ever more so crucial.

In my opinion, at least.

Agreed.

Yeah.

It's embarrassing, but I actually haven't started playing with this stuff yet because I just got my new laptop sorted out.

My old laptop couldn't run the agent mesh because my...

Whatever, it doesn't matter.

My WSL was too old.

I got a new laptop.

I got all modern.

Anyway, and so I'm going to start playing with this.

So I don't have a...

I don't like to talk and have an opinion about something that I haven't really used and touched myself, you know?

Like I could not tell you much about...

I don't know, Boomi.

Because I've never really played with Boomi.

I kind of know a few high level things.

But if you ask me about Solace APIs, I can go...

I can talk to you for hours about because I played with them I don't like to talk about things that I haven't...

Otherwise, you just sound like a marketing person who's just reading the slides and, you know, whatever, right?

So yeah, I don't want to say too much about...

But then again, you're very much a hands-on kind of person, hands-on specialist.

Well, I'm more of an architect and I deal with qualitative analysis all the time and contextualising that to business.

So a completely different scope.

So we're looking from different perspectives.

For you, it's important to experience that so you can have an opinion.

For me, it's important to ponder upon the topic to have an opinion, to actually think it through.

Two different perspectives on the same thing.

And well, I gave them our business opinion, right?

Really?

So yeah, it's like we see agentic AI systems as distributed architectures and that they are made better by something like us.

So that's...

We're not going to build LLMs, that's for sure.

We're just trying to make them talk better, that's all.

So yeah.

And that's absolutely going to be extremely important in proper adoption of Generative AI and later models to make them talk properly with proper boundaries and proper context of semantic understanding, semantic coupling of what they're doing.

Because otherwise, that's going to be just hallucination after hallucination after hallucination.

And it's not going to be pretty.

Yeah.

And it's like, obviously, proper authorisation from agent to agent is also super important, right?

If I'm querying, if I'm talking about my HR system, I should not be able to look up other people's information on HR, right?

Like, I mean, it'd be very locked down to that versus a weather system.

Who cares?

I mean, there are very customer details.

They're like, yeah, these kind of things need to be part of it.

Passing along tokens, however, authorisation authentication is done.

But we're getting into this now because this is actually useful stuff.

And it's like, now the guardrails have to go up because it's a little bit wild out there.

So yeah.

Yeah.

We're biased as well, though.

We're all biased here.

So and AI and generative AI LLMs only amplify our biases, basically.

And that's been already established all over the place that if we have use cases that can produce bias, that bias will definitely be amplified because the training data, especially if we're using large language models that were trained by a different company.

So whoever is training them, like Anthropic, Google, doesn't really matter.

They're training the models and we're contextualising them to our use cases.

We have no idea on what data those models were trained, which means that we have no idea what kind of biases will be surfacing through those models in our use cases.

We know there will be some, but we don't know what they...

Yeah, that's something that needs to be accounted for.

So any use cases that are prone to bias already should be disqualified from use because you would have to actually sit down in your company, take a vetted data set, check it for biases, train the model yourself, build the model yourself and then utilise it for your use cases.

That is a multi-million dollar endeavour.

Easily.

Not a lot of companies can afford that.

So we're kind of forced to use those models, but that comes with that risk of those kind of cognitive biases presented in the data sets.

And we've already seen that in different ATS systems.

Workday was recently sued for their ATS system.

What's ATS?

ATS, I don't really remember what the abbreviation stands for, but that's basically a system that's vetting candidates.

Okay, okay.

Analysing CVs, vetting candidates.

And it turns out that their ATS, presumably, was already biased towards, for example, people over 50.

Okay.

Right?

And they got rejection emails just automated rejection emails within minutes after applying or at hours that are not operating hours of said company.

These kinds of things.

So those biases based on the data sets already went amplified and made actually companies lose talent because of that.

So there are many, many other examples where that funnel of data and providing proper data for agents is very, very crucial.

Over 50 is not that old, by the way.

I just want to point that out there because that's creeping up pretty soon.

Yeah, yeah, yeah, yeah.

That's, yeah, mine is the 40 mark and that's also creeping on me.

It's, ooh, that's going to be...

I don't care.

I'm young at heart.

And so are you.

So yeah.

There's plenty of problems with that.

We don't know.

We'll see.

We're trying to make it work with what we have.

And integration is going to be only, only more important in general.

Whether or not it's going to be agent to agent or it's going to be our IT ecosystem to those agents to exchange proper data for them to work with that data or utilise services.

We'll see.

So, yeah.

Yep, yep.

And hopefully within a year or so, I'll have a much stronger opinion on all this stuff because I've been kind of...

Hopefully.

I've been using AI, but I haven't been doing AI.

You know what I mean?

I'm still waiting for a lovely project somewhere of a client that actually wants to utilise MCP so I can jump into domain-driven design and actually building that...

Get a project.

Yeah, MCP-related integrations and actually design that properly so that we can work that.

Right now for me, it's a theoretical work, but I see that there is a strong connection to domain-driven design that the domain-driven design field will grow due to agent AI and the utilisation of data by agents.

That's definitely going to be a mandatory thing to handle that data in that specific way.

Yep.

That said, we are way over two hours.

Yeah, we are actually.

Yeah, I'm like, I got to go pee soon.

it was such a rush to get here.

Honestly, I'm glad our planes were late this morning because there was more thunderstorms rolling around Ottawa.

So my plane into Ottawa was a bit late.

So I'm very happy, even though we're two hours past when we said we were going to do this, that you were flexible enough to allow me to get home.

Otherwise, well, I could have just hung out at the airport like in the corner.

That would have been a little bit cringy to talk about these things with such excitement at an airport.

That would be a little bit difficult.

Thank you for being flexible and allowing me to get home.

And thank you for hosting me on your cool podcast.

Maybe you should come on to my podcast at some point.

Oh, definitely.

Office hours for the win.

Office hours.

All those office hours.

And for those, I'm going to switch for a moment.

Take us, well, okay.

Take us off screen.

Coming up next week.

Exactly next week.

We're going to be having a bit of a discussion.

Topic that's very related to our now discussion about AI.

So we're talking data mesh versus application integration.

I'm having two lovely guests.

Rachel Barton, who is a principal data and domain architect.

And Andrew Jones, who's a principal engineer and author specialising in data contracts.

That's going to be a very interesting conversation to be had.

Because we're going to be looking at what are differences in the field between data mesh and application integration.

What the hell is going on?

Because I've had some really interesting comments on the topic after a workshop that I ran at DDD Europe this year.

And I think there's a lot of misconceptions and basically differences in language.

How people describe the data architecture and data mesh versus how we talk about application integration.

And this is something we're going to be exploring in the next stream, which is going to be exactly in a week.

And if you're still with us, if you're still watching, and if you want to catch all the interesting topics on live streams, but also articles, you can read more on application integration, enterprise application integration, event-driven architecture, API-led architecture at bridgingthegap.eu.com.

You can subscribe to our sub stack where we release all those live streams as posts also release their articles with a little bit of a delay and sometimes have some conversations.

And if you're still with us, subscribe to the YouTube channel.

You'll get a ping about new streams coming up.

I'm also planning a few video series, well, still in planning, but materials pending to be built up and recorded.

So hopefully a little bit more content will be there alongside these live streams.

So yeah, that said, we're going to be ending.

What time is it in Ottawa right now?

It's coming up on 5.30 PM.

5.30, oof.

Yeah, time for dinner, lunch.

Time for dinner.

Actually, so yeah.

For you, time for dinner.

For me, it's time to go to bed and get some sleep before I go to the office tomorrow.

Thank you all for joining.

And Aaron, it's not the last time that we're going to be seeing you.

It's definitely the first of many.

Yeah, we'll be chatting about more event-driven topics later on.

I'll ping you offline to chat about those.

We'll figure something out.

I didn't show my sunburst.

I have to come back and show the sunburst diagram at some point.

So we already have a new topic to talk about.

So we'll jump into that.

So observability, analytics, these kinds of things, these are also EDA related, very important.

Welcome back to EDA.

There are also more specialists in the field of EDA that we could talk together about EDA.

That's going to be probably interesting.

So first occurrence, not the last one.

So we'll see each other on the stream somewhere later.

And yeah, thank you for watching.

Thank you, everybody.

Have a lovely afternoon, evening, night, whatever it is in your time zone where you're watching this.