The Kafka Cargo Cult
Overview
About this stream
“We don’t do message brokers and Event-Driven at this company” is what I heard from one client some time ago. I was puzzled, because why would someone discard a whole valid set of architectural patterns? Turns out a few years back they suffered heavily because someone thought they could be like Netflix or LinkedIn and use Kafka. “It worked for those companies - right?” Absolutely! The problem was that this client was not a streaming service and had no use case for streams! The result? Heavy implementation, high cognitive cost for all teams, serious operational problems.
“We don’t do message brokers and Event-Driven at this company” is what I heard from one client some time ago. I was puzzled, because why would someone discard a whole valid set of architectural patterns? Turns out a few years back they suffered heavily because someone thought they could be like Netflix or LinkedIn and use Kafka. “It worked for those companies - right?” Absolutely! The problem was that this client was not a streaming service and had no use case for streams! The result? Heavy implementation, high cognitive cost for all teams, serious operational problems.
We often look at the success of various companies, their tech stack and think: “Hey! We can do the same thing! It will work because it worked for them!” The problem is that we often do that without looking at trade-offs, added complexity, or even having a valid use case for such technology. This effect is dubbed a “cargo cult”, and while it originates from people’s behaviour in Melanesia during the 2nd World War, it has found its home in IT!
Let’s have a chat about one of the most famous cargo cult in Integration Architecture - Kafka! For that we’re bringing to the stream two returning guests:
Chris Simon, a technology coach and advisor helping technology teams drive business success. Chris talked about this this year at DDD Europe “It’s like 10000 streams when what you need is a queue”
Aaron Lee, a Senior Solutions Engineer at Solace. Aaron will happily discuss event-driven architecture, networking protocols, application design, monitoring, time-series data, and visualization frameworks for as long as you’ll listen.
Together we’ll dive into Event-Driven Architecture and when to use Kafka and other message brokers! Join us live!
People
Speakers
Chris SimonTechnology coach and advisor
Aaron LeeSenior Solutions Engineer at Solace
Karol SkrzymowskiHost / Integration Architect
Watch and read
Transcript
1771 transcript lines
- Karol
Good morning, good afternoon, and good evening, everybody, and welcome to another episode of Loosely Coupled by Bridging the Gap.
- Karol
My name is Karol Skrzymowski, and I'll be your host tonight.
- Karol
And today, for the first time, I think, ever, we actually can welcome our guests and myself with a good morning, good afternoon, and, well, good evening, because the guests today are from completely different time zones, and we're going to be talking about something quite interesting, a little bit of psychology, quite a lot of tech, quite a lot of architecture.
- Karol
The topic today is Kafka Cargo Cult, but we're not going to be limiting ourselves to Kafka.
- Karol
We're going to be talking about event-driven architecture.
- Karol
That's just the title.
- Karol
So, without further ado, joining me today in the studio, we have Chris Simon coming to us from Sydney, Australia.
- Karol
It's 3 a.m. in Sydney at the moment, and Aaron Lee coming from, was it Toronto, Canada?