Why Kafka + TIBCO === A Better Event-Driven Architecture
Ok, where do I start … This relatively small title covers a whole gamut of concepts, technologies, and — dare I say it — a little bit of technical religion too.
Let’s start with Kafka. At its core, it’s an Open-Source Event Streaming Platform that was created by our friends here at LinkedIn. It was designed as a highly scalable distributed platform that can handle trillions of “Events” per day, allowing you to capture literally any event as it happens in real time for processing. Kafka is on every single Developer resume today, and it’s being used in nearly every Company/Project that you can think of. Super cool tech and everyone wants a piece of it.
I mentioned “Events” in the above paragraph too. What’s an Event? In its simplest form — “Something Happened, at this date/time.” It could have been a new order placed on your website, an IoT sensor reading, or a change of state — an Order moved from being Processed to Shipped for instance.
Coupling Kafka and Events together, you get the basis for an Event-Driven Architecture. Something happens — that gets published to Kafka, something picks up that message and does something with it. All in real time and as fast and scalable as you want.
And this is where it gets interesting. What do you do with those events, and how best to process them?
Kafka includes something called the Streams API, giving you the ability to write Java code to process those events, allowing you to aggregate events, create joins of data within a stream, create windowing parameters to look at time slices of events.
But at the end of the day, you have to write code, and while this is great for Developers who love to write code, this isn’t necessarily the right thing for a Business. A Business wants to be agile, respond to market changes quickly, make changes to their IT systems to cope with new business strategies overnight, and if you are relying on Developers handcrafting code, it’s never going to be as quick as the Business wants it. This leads inevitably to the great Business/IT divide that we’ve all seen.
Kafka is encouraging the developer mindset and merely helping create new ways of writing code.
So how can we do better?
When I think about Event-Driven Architectures, there are always 3 key aspects to it:
- Event Distribution
- Simple Event Processing
- Contextual Event Processing
Event Distribution is precisely where Kafka shines and what it was built to do. There are of course many other Messaging platforms (of which TIBCO is a notable vendor), but I would not put Kafka in quite the same bucket. There are many different types of messaging patterns such as Publish/Subscribe, Point-to-Point, Queuing, etc. and Kafka only really supports Publish/Subscribe — which is perfect for an Event-Driven Architecture. I’ll look at another blog in the future that discusses many of the patterns and which messaging platforms would support each one.
Simple Event Processing starts to look at what those Events published on the Event Distribution platform actually mean. Usually implemented as simple if/this/then statements and are generally reactionary. So, if we receive an Event that says an Order is Shipped, then we might send an email to the customer.
Contextual Event Processing starts to bring in other techniques to help learn more about the Event and the Action that needs to happen based on that single Event — or crucially an aggregation of Events over time, even the absence of an expected Event is classed as an Event in its own right. Quite often, this is where AI/ML algorithms also come into play.
As I mentioned previously, this is commonly achieved utilizing Kafka, the Kafka Streams API, and Java coding. (Although I’d suggest that Contextual Event Processing would be difficult to do in this scenario.)
And this is where TIBCO starts to add some real value to Businesses. Whilst TIBCO provides the same level of commercial support for core Kafka as other vendors, TIBCO also has a huge amount of experience in coupling Event Distribution with Simple and Contextual Event Processing.
In particular — all Event Processing is 100% GUI-Driven. Because all the Event Processing Capabilities provided by TIBCO are GUI-based. Developers spend less time building the business logic required than if they were handcrafting code which means that they can respond to business requirements much faster.
TIBCO also has different solutions for Simple or Contextual Event Processing depending on the complexity of requirements. One size does not fit all and we always advocate using the right technology for the job at hand. From simple rules engine implementations through to State Machine Models.
Everything in the software world these days needs to be stateless but any hardware developer will tell you, understanding the current state is absolutely vital when building real-time hardware systems and the same is true for understanding how you react to events in real time in a software system. So, having a robust state machine implementation is crucial in a full Event-Driven Architecture.
All of this brings me to my last point — Reacting to Events. I’ve kind of skirted around this already but being able to react intelligently to an Event or set of Events or even the lack of an expected Event could be crucial for a Business. We often call these Business Moments, and these usually relate directly to the Business KPIs and have an immediate impact on the Customer Experience. Think of a new Customer Order or a sensor reading that passes a threshold over the last 5 minutes. These are examples of Business Moments that we need to react to as they are happening. Even understanding that 5 minutes after we update the CRM system, we should see an email being sent to the Customer — and detecting that didn’t happen (as a non-Event) could indicate something important that we want to know about.
In fact, Events also have a valued lifetime. In most situations, the data is most valuable “right now”. But over time that value decreases. It would be great to know that we are being defrauded right now because we can stop the transaction from happening, rather than the more traditional — we were defrauded last week according to our weekly transaction report, and there’s nothing we can do about it now.
There are lots of other examples that I could point to that our customers are doing right now with Kafka (and others) + TIBCO to build out some amazing Event-Driven Architectures, check out the TIBCO website or look out for our annual TIBCO Now conference where many of our customers present their stories.
One last thought I wanted to leave you with:
- Kafka! = Event-Driven Architecture
- Kafka ∈ Event-Driven Architecture
- Kafka + TIBCO === A Better Event-Driven Architecture
