Streaming Audio: Apache Kafka® & Real-Time Data

Using Event Modeling to Architect Event-Driven Information Systems ft. Bobby Calderwood

September 30, 2020 Confluent, original creators of Apache Kafka® Season 1 Episode 120
Streaming Audio: Apache Kafka® & Real-Time Data
Using Event Modeling to Architect Event-Driven Information Systems ft. Bobby Calderwood
Show Notes Transcript

Bobby Calderwood (Founder, Evident Systems) discusses event streaming, event modeling, and event-driven architecture. He describes the emerging visual language and process, how to effectively understand and teach what events are, and some of Bobby's own use cases in the field with oNote, Evident System’s new SaaS platform for event modeling. Finally, Bobby emphasizes the power of empowering and informing the community on how best to integrate event streaming with the outside world.

EPISODE LINKS

Tim Berglund (00:00):
If you listen to this podcast, you probably believe event-driven architectures are a good idea, but how do you design them and how do you document them? This stuff's all pretty new and these tools aren't all mature yet.

Tim Berglund (00:12):
Well, once and future Streaming Audio guest, Bobby Calderwood, is here to tell us about event modeling, a framework and toolset for such a time as this. Now, we've had a couple of episodes with some audio trouble recently, this is one of them.

Tim Berglund (00:25):
So please bear with us on today's episode of Streaming Audio, a podcast about Kafka, confluent, and the cloud. Hello, and welcome to another episode of Streaming Audio. I am your host, Tim Berglund joined again, as a returning guest, by my friend Bobby Calderwood. Bobby, welcome to the show.

Bobby Calderwood (00:52):
Thanks Tim. Great to be on again.

Tim Berglund (00:54):
Yeah, I really enjoyed our previous conversation. Something like a year ago. I didn't look it up and I don't remember these things precisely, but we talked about a lot of some of the financial systems you work on and the intersection of functional programming and event-driven architectures. Does that all sound like it was real?

Bobby Calderwood (01:13):
Yeah. No. It sounds like a dream. It sounds like a far off dream. Yeah. We [crosstalk 00:01:17]

Tim Berglund (01:17):
[crosstalk 00:01:17] a dream but a good dream.

Bobby Calderwood (01:18):
That's right. No, that was one of my favorite things I did last year in my career. It was so fun to be on the podcast. And I always enjoy talking to you whether on the podcast or conferences or whenever we meet up. So yeah. I know it was great. Yeah. We talked about functional programming, functional microservices, and a lot of, I think what we'll talk about today kind of extends and enriches that particular topic.

Tim Berglund (01:40):
Yeah. Yeah. Hard to go wrong with those things. So you recently spoke at Kafka Summit, as recent as of the time that we're recording this. That was not quite a month ago as of this recording, and you talked about event-driven information systems and particularly how to model them. And that's just so there's no suspense. That's what I want to talk about today, but I'd love an update. We talked about all this financial stuff. Are you able to talk about what you've been working on recently?

Bobby Calderwood (02:11):
Yeah.

Tim Berglund (02:11):
But just before you do that, I sometimes cite you by name as one of the few people who's built more than one event-driven system. Everybody's on their first right now.

Bobby Calderwood (02:20):
Right.

Tim Berglund (02:22):
[crosstalk 00:02:22] puzzling through all these hard problems to kind of learning this new discipline, I'm like, "Well, some people have done three and..." So what are you working on right now?

Bobby Calderwood (02:30):
Yeah. Yeah. So I think we're midway through the good news, a good report on kind of the last podcast episode. We talked a lot about real-time payments and the system we're working on at the time was this realtime payment system to help US banks to plug into this really great new infrastructure that's available from the clearinghouse, the realtime payment system.

Bobby Calderwood (02:53):
We built that out. We were kind of in the market selling that to banks for quite a while. And just recently we've started very serious talks about acquisition where a business wants to acquire kind of the part of the realtime payment of our company, so the system that we build or the IP around it, et cetera. So yeah, and acquisition, that's really, really great. So we're [crosstalk 00:03:13]

Tim Berglund (03:12):
That sounds all kinds of not bad.

Bobby Calderwood (03:15):
Yeah. No, it's really good. I can't talk about it in detail obviously, because the negotiation's still ongoing-

Tim Berglund (03:17):
Of course.

Bobby Calderwood (03:18):
... but yeah, that's been a really exciting thing for our business and I credit a lot of that success to the fact that it is event-driven. It is built on top of Kafka. It's sort of this Bulletproof infrastructure and architecture that's been really influential in our discussions and negotiations, both with the banks and with these other companies that have talked about acquiring us too.

Tim Berglund (03:38):
Super. You said something in your talk about the way we think about event-driven systems. I think it's hard and this is just based on my experience in the time I get to spend with architects and developers and folks who are trying to do this. And it seems like a lot of those actual sessions I've been in have been with financial companies.

Tim Berglund (04:02):
That's probably just a function of the fact that Kafka and a well-drawn architecture uptake is high inside that vertical, but it seems like people have a hard time, right? It feels like we were taught... At one point, it was hard to build a system on top of a database and build a schema and all that.

Tim Berglund (04:22):
You had to learn that. Nobody is born being good at that, and we went through that process of learning it and everybody who's been around for a little bit has built a few of those and people who are new at it have learned and they're building their first one and there's this momentum behind that. And I feel like it's hard then to think about systems in terms of events. What do you think of that? Am I... I'm kind of questioning myself now and maybe I'm identifying the wrong hard part, but what do you think?

Bobby Calderwood (04:53):
No, I think that is really hard and it's both hard to think about and hard to communicate about. And part of that is the issue that you brought up exactly, right? Part of it is engineer training, right? We were trained by brand new hatchling engineers to think about the third normal form and the entity-relationship diagram and the database.

Bobby Calderwood (05:13):
And then sort of projecting that up a layer with a whole bunch of impedance mismatch into our code and thinking about the objects and their relationships and all of that stuff. And that look it's an unnatural way of thinking to your exact point. It took us a lot of training to think about things in that way. But to a man of the hammer, everything seems like a nail, right?

Bobby Calderwood (05:36):
Once you have that pattern and set of paradigms in your brain, that's the way that you sort of look at the world and it becomes sort of a worldview and it colors every problem that you approach, et cetera. So I think at this point, the difficulty for the engineering side of the organization is exactly what you said.

Bobby Calderwood (05:53):
When we're kids, we're used to thinking about things in terms of events. We think about in narratives like, "Hey mom, I went outside and this happened and then this happened and that causes this other thing to happen." Right? That's the story we tell our mom when we come back in from playing.

Bobby Calderwood (06:05):
So, it's easy for us as humans to think about systems or to think about events that way. But then all of this engineering training kind of gets in the way and we were trained in a very unnatural way to think about problems in this very sort of Silicon-first sort of way, right? So I think for the engineers, that's what gets in the way is our training.

Bobby Calderwood (06:24):
On the other side of the house the business stakeholders and the potential users of our systems, the things that we build, I think we have trained them to think about these things wrong too. We train them to think about where's the data? How's it stored with? What's the database? How do I curate it?

Bobby Calderwood (06:43):
I think it's more natural for the business side of the house to think about things in terms of events and narratives and the customer comes and he asks, or she asks us to do this, and then we think about it and then we do this other thing instead, right? So, I think the business side of it still thinks in terms of events, but then when they start to think about the systems, they're like, "Oh, okay, now that event-driven thinking breaks down but it's not this explicit in their brains."

Bobby Calderwood (07:05):
But this is sort of the conversation they have with themselves is I can think about it like a normal human up to this point and now I've got to talk to the engineers and now we've got to talk about it in terms of databases and the stuff that I sort of understand in a very fuzzy way if at all, right? So, I think that's the issue. I think it's a self-inflicted wound on our part. I think we've done this to ourselves.

Bobby Calderwood (07:24):
Fortunately, I see light at the end of this tunnel. I think as we do start to sort of ensuring the business narrative as the thing that we think about and then the systems we build, the business processes we build, et cetera, are subservient to that, right? They are built in the service of this business narrative. I think as we get back to that as an industry, it'll be a lot easier for new engineers to come on into the industry.

Bobby Calderwood (07:48):
They won't have to unlearn a whole bunch of bad thinking. They'll be building event-driven systems from the first day, and they'll be able to think about it in those terms. So that's kind of the thinking part of it, how we envision and talk to ourselves about these systems, the way we talk to other people about these systems is also really hampered by this, right?

Bobby Calderwood (08:07):
So there's this constant problem in software engineering. Agile tried to solve it. User stories, jobs to be done, behavior-driven development, they're all efforts to solve this. How do we capture the requirements of the system? How do we describe the system to each other in a way that the regular humans can talk to the engineers and the engineers can talk to each other and talk to the UI UX people and talk to the architects and... Right?

Bobby Calderwood (08:32):
So, how do we describe the system to ourselves during the design phase before any code is written? As we're writing code, how do we imagine that code to ourselves and describe it to each other? "This is the part of the system i'm working on today." And then when we're in operations, if something goes wrong there's an IT incident, how do we communicate that on the incident call?

Bobby Calderwood (08:54):
How do we talk about what went wrong on the incident call without having the proverbial Brent from Phoenix Project on that call, do we have enough context and system understanding on that incident call to be able to address that incident? Right? So throughout the software development life cycle, there's this pervasive communication problem about how we envision these systems and how we describe them to each other. And there's no good solution, right?

Bobby Calderwood (09:18):
I mean, there's been tons of attempts throughout the years in the industry to try to solve this problem and it remains unsolved, right? I mean, user stories at some level are great to sort of during the agile kind of planning and design process, but they're not going to help you during an incident call, right? You're not going to bring up a user story in the incident call to try to say, "Yeah, this part of the system is broken." [crosstalk 00:09:43]

Tim Berglund (09:43):
I'm sorry, but when you say it like that. [crosstalk 00:09:46] I know you're not ever going to do that.

Bobby Calderwood (09:49):
No.

Tim Berglund (09:49):
Stories are a good way of understanding some things. Just that's very funny. You're right.

Bobby Calderwood (09:54):
Yes.

Bobby Calderwood (09:54):
So, we lack this holistic system understanding and a holistic way of communicating about systems. Yeah. And it's a problem.

Tim Berglund (10:06):
Yeah. I know. I mean, in the '90s once I was involved in a big project that was written in C that used a structured analysis, structure design tools that could round trip. I mean, so you build this particularly, I'll say terrible kind of diagram, and it would generate code and you fill in the functions with the good bits.

Tim Berglund (10:32):
And if you had to change method signature or function signatures or something like that it would do a round trip and update the diagram. It's super space age so... There was a tool for visualizing the system, for designing the system that attempted to stay up to date, but it's... You know what? No, I can't even. It's too much of a strong question, I can't bring myself to ask, "Why doesn't that work?" Because we all know that just doesn't work.

Bobby Calderwood (10:54):
Right.

Tim Berglund (10:56):
Yeah. That's actually... It turned out to be a terrible idea. But you're right. We don't have a standard thing. Let's talk about some other options though. You mentioned user stories and I don't want this to come off like we're bagging on user stories. There is wonderful mechanisms, but as you say, there are things they don't do.

Tim Berglund (11:16):
I think the agile answer to that is that you shouldn't want, or it's somehow a quest for an illegitimate thing to try to find something like a story that you can bring up in the support call. It doesn't exist. The code is all that exists at that point.

Bobby Calderwood (11:33):
Right.

Tim Berglund (11:34):
Probably shouldn't debate that right now. We may agree anyway, but what else? I mean, UML comes to mind, and that's... I think many people are thinking that right now.

Bobby Calderwood (11:44):
Yeah. And you bring up a good point with sort of the agile answer this... In theology there's sort of a parallel argument, right? Is it scripture or code only? That as the source of truth, or is there scripture and tradition? Which is code and tacit knowledge in the minds of the development team, right? Yeah. It's a big question, right?

Bobby Calderwood (12:03):
It's a sort of fundamental epistemological question that arises in software engineering but also arises in these other domains of human endeavor. How much can we know? How do we know it? Are we sure we know it, right? This is epistemology. It's the field of philosophy where we talk about these things. Yeah. And it's a huge problem. It's one that has befuddled philosophers and theologians for many centuries, right?

Tim Berglund (12:28):
I was going to say it's an enduring problem because-

Bobby Calderwood (12:31):
Yes [crosstalk 00:12:31]

Tim Berglund (12:31):
... there isn't an epistemological consensus.

Bobby Calderwood (12:33):
Right. Right.

Tim Berglund (12:34):
There isn't. You can't go ask philosophers and they'll all tell you the same answer of how we know things.

Bobby Calderwood (12:38):
That's right. And I don't think we'll get there with code, especially not on a 45-minute podcast either. So we're not going to solve this question.

Tim Berglund (12:45):
I'm optimistic, but keep going.

Bobby Calderwood (12:48):
But yeah, there have been these really... And I certainly don't want to come off as denigrating any of these things we're talking about. Look, they're engineering tools. They all have features and they all have trade-offs, right? And I'll probably emphasize the trade-offs because I'm looking for a better way, but the UML community, you sort of alluded to it earlier, some really fantastic work.

Bobby Calderwood (13:08):
I mean, some almost magical engineering to your point, this ability to round trip from code to diagram. That's magic, right? That's basic stuff. User stories, jobs to be done, behavior-driven development was sort of given when then Gherkin and phrasing and all that stuff, all super great attempts to solve this fundamental problem, this communication and no ability problem about how we collectively understand a complex thing.

Bobby Calderwood (13:40):
There are some drawbacks, right? So UML, I think for the problem that I am discussing where you're trying to get this holistic system view across multiple skillsets, right? So you want your business stakeholders even potential users, UI UX folks, software architects, project managers, software engineers. You all want them on the same page, understanding the same things and knowing the same things.

Bobby Calderwood (14:06):
UML is not for that, right? UML is very focused on engineers. It about specifically sorts of object-oriented programming disciplines. There are parts of UML that are not specifically object-oriented obviously, but a lot of it is sort of geared towards object-oriented code design right down at that sort of code level.

Bobby Calderwood (14:24):
And it's really by engineers and for engineers, it's not about communicating how the system works to everybody else, right? Because you show a sequence diagram or some of these other things, you show that to other people and they're like, "How does this help me make more money?" Right? This doesn't [crosstalk 00:14:39]

Tim Berglund (14:38):
It is a specialized form that you actually have to learn not just how to create, but how to consume.

Bobby Calderwood (14:44):
That's right. That's right. And at that point you might as well be writing code, right? Because it's by engineers for engineers, it's only consumed by the engineers, right? At that point why not just be writing code? At some level, maybe it helps you visualize it because then it sort of engages that visual cortex, that part of your brain is super well developed.

Bobby Calderwood (15:03):
But as an industry, I think we've mostly left you all behind. There's a few artifacts, like sequence diagrams that sort of assist in kind of modern agile software practice. But a lot of that stuff has gone by the wayside and to your point kind of for good reason, right? It adds overhead to the engineering process without this sort of holistic communication benefit of being able to describe what we're doing to non-engineers. Then there's...

Bobby Calderwood (15:28):
So that's kind of by engineer for engineer type of diagramming and modeling disciplines. On the other side of the spectrum, there's sort of this low-code world where our IT departments are a pain to work with. And so we're going to end-run around them and empower this sort of a legion of citizen coders to draw a picture and then press a button and then that picture turns into a system.

Bobby Calderwood (15:52):
And all of these things are sort of in these black box runtimes. And so from the IT department, it's sort of a double slap in the face, right? The IT department is like, "Okay, so you're end-running around us. Cool. That feels really great. And then you're going to dump onto our laps this black box platform that doesn't fit any of the ways that we do any of the rest of this stuff, right? It doesn't integrate with our cICB pipeline.

Bobby Calderwood (16:14):
It doesn't run on the same hardware, the same platforms. We don't know how to keep bugging. When something goes wrong in production, we don't have the same level of monitoring and instrumentation and stuff that we're used to with our regular real coding language things like the JVM or whatever we're running on."

Bobby Calderwood (16:30):
And so now it's sort of this double slap in the face like, "Okay. A, you showed us that we're hard to work with and you'd rather just write the code yourself. Okay, great. And we have to operate this thing that is sort of pain and that doesn't fit with the rest of our IT world."

Bobby Calderwood (16:43):
So I think both of those fall short of the sort of whole system understanding and communication and bringing together kind of in that DevOps spirit, bringing together the different parts of the organization to work in harmony and build common understanding. But I think the low-code is a bit of a nuclear option, right?

Bobby Calderwood (17:02):
It's a bit of the... Like, "If you guys don't get on the same page with us, we're just going to forget about you and just go do low-code. Do you know what I mean? It's more of an alternative than it is a thing that brings parts of the organization together in my view and in my experience.

Tim Berglund (17:15):
Right. Which I guess brings us to event modeling as you understand it. So you've got... I say you. This is a tradition, I think, that you're entering and also helping build. So by all means, talk about the landscape a little bit, but tell us about this emerging visual language and process for-

Bobby Calderwood (17:34):
Yeah. Absolutely. So in my topic in my [inaudible 00:17:36] on the talk, I mentioned this a little bit. On the one hand kind of in the old days, we had this big design upfront moment where we're spending a lot of time on the design. And then as we moved over to agile because a lot of these waterfall type processes didn't work, they postponed risk and all the things that we don't need to go into now about why sort of waterfall processes don't work for a lot of types of software engineering.

Bobby Calderwood (17:59):
As we swung over to Agile, we sort of threw out that design discipline in a lot of ways. We sort of went from big design up front to no design up front and now we're just sort of like, "Oh the design will emerge as we test code refactor." Test code refactors, surely a design will emerge in our code. I've never seen it happen so I don't believe that story.

Bobby Calderwood (18:19):
Maybe within a single code base when you're talking about how you structure your classes and function calls and method calls and stuff within a codebase, maybe that can happen to be a TBD, but not a distributed systems design, not a, like I said, sort of a holistic and well-thought-out way of solving business problems. That's not going to emerge from any particular code-based test code refactor cycle, right?

Bobby Calderwood (18:45):
You have to have a design, right? And when software engineers code in the presence of a design, they are a lot more efficient and they're able to solve their problems a lot better. And they spend a lot less time thrashing in the, I don't even know if I'm close to solving a problem because I don't know what problem I'm solving, mode.

Bobby Calderwood (19:03):
Early in my Agile software experience back in the late '20 tots and early '20 teens, I spent a lot of time as a junior developer in that mode where I don't know what problem I'm solving and so I can't really measure if I'm making progress. I'm doing stuff and I'm keeping busy and my head's down and I'm reporting stuff on standup, but I don't know if I'm helping or if I'm hurting.

Bobby Calderwood (19:27):
So those are the two extremes. The big design upfront and the no design up front. I think both are really flawed. So in the search for this lightweight language that will allow us to do some design up front, but sort of just enough design upfront. And that serves us downstream with design and implementation and operations in the ways I described getting everyone on board, getting the ops people and the business users and the stakeholders. Kind of getting everyone speaking the same language.

Bobby Calderwood (19:59):
The best thing I've found in my career to this point is event modeling. Event modeling is a discipline that was invented sort of pioneered by Adam Dymitruk with help from Greg Young. Greg Young is sort of the coiner of the term, event sourcing, and CQRS. He sort of invented that at some level.

Bobby Calderwood (20:18):
Yeah. He's very, very good. In fact, I was just on a meetup with him a couple of weeks ago where he gave his sort of original event sourcing talk. And it's sort of this talk that he's been giving for, I don't know, 15 years, 12 years at least. And it's about a financial domain. So that kind of really spoke to me from my past experience. And he talks about why event sourcing is superior to thinking about the current state. Thinking about the events really helps you in a lot of these ways.

Bobby Calderwood (20:47):
So Greg and Adam work together. They're kind of up in the Seattle and Vancouver geography. And so they've been fellow travelers for a long time. And then Adam sort of distilled there is very extensive engineering experience, sort of distilled a lot of these thoughts around CQRS, event sourcing, how to do it in practice, what does it mean for coordinating with other teams and describing the system to your users and how do you incorporate the user journey into this thing that is sort of backend in terms of event sourcing.

Bobby Calderwood (21:18):
So, he distilled all this down to this really beautiful and very simple modeling language called event modeling and it's super simple. And that's really the benefit of it is its high level enough and descriptive enough for non-technical skillsets to be able to engage and contribute meaningfully to the design and sort of system understanding. But it's also well-specified enough to be immediately actionable by the software engineer.

Bobby Calderwood (21:44):
So, it really is a sort of Goldilocks zone of just enough design, general enough for non-technical skillsets but specific enough for software engineers whereas UML is much too specific and doesn't speak to this nontechnical skillset. So it's this really, really great distillation, I got to give Adam a lot of credit for what he's built. So I discovered this while I was doing professional services work.

Bobby Calderwood (22:09):
I was doing an event storming, which is sort of an ancestor technique to event modeling. Alberto Brandolini did event storming, wrote some really good thought there as well, came from the DDD community. And I had come up with my own kind of hackneyed simplification of event storming that was more... Event storming tends to be very high level. How do you map out kind of a whole domain? I only came up [crosstalk 00:22:33]

Tim Berglund (22:32):
When you're trying to explore a domain and generate shared understanding about-

Bobby Calderwood (22:36):
That's right. That's right. And event storming explicitly is sort of about exploring the problem space. Like, "Hey, what's the problem we're trying to solve? How do we all work together to solve that problem?" And it's much less about solution space, whereas event modeling is really truly solution-oriented, we're speaking out a piece of software we're going to build when you're approaching event modeling rather than exploring a business domain.

Bobby Calderwood (22:57):
And so I had to come up with my own kind of hackneyed simplification to event storming. I called it a low level or whatever. I don't know. It was a terrible name, but low-level event storming, and we were doing that... When I was at Capital One when I was at... As soon as I started evidence systems and doing my professional services work with a bunch of big companies doing this, that's what we're working on.

Bobby Calderwood (23:18):
And in fact, actually, two colleagues of mine and I at Capital One were sorts of given the job of coming up with the microservices curriculum there and we incorporated event storming as sort of an explicit part of the two-day training that we developed. We developed the curriculum and talked about all that. So event storming I've been doing it for years since I don't know, 2016, 2015, 2016 timeframe and it was great and I really liked that.

Bobby Calderwood (23:46):
As I found event modeling I was like, "Yes, this is what I've been looking for. This is the kind of simplification, the lower level kind of zoom in and starts talking about a specific system kind of focus that event storming didn't give me." So it's great and so I've been doing it ever since kind of 2017-2018 timeframe. Yeah.

Tim Berglund (24:06):
Nice. Nice. You're good. Saw a little bit of that under your belt at this point. I want to revisit, just because you're making me think things and not because this is the time to debate it, but the question of agile architecture.

Tim Berglund (24:19):
You have a settled opinion that it is substantially not a thing. And I know there are people who say that it is, and I don't even know if this podcast is a place to work that out. I might go and invite some people to just talk about that because it's interesting and it kind of relates to this.

Bobby Calderwood (24:38):
Yeah. I'd love to be involved in that conversation. I think it's an important one. Just anecdotally and personally, I've met software architects or enterprise architects at companies that adopt agile and they feel hopeless. They feel disempowered and disenfranchised at some level where they're like, "I used to know what my job was and I drew pictures and I thought about latencies and SLS and that's what I used to do.

Bobby Calderwood (25:06):
And now in this agile world, I don't know what I do. I serve as sort of a consulting developer on teams and I sort of visit three or four teams and help them with hard things. But that doesn't feel architect to me. That sort of feels engineer SWAT team to me." Right? So I've seen a lot of architects struggle with this. So, I think it's an open question. I don't think we've solved it all the way.

Tim Berglund (25:25):
We haven't. I'm now just imagining the agile architecture proponent answering what you just said with, "Good."

Bobby Calderwood (25:31):
Yeah. All right. And that's fine. If that's your answer, that's fine. But look, there's a real human cost there, right? I mean, you've got these people who have been doing a job for a lot of years and they have a ton of wealth of experience and depth and they understand. They know where all the skeletons are buried in these systems and to disenfranchise them and lose them, you're losing a lot. There's a real human cost there.

Tim Berglund (25:52):
Yeah. The human cost to the organization. I mean, the human cost is one thing but-

Bobby Calderwood (25:56):
Yeah. Absolutely.

Tim Berglund (25:57):
... there were once a lot of blacksmiths and there are fewer of them now. Sometimes that happens.

Bobby Calderwood (26:01):
Right.

Tim Berglund (26:01):
And there's a human cost and somebody's got to love those people but in economic transitions like this one could argue that it is such a time for the enterprise architect, but maybe not. It's very much an open question and that would be an interesting panel to do.

Bobby Calderwood (26:19):
Yeah. [crosstalk 00:26:20]

Tim Berglund (26:19):
So for the future, if you're listening at me, if you think that sounds a good idea and where else do I want to go with that? We talk about architecture and... Oh yeah. Yeah. So, the other insight I had has to do with a book I'm reading right now called The Fifth Discipline by Peter Senge. It's a systems thinking. It's really a management book, I think, but a super intellectually interesting one.

Tim Berglund (26:50):
The subtitle is The Art and Practice of the Learning Organization. It was originally written 30 years ago but updated maybe 15 years ago. So some of the examples feel like I'm young again, but the ideas are kind of blowing my mind and the whole architecture emerging thing, and on what scale does architecture emerge and you made the point here and in your talk when you've got multiple systems that are interacting.

Tim Berglund (27:19):
It doesn't seem like the agile architecture thing really happens then. Maybe if there's one system, there's enough mental state that implicitly or otherwise people have the structure of the thing in their minds, they're able to reason about it, but then there are these...

Tim Berglund (27:38):
You're building a composite system of individual subsystems that have fairly primitive interfaces and all you really know is that primitive interface but not enough of the structure and interaction of the other system. And if you never know enough about the structure and interaction of the other system, then it's maybe impossible on principle for that architecture to emerge.

Bobby Calderwood (28:00):
Yeah. Yeah. And there's a lot of really great writing and thinking about the dynamics of complex systems, et cetera. I really love the DevOps community for embracing and really John Allspaw and some of those other folks that look at complex systems and complex system failure specifically. And they're like, "Look when a complex system fails, you can't just fire the person who was the proximate cause, because it is not their fault." I like it if you look at the dynamic-

Tim Berglund (28:28):
[crosstalk 00:28:28] such thing.

Bobby Calderwood (28:30):
Right.

Tim Berglund (28:31):
That's [crosstalk 00:28:31]

Bobby Calderwood (28:31):
Yeah. Yeah. And there may not be such a thing as a fault, but complex systems are definitionally complex. There's a lot of different concerns and they are twisted and intertwined with each other in ways that make it impossible to predict.

Bobby Calderwood (28:43):
That's the definition of a complex system is you cannot predict the outputs of the system from the inputs, right? That's how we define a complex system, and... So to that point, it is unethical to fire someone because they put in the wrong inputs and got an output you don't like because that's not [crosstalk 00:29:00]

Tim Berglund (28:59):
Yeah. [crosstalk 00:29:00] Kind of coming to the conclusion that it's a little sketchy to say root cause.

Bobby Calderwood (29:05):
Yes.

Tim Berglund (29:05):
That kind of doesn't look like [crosstalk 00:29:06]

Bobby Calderwood (29:06):
Yes. Right. The Five Whys Analysis and all of that. That works at some level, but not in an absolute sense when you're dealing with complex systems. The real trick to this is... Look, and this is the trick for political science and political economy too, right? Where you've got this infinite possibility, all of these economic agents interacting with each other.

Bobby Calderwood (29:31):
And a lot of the times that turns out really well and capitalism, trade, and markets, I'm a huge proponent of those things. For good reason, I think we've seen historically that those things work really well and they work better than a planned economy. So there's this real tension in these complex systems between teleology, which is like, "I have a goal and I want to get to that goal." So that's kind of teleology. We're doing a lot of philosophy on this podcast [crosstalk 00:29:55]

Tim Berglund (29:55):
We are. I like [crosstalk 00:29:56]. Yeah. Teleology is an orientation towards a purpose.

Bobby Calderwood (29:59):
Yes. Orientation towards a purpose and a goal.

Tim Berglund (30:01):
There's an end or a goal.

Bobby Calderwood (30:02):
That's right. That's right. So beginning with the end in mind there's a lot of management thinkers that talk about that. So teleology on the one side and I put the kind of UML and a lot of these big design upfront kind of waterfall things in that teleology first camp, teleology is being the most important thing. We want to know where we're going.

Bobby Calderwood (30:19):
And then there's sort of the emergent... Hey, good things happen when you release the constraints in the system and allow the agents in that system to interact with each other however they choose. A lot of times those systems will come to a steady-state that is superior to anything you could've arrived that teleology. So like a planned economy is teleological, a market economy is emergent.

Bobby Calderwood (30:42):
And again, historically I mean we don't want to turn this to a political show, but historically we've seen that markets are superior in many ways to planned economies because the unforeseen steady states that our market economy can achieve outperform the narrow of a planned goal that I had for this planned economy.

Bobby Calderwood (31:03):
There's something. There's a real similar dynamic in software where there's this real tension between where I want to go and if I have too many constraints in the pursuit of that goal, I eliminate the possibility of these really good steady states that I couldn't have envisioned beforehand, but that would be a much better place for my company to be than the goal that I set at the beginning, right? So it's this real again. Epistemology, teleology, it's this real thorny question.

Bobby Calderwood (31:32):
You don't want to lock down that process too much. And I think that's why a lot of people from the agile community would say, "Good, get rid of those enterprise architects. Those guys were really cramping our style, right? They're harsh on our mellow. They put all these constraints on us and as a result, we couldn't do our best work."

Bobby Calderwood (31:46):
And that's fair. I think there's an absolute truth to that at some level. When you put constraints on a system, especially if you put the wrong constraints on a system, you end up tamping down a lot of those hypothetical potential good steady states that that emergent system could obtain and achieve.

Bobby Calderwood (32:04):
On the other hand, if you have no rules of the road, if you have this sort of pure laissez-faire let everyone do what they want the thing you could end up at sort of a local optimum, but vastly miss sort of the global optimum. So you've got to figure out a way to sort of set the right rules of the road, set just enough constraint and just enough rules to promote the incentive structures and the emergent states that you want and eliminate some of the emergent states that you don't want or that would be very suboptimal.

Bobby Calderwood (32:37):
And that's the role of regulation in the economy. But doing that well is a dark art, right? There's no scientific way necessarily to sort of think about these things and pick among the outcomes of these complex systems that you want and eliminate all the bad ones, right? You can't really do that because again, the nature of complex systems. Yeah.

Bobby Calderwood (33:00):
But you can set reasonable rules of the road that are based on values, that are based on principles, right? Not a tactical like, "Do this, do that," kind of a mechanical set of constraints, but rather a set of constraints that are around values and around principles and around the kind of these broad ideas that will guide us to the right set of emergent states in this complex system.

Bobby Calderwood (33:23):
So that's what I think... Well, we are really deep there for a second, bringing all the way back to event modeling. I think that's what we have here with event modeling is this sort of... Let's just set a couple of rules of the road. Let's put the business process and the business events and the narrative. Let's make that the central thing, right?

Bobby Calderwood (33:39):
Not the needs of the Silicon or the training of the engineers, but the business process. Let's make that the center of concern. Okay. That's kind of principle one of event modeling. So, when we do event modeling, we talk about the events first. So in event modeling, there's kind of these four visual words. There's commands, events, read models and interfaces or wireframes and that's it, right?

Bobby Calderwood (34:04):
There's just these four words, and there's a set of sort of structured data flows among these kinds of components, so it makes this sort of graph. And the first thing you do in the practice of event modeling is just to get the events up there. So they're represented by these orange sticky notes and you get them up and sequence them into kind of a plausible narrative. And you say, "Okay, these are the major business outcomes that we need to record in order to understand what's going on here."

Bobby Calderwood (34:29):
So we talk about how to record those things in terms of events, and then the other sort of principle or guiding value in event modeling is let's talk about the user's experience. Let's think about their flow. Let's think about kind of what they experience. So, the next thing we do in our event modeling practice is we talk about the wireframes. So we wireframe out the system.

Bobby Calderwood (34:50):
It's similar in principle to journey mapping, user journey mapping and customer journey mapping, but you think about what the users are experiencing. Okay. At this rough part of the chronology here, the timeline is running from left to right along the bottom. At this point in the event chronology, what is the user seeing or experiencing or doing? And we sort of map that out.

Bobby Calderwood (35:12):
So those are two of the big values. Let's think about the business narrative and let's think about the user's experience. Those are the two big values that set the rules of the road for event modeling, except these are the basic structures for event modeling. And then in order to get well specified and be able to talk about specific software components we need to build, then we talk about commands and read models.

Bobby Calderwood (35:38):
So commands are how users introduce change into the system. It's, "Do this for me, please," and the user clicks a button in the wireframe that sends a command to the system saying, "Hey, please do this for me. Place my order. Sign up for my account. Transfer money from one account to another." That's a command.

Bobby Calderwood (35:57):
And then that command we think about it and we process it and we reason about invariance and say, "I don't know. Do we have an account with this customer? Does this customer have an account with us? Do they have enough money in their account to cover this transfer?" We assert all the invariants there when we're sort of processing this command.

Bobby Calderwood (36:14):
And as a result of that command, we either return back to the interface like, "Nope, sorry. We're not going to do that." Or we write out an event and say, "Yep, money transferred. That event has happened. We're going to write it down." And then...

Bobby Calderwood (36:27):
So, that empowers our users. We empower our users by putting the commands on the board so that they can make a change. And then we inform our users via the read model. So as those events are happening, we aggregate those events up using technologies like Kafka and Kafka Streams especially or KSQL.

Tim Berglund (36:44):
Maybe some kind of stream processor like that [crosstalk 00:36:46]

Bobby Calderwood (36:46):
Yeah. Some sort of stream processing thing that can observe events going by and aggregate them, or take action on them.

Tim Berglund (36:52):
Provide efficient key lookups to certain aggregates. Sure.

Bobby Calderwood (36:55):
That's right.

Bobby Calderwood (36:55):
Yeah. One might need such a thing. So, you build this reading model in order to answer your users' questions about the state of the system. So I place my order and then I go check on the status of my order. The read models, what it informs me about. Has my order shipped? Is this order the right one? And all those questions that users may have. So we empower, we inform our users. Those are the other kinds of values that shape this.

Bobby Calderwood (37:23):
So we're focused on the business narrative, we're focused on users' experience. We empower our users, we inform our users and then we also need to add to integrate with the outside world because our system doesn't exist in a vacuum. So we need to be able to export or import state from other systems so there's a pattern in event modeling for external state import where we listen to the events of some foreign system and then incorporate those events into our own narratives and then take some action on that.

Bobby Calderwood (37:54):
So that's all about kind of listening to some foreign stream, again, Kafka streams and other related stream processing technologies are great for this because that upstream system doesn't even need to know that I care. They don't even need to know about me. They're just doing their thing going about their work the way that they're supposed to and generating events as a result. And then I can subscribe to those events and import their state into my world and inform what's going on in my system. So that's external [crosstalk 00:38:17]

Tim Berglund (38:17):
Yeah. That can be some terrible thing. You're scripting a file or something where you're doing CDC on a table, whatever it is. The mechanics of the integration happen and you turn that into events and there's a sense in which you have to maybe make sense of that external system's event in terms of the context of your system, right?

Bobby Calderwood (38:36):
That's right. So we call that translation. Adam calls that translation in his world. So if you're getting low-level events, so his canonical example is geo events. So your phone is just slinging off these geo coordinate events because it's kind of creepy and striking you everywhere you go, but it's throwing [crosstalk 00:38:54] off these geocodes. But those geocode by themselves don't mean anything to your application necessarily.

Bobby Calderwood (38:59):
So you have to translate those raw geo events via some sort of business logic to contextualize them and make them relevant to your application. So oftentimes that's something like geofencing where you say, "Hey when the customer enters my store or when the customer interests the hotel, that's a business event."

Bobby Calderwood (39:16):
So I'm translating these raw events via this geofencing piece of logic and then I emit this more contextual event to my kind of local event topic saying, "Okay. Customer showed up at the hotel, we should do something as a result of that. Or maybe we just record it and figure out what to do about it later," but that's an important thing in our domain. So yeah, exactly. There's sort of a translation process.

Bobby Calderwood (39:39):
And then the last pattern that goes with this value of we don't exist in a vacuum and we need to integrate with other systems is internal state export. I always mess that up. Internal state export where we've got stuff going on in our world. We can't just rely on someone else to listen to us and do the right thing. We need to go out and issue a command into some foreign system, right?

Bobby Calderwood (40:02):
So a lot of times it's like payment processors. That's kind of Adam's canonical example of this. Things are happening, users are checking out and doing all their stuff in our application, and then we've got to go talk to that dreaded payment gateway and it's a foreign system and we don't control it and we don't even know if it's event-oriented or not. We just need to go cause some change in that system.

Bobby Calderwood (40:21):
So right now, I use Stripe in my application event building right now, which we'll talk about in a minute. And that's how we treat Stripe exactly. We have stuff going on in our application. We reach out and talk to Stripe. Stripe is fantastic. And they do have an event-oriented interface, which is their webhooks. And then they basically call us back and say, "Yep. A good happened," or, "Oh, oops. There was an error, a bad thing happened."

Bobby Calderwood (40:42):
But we get an event back as a result of that. If you don't have an event-oriented foreign system, you just sort of call that command or web service or whatever. And then the result of that web services call will include a lot of information about whether it succeeded or failed, and then you write that down as an event locally as well.

Bobby Calderwood (40:59):
So we're still always thinking in terms of events, even when we're dealing with foreign systems sometimes we have to generate those events ourselves as a result of the web services call, whatever the result of taking that action was, but for good foreign systems like Stripe we can call them and count on the fact that they're going to sling an event back to us [inaudible 00:41:18]

Tim Berglund (41:19):
Good external foreign systems that do some sort of financial transaction in an event-driven way. I don't know if you know anything about that, but I know a guy I could introduce you to who could tell you all about it. And just if you're listening and didn't pick up on that, that guy's Bobby. He's the guy who knows everything about that.

Tim Berglund (41:44):
So anyway I imagine in the external state import and internal state export pieces, I mean, those get reduced to code at some point when you're interfacing with things that are not so well-behaved, meaning things which are synchronous, that's where you get those little impedance matching transformers doing polling or keeping tables of things and doing the adapting from the internals of the event-driven system to the unfortunate synchronous externals of the world.

Bobby Calderwood (42:16):
Yeah. And look, that's a reality and any solution which has to be a big bang, and you have to apply it all at once across every piece of your organization and rewrite every piece of code you've ever had, it's never going to work, right? That's not a viable strategy. So yeah, event modeling acknowledges sort of the realities of that. And in the internal state export pattern, Adam explicitly talks about, you've got these events going by, you actually build up a read model solely for the purpose.

Bobby Calderwood (42:42):
The read model is not for informing your users, this reading model is just kind of the to-do list of work that you need to do against some foreign system. And you may be able to do that piecewise. Each event that flows through you do your side effect and then you record the result of that side effect. But look, sometimes you've got to batch those things up and send them once a day as a file or...

Bobby Calderwood (43:02):
It's just some ugly thing, right? But building that read model is sort of a key part of that internal state export pattern because having that read model allows you to reason about the SLA and reason about are we doing this in terms of a single piece flow of each event or are we matching them up and sending them in micro-batches?

Bobby Calderwood (43:17):
Or are we doing a big file at the end of the day and we're reconciling some external ledger or something? These patterns are sort of sufficiently flexible in general that allows you to do a lot of things at the implementation level, which satisfy the design, but are ugly and dirty and you've got to sort of hack around the fact that you're dealing with some system that doesn't deal with this very well.

Bobby Calderwood (43:39):
Adam has a whole article about event modeling for legacy systems or event modeling for existing systems, where he talks about you can take an existing system and then start to sort of model it out, what it does today. And then start to sort of change and evolve that model and slice off features from that core system and start doing it yourself and building these little event-driven slices of the event model outside of that foreign system. While still relying on that legacy system to kind of do the bulk of the work or some part of the work that is regulated or whatever you need it to do.

Tim Berglund (44:14):
That's refactoring to events via event modeling.

Bobby Calderwood (44:18):
Yeah. Yeah, exactly. [crosstalk 00:44:19]

Tim Berglund (44:20):
And as you say that sounds like when it comes time to actually put this into practice a critical discipline, because you're usually not greenfield. Sometimes you are safer those times, but there's probably something else in the world you have to talk to. That's when it's probably not event-driven.

Bobby Calderwood (44:37):
Yeah. Sadly not yet. Not yet again.

Tim Berglund (44:39):
Not yet.

Bobby Calderwood (44:40):
Yeah. Even in that future date that we talked to at the beginning of the show when engineers are coming out of school and the first system they work on is an event driven system and that's the only way they know. At that future date, you don't have to worry about those things quite as much.

Tim Berglund (44:52):
That's right. And we'll be old men and talking about how we started with ERDs back in the day and the kids [crosstalk 00:44:59]

Bobby Calderwood (44:59):
That's right.

Tim Berglund (44:59):
... "What's an ERD." Let me tell you-

Bobby Calderwood (45:04):
Oh back in my day.

Tim Berglund (45:05):
Now there's, I guess a couple of things and we're coming up against time, which is... I think one of the challenges is that it's interesting to talk to you about things. So we are just going to do that. But there's the actual process you gave us the visual language, and then there is...

Tim Berglund (45:21):
You said this almost was born as a souped-up version of events storming and event storming if you don't know, is a thing a group of people does together. So, there are workshops.

Bobby Calderwood (45:32):
The workshops. Yeah.

Tim Berglund (45:33):
There's a workshop process of doing this and then there's the visual artifact. But I think if I may, and I just want to flag, yes, of course, there is a way that you do this and we'll link to resources. You might check out more about that, but I'd love to hear about where you've done this. What successes can you talk about that you've put this in practice?

Bobby Calderwood (45:55):
Yeah. Yeah. So the one I refer to over and over again in my [inaudible 00:46:00] talk is with a big tax preparation service we're doing the kind of helping them re-envision and re-engineer and re-design their tax platform kind of the core piece of their business. And we did event modeling with them. Really successful.

Bobby Calderwood (46:15):
We got all the kind of business folks in the room with the engineers and just from a communication and context sharing-perspective, the workshop itself was a huge success and it was great. The artifacts that come out of it we've got a big piece of butcher paper covered in sticky notes. Leave something to be desired, which is why we ended up building our own sort of event modeling platform, which we'll talk about in a second.

Bobby Calderwood (46:38):
But the artifact leaves something to be desired. We did end up digitizing that artifact in [Vizio 00:46:43] and making stories out of it and building to those stories. But that process at the end was really hard, but the actual workshop moment itself was great. We also did one with a pharmaceutical research company. They try to find new drugs and treatments for genetic and other types of disorders via this big data processing pipeline.

Bobby Calderwood (47:07):
The super cool company just raised a bunch of money, but they're really doing well. We went and did a workshop with them to do exactly this, kind of map out their domain. And part of the day was sort of this high level more event storming kind of exercise.

Bobby Calderwood (47:21):
And then part of it, we sort of got down to brass tacks on a particular piece of the system and then talked through, "Okay, how would we build this piece? And what would it look like and what would the user see?" And we did that via event modeling. So both really successful workshops. Really loved that.

Bobby Calderwood (47:34):
In my own software engineering for building our products internally at Evident Systems, that was all professional services staff. But on the product side of Evident Systems, we also use event modeling. So for our real-time payments system, for our event modeling product itself.

Bobby Calderwood (47:50):
We do event modeling, which is super meta and cool, but we actually build event modeling to talk about how to build event modeling in our event modeling platform. So we do all of that kind of stuff. We eat our own dog food with the product itself to do those things.

Tim Berglund (48:05):
You're building that tool. So tell us about that tool.

Bobby Calderwood (48:07):
Yeah. Yeah. So as I said, sort of getting to the end of event modeling workshop and having seen a bunch of people try to take the panoramic mobile shot, a mobile photo of the big piece of butcher paper covered in sticky notes, [crosstalk 00:48:25]

Tim Berglund (48:25):
And then you're trying to get to Vizio. So it's pretty [crosstalk 00:48:27]

Bobby Calderwood (48:27):
Yeah, that's right. And then it's two or three BAs for three days just trying to digitize this thing and squinting and looking at other people's handwriting through this blurry panoramic shot. It's not great, right? It leaves something to be desired. Event storming was sort of meant to explicitly be a co-located workshop experience.

Bobby Calderwood (48:46):
When Adam designed event modeling, it was a little looser. He was like, "No, this can be done effectively, remotely." This can be done effectively with everyone sitting on their own computer kind of in some sort of collaborative environment building this event model. So our experience with some of these successes that we talked about and understanding that part of the philosophy from Adam that this doesn't have to be an in-person workshop, that this can be a thing that evolves over time asynchronously as well we built our own event modeling platform called oNote.

Bobby Calderwood (49:19):
So, oNote.com. It's free. You can sign up for it, please go do. It's great. We love it. It was something that we built just to scratch our own itch and to help the event modeling community. But as we got into it and started building it we were like, "This has huge potential." Right? So a lot of folks were using a kind of collaborative whiteboards for doing these event models and those are great.

Bobby Calderwood (49:43):
They're great sort of general-purpose tools, but oNote is a more constrained modeling environment so you can't end up with an invalid event model, right? You can only draw arrows from things that make sense to other things that make sense, right? You can't make invalid connections-

Tim Berglund (49:57):
No. It enforces semantics of it.

Bobby Calderwood (50:00):
Yeah. It enforces the semantics, the actual structure in terms of the visuals, the vertical space on canvas is constrained so that only events can go down in the bottom section and only wireframes can go in that top section, and only commands and remodels can go in this kind of center, spinal column section.

Bobby Calderwood (50:18):
Anyway, so we constrain the model in canvas, but then behind the scenes of that, we've got sort of the visual projection of the event model, the visual piece of it, but behind the scenes, we're building this data structure, right? It's just a data structure that we have in memory. And then we store it to a disk locally in your browser and using this data structure, we can do all kinds of really cool stuff, right. So we can...

Bobby Calderwood (50:41):
So as you move from the design phase where we can do sort of that collaborative design, where we're all working on oNote together, collaboration is a feature we're going to release very soon within the next couple of weeks. Probably before this podcast episode drops we will have a collaborative environment. But then as you transition from the design kind of workshop moment into this implementation phase we can generate code.

Bobby Calderwood (51:04):
So we can actually generate based on the commands and remodels and the schema that you associate with those things. We can generate GraphQL, gRPC, OpenAPI Swagger, maybe even AsyncAPI. We're looking at that as well. So we can generate kind of the interface definition code, and then you can take that specification and use it in your toolchain of choice if it's Java or JavaScript or Golang whatever.

Bobby Calderwood (51:26):
There are already existing toolchains for ingesting that interface definition language artifact and getting to runtime code. We can integrate with Confluent Cloud. That's one of our targets is being able to say, "Hey, we've got all these lanes of events here, we can just create a topic for each lane, and then we can put the schemas that we define in the event model into the schema registry and confluence cloud."

Bobby Calderwood (51:52):
So there's a lot of integration level stuff we can do to speed the implementation process along. Another cool one there is rather than having to sort of separate user story generation exercise or sort of process that you do in agile, or ritual, sometimes they call it, we can actually just carve up this diagram into the set of tasks that are required to implement this diagram.

Bobby Calderwood (52:13):
Adams calls those slices. They're kind of these vertical slices of functionality that have a well-defined contract. They have well-defined inputs, well-defined outputs, and that represents sort of a discreet unit of developer or engineering work.

Bobby Calderwood (52:26):
And so rather than having a separate user stories exercise during your agile planning, we can carve this diagram up, dump it into Jira for you as you burn those tasks down in Jira or whatever you have issues or wherever you track your work. As you're burning those things down, we can show in an overlay or something on top of your diagram, "Here's the progress we've made on this," right?

Bobby Calderwood (52:46):
So, we can start to really integrate with the design process, the implementation process, and then during operations, as I said, we can integrate with your data platform, pull out some metrics and monitoring, and actually maybe start to visualize the data flow through your system in real-time. You can start to see people are stuck at this part of our checkout flow, and we can see that on the metrics visually and all that stuff.

Bobby Calderwood (53:09):
So there's a lot we can do with this event model that's not just about design. It's valuable from the design through the implementation, into operations. Again, another one is [inaudible 00:53:20] management, right? Something bad has happened, our system's broken. The people who built the system aren't necessarily on the call so we're missing a lot of that mental context.

Bobby Calderwood (53:29):
Can we use this artifact, this design artifacts, to reason about where the system's broken to visualize the metrics and see what's happening, and then fix the system without needing the original team that has all that shared tribal knowledge in their heads. Anyway, there're a lot of really exciting things on our roadmap.

Bobby Calderwood (53:47):
Right now oNote is sort of just the modeling canvas piece of it but we have a roadmap that gets it a lot farther along and becomes a sort of an integral part of accelerating the software engineering process. It helps you to adhere to these broad rules of the road that we sort of outlined earlier, that if all the systems in your enterprise are just sort of following these basic rules of generating events and informing their users, empowering their users, caring about their journey and having structured ways of integrating with other systems.

Bobby Calderwood (54:19):
If everyone's following those rules, you don't need real strict architecture from the outside. Those rules are enough to where you end up with this really great emergent type of thing where people are able to use these data and build extensions and really cool alternative views of these data without having to be so teleological about it, right?

Bobby Calderwood (54:41):
You can be emergent, you can be iterative and everyone kind of doing the right thing and you get to this really cool steady-state emergent world, the ecosystem, but you do have these broad rules of the road so you don't just end up with this senseless chaos, right? You've solved a lot of the hard distributed system problems. You've solved a lot of the consistency and all of those difficult things just by adhering to the rules baked into event modeling.

Tim Berglund (55:04):
My guest today has been Bobby Calderwood. Bobby, thanks for being a part of Streaming Audio.

Bobby Calderwood (55:08):
Thank you.

Tim Berglund (55:09):
Hey, you know what you get for listening to the end? Some free confluent cloud. Use the promo code 60PDCAST. That's 6-0-P-D-C-A-S-T to get an additional $60 of free confluent cloud usage. Be sure to activate it by December 31st, 2021, and use it within 90 days after activation. And any unused promo value on the expiration date will be forfeit, and there is a limited number of codes available so don't miss out.

Tim Berglund (55:37):
Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me @tlberglund on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community, Slack. There's a Slack signup link in the show notes if you'd like to join.

Tim Berglund (55:59):
And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribed through Apple Podcasts, be sure to leave us a review there. That helps other people discover us, which we think is a good thing. So thanks for your support and we'll see you next time.