It’s Time for Streaming to Have a Maturity Model ft. Nick Dearden

Nick Dearden explains the five stages of streaming maturity. They are not denial, anger, bargaining, depression and acceptance—that’s the Kübler-Ross model, and it’s for bad things. This one is for awesome things, and takes you from the first streaming project you ever build all the way to a state where an entire organization is transformed to think in terms of real-time, event-driven systems. If you have ever found yourself trying to get streaming technology adopted, this episode is for you!

EPISODE LINKS

• Five Stages to Streaming Platform Adoption

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

Copy Transcript Copied

SPEAKER_00 0:00

Between your first streaming application and the time that your entire company is transformed around an event streaming platform that forms the DNA of the whole organization, you know, there's probably a few steps. Nick Dearden and I discuss the streaming maturity model on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Nick, welcome to Streaming Audio. Thanks, Tim. It's great to be back. Always good to have you on the show. We have been talking uh recently about streaming as a paradigm shift and streaming as a set of development practices. And you know, I talk to people and you talk to people, and like everybody knows streaming is the thing they're supposed to be doing next. Uh, but everybody also rightly has the idea that it's kind of big and it it clearly changes a lot about the way that we're gonna write software and we're gonna organize ourselves. And it feels like this big, scary ocean that has to get boiled. You know, if we properly articulate our vision for what the world should be like, sometimes people kind of say, Oh, that's huge. Well, how do I even start? I mean, you ever you see that?

SPEAKER_01 1:28

All the time. That's that's absolutely right. I think once I've certainly been through this journey personally, once you start to wrap your head around the potential of the things we talk about as becoming, you know, uh driving everything in your business around this event streaming platform. And what that really means, it does seem like quite a scary set of change. And not to be all sort of business schooly about it. Though there is a set of patterns to that, and like any transformational journey, you get to start from wherever you need to start from. You have to go a step at a time in a direction you know where the direction is until you get to the destination that makes sense for you. I think we can probably usefully uh drill into what those stages look like.

SPEAKER_00 2:17

Yeah, because there's there's kind of uh it almost goes from awareness of uh, hey, that's a good idea, to you know, there is there is some streaming nirvana place that we can really streamy, streamy end state that that maybe we can come up with some account of. But let's just start. Like, what do you where do you see people at the beginning? What what is what is the beginning of the streaming journey?

SPEAKER_01 2:43

Aaron Powell Well, I I'd actually preface the answer to that by saying there's there's probably a few different starting points, depending on where you come from, and what your initial motivations for discovering Kafka and the Confluent platform are all about. So usually, though, what's common about those things is that it starts on a relatively small scale within some small area of potentially a very large organization. And I guess we're going to talk mostly about organizations that are not what we would traditionally classify as digital natives, who might be free to operate with only 18 months worth of legacy systems that they've accumulated, you know, and they don't have a lot of these challenges, right? So we'll just put those lucky folks to one side for a minute and say you start out because you know you have an area where you think adopting streaming might be able to help you. And perhaps you you've only got the vaguest inkling at this time of that Nirvana streamy, streamy place, as you so nicely called it. Uh maybe all you're looking to do is to either I'd say there's there's two places here. One is you might just be looking for a way to unlock some data that you already have in your systems. So that might be you have some set of mainframe applications, for example, if you've been around a while. Getting data out of mainframes can be quite challenging. You know, a lot of time you have to pay for every additional call that you make into the mainframe, and so that puts a you know a procedural block on you doing new projects. You have to go to the committee that agrees to spend money on getting more data out of the mainframe before you can begin. You know, that kind of thing slows folks down and becomes quite frustrating. Or perhaps you have a Greenfield project where you have some new initiative that wants to take advantage of newer technologies and approaches like event streaming. Perhaps it's some IoT uh exercise. One example I might cite there is I know we've done some work with a couple of different insurance companies who want to offer their car insurance customers lower premiums if they're willing to attach a small device to their vehicle that you know reports what good drivers they are, for example. Those systems, of course, generate a lot of event-centric data in real time that needs to be collected. So these are essentially new ingestion kind of projects. What you're doing is just scooping up all that new real-time data to do something with it. And the only integration or touch points that you might have with your existing systems then are about pulling some data out of them to be combined with this event streaming. So you might need the customer profile and then something that monitors their driving behavior and occasionally updates the profile to say how good, hopefully, uh a driver they are.

SPEAKER_00 5:36

Make sense? Right. It does. So the I'm trying to think of um what it feels like to do this, right? Like from the first person experience of building this. And those examples you gave were, I think, a s across a spectrum of aggressiveness as business initiatives. Like the connected car thing is huge. The I have this legacy system and I need to get stuff out of it is small. But what they have in common is that they're they're kind of isolated applications. So this could feel like, if I understand you, I'm a senior developer or I'm kind of an application architect type of person. I'm not, you know, the enterprise architect reporting to the CIO, setting the tone for the whole company. But I'm building a thing and it's self-contained and it's inside this part of the business. And it's okay if other people might not even know about it in other parts of the company. We're just doing this thing. And maybe, you know, my experience as that senior developer or application architect is even colored by the, oh hey, streaming's cool. Um, I should try some streaming. But it's it's the kind of thing where that kind of mindset is potentially a sufficient driver for the adoption of the technology. In other words, it's relatively tactical at this point.

SPEAKER_01 6:52

Absolutely it is. You you don't have to climb the hill of convincing everyone in your entire organization that they should solve every problem in a certain way. You're just going to prove that you can do one thing using this kind of new approach, and that success will demonstrate itself and help you then evangelize that, hey, maybe this is a good idea, a good set of approaches and tools and techniques that we can use to solve some adjacent problems. So it's very much contained within that one team at the beginning. As you say, a couple of senior developers, maybe an associated DevOps person, or an architect, and maybe a business sponsor who's driving that project, but maybe doesn't necessarily have strong opinions about the actual technological implementation. Trevor Burrus, Jr.

SPEAKER_00 7:37

Right. It's more like there's this thing that they need done, and oh, okay, the developers want to do it in this way, fine. So it's uh it's a it's a foot in the door for streaming. And you get technology early adopters who do this kind of thing. And you know, those technology early adopters happen to be like some of my best friends uh in the business. This is this is not a bad thing at all, but that's kind of what drives it at this stage.

SPEAKER_01 8:03

Aaron Powell That's right. And it doesn't it also bounds the scope of the risk of doing something different. I think that's an important thing to call out. You can start small, maybe you're thinking big, but you can still act quickly here. And if you need to experiment a few iterations, hey, it's not the end of the world because you've bounded your scope to not boiling the ocean. Aaron Powell Right.

SPEAKER_00 8:23

So where does it go from here? What's the next step?

SPEAKER_01 8:27

So the next step is maybe we need to get into I I think when we talk about this within Confluent at least, we call the next step production streaming. That's that's usually the sort of the label for it, which is we've proven to ourselves that this is gonna work for at least an initial use case or a collection of use cases. What we want to do now is worry perhaps a little bit more about some other concerns, though. We're not proving technically we can do this anymore. Now it's about where the technology folks and the business folks are gonna have a bit of a coming together of the minds where we can demonstrate to our application owners there's some clear business benefit to having real-time data. It could be about improving the customer experience, it can be about enabling those new business models, uh, or of course it can be about ways to do compliance and risk management. They're always the three, this is the business schooly part. I apologize to your listeners, Tim, ahead of time. But there's only three, three drivers for any project, aren't there? You can save money, you can make money, or you can reduce the risk attached to your money.

SPEAKER_00 9:31

That's exactly right. And you know what? Since this is a technical podcast, and we do have a technical audience filled mostly with developers, it's not bad to learn these things. Um I was just listening in another podcast, uh Software Engineering Daily, earlier today, before this recording, and Jeff Myerson, the host, was making the point that, you know, you might want to move up in the organization at some point, so it's actually good to learn this stuff. So that trifecta, everybody. Uh, make money, save money, reduce risk, attach to money. When you're trying to build something, you know, you you've got your own motivations for building the thing, and maybe they're not one of those things. But if you can't articulate uh your initiative in those terms, or even partnering with somebody in the business and articulate it in those terms, you're not going to get it done beyond that first stage, right? Like if you want to, if you want to go bigger, uh it has to be here because this is now business decision makers, and they have an ethical duty to make decisions that are that are broadly constrained by those three things. Aaron Ross Powell And a legal duty in many countries where yes, yes. I say ethical, I should use the magical word fiduciary. But I don't want to sound like I think I'm a lawyer, because I I know I'm not one. But honestly, yeah, you do. Yeah, that's that's that's a very real thing.

SPEAKER_01 10:48

Aaron Powell That's right. And so we talked before about the kind of project where you begin, and you can broaden the scope of those kinds of initiatives. So it can be about just liberating the data that's stuck in external stores or the legacy systems that you have. Maybe they're mainframe-based or MQ systems, or they are we often talk about um databases that cannot support thousands more queries per second, right? And you can't scale up that uh uh source system. But what you can do is liberate its data as a stream of change events and make it available to many more systems. And this is the foundation of uh what anyone who wants to do a customer 360 or holistic view of the customer kind of project will find themselves doing almost immediately. All those projects fall into this kind of category. And that's largely in the saving money camp. Right? We can do this thing, or we can make possible um the new project that you want, and we can do it in a much more cost-effective way than perhaps we could have done using the same tools and techniques we were using before.

SPEAKER_00 11:54

And how would this is still that second just second step up, or have you?

SPEAKER_01 11:58

No, I think it is the second step. What you're gonna do this at this point as a technician, technology enthusiast like we are here, is you're gonna teach your business partners the value of event-centric thinking. And just show them the value in one of those three terms that we we described, saving money, making money, and protecting money, of what this kind of technology can do for them. You have an example on hand now to point to, could be in your group or in an adjacent group where they built- You built a thing and it's being successful. That's right. And you can point to it and say, look, this is not theory, this is practice. We have practiced this. Here's the result. Now let's work together, Ms. Business Partner, on what that means for the projects that you're particularly interested in advancing. How about if we help you imagine them as something centered around a real-time flow of events? Because very often, you know, one of the things I found we accidentally do as technologists is we teach our business partners and sponsors to think in terms of what we currently believe to be possible. And we remove their capacity almost to imagine new things. Because we train them by just beating them down every time. No, no, you're being stupid. You know, you can't possibly do that. We can't be done for XYZ reasons. Right. Right? We've all seen it happen. Nobody intends to do that. None of us intend to be that. We all want to be enablers, don't we? Uh I actually think that's one of the things that really attracts people to what we're doing at Confluent, is it lets us unleash that creativity that we have and say, look, this is possible. We can do this, we can transform the business, we can make that project a reality that you thought was only a dream before.

SPEAKER_00 13:50

Yeah.

SPEAKER_01 13:50

It's super, super powerful. And so if you teach your business partner that there is a new art of the possible and they can do these things, then all these projects that perhaps you'd never even really heard of come crawling out of the woodwork. All these ideas get dusted off and potentially reimagined. Okay, and you can get more of these quick wins under your belt quite quickly. So at this point, what you're doing is broadening the constituency, not only to your business partners, but maybe to some other teams of architects and developers, maybe some more operators. And they need to gain their own skills in thinking of problems in this way, and also of building these streaming applications that you and I are so often talking about. So this is really a skills and experience gathering phase where you're teaching each other, you know, drawing in external information wherever you can get it. You find the case studies, you go and listen to the excellent folks who've talked at places like Kafka Summit about the journeys that they've been on as an organization. I see that. I know you're on the program committee, Tim, you must see this even more than I do. But there's a there's a big upswell in the number of talks that get submitted to things like Kafka Summit for, hey, let me tell you about the journey that we went on as an organization.

SPEAKER_00 15:08

Yes. I love those kind of talks, by the way. Um I mean, it's it's we must have also, you know, here's a deep dive into the consumer group protocol. But uh it it's just encouraging to me that we're at a point in our own, you know, the maturity of the technology that those stories exist. And it's it's started to change businesses. Which is is not quite where we are in the second phase. The first phase, we built a thing. Yeah. The second phase, it hasn't really transformed the business, but it sounds like to summarize it, like we're starting to educate the business in the value of doing more of this.

SPEAKER_01 15:45

Yes. Uh maybe we could, if we really needed a a one-line summary, we could call it the internal evangelism phase. Where you you show the people you did the thing and the thing was good. And then they they recognize that it was good and they do some as well. Um and that leads us towards we're we're blended, they're not discrete phases, obviously. They're a blended uh phasing of things that happen. But as time goes on, what happens is that these streaming projects that you have gain a couple of additional characteristics. So I think the first one is that the mission criticality of them rises. As you bring more applications into this kind of paradigm, then what happens is you move away perhaps from thinking about managing individual applications and thinking more about, hey, you know, maybe as an organization we could share all this stuff, right? It's not just about me and my business unit, but there's value in it working across even lines of business or other departmental silos that exist. And what we're doing now is we're thinking about we need completeness, we need the emergence of some sort of governance processes around what goes through here, we need SLAs for what when things are made available. This is perhaps when you start to consider a move to having a shared services team or someone of that nature that operates the streaming platform on behalf of many application groups. So what you're doing now is saying rather than the app teams operating their own infrastructure that supports only their apps, now we're going to move that responsibility to a shared group that can do it for many of us. And that means we can centralize the skills and expertise that we have. And so what you'll often see is as these kind of initiatives get more awareness, there's some sort of new team that gets spun up. Perhaps it's in a DevOps group or something of that nature. And you have to be able to negotiate what the contract between that shared services team is and the various groups that are going to make use of the shared streaming service. And that can be okay, what's the process for provisioning a new topic of data? Well, what do we need from a governance and compliance perspective so we understand what's in it? So we can appropriately set sharing and security rules, we understand where the PII is in here. You know, we maybe we operate internationally, we've got sovereignty rules to be concerned about. There's always a specter of GDPR and other such initiatives out there.

SPEAKER_00 18:16

There's a schema, maybe that that sort of thing could could exist at this point.

SPEAKER_01 18:20

Aaron Ross Powell That's right. And then maybe, oh, maybe we should use all those schemas. You know, we have a catalogue of them, right? We need to make other developers aware of the catalog of what we already have, so they don't just reinvent the wheel. We can leverage each other to some extent. That's what having a shared service is kind of about.

SPEAKER_00 18:37

Right. And so for the benefit of the listener, we are not making this up uh at this point, right? This is a thing that we see uh with customers and early adopters of Apache Kafka and customers of Confluent. It's a very real thing. Like the Kafka Summit San Francisco, uh, one of the keynotes was by a gentleman named Chris D'Agostino, who talked about what's going on at Capital One. And Capital One may even be more mature than this, but he was talking about their shared event infrastructure. Contracts are enforced, and so you get that kind of shared or that kind of architectural layer at uh once you've got the shared infrastructure, that architectural layer sort of emerges.

SPEAKER_01 19:19

Absolutely, it does, and uh just to echo your point. Firstly, Chris's talk is is a great watch on YouTube for anyone who didn't see it the first time around. I'm sure you'll put it in the show notes. Yes. Uh and the other thing I would add is this is absolutely you know, observations that I spend a lot more time recently working with our pre-sales and our post-sales and our professional services groups. This is absolutely a set of patterns that we are drawing based on the experiences of many, many actual customers and organizations doing these things for real. Yeah. So however far you think you are from this state today, trust me that these organizations like Chris is that have been out there, they started from where you are now. And you know, they they are the great thing about people like Chris, he's willing to stand up and share his experience of what it took to get from zero to where they are now, way down the road of streaming maturity, let's say.

SPEAKER_00 20:21

And I yet yesterday met with a customer uh and they they haven't done a public keynote, so I I won't disclose their name, but same exact story. Shared infrastructure team, API wrapping it, built-in schema management, governance, all that stuff. It has to happen. So when you're there, where do you go from there? Because that seems fairly mature, but what what I would I would call the next outcome.

SPEAKER_01 20:45

Um maybe this is a would be a rather colloquial way of saying it, but I'd call it the snowball phase. So this is where now many other groups are of aware that there is this shared service thing. We've done some education of the possibilities, and suddenly lots of other groups in your organization want to come and try it out. And primarily they come not necessarily because they've been sort of philosophically convinced yet, but because they see an easy way to get access to lots of real-time data about the business they could never get before. And what happens then is when they come and build their applications that want to make use of that data, they almost invariably end up contributing some real-time data back as a as a byproduct or a result of what they're doing. Sure. When you and I talk about the more technical aspects of streaming applications, usually, you know, it's we can describe it as a process or a set of processes which read some data in real time, they do something, and some more data comes out of the backside. And so as more and more of these apps come and they contribute their own data to the pool of what's available, it increases the gravity well around this thing and it drags more people in faster. And what organizations will typically report is that this The phase where, in maybe just a six-month uh timeframe, dozens and dozens of projects and teams will suddenly turn up asking to be onboarded to the shared services platform. And if you're not prepared for that at this phase, that can almost be a risk to your overall success. Because if you're not prepared to say, okay, we understand how to do some governance and some capacity planning and some SLA provision and reporting across all these things that are going to come at us real quick, then you may not be successful. You may not be able to provide that shared service in a reliable way. And that's where making sure that you have the expertise in-house or you can supplement what you already know in some way, making sure you understand the ins and outs of the operation of such a kind of platform are key to being able to go forward successfully.

SPEAKER_00 22:54

Now, let me play Devil's Advocate a little bit at this point. So imagine a time when you and I were younger men and people uh younger than that. And people were using terms like open systems and uh you know departmental databases and and you know, relational databases were the disruptive uh data infrastructure technology because it wasn't a mainframe and it was easier to get access to, and you know, the mainframe was this this thing in a tower guarded by a priesthood and the data was all locked up. Um it seems like one might have made the the argument then that those systems could also have led to this new era of sharing. And we know in the time between then and now that that that story was uh not always great. Like relational databases led to a proliferation of siloed uh data that was very expensive to connect and giant cottage industry, it's not even fair to call it a cottage. It's like a more of an 18-room villa than a than a cottage. But uh the the big giant cottage ETL industry grew up around getting all that data integrated. So I think we would have said the same thing then that we're saying now. Why is it more likely to be true now?

SPEAKER_01 24:21

Aaron Ross Powell Well, in the general sense, I think you're right that the technology pendulum swings back and forth, doesn't it? We went away from an era of centralization to an era of very much decentralized data. And the cost that came with that, because of the technologies we ha had to hand in the day, uh, was this fragmentation or siloization, if that be a word. Um silofication. Whatever the word.

SPEAKER_00 24:47

Yeah, I like that one.

SPEAKER_01 24:48

We're gonna go silofication. We'll go with that and press ahead. Um and so yeah, it those problems got created as a as an accidental byproduct because those were the tools that we had at the time. And we thought the trade-off was worth it. We maybe didn't foresee the ultimate consequences, but I think if we looked back, we'd probably still say it was worth it. You know, we could build all those applications faster, even though there were these hidden costs that resulted in a business where I've spent 20 years working in sort of data warehousing and analytics, and where we went out and tried to re-gather all of that data and reintegrate it in some painful way so that we could get this holistic view. And in my mind, what's changed with the advent of the streaming platform is that it's it's an almost magical hybrid of centralization and I don't want to say departmentalization, but localization. So you can think of this as a shared service, you can implement a shared service that means there is no duplication if you don't want any. But you still have, because now we've understood how to unlock the power of distributed systems for folks. That's you know the idea that powered the Hadoop movement for such a long time. We don't have to be constrained by those giant monoliths with the priesthoods that you mentioned. Right. You know, we can build systems where, oh, we need some more capacity. Okay, I plug in five more cheap boxes and off we go. Uh that is, I think that revolution is what is powering this idea. So any number of teams can now come and consume data that is already present as an event stream while incurring very, very marginal additional uh hardware costs. And even if you do need to plug some more boxes, well, that's easy and cheap to do. I think that's the difference that I would call out.

unknown 26:37

Yeah.

SPEAKER_01 26:39

What we're gonna see in the next few years is you know, stronger patterns of governance around that kind of operating model, I think.

SPEAKER_00 26:45

Yeah. And uh I would add, uh, reversing my own devil's advocacy, so we've we've learned how to scale the data infrastructure. We we were not uh that story was not great um with relational databases. The other thing that we're doing now with event streaming is modeling data as immutable objects. Events are immutable objects. And that this is a longer story, but that unlocks a bunch of simplifying assumptions that let you effectively copy things without worrying about data integrity. You know, so we're able to scale the infrastructure itself, and the fact that what we're doing is logging immutable events means it ends up being easier to share things. And anybody who's tried like concurrent multi-core programming knows that when your data structures are immutable, sharing them between threads is dramatically easier. Same thing with data items and distributed data infrastructure. It's the same principle applies. So I think that's a that's an important point.

SPEAKER_01 27:52

I think any advocate of EDA or any other flavor of being event-driven would would say something very similar. Your system of record should always be what actually happened. And there's a uh interestingly, a byproduct series of events. You might call it the audit log or the audit trail. There's an event which is that I, as a system, did something in response to some other events that were you might think of as real-world events. And then the next system ingested those, it took some action, which can itself be recorded as an event. So there's this interesting meta spiral that you can get into about what is an event.

SPEAKER_00 28:33

Yeah. Uh there is some inciting action, but then any computation and enhancement and extra value you add to that thing from then on is still an event. So uh where do we go from there? What's the next level of maturity?

SPEAKER_01 28:49

Aaron Powell, I think beyond here, as as we we move towards that nirvana, uh we're gonna talk about more about now. You're really into the era of I don't want to say business transformation, that's too grandiose, but you might need to rethink how you align your technology initiatives and your business initiatives to get the maximum value out of this shared infrastructure that you now have. Because as we know, you know, most org charts and systems design charts are pretty strongly parallel each other in an organization. And teams and initiatives grow up around systems. And those systems grow up around the teams, and it's a sort of a self-reinforcing structure. And when you do something that breaks that model, you open the door to yourself to be able to change the way you structure your actual business activities. Uh and to be more, for example, customer-centric rather than centered around one particular type of interaction with a customer, like uh a help desk, for example. That's now becomes uh an activity that makes events about customers. But maybe you have a technology group whose focus is we do all the customer events. And so that then allows their attached business groups to start to think about themselves more as, hey, we're the customer experience group. We're not the group that runs the help desk. That might be something that we do, you know, sort of along the way. But really, our focus is about how we can improve customer experience across the board and generate better revenue streams as a result of that. So that's a very liberating thing, but it's a thing that takes some time and it takes some high-level sponsorship. So by this stage, you've got your shared services up and running, you've got this snowball effect in full flight. And I think these conversations then start to spark. As all this activity inevitably comes to the attention of your sea-level executives, uh, and they're going to start to have these conversations about we can really take advantage of what we see now by changing ourselves in a more fundamental way to unlock even further value from the kind of technological approach that we're adopting.

SPEAKER_00 30:59

So at this point, this is really you know, the subjective experience of the developer or the architect is really that culturally, you know, the engineering culture here is event-based, and there is a shared event infrastructure that we build our applications off of. But really, what this phase is about is a couple steps above that. Like maybe from an engineering standpoint, it's felt like that for the last two years. But when we're at the streaming nirvana, it really is executive leadership sees the business as a thing which processes events, right? They they see their business in an event-centric way, and that's aligned at a deep level with the technology infrastructure as well.

SPEAKER_01 31:41

That's absolutely a great summary of what I was trying to say there. I think uh we probably have some great quotes from Kerry Joel at Royal Bank of Canada, actually about the moment when that transformation had suddenly become apparent within their organization. Yeah.

unknown 31:58

Yeah.

SPEAKER_00 31:58

Yeah, this is like you said, these things blend together. It's and and I think we've defined a five-stage streaming maturity model here, just to use that language. And the absolute worst thing anybody could do would be to have like debates about are we in stage four or stage three? I think when we passed over to stage three, it's like listening to movie buffs talk about when the transition between uh act two and act three was. You know, it's it's you could just watch the movie too. Uh so you know the work of building streaming, you're pursuing this goal, and the maturity model is a very helpful schema, but these things blend. And and you may not even know uh when you get there. But you you look back and you realize, oh wait, yeah, we're no, the business is aligned. This is what we do. That's right.

SPEAKER_01 32:43

I think we're we've reached a stage now where we're we're talking about it in very sort of abstract, high-level terms. But what's interesting to me is that that conversation started just 20 minutes ago with a discussion of, hey, I do my pilot project. And I wasn't thinking in these terms at all. What's interesting is how it evolves along that journey from just uh three or four of us, we've got this Friday afternoon project, you know, we think we can prototype doing something in a different way. Uh it can really, the snowball does start to go faster and faster. Uh it's and it's super exciting for us here at Confluent to be in the in the center of that. That's one of the things that keeps me very engaged in what we're doing.

SPEAKER_00 33:27

Indeed. Uh so let's real quick summarize the five stages. We had uh the beginning, you build a thing and you're trying to solve some particular problem, and maybe nobody knows about it outside your team. Yep. Next phase.

SPEAKER_01 33:40

Step two, I would say, is when you're talking about connecting systems together via event streams. So this is still we've still got small teams. Maybe what we're doing is unlocking data from an existing silo. And then we can use that to chain together two or three different systems that need each to operate on with some awareness of what a preceding system has done. Got it.

SPEAKER_00 34:02

So that's connectivity via event streams, let's call it. There you go. Step three, shared infrastructure, right?

SPEAKER_01 34:10

Let's call step three um event-driven applications. Okay. So where in step two you're saying, you know, I have this stream of events and I can use that to get data between systems. Probably at this stage your your thought process hasn't fully shifted to making event-driven systems. I think there's an architectural change in the way you think at this point, which is important to call out, which is that you understand that centering your system around operating on a stream of events, that's that's a different and more valuable idea than saying, hey, there's some data over there, and the only way I can get it is by um getting a series of change events fed to me. Got it. Does that distinction make sense? It absolutely does. Absolutely does. And from there, step four. And then we're going to move up to the shared service. Uh and this is where the flywheel, or the snowball or gravity well, call it what you will, where that process kicks in. So this is full shared service, governance, you've got senior management visibility into what you're doing. It's you've got the mission criticality of the use cases that are being adopted. And you're breaking down those silos that exist within your technology spaghetti diagram, you know, that you would draw out of all the things that you have today.

SPEAKER_00 35:26

Right. And finally, stage five is that long hoped for alignment of the structure of the business and the thinking about the business and the technology infrastructure that that undergirds the business, all that at once. Aaron Ross Powell The event-driven business, yes. Yeah, that's quite a goal. It certainly is, but it's an exciting one. It is. It's good to be a part of. Well, my guest today has been Nick Dearden. Nick, thanks for being a part of Streaming Audio.

SPEAKER_01 35:54

It's always a pleasure, Tim. I promise to talk about more APIs and code next time. And there you have it.

SPEAKER_00 36:00

I hope that was helpful to you. If you've got questions, you can ask me at TL Burgland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on any of our YouTube videos. Your question might be featured on the next episode of Streaming Audio. And feel free to subscribe to our YouTube channel and this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast and just generally helps us get the word out. We appreciate your support. See you next time.