From Monolith to Microservices with Sam Newman

Author Sam Newman catches up with Tim Berglund (Senior Director of Developer Advocacy, Confluent) in the virtual studio on what microservices are, how they work, the drawbacks of microservices, what splitting the monolith looks like, and patterns to look for. 

The pair talk through Sam's book “Monolith to Microservices” chapter by chapter, looking at key components of microservices in more detail. Sam also walks through database decomposition, integrating with new technology, and performing joins in event streaming architecture. Lastly, Sam shares what he’s excited for in the future, which includes “Monolith to Microservices Volume II.”

EPISODE LINKS

• Monolith to Microservices
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage* 

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_00 0:06

You know, it's one thing to prevent the vision of microservices communicating through Kafka topics. But how do you actually get there? Sam gets us started on that journey. Today's episode of Streaming Audio, a podcast about Kafka, on the cloud. Hello and welcome to another episode of Streaming Audio. I am, as always, your host, Tim Bergland, and I'm joined with me in the virtual studio from across an ocean, Sam Newman. Sam, welcome to Streaming Audio.

SPEAKER_01 0:41

Thanks for having me, Tim.

SPEAKER_00 0:43

Great to have you here. Now tell us a little bit about yourself, Sam. And I could I can certainly say you are speaker, author, consultant, trainer, thinker about microservices. What have you been thinking about recently and um what got you there?

SPEAKER_01 1:01

I guess I've I wrapped up uh a new book at the end of last year called uh uh Monolith to microservices, which was me taking a look at what what are all the nasty things that have to happen if you want to take an existing system and sort of break it apart into microservice architecture. And I kind of put that to the bed at the end of last year, but obviously the book's just getting into people's hands now, so that's been nice. I'll be getting feedback about that. But today I was literally working on the.

SPEAKER_00 1:26

Spoiler, we might talk about that book today, by the way.

SPEAKER_01 1:28

Well, I I have no problem with that whatsoever. Um but literally today I was actually working on the second edition of my first book, Building Microservices, which has me going down various rabbit holes looking into structured programming and different types of implementation coupling. I I will not bore the listeners of the podcast uh with with what I've been finding because my thoughts are not yet fully thought uh fully formed. But I've been reading an awful lot of stuff written in the 1970s, um uh uh and uh trying to put it into a modern context today.

SPEAKER_00 2:00

Interesting, yeah, because the phrase structured programming, you don't say that a lot. I mean that's a historical phrase at this point.

SPEAKER_01 2:07

Yeah, but but I mean we we talk a lot about you know coupling and cohesion, coupling cohesion, coupling, cohesion. I you know I talk about it in the context of of services-based architectures, but I know some of the but they all have their root in you know that the concept of structured programming, which and that of course, even before that we have information hiding from David Parnasse. And these are ideas which, although we talked about in the concepts primarily of modular-based software, I mean microservices are very arguably a m a type of modular architecture, the difference being that you know the the call barrier between one module and another is a distributed system, it's a it's a it's a remote network call, right? So we've taken modular architectures and then thrown in all the fun of the distributed systems world. Um, and I think while a lot of people, my myself included, I've been going back looking at all that distributed systems work as well to understand are we doing things in the right way, I think way more attention has gone into looking into things like you know, distributed transactions and all this sort of stuff than actually saying, well, actually the other part of this is the idea of modules. And so there is a lot of stuff around modular, the thinking around modular design and structure programming that we could bring into a microservices worldview. Um but some of those terms also don't make sense. A lot of the stuff there is about code level stuff, and really it's a bit different than somebody saying, well, so I'm trying to bring those ideas a bit in at least in the second edition, trying to bring in as much of those thinking as possible, so I have a bit more of a rigorous foundation for some of my arbitrary back shit crazy ideas that I have.

SPEAKER_00 3:44

That is uh actually exciting. You've kind of sold me on that because it's you know, again, you say structured programming, and you're you're very much like the haircut I'm picturing changes. Uh and the music sounds different, and the photos have this kind of different color cast to them. Like all of that puts me back in somewhere between 1979 and 1985, and I'm I'm at an age that I I frankly don't want to go back to. So this is difficult to talk about. But the yeah, I mean you I you just made the case, those concepts apply. We used to talk about them in terms of you know C functions or Pascal uh methods, but now um they absolutely apply to microservices. That's something to look forward to. I think, Sam, I'm gonna invite you back to talk about that when that thinking has matured a little bit.

SPEAKER_01 4:32

Perfect.

SPEAKER_00 4:33

Yeah, yeah. No, that's good. But anyway, let's let's talk about uh the book, uh monolith to microservices. Um I would really uh I mean it's uh it's the sort of book that I kind of want to just walk through the chapters, and I think that's gonna be a very interesting discussion. So you uh you start out um in the first chapter, uh you actually have a section on coupling and cohesion, so clearly this was in your mind already. Um but you def you you start out defining microservices. And I think this is one of those things where like everybody knows what a chair is, but it's difficult to identify the set of necessary sufficient conditions under which chairs obtain. What how do you define microservices?

SPEAKER_01 5:15

Yeah, yeah. When is a burrito not a burrito?

SPEAKER_00 5:17

When is a burrito not a burrito?

SPEAKER_01 5:19

Yeah, um uh I it's interesting I've been trying to describe them for five years, and I feel like I get a bit better every time. And and you can describe them at different levels, right? So uh we can look at them purely at a technical level and say a microservice is a collection of business functionality which is exposed over network endpoints, but that makes it feel very cold and doesn't really speak to maybe what distinguishes a microservice from an any other kind of service-based architecture. Um so I tend to sort of if I try and bundle using that, but a microservice is an independently deployable service uh which is modelled around a business domain. And so it's a the idea is that the the prime way we find our service boundary is by defining that around sort of business concepts. We expose those business concepts via these different networked endpoints. They could be, you know, rest-based calls, they could be, you know, invented endpoints. Uh, but we're always always striving for the ability to make changes to these services independently from other services. So I sort of compare microservices to a monolith, and I say a monolith, at least in terms of how I define it, our monolith is a unit of deployment. And what we're looking for with a microservice architecture is to be able to make changes to those uh microservices independently from one another. The idea if we can do that is we it makes it easy to do things like zero-down time deployments, it makes it easier to roll out functionality more quickly. And that's maybe one of the distinguishing characteristics in terms of microservices compared to maybe other types of modular decomposition, which is if you think about a traditional module-based system, you might want to roll out a new version of that module, but you've still got to redeploy the whole process that contains all the other modules as well. Whereas the microservice to the deployment pattern. Exactly right. Whereas with a microservice architecture, our deployment pattern is such that we're we can make a change to one service, deploy that service, release that functionality to our customers, but the rest of the system can be remained untouched. So that's that's sort of the idea that the overall idea. I suspect I explained it even better in the book where I had time to actually edit my thinking a bit more. But I hope that makes sense.

SPEAKER_00 7:27

This is pretty good. I think the key concepts that you brought out there are uh you know organizing organizing the definition of the microservice around a business domain. That seems like one axis, and the other axis is the deployment pattern. And I forget, I'm saying deployment pattern, that's not the word that you said, and I can't recall the word that you said. A monolith is a is a deployment structure or something. Yeah, yeah, yeah.

SPEAKER_01 7:49

So I I well, I because I'm very aware of the of how the word monolith in a lot of circles has now well, I I would actually say that what what I'm seeing in the wider industry is we don't hear the word legacy anymore. We hear the word monolith instead. And so for a lot of people they've just replaced the word legacy with monolith, which I think is and that's really dangerous because it presupposes that all monolithic architecture is somehow bad or to be rejected. I I think instead I want to s I want to talk very I I would like to talk very concretely about the idea that a a monolith refers to a you know a monolith is a unit of deployment. So uh and that is a s and and because I think because if you s if you assume that monolith means legacy, then it means that you would never build a monolithic system. And if you have a monolithic system, it's something you want to migrate away from. And I think that's really dangerous because that you know this idea that microservices therefore become the default way of building software. Look, it's great for me, I sell more books, but it's all fine. Yeah, yeah, it's great, but it's not it's not for everybody. Like the you know, you're when you're going into the world of microservices, you're going into the world of distributed systems, and that is a that is a crazy ride, and and you've got to be really sure that it's right for you.

SPEAKER_00 9:09

Yeah, it's it's it's it's bad, it's a bad life choice, is what it is. You absolutely shouldn't do it if you can possibly avoid it. And I that's again speaking as one who makes his living in distributed systems. So uh it's yeah, I think we we tend to to do this kind of manichaean thing where the thing I like uh is is the good and light thing, and the other thing is uh darkness and shadow and bad and can never be good. And monolith is that new mustachio twirling villain in our world, and I I I agree that we we must not do that because oh go ahead.

SPEAKER_01 9:45

Because I think it's it's it's our our industry is riddled with dogma and this becomes part of the latest dogma. And the you know the problem with you know dogma is that you know people gravitate towards extreme positions as a as a replacement for any critical thought. So I don't actually have to apply any critical thinking about my situation, I just say, well, it's a bad microservices are good, or I my microservices have to be reactive, I've got to do it according to what the reactive manifesto says. But because dogma is simple in a way. We don't we can sort of abdicate responsibility for our own choices. We do what everyone else does or what this shiny thing is, and we assume that that person's done the critical thinking on our behalf, and it's like I don't I don't think, especially with decisions like this, that we can make things as those decisions as simply as that.

SPEAKER_00 10:36

Um probably not. There might be uh times in the you know the scope of of uh human life where that sort of thing is is the right thing to do, but we're building tools ultimately, and that's uh uh fundamentally a pragmatic undertaking. And you know, trying to find a way that works well to build a thing that works well to help a business do its stuff better, uh, is really what we're doing, and and we have a tremendous amount of flexibility to to pick the thing that works, and that might that might be a monolith.

SPEAKER_01 11:05

Yeah, and it's also so so different for different people. I mean, this is you know, just what works for some another organization isn't necessarily what's going to work for you. I remember when um like Monzo came out, and I I know some people at Monzo based here in the UK, there were so Monzo for the people that don't know are a fairly successful uh some people call Challenger Bank here in the UK, so fairly new banking organization, got 1.5, 1.6 million customers, which is a pretty good challenger in my opinion. They mentioned that they have over 1,500 services, and they got a lot of flack from people saying that's crazy, that can't work. You're you're you know, you are all clearly idiots, was the subtext of a lot of the responses. And you had loads of other people saying, Oh wow, you must be really great if you've got that many microservices, and it's like no no that's not the point. Uh that company seems to be doing well, uh the the software seems to be working, 1500 microservices works well for them. That's not gonna work for you. Even if you're a bank, if you don't have the skills, the experiences, if you're not trying to build build products in the same way, if you're not organized in the same way, uh then their architecture is not gonna work for you. Um it's in the same way as you know everyone runs around saying, Oh, I'm doing the Spotify model, which is uh again, you know, putting to one side the fact that the Spotify model doesn't describe what Spotify does anymore. Um why would you copy the you know the the way a a web uh streaming comp a Swedish uh music company creates its products if you're not exactly the same company with exactly the same people and exactly the same challenges? Like you if you you shouldn't copy the outcomes of other people's architecture, but you should learn from the processes they go through to come up with that architecture. There's always dangerous just by copying what you see at conferences in this way, you know.

SPEAKER_00 12:54

Try to try to understand uh what it is about that system that worked and didn't, and what might be lessons you could learn that you could apply to your own. By system, I mean human system, that is that is understanding the business, executing the business, generating the software that supports the business. Uh, what can you learn about those other successful things um in terms of of their process, not their content, uh that then you could adapt to your own. Uh reminds me, um, and I uh this this sounds like a political thing, I dare not uh mean it that way, but a common discussion about uh public schools, Finland is held up as a paradigm case of extremely high functioning public schools, which uh to my understanding is absolutely the case, right? Like we should we should buy them a drink and congratulate them because they've done a really good thing. And you know, other other non-Finland countries say, well, this is what Finland did. Let's do that thing and we'll get that outcome. Yes. Uh it's and and to be clear, there could be definite lessons to learn there, but you have to do that same thing of like, well, you know, that's a different system. Like, I don't I don't go into saunas that are 180 degrees Fahrenheit, for example. Um, and and and Finns do, and we love them for that. Um, but you know, they're different systems.

SPEAKER_01 14:10

It it's also you you you you take one thing without understanding the rest. Like, you know, part of the finished system is that kids don't start school till seven. That works because there are other social structures in place, right? If you just take one of those ideas without the other ideas, it might all fall apart. Yeah, it is.

SPEAKER_00 14:27

Oh, it don't start till seven, okay.

SPEAKER_01 14:29

But also taking it to its worst extreme, this becomes cargo culting, right?

SPEAKER_00 14:33

Exactly.

SPEAKER_01 14:34

You you build the runway because you think that means a plane's going to land and deliver you food, right? But that's you you well, by all you're seeing is someone building a runway, you don't realize there's other things that are going on behind the scenes.

SPEAKER_00 14:47

Yes, because generations ago there was a runway, there were planes that landed and brought food, uh, and you know, profound global systemic things happening that caused that to happen, it wasn't the runway. Yeah. So with that, oh boy, that's those are just all great ideas that we don't even need to stop talking about. But I do want to ask you, when might microservices be a bad idea? You you've kind of teed this up, but what do you say?

SPEAKER_01 15:14

Um I I guess okay, at one level very generically, I sort of see microservices to an extent as a last resort architecture, which is have you tried the really simple things first? So microservices, I think, are a bad choice if you haven't tried the simpler things first. So a really simple example of that would be you've got uh a situation where you want to scale your application to handle more load. Um now microservices can help you do that, right? Because I can I can basically independently scale different components, but I could also potentially handle more load by you know getting a bigger machine. Have you tried that to see does that work? Does that solve your problem? Uh you could also do that through horizontal duplication, having more copies of your microservices have of your monolith. Did that work? Because if it did, guess what? You don't have to have a microservice architecture. So I think the first thing is have you exhausted the other possibilities? I think there are also probably two uh other kinds of there's sort of I say three other specific cases where I think microservices tend to be a bad idea and that I would need even more convincing that it was worthwhile. Um the first is with really small teams. Um, there tends to be a an amount of increased uh operational work that's required in a microservice uh ecosystem, and with small teams, it's harder to justify that work. Um, second is for people creating software that is going to be deployed and managed by their end customer. When you create that microservice architecture, you do push complexity into the operational space. You can offset that complexity by adopting new tools, techniques, uh deployment platforms like Kubernetes, if you want to, that help offset that complexity, but that requires that the people owning and running the software learn those new skills.

SPEAKER_00 16:59

You know, then like they say about regular expressions, then you have two problems.

SPEAKER_01 17:03

Well, well, yeah, but also now it's worse, right? Because if your customer is used to receiving like a monolithic window service on a on a DVD, and now you turn up and say, Oh, here's the 200 uh Kubernetes pods, put this on your Kubernetes cluster, and they go, What's a Kubernetes? What's a cluster? What's the pod?

SPEAKER_00 17:20

My what now?

SPEAKER_01 17:21

Yeah, exactly. Right. Um and and I mean there are companies like Replicated, for example, that try to make that process easier, but that's not fit for everybody. And the the third bucket, I think, I should have a YouTube video on this actually, was um people doing startups. You know, I don't think Microsoft is a good choice of startups. Because you know, if you think about where you are as a startup, you haven't yet worked out what it is that that you're trying to build. You might have an idea about a product, um, but you don't yet know if that product you know actually resonates with your customers. You're trying to find market fit. In those situations, you tend to have a smaller number of people, you tend to be constraining your resources, but the fundamental nature of the product can be changing quite rapidly, and that means you don't have a stable design or stable sense of your domain. You don't even have necessarily a core through line of what your product is. So you have a high degree and pace of change in your services architecture. You're much better off just building a very simple monotheic architecture. Rebound Rails is a great tech stack for startups. Triangulate what your product is you're trying to build, and then once you've got a firm idea, then you might start looking at now a microservices right for us. Microservices can be great for scale-ups, but for startups, I think they're a pretty bad choice.

SPEAKER_00 18:35

Excellent. So after you've gotten a better understanding of your domain, um, you have say validated a market, uh, and you're beginning to scale into that market, uh, you know, the let's say it is a startup, and um startup has found product market fit, and you know, from a business perspective, they're probably thinking about scaling their sales organization and they're hiring sales reps and building out, you know, the the functions that are super uninteresting to developers, like uh sales enablement and things like this that are that are like actually really critically important parts of a business uh as the business grows. And when when maybe when you start to hear those things being talked about and getting traction in the organization, you're like, all right, we're scaling now. Um maybe it's time to do this. By that time, of course, you've got a monolith. I mean, that's you'd be two years into the game. Yeah, yeah, absolutely. And there's a bunch of code you hate and it's unwieldy and it's hard to deploy, and all the stuff that that we like to beat monoliths up for. So once you're there, talk to me about splitting the monolith. What are and I I'm so glad to ask you this because um, you know, we're not really talking about Kafka directly today. There's plenty of stuff on the podcast about uh really the the messaging and or you know event event streaming application and microservices story and how Kafka and microservices go together. And I think that that idea is out there. We'll talk about it more in the future, but you know, just kind of that as the context. What I never talk about, me just thinking me personally, is how do that how do you actually do it? You know, sure, you want to refactor your monolith, and it's a good thing you have Kafka because it's magic, and I can explain why, but then how?

unknown 20:17

Yeah, yeah.

SPEAKER_01 20:19

I I think the very first thing to accept is I don't think actually this is not tends this tends not to be a problem when I talk to startups and things, but certainly for enterprise organizations. Um like when you're when you've built something and you don't like it anymore, there's a temptation to say we made a real mess of that. Well, let's build it again from scratch, where we don't have to deal with the problems of the existing system. This is what uh a lot of people call the big bang rewrite of the system. Uh-huh. Now most startups and big bang on your resume. Well, well, yeah, a big bang in terms of how long it takes and how much money it costs and all these sorts of things. Now, you know, most startups and scale-ups and don't have the luxury of even thinking about those because they've got a real running system and that's too dangerous in operation. I the whole central ethos of the book and the way I think about this stuff is this idea of taking the system and incrementally migrating it to become a microservice architecture. And so, really, what we're looking at here is you're you're you're using like this is a refactoring exercise. This is not a we're gonna rebuild a whole brand new system from scratch exercise. This is a we're gonna take an existing running system and we're gonna apply incremental changes to make the architecture uh fit for what we want now. So I think there is this thing like the architecture we've got, right, it's not fit for what we want now. It doesn't mean it's bad, it just means that what we now need out of it, it can't deliver. So what we need to do is we need to apply changes to make it do what we want. The goal isn't to necessarily get rid of the monolith, the goal isn't to make everything microservices. And yet if you look at all of the I've looked at so many different large scale migrations from large scale monolithic applications, it's very rare that monolith dies. Often all those organizations need to do is change the existing system enough. To help them do what they need to do today. So that's the first thing is that prioritization. It's like, well, what is it? What's the gap? What does your current architecture not let you do? And be hyper focused on that outcome. Because the bits of functionality that I'm going to want to extract from your monolith, if it's about scaling your application, are going to be quite different if it's about um uh segmenting data so that you can run in different countries. That's gonna be different again if your goal is more about faster time to market, or if it's about adding more developers to scale up the development for. So you kind of that the your goal is very much gonna determine where you start that and you know which which pieces of functionality you want to extract. Um I think an awful lot of people when I start talking to them have already decided to do microservices, but they've lost sight of why. They don't remember anymore what the reason was for them doing this. And as a result, when they start trying to think, well, where do I start? They only sort of the only driving force they have is the things that look easy to extract or the things that look easy to pull out. And that might be the right place to start, but if it doesn't actually help you get towards your end goal, then you know what's the point? Um there's no point in extracting functionality that you know hasn't changed in 10 years if your driver is to make it easier to you know make changes to your software because you're probably extracting functionality that is unlikely to change in the future. Um, but that might be the right functionality to extract if that is the stuff that's causing your system to have a bottleneck in it and you're trying to scale the application.

SPEAKER_00 23:36

So uh if I hear you correctly, you're saying the monolith survives in some modified form uh and certain bits of it become microservices.

SPEAKER_01 23:46

Yeah. I I mean setting aside all the other reasons why big bang re rewrites the systems are bad, and there's a lot of them, and we could have a whole hour about that. You know, uh Martin Fowler says if you do a big bang rewrite, the only thing you're guaranteed is a big bang. Um the the one of the biggest reasons why you want to make this an incremental journey is that that the vast majority of the sort of pain, suffering, and horror of a microservice architecture are only things that will only be evident to you once you're actually in production. So many of the issues are sort of invisible to you when you're running on your laptop. The moment you hit production, that's where all of the your, you know, all of these uh all these chickens come home to roost, so to speak, right? And so you're going to learn an awful lot about that once you get into prod. So if you've made a mistake, it's good to make a small mistake. So it's so what I always try and focus with teams is get something, extract one piece of functionality that you think is moving you in the right direction, have that integrated with that monolithic system, have it running in production, and you'll learn from that. You'll start to understand, okay, uh oh, oh, actually, you know, now we've got a latency issue, but we've only made a small change, so it's easy to spot where that latency issues come from. It's it's it's a more of a gradual incremental change, and and you know, again, coming back to that idea that the operational world is going to change, you're not asking it to change suddenly overnight, you're making a small incremental change in your working practices. A small incremental change is maybe some of the tools or technologies that you need. And it's like uh so I talk about the adoption of microservices like turning a dial. You know, you turn that dial up and you have a few more when you see how that goes. Um, so the analogy is like you know, you've got a stereo, right? You want to find the right volume for your stereo. So you you turn the volume up gradually so it gets to a happy level. What you don't want to do is crank the volume round to 11, then plug your headphones in, and then find that your ear jumps explode. And so for me, this is the same idea. We slowly turn that dial up, extract the service, get it into production, have it serve real use and actually be useful for you. Learn from that and use that to adapt the next thing you do.

SPEAKER_00 25:51

You uh you identify some patterns for splitting the monolith. Give us a taste of a couple of those.

SPEAKER_01 25:57

Yeah. So I kind of the the the main bulk of the book in terms of patterns is like chapter three and four. So the chapter three is about application code decomposition patterns, and chapter four is sort of data decomposition patterns. So in chapter three, there's sort of two patterns that are quite useful. There's one which is called the Strangler Fig application pattern, and that's this idea that you sort of wrapped a new system around the old system. So I I I from the outside world, this could be from a user interface, could be from an API, I make a call to a system, and that call gets served. It's opaque to me whether or not that call is being served by the existing application, or if it that calls actually being diverted to my new, in our case, microservice functionality. So that can be implementing something like a stranger fig application uh can be something as simple as sticking a proxy between the monolith and upstream calls. Uh you can do it at the user interface level. But the idea is that from the outside world point of view, you you don't see any difference. Uh fundamentally, you know, most of the the patterns I share are refactoring. You know, a refactoring is something which changes the structure of a code but not the behavior. And so normally when you do a microservice migration, you're looking for like-for-like functionality that you'll then change later on. So the Strangler Fig application pattern, because you don't, you know, you sort of intercept the call, bypass it, you don't actually have to change the underlying monolithic system. And that can be really useful because it means you've got both sets of implementations running that opens you up to doing things like canaries or parallel runs, uh, which allow you to actually compare the two pieces of functionality and make sure they're working appropriately.

SPEAKER_00 27:32

Uh another pattern could you for people who don't know it, could you define Strangler Fig?

SPEAKER_01 27:37

Oh, so Strangler Fig, uh, well, it it's basically named uh the the inspiration is actually a type of plant, which is a type of fig tree in in in Queensland. And it literally that the fig itself only can exist by by wrapping itself around existing trees. So you get a tree that grows in the rainforest, the strangler fig sort of takes root in the canopy of the tree, it wraps itself around the tree and it gets bigger and bigger and bigger over time, and eventually it's sort of strong enough to stand by itself and it no longer needs the underlying tree. And so with a strangler fig uh migration pattern, it's the idea is the same. The monolith does everything we need it to do in our case, or the existing system does everything we need to do. Our new system doesn't do everything we need to do right now, it only maybe does like a 1% of the functionality. And so what we do is we wrap that new functionality around the old. So when a call comes in, if that call is for the new functionality we've migrated, we divert that call to our new system. Otherwise, we allow that call to continue to the old monolithic application. So from the point of view of the upstream observer, which could be a user interface, could be an API, you don't know anything's changed. But behind the scenes, you can dynamically just divert those calls. Uh and that gives you a really kind of quite straightforward way to migrate functionality. It doesn't work for every piece of functionality, it only really works for things that you can kind of divert at the perimeter of your existing uh uh monolithic application, though.

SPEAKER_00 29:00

Uh and it's safe because you have the old thing that still works while the strangler fig is working on killing it. Absolutely. Uh yeah. And you mentioned uh database decomposition. There are a number of patterns there. Talk to us about uh and and uh I'll just uh to give my context on this. Um it always seems easy. Of course, it's not easy in practice to refactor a monolith while it's still supporting a business, but it's easier to think about, right? Because you already have probably a modular architecture and you know what the pieces are functionally. Yeah. Um and and those functional pieces are uh you know, it's difficult to implement, um, it's it's it's hard to operationalize, and you know, nothing in software is ever easy, but uh it it's relatively less of a cognitive load to think of where the cleavage lines are in the monolith where you're gonna split it and get nice little gems out of it. Um not so the data, because data uh generally likes to be shared among modules.

SPEAKER_01 30:01

Yeah.

SPEAKER_00 30:02

So talk to us about about decomposing the database.

SPEAKER_01 30:04

Yeah, I mean it's it's hard, it's painful, but you've got to do it. Um, you know, we we we we talked earlier about coupling and you touched on coupling and cohesion at the beginning of the podcast, right? You know, we don't want shared databases because they become effectively one of the worst forms of implementation coupling between services. So uh this is especially problematic if you've got a relational database. So relational databases are amazing things, they're incredibly powerful, and the more I spend time with them, the more I appreciate how smart they are. But we've got a lot of data which has relationships between things, so teasing that stuff apart can get really problematic. You know, it's thinking about breaking foreign key relationships, for example. Um, when you start getting into the world sort of refactoring databases and changing those things, you need to need some good tools. So I like things like FlyAwayDB, which allow me to make refactorings to my databases in sort of version controllable delta scripts that could be run in deterministic fashion. But so one of the patterns I talk about really is how you take a foreign key, how do you move a foreign key relationship effectively or move a join relationship from a database tier to an application tier? So, you know, I might uh I gave the example that I've got in the book, I think, where I've got like uh sort of I've got some code that has stores that that manages all the sales that we've made for a particular online store, and it says, okay, we sold an item, and the idea that item was one, two, three. We have a different bit of our functionality which manages the items that says, Okay, item one, two, three, it's the greatest hits of Justin Bieber, right? Now I could easily in a database have those two bits of data relating to each other so I can extract a nice uh you know set of data which says what's the top test what what the top ten best sellers this week, and it would pull out both pieces of information doing a join in the database. Now, if my catalog information with the information with the critical information that the item one two three is the greatest hits of Justin Bieber, is now in one service and the information about the item that got sold is now in a separate service, where does that join happen? That join now has to happen up into the application tier. Um and so when you start doing joins across application tiers, you've got a whole host of questions to ask, like, well, well, what does a foreign key relationship look like in that world? Uh what happens when I need to make lots of joins? You know, is is caching my answer? And the answer is hopefully not, but probably yes. And so what does that mean? And then you've got to deal with the fact that you've potentially given up on sort of things like referential integrity. Uh, in many ways, that one example is the worst example in the book because it's the most horrif horrifying for many, but it it can be solved.

SPEAKER_00 32:37

I I wasn't feeling good, by the way.

SPEAKER_01 32:41

But the the thing is, these these aren't see, I mean, the thing is people do solve these problems every day, and people build systems without those kinds of relationships in them, and they build distributed systems at scale without those kinds of relationships. They do joins in effectively, they do joins in different ways. So there are I I thought that was one of the most more complex examples in the book, but I also talked through situations like how to do things like share static reference data and stuff like that as well. There's an awful lot of roles for sort of more what we'd consider traditional database uh technologies, you know, there's a there's absolutely a role for traditional ETL type processes, things like change data capture. Uh, database sort of materialized database views can be helpful here because you can use that as a migrationary path. So uh, you know, rather than a service can you know being coupled to an entire database, it can only be coupled to maybe to a view, which is maybe a bit better. Um, I I also talk about sort of well, what happens when you do need lots of data in one place, you know, when I do need to do large-scale joins. So, what is the role of a traditional data warehouse? I mean, this is where stuff like CAF gets really useful, right? Because I can actually, you know, I try and invert the models a bit here. Um, in a traditional enterprise organization, the poor data warehouse team has to go to the services to get the data. I think in a microservices world, it should be the inverse, right? The the data warehouse or something, whatever that is, you have to get your data to it so that you can do these large-scale sort of reporting operations, in which case, you know, streaming technology like Kafka becomes really useful, but so can just good old-fashioned standard ETL processes too. Um just because it's maybe old-fashioned database technology doesn't mean it's not incredibly useful for these sort of migration uh approaches.

SPEAKER_00 34:23

And I wonder, um definitely Kafka plugs in there as an integration technology uh in a in a well-understood way. Um we're talking about joins, and and uh everybody there's looking about a dozen database decomposition patterns in chapter four of the book. So there's a lot more than what we're talking about, and I think you should probably read it if you do microservices. Uh, and Sam has not compensated me to say that, but um it's just true. Uh the the join thing, I wonder if uh there is uh an angle there where if you've got the two um data sources that that you need to join that you had been joining in the relational database that now you can't because you've decoupled, you know, you've decided that it's the right thing to do to um refactor those parts of the monolith into services. And if those services are consuming events from a Kafka topic, which is a great way, you know, if you're if you're doing the reactive thing, you're not cargo culting it, but you're doing it because it's a good idea in your context. Um I wonder if there's a case there for doing the join in a stream processor, like in a Kafka streams or KSQL stream processor and having the service consume that joined.

SPEAKER_01 35:39

Uh uh absolutely. And and so I mean, um I think I actually talk, well, the example I give in the book, I think, is talking about how you would create effectively a uh the ability to work out who is the top best seller in the last week. And I think I do give the example that something like KSQL would be a really good fit there because um if you're emitting events when the catalogue changes and you're emitting events when sales are made, then effectively projecting across those two things to create to project out a effectively a materialized view across those two streams would work quite well. Um and I mean you could even uh you there you could also you'd have to go all the way in with KSQL to do that, you could create a dedicated service and sort of re-implement that uh that logic inside it. But absolutely, I think the um and KSQL would fit really well there where you kind of know the result or where you kind of know the join up front. Quite often, though, um in organizations people want to be able to run ad hoc queries and run ad hoc joins, and that's the kind of place where data warehouses work, right? So I do think they're quite exactly they're quite different. I don't know how platform. Yeah, although they sort of from a technical point of view they're the same, actually, from an end user point, they're quite different. So if I know I've got to generate a top 10 VIS, KSQL's a great fit for that. But you know, uh you know, a lot of organizations, people have got Click and Tableau, they expect to be able to connect these things to an existing Nesq SQL data store. You may just need to stick it in a big, big old SQL database. Um so that I think uh you know it's quite possible you end up doing both, to be blatantly honest, in some situations. And again, you know, Kafka made Kafka's a great fit then to actually getting your data to the data warehouse, but also potentially giving you the ability to then use something like KSQL to do that sort of um streaming projection.

SPEAKER_00 37:20

Indeed. You're uh I um this is sort of classically opinionated Tim at this point, but you're you have a shot at succeeding with uh a Kafka-based architecture at that point. Because you're you're doing a tricky thing. Kafka's good at that tricky thing. It's still tricky. Uh it's just you've kind of got the right tools there.

unknown 37:38

Yeah. Yeah.

SPEAKER_00 37:39

And again, where uh remind us, where are you thinking next? We talked about that a little bit at the beginning, but I'd I'd love to leave people with uh a picture of that.

SPEAKER_01 37:47

Uh well, quite aside from the fact I'm doing small stuff online and all those sorts of things, I'm trying to get my YouTube channel up and running because I I've got to do that now, stuck in the house. Uh, but I'm also working on the second edition of building microservices, so I've just wrapped up the first sort of three chapters of that book. So that should be available in early access to people that are on the I Riley platform um in the next sort of three uh next month or so. And uh I'm going through chapter by chapter from the original book, basically revisiting everything and rewriting a lot of stuff. Uh, there's some areas where I wanted to reflect a shift in technology. So, you know, what does the you know if I think about when I wrote the first edition of that book, you know, containers were you know, Docker was just a thing like five or six years ago when I wrote that. So, but now Docker's come and gone again, right? And now in the world of Kubernetes. So, what does that mean? So I talk a bit about Kubernetes and FAS and those sorts of other aspects. Uh so in some areas it's about new technology, in some areas it's where I want to go back and refine how I explained concepts where I think the concepts were right, but I want to explain them in a better way. And there were some areas I just think were underserved. I didn't talk enough about user interfaces, for example, so I'm gonna do a whole chapter on user interfaces in the new book. So I'm working my way through that. Early access should be available in the next sort of four weeks or so, but I'll be dropping chapters pretty regularly through between now and throughout the rest of the year, really. So probably have that wrapped up by the end of the year. And then I've got more books planned beyond that when I can find the time to do it.

SPEAKER_00 39:16

Well, don't be surprised if we ask you to come back on to talk about it. Our guest today has been Sam Newman. Sam, thanks for being a part of Streaming Audio.

SPEAKER_01 39:24

Thanks so much.

SPEAKER_00 39:25

And there you have it. I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at TL Bergland on Twitter. That's at T L B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in Community Slack. There's a Slack sign-up link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast, wherever find podcasters are sold. And if you subscribe to iTunes, be sure to leave us a review there. That helps other people discover the podcast, which we think is a good thing. So thanks for your support, and we'll see you next time.