Containerized Apache Kafka On Kubernetes with Viktor Gamov

Kubernetes provides all the building blocks needed to run stateful workloads, but creating a truly enterprise-grade Apache Kafka® platform that can be used in production is not always intuitive. In this episode, Tim Berglund and Viktor Gamov address some of the challenges and pitfalls of managing Kafka on Kubernetes at scale. They also share lessons learned from the development of the Confluent Operator for Kubernetes, and answer questions like:
-What is Kubernetes?
-What are stateful workloads?
-Why are they hard?
-Will Confluent Operator make it easier?

EPISODE LINKS

• Join the #kubernetes Slack channel
• Kafka on Kubernetes: Does it really have to be “The Hard Way”?

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_01 0:00

You've probably heard of Kubernetes, and maybe even know something about it. Maybe you know that there's this idea that stateful workloads are hard. Well, why is that? And what does it take to succeed running a workload like Kafka or the Confluent platform? I'm gonna talk to Confluent Developer Advocate Victor Gamov about just that on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the Cloud. Welcome back, everybody. I am your host, Tim Burglund, and I have with me today in the virtual studio spanning two time zones here in the United States, Victor Gamoff. Victor, welcome to streaming audio.

SPEAKER_00 0:43

Longtime listener, first time caller. No, I'm just kidding. I'm not the first-time caller. I've been to the show and I'm really excited to get back. And uh hi everyone. Um, just a friendly reminder, Victor Gamoff, uh developer advocate here at Conflict, talking specifically about the Kafka things. And today we're gonna be talking about Kubernetes side of things of Kafka, right?

SPEAKER_01 1:03

That is correct. Um, yeah, I think I don't know, this is second or third time you've been on. Not not often enough, in my opinion.

SPEAKER_00 1:10

So we need to I mean just the second one. The first one is was like an inaugural when I just joined a developer advocacy team. Now we've got Gwen. Now it's a good time.

SPEAKER_01 1:21

Yes. So we should put a link to that in the show notes so people can see the whole Victor corpus, as it were. But like you said, we're gonna talk about Kubernetes today and Kubernetes and Kafka and uh whether that's a good idea or a bad idea, and if it's a bad idea, how to make it into a good one. And I know this is a thing that you spend a fair amount of time on, Victor. To get us started, I think at this point, uh just about everybody knows what Kubernetes is, but just in case there's anybody in the audience who is really new and has heard it as a buzzword, but doesn't know, what's the like 60 to 90 second account of what Kubernetes is?

SPEAKER_00 1:57

Oh wow, it's a very difficult and uh challenging question. So I'll try to answer this. Um usually the people, when the people talking about Kubernetes, they're talking about the orchestration uh engine that allows you to schedule your containers. Um and but essentially what but what Kubernetes is it allows you to run your apps that containerize. Um typically it's a Docker, but it's not uh not only a requirement. And uh once your apps are running in Kubernetes, Kubernetes allows them to be fault tolerant. If something fails, it allows them to restart. Uh Kubernetes handles uh situation when you need to increase load dynamically. Um, if the load to your application is is growing, you can increase the number of uh instances of your application very easy. But from a programming perspective, uh Kubernetes is also an API server and that provides like very rich uh programming model. Usually I like to talk about Kubernetes as a like modern distributed operating system that allows you to run this, uh the massive um the distributed operating system and the schedule your containers, like your small apps that running in the regular operating systems. And uh there's there's also like a system uh-wide uh utilities that allows that run inside the Kubernetes and some of the user space apps, it's basically your applications that you're running. So you tell me if it was uh clear enough uh or you want to dig that uh somewhere, take that deeper.

SPEAKER_01 3:30

Take that somewhere, or less deep. I I would even say that if you know, given that containers are a good idea, like you decided, and I know it doesn't need to be Docker, but let's just say Docker, because it is 99.9999% of the time, you you decided Docker was a good idea and you want to deploy all your applications in Docker containers, like that's great for development. Uh everybody learns that workflow and it's super cool, but then you want to run those programs in production, and you have you have computers that you want them to run on. Getting those Docker containers to get scheduled on actual server resources is like a whole nother problem above containers. And I think of Kubernetes as that scheduler that takes those containers and deploys them to actual machines, and then there's this whole other set of problems that come up when you say, Let me schedule my containers, and it's got all these things in it. When you refer to it as an API server, it has to it has to have all these other functions in order to make that happen, in order to get that scheduling done. Uh it it tries to be this full stack, like you said, sort of operating system-like thing.

SPEAKER_00 4:35

Yeah, yeah. So if you think about this, uh when you have your computer, this this is why I like this um the kind of like a mental model about the operating system. You have your computer, you're never kind of thinking what kind of detail, what kind of uh the chips you have there, what kind of RAM you have there. You know there's some resources, you know you need to run these applications there. And if there's not enough resources, you will see that application will crash and there will see some message. So same thing with the Kubernetes, right? So you run this on some pool of nodes, like physical nodes or virtual nodes, it really uh really doesn't matter. It really you don't care about this, right? So you run this in the pool of nodes, and you know that your application, your container thing will require certain amount of CPU, certain amount of disk, certain ports need to be open, and after that, you define this as a as a manifest. So you you're providing these requirements to your Kubernetes server and saying, hey dude, I really need you to schedule this. And uh this Kubernetes responsibility to read this manifest and check if it has enough resources to satisfy your desire of deploying this thing. Got it, got it.

SPEAKER_01 5:53

And here's another thing. Let's get this into the uh kind of like why do streaming and Kafka and Confluent people care about this. The whole vibe, right? If you just if you just kind of step back and you look at the kind of system Kubernetes is, like I framed it as okay, containers are a good idea, and now I've got some number of servers, and I want to run all these containers on those servers. And there's this assumption that, like, oh, I don't care where like what actual IP a particular container runs on. I, you know, I've got my my microservices all Dockerized, and I just want Kubernetes to go schedule them, and it can it can you know move containers around or deploy new instances, or if one dies, it'll just deploy in it in another instance somewhere else. So it's kind of like this frictionless, massless moving of little bits of application software all over a cluster. Is the dream, right? That's automated scheduling of these containers, yeah. Which is awesome when you've got relatively stateless microservices.

SPEAKER_00 6:59

Right, exactly. I I I like I like the way how you started this with microservices when when we're talking about these uh small applications, right? So the microservice itself, uh like a share enough architecture. You don't need to do anything inside your microservices, usually will depend on some other service to to get data and stuff. But for communication reasons, right, you still need to have uh some sort of medium that allows you to um share some state for you know, for that matter, or like even it can be even you know some some sort of like external API, but you know, still you need to get some data from somewhere, right? And this is where we're going into the world of stateful applications, meaning that we're talking about databases, we're talking about messaging systems, and even if we do this uh microservices things uh using you know the Kafka streams, uh we can we can talk about this uh also uh a little bit, but uh it still requires some some state. Maybe you will require some state. And yeah, and this is this is actually a very interesting uh task. So I'm trying to avoid the saying uh words like tricky or difficult or problematic or don't or wrong exactly or or death first or anything.

SPEAKER_01 8:19

And that's not and that's not the message. Obviously, that's not our message here, but it it it's worth you know, when we're going into this discussion of Kafka and Kubernetes and just sort of framing, okay, everybody, this is what Kubernetes is, it's worthwhile to remember that the whole sensibility of the thing was here are these massless Docker containers, and I'm just gonna schedule them. And it's gonna be automatic and it's gonna be great, and we're gonna share secrets among them and you know, little bits of config, and we'll have all these waves of doing that. And so, oh good, my services are all scheduled everywhere. But when you have now data infrastructure like a database or a Kafka cluster, uh that's uh, as they say, a horse of a different color, right? Because it's not massless. You can't just pick this stuff up and move it around because there's all this data. Um and we don't this this is not a deep dive on on Kubernetes, and Kubernetes has interesting answers to things like this, but uh it's controversial, I guess, is where I'm going, right? What what what do you think you, Victor, and where do you go when you begin to consider the question of stateful things like Kafka on Kubernetes? What what's your view on that?

SPEAKER_00 9:29

So I was really blessed, I would say, uh by um hashtag hashtag blast, yes, by um being in the the world of like people who much smarter than me, and this you know tend to be um kind of like chain reaction when you're talking to smart people, they introduce you to another like smart people, and you know, you you start you know talking to those people, and uh one of the um we can we can call him as uh one of the like uh founding fathers of of Kubernetes, uh the Kelsey Hightower, um he actually have very uh very good Twitter when he's uh he's talking about different opinions, and he's very opinionated, and his opinions very controversial, and it creates a lot of discussion. And one of the things that he mentioned that sparks some of the discussion around the possibilities to run stateful applications is where I quote Kubernetes has made huge improvement in the ability to run stateful workloads, including databases and message queues, but still I prefer not to run them. So, meaning that Kubernetes supports your stateful application, Kelsey Heintower doesn't. But yeah, this is what I like about the the Twitter thing that people can you know grasp on like first thing, but actually there's a thread when he explains what he meant is that yeah, so you're building house, you're getting bricks, you're getting your wood, you're getting your plywood, you're getting your some other materials. Um, but you're not you're never building house that's gonna be you know exactly the same. So it's the same thing with uh running uh different databases and Kafka in Kubernetes that it's not enough to use all these low-level concepts uh and small build-in blocks that Kubernetes has to support that. Like Kubernetes provides the way how you can store the data for your pods. Pods is just a group of uh of containers, just the way how they call um this kind of like small apps that you deploy. Uh, in most cases it's just one container per pod, but you know, there's some exceptions. Um so once you run this pod, pod is ephemeral, right? You can kill it and it will restart. So this is this is what these tools are teaching us to treat your application not like your pets, rather than how you call it herd, cattle, cattle, yeah. Cattle, yeah. And um when you have this uh you know application when it what you what you what do you do with the one of the uh individual uh animal that is sick? Um you remove this from the cattle, so it will not affect other so the same thing that Kubernetes does is essentially allows you to if your container or your pot is you know experience some problems, we'll just you know remove it. But you need to store the state. So Kubernetes provides the way how we can store the state using uh persisting volumes. Uh there's kind of um application or your your your manifest that requires Kubernetes to provision this volume. It's called persisting volume claim. And uh, yeah, this these things are good, but you know, we deployed Kafka there because what we need for Kafka, we need storage and we need to have a stateful network identity, meaning that we need to have an address. And this address should not change when we restart one of the nodes, right?

SPEAKER_01 12:59

One of the even when, like I said, one one so a broker now is a containerized thing, and that container is going to be scheduled in a pod and it's gonna go run on hardware, but it needs to have a stable address, a stateful address that applications can count on.

SPEAKER_00 13:16

And also cool thing about you know things like Kafka. Uh Kafka has um certain way how you configure things, right? So you need to put these names of uh the servers, like Bootstrap servers in like certain format, and like you can kind of sort of try to dynamically scale it, but other nodes need to know where to find those nodes so they can join the cluster. Plus, Kafka depends on another distributed system uh called Zookeeper to storing this metadata. This is how other Kafka nodes will discover. So you also need to provide the way how Zookeeper will run in the full tolerant way. And Zokeeper historically, I think it's right now it's changing and we're moving to the world of uh 3.6 Zookeeper when Zookeeper will allow to do like dynamic configuration. Before that, if you want to add something into Zookeeper, some node into Zookeeper, you need to change configuration. When you change configuration, you need to restart Zookeeper. So without going much of the details, um the uh the thing with this is actually you think about right now you're running distributed system that depends on another distributed system that depends that runs on top of distributed system because Kubernetes is also a distributed system. You have uh pool of nodes where all these like a small agents are running and they're scheduling your pods, and it also depends on distributed storage, right? So and things getting very, very interesting, and uh things can get very very hairy very fast. So for perspective of deployment, uh, if you think that deploying Kafka in Kubernetes is difficult, it is not. Actually, it is not.

SPEAKER_01 15:06

You need to actual the actual deployment is is easy to do.

SPEAKER_00 15:10

Yes, sir. You're right. You need to have uh just a couple things done in certain order, right? So you need to have a provision zookeeper, you need to have persistent volumes and persistent volume claims for um for storing this data, you need to have uh things called services in order to other nodes will discover this, and the your Zookeeper cluster will be routable, route routable. Um, and uh Kafka can connect to Zookeeper, and also you need to expose Kafka port so clients can connect to Kafka. So this is easy, more or less. You know, you can you can master this, and uh to help you doing that. I can I can write the YAML to make those things happen. Um, if you like to do so. Um yes, and this is uh also a very interesting discussion, like what uh uh when we're having with uh the people who dealing with this kind of stuff, do you like personally, Tim? Like we in a shame free environment, uh do you like YAML? I really don't. And and it's it's very interesting that uh every time when we you know, if you in the industry like long enough you remember uh rise and fall of different file formats, XML, uh JSON, now we have YAML. It's it's more like the huge selling point that people say, Oh, this is human readable. Um yeah, but is it human writable? Um if you using like YAML without any third-party tools and trying to just format it correctly, oh boy. Um, but uh nevertheless. So yeah, you write a bunch of YAML. You need to for every set of components that um remember I when I when I started this conversation, I mentioned that uh Kubernetes is actually an API server. And inside this API server you can invoke different APIs in order to get what you want. So when I say I need Kubernetes to provide me with storage, I need to call the system volumes API and send this uh information about what what kind of storage I want to have. So Kubernetes will take this API and make it happen. Same thing with other uh components. If you want to do service, you're calling the service, you provide this uh manifest to this service, and and and the service will you know the the materialize it. So and this is uh this is where we go into a very interesting place. Um that's called this this all this thing called resources in Kubernetes. And uh because this OPI server, you can ask me, Victor, can we extend this? Tim, you need to ask me now.

SPEAKER_01 18:01

Victor, can we actually no? I'm not gonna I'm not gonna ask that yet. I don't want to ask that. I want I want I want you to summarize. Uh you said that deploying Kafka in Kubernetes is easy, but there are hard things about this. And I just want to summarize what what is hard. Okay, this is you mentioned a bunch of things like you have to have persistent volumes and persistent volume claims. Like uh just kind of briefly walk us through that again. What what's why is this harder than your microservice? I mean there are components, but but fundamentally, what's the hard part?

SPEAKER_00 18:35

Fundamentally, you know, once you deploy it and everything works fine, you're good, right? So you deploy things and everything works and life is perfect. But as we know, uh any possible movie, any possible book, this like perfect ideal situation doesn't last long. So this is why uh I mentioned that we're running massive distributed systems on top of another distributed system. And as we know, uh things can go very, very south in distributed systems. You might have some of the network glitches, you might have some of the computers will run out of resources, they will fall out of this pool of nodes. Uh, you might have some of some of the exceptions in your pod, and your pod needs to be rescheduled. So all the situation in running Kafka in Kubernetes and maintaining maintain this stable Kafka cluster. Or next thing is to is to upgrade Kafka cluster because okay, you convince your boss, you convince your team that you're going into the Kubernetes world, you deployed everything in Kubernetes. Now you need to include the mindset of uh maintaining all the stuff in Kubernetes and how you can have this um you know, upgrade of your applications, upgrade of Kafka applications. And if we're talking about more or less uh more or less big organization that cares about the data that they put in it, uh things around security and uh access restriction and uh all these kinds of things also need to be taken into account. So, meaning that uh need to be integrated with uh role uh role-based authorization control uh that built in in uh in Kubernetes, how to provide access um not only for uh your pods to to to work correctly, but also um uh the because La Kafka has its own things to to restrict access. So this kind of things is is becoming you know really, really burning questions for people. How I do that, like okay, I deployed this, cool, it's running, but how to sustain this life? I think this is great.

SPEAKER_01 20:51

And there are people who do it, right? There are people who are very good at Kubernetes who are able to uh kind of wrangle it and and deploy these stateful pieces of infrastructure, which are, again, like we said initially, these this is slightly counter to the original notion. You know, they were this was not built, this was not built for stateful things, but people, once they get into it and they're deploying their services on it, they say, Well, I want one push-button place for all my container scheduling to live, which I think is very reasonable. And they they the the community has really taken Kubernetes in this direction of just insisting that we deploy stateful workloads on it. And the the leading edge, cutting edge people who are very good at it have gotten it done. But most teams find it tough. You you can do all these things, but dang it, you have to. And it's not easy and requires you to go a lot deeper on the infrastructure than you might want to.

SPEAKER_00 21:49

And so, what do we do about that? Yeah, and what what actually happened? Um a couple things actually happened. So, first of all, cloud happened. And uh people start thinking, yes, so like I can focus on these like virtual machines and have uh dedicated people to deploy this, but hey, can we automate this? So it's why we we saw rise and fall, all this automation, uh deployment automation, or like uh they start called like terraforming, you know. You you you you go into the cloud, uh cloud formation, terraforming, all this stuff, right now, and you start thinking, okay, so I I I've done enough of this, and I want to focus on actually you know doing things. Um and uh like I want to go say Kubernetes. I don't want to go and do deploy stateful sets, deploy um services, network, um my my deployment, whatnot. I want to deploy Kafka. How I can tell my Kubernetes, just deploy my Kafka. So, and in this case, like I want to transition this from uh mental model of traditional enterprise when you can say, Hey, we have a Joe, and the Joe, let's call him Dev Kafka ops specialist, right? Um, I wanna I really want to coin this term dev Kafka ops. So the Joe knows how to deploy this, and uh, but unfortunately, there's a thing that we uh right now we're living in the world where the human cloning is still prohibited, so we cannot like physically take this Joe and uh scale him, yeah, and something happens to him, you know, the the the you know bus and stuff. Now we need to transition this. Or I'm really I'm really excited about this like a Doom Patrol uh movie where this there's a character called Robot Man and uh he died in a car crash. So the doctor takes his brain and put the in uh inside the robot. So think about this. We take the the information from the brain and put this inside Kubernetes, and now uh we can teach Kubernetes to you know do things for us. So Joe knows exactly how to you know deploy Kafka, knows how to scale up, scale down, how to do rebalancing, how what to do in case of some of the you know uh unreplicated uh uh partition happened. We know that this is one of the important metrics, right? And uh so we we we take his knowledge, it was like in there was some like a thought process, and after that, um people at Confluence start thinking, hey, we actually can do that. And you know, we know how to run Kafka, we uh we know how to transition this knowledge, automate this knowledge from the things uh the brains of our people who were running this consultancy around Kafka back in LinkedIn and back on the early days of Confluent. Can we put this into works? Right? So this is how this is how we start uh this idea of uh confluent cloud. And actually, confluent cloud is powered by Kubernetes. And at some point, you know, there's few iterations happened, we learn a lot uh about how to run this at scale, and we start thinking, okay, now how we can transition this knowledge from the from the brains of our um cloud into the brains of people's uh um like uh uh enterprises, they want to run their own uh the private or hybrid cloud, they want to have the same ability. They cannot have uh you can have a consultant of of Confluent who go there and sitting in their data centers and will deploy clusters for them. This is also uh you know valid choice. But hey, right, can we take this automation and you know um software uh how it's called uh when we're talking about um software development and how the economy of the software development is different from like traditional manufacturing, right? So we can develop one thing once and can sell it multiple times. So this is exactly what um what uh we did uh and we continue doing, and uh very soon you know people will will have a chance to touch this awesomeness.

SPEAKER_01 26:22

Um Kubernetes has a I think a really neat way of modeling what you just described, you know, all that expertise and knowing how to operate a thing, it it reduces it to this really simple abstraction, which is that you've got your desired configuration, and you know, which is a fairly complex vector of of a bunch of things. These containers should have these secrets and should have this storage and you know these uh stateful addresses and and all of that stuff. You know, these this is your desired configuration. And then there's the real thing out in the world, right, that's running on the Kubernetes cluster. So there's what it should be and there's what it is. And when you've that that's that knowledge of turning what there should, what should be into what is is Kubernetes way of encoding all that operational expertise you were just talking about. And I think this is really cool because you know when you say it like that, that sounds like uh the transient response of the system or the initial conditions. Oh, okay, that's the configuration. Let me go make sure that's running, let me start everything and make it like the configuration. But in the the world of Kubernetes, that you know, the the actual running state of the cluster is changes. Things break, services die, servers go down, or instances blink out of existence, or whatever. So there's this constant process of saying, all right, here's my configuration, and I'm gonna just constantly make sure that what is actually deployed in the cluster looks like that configuration. Um and that is called in the world of Kubernetes, we're introducing a new concept, a controller, right?

SPEAKER_00 28:06

Correct. So things that we um touched briefly on um the on the previous like the 20 minutes that we're talking, that you know, it's still API server. And you're supposed to ask this question, Victor, can we extend this API server? Um, but hey, this is uh this is how we're getting to the uh to the IP.

SPEAKER_01 28:28

How do we extend that? So this API server is the controller, right? That's the thing that animates that.

SPEAKER_00 28:34

We do intend these controllers. Yeah, so we have this uh the concept of um resources that I mentioned, right? You have uh something to operate on, and uh it's good. Okay, we say yeah, there's some resources, but essentially resource is just basically definition. There should be some handler that you just uh explained, like how who reacts on this one. Uh, and this concept of controller. So basically, uh when you want to extend uh uh Kubernetes uh capabilities, you do in the same way as you did with uh, for example, um um writing some system software. So basically, you're writing your software that listens this API server event, okay. Hello, event uh driven uh architectures inside Kubernetes. Uh, when some resource changed, Kubernetes will fire the event, and someone who is interested in this kind of events will react on it. And it's it's it's kind of brilliant because it's quite easy and this model is uh quite um quite easy to understand to developers, right? So they can just uh you know extend these endpoints that they need to implement, implement this logic that they need to react on, right? And essentially it's uh it's basically like a loop, a reconciliation loop, right, that happens inside Kubernetes. That like if my world is in a state that uh my config just passed me when it's in the same state, so we're good. We're in the world where everything works. Now, so when I said that, okay, so can we can we do like something about the maintenance? So how we can react on certain things, for example, like uh we have a number of uh of uh underreplicated partitions increase. Can we uh can we listen to this one? So it appears it's not like that uh simple because it will require some knowledge of underlying technology. So this is why um the standard tools provide you just the building blocks. You need to have this logic, you need to have this uh uh brain to build it. So on top of this concept of controllers and resources, um there is a the concept of operator appeared, right? Remember this this robot man that was you know died in a crash and we took the brain. So this is what the the operator does.

SPEAKER_01 30:59

This is actually the this is the person who was very good at operating Kafka in Kubernetes.

SPEAKER_00 31:05

Yes, exactly. It's uh like one of the the conflict consultants. I'm sorry. Uh um, yes, and I just cannot uh I I cannot uh cannot help to remember my my times uh in the professional services when we working with customers and we explaining like certain like practices and best practices. So like instead of explaining this, like do not tell, like show, right? This is what the the mantra of videographer. Unfortunately, we cannot do the video today, but uh I promise we will more. Um, and um so yeah, but essentially like with this logic was implemented and uh in our cloud solution, and after that we take this uh as a as a product and we start testing with a bunch of our existing customers. Right now we do like a real-world test, like some of the assumptions about real world that we have in our cloud environment, we can control it, right? Because you know, we can control environment, we can control things, but when you deploy this into the wild, and uh right now, situation around Kubernetes reminds me um the time where there were a bunch of different uh Linux distributions, and everyone, I don't know, Tim, if you uh was kind of enthusiast about the building your own kernel back in the days.

SPEAKER_01 32:25

Um I never was, but I was certainly aware of it.

SPEAKER_00 32:28

This is exactly what happens right now. You can choose different distribution. So if you want to run this in cloud using Google, if you want to run this in uh Google has uh this uh the Google Kubernetes engine, yes, the GK. This is simple as that. And uh the like you want to run this on-prem, there's like distribution of Kubernetes that you can uh deploy, or there's OpenShift, uh which is another kind of you know, remember there was other Linux distribution and there was uh like a Red Hat Linux. Uh the same thing with the OpenShift, there's like the Kubernetes distribution from other vendors, and there's like OpenShift. Um, there's a uh the tool from uh from from from Red Hat that allows you to you know deploy your Kubernetes thing now. And uh and it and it turns out like a lot of people using different Kubernetes installation, and and we need to you know work with the customers to understand this real world and what kind of challenges they they facing. And uh remember when I say um when you more or less um big enterprise, uh and especially if you're doing something around um information like a finance information or health information, you need to think about other things like encryption, security, and uh the the you know encryption in in motion, encryption and rest. So this kind of things that we're we validating our design right now with customers, and this is what uh that we're looking forward to kind of deploying the first version where we will take care of the deployment, we will take care of the upgrades, um, some security configuration. Another interesting thing that oh, okay, I have this uh I have this Kubernetes thing, but my application not in the Kubernetes. How you can do that? You need to you know provide external access to your Kubernetes cluster, to you can you need to configure your uh load balancers to provide this access, and uh it's also part of deployment. It's like it's easy to do, kind of sort of once, you know, when you're doing this, when you know what to do. Uh but when you know, yes, exactly. And uh when you're trying to have this kind of internal private cloud of Kafka clusters that you want to provide to different teams, it's getting tedious, and you know, you don't want to do this every time. You you cannot automate this. There's tools like Helm that allows you to template out certain things. Um, but Helm is a declarative tool, and it cannot you know have a logic like saying, okay, how we can do how how I can do um how we can do a rolling restart. Because we need to do rolling restart in particular order, because you know, you don't want to kill a controller. One of the Kafka broker brokers is the controller. You don't want to kill a controller first, because in this case it will take uh some some some others nodes time to like new controller. After that, this controller needs to rehydrate this data from from from Zookeeper and so far so on. So when you do rolling restart, you need to follow certain order, right? And this is kind of logic that not easy to do with stock components. You need to have a controller, you need to have operator.

SPEAKER_01 35:51

And confluent operator is um you you're kind of you're kind of hinting at it, but it is this custom controller that embodies all of this operational expertise. So it's it's the thing, the you know, the API extension, or just custom controller in Kubernetes terms, that makes it a whole lot easier to run a workload like Kafka on Kubernetes, right?

SPEAKER_00 36:16

I think, yeah, yeah. And I think uh last thing that I want to share today um is uh we're gonna be talking about uh Kubernetes and operator and all these challenges on uh Kafka Summit. Can uh can we talk a little bit about Kafka Summit team here on this podcast? Uh yeah, the we have uh I'm really excited because it's uh this year uh Kafka Summit happens, uh one of the summits uh will happen in my uh on my turf uh in New York in uh Big Apple. And I'm really excited to talk about uh Kubernetes and uh Operator. I will uh take the stage with my friend Michael Ng, who is uh running uh product here for Operator and other things uh uh related to Confluent platform, and we're gonna be talking about all these challenges and how to overcome this. Um so I really, really, really, really hope I will see many of you there. And um can we uh can we throw away some uh some uh some promo?

SPEAKER_01 37:20

You know, here's what we should do. If you're listening to this, you should reach out to Victor on Twitter. That's at G-A-M-U-S-S-A. Gamusa, G-A-M-U-S-S-A, on Twitter. Find him on Twitter, and he'll give you a discount code. Nice. You gotta you gotta want it. You gotta you gotta reach out just a little bit. Um, but it's a it's a decent discount. So you should join us. If you're listening to this uh before, what's the date of the New York Summit, Victor? Why do I not just know that April 2nd. April 2nd in your hometown or very close to your hometown. If you're listening to this before, significantly before April 2nd, reach out to Victor on Twitter, get a discount code because you're gonna want to hear him and Michael talk about this uh at Summit. They're gonna lead you through all of this stuff. But basically, you know, the problem is we've laid it out that Kubernetes is uh a good thing, and in any event, it's a force of nature. It's kind of a thing that you have to have a position on, and a lot of people are adopting. You have to appreciate why a workload like Kafka is difficult, and basically appreciate that it's probably a good idea to get some help. Uh, and Kubernetes is built to be extendable in this way, to have these pluggable custom controllers uh that can run workloads that aren't just little massless microservices that you can schedule on any server you want, but actually complex distributed systems. It's adaptable. Kubernetes will run these things and it will do it successfully. But a thing like Confluent Operator is a good idea there. And also, if you're listening to this, you know, in this part of the year, uh in a timely fashion after the release, uh, it's a pre-release product. So we're we're talking about a thing that Confluent, you know, it is it is a Confluent product, or rather will be, uh, but it's not shipping yet. So this is just to kind of get you prepared to think about how it works and why you need it and why it's a good idea.

SPEAKER_00 39:17

Yeah, so I I would say that uh we do have some recommendations on how to you know deploy things. We do have this uh you know YAML files if you like it. Um you can even do it today, like I said, uh you know, deploying Kafka on uh Kubernetes is is is easy. Um it's it's not a rocket science. However, you know, have sustaining this um and uh running this successfully in production will take some effort. So I would like to say that uh we do have, if you do these kind of things, we do community uh Slack where you can reach out to, you know, I'm always hang out in the Kubernetes group when the people have different questions around the certain things. Um, you know, apart from that, we do have other awesome groups there and where you can ask uh questions about stream processing and some some other components like schema register or control center. So um I would say that you know join our Slack, um, community Slack if you're not there for some reasons. I don't know. If you listen to this podcast and you probably in the this Kafka world, um so please do.

SPEAKER_01 40:20

Yeah, there'll be a link in the show notes for how to get there. Uh but we definitely encourage you to do that. You can reach out to Victor in the Kubernetes channel if you have more questions about this. My guest today has been Victor Gemoff. Victor, thanks for being a part of Streaming Audio.

SPEAKER_00 40:36

I really appreciate uh that uh you know you bring me back. Hopefully we'll uh see many of you around the world. Thank you for your time and thanks for our awesome host, Tim Berland. And there you have it.

SPEAKER_01 40:49

I hope that was helpful to you. If you've got questions, you can ask me at TL Berglund on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on any of our YouTube videos. Your question might be featured on the next episode of Streaming Audio. And feel free to subscribe to our YouTube channel and this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast and just generally helps us get the word out. We appreciate your support. See you next time.