Apache Kafka 2.5 – Overview of Latest Features, Updates, and KIPs

Apache Kafka® 2.5 is here, and we’ve got some Kafka Improvement Proposals (KIPs) to discuss! Tim Berglund (Senior Director of Developer Advocacy, Confluent) shares improvements and changes to over 10 KIPs all within the realm of Core Kafka, Kafka Connect, and Kafka Streams, including foundational improvements to exactly once semantics, the ability to track a connector’s active topics, and adding a new co-group operator to the Streams DSL.

EPISODE LINKS

• Check out the Apache Kafka 2.5 release notes
• Read about what’s new in Apache Kafka 2.5
• Watch the video version of this podcast
• Join the Confluent Community Slack
• Learn about Kafka at Confluent Developer

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_00 0:00

Welcome to Streaming Audio, a podcast about Kafka, Confluent, and the cloud. We got some great stuff for you today. I hope you enjoy. Hi, I'm Tim Berglund with Confluent. I'm here at Bandolier National Monument near Santa Fe, New Mexico, to tell you all about Apache Kafka 2.5. If you remember, 2.4 was a really action-packed release. There was a lot to talk about. 2.5, still very exciting. There's great things, but fewer details. It's a little bit easier to digest, which I think we can take as a good thing. Now, as usual, I'm going to divide the changes up into three categories. We're going to talk about the core Kafka changes, the Kafka Connect changes, and the Kafka Streams changes. And all of these are going to be with reference to KIPS, Kafka improvement proposals. So I'll always refer to the KIP number, and you can Google that, just KIP and that number. That'll get you to the Kafka wiki page with more details on that KIP. And as always, you want to check out the release blog post and the release notes for all the details. But here's the summary. Let's dive into core Kafka. If you remember in the summer of 2019, we announced the approval of KIP 500. Now, KIP500 is Zookeeper removal. And like we said at the time, it was gonna spawn lots of other smaller kips that were specific things that had to happen in the code to get rid of Zookeeper. And that's happening in 2.5. We got a couple of those landing. We've got Kip555, which affects uh certain admin tools. Some of those admin tools would communicate directly with Zookeeper, and that's already started to happen. Uh if you look back at 2.4, there were some changes in the admin client interface, a few things that are getting added to that interface that aren't Zookeeper connections anymore. Well, 555 is just more of that sort of thing. Admin tools, getting the Zookeeper connection severed, and you know, still, again, Zookeeper's still here in 2.5, but uh we're kind of cutting those apron strings so that eventually a while from now it's be able to take Zookeeper out and all those other interfaces won't have to change at that time. There's also Kip 543, which says that dynamic configuration changes won't require direct Zookeeper access. So this is another, as I say, apron string being cut so that when Zookeeper goes out of the picture altogether, uh these things, these interfaces have already been refactored such that the direct Zookeeper access goes away. KIP 515 enables Zookeeper to use the new TLS authentication support that had already been there. Now in 2.4, you already had Zookeeper 357 that had the new TLS supported authentication. Uh so that was in place. Now, as you can see, we've got configuration parameters that let you do that. That's what Kip 515 does. We can set things like the enabling the Zookeeper SSL client, setting the key store, setting the trust store, all those things that you need. So now brokers talking to Zookeeper are doing it securely. So as you can see, even though we're in under the auspices of KIP500 in the process of deprecating Zookeeper, uh this is an important security consideration. The broker to Zookeeper connection had not been securable and now is. So that's important that that stuff happen since the process of removing Zookeeper is going to take a while. Security is still a concern in the meantime, and so we have KIP515 landing. KIP541 gives us a fetch max bytes at the broker level. All right. Historically, that is a consumer client side config, and you can set that, you know, here's how many bytes I want to fetch when I go to do a poll in the consumer. But a badly behaved consumer could set a very large value, right? You could say, like, oh, I want to fetch 100 megabytes, I'd like a terabyte, I just want all the things. And, you know, maybe unlikely that you'd have all that in a topic, but it's still possible for a badly behaved consumer to set that to a pathologically large value, and you can get bad behavior. Uh, you can get timeouts if the stuff just takes too long to transfer. Uh, various things can go wrong if that consumer is, we'll say, a little ambitious with its fetch max bytes. So now at the broker, we can say, yes, consumer, you can do what you want with your max bytes fetch configuration, but we also will have a maximum. So, like we're not going to go larger than two megabytes or whatever you set that to. So if the client tries to exceed that, the client is not allowed to, and the broker's max configuration wins at that same level. Another important change is that Apache Kafka 2.5 drops support for Scala 2.11. Uh 2.11 has been EOL'd for some time, and so hopefully you are off of that. It's a binary dependency of the broker to have this version of Scala around. And if you have a reason why you need to have 2.11 as your Scala binary, it's important to note you should not upgrade to 2.5 because it's not supported anymore. There are APIs that the broker is using that are simply not compatible with 2.11 anymore. So this is an official deprecation there, and be warned, hopefully it won't affect you. But if it does, you can plan ahead. That does it for the core Kafka kips. We have one uh connect kip of note, and that is kip 558. Track a connector's active topics. Now, as a sync connector, what you do is you're subscribing to, let's say, a topic and you're getting messages from that topic and you're putting them somewhere. You're putting them into a Cloud Blob store or Elasticsearch or whatever it is as a sync connector. And you might not be subscribing to just one topic. You might be the sort of person who uses a regex to describe a set of topics that you want to subscribe to, and that's perfectly fine. But if you do that in your connector config, you kind of don't know what the matching has been, what topics have been picked out. Well, now with 558, there is a new URL, as you can see there on the slide, in the REST interface, the connector, connector name, slash topics, and that's going to return a list of the topics that that regex has matched. So if that's a thing that you do, now you can find out explicitly which topics have been matched. Hey, you might be surprised. So this is a good thing to be able to observe. Let's move on now to Kafka Streams KIPS, and we'll start with KIP150, and that's adding a method called co-group to the DSL. Now, imagine you've got a number of streams that you want to group and join. All right, I'll make up an example off the top of my head. Let's say you've got a stream that's keyed by user ID and it's purchases that the user has made. And you've got another stream that's keyed by user ID, and that is times the user has interacted with the mobile app, and then you've got a third stream, and that is uh times the user has interacted with the website. It's like clickstream data or something like that. Those are all three things that are keyed by username, and you might want to group them by username and maybe add some window. But really what you want to do is you get all wanna get all those together. You've got some analytics, uh, you want to try to understand what the person is buying in relation to how they use a website, how they use the mobile app, and grouping those three streams together is a reasonable thing to do. Previously you had to make them into tables and like do left joins, and it was kind of a pain. Now we have this method called Co-Group. You can see the code there on the slide. Uh, and co-group lets you do that in a more fluent way when explicitly what you're trying to do is, you know, you've got these three things that are grouped and you want to join them together. This is the method that lets you get that done. And there's KIP 523, which adds a two-table method to the DSL on the K-Stream class. So previously, if you had a stream that you wanted to turn into a table, it could be a little cumbersome, right? And let's assume this is a stream that is the intermediate product of some stream processing topology. You'd have to two it, you know, save it off to a topic and then create a table from that topic. Well, now there is an explicit way to say, no, really, I want this thing to turn into a table, return me a K-table, and that's what the two-table method does. So huge, I always refer to this kind of thing as syntactic sugar. It's kind of API sugar, uh, but huge improvement in the way that flow works. KIP535 allows state stores to serve stale reads during a rebalance. So let's back off a minute and explain what all those things mean. If you've got a streams app, this streams app, of course, is a clustered thing. There can be multiple nodes that are cooperating in the stream processing work, and there's all the partitions from the topic that are assigned to those nodes. I'm assuming you kind of know how those basics work. Sometimes those partitions get rebalanced, right? A node fails, one of your nodes goes away, partitions have to be reassigned somewhere else. Or you add a node, you're horizontally scaling your streams app out, and those partitions that used to be assigned to say your first two, now some of those partitions need to move over to your third node, the one you've just added. And that's not instant. That work actually has to happen. There's data that needs to get moved into that state store. So there's going to be the state store inside of each one of those three nodes, and there's this process of minutes, maybe hours, I don't know, this long rebalance taking place where that state store on the new node is getting populated. And now with this kip, Streams has a little bit more flexibility of which node is going to serve interactive query requests during the rebalance. Like we don't shut off interactive queries during the rebalance. It'll be able to pick. And if you look on the slide, you'll see that one of those nodes is at offset 100. Its state store is, say, all the way caught up or caught further up, uh, and the other one is just at 32. Well, this is still stale to some degree because the rebalance isn't complete. We're kind of not sure what's going on. But if we're looking for a record uh that can be served by that top node there, that app instance one at the top, the caller doing the interactive query request will get the answer from that, even while the rebalance is running. And as usual, there are always more kips. You want to read the release blog, that'll have a little bit more detail. Always read the release notes if you want to know it all. The release notes are going to tell you all the kips that are there. Check out Apache Kafka 2.5. And as always, you know what? I want to know what you're building. I love hearing stories. So hit me up on Twitter. Uh, tell us what you're working on, tell us what you're using Apache Kafka 2.5 for, and I look forward to talking to you soon. And there you have it. I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at TL Burgland on Twitter. That's at T L B E R G L U N D. Or you can leave a comment on a YouTube video or reach out in Community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which we think is a good thing. So, thanks for your support, and we'll see you next time.