Kafka Screams: The Scariest JIRAs and How To Survive Them ft. Anna McDonald

In today's spooktacular episode of Streaming Audio, Anna McDonald (Technical Account Manager, Confluent) discusses six of the scariest Apache Kafka® JIRAs. Starting with KAFKA-6431: Lock Contention in Purgatory, Anna breaks down what purgatory is and how it’s not something to fear or avoid. 

Next, she dives into KAFKA-8522: Tombstones Can Survive Forever, where she explains tombstones, compacted topics, null values, and log compaction. Not to mention there’s KAFKA-6880: Zombie Replicas Must Be Fenced, which sounds like the spookiest of them all. 

KAFKA-8233, which focuses on the new TestTopology mummy (wrapper) class, provides one option for setting the topology through your Kafka Screams Streams application. As Anna puts it, "This opens doors for people to build better, more resilient, and more interesting topologies." 

To close out the episode, Anna talks about two more JIRAs: KAFKA-6738, which focuses on the Kafka Connect dead letter queue as a means of handling bad data, and the terrifying KAFKA-5925 on the addition of an executioner API. 

EPISODE LINKS

• KAFKA-6431: Lock Contention in Purgatory
• KAFKA-8522: Tombstones Can Survive Forever
• KAFKA-6880: Zombie Replicas Must Be Fenced
• KAFKA-8233: Helper Classes to Make it Simpler to Write Test Logic with TopologyTestDriver 
• KAFKA-6738: Kafka Connect Handling of Bad Data
• KAFKA-5925: Adding Records Deletion Operation to the New Admin Client API
• Streaming Apps and Poison Pills: Handle the Unexpected with Kafka Streams
• Data Modeling for Apache Kafka – Streams, Topics & More with Dani Traphagen
• Distributed Systems Engineering with Apache Kafka ft. Jason Gustafson
• Kafka Streams Topology Visualizer
• Follow Anna McDonald on Twitter
• Follow Mitch Henderson on Twitter
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_00 0:00

We're doing a special Halloween-themed episode of Streaming Audio. So we thought we'd dive into some Apache Kafka internals and this year's scariest Kafka issues with Anna McDonald, a technical account manager here at Confluent. From lock contention in purgatory to tombstones that live forever to zombie replicas, you are in for, I'm going to say it, folks, a spectacular episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello, and welcome back to this special Halloween-themed episode of Streaming Audio. Recording this just a little bit before Halloween 2019. And I am very pleased to be joined in the virtual studio by a brand new coworker of mine, Anna McDonald. Anna, welcome to the show.

SPEAKER_01 0:54

Thank you very, very much. And a happy Halloween to you.

SPEAKER_00 0:58

Happy Halloween to you. Anna is a uh presenter at Kafka Summit, active Kafka community member. And hey, tell us about your new job.

SPEAKER_01 1:09

Yeah, it's this really cool company called Confluent. You might have heard of it. Uh no, I'm gonna check them out. Yeah, you really should. They're they're pretty awesome. I am uh I'm absolutely thrilled um to be joining Confluent, specifically the uh the TAM team as a technical account manager. Uh it's it's I've been uh I was uh employed at SAS for 16 years, so it it takes quite a bit to uh for me to leave, and I I will miss miss my friends dearly, but I am super excited about getting to work um on Kafka all the time.

SPEAKER_00 1:40

Yes, I love it. I'm glad that you have uh joined us. And um everybody, this was uh I want to be very clear, this was Anna's idea to do a Halloween-themed episode. And I don't know if you follow Anna on Twitter. Uh her Twitter link is in the bio. It's a thing I recommend. Um the puns, I mean, I like to think I'm good, honestly. Okay, I I I think that of myself. I think I'm pretty good at puns, right? Absolutely out of my class. So, uh, when it comes to Anna. So when if if there are, I don't know, some puns today, I just want you to be ready. It's gonna happen. Uh nobody's gonna apologize for it. It's gonna be amazing. Um, and so Anna, yeah, it's your idea to do this Halloween-themed episode and kind of talk about some scary things that happen in Kafka. I'm gonna bring these up, and a lot of them have associated Jiras, all linked in the show notes. Um, but I'm I'm just gonna kind of ask you what they are, and I want you to tell us what the problem is, and then we'll talk about how to fix or at least mitigate the problem. Uh like is it you know a silver stake through the heart? Is it a silver bullet? Is it garlic? You know, whatever these things might be. Sound good?

SPEAKER_01 2:54

That sounds awesome.

SPEAKER_00 2:56

All right. Number one, the Jira, known affectionately as Kafka 6431, lock contention in purgatory. Now, um I can you tell us what purgatory is for?

SPEAKER_01 3:13

I can, and and purgatory is fascinating. I'm um, you know, I have these things where I go down uh rabbit holes because I find pretty much everything interesting, with the exception of cooking. I do not find that interesting. I've tried for years. No, not at all.

SPEAKER_00 3:27

I I just I'm not like good at it. I I enjoy the process of it. Really? I do.

SPEAKER_01 3:33

That's that's fascinating to me. That's it, I do not have that that penchant whatsoever. I would rather do a Sev1 than a grilled cheese sandwich, legitimately.

SPEAKER_00 3:42

Okay. All right. Well, it's a good thing here, Tam. Um, so what's what's what's Kafka Purgatory?

SPEAKER_01 3:47

Okay, so purgatory is perhaps the best name that I've ever heard for something. Uh it's it's the place where things go to wait. So in Kafka, and and this is why I love it, there's there's all these different things that are magic unless you know how they work. For example, uh purgatory is used for when you you set axe equal all on a produce request. When you say, I want to minimize any sort of data loss or the chance of data loss, so wait until not only the leader for this partition that I'm writing to has acknowledged it, but I want all the replicas there too. I want to make sure that I have that level of uh safety. But the question is, okay, that's great. I love that that works. Like, how do you do that? How does that work? And it works because of a thing called purgatory. Uh purgatory is basically a holding center for delayed operations. And I called it delayed operations because that's the name of the class that implements uh or that the the uh abstract class that implements purgatory. So what it is, is it's just kind of a a place where, hey, I'm not ready to go yet. I can't send back an acknowledgement to this producer until all the replicas are done. So I'm gonna wait around in here. This operation's gonna wait until I hear back from all the replicas, then I can go. So that's what purgatory is. And it took me, uh I enjoyed looking at this way too much.

SPEAKER_00 5:10

Um because more so than making a real cheese.

SPEAKER_01 5:13

Oh, yeah, yeah. Hi, tons. I mean, there's just no comparison. Uh and there's there's lots of like little implementation details about purgatory that are fascinating. And I'm not gonna go over all of them because I think you know that's a whole different show. And plus, implementation details are usually not something that are openly documented either.

SPEAKER_00 5:31

Uh but but you can come back on after this. So it's save save some for next time.

SPEAKER_01 5:36

But I would reserve judgment on that till we're done. I mean, I'm just gonna give you that. You may not want to say that yet.

SPEAKER_00 5:43

I'm I'm I'm I'm hopeful.

SPEAKER_01 5:45

I've behaved myself so far. Um but like for example, heartbeats during a rebalance. You know, you you pause that stuff. Uh what during a rebalance, what about joining groups? Like all that stuff is delayed, and it it kind of is part of the magic that makes Kafka work. And a lot of them, the delayed operations are implemented with purgatory. So that's that background on purgatory. And without getting into the guts of it, it's a place that that delayed operations go.

SPEAKER_00 6:14

Does it take the form of a queue?

SPEAKER_01 6:16

Yeah, actually, uh well, okay, see, now you're you're tempting me, and I am I have zero, absolutely zero restraint. So it actually, there's a concurrent, uh, there's a uh concurrent hash map that has like um a key, and then it points to a concurrent linked queue that has that list of operations that are waiting. So these things are waiting on this. And there are there are uh per different types of purgatory. So there's a topic purgatory, um, there's a produce request purgatory, there's actually a brand new one, ACL uh updates. If they're using async auth, they're also using purgatory now as their uh their delayed operation structure. And all of those things live in um the the operations that are waiting on there live in a uh uh concurrent linked queue, I I believe, yes.

SPEAKER_00 7:04

Yeah. Okay. Which I mean that would that would make sense. It would indeed. That's a that's a I I I don't feel like asking really um on whether that's the deal. So that okay, cool. So purgatory is this set of or sort of map of cues where various kinds of operations go not to die, but to wait until we can determine, as it were, their eternal disposition.

SPEAKER_01 7:27

Well, sometimes they die if they time out, and then you know you get a we get an error back.

SPEAKER_00 7:32

And that sounds like that gets us into the actual problem, which is lock contention.

SPEAKER_01 7:36

Well, yeah, lo see this.

SPEAKER_00 7:38

And that might set up uh the conditions where you might get back.

SPEAKER_01 7:41

And not the outset. I'm gonna say that I chose these based solely on title. Absolutely. That that's it. So I'm hoping maybe if if you know I'm allowed back and we can do this next year, people will title issues trying to get on the show. Because I think that that would be awesome. Um so for lock contention in purgatory, I think, and I'm gonna I'm gonna mess this up. I should have written this down, but I think it was it was Uber uh who introduced this, and I I have it up. Uh or it could have been Lyft. Which one was it?

SPEAKER_00 8:14

This is 6431, the jury.

SPEAKER_01 8:15

Yes, it is. Um and they had been running this and they had seen some, I believe it's part of their work on Exactly One semantics, where they had tried to, you know, put in a whole host of performance improvements. And one of the things they were finding, and this makes perfect kind of sense, is there was a lot of lock contention in purgatory for topics that, for example, you got axe equal all, which would make sense for you know when you're doing something like exactly once, and every single produce is going to throw something in in purgatory. And what they were seeing is there was a lot of contention in the broker trying to get that lock. And so what they decided to put in is they um you know sharded purgatory into smaller partitions, which would allow them not to have to get a global lock on the entire purgatory and hence be able to not have that level of lock contention.

SPEAKER_00 9:04

Okay, so and by the way, everybody, there's a lot of things that we're gonna have to talk about today that um are internals, the JIRAs are linked, we're giving you the JIRA number, so some of this might be sort of an exercise for the student to dig into some details. I'm not going to ask Anna to explain absolutely everything she says, but some things is that a lock on I mean the the data structure is a concurrent hash map and there are concurrent queues, so you don't need to take a lock. Uh you don't externally manage a lock. So what what lock uh what what is the lock that is being contended for?

SPEAKER_01 9:41

There I think it's a it's implemented was used to be at least until this is this fix went into like 2.2. Um it was a re-entrant read-write lock and it was global. So I think that managed, you know, it for the best I can tell, and and again, this is it's an implementation detail, so so it's pretty much me reading the history and the code and the commits to figure this out, which which was super fun, actually. I really enjoyed. Um kidding. Yeah. And so so what what was ending up you know happening is the entire purgatory was was having to be uh was having to be shut down in order in or the lock.

SPEAKER_00 10:21

There was contention, contention for just the purgatory data structure. Correct. Yeah, okay. That's a setup for that's a setup for lock contention, right there.

SPEAKER_01 10:40

Yes, yes, indeed. Um and full disclosure, uh, again, like this is this I had a great time learning about these things, but I did not pick them based on knowledge that I currently possessed in any way. Right. We don't want things. Exactly. So some of this I know, and I'm gonna fully disclose when I'm like a sketch on something and and needs to be, you know, more more more research done by me or you, because it's again fair.

SPEAKER_00 11:04

Yeah, hit me up on Twitter and listeners, you should tweet at us. Uh yes, including Anna. And if you have to set us straight, set us straight.

SPEAKER_01 11:12

And maybe I'll make some crazy hypothesis and you could tell me that I'm wrong. Because people love that. They love that makes people feel like, wait a minute, that's not correct.

SPEAKER_00 11:20

Right.

SPEAKER_01 11:21

I'll do that.

SPEAKER_00 11:22

Yeah, no, totally. These are chosen because they have spooky names, and and purgatory is at least a little spooky, um, I think.

SPEAKER_01 11:29

I think it's super spooky because I don't like to wait. Like, who wants to wait? That's scary.

SPEAKER_00 11:35

I was waiting uh just a few hours ago. It was lunchtime, and I was waiting at a buffet line, and this is a confession about me. I hate doing that, and it's I just I'm so impatient. Right. So, no, I don't like waiting either.

SPEAKER_01 11:48

Yeah, it's like a nightmare. So, this to me, like I saw this one, I was like, ooh, that's scary.

SPEAKER_00 11:53

Right, right. Okay, so that's lock and t anything anything else to say about lock intentional purgatory?

SPEAKER_01 11:58

No, I don't, yeah.

SPEAKER_00 11:59

Okay, all right, let's move on. Um tombstones can survive forever. Now, uh that's this is uh Kafka 8522 is the Jira. Um tombstones in the spooky graveyard sense, of course, are relatively durable things. If you like me are in the habit when you're visiting an older city of wanting to see the graveyards, I don't think I'm all that dark of a person, but I just love doing it. Usually we kind of when you get to like the 400-year mark, um, but there's just an erosion in the gravestone, it can be really hard to read the writing. Uh, you know, you start to get into maybe it was a different language or the writing system was a little different, and that can be a struggle anyway. Um, so you know, tombstones aren't really eternal things, but we're talking about Kafka tombstones. And so to start with, what's a tombstone?

SPEAKER_01 12:50

Right. So a a tombstone is a way to clean up a key uh in in Kafka. You know, if you if you have a compacted topic, which compacted topic means that right, you know, you you have uh compacted topic. I shall. So a compacted topic means that you want one per key, you only you you kind of like overwrite it. So it's a way of kind of saying, okay, well, if multiple people write, you know, produce values for this key, I only want the latest one. Um and there's a a mechanism you know that that goes through and cleans up those old previous key values, you know, values that were produced for that key. And so that topic becomes compacted. So at any given time, um, in the active segment, there there's a you know that that's where you're gonna go and you're gonna look to find your latest key value, hopefully for the most part.

SPEAKER_00 13:42

That's and that that process will look something like let's go read all of the old segments and discard uh things that aren't the most recent key.

SPEAKER_01 13:51

Exact a mundo.

SPEAKER_00 13:53

And um, if you want a key to go away, you can from the API perspective, it's it's possible to delete a key from a compacted topic, right?

SPEAKER_01 14:01

Mm-hmm. Yes, yes, yes.

unknown 14:03

Yep.

SPEAKER_00 14:03

Um and you do that by just you produce a null value, right?

SPEAKER_01 14:07

Mm-hmm. Exactly. Yes, yep.

SPEAKER_00 14:09

So how does that get done?

SPEAKER_01 14:13

So this is this was really interesting, and this is all Mitch's fault, who is another Tam.

SPEAKER_00 14:18

There's so many things are Mitch's fault.

SPEAKER_01 14:20

I know.

SPEAKER_00 14:22

Yes, it's I don't even want to start with that.

SPEAKER_01 14:24

There's a long list. It's like it's a I mean, he's like, you know, very like I told him, him and Danny really are are primarily the reason why I am working at Confluent and Dick.

SPEAKER_00 14:35

And we only we run, you know, we shoot for like a 45-minute kind of runtime. So if we get into the things that are Danny Trapagon's fault or Mitch Underson.

SPEAKER_01 14:42

No, no, no, Danny, nothing's Danny's fault. Yeah. Nothing's Danny's fault.

SPEAKER_00 14:46

Okay, Danny's also previous guests on streaming audio.

SPEAKER_01 14:48

Danny is my ride or die, so it defaults to Mitch, right?

SPEAKER_00 14:54

Mitch's fault. Uh we'll link Mitch in the in uh the show notes, his Twitter handle.

SPEAKER_01 14:58

I agree with that. So so so back to tombstones. Um so it it actually what we were talking about, we were talking about log compaction, we were talking about log cleaner threads. There's a there's a problem with log cleaner threads where if they die and you don't know it, obviously that can be bad because nothing gets cleaned. Nothing gets cleaned up and and things, you know, the things start to fill up, and it's just it's not it's not fun. And so we were talking about the intricacies of log cleaning, yada yada yada. And I started to look into this, and then I stumbled upon this Jira, and that was where the whole idea for Halloween came out. I was like, this is the best title ever. Uh, Evelyn Bayes opened this ticket. I, you know, best title ever. Tombstones can survive forever. It just sounds spooky. So I was like, this is fascinating. And at the end, it's really neat because this kind of covers this JIRA almost all the like you have to, in order to understand this problem, you have to understand pretty much almost everything about log compaction. So I think it's a really great way to dig into it and make sure that you, um, if you're interested, you know, make sure that you're well versed in in kind of that the edge cases, which as a TAM, you you should be. So for me personally, right, this is kind of an exercise in gearing up to do some super fun stuff. So um, in this specific case, what you want to happen is in a compacted topic, is you want to say, okay, I've produced you know a bunch of values for this key. I'm done with it now. You know, and let's use a concrete example. We'll take an address. I have this customer, and I've produced, you know, lots of address updates when they change, blah, blah, blah. But all of a sudden they're not a customer anymore and I want to get rid of them, right?

SPEAKER_00 16:39

Or they or they they GDPR themselves out of existence.

SPEAKER_01 16:42

Great, yes, example. Right, exactly. Right? So we need to get rid of that. So we send a tombstone in. Now, you can't just immediately get rid of the tombstone message. Because, and you have to be very careful with how you clean it up, because there are a lot of probably, hopefully, app, you know, consumers that have consumed information about this key. Those consumers, let's say one's offline, one's having maintenance, something's happening, you want to give them the opportunity to consume that tombstone so they can clean up local state stores, right? So there's a little bit of magic that happens in terms of cleaning up tombstones. And I think I, you know, that that hopefully makes sense.

SPEAKER_00 17:27

Is it like a a uh grace period um after the tombstone is written that you you want to keep propagating it for some period of time?

SPEAKER_01 17:36

Right, right. Yes. So so you can't um there's this concept of a delete horizon, and this is where it kind of gets a bit like I said, if you want to understand log compaction, this is a great ticket. You know, read read this ticket. Um and it make what you're doing is you're saying, I want to wait to clean up this tombstone um until like the earliest time it could be cleaned up is the delete horizon. And that is actually not something that's concrete, which is what causes this problem. So it's supposed to represent the last mod time of the segment containing the first dirty offset, and that's an offset that isn't that hasn't been cleaned. So that's the first, you know, the first first one that that might not be in the latest kind of compactedness. And so so and that that's all well and good, except for there's a thing that happens, right? So when segments are clean, they merge. So you could clean three segments and you would merge it into a to one, right?

SPEAKER_00 18:44

That's the whole point of being a compacted document.

SPEAKER_01 18:46

It is other right. It doesn't make sense to keep around half empty things, kind of like boxes of tissue. I personally merge tissue, like if I because I hate that when there's like three in one box, so I'm always putting you know tissue into there's like a discontinuity, there's a point at which you pull out a tissue and the next tissue isn't the same. I'm willing to go, yeah. That doesn't bother me.

SPEAKER_00 19:10

But you're willing to put up with them to have it.

SPEAKER_01 19:11

Yeah, that doesn't bother me. But anywho, so it's kind of like you're compacting things, right? And so the issue that you have there is added on to the delete horizon time, that that last mod time is the retention time. And so they say, okay, we're gonna that this tombstone can't be deleted until this much time after the delete horizon. Well, if you compact multiple segments and it's the last one, that last one could have offsets that were written right before the tombstone. Or sorry, after the tombstone. So what ends up happening is the date never hits. So you've got a newer offset than the tombstone, it gets the last the um delete horizon from that offset.

SPEAKER_00 20:00

Okay, how can that happen when I've got older segments? If I've got older segments, won't their values of the key always have older timestamps than the tombstone?

SPEAKER_01 20:11

No, because let's that you gotta think about it this way, right? That tombstone, it has to wait anyway, right? And it it depends on how there's like some specific conditions, right? So so you have to have a small amount of throughput because if you don't, then like you, you know, you said you're gonna be consistent, you know, cleaning these. So you have to have a small amount of throughput. And what ends up happening, if you have a min clean uh dirty ratio, which is something you use to make sure that you're not cleaning all the time, because that can be very intensive, but some people set it very low, then what ends up happening is when those two things meet, you end up having um basically constant cleaning. Everything that comes in is gonna get compacted very quickly. And so new so segments like you that have a later offset than that tombstone, those are driving that delete horizon that time. So, so and that's kind of kind of a uh you know uh a bad thing, obviously.

SPEAKER_00 21:14

Because that means that that tombstone gets propagated to the next clean segment.

SPEAKER_01 21:19

Right. Um and the Jira says it best says what happens is all the data continuously gets cycled into the oldest segment, which is not good because then it's newer and bam, and that tombstone will just live forever.

SPEAKER_00 21:33

And we uh we don't want that.

SPEAKER_01 21:35

No, no.

SPEAKER_00 21:38

That's uh Kafka 8522, the JIRA, tombstone tombstones can survive forever. And the the whole I think we talked about this enough at the beginning, but the whole idea of of a tombstone um it just makes me think you see this also in um uh log structured merged trees, uh like RocksDB and like the same. This thing where you can't really and so I guess fundamentally when the thing that you're writing is an immutable data structure, which a log is after you've produced it, you can go back and change old things. The way you delete has to be by you write this special thing. And when you dig into these systems, there are always these funky edge cases with tombstones. That take a while to get cleaned up.

SPEAKER_01 22:20

And this is the coolest one because the what ends up happening is, you know, if you look at this here, it's great because people are like, oh, well, we'll just do this and it'll be great. And then I like to think of Jason Gustafson. It's Gustafson, right? Is that right? Yes. Score. Um also he's awesome, obviously. I wish he could narrate everything in my life. He's like so rational. Um, but he comes in like Vincent Price style at the end, and he's like, I think there's a little more to this. And they end up actually in the KIP, and it's not you know f finalized yet. I don't know if it's been accepted, but um, in the kip, they're actually gonna rewrite the checkpoint file structure. Um yeah, yeah. Yeah, to cover transactions too, because there's and that's that's kind of what I like about this the best, is because everyone's like, all right, let's do it, let's do it. And Chase's gustoson's like, I think there is a little bit more to this. Dun dun, which makes it even more Halloween-y. Like lightning crash.

SPEAKER_00 23:19

Creaking wooden door the outside of the crypto.

SPEAKER_01 23:22

Totally.

SPEAKER_00 23:25

Yes. Um, all right. Uh, so that was uh Tombstone's Living Forever. Next, um we have Undead Replicas.

SPEAKER_01 23:35

Okay.

SPEAKER_00 23:35

This is Kafka 6880. Zombie Replicas must be fenced.

SPEAKER_01 23:42

That again, what an awesome name. That's hilarious. It is, right? It's so good. And this one is pretty straightforward. Um adding um the epoch stuff is is key. And and I think it's really been successful at handling a lot of these cases where um where things go where where things go awry and then they come back, and and when they come back, they don't know things have changed. And that's kind of I think underlying this ticket is in.

SPEAKER_00 24:15

So walk us walk us through that. You can assume everybody knows what a replica is. We're we won't cover that.

SPEAKER_01 24:19

Okay.

SPEAKER_00 24:20

But what's that epic thing?

SPEAKER_01 24:21

So so that just means that like there's a generational aspect um to a bunch of different operations. Like uh, there's a generational aspect to uh uh like uh a partition leader. And in this case that's important because let's say that and and this is a great example, uh broker one is the leader for this topic for partition zero. There was used to be nothing to distinguish broker one being the leader, you know, the first time, and then someone else being the leader, and then broker one being the leader again. There was nothing to distinguish between those two times.

SPEAKER_00 25:02

And that's okay.

SPEAKER_01 25:03

Yeah, and that's a problem.

SPEAKER_00 25:05

Yeah, no state in the replica that said, I was and then I wasn't, and then I was again.

SPEAKER_01 25:10

Right.

SPEAKER_00 25:11

There was why is that a problem?

SPEAKER_01 25:13

Well, that's a problem. Uh, and I think you know the the description in this gear is is very good if you if you go read it, and it because sometimes things think they're okay and they're not. And in this example, you have um and the concept of a high watermark. A high watermark is the highest offset that's available to be consumed. It's the the highest committed, the last committed message, right? And um what ends up happening is is we know that this all happens in an you know in a fashion where you the sometimes broker one will be, you know, broker one's a leader, and um, and it's up to this offset, broker two is up to this offset, and broker three might be a little behind, right? And so what what what let me see if I'm trying to think of a good way to say this, and it's hard for me, like I love pictures, and so this is very this is very fun, this podcast with the no pictures.

SPEAKER_00 26:11

Yeah, it's no pictures.

SPEAKER_01 26:12

So I mean yeah, it takes some getting used to. So what ends up happening is the and and I'm just gonna go through this scenario. Broker one is the leader, and let's say that broker one is super speedy and it's up to offset 50. Broker two, replica, so also super speedy, offset 50. But broker three is a little slow, right? It's a little slow, so it's behind at 40. So the high watermarks 40. And then broker two goes off the reservation for some reason.

unknown 26:43

Okay.

SPEAKER_01 26:44

Something happens, it fails to reestablish a session, and while it's off the reservation, it misses a real it misses a re-election. So all of a sudden, it still thinks that broker one is the leader, but it's missed a re-election. Now it's broker three.

SPEAKER_00 26:59

Okay, so it um was partitioned, basically. It was just off for having a long GC pause or say it was Right.

SPEAKER_01 27:07

Who knows? There's bugs and things that cause it, right?

SPEAKER_00 27:10

Right.

SPEAKER_01 27:11

Yep. Um, and and so so what ends up happening is when it comes back in, it thinks leader, it thinks broker one's still the leader. So it's gonna say, hey, I'm here, hello, and broker one's like, I'm not the leader. I'm not the leader. And broker one and broker three are gonna go ahead and continue to be like da-da-da-da-da-da, we're fine. So the issue then comes in. What if broker one is then again re-elected to be leader? When it's re-elected to be leader, if there's data that's kind of built up since then, and remember broker two, the one that went off the reservation, was pretty speedy, right? Uh-huh. Yep. So if broker one gets leadership back, and in the meantime, the offsets have climbed to be up to where broker two was before, because remember they were both at 50 and three was a little behind. You've got those 10 offset, or you've got those offsets that broker two had that never were committed, but it thinks now that they have been. As soon as broker one comes back online, it's like, oh, there's my leader.

SPEAKER_00 28:18

Oh, and and he's a 50 and everything's fine.

SPEAKER_01 28:21

And we're so cool. And the problem is it has no idea that there were two different leaders, that there was even nothing, you know, it's not the same leadership because there was no so there was no sense of an epoch.

SPEAKER_00 28:32

So it's this is a legitimate split brain right uh edge case here.

SPEAKER_01 28:36

Right, exactly. Um, and so that's what that this is that's what this is handling. And I think you know, the the more a lot of things are going, you know, this way more and more and more and more, and it's just building in more better and better and better and better safety um in terms of these edge cases. And I find I love, I just I love this stuff. I find this fascinating.

SPEAKER_00 28:54

It's so good. And so the fencing, can you tell us how that works? Like what well exactly.

SPEAKER_01 28:59

So what will end up happening is it'll it'll when it when it goes to try and fetch, broker one will be like, you're the wrong epoch. So no, it's not gonna happen, right?

SPEAKER_00 29:10

Oh, and and that epic is a counter of the number of times we have elected.

SPEAKER_01 29:15

Yes. So so it's gonna know it it's gonna say, okay, well, I'm the leader, but I'm a different generation of leader. I may be the same broker, but I'm not the same generation that you have. And so it's gonna know that that's garbage. Like you left, you know, you weren't here for all the troubles that we had. You you're not, it's not gonna happen.

SPEAKER_00 29:35

Right. And so it it you're you're able to keep it, keep the zombies uh outside.

SPEAKER_01 29:41

Right, exactly. It knows that there's something wrong, it knows that that that there's a mesh, there's a mismatch mesh, I should say.

SPEAKER_00 29:47

Yes, and nice, and there's some remediation process, which we will leave for a future episode, maybe next Halloween. Yes. You don't just want to, I mean, I guess you could just kill the zombie, but it'd be great if you had like some antidote. You could just bring it back to its normal self, its previous self.

SPEAKER_01 30:03

It would yeah, it would be if you truncated to like the high water mark that there was from that epoch and then just pulled in, right? I mean, that would work. And I think that might be what it is. I don't know, yeah.

SPEAKER_00 30:14

We we don't have to talk about that in the future. You just solved it. Okay.

SPEAKER_01 30:18

Until Jason Gustavson's like, but wait, that's yeah. I don't think so.

SPEAKER_00 30:28

Telling us of the the uh uh evil creatures lurking.

SPEAKER_01 30:33

He's the best, though. Like I love yeah, I love reading. This stuff's so good. He's good.

SPEAKER_00 30:40

He's uh uh very sharp guy. Okay, next scary issue. This is the Jira Kafka 8233. It's a new test topology driver.

SPEAKER_01 30:55

Yes, that's don't read that title. See, this one I made an exception for because it's a it's a wrapper, so I figured I could call it a mummy.

SPEAKER_00 31:03

See, that's the thing. It's it's it's called it it's a wrapper class. We're not gonna call it a wrapper class. We're calling it a mummy. It's the mummy, it's the test topology wrapper because mummies um are wrapped.

SPEAKER_01 31:17

Yeah, they are, yes, in in some kind of fabric with individually wrapped for freshness.

SPEAKER_00 31:22

So um tell us about, by the way, if if you could just start with a little bit of uh background on test topology driver, I think we've talked about it on the show before, but it's been a while and it's so important. Yes. And then tell us about what's what this jurisdiction is.

SPEAKER_01 31:37

Well, and now we're kind of in my wheelhouse. So I'm learning, you know, a lot of kind of but Kafka Streams is kind of my sauce. Like that's you know what I spend a lot of my time doing. And so the test topology driver is a way to set up your topology, which to define topology, it's the sync sources and nodes that go through a Kafka Streams application. Uh also on Twitter, and and I should have written this down, there's somebody who did this. It's really cool. It's a way for you, you go in and you paste the topology that you can get printed out of a Kafka Streams application, and it makes like a super cool graph. I did link it on Twitter, so there's a tweet out there on it. Um maybe I can tell you, Tim, after we could could we add it in the show notes. Yeah, we can do it in the show notes. I think it's great. I really think it's great. And so to understanding topology is very important.

SPEAKER_00 32:25

It's it's what the flow is of your Kafka Streams application, and obviously that's it's good to visualize that too, because when you're in Kafka Streams development mode, you're lost in the API and in the code, and you're trying to get stuff to work and doing this, that, and the other thing. And it's not like you have that many knives you can cut yourself with, but you it's code, you know, you can. And so seeing a picture of it.

SPEAKER_01 32:44

I'm thinking Edward scissor hands. I mean, I'm just saying, you know, it's there's a little flailing there. Like, woo.

SPEAKER_00 32:52

And then you look at a picture, you're like, oh, I'm actually really bad at this.

SPEAKER_01 32:56

Yeah, yeah.

SPEAKER_00 32:57

So right.

SPEAKER_01 32:58

You're like, what have I done? It's Frankenstein. What have I created? Kafka screams, but sorry, hashtag pun. But but part of that, here's the thing, though, part of the reason why I think people do flail so much is because it's really it's it's really hard to test in a way that straightforward. Um the with the the cur with the current topology test driver. You need to to, it's and it's not even you need to understand, because I think you could have like a really full understanding and still think this is just a frigging pain in the butt. Um and and part of the you know, the problem is in order to kind of set this up, you you have to create these consumer records, uh class, you have to create consumer records, you have to pipe those into a topic, then you have to retrieve producer records, the um CERTI stuff, like serializate, like your serializer and deserializers, those can be a pain to figure out like how to specify. Like it's just it's a lot to set up. And if you imagine you have a Kafka Streams um application that has a very complicated topology, those input topics, I mean, it it can really be a burden.

SPEAKER_00 34:08

It's not like there's one and you do some stuff with it and there's one output topic. It's it's right. It's an interesting graph.

SPEAKER_01 34:16

Yes, it's rarely that. Rarely.

SPEAKER_00 34:18

Right.

SPEAKER_01 34:19

Um, and so this one is kind of very uh very near and dear to my heart. I was talking to um Tayus about this actually be right before this was um written, because uh for my Kafka Summit presentation I did a demo, and in the demo, I used like a lightweight refactored fluent wrapper for the testing because it was I just didn't want people who were new with it to have to take that on as well. So I you know, there this opens doors to me for people to build better, more resilient, more interesting topologies because now you can kind of play around with it, it's more friendly.

SPEAKER_00 34:58

Um so the the but the essence of of the JIRA 8233 is that there's a new wrapper class, or mummy, we're saying.

SPEAKER_01 35:06

Yes.

SPEAKER_00 35:06

For test topology driver, and um it's just like sum up, sum up what's better about it. It's a better API, but how is it better?

SPEAKER_01 35:14

So so in this API, you can just specify a topic and you just say, boom, I'm going with this. I'm putting stuff in this topic. You don't have to say, I'm gonna go and I'm gonna create a consumer record. Let me give this consumer record a four you don't have to do that. You just specify that it's more straightforward.

SPEAKER_00 35:30

Um that's all that is all boilerplate. I mean, you're doing that all the time, but you know, computer uh that I guess is one of those things where you step back and you say it's too bad we're not using computers for this.

SPEAKER_01 35:40

I know. What how unfortunate.

SPEAKER_00 35:42

Right.

SPEAKER_01 35:44

And I and it and I think um, you know, that I think this class, like this is a great start. I think we need to extend this further, um, especially because I'm kind of a huge fan of, and and full disclosure, I did not have to struggle with this. Uh, there's a very talented developer named Sat who did this for us. And uh I remember that and I felt super guilty um because it wasn't fun. And so I think about I have a list still of the troubles that that he had, and I think there's more room to improve on this. And this is like a case where if you're somebody and you want to start contributing, um, especially if you're someone who likes fluent style programming, like that's kind of selfish of me because I do, but oh well, whatever. Talk to streams. Yeah, yeah, yeah. Totes. Um, then I would, you know, building on this and and and kind of fleshing out more of a fluent um rapper rappers and and stuff like would be great. I would love that. Um, maybe I'll do who knows? Maybe I have enough time.

SPEAKER_00 36:43

We'll see. You will. And that's and that's I think a good reminder because this kind of improvement, you know, when you see this kind of improvement, and when you've used the previous thing a lot and you're like painfully aware of its shortcomings, and like, oh, it seems so obvious, of course. Now this is here. I mean, it never does the first time through, right? The first time through, you're like, hey, we have a test class, and that's great news. And like you make the best one you can, and then you realize, oh, this needs to get better, and and you get to make it better over generation.

SPEAKER_01 37:09

Yeah, it's I mean, uh like liter when you were saying that all I could say is, hey, look, there's a way to test it. That's awesome. Right? Like score one. Move on.

SPEAKER_00 37:20

Docker compose up is not a part of running my unit test.

SPEAKER_01 37:24

Hey, check it, right? And I'm not I'm not hating on that, you know. I just think, especially now, and again, full disclosure, I'm like the huge Kafka streams fan of the wazoo. Um, but it's I think there's just so much room to do so many cool things. And the more we can let people play with topology and you know, find those edge cases for themselves and say, what if I change this order? Well, what if this does this first? Well, what if I do, you know, I think it's gonna be will move faster and faster. I think it's gonna be great. I'm very excited about it.

SPEAKER_00 37:56

Nice, nice. It's good to see it happen. Okay, next one.

SPEAKER_01 38:01

Okay, this, yeah, go for it.

SPEAKER_00 38:03

Kafka Connect dead letter Q.

SPEAKER_01 38:06

Yeah.

SPEAKER_00 38:08

Go ahead.

SPEAKER_01 38:09

I was just gonna say this one is is all Mitch. This was his suggestion. This one, yes, yep. All right, this is Mitch's contribution.

SPEAKER_00 38:19

This is Mitch's contribution to the list. So if it doesn't seem like it's as good as the other ones, uh. Right. Hashtag blame Mitch. That's right, because it's because Mitch contributed it, but it's pretty good still. And like I guess if you're used to you know old school messaging or anything like that, that letter Q doesn't seem scary. But you need to think about it. That's actually really spooky. You're talking about these things that are dead, and you're keeping them around, and that's morbid. But anyway, Anna, tell us about the Kinect deal here.

SPEAKER_01 38:48

Well, you know, and I looked at this and I I actually think that it's pretty cool. And and the reason I think this is pretty cool is because to me, it's it's it's kind of okay, you call it a dead letter, but it's almost automated error reporting for these bulk transfers that you would do via Kinect. So apparently what used to happen with Kinect, and again, full disclosure outside my wheelhouse, but if something like failed on, you know, there was like a serialization error or it just blew up, which is bad, obviously. That's not ideal. You don't want your whole thing to fall over, which will also, you know, happen in Kafka streams. And um Loick did that great presentation at Kafka Summit about poison pills. I think that was excellent if you want to know how to handle you know those kind of of errors. That he did a fantastic job at that.

SPEAKER_00 39:32

We will put a link to that episode up.

SPEAKER_01 39:34

Yeah. So um, and this is a way to automate that. And in the beginning, I was like, yeah, it's a dead letter Q. I dislike all other messaging systems, yeah. And then I looked and I'm like, oh-h. Because the coolest thing about this is there are headers that you can add to the message that you know broke everything that give you more information about what went wrong. And that is cool. Um, so this is connect. Connect is in a when it dies, it adds headers.

SPEAKER_00 40:04

Right. Like what kinds of things.

SPEAKER_01 40:06

And how cool is that? I that is you get like a stack trace, or what you get like, okay, so so you get um the topic that contained the message, the numeric idea of the partition, the offset, uh, the class name that caused the error, um, the fully qualified class name, the message in the exception, and the stack trace. Yep.

SPEAKER_00 40:24

And Mitch's phone number and email.

SPEAKER_01 40:26

Indeed, right. Yep, yep. Direct line.

SPEAKER_00 40:29

Excellent. That's the best juro yet. All right.

SPEAKER_01 40:34

Yeah, yeah. So very cool. Go ahead. I was just gonna say, so very cool, yeah. I mean, that was a win for Mitch. I like that one.

SPEAKER_00 40:41

It is, it is, he needs one. Um, last one. This is uh this is a good Halloween costume. If you want a Kafka-themed Halloween costume, you could dress as the Executioner API. And for more details on this, you read uh the Jira Kafka 5925. Anna, tell us about the executioner API.

SPEAKER_01 41:00

Okay, so this one, this one uh, you know, also was recommended by Mitch, and I have to concur that this is kind of scary to me. Um, you know, and and and I this is why I call it the executioner, because what it's trying to do is is the give you the ability to delete messages from a specified offset inside of a partition.

SPEAKER_00 41:20

Um talk to me about that. That sounds like yeah.

SPEAKER_01 41:27

Yeah, I don't, I'm like this is this is a scary one because it scares me that people would do this kind of.

SPEAKER_00 41:34

That's the I mean, this is like uh what I always tell people about the immute immutability of logs. If you're editing making a change to an application log file that's not appending a line to the end, you're probably a criminal. Uh-huh. I agree.

SPEAKER_01 41:49

I concur with that opinion.

SPEAKER_00 41:50

So this is for crime. Uh let's establish that. All right, so good. Indeed. Tell us more.

SPEAKER_01 41:56

Uh so I think part of it is that you used to be able to do this, as far as I can tell. Um, and so so there this is this is kind of a way to put that back in without relying on time-based or size-based log retention policies. I'm like, ah, you know, because why does it bother you that it's there? You know, tombstone, the individual record, if it's like something for it.

SPEAKER_00 42:21

So I'm I need to read that would be for a compacted topic. We don't need log compaction on for this tool.

SPEAKER_01 42:27

No, right. That's true. That's true. Yeah, there you go. That's true, that's true.

SPEAKER_00 42:31

Um potential um is is is GDPR in view here? Is that discussed in the JIRA?

SPEAKER_01 42:40

It doesn't really say it's it doesn't really say so. And the thing that gets me about this is it's uh it says deleting messages starting from a specified offset. So the the GDPR thing would to me be more of a targeted thing where I want to delete this. This is like I just want to start here and just execute, just you know, decimate. Gotcha. Just take off. And I don't know why. Um yeah, I'd be interested. If anybody is, you know, I'd be interested to know more about the use case for this.

SPEAKER_00 43:08

The the use case that gave rise to this. Indeed.

SPEAKER_01 43:11

The backstory. I think we could call it a backstory. Let's we could.

SPEAKER_00 43:15

Because you know, he's an executioner and right, like the punisher. There, he's got a great backstory. He's definitely an executioner.

SPEAKER_01 43:23

What happened to your topic that would make you want to do this?

SPEAKER_00 43:26

Yeah, and you you learn the story, you're like, oh, okay, you know, I bet if that was me and I had those skills, I'd want to do that too. Yep. Uh and like you wouldn't, but you at least you can see where it comes from. And so we um are interested in that story. So if you're a person who has background on the use case to this juror, we'd also love to hear from you and the appropriate Twitter handles uh for Anna and for Conflict are in the show notes. So please uh talk to us. Anna, this is such a great idea.

SPEAKER_01 43:57

I just can't get over it. The way I feel about it is after this, there's nowhere to go but up. You know, as we gain more, as I gain more knowledge. This was really fun, though. This was so fun.

SPEAKER_00 44:09

I actually have learned a a number of things. Uh didn't quite understand zombie replicas, um, didn't know what purgatory was, straight up, everybody. Didn't know what purgatory was before we started this. These are the things I always say when I ask questions. Sometimes I'm asking questions to clarify for listeners, sometimes I'm asking because I don't know, and I learned uh several things here. So I really, really appreciate it, and I am so glad that uh I have you as a coworker now and look forward to the next time you can be on the show.

SPEAKER_01 44:40

Sweet.

SPEAKER_00 44:42

My guest today has been Anna McDonald. Anna, thanks for being a part of Streaming Audio.

SPEAKER_01 44:46

Thank you very much.

SPEAKER_00 44:48

And there you have it. Before I go, I want to tell you that we have a pretty cool new offer to help you get started with Confluent Cloud without you having to pay for anything. If you're a new user and you go through the regular sign-up process and start using Confluent Cloud, your first $50 of usage per month are free. This will last for the first three months after you sign up. So that's $50 per month of serverless Kafka for three months at no cost to you. So go to the sign-up link in the show notes. I don't want to read you the URL, and sign up now. I think the only thing I could really do more is write your code for you. And I think we both agree that's too much to ask. So check it out and hey, let us know how you like it. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can reach out to us on Twitter at Confluent Inc. or reach out to me at TL Burgland. That's T-L-B-E-R-G-L-U-N-D. Or you can hit us up in Community Slack. There's a sign-up link for that in the show notes as well. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks a lot for your support, and we'll see you next time.