Streaming Audio: Apache Kafka® & Real-Time Data

The Present and Future of Stream Processing

• Confluent, founded by the original creators of Apache Kafka® • Season 1 • Episode 250

Share episode

The past year saw new trends emerge in the world of data streaming technologies, as well as some unexpected and novel use cases for Apache Kafka®. New reflections on the future of stream processing and when companies should adopt microservice architecture inspired several talks at this year’s industry conferences. In this episode, Kris is joined by his colleagues Danica Fine, Senior Developer Advocate, and Robin Moffatt, Principal Developer Advocate, for an end-of-year roundtable on this year’s developments and what they want to see in the year to come.

Robin and Danica kick things off with a discussion of the year’s memorable conferences. Talk submissions for Kafka Summit London and Current 2022 featuring topics were noticeably more varied than previous years, with fewer talks focused on the basics of Kafka implementation. Many abstracts featured interesting and unusual use cases, in addition to detailed explanations on what went wrong and how others could avoid the same issues.

The conferences also made clear that a lot of companies are adopting or considering stream-processing solutions. Are we close to a future where streaming is a part of everything we do? Is there anything helping streaming become more mainstream? Will stream processing replace batch?

On the other hand, a lot of in-demand talks focused on the importance of understanding the best practices supporting data mesh and understanding the nuances of the system and configurations. Danica identifies this as her big hope for next year: No more Kafka developers pursuing quick fixes. “No more band aid fixes. I want as many people as possible to understand the nuances of the levers that they're pulling for Kafka, whatever project they're building.”

Kris and Robin agree that what will make them happy in 2023 is seeing broader, more diverse client libraries for Kafka. “Getting away from this idea that Kafka is largely a Java shop, which is nonsense, but there is that perception.”

Streaming Audio returns in January 2023.

EPISODE LINKS

Kris Jenkins (00:00):

Hello, you are listening to Streaming Audio in the last days of 2022. Yes. The winter solstice has just passed. The planet begins to turn a corner and make its way to 2023. And over here in the pod cave, we're digesting a few large, rich meals for the holiday, and I guess attempting to digest a large and rich year just gone by. Yeah, it's time to get reflective about 2022. For me, it was the first year in several where we've been able to get out into the world and meet up with people in person, find out what's been actually happening, take the pulse of the industry. So I thought it'd be fun to bring in a couple of my colleagues to wrap up the year, have a bit of festive reflection, and the New Year's look at the year to come. So joining me are my fellow developer advocates, Robin Moffatt and Danica Fine. And they're here with me, Kris Jenkins, for a 2022 wrap-up edition of Streaming Audio. Let's get into it. The end of the year is upon us and joining me to wrap it all up is the excellent Robin Moffatt. Hey, Robin.


Robin Moffatt (01:15):

Hello, Kris.


Kris Jenkins (01:16):

And the ever present Danica Fine. How you doing?


Danica Fine (01:19):

Oh my God. Ever present. Amazing. So well.


Kris Jenkins (01:22):

Well, that's one of the trends of the year. I think you've been everywhere this year, haven't you, then?


Danica Fine (01:26):

Oh, man. Actually, later this week I was supposed to go, I planned to go count up all the conferences I went to. And I think it was quite a bit, like-


Kris Jenkins (01:37):

You-


Danica Fine (01:37):

... 13 or 14.


Kris Jenkins (01:37):

You've traveled-


Danica Fine (01:39):

A lot of conferences.


Kris Jenkins (01:39):

... a lot.


Danica Fine (01:40):

A lot.


Kris Jenkins (01:41):

And all sides of the world.


Danica Fine (01:42):

Was pretty good for first year as a DA. You know?


Kris Jenkins (01:46):

Yeah. You have got the air miles in a way that Robin has stopped doing.


Robin Moffatt (01:50):

I've completely, well, almost stopped doing it.


Kris Jenkins (01:52):

Almost.


Robin Moffatt (01:52):

Just two conferences this year.


Kris Jenkins (01:54):

Which two did you go to?


Robin Moffatt (01:56):

The only two that were worth going to, Kafka Summit London and Current 2022.


Kris Jenkins (02:02):

You may have been biased, as you had something of a hand in those.


Robin Moffatt (02:06):

Somewhat biased, yes. And other good conferences do exist.


Kris Jenkins (02:10):

You were chair of the program committee for both of those this year.


Robin Moffatt (02:13):

Oh, yeah.


Kris Jenkins (02:14):

Is that the first time you've chaired a committee?


Robin Moffatt (02:17):

Yes. Yeah, so I've been on the program-


Danica Fine (02:18):

Oh, really?


Robin Moffatt (02:19):

... committee before, first time chairing it, which was quite an experience. It was good fun, actually.


Kris Jenkins (02:25):

And you are handing over that mantle to Danica, so-


Robin Moffatt (02:29):

I am.


Kris Jenkins (02:30):

What's it like? Because we've all been on the committee, like rifling through abstracts trying to find good ones. But what's it like being a program chair?


Robin Moffatt (02:40):

It's quite fun, actually, because I've been to many conferences over the years and you actually get to put your own mark on it. So you have hopefully a good feel for what the community are going to be interested in and you can actually push it in that direction. And so a bit more of this kind of topic, a bit less of that one, just a bit more opinionated. Sure, I do have opinions now and then, which I generally hold back on, except on Twitter.


Danica Fine (03:07):

Robin's never had opinion in mind that I've seen. [inaudible 00:03:10]-


Robin Moffatt (03:11):

It actually says it on the job posting, like, "Shit posting and memes." It's there.


Kris Jenkins (03:15):

We'll bleep that out, thank you very much.


Robin Moffatt (03:16):

Yeah, okay.


Danica Fine (03:16):

It's on the fine print.


Robin Moffatt (03:17):

There we go.


Kris Jenkins (03:18):

Well, handing over to Danica. I think Danica has opinions, too, so that'll be just fine.


Robin Moffatt (03:21):

Yeah.


Danica Fine (03:21):

Oh, man.


Robin Moffatt (03:21):

Danica's going to be awesome at it.


Danica Fine (03:24):

Yeah.


Robin Moffatt (03:24):

Occasionally, I think.


Kris Jenkins (03:25):

You are doing Kafka Summit London '23, that's coming up.


Danica Fine (03:29):

Yeah, looking forward to it. But it is pretty reassuring. I didn't realize, Robin, that last year was your first year chairing.


Robin Moffatt (03:35):

Yeah.


Danica Fine (03:36):

So that makes me feel a lot better. So I can definitely-


Robin Moffatt (03:37):

Seamless.


Danica Fine (03:37):

... do it


Kris Jenkins (03:37):

Seamless.


Danica Fine (03:37):

That's great.


Robin Moffatt (03:43):

Yeah, definitely.


Danica Fine (03:43):

Yeah.


Kris Jenkins (03:43):

So that-


Robin Moffatt (03:43):

It's definitely the kind of thing, having been on the inside of it as a program committee member before, even if just for the boring stuff, like the platform that we use for abstracts, and who the people are, and all that kind of stuff. So having that experience already then makes stepping up to chairing less of a daunting step than if you just come in completely from the cold.


Kris Jenkins (04:03):

Yeah, decent tools, there.


Danica Fine (04:04):

Already, I feel like I didn't know what I already knew, and it's quite a bit. So just having experienced it for the past year. Yeah.


Robin Moffatt (04:11):

Definitely


Danica Fine (04:11):

It'll be good.


Kris Jenkins (04:11):

So-


Danica Fine (04:11):

It's going to be better.


Kris Jenkins (04:14):

Let's go onto that theme then, Robin. If you say you've put your stamp over which kinds of talks to emphasize, do you spot any patterns over the past year? What trends emerged in the what people want to talk about field?


Robin Moffatt (04:33):

Yeah. And it's changed slightly with Current, because Current, we broadened the scope. So it's not going to like for like. There's less of the, "Here's just how we run Kafka." Obviously, Kafka's well adopted now. Lots of people are using it. So simply the fact that you're running Kafka in itself isn't so interesting. And even Kafka on Kubernetes thankfully is also no longer particularly interesting. It's just like, "You do it or you don't do it. Either's fine by me." But conference talks need to be a bit more interesting than just that. And-


Danica Fine (05:06):

But it-


Robin Moffatt (05:07):

... so there's-


Danica Fine (05:07):

... makes it really alliterative. It's great.


Robin Moffatt (05:08):

That's- [inaudible 00:05:10]


Danica Fine (05:10):

Is that not enough? Is that not enough?


Robin Moffatt (05:12):

Not for me, it's not, no.


Kris Jenkins (05:14):

If you want to get into the Kafka Summit London program for next year, you've got to make your talk title alliterative for Danica's sake.


Robin Moffatt (05:23):

Inside tip.


Kris Jenkins (05:25):

But yeah, we're going beyond the stage one thing of Kafka, I feel.


Robin Moffatt (05:31):

Yeah. So there's still interesting talks about running it, but it's much more like real experiences that people have had beyond just like, "Well I installed it, and ran it, and here's what I do, here's what I clicked on." But actually, "Here's the kind of things that went wrong, and how we spotted it, and how you can avoid it." And real useful stuff, which is why we go to the conferences in the first place.


Kris Jenkins (05:51):

Yeah.


Robin Moffatt (05:52):

So stories about stuff that people are building with it, we've always had those, they'll always be interesting stuff that people are building. I think one of the trends with Current, with opening the screw up much more, there were alternatives to Kafka there and other stream processing technologies, but I think one of the big things there was the way in which people are looking at stream processing and lots more people are getting into that now, lots more technologies, lots more companies sometimes behind those technologies. Ways of either getting data into a streaming platform or processing whilst it's there. So that's definitely one of the ones.


Kris Jenkins (06:28):

It definitely feels like it's starting to go somewhat mainstream, as not replacing batch, which some people would like to do, but certainly as part of a nutritious breakfast as they used to say.


Robin Moffatt (06:40):

Yeah, definitely.


Danica Fine (06:42):

Well, Kris you had your panel, right? So you know-


Kris Jenkins (06:44):

Yeah.


Danica Fine (06:45):

... your-


Kris Jenkins (06:45):

We did that panel at Current, people chewing over the discussion, "Is Streaming the future? Is it part of the future? Is it a fad?" And it was definitely the sense that we are going to converge on something where streaming as part of everything we do.


Robin Moffatt (07:01):

Yeah, and I think it's definitely part of. I'm going to be really interested to see how big a part that is. I was watching just the other day, Benn Stancil did a really great talk at Current, I think it was, it's called Big Data is Dead, the End of Big Data, something like that. And the last part of the talk, I think it starts at about 24 minutes in, and he's talking about why doesn't streaming have this widespread adoption? And his general point was, "Well once it becomes boring and like, 'Well, I may as well stream than not.' That's quite useful." But a lot of the use cases, like the realtime fraud detection or the notifications, not that many people need it, particularly if you're looking at with a data engineering slant. So until it becomes like, "Well," he just put it in the context of the technologies and the conference, "It's all about the technologies."


Robin Moffatt (07:51):

Like I've just said, like, "Is it this one or is it that one?" And people go to the conferences and talk about the technologies. Once that bit becomes just a given and it's like, "Well, it's as easy to stream it, so as well just use Streaming instead of the batch option, because there are benefits like your reports show more up to date data, right? That's a useful thing."


Kris Jenkins (08:11):

Yeah.


Robin Moffatt (08:12):

It's not necessarily as useful or it's not necessarily worth the effort of adopting a stream thing if you need to understand all of these things behind it.


Kris Jenkins (08:21):

Yeah, it does need to get to the point where it's boring infrastructure in the best possible way.


Robin Moffatt (08:26):

Yeah, definitely.


Kris Jenkins (08:28):

Danica, you've been keeping a close eye on the KIPs this year. Anything?


Danica Fine (08:33):

More or less.


Kris Jenkins (08:34):

Because you're the queen of our release notes these days. Is there anything in there that's helping it to become more mainstream and boring in a good way?


Danica Fine (08:43):

Ooh. Yeah, that's a great way to phrase that. Yeah, honestly I like that it's not boring yet because my background in using Kafka was like every project we did was, "How do we make this the most complicated thing ever? And let's dive into it and figure out all of the nuances of everything." We were pushing the limits of Kafka streams, which was just so much fun. But yeah, no, just on that note, I was trying to use interactive queries quite a bit for Kafka streams. And even a couple of years ago that was a great feature. You can access the information from your state stores and hit it with a rest endpoint and whatever you want to do with that information. But it was still kind of yet a, "Finesse it." And so there were a couple KIPs that have been coming out last year or so that just make interaction queries better. So we're at version two of those and fetching a specific range of information. So it just makes it easier for the average user or maybe someone who wants Kafka to be boring, right-


Kris Jenkins (09:48):

Yeah.


Danica Fine (09:48):

... to get access to that information from a relatively complicated application. So there's a lot of KIPs like that are making it boring, which is really good for the wider range of users that we want to be using Kafka, right?


Kris Jenkins (10:03):

Absolutely. And it frees us up to do the more interesting stuff. We've had some really interesting, what I would've thought were fringe use cases this year. Particularly, I'm thinking of on the podcast we had Simon Aubrey, who I'm sure you both know, talking about using AI to track animals in his back garden in real time. And that was really cool.


Robin Moffatt (10:26):

I read that blog, that was brilliant.


Kris Jenkins (10:27):

Yeah.


Danica Fine (10:27):

Yeah.


Kris Jenkins (10:28):

Yeah. It's like, "I'm getting an alert because there's an 89% chance there's a koala wandering through the backyard."


Danica Fine (10:36):

Did he show a picture of his backyard? I'm very curious. Is that something that we're at risk for, like a koala just wandering through his backyard? That's-


Robin Moffatt (10:43):

Not in Yorkshire-


Danica Fine (10:43):

... a question-


Robin Moffatt (10:43):

I'm pretty sure.


Danica Fine (10:43):

... for additional podcast.


Kris Jenkins (10:47):

I think there's a larger risk of a giant poisonous spider wandering through his backyard, but that's also worth having realtime data about, right?


Danica Fine (10:54):

Yes.


Kris Jenkins (10:54):

Yeah. But meanwhile, again, I don't want to use the book word boring as a pejorative, but we've had some of the more living with it type topics, like we had a podcast with Nikoleta Verbeck on The Things You Need to Know Once You've Got Past the Basics.


Danica Fine (11:14):

Yes.


Kris Jenkins (11:15):

Right. And I know you did a blog series-


Danica Fine (11:17):

There's-


Kris Jenkins (11:17):

... with every- [inaudible 00:11:17]


Danica Fine (11:17):

... so many of them.


Kris Jenkins (11:17):

Yeah.


Danica Fine (11:19):

So many of them. Yeah, that podcast was so, I'd say inspiring, just because of my experience with Kafka. Because you wish that you knew ahead of time all the things that not can go wrong, but all the things that you might just miss, right? Because it's a pretty-


Kris Jenkins (11:32):

Yeah.


Danica Fine (11:33):

... complex system for better or for worse. And so we got together and put together a blog series that we hope to continue indefinitely because there is just that much information to cover. But really taking people back to just the beginning, the absolute beginning. And like, "Oh, if you're encountering an issue with Kafka, well, I'm not just going to give you a bandaid, because, well, most of what we want is, 'How do I fix this specific issue that I encountered right now?'" But when we do that and I've been guilty of that, "I just want this fix right now. Help me." I don't understand. I don't understand what caused that problem or if there's something else that I'm doing in my system that made this issue arise. So that's what the blog post walks through is, "All right. How do we take everyone back to the beginning and actually understand the nuances of the system, and some configurations that we may be playing around with, and not knowing that they can cause issues down the road?"


Kris Jenkins (12:25):

Yeah.


Danica Fine (12:26):

And maybe that will make things more boring as well. So that's the theme. Let's make Kafka boring.


Kris Jenkins (12:32):

Yeah. We might have to try and rephrase this for the title of the podcast, but it's like-


Danica Fine (12:40):

I don't know if that's going to fly.


Kris Jenkins (12:41):

We're reaching that point where you can build a producer and a consumer and get real time data, and think, "This is great. Suddenly my data is being processed in real time." And then to pick on a victim, linger.ms, you set linger.ms to 100 instead of whatever the default was, and things get a bit better. And you think, "Oh, great." But you've got to start developing that deeper mental model of what's going on in the way we probably have as an industry, quite a deep mental model of relational databases without realizing it. You know.


Danica Fine (13:15):

Mm-hmm. And with things like linger.ms, as soon as you start playing around with those levers, they're not just in isolation. Right?


Kris Jenkins (13:24):

Yeah.


Danica Fine (13:25):

That's going to affect 14 other configurations that you've never heard of and you never wanted to deal with.


Kris Jenkins (13:30):

Yeah. And you have to have the mental model of what it's doing, because otherwise you're just randomly setting numbers and hoping. Yeah.


Robin Moffatt (13:38):

Well, it's that great quotation, isn't there, about, "Randomly jiggling things until they unbreak." I can't remember who said it, but it's this idea that you actually have to understand from the bottom up, like, "What's actually going on." You can't just find the top hit on Google, and make that change, and then hope that things will just fix themselves.


Kris Jenkins (13:55):

Yeah.


Danica Fine (13:55):

But usually that top hit on Google fixes it. And then we just move on with our lives.


Robin Moffatt (13:59):

Yeah.


Kris Jenkins (14:00):

Yeah, but you always end up-


Danica Fine (14:01):

And then it breaks two weeks-


Kris Jenkins (14:02):

... with a better longer-


Danica Fine (14:02):

... later.


Kris Jenkins (14:02):

... term fix if you understand just one step below the abstraction. Right?


Robin Moffatt (14:07):

Yeah.


Danica Fine (14:07):

Mm-hmm.


Kris Jenkins (14:08):

Have you seen any good talks about that, Robin, the getting past Basecamp into stage two of Kafka adoption?


Robin Moffatt (14:19):

I'm not sure if it's what you're asking. I saw a great talk just last week from Brendan Gregg, I think from 2012, about-


Kris Jenkins (14:26):

Oh, really?


Robin Moffatt (14:26):

... performance analysis.


Kris Jenkins (14:27):

Right.


Robin Moffatt (14:28):

So nothing to do with Kafka, but about troubleshooting performance problems. And it was fantastic. And it's funny, 2012. God, that sounds like, "Yeah that's ancient." In technology years, it's like, "Surely that's out of date." But it is as relevant now as it was then. And it always will be, just like talking about how you take a methodical approach to troubleshooting performance, and going through some of these great anti patterns. There was the streetlight anti pattern, and this idea that you hit a problem, and you obsess over the things that you can examine. So you pull up a WebGUI, and it's like, "This thing has got this spike. Oh, wow. Here's this spike. I need to go and look at that."


Kris Jenkins (15:07):

Yeah.


Robin Moffatt (15:08):

And the story that goes with it is there's a drunk on a pavement, he's looking for his keys underneath the streetlight. And the policeman comes along, he says, "Oh, can I help you, sir?" So he's like, "I've lost my keys." The policeman says, "Oh, did you drop them here?" He says, "No, I didn't drop them here, but it's the best light here." So it's this idea that you don't want to be looking underneath the streetlight. You need to actually understand what it is. And it's a really, really good talk. I'll make sure we put it in the show notes.


Kris Jenkins (15:33):

Yeah. But sometimes so hard to know if, to stretch the metaphor, there's only one streetlight in the area, that's the only place you really can look. It's often hard to know what you should be looking at under the hood, like Nikoleta's-


Robin Moffatt (15:50):

I just-


Kris Jenkins (15:50):

... list of tuning parameters, you wouldn't know to go and look for those unless you had someone guiding you, right?


Robin Moffatt (15:56):

Well, I disagree. I think you need to-


Kris Jenkins (15:57):

Yeah?


Robin Moffatt (15:57):

... understand. Okay, so advertised listeners in Kafka, let's pull it back onto topic, as it were. And that's one of the advertised listeners, is people listen to this haven't come across it. It's like to do with how Kafka brokers communicate with the client and you have to configure it correctly, particularly on things like Docker, or if you're ECT or whatever, because otherwise things don't work, or they work a little bit and then they stop. And it's really confusing. And it comes up again and again over the years on Stack Overflow and everywhere. And there'll be answers like, "Well, just hard code your hosts file." Which is a horrible approach, but it kind of works until then it doesn't work when you input in environments. You have to understand what's going on.


Robin Moffatt (16:37):

And this is one of those things I get a little bit, I won't negotiate it, or whatever the word is, because whatever you're working with, you have to understand it enough to be able to troubleshoot it. And that's sometimes a bit idealistic, and sometimes people have just got a deadline, and they need to get the thing written, and like, "Just give me the answer."


Kris Jenkins (17:00):

Yeah.


Robin Moffatt (17:00):

But if you're serious about understanding technology, you need to understand, like, "Well, if you've hit this error, what's the error mean?" And go and look up how the listeners work in Kafka. And I wrote a blog about it. And it gets a ton of hits because the actual documentation in Kafka, it literally explains it but it doesn't explain it enough. So it's, I guess, a gap in the docs. But that's why I like blogging. You find something that's not documented so well, "I've got to write a blog about it and that'll help a bunch of other people hitting that problem."


Kris Jenkins (17:28):

Yeah.


Robin Moffatt (17:29):

And it's the same thing with performance and everything else. It's not like if you don't know it, then you're daft. But I do personally feel like if you're going to work on a technology seriously, you do need to understand under the covers, the next level down. Like, "Okay, so what is the end-to-end flow here?" Or, "When it does this, which bit happens next?" And that's a beauty open source. You can actually go and look. And if you can't figure it out, there's great community out there who will help you figure that through.


Kris Jenkins (17:54):

Yeah, sometimes I think you have to dig until you find the fix. But to future proof yourself, you want to keep digging until you really understand why the fix fixed it.


Robin Moffatt (18:06):

Yeah.


Kris Jenkins (18:06):

And sometimes you can't find the fix until you understand it. But we've all been guilty of just setting it to the magic number that seems to work.


Robin Moffatt (18:13):

Definitely.


Kris Jenkins (18:14):

Right? Yeah.


Robin Moffatt (18:14):

Oh, yeah.


Kris Jenkins (18:16):

Yeah. But another trend, there's the technical trend of getting to stage two. But I've definitely noticed the social trend of streaming technology maturing, which is you start to get companies now, Data Mesh is the buzz term, but thinking about treating their data as something that you've got to talk to other departments with. Instead of talking to other departments with APIs, we're talking to each other with data now. And that's definitely becoming a bigger trend. I don't know, I'm going to put Danica on the spot, there.


Danica Fine (18:53):

Oh, man.


Kris Jenkins (18:54):

Yeah, because you've been speaking to customers a lot, you've been out there, you've been dealing with KIPs. Do you think this is going to be part of the journey going forward?


Danica Fine (19:06):

Absolutely. Okay. So most of the conversations that come up around that, especially with customers and just people out in the field, it's always folks that haven't, they're like, "Oh, this seems intriguing, Data Mesh. It's going to solve all of our problems. This is the next best thing." That's what happens with buzzwords. And then asking questions about how to actually implement it. And of course, we have a Data Mesh demo, and, "Here, this is exactly how you might implement it and use it." But as we get deeper into the conversations and like, "What are you actually getting out of Data Mesh?" Treating your data as a product and changing how you communicate with other areas of your company. They all seem to, and this is my personal belief, is that it's a set of best practices, right? You don't have to-


Kris Jenkins (19:53):

Hm.


Danica Fine (19:53):

... implement into any certain way. It's just understanding how to operate in this data-driven space in a more responsible way. And when you get to that point, and that's what it is, it's like, "Okay, is the fix Data Mesh? Or can we dig a little deeper and understand the best practices that are actually supporting Data Mesh and just start to work with those instead of just focusing on this buzzword, that fix." Right?


Kris Jenkins (20:18):

Yeah.


Danica Fine (20:18):

So that actually comes pretty neatly full circle in that conversation, because that's what I've been pushing for. And it's really cool that in having conversations with people, they come to that on their own in the middle of the conversation. Like, "Oh, wait, okay. We can just focus on those couple things and if we do that we're going to get to that point." And I like that people are excited about that because those best practices are good, and they are best. Right?


Kris Jenkins (20:45):

Mm-hmm.


Danica Fine (20:45):

It's always nice when the best practices actually are best, and they're not too complicated, and they make sense, right?


Kris Jenkins (20:53):

Yeah.


Danica Fine (20:53):

They're intuitive. And so I think that is going to be part of the journey moving forward. Because if you want to use your data efficiently, I think that's a really big part of it.


Kris Jenkins (21:05):

Yeah. You say best practices around solutions. I almost think that Data Mesh is partly a solution, but there's no codified way of doing. It's partly way of codifying what shape your problems look like. You have a problem that if you can't deliver data to people, things are going to be more difficult for them to cooperate without coordinating.


Danica Fine (21:25):

Yes.


Kris Jenkins (21:25):

You have a problem that if you are publishing data to them, then you need some kind of SLA for that. You need some kind of guarantees about quality. And it's almost, is Data Mesh a solution or a description of the trouble you've got ahead unless you solve it?


Danica Fine (21:42):

Maybe it's just the streetlight. Is Data Mesh the streetlight?


Kris Jenkins (21:48):

There's your title for the podcast.


Robin Moffatt (21:48):

Love it.


Danica Fine (21:53):

My new talk with Adam Bellemare.


Kris Jenkins (21:55):

Oh, yeah.


Danica Fine (21:56):

Is Data Mesh the Streetlight?


Kris Jenkins (21:59):

He's got a book out. Anyone get a signed copy of that book at Current?


Danica Fine (22:02):

No.


Robin Moffatt (22:02):

Missed that one last.


Danica Fine (22:03):

The line was-


Robin Moffatt (22:04):

There was a big line, too, for it, though, wasn't there?


Danica Fine (22:05):

Yeah.


Kris Jenkins (22:06):

There was. Yeah, it's hot topic.


Danica Fine (22:08):

Man. I'm just going to set up my own line at the next conference, set up my own table. I don't know what book I'm going to sign, but it's not mine. I don't have one. But-


Kris Jenkins (22:14):

Signing people's house plants.


Robin Moffatt (22:15):

Yeah.


Danica Fine (22:19):

"Here. Here's a Leaf from my plant."


Kris Jenkins (22:20):

That's another one of my favorite things that I've seen a bit of, again referencing Simon Aubrey but also you and other people. IoT related fun stuff. I know you have a future IoT related project. Tell me about that and how people have reacted to your plant project over the year.


Danica Fine (22:41):

Oh, man. I didn't expect people to be that excited about it, first year as a DA, just trying to find an interesting project, one that I find interesting, because then that's going to be a driving factor for me. I want to build this, selfishly and it's going to be useful. And I think it checks a lot of boxes as far as what should people understand when they're trying to build an end-to-end pipeline. So it's been really, really well received. I think people are excited that it's a fun way to learn the end-to-end pipeline. And then inevitably, at every conference or speaking engagement, have at least one person come up and say, "Okay, how do I build this in my house? Because I need it." Or like, "My girlfriend has too many plants, how do I build this for her and stuff?"


Kris Jenkins (23:28):

Yeah.


Danica Fine (23:28):

Like, "All right, let's do it." It's so much fun to me that people are that excited about it, that they want to build this, and even if they haven't used Kafka before, they are inspired to do that or something similar. And yet I think I've got a whole backlog of random IoT projects that I want to build out now because, because it was easy. It was pretty simple to integrate it and there's so many random things that you can put together.


Kris Jenkins (23:57):

Yeah, it's fun to do stuff you can actually put your hands on when we're dealing with a lot of abstract ideas. And it's fun to scratch your own itch in a way. This is completely off topic, but one thing I found this year that's really interesting me is have you heard of a program called PICO-8?


Danica Fine (24:17):

No.


Robin Moffatt (24:18):

No.


Kris Jenkins (24:20):

You remember the age of eight bit consoles, like Nintendo Entertainment System and the-


Robin Moffatt (24:25):

Yeah.


Kris Jenkins (24:25):

... Game Boy. PICO-8 is a development environment for writing those, for a fantasy console that never really existed. But it's this wonderful, tiny idea that you can draw sprites, and pixel maps, and compose music. If you have kids-


Robin Moffatt (24:41):

Yeah, wow.


Kris Jenkins (24:42):

... start them on learn. It's based on lure and you can start them coding with a complete build your own games environment. And the one thing it lacks, which galls me, there are no network primitives, so I can't connect it to Kafka because I really want to do a-


Robin Moffatt (24:56):

That would make an amazing talk, right?


Kris Jenkins (24:57):

Wouldn't it? Like fantasy console game browsing-


Robin Moffatt (25:00):

Oh, cool.


Kris Jenkins (25:00):

... your topics or something. That'd be awesome.


Danica Fine (25:00):

Oh, man.


Kris Jenkins (25:00):

But-


Danica Fine (25:00):

Yeah, that would be-


Robin Moffatt (25:00):

I- [inaudible 00:25:05]


Danica Fine (25:06):

... super cool connect to your choose your adventure game. That would've been super cool.


Kris Jenkins (25:09):

Yeah.


Danica Fine (25:09):

Oh, man.


Kris Jenkins (25:11):

One thing I meant to ask you, Robin, for ages, you had a matrix display on your back wall, like IoT thing.


Danica Fine (25:18):

It's a-


Kris Jenkins (25:18):

Did you ever connect that to Kafka?


Robin Moffatt (25:20):

No, it's called, it's a Divoom, D-I-V, double O-M. And it didn't actually have an interface that I could find that I could use.


Kris Jenkins (25:31):

Oh.


Robin Moffatt (25:31):

I don't know if there's an unofficial API somewhere. But you had the phone app, and you could draw a sprite or find one, and then send it to it.


Kris Jenkins (25:37):

Oh, okay.


Robin Moffatt (25:37):

So I guess you'd have to-


Kris Jenkins (25:38):

There's no SBK?


Robin Moffatt (25:38):

... sniff something. Not that I could find, no. Maybe there is and that would be fun. But yeah, definitely, that would be fun.


Kris Jenkins (25:41):

Because I have an Adafruit Matrix portal, which I've been trying to do something with, but it runs a version of Python. And this is getting onto something, a trend I'd like to see in the coming year. It runs Python. I can't get a Kafka client onto it because the Kafka Python library depends on librdkafka to C bindings and I can't get the C library on there because it ... You know.


Robin Moffatt (26:13):

Yeah.


Kris Jenkins (26:14):

Too much pain. And it's like, I would like to start seeing next year, this is my hope for the future, that we start seeing more Kafka clients and more language libraries that aren't necessarily dependent on the underlying C library that we publish.


Robin Moffatt (26:31):

Yeah.


Danica Fine (26:31):

Yeah.


Kris Jenkins (26:31):

Like-


Robin Moffatt (26:31):

Did you try the REST API?


Danica Fine (26:31):

Yeah.


Kris Jenkins (26:35):

That's my backup plan. So far I've been struggling with SSL.


Robin Moffatt (26:39):

Ah.


Kris Jenkins (26:40):

I keep chipping away.


Robin Moffatt (26:41):

That's another fun thing.


Kris Jenkins (26:42):

Yeah. SSL on IoT devices, that can be a lot of fun.


Robin Moffatt (26:46):

Yeah.


Kris Jenkins (26:47):

But-


Robin Moffatt (26:47):

That's one-


Danica Fine (26:48):

That was one of the things that took me a week to set up on my [inaudible 00:26:52]. I remember just like, "All right, Google, tell me the fix." Yeah, that was just one of those things again. So good stuff.


Kris Jenkins (27:00):

But that leads me to where I want to go. Just finally, that's my hope for next year. Broader, more diverse client libraries. They're getting away from this idea that Kafka is largely a Java shop, which is nonsense, but there is that-


Danica Fine (27:15):

Definitely.


Kris Jenkins (27:16):

... perception. What in your fantasy Kafka 2023 streaming world, do you hope to see, Robin?


Robin Moffatt (27:30):

Oh, like you say, more, better support for other languages. It's always been a problem. Still, it's getting better, but there's a huge amount more to do in it. Probably better connectivity, just general open source, like [inaudible 00:27:46] fantastic example, more stuff like that. More connectivity in that area. But just general, it's not a Java thing. It is a Java thing, but once you've got managed services, who cares what the broker's running in, right?


Kris Jenkins (28:02):

Yeah.


Danica Fine (28:02):

Mm-hmm.


Robin Moffatt (28:04):

Let's make sure all the [inaudible 00:28:04] clients can use that as whatever people want to code in.


Kris Jenkins (28:06):

Them. Yeah. Postgres is-


Robin Moffatt (28:06):

Even like-


Kris Jenkins (28:06):

... an NFC thing, right?


Robin Moffatt (28:08):

... Haskell law or something like that.


Kris Jenkins (28:09):

Haskell. Yes. I'd like to see Jay Kreps announce proper support for Haskell at Current '23. Jay, if you're listening, I'm prepared to help out with that effort.


Danica Fine (28:23):

Dream big.


Kris Jenkins (28:25):

Dream big. Danica, what's your hope for '23?


Danica Fine (28:30):

Oh, man. Can you repeat the question? What was the wider, it was for streaming, or for Kafka, for everything?


Kris Jenkins (28:35):

For-


Danica Fine (28:35):

What's my hopes-


Kris Jenkins (28:36):

For this world we-


Danica Fine (28:36):

... and dreams?


Kris Jenkins (28:37):

... are in, what innovation do you hope that we'll see? What's the main thing-


Danica Fine (28:41):

Nothing.


Kris Jenkins (28:42):

... you're crying out for, that you think might happen next year?


Danica Fine (28:51):

Oh, man. No. Oh, man.


Kris Jenkins (28:53):

It's the end of-


Danica Fine (28:53):

I don't-


Kris Jenkins (28:54):

... the year.


Danica Fine (28:54):

... think it needs to be a wider innovation. No. You two covered that. You two covered that. What I want for my end of year, my winter gift that just keeps on giving for the next year or so, and I'm trying to deal with the blog posts that I write from now on, is just teach a man to fish. No more bandaid fixes. I want as many people as possible to understand the nuances, that the levers that they're pulling for Kafka, or whatever project they're building, I want them to understand. Even if it's only one person that will make me happy.


Kris Jenkins (29:28):

Right. Moving away from-


Danica Fine (29:29):

That's what I want.


Kris Jenkins (29:30):

... enough rope to hang yourself into the enlightenment era of Kafka.


Danica Fine (29:34):

Yes. That's what we're hitting. Yes. The enlightenment, the streaming enlightenment. That's it.


Kris Jenkins (29:41):

That could be the catchy subtitle for Kafka Summit London, which will be the next time we all meet together, I think.


Danica Fine (29:50):

Yes.


Robin Moffatt (29:51):

Yeah.


Kris Jenkins (29:51):

Cool.


Danica Fine (29:51):

I'm so excited, looking forward.


Robin Moffatt (29:51):

It'll be here before we know it.


Kris Jenkins (29:53):

Yeah. Well, good luck with the program committee, Danica.


Danica Fine (29:58):

Thank you.


Kris Jenkins (29:58):

Robin, pass the torch successfully.


Robin Moffatt (30:01):

I shall.


Kris Jenkins (30:01):

And thanks very much. We'll speak to you again soon.


Robin Moffatt (30:04):

Cheers, Kris.


Kris Jenkins (30:05):

Danica, Robin, thank you very much. I'm looking forward to seeing what they both do in the year to come, and I'm quite looking forward to some new guests telling me what they're doing in 2023. So if you are one of them, drop us a line. I'd like to hear from you. As ever, Streaming Audio is brought to you by Confluent Developer, so if you're spending this festive period taking a break to work on a pet project or brushing up on your technical knowledge in the downtime, then developer.confluent.io is there for you as a resource to teach you everything we know about streaming technology. But until next year, it remains for me to thank our guests, Danica Fine and Robin Moffatt for joining us today, and you for listening to our technical explorations all this year. I've been your host, Kris Jenkins, and I will catch you in '23.