Machine Learning with Kafka Streams, Kafka Connect, and ksqlDB ft. Kai Waehner

In this episode, Kai Waehner (Senior Systems Engineer, Confluent) defines machine learning in depth, describes the architecture of his dream machine learning pipeline, shares about its relevance to Apache Kafka®, Kafka Connect, ksqlDB, and the related ecosystem, and discusses the importance of security and fraud detection. He also covers Kafka use cases, including an example of how Kafka Streams and TensorFlow provide predictive analytics for connected cars.

EPISODE LINKS

• How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka
• Learn about Apache Kafka
• Learn about Kafka Connect
• Learn about ksqlDB, the successor to KSQL
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_01 0:00

Machine learning grew up as a batch mode discipline. But is it compatible with the world of streaming? Kai Vaina, a systems engineer and technology evangelist here at Confluent, tells us about his experience integrating Kafka and various machine learning APIs and how the Twain end up meeting. It's all on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. I am, as ever, your host, Tim Berglund, and I'm joined in the virtual studio today by my colleague, Kai Vahner. Kai, welcome to Streaming Audio.

SPEAKER_00 0:43

Thanks, Tim. Great to be in the podcast.

SPEAKER_01 0:46

Yes, I'm excited to talk about what we're going to talk about today because I know it's something that is uh near and dear to your heart and always has been. But first, as always, I like uh listeners, this is the first time you've been on the podcast. I like listeners to know who you are. Uh what is it that you do? Um tell us.

SPEAKER_00 1:03

Sure. So my name is Kai Vena. I'm based in Germany and I'm with Confluent for almost three years. My main focus is talking to customers and seeing their Kafka use cases and what they do in production or what they want to do. And that's just part of my story because I also um go a lot to conferences and take a look at new technologies like machine learning or IoT and think about how to combine that with the problems of our customers. And so that's the two parts of my job. And the second the third then is to talk to product management and our engineering and discuss how we can improve our product and features regarding what I learned from the conferences and from our customers. So I'm at different customers and events every week, and that's really exciting to see all these things.

SPEAKER_01 1:47

You are indeed, yeah. So for your and and I'll just say for the benefit of everybody, yeah, Kai does have this customer-focused role, which is uh very much like unlike mine and my team, our interactions with with uh paying customers are they happen, but they're occasional, right? We're more focused on the broadly developer community. But but Kai does a lot of that work too. Uh he's uh uh based in Germany, and so you're most likely to see him in Germany or elsewhere in Europe, but at a lot of conferences in the US and in uh Europe and at a lot of meetups, and he just does a lot of that uh you know, really advocacy and evangelism work uh that I appreciate. So thank you for doing that, Kai. Yeah, sure. And the community benefits from that. Happy to do that. Now, uh the topic that I alluded to before that uh has always been a thing that uh is an interest of yours is machine learning. And you've spent a lot of time um I think trying to bridge the gap between machine learning and Kafka uh which is good because I think I think it's kind of grown up as a batch discipline. Would you say that's fair?

SPEAKER_00 2:56

Yes, absolutely. And I think the main problem really is that people either take a look at machine learning and its technologies or they work on Kafka, but connecting the dots is really hard for many people. Um while actually it's really easy, and I think that's what we want to discuss today, though that people can connect these dots and combine these technologies and use cases.

SPEAKER_01 3:16

Nice. And I want to do that, uh, because you've got a lot to say on the topic, but um I want to define machine learning. Like I think we all kind of know what it means, but this is one of those terms that if we did that computer science family feud game that I talk about sometimes where you just like surveyed a bunch of people and had them give short definitions and then you know tried to guess what those were. I think I think the survey would be a little bit all over the map. I I don't know that we've all got, you know, we we have a sense of it, but if you don't work in it directly, you might not have a tight definition. So define machine learning for us, if you would.

SPEAKER_00 3:51

Yes, I will do, because that's really very important because everybody will have a different definition. So when you talk about machine learning with someone, first define what you mean. And for me, always I explain machine learning, it allows computers to find hidden insights without being explicitly programmed where to look. So that means you don't write if-else loops like you did before in your Java or Python code, but you really send a lot of historical data from a database or something else into an algorithm, and that finds insights for you. And that in the end, from a technical perspective, really is what machine learning is, and that can use many algorithms under the hood. This is not new things, this is decision trees or clustering, or now the cutting edge deep learning is neural networks where you use something like an autoencoder or a convolutional neural network. But the point really from a technical perspective is you use historical data and send that into an algorithm which finds insights from the data. And with that training, then this is what you train a model with, and then you get a model which you can use for your predictions of future data. And so that's in the end, from a technical perspective, what machine learning is. And with that, if you think about use cases, um, there is two different kinds of use cases. First of all, and that's typically what you see in the news, is the really fancy stuff. So, for example, with machine learning or deep learning, you can do things like speech translation or image recognition. And this creates a lot of new use cases which could not have been built without that. But on the other side, um, and that's really what in the in the field at our customers, maybe 70-80% of today's application of machine learning is is really act on critical business moments and improve existing use cases like fraud detection or cross-selling in the retail store or inventory management, predictive maintenance. That's all use cases which exist for a long time, because for example, fraud detection banks do that for 20 years already, right? Um, but now they can leverage machine learning to build better models to find better insights to do, for example, the fraud detection. So that's in one minute or so what I would define machine learning on a high level.

SPEAKER_01 6:05

I like it. I like it. And the like, for example, your fraud detection use case. I mean, nobody needs machine learning to do fraud detection. You can just decide uh you know, you can have you got any sort of algorithm you want that is constant and untrained, like, well, you know, this this uh purchase was too close in time to the last purchase, therefore it's fraudulent or something. You know, that's a stupid algorithm, but that's that's not a machine learning algorithm, that's just fraud detection. Exactly. But machine learning, I think to summarize what you said, is basically you want to think of it as there's there is a function and that function has parameters. And we don't know what kind of function it is, we don't know what the parameters are, that all depends on whatever the the algorithm is on the inside, and you you named a few. But there's some parameterized function uh that we you know, we give it an input and we get a predictive output. We set the parameters though by looking at other examples of past things whose outcomes we know. So like we know what a fraudulent what past fraudulent transactions are and past good transactions are, and so we can set the parameters of that function by by training it with that data. Is that a good restatement?

SPEAKER_00 7:14

That's a good restatement, and actually this also shows how easy now you can integrate this machine learning model into an existing application, right? So where in the past you had other um rules, for example, which you built for fraud detection, and now you can replace or combine these existing rules with the new analytic model, and then do things like A B testing or combining both, and therefore that's what's why so important that machine learning is not just for the new fancy stuff like autonomous cars, but really just to improve your existing business processes.

SPEAKER_01 7:46

Absolutely. So, what does this have to do with Kafka?

SPEAKER_00 7:50

Yeah, that that's that's the other part, and that's really not to connect the dots. And that's um very first have to explain. Um, most people, when they think about machine learning or Google for it, then for machine learning you find a lot of different frameworks and solutions. Like the one of the most famous ones today is TensorFlow, but there is many others like H2O or uh PyTorch or many proprietary solutions for machine learning. And the point is when you try this out, typically there are great tutorials, right? And then you see um tutorials with 20 to 50 lines of Python code where you can do train a model for image recognition, and that really works and works very well. But this is just a small part of the machine learning infrastructure, and therefore I always point people um to a great paper by Google, which we will also add to the to the podcast um links, um, which is called Hidden Technical Depth and Machine Learning Systems. And this paper explains, and that's really what we see in the real world, that all our customers, that writing them machine learning code, so this kind of 20 to 50 lines of Python code to train a model, that's a small fraction of the overall infrastructure for machine learning. You also have to think about things like data collection, um, serving the infrastructure, monitoring the whole process, and many other steps. And all of that typically at scale and in real time in many use cases. And if you think about, for example, a connected car infrastructure or millions of mobile apps which are doing PayPal payments, that's nothing but a 20-line Python script can do. So this is really just a small fraction. And where Kafka comes into play is really to build the whole infrastructure, both for training the model and for model deployment. And that's why we how we connect the dots. Um, you don't have to use Kafka for everything, but we can discuss in more detail now where we see customers and other companies combining both and why that makes sense.

SPEAKER_01 9:46

Absolutely, yeah, because there's a you you were really describing the data engineering that uh kind of goes around the machine learning, and there's this little machine learning algorithm at the heart, which is like technically and theoretically sophisticated and does impressive things. And we're kind of at a point in the development of this technology where everyone's mind is being blown by it. You know, there are all kinds of really impressive predictions that machine learning algorithms are pulling off, and things like speech recognition is working better, and that's all that's all driven by machine learning at the heart of it. So we we're all kind of getting wowed by that algorithm, and we ought to get wowed by that algorithm. But like you say, maybe that's just 20 lines of Python code or some library or something like that. Um, and that's not really what we're here to talk about. There that thing is there, you know, that little function that you trained with data is there, but you need all this infrastructure around it. You need to kind of engineer a data pipeline around it. And that's that's uh that's really what where Kafka and the platform more broadly comes in.

SPEAKER_00 10:52

Yes, and and it's actually even more complex because there's really this kind of impotence mismatch between the data scientists and the production engineers. So we know that the same thing from a database, right? Where you have the relational database and you have the objects and Java, for example, and the same is true for machine learning because um what you said is is actually also partly true. So, for example, the data scientist builds a model in Python, and that Python model works well and does great predictions. However, um, and that's really what we see at our customers, then this data scientist, which is in a completely different team than the production engineers, he hands it over, and the production engineers say, Well, it's great that this model does get good predictions, but we cannot deploy this because we deploy Java, because that's highly available in our infrastructure, and the Python code does not work here, or it does not scale well, it is not um efficient enough because Python is a dynamic language, so it's much slower than Java, for example. And therefore, we have seen customers, um, and that's really in all big companies where they are separated teams, say the data scientists write Python code for a working model, but then the developers have to re-engineer the same model and rewrite it in Java code. And that's really what creates a lot of complexity for the overall process, and therefore it's really not just writing the right model, but also writing it in the in the right technology, or at least being able to export it into another technology which is used for production. And so this is really a very big challenge, not just writing the code, but also delivering it in the right uh technology.

SPEAKER_01 12:21

That sounds actually terrible uh from an operational perspective. Like um, but this is a thing that you see commonly is that there's a data science team that is experimenting, let's just say in Python, and I mean Python is the fastest growing language on the planet right now, no nothing but love for Python, but you know, they're doing their work in Python, and the the people who will run code in production know how to run Java code and aren't willing to deploy Python. So you actually see the the people you know, the the quote unquote scientists building one thing, and then there's another team that productionalizes it by rewriting it. Yeah.

SPEAKER_00 13:02

That's one option, and that's really what we see often in the real world, or even worse, um, the models are not deployed at all because they do not get it into um a scalable infrastructure. And so there are different workarounds for that which we can can cover regarding how you deploy your model. So therefore, it's really important to understand. Machine learning is always two things. It's training a model, and and that's fine if you do it in Python to write the code, but the trained model has to be in a in a kind of code or or data format which can be deployed a production system, which often is not Python. And that's therefore important and also good to separate the model training from the model deployment part.

SPEAKER_01 13:40

Yeah, yeah, that's a super good point. Um and I guess that's but but culturally you you do see that where there's maybe something like a data lake and uh you know data scientists in picture them in white lab coats with clipboards, you know, hacking on Python, building these amazing predictive algorithms on this big pile of data. And that's not that's not that's not the goal. The goal is to actually productionalize the model and have it do things in real time in the business.

SPEAKER_00 14:10

Exactly. So our key recommendation is from our lessons learned with customers, um, talk to each other from the beginning. So especially in the big companies where it's different departments, from the beginning that data scientists have to know how does our model have to scale? Um, how does it have to perform? And that's then where Kafka again comes into play because if you think about uh hundreds of millions of payment users in your mobile app, or about the connected car infrastructure with hundreds of thousands of cars, um then this has to scale. And if the data scientist knows that from the beginning, he can uh do the process from the beginning in another way. And that's then also where at our customers Kafka comes into play also for the data engineering parts, because with Kafka it scales well, and that's how you have to think about that from the beginning to be successful in your project long term.

SPEAKER_01 14:55

Walk us through the architecture of your dream um machine learning pipeline. Uh tell us tell us how it ought to be done.

SPEAKER_00 15:05

Yeah, sure. Um so in the end, as I said, um this is typically two parts. So um the model training part is where you ingest the historical data or the events into the model training framework, like TensorFlow. And therefore you have to get it from all the different systems. This might be mobile apps or cars or sensors or whatever. So you need to uh create this pipeline at scale for millions of events or terabytes of data. And that's of course where Kafka comes into play already. So typically our customers use things like Kafka Connect to integrate to different source systems to be scalable and integrate the data in real time. And then um you can ingest that directly into a framework like TensorFlow, or if needed, or if you want to do that, you can also do a lot of the data pre-processing with Kafka technologies. So I think we have talked about that in other podcasts, like technologies like Kafka streams or KSQL, um, where you can do similar things like in a Python machine learning framework. So um you can do things like filtering or enrichment or aggregations in both in Kafka or in the machine learning Python code. And if you want to do that at scale, then ideally a lot of that is done in Kafka already, so that only the last part of the model training, which is really um using the algorithms for model training, is used in the Python code from TensorFlow or something like that. So this is on a high level the model training part. Get the data, for example, with Kafka Connect, do the data processing as much as possible, also with Kafka native technologies, and then ingest the data into the machine learning framework to train the model. And that's definitely where the ML framework then and its code is used, like for example, TensorFlow. And then this is the yeah. Oh no, go ahead. And this is the model training part. And then for the model deployment, and that's really important, it can be completely separated from the training part. And in model deployment, this can, for example, be also done in another Kafka client, like a Java client or.NET client, or maybe even a stream processing application like Kafka streams or KSQL. Because what many people don't know that the model itself, it's just a binary. Depending on the technology, it can be an XML file, or in the case of TensorFlow, it's protobuf. And therefore, this really separates the training from the model deployment. And even if you use, for example, a Python pipeline for model training because it works well for you for whatever reason, you can still deploy that model to a Kafka streams application and scale this up and down as you need. So while you might have, for example, a data lake with Spark or other technologies to train a model, you still can take this binary model and deploy it into a scalable Kafka streams application and run that at scale and scale it up more and more.

SPEAKER_01 18:01

That sounds great. I uh that's funny. You you just said this this is a little bit of a rabbit trail, but you just said the model for TensorFlow is delivered in the form of protobuf. Yeah, it's interesting. So uh and that then uh in fact, if you could you said TensorFlow just as an example of one of the you know, so this this is the algorithm that goes in the box that we said we're not really talking about today, but but of course we're gonna talk a little bit about it. Um to help get some understanding of the landscape here, tell us what TensorFlow is and tell us what uh the other options are for going inside that that you know that that function where you're doing the actual machine learning predicting. Uh TensorFlow is one, tell us what that is and tell us what it's not.

SPEAKER_00 18:46

Yes, exactly. So TensorFlow is a specific machine learning framework. Actually, it mostly focuses on deep learning like neural networks. And that is really used especially to remodel training. Um so this means you ingest all the data and then you train a model with the algorithm to get these insights, find hidden insights in existing data. And then you get this model with TensorFlow and can deploy the model somewhere else. And that that is pretty similar for all the other frameworks in the market, like um besides TensorFlow, the second big one is PyTorch. And we really see that um on the market, these are the two ones which are used most in the meantime: TensorFlow and PyTorch. It's similar to um Docker and Kubernetes and so on, right? And in the past you had many frameworks, and then now you everybody's using some kind of Kubernetes.

SPEAKER_01 19:35

There were bitter wars, and now there's just kind of like one or two. Yeah.

SPEAKER_00 19:38

And the same is true for um TensorFlow and PyTorch, because now everybody's using them. Even for example, uh Amazon had its own similar framework, um, partly as they reported it's better than TensorFlow, um, but now even they support and use TensorFlow because it won the battle, let's say. And therefore, um, this is what TensorFlow or the others do. They train models. They also have additional features, like for example, doing pre-processing. And that's then the part where you have to evaluate. Um, should I use the pre-processing features of TensorFlow, or should I do it in Kafka? And that's in the end the trade-offs of doing it in a Kafka native but uh reliable and scalable way, or should I use the features of TensorFlow, which probably provide better pre-processing functions? So that's where you simply have to evaluate what to do where. But the main part of TensorFlow or other frameworks really is to train the model, and then you get the binary. Like as I said, it's uh typically protobuf and TensorFlow. And then the second part, of course, is also that um TensorFlow or other frameworks provide an API um to deploy the model somewhere, and that's then where also different options come into play. So um for TensorFlow, um like every other machine learning framework um which has big success, um it's typically all in Python, and Python is the primary language. However, still it has, for example, a Java API. So with that, you can train and build the model with Python, and that's what the data scientist uh wants to do, and still you can load the model from a Java application with the Java API. And this is how how these frameworks now try to solve this impedance mismatch because you cannot rewrite and re-engineer the model. This is not uh practical in the real world. But with these kind of different APIs, you can make both happy the data scientist and the production engineer.

SPEAKER_01 21:29

That sounds wonderful. Uh so you you don't have to do that extremely painful redevelopment. You just take the artifact that the training process delivers and basically deploy it to production. And that being a Java API, now that is uh I I hope I could say trivial to integrate with Kafka streams or just a consumer or whatever, you know, whatever it is that you're doing to get messages out of a topic and into your model and and place the predictions in a in a Place where the application can see them. And we'll put up links if you're seeing if you're just coming into this episode new to the podcast and you don't have much history with Kafka, we'll put up links to Kafka Connect and Kafka Streams and KSQL. Those are some of the technologies that Kai has mentioned. At least I I I think you've mentioned Connect. Am I making that up? I was thinking of Kinect. Why don't you tell us what role Connect has in this?

SPEAKER_00 22:23

Yeah, sure. Kafka Connect is used to on both sides, on the source and on the Think side. So on the source side, you can use it to get all the data into Kafka and then to TensorFlow. So for example, if you think about connected cars, you can use the MQT connectors. Or if you go back to even an Oracle database with change data capture, typically in machine learning, you also integrate data from different systems because the more data you get, the better your models get. And therefore you want to integrate as much different data as possible. So if you think about the tech companies from Silicon Valley, they use all the data they get about you from different systems. And to integrate that first easily and also at scale in a reliable way with the same technologies, that's where Kafka Connect can help a lot. And the same is true, then of course, after you have it in Kafka in a reliable and scalable way, then you also can use the connectors to store it in a data store where the machine learning framework consumes it from. Because that's also very important to discuss. Traditionally, machine learning frameworks get the data out of a data lake or database that can be something like HDFS and Hadoop, or this can be something like Google Cloud Storage or AWS S3. So that's the common architecture which people use today. So if you ask a normal data scientist what he wants to do, he gets all the data at scale and stores it in an object store or so, and then he uses TensorFlow or another framework to train the model on that. Because model training is always almost always a batch style, right? So um this is this is fine, and it takes hours or even days to train a very good or complex model. So therefore, Kafka Connect is also used on a thing site often. Um that that's the normal case, let's say to get the data in.

SPEAKER_01 24:15

So it really looks like a regular data pipeline. Uh there's data out in the world that's unfortunately not in Kafka. Connect gets it into topics, um, you do processing on it. Um and then the results maybe you use Connect to get to somewhere else. Uh so it's just that that processing you do on it includes, in this case, a machine learning algorithm. So you're just plugging in um this very interesting extreme processor in the middle of an otherwise conventional data pipeline.

SPEAKER_00 24:44

This is the normal process, yes, but I I want to mention one more thing because that really I would say is cutting edge and also simplifies this machine learning infrastructure. Because the normal way, as I said, is you um store the data in a data lake and then you put the model training on top of it. What you can also do today is you can directly stream the data from Kafka into TensorFlow. So therefore, this is a great example, what we use all the time. But um TensorFlow has an additional plugin which is called TensorFlow IO. It's part of the TensorFlow project. And with that, you can use its plugin for Kafka. And with that, you can avoid the data store in the middle. So you don't have to store it into S3 or HDFS anymore. You can directly stream the data from the log of Kafka into TensorFlow. So in the end, TensorFlow has its own Kafka consumer, if you want to describe it in a simple way. And this completely um simplifies the architecture. So in the talks and demos I do in the next month, I call this streaming ML because you stream it directly into TensorFlow. And you can do that both for model training and for model inference. And that's really exciting. And I'm working also right now with one of the TensorFlow committers to build a cool example for that. We will share that in the links. And there really we can show that one, it's really a streaming architecture, and uh secondly, it's also real time and scales well because it's just Kafka under the hood. So in our example, we really connect to 100,000 cars, simulated cars, of course, which send data in real time to Kafka, and we ingest that data directly into TensorFlow without using another data store in the middle. And that's really exciting and cutting edge. So and this a lot a little bit also um is new instead of using the traditional pipeline.

SPEAKER_01 26:32

I guess it makes sense that that consumer would have gotten pulled up into TensorFlow APIs or TensorFlow framework code because probably everyone was writing the same boilerplate over and over again. And anytime a community sees that kind of thing happen, the community tends to just like like an Amoeba sort of uh you know, grab onto that little chunk of boilerplate that uh 2013 people wrote in slightly different and slightly differently buggy ways to pull that up into the framework. So that's that's good. Uh you said, okay, you said one uh thing that really got my interest, um, that uh this can be done for model inference, obviously, right? You've got your model and now you plug it in and it's you know TensorFlow for one has gotten good at interoperating with Kafka. So you deploy a model to the to the engine and it just runs it, uh, which is great. That's model inference that's running the model that you trained. Um but uh as we know when you look into machine learning, you know, you you train a model and there can be some drift in the data, and things can just you know, you you don't that's not a one and done proposition. You don't just train it and then go away and let your data run for 10 years. You have to retrain your model on some kind of schedule. And if that's a batch process, fine, it's a batch process, but you just said you can do that continuously. Talk about that.

SPEAKER_00 27:52

Yes, absolutely. I mean, in the real world, you always have to monitor your predictions and your model, and you have to improve it. Um, otherwise, machine learning does not make much sense in the real world. Um, the only difference is depending on the use case. Um, in some use cases, it's fine to retrain and redeploy a model every month. Um, in other cases, like for example, in ad advertising or in fraud detection, we have customers where this has to be really done um in an hour or even in minutes if possible. And that's definitely an important uh scenario. And so um in in in theory, it's not really hard because um in the end you consume new data and then you train it again with the same algorithm, and then you get new data in it and get new wrestles for the model. However, in the to realize that in the real world, it's really hard because this in the best case, of course, has to be automated. And um it has to be automated not just for the model training, right, where you train new versions of a model, but also then again for the model deployment, where you deploy the model, and then also for the monitoring of all of that infrastructure. And that's the other reason why so many people use Kafka here more and more, because with Kafka um it's all real-time and um it's built for cloud native infrastructures like Kubernetes or going really to a cloud provider. So you can easily do that by deploying a new version of the model and retrain it, and then also deploy it into an existing application. Um, there's different approaches how to do that. For example, we've seen customers which really train new models in their data lake infrastructure, and every hour or whatever the timeline is, they send this new model to a Kafka topic because the model is just a binary, right? You can attach it to your Kafka message and then you you send it to a Kafka topic, and then either the existing application um internally consumes from this topic and updates its model, or maybe it does things like A-B testing and tries out different models. Or maybe you start a new container in your Kubernetes infrastructure and run the old model in one container or pod and the new model in a new one. So there is many different alternatives how customers do that, um, but it's definitely part of a real-world machine learning infrastructure. And that's again why in the beginning I said um there are many hidden depths in machine learning infrastructures, and this is definitely what you have to solve to do machine learning in the real world, because um in machine learning it changes all the time. Um if you think about um how you build a traditional application, you have unit tests and you have integration tests and you have system monitoring. So that's pretty straightforward. And when you build something like a payment app, you test it, you do the integration tests, and then you deploy your application. However, when you do the same for machine learning, you have to do much more because the data which you used for training the model is different from the new data, and you have no idea what the people will send in. And therefore, in addition to unit and integration tests and system monitoring, you have to do data tests, data monitoring, prediction monitoring, model tests. And so you see that the complexity is much higher, and so um, it's much more challenges to build a really good architecture for machine learning than for traditional systems.

SPEAKER_01 31:13

Where um this question is somewhat external to the pipeline itself, but to do continuous retraining, you need labeled data to train, right?

SPEAKER_00 31:24

Uh not necessarily. I mean, there is in the end two different kinds of machine learning algorithms. There's supervised machine learning where you use labels. So, for example, um in fraud detection, um, experts or in ideally tools um use labeling to define what is fraud. In the example of payments, this is typically done by by humans. For example, in image recognition, um, we have many customers which build their own automated tools for doing that, and then only the humans have to do less work for um correcting things.

SPEAKER_01 31:56

On the other side, yeah, so you have like a thousand pictures of dogs, exactly, and you said these have dogs, and then you have a thousand pictures of cats, and you say these are cats. Exactly. So a human being does that and trains the algorithm with that data.

SPEAKER_00 32:09

Exactly. That that's the one part. And on the other side, also you have unsupervised machine learning, for example, an auto-encoder to detect animalies, and that's things where you don't need labels because it's unsupervised. And there it's of course much easier. Um, and therefore, also the improving the model and retraining is easier. However, unfortunately, um most of today's use cases um require labeled data and supervised learning, especially what we see in in the big companies today, like fraud detection or these kind of things. And therefore, yes, here you need labels. And if you build this type pipeline, that's definitely a bigger challenge because the question is um, how how do I um retrain the model with new labels? And that's typically where in parallel, people see fraud in existing transactions and then they label this data again. So this is definitely harder to do for supervised machine learning in a real-world example.

SPEAKER_01 33:02

Got it. And in that case, then there would be humans applying those making decisions, applying those labels and feeding them into the pipeline, and that would go into you know wherever the retraining data topic is, the model would consume that and and you know continuously adjust itself based on those new labels and those new inputs.

SPEAKER_00 33:24

Yes. But we see more and more automation here. Um this really depends a little bit on the use case and on the resources that people have. Um especially in use cases, for example, for images for Airbnb or so or these kinds of um uh uh internet sites, um, here it's it's not possible that humans do all of that because it's too many images. And therefore, these kind of internet shops, they have built automated tools for that. Um for fraud detection, it's much harder, but still people want to automate that. It's always a trade-off of um how correct has it to be and and all these things. Because in an Airbnb, if you see maybe a wrongly labeled picture by the tool which shows the bathroom instead of the um another room, um it's not good, but it's still okay-ish. Um if you have fraud detection or maybe think about healthcare, then here the data is much more critical, and therefore it's um it has to be more correct, let's say, and therefore here typically the humans do more work. Got it.

SPEAKER_01 34:20

Now I'm about to uh show my ignorance of machine learning generally, and that ignorance is frankly considerable. But when it comes to unsupervised learning, um you don't need labels, and I get that, but then what does retraining mean? Like where is the source of feedback into the model?

SPEAKER_00 34:38

The feedback is that in the end you um ingest more and more data, and um the the model can detect uh different changes. So a good example for unsupervised learning is an autoencoder. An auto-encoder gets all the data in and then it compresses the data to a smaller fraction, and then also it creates the full data set from the compressed data again. But that's not possible because we compressed it before, so we lost some data. And if there is no anomalies, then it can create the exit the incoming data again very well, so mostly correct. If there is anomalies, then it's not possible to do that in the right way. And therefore, you can detect anomalies by that. And if you think about use cases like predictive maintenance, um, where the sensors of cars or other engines create new data all the time, and you have no idea what exactly the data will be, like the measurement of the engine or different values of different sensors. The unsupervised machine learning can still see if there is anomalies, even if you have not seen these anomalies before. And this is a thing which you cannot really label because there's always new things coming in. But with unsupervised learning, you can still detect them. And um that's how how this kind of unsupervised machine learning works and and helps you with some kind of processes.

SPEAKER_01 35:58

But you're saying um most of the interesting loot use cases are and you know the mind-blowing things are supervised algorithms.

SPEAKER_00 36:05

Today, yes. Uh, there's a few use cases, like I said, for auto-encoders for anomaly detection or clustering, for example, where you don't need labels for that. If you want to get all the data into different clusters, like clusters of different kinds of customers, which have different kinds of money or so. But most of them are still supervised, um, yes. But here also we see a lot in research, um, of course, uh, people want to do more and more unsupervised because it's um much easier to train, because you don't have to get the label data. But um, outside of research, most of the use cases today use supervised machine learning.

SPEAKER_01 36:40

My guest today has been Kai Vayner. Kai, thanks for being a part of Streaming Audio.

SPEAKER_00 36:44

Sure, it was great to be here.

SPEAKER_01 36:46

And there you have it. Before I go, I want to tell you that we have a pretty cool new offer to help you get started with Confluent Cloud without you having to pay for anything. If you're a new user and you go through the regular sign-up process and start using Confluent Cloud, your first $50 of usage per month are free. This will last for the first three months after you sign up. So that's $50 per month of serverless Kafka for three months at no cost to you. So go to the sign-up link in the show notes. I don't want to read you the URL, and sign up now. I think the only thing I could really do more is write your code for you. And I think we both agree that's too much to ask. So check it out and hey, let us know how you like it. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can reach out to us on Twitter at Confluent Inc. or reach out to me at TL Burgland. That's T-L-B-E-R-G-L-U-N-D. Or you can hit us up in Community Slack. There's a sign-up link for that in the show notes as well. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks a lot for your support, and we'll see you next time.