Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

IoT Integration and Real-Time Data Correlation with Kafka Connect and Kafka Streams ft. Kai Waehner

• Confluent, original creators of Apache Kafka® • Season 1 • Episode 97

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 40:55

There are two primary industries within the Internet of Things (IoT): industrial IoT (IIoT) and consumer IoT (CIoT), both of which can benefit from the Apache Kafka® ecosystem, including Kafka Streams and Kafka Connect. Kai Waehner, who works in the advanced tech group at Confluent with customers, defining their needs, use cases, and architecture, shares example use cases where he’s seen IoT integration in action. He specifically focuses on Walmart and its real-time customer integration using the Walmart app. Kafka Streams helps fine-tune the Walmart app, optimizing the user experience, offering a seamless omni-channel experience, and contributing to business success. 

Other topics discussed in today’s episode include integration from various legacy and modern IoT data sources, latency sensitivity, machine learning for quality control and predictive maintenance, and when event streaming can be more useful than traditional databases or data lakes.

EPISODE LINKS

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

  •  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
  • ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
  • 👍 If you enjoyed this, please leave us a rating. 
  • 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
SPEAKER_00

My colleague Kai Vayner is back on the show to talk about the Internet of Things and specifically how Kafka and event streaming fit in, not just at the edge where you'd expect them, but also in the back end. Listen in on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. I am as ever your host, Tim Bergland, and I'm joined in the virtual studio, separated by a distance of, I don't know, around 5,000 miles by uh my friend and colleague Kai Vayner. Kai, welcome to Streaming Audio.

SPEAKER_01

Yeah, hi Tim. Great to be here again.

SPEAKER_00

Yes. I think since the last time you've been on, I think your role has changed. Um maybe it hasn't, but uh so Kai and I work together at Confluent. And Kai, you're a part of the advanced technology group now, which sounds awesome. But tell us what you do.

SPEAKER_01

It is. So actually, what I'm doing did not change that much, but it's a little bit more global now. So I work with many customers. So I face really around 100 or more customers per year. And to most of them, I meet them on site, normally if in Europe or US or Asia. And I talk to them about the different use cases and architectures and our roadmap. And in the end, I have two specific topics which I focus on a lot if it's not just about the general stuff. And one is machine learning and the other one is internet of things. And therefore, we have a lot of customers in both of these areas, and therefore I'm excited to talk about IoT and Kafka and event streaming today.

SPEAKER_00

Yes, and that I guess I should have said that is what we're talking about. We're talking about IoT today. So yeah, if you follow Kai at all, like on the Confluent blog or elsewhere or any of his conference talks, um I mean Kai, you you do have a fairly substantial internet footprint. So uh if you follow Kai, you know that machine learning and IoT are things that he is super interested in and has been for as long as I've known him. Uh in fact, Kai, you have been interested in Kafka and machine learning since before Confluent was interested in Kafka and machine learning. I believe you were one of the people who was kind of pushing us to care about that. Um and there's been uh on on the podcast recently we've had actually a lot of machine learning episodes. So it's been a theme, but not so much IoT. So um everybody it's IoT is one of those things that I know everybody listening knows what it is, but it's always useful to ask somebody who spends their time thinking about it, how do you define it? What what does IoT mean?

SPEAKER_01

Yes, and and and that's really very important to understand also the different use cases and architectures. I mean IoT, it means inside of things, but in the end, I would say on a high level, there is two different IoTs. The first one is industrial IoT, that's what most people think about. That's things like manufacturing, where you produce new products and then you have all the sensors and correlate and integrate it and maybe build analytic models. And uh, industrial IoT is then also related to automotive companies where they produce the cars and so on. But the second part is also consumer IoT. And this means things like when you have your mobile app and walk through the retail store, and then in the back end, all the data has to be correlated to send you a coupon to buy something or something like that. And therefore, this is the two high-level discussions about IoT with often very different technologies and architectures under the hood. That's what we can discuss today in more detail. But on a very high level, therefore, you have many different devices and things and connect them typically in real time to correlate all the data and get value out of that.

SPEAKER_00

Awesome. What I think of is um the the just the first application I think of when anyone says IoT is I automatically think of smart thermostats. And then I just feel lame because that's I mean, smart thermostats are great, but I mean, come on, there's got to be better examples. But actually, robots in factories are the very second thing that come to mind. So that's uh that's that's the mental picture I get is uh either the thermostat in my hallway and you know, 10 million of those all over the place, or uh an assembly line full of robots with all kinds of instruments and and measurements on them.

SPEAKER_01

Yes, exactly. And the interesting thing is when you think about these two use cases that this already explains two very different architectures, because if you think about IoT manufacturing, you actually have to deploy all the processing in the factory and do everything there, and it's very critical. On the other side, if you think about the smart home, then everybody has a little bit of stuff in their own home, but then you also have to send this data to a central cluster or cloud infrastructure where you can leverage and combine it together. So therefore the architectures and use cases are very different. And that's really also what you described. One is industrial IoT and one is more like consumer IoT.

SPEAKER_00

Yeah, that's that's funny that the two the first two things could that come to mind really cover those two broad categories. And there's so much cool stuff, uh, you know, like uh sensors. Uh there's there's often some kind of sensing technology, and uh people who deep dive into the physics of sensors can get into that. There's the embedded systems aspect of IoT. I started my career as a firmware developer, so I I uh my mind always goes to you know that that purpose-built small computer. Uh but I think what concerns us here is uh IoT and event streaming. These are these are sources of events and those events go somewhere. So talk to us about that.

SPEAKER_01

Yes, exactly. I mean, so first of all, devices produce data, the sensors create information, but then you also have actors where you can do something again when you consume data from a device. And this is still just messaging in the end, and there is a lot of products on the market for IoT messaging, like for example the MQTT standard, which is used more and more. But this is just for sending and consuming messages. The other thing is what we talk about a lot is event streaming. So you also want to use the data and do something with that. And that might mean you process the events or you even correlate it with other information. For example, you correlate information from different sensors. For example, in a car infrastructure, you want to correlate the information from all the different cars on a street together. And that in conjunction with the traffic light information and with the weather information, that is what creates the real value if you correlate the data. And that's true for most IoT use cases. It's not just about getting information into another system. That's what it was all the time with messaging, but now you want to correlate the data while the information is important, and that's what we do with event streaming in IoT with our technologies.

SPEAKER_00

Got it. And that makes a certain amount of sense. I mean, it's interesting. When you start with a consumer IoT application, like a smart thermostat, um there are some aggregations that you can compute in a fairly straightforward way to know like average temperature in a postal code or in a city or something like that, or uh, you know, daily high and daily low interior temperatures. That's nice, you know, but uh and I could imagine that being on a dashboard somewhere, but that's not the most valuable insight you get. You're saying those valuable insights come when you're connecting the events that you're capturing with any other data source.

SPEAKER_01

And and I want to give you one example here. I think we have so many great reference um use cases for IoT scenarios. So um, if you talk about consumer IoT, we had a great example from Walmart. They were speaking at a Kafka summit about real-time customer experience. And you can imagine there is plenty of these end users. They are people with their mobile phone and their mobile app from Walmart and they are walking through the retail store. And so, one use case here is that Walmart is able to take data from your past buying patterns, their internal stock information, and also combines that with your mobile phone local location data. So while you're walking through the store in real time, and it correlates it with some other back-end systems, like the weather information or your social media information. And all of that can be correlated so that Walmart makes the right recommendation to you. And that depends on many other internal things, like what's the inventory, what Walmart has to sell in their stock? Where have do they have too many? Or what do you want to sell for a less price at this point of time? And that's all these things which are correlated. And then while you're walking through the store, I mean, we all know in the US, Walmart is a big store. So you need to get the right information in the right context. And that's what you get then as a push message in your smartphone app. And with that, um, Walmart is able to, on the one side, increase the customer experience and make them more happy. But on the other side, of course, it's also a business case because they make more money and um have lower costs for inventory and so on.

SPEAKER_00

So without and I I want to get as specific as possible, and I know in your role, you work with actual customers and people trying to build things, and um the the names that you talk about are all people who don't mind being talked about. Um and I want to drill into this, but obviously I know there's a certain amount of detail that you can't give about the way people actually build things, but you went through a few data sources there. There was my location. So I'm using a retailer's mobile app, and that uh that device has location data, and in a big box store like say Walmart, and again in the US, everybody is familiar with how big a Walmart is, uh you could get uh you know cell phone location data is precise enough to say what part of the store I'm in, maybe even what aisle I'm in. I don't know how good that would be, but uh you know, inside the building. But you could certainly tell am I on produce, am I on automotive, am I an outdoor, am I on clothing, uh, am I near uh the pharmacy or whatever? You know, that's those big chunks of the store. So the thing in this case, we're talking about IoT, the thing is my mobile phone, and I'm using that app and it's sending location data telling me what I'm doing. Walk us through again, because you kind of listed off some other data sources, but I want to know like what's the actual computation that would get done on the back end that would result in something useful to the business.

SPEAKER_01

Yes, so in this case, um the scenario is that you are walking to the Walmart, and in this case, the app knows that you are walking into the store, and then the back end can already start doing correlations. So it can take the data from your past uh buying patterns, what you did in the past. So that it or it takes a look at your loyalty platform, for example. And with that, it it might know what you're interested in. For example, also you have searched on the Walmart website before about a specific product, and so it can um start thinking about what you might be interested in, even if you're not in the store yet. And then it can think about the inventory Walmart has and other back-end systems, and this is then the computation where it's calculating the things or even using analytic models and machine learning to predict what I might want to buy. Um, and then it makes you a push recommendation, maybe for example, to get 10% off if you buy it today, because then you don't have the risk to go to another competitor and buy it there. And um this is then back as a push message to you and you can make a decision. Or another example is then later. Um, when you actually have bought something and while you're walking out of the store again, then again, calculations are happening in the back end. And here it's also so critical that this is happening in real time, which means at least in a few seconds or so, because then it calculates that, well, I have bought a video game console with two games because maybe it's the birthday of my son or something like that. And so um, then you can say, well, um, hey dear customer, if you now buy a third game within the next 60 minutes in our store, then you get 30% off. And this is these kind of correlations. You need to connect a lot to different back-end systems. This can be either a remote store in the cloud or something cached in at the edge in the store. That's what we can discuss later for the architecture. But but this is the process which is happening, and the important thing is it's different data sources, and the data is correlated in real time to make the best recommendation for the customer, to make the customer happy and to sell more to them.

SPEAKER_00

And I suppose um with correlations, like you're talking about, we could bring in other data sources like it's hot out today, and I'm near the refrigerated foods. Uh hey, wouldn't you like some hot dogs and popsicles so you can have a cookout, or you know, it's cold and you're near the beverages, so you know, uh, here's 10% off on hot chocolate, or you go over to the outdoor stuff, and you know, here's 10% off on fire logs or something like that. So that that kind of bringing in and I get that would that that also all sounds like the sort of thing where you bring in an analytic model to make predictions about about the way those purchases work.

SPEAKER_01

And and the big question here is always where do you get this data from? Um let's take two examples. So the weather information, it um if you have thousands of people in walking through the store every day, it doesn't make to ask for the weather every second for every um customer. So you ask for it with a web service, maybe um once every 10 minutes or so, and then you store this information somewhere locally, because it doesn't make sense to have that many communication to the outside world for that. On the other side, um, if there's this more short-term information, like where exactly is the customer walking, or is the customer talking to a salesperson which is using its own computer to give answers? Um, this is the data which you correlate um in real time, but it's not necessarily always a remote call. Um, so some data is often cached, like um it could be either just on in a client application in Kafka streams, for example, um, so that you can correlate it quickly without doing a lot of um request response or even remote procedure calls to external sites. Um that's then again for latency and for cost and many other um just questions you have to answer.

SPEAKER_00

In consumer IOT, and I'm I'm calling this consumer IoT because I'm holding my phone and I'm a consumer and I'm buying things at Walmart. Um latency, do you run into very latency-sensitive uh applications and and I'll say challenges or our customers saying, like, wow, latency really matters here. How do we crush it?

SPEAKER_01

Yeah, it depends on on what you really then mean with real time, right? That's what we always discuss with our customers. I mean, of course, it's it's not really important in most of these use cases if the um information is correlated within a milliseconds, but often it should be in a few seconds at least. And that's really so many examples because the the basic rule of thumb is if the customer is already out of the store again, um, then he will probably not come back because then he goes to another store, maybe, which might be the competitor and not the next Walmart. So um it's definitely um it cannot be a minute or so or two minutes, it should be within a few seconds. That's uh for most um of these kind of use cases.

SPEAKER_00

Yeah, and in this use case, that totally works because I'm walking, right? It's at it's at human walking speed. Uh now I'm approaching frozen foods, you know. I kind of have 10 or 15 seconds before I'm past that. That that that whole thing um doesn't doesn't you don't need 100 millisecond latency on that computation.

SPEAKER_01

Yes, exactly. And but but also the other question is then do you store the data in the retail store or maybe somewhere in a central cluster in the cloud? But here it's the same question. So if you always store this data in a database or in a data lake, um then it's not that easy to correlate that in real time and send information back to the to the retail store. And that's exactly no matter if you run it at the edge or if you run it in a cloud infrastructure. This is really where event streaming shines because you want to correlate the moving data while it is hot and before you store it to another database or data lake. So no matter where you want to do these computations or what makes more sense for you, it's really important that you can continuously process the data without using a database in the middle all the time. That's for the um latency problems, but also for scalability and so on. I mean, that's the reasons why people use event streaming more and more.

SPEAKER_00

Right. Yeah, sure. So you don't maybe you don't, at least in our in our particular consumer application that we're kind of chewing on right now, you you don't have crushing latency problems. Um like if you if you can't get this stuff done in in 10 seconds, um you're doing something very, very wrong. Um but so maybe maybe you're not driven by latency, but if you've got all this data in a data lake or even in the best case in a database, you probably and I'm gonna make some assumptions on behalf of listeners that uh you know some Kafka streams. We've got some past episodes, and I'll I'll link to one that gives you enough of an update to bring you up to speed here. But you might want to say Kafka connect uh some of those data sources into Kafka topics and then be able to materialize K-tables out of them and do stream table joins so that at least everything is uh is within the context of that that stream processor. All the all the computation is happening within the context of that stream processor. And you don't have, for example, synchronous database calls in the middle of a Kafka streams topology. Exactly. Yeah, that makes a lot of sense. So give us a give give us an industrial uh IoT use case and how how do those tend to differ from the consumer ones?

SPEAKER_01

Yeah, so let's talk about um our a more classical one in manufacturing. So as one example of one of our customers, it's Savastal, which is a steel uh company, so they produce steel all the time, and they have a few factories there in different countries. And their main use case is that they want to do real-time analytics to increase the um quality of the product and to reduce the cost with that. So in the end, what they have, they have some different assembly lines and production lines, and they have a lot of sensor information, as you can imagine, continuously, and it's a lot of data. And their use case is that they now consume and produce this data and correlate all this data in real time to make the right decision. So, for example, if you um produce something in manufacturing, you can scrap parts early, and this means you can save a lot of money if you can find out early that you have to scrap it later anyway. And this is in the example of Zebrostahl a great example of real-time machine learning because what they do, they really deploy an analytic model in a Kafka application for every single sensor information which comes in, they apply this analytic model, and this model then does a prediction, let's scrap this part or um continue with it, and then you can make the cost calculations with that. So here it's really the business case is saving money and increasing the quality. And the big different thing is here, so in in contrary to um consumer IoT, this is not about thousands or even millions of users with their mobile apps, this is a factory. So um, this is maybe a few hundreds or maybe one thousand machines which produce data and sensors. And so it's not the big problem about having too many um clients for the for the streaming data. But the challenge is really that you still have a lot of data because these sensors continuously process data all the time and you want to correlate it all the time. And that's one of the very common use cases where IoT is really Kafka or event streaming at the edge. Because here it doesn't make much sense if you replicate all this data into the cloud or another data sender, do all the real-time streaming analytics there, and then send the result of the calculation back to the factory. So this is a very common IoT scenario where all the action is happening in the factory and not leaving it. I mean, still you can um send some data out for reports to the cloud later, but um most of the critical processing is happening in the factory in this case.

SPEAKER_00

Got it. So you've got infrastructure, and there's actually hardware running, I guess I should just say a Kafka cluster running in the factory.

SPEAKER_01

Yes, exactly. So in the end, it's nothing special, right? It's like in your normal data center where you build your, I don't know, banking application or something like that. And the same is true here, but here everything is then deployed in the factory. And um, of course, this is only happening if it's a bigger factory, but then you have more something like a small data center there, um, because you have to run it mission critical, and therefore it is typically then a distributed um confluent platform in our case, where you deploy and monitor and run all the mission critical client-side and server-side systems and connect to all the sensors and so on.

SPEAKER_00

Yeah, so that is about reliability more than latency in in your observation. Like it it's not a good thing.

SPEAKER_01

They don't want in in most of these cases, it's really both, because um here really um time matters also. So um, if if a machine in IoT in manufacturing is down for for because you have a problem because uh an engine breaks, um therefore um you have to act quickly before that. And therefore, you want to predict uh, hey, my machine is breaking in the next five minutes, um, please shut it down and re replace a part which will break soon with a specific probability. So it's it's still also about latency, and uh, often that's also important. But of course, um mission criticality is the even more important point because most of these factories are built for um 24-7 uptime, so it's critical. However, having said that, on the other side, we still see um edge deployments where in the meantime we often have a discussion about really having just one Kafka broker. There, of course, you lose the high availability, but often it's good enough if you just want to um send and replicate the data to a central cluster somewhere in the cloud, but still want to do some local processing. This is so often a discussion about your SLAs and um about how much downtime you can live with and all these kind of things.

SPEAKER_00

Right, right. Uh yeah, because I guess downtime, factory downtime, if a line stops, there are a lot of costs associated with that. You're still you know accruing all the capital costs of all the equipment and they're not producing anything. And sometimes you have strange process dependence dependencies where a line going down in one part of the process causes you to have to stop something that's expensive to restart elsewhere, particularly in steel, uh or things with metal. You know, you have big things that get real hot. And uh, if you if you let them get real cool, it takes a long time to get them hot again. And so they exactly.

SPEAKER_01

And therefore, in this case, it's really about both mission criticality and real time. Real time in terms of really um less than a second, so really fast, all the time, all the time continuously. So therefore, this is actually pretty common for most scenarios we see in IoT. They are typically um they have real-time requirements and are mission critical. So and that's also what makes it so interesting. And maybe uh last point about this um use case is nevertheless, um, while you want to deploy the event streaming directly in the factory, and I mean they have different factories all over the world where they deploy it, however, um then they also still often have the aggregation pattern. So that means that they want to replicate some of the sensor data or information back to a central cluster in another data center or cloud, so that they can can compare and analyze the data from the different factories. So that could be, for example, one data lake which you have behind one Kafka cluster in the cloud. And based on that, you can, for example, train the analytic models because here then you can find pattern and insights to answer questions like why do we have more problems in this factory in China compared to the factory in Germany, which uses exactly the same hardware and assembly lines. And so this is also another very common scenario that if you are distributed over different countries or regions, that still you want to have some kind of aggregation. So um, first of all, you need to make it mission critical in the factory on site to run everything for your daily life. But then also you can replicate it to a central analytics cluster, which often is not that critical because it's okay if it's down for an hour, right? But there you ingest all the data from all the other factories, and there you can run your long-term analytics, which might take a few hours to train a new model.

SPEAKER_00

Right. Ah, that makes a lot of sense. Okay, so you still do need that data in the cloud, as it were, or or outside of the factory in a central location.

SPEAKER_01

Of course, it depends on the use case, but typically um the factories are not also the IT um um back ends, right? So the IT backend is somewhere in one headquarters or so for most big companies. And so it's not where the factory is. So it always depends on on the use case and what exactly you need to replicate. But in most cases, I see it's really a hybrid architecture, not necessarily in the cloud everywhere, but um at least you have edge deployments, which means the factories or the retail stores, and you have a central deployment somewhere in a data center or cloud where you want to do your correlations and also um what we did not talk about yet. Of course, you also need to integrate the information from the IoT uh backends and machines to other systems. That's also where Kafka and Confluent does a lot of work. Because you can imagine, I mean, one thing is to run the assembly line well, that's where you produce your things, but on the other side, you also have to integrate this to many other systems, like for example, your ERP system or your CRM system for the customer management and many other back-end systems. In the big companies, it's not just two or three systems, but tens of systems you have to integrate. And this is typically where event streaming with Confluent also comes into play, because this is also mission critical. The central thing to integrate um the IoT uh machines and information with the back end of the enterprise, this has to be critical and also real time. And that's why we also are part of the architecture here very often.

SPEAKER_00

I was gonna ask, so you you have mostly anticipated the question and answered it for me, but I wanna I I still want to take it head on. Once the data so the having the data be event data in Kafka, in in the factory, makes all the sense in the world. Because again, for this industrial use case, just to back up a moment, you've you've said it's not that there are so many devices, like in the consumer case, but often the data rate from the devices is higher. So we've got sensors on machines, let's just say, to keep it simple, uh reporting data at a high rate, and maybe there are a smaller number of those sensors, but um there's still a lot of events. And so we've got some kind of you know, you've you've you've got Kafka or Confluent platform or whatever it is managing those events. You're doing real-time computation in Kafka streams or KSQL or your own bespoke consumers if you're into that kind of thing, to do predictive analytics to tell you, you know, ahead of time when a machine needs to be fixed, or early in a process when a part needs to be scrapped. And that all makes sense. And it all makes sense as event-driven stuff and real-time stuff. Once it gets back into the cloud and we're talking about ERP integration and you know, now let's do reporting across factories in various geographies and kind of get a view across the business, why not just get that into a database or a data lake and start using traditional analytics tools at that point? Make the case for events on the the very back end.

SPEAKER_01

Yeah, I mean, this is really actually where where um people come to us very often. So um the case for this is that if you use event streaming in the middle here, you still have um all the connectivity to all the backends. So not every backend is is real-time streaming. That's of course important to understand. So for some use cases you store the data in your relational oracle database, but then you also want to ingest the data into something like Elasticsearch for text search, and you also still want to build a new application based on a time series database. So the huge advantage why people use event streaming here to integrate to the rest of the enterprise is that this central streaming infrastructure can do everything in real time at scale. That's what Kafka is built for and battle tested. But it also decouples the systems and can integrate with I call it your legacy communication, like request response or data addressed in a database. So you can also connect to all of that. But the big benefit if the middle thing is real-time at scale, you can also build your next business application in real time if it needs to be. And that's why people use Kafka in the middle instead of any other batch integration tool or ETL tool or something like that. So that's very important why people um use it on this side of the integration. And with that, it's also important um to understand that while we talk about sensor integration, so now back to the other side of the machines and so on, to Kafka. Um, here we have so many different options, and often it's not really just Kafka or Kafka Connect alone, or um we also even have tools like an MQTT proxy to integrate with devices directly with the MQTT standard. So that's one option. However, there is many other options, like it's in the business backend system, the same is true for IoT. Either there is a specific proprietary integration tool already in place. So I'm based in Germany, so in Germany, manufacturers have a lot of Siemens devices and machines, and of course, they also already have a Siemens integration IoT platform in place to do that. So that's where this Siemens technology is great for. And then there is other IoT platforms on the market. Some are specifically for one technology or standard, like we work with a lot of partners for MQTT brokers, right? Like Hyphen Q, for example, which is dedicated to integrate with MQTT infrastructures. And then we ingest it from there into Kafka and from there to the rest of the world. But also on the other side, we are very complementary to these uh generic IoT platforms like Siemens Mindsphere or Cisco Kinetica or the hundreds others on the market. So often it's a combination of the different technologies, and everyone does what it does best. And really, event streaming is often the central layer which integrates both with the machine side and with the sensor side, and also with the rest of the enterprise. And a huge advantage again is if this single thing in the middle is highly available, highly scalable, and real-time, then this is really a back pressure system, so you can integrate with everything, but you can also build new applications in real time on top of that. And that's how how we fit often into the picture. So it's often not Kafka alone, but it's a combination typically with other technologies which are built for the specific problem to solve. However, and one last word, I know it's a long sentence here now, but um one last thing to understand here is also that um from a marketing perspective, all the platforms can do everything. And um, when I talk about this example with Siemens machines in Germany again, then there is this IoT platform from Siemens, and it does one thing very well. It integrates very well with the Siemens machines. However, what it's not really built for is to process all the data and send it back to the other backends like an SAP system or like a mainframe or Elasticsearch. And that's where we often are very complementary and we are combined with these IoT platforms, which are often already in place at our customer.

SPEAKER_00

So the answer to why events on the back end really turns into the event the answer to that question in general. Uh, why why are events why why is event streaming a better architectural substrate on which to build traditional enterprise applications? And uh I will say so. I think that's kind of the case you've just made, which is obviously a case I believe.

SPEAKER_01

Yes, it is very similar. I mean, in the end, it's just different technologies, right? So on the on the machine side, you often have proprietary or very complex legacy interfaces. And in the enterprise world, um you either have modern uh cutting edge technologies and products like let's say Elasticsearch and Hadoop and Spark or whatever, or you also have legacy to integrate, like mainframes and edifact and so on. So the story is very similar. One one big reason here from when you really think about IoT on the one side and the rest of the enterprise on the other side is that one strong thing about Kafka, while it's real-time and scalable, the other big advantage compared to many other middleware tools which are used here or which people try to use here, is that it also decouples the different systems. And this is the most important thing for IoT scenarios. And because if you think about that, machines or assembly lines or whatever you have in the IoT scenario, they always produce data continuously, if they are not down, right? So and you want to produce them data 24-7 because they are working and creating new sensor information continuously. And if the back-end system cannot consume all the data, then also the even streaming platform is the back pressure system. So even if the sensors produce data continuously and the back-end system cannot handle it, um that's okay because that's where Kafka is in the middle. So one consumer application might be able to consume everything in real time, but another one takes overnight or some more time to process all the data. Or maybe then one uh consumer application is down for an hour for whatever reason. If it starts again, it can consume from the event where it stopped before. And um, this is a perfect scenario, especially in IoT, because in IoT um most of the data um is is uh time sensitive. It's sensor data, which is time series-like and with timestamps. And that's a perfect fit for an event log like Kafka with a timestamp with unchangeable events. So um it simply makes much more sense to use this in the middle instead of like a database or a data lake.

SPEAKER_00

It does indeed. Uh that that that case is, I think, very well established. Um you mentioned MQTT, and that always comes up in discussions of IoT. So tell us what MQTT is and why it is strongly correlated with IoT discussions.

SPEAKER_01

Yeah, so so um first of all, I want to say it's it's it's not coming up in every IoT discussion in the real world. It's really more about um in the IoT world, which many um people here which are not working in IoT. Um that might sound a little bit strange, but um if if we go to conferences and so on, of course, everybody talks about MQTT and it's it's a very established um um uh standard for many different IoT projects. So um it's it's great because it's built if you have, for example, a bad communication or a bad network. That's where Kafka is not a good fit, and that's why often you complement MQTT and Kafka. So MQTT is used a lot to integrate um, for example, with driving cars, because when you go through the tunnel or when your LTE connection is not working well, that's exactly what MQTT is built for. And it's really great for these use cases. However, and having said that, in industrial IoT, where you talk about the factories, here often you use completely different standards. Another standard which is emerging in industry 4.0 is OPCUA. So um that's not really related to I MQTT, it's a very different standard, um, which speaks the language of the machines. And you can combine it with MQTT, like you can combine it with Kafka, um, but it's a very different standard. So I I would say in addition to MQTT, OPCUA is the third, is the second standard which is used as much as MQTT in as especially in industrial IoT. And in addition to that, and the real world today is that I would say 18, 90, 95% of IoT projects today in industrial IoT actually don't use a standard at all. And that's exactly why these IoT platforms exist, like Siemens MindSphere, which can integrate very well with the Siemens protocols, which are proprietary and not easy to integrate. And therefore, from our perspective, as events-drewing platform in the middle, we typically don't see just one integration pipeline or one standard. Um, MQTT is one, for example, for connected cars or for consumer electronics, and that's perfect. But for something like um manufacturing, there it's often either proprietary protocols and another IoT platform in the middle, or maybe a framework like PLC4X, which is an Apache framework which has a Kafka Connect connector and can also directly connect to the machines. Or really um it's some proprietary stuff where you have to use another product for that. So um MQTT is one of the three big ones we see a lot, but it's not the only one in IoT. I think that's important for many people in the audience which think it is.

SPEAKER_00

Yeah, yeah, it's it's the one uh so obviously I'm not an IoT specialist, and I I'm I'm sort of with respect to IoT, I'm the the technologically literate layperson, right? I mean, I know some of the relevant protocols and kind of how you make computers and connect them connect them and things, but not knowing the details of those protocols and how often they come up in the real world, the one that gets the press is MQTT. So that's the protocol. And do you happen to know off the top of your head the transport that these generally use? Are they over TCP?

SPEAKER_01

Um yes. So so they they typically are there's also different options, but typically that's also TCP, yes.

SPEAKER_00

Which which makes sense. And then when you get to that, uh and kind of the question I've always had in my mind is um yeah, okay, fine, they're lightweight, you can put them in a small processor on the edge if if you have a device that that is resource poor. Um but pretty much anything you're gonna put in the field is gonna have a TCP IP stack in it. And if you can do TCP and you can concatenate strings, right? Just just take do you have those two capabilities? Can you concatenate strings? Can you open a TCP socket? Then I think that and you know the answer is yes, for whatever device you have, you're gonna be able to do those things. Uh and I think that's probably why uh you you have so many non-standard protocols, right? People are just gonna do their own little HTTP interfaces and call them RESTful and be done with it.

SPEAKER_01

Yes, I mean for IoT and especially for industrial IoT, it's also simply um the strategy of the vendors, they don't want to use standards because they make more money this way. So it's it's really more like that. Um not because they cannot use standards, they are going there more and more because they are forced to from from the others from the customers. But in the end, the real reason is that they want to make it proprietary so that they can that they that you have to use their IoT platform in the middle, also. That's I think um part of the big problem here. Um, I I want to say one last thing one thing again about this MQTT discussion in Kafka. So we will also share in the in the links into this podcast a few documents because I also um have done a talk about exactly this question. Um, MQTT and Kafka, how are they complementary? Because um both have their strengths, right? So it's not that Kafka can do everything. So and we also want to be be honest here, right? Or say the right thing into the public. So um one reason why event streaming and Kafka and Confluent is combined with these technologies like MQTT is because Kafka is great as a central event streaming platform at scale, in real-time, highly available mission critical systems. But um, one of the things it does not do well, it cannot connect to thousands or even millions of devices, and also it cannot handle bad network continuously. And so that's the two big things where MQTT is built for. And therefore, for many use cases like connected cars or um um IoT in the retail stores and all these things, that's where it's a perfect combination with MQTT.

SPEAKER_00

What a great way to end things. My guest today has been Kai Vahner. Kai, thanks for being a part of Streaming Audio.

SPEAKER_01

Yeah, you're welcome, Tim. And people please connect to me if you have your own IoT use cases. I'm happy to chat with you and also about your challenges and and maybe how confluent and event streaming can help. Thanks a lot for listening.

SPEAKER_00

And there you have it. I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at TLBland on Twitter. That's at T L B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in Community Slack. There's a Slack sign-up link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which we think is a good thing. So, thanks for your support, and we'll see you next time.

Head Shot

Tim Berglund

Host
Head Shot

Adi Polak

Co-host
Head Shot

Viktor Gamov

Co-host