Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

Why Kafka Connect? ft. Robin Moffatt

• Confluent, original creators of Apache Kafka® • Season 1 • Episode 36

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 46:42

In this episode, Tim talks to Robin Moffatt about what Kafka Connect is and why you should almost certainly use it if you're working with Apache Kafka®️. Whether you're building database offload pipelines to Amazon S3, ingesting events from external datastores to drive your applications or exposing messages from your microservices for audit and analysis, Kafka Connect is for you. 

Tim and Robin cover the motivating factors for Kafka Connect, why people end up reinventing the wheel when they're not aware of it and Kafka Connect's capabilities, including scalability and resilience. They also talk about the importance of schemas in Kafka pipelines and programs, and how the Confluent Schema Registry can help.

EPISODE LINKS

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

  •  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
  • ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
  • 👍 If you enjoyed this, please leave us a rating. 
  • 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
SPEAKER_00

One of the distinguishing features of the information infrastructure in the modern enterprise is that there's still a lot of stuff that's not Kafka. There's relational databases, there's elastic search clusters, there's graph databases, there's all kinds of things out there that really have a reason to exist. What's a sensible way to get these integrated with your Kafka-based system? We'll talk about that on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. And welcome back, everyone. I'm joined today by my colleague, Robin Moffitt. Robin, welcome to Streaming Audio. Thanks for having me. You bet. Hey, Robin, where are you from and uh what do you do?

SPEAKER_01

I am from the north of England, uh, a little town called Elkley, uh in North Yorkshire, West Yorkshire. Um, and I am a developer advocate at Confluence.

SPEAKER_00

That's right. So, folks, Robin actually works directly for me. Um, I interview a lot of confluent people on this show, and I've been thinking about that recently. This that's not like necessary. This doesn't need to be uh a 100% confluent organ. We have external guests sometimes, and that's a great thing. Um, and I'm probably gonna try to push to do more of that in the future, but uh it turns out that there's a lot of interesting people here with a lot of interesting things to say about uh streaming, event streaming platforms, and Robin is one of them, and not only does he work uh at Confluent, but he is a member of the Confluent developer relations team, and he is I'll let you know. He's not gonna say this, but I'll let you know, uh, dear listener. He's an outstanding developer advocate. So if you have uh he tends to operate more in Europe for obvious reasons, uh if you've seen him speak, you know that. If you've read his blog posts. Uh in fact, if you haven't read his blog posts, I think you're like a minority uh just of among human beings. So I think most people have read Robin's blog posts on the console blog. That might be an overstatement. But anyway, um I can uh I can I can pump Robin up because he's on my team, and I'm great, I'm really glad he is. Robin, I'm glad you're on the show today, too. That's a pleasure to be. Thank you very kind words. Oh, you bet. So uh we are gonna talk about uh Kafka Connect today. And um this is I think a specialty of yours. And uh before we even get into the what is Kafka Connect, um what tell us what technology background do you come from? Before you were a developer advocate here, uh you did some other things at at Confluent, but like prior lives, what what were the technologies you specialized in?

SPEAKER_01

So my background is very much in data um and analytics. So I started my life as a uh as a junior programmer, as a retailer here in the UK, um building um data warehouses and BI systems, as they were called back then, or MIS, um, on a mainframe with DB2 and COBOL programs, um, which I always think kind of lends a bit of geek credibility when you say that you actually worked on a mainframe. Um but it was it was cool. You kind of you learnt how to build things the proper way, um, and then learnt how to burn it all down a few years later. Um so I did that, and then I've been a consultant. I did a bunch of stuff in the Oracle space, although never actually worked for Oracle, which uh a lot of people assume. Um so consultancy gives you chances to see lots of different ways that people make the same kind of mistakes. Um and then I came to confidence.

SPEAKER_00

There you go. You know, it's kind of funny. I also um so I I don't I don't know the details of your CV. Uh and if you had asked me, if someone had asked me, hey that Robin Moffat, has he ever worked for Oracle? I would have said, Yeah, I think so. So it's just kind of funny. It's this this Oracle DBA thing, and people assume, oh, you must have worked there.

SPEAKER_01

Yeah.

SPEAKER_00

No, I I kind of I talk about Oracle a lot, but I never actually worked there, just with that stuff. There you go. And I think uh I I ask about your background because I think that is a reason why you are uh that that you tend to specialize in Connect. If if you look at the uh developer advocates at Confluent, um everybody sort of has uh things that they love and product areas that they're responsible for giving feedback on. And uh you spend a lot of time with KSQL and you spend a lot of time with Connect, and I think your your you know very formal data background uh is just kind of gives you this affinity, like you know the things and you kind of love the things. And so it it seems like uh Kinect is is a legit place for you to specialize.

SPEAKER_01

Yeah, definitely.

SPEAKER_00

It's uh So tell us, given all that, um what what is Kafka Connect? Imagine uh the uninitiated person who knows what Kafka is, doesn't yet know what Kinect is. Uh what is it?

SPEAKER_01

So Kafka Connect is I I do this this an awful lot at uh meetups and conferences, so I'll I'll kind of uh I'll try not to make it sound like a pitch, but um Kafka Connect is uh is how you do streaming integration between other systems and Kafka. So where you've got data upstream that you want to get in from somewhere where it's stored uh or being produced, you want to get that into a Kafka topic, you've got data from a Kafka topic, you want to get it downstream to somewhere that you want to store it. Um and the killer thing is that it's a framework, so it does all of the tricky things about that integration for you. And as a user, as a developer, it's just configuration files. And you can kind of go much deeper on it, but in essence, as a developer, you want to get data in from somewhere, you set up a JSON file and say, get the data from this place and stream it into this topic. You want to get data from a topic, you say JSON file from this topic down to this place. And it's it's kind of as simple as that.

SPEAKER_00

Cool. So connecting, sometimes I say it's for connecting legacy systems to Kafka. Uh, and I that's usually with a bit of a wink and a nod because you know it's implying that everything that's not Kafka is legacy, which is of course a little over-opinionated. But just anything that's not Kafka, yeah, getting data in, and then from Kafka getting data out to that other system.

SPEAKER_01

Yeah. And it kind of solves this problem that people like to reinvent the wheel of over and over again of, oh, I've got data that's sitting in Kafka, I want to go and put it into HDFS. I know, I'll write a Spark job to go and do it. And that's fine the first time, and then they realize they want to scale it out, or they want to handle restarts, they want to handle the schema, and bit by bit they end up reinventing Kafka Connect.

SPEAKER_00

Which is a terrible trap. And that's every time I talk about Kinect, I I warn people of the framework trap. And every once in a while, I'll actually get that question. Um there was a recent interview uh on this podcast we were talking about language libraries. You know, hey, why not write my own? Well, that's crazy. But uh when it comes to Kinect, this is tempting because you think, oh, come on, I just need to go pull this table in this relational database, and when I see an ID higher than the last time I pulled, I'll just take that Java result set and produce it, use the producer API and produce it into this topic. You know, come on, everybody can sketch that on the back of a wet paper napkin. Um, and so there's this temptation to go write that as a framework. But you mentioned a few things in there. You said handling restarts and scaling out, and it's actually not trivial, right?

SPEAKER_01

It's not. And I can see the tempting trap because it you kind of you start off with one thing. And I had a conversation just last week. Um, and this is one of the things, just as a side note, I love about developer advocacy, because you get to have all these great conversations with people who are doing this stuff and they want to know about your product, and you can take these conversations back internally. But they we're talking about Kafka Connect, and they said, So is there a situation where you wouldn't use Kafka Connect? And you could kind of tell that they had a reason for asking this, and you kind of like chat a bit more. And they'd basically done what we said, they'd they'd written their own solution, and it was very cool, it was all Kubernetes and all this kind of cool stuff, but they'd reinvented it. Um and they kind of they got that what they'd done and what they'd done sounded fantastic, but they'd kind of just done what was there already. And I think one of the things about Kafka Connect is people don't realize it's there. They think, oh, we've got Kafka, we've got the PubSub, uh, we've got the producer consumer APIs. Kafka Connect is this separate thing, but it's actually part of Apache Kafka.

SPEAKER_00

Right. And very good point. It is. Uh this Connect is as an API and as a framework, it's uh a part of Apache Kafka. Now, connectors, we'll talk about what connectors are in a little bit. Uh that's a different story. So once we get kind of the architecture on the table, that whole question of um, you know, people will ask, like, is Kafka Connect open source? And the answer to that question is nuanced, right? Like, yes, Kafka Connect, the API and the framework, is a part of Apache Kafka. It is absolutely open source. Um, but then you plug these connector things into it, and some of those are commercial, right? Some of those you actually pay for, some of them are open source, and we'll we'll get to all of that. But uh yeah, that's a that's a that's a nuanced thing because Kinect is is uh an ecosystem, really. Um so give me some examples of things uh by the way, I also want to react to the you know the the is there ever a time you wouldn't use Kinect uh person really looking to be told that it was okay that he or she accidentally re-implemented Connect. You know, like people do that. Sometimes people who are not aware that the thing exists accidentally rebuild it. And you you always you always want to avoid that. I say this all the time. Regular listeners of this podcast will recall me talking about what our friend, the software architect, Admiral Akbar, has to say about that. It is a trap. You don't want to rewrite a framework because you probably are not in the business of frameworks, you're in the business of some other business. And so using something like Kinect is the smart move. But what give me some examples of uh things Kinect typically connects to? We've talked about HDFS and relational databases, but what are what's the landscape like?

SPEAKER_01

So and relational databases are a huge part of it, particularly upstream, but downstream also. Um so I like you say, like legacy systems databases, right? Um they're still they're still used for very good reason in some places and sometimes less good reasons. Um a lot of the time people will have data coming from databases maybe they they want to offload. So like a bunch of relational databases get the data into somewhere cheaper where we can process it and store it cheaper. Um so doing that that just traditional database offload type stuff down into HGFS or up into S3 or BigQuery or wherever, that's that's one of the classic things, and just kind of building out that's just that pipeline around it. Umes are where they want to get the data into a database. Um, you've got a sync connector as well for databases, or into places like Mongo or Elasticsearch. Um, that could be data that's they're generating within their own services that they want to get out somewhere else for analytics or for monitoring. Um, but anywhere where you can think of data coming from or data going to, there's almost always a connector available for it.

SPEAKER_00

Gotcha. And there really is. I mean, it's so true. It's uh there's even um, in fact, I guess maybe I should ask, uh, where do you go to find connectors? Uh like if you want to know, obviously you could just Google, but uh what's the what's the efficient search path there?

SPEAKER_01

Well, it's funny you should ask that. Um the uh Confluence Hub, um so hub.confluent.io um is a is a good source of them. Um so that's uh a curated resource. It links to a bunch of uh connectors that you get from Confluence and a bunch of community connectors as well. And so anyone can submit their connector and have it listed there. Um so that's a good place to go. Not all of them are there, so there's also Google, like you say, um, which is obviously a bit more uh less curated. Um but sometimes if you're looking for a particularly niche one that maybe the author's not submitted it to the hub, uh you might find it on there. But typically just the Confluent Hub is the place to go.

SPEAKER_00

Yeah. If uh if by curated you mean paid rank, then yes. Yeah. And and probably in that case, if there's a connector that's not on Confluent Hub, um then uh my guess is going to be that the thing you're connecting to is somewhat unusual and is therefore going to be a very useful search term. So um this is the the whole point of Hub is that the the common connectors, and even some of the not very common connectors, like let's put them all in one place uh so that they're easy to find and it's very straightforward what the license is, and you know, can I use this? Do I have to pay for it? What are the conditions of use, all that kind of stuff is all right there. Um and you know, for things that would be noisy, like JDBC, you know, that that doesn't narrow things down very much, but uh, you know, some odd mainframe protocol or something that's not there, but there's a connector somewhere maybe on GitHub, that search will probably be better. Uh I realize we're we're um using some language that we haven't quite defined. We're talking about connectors, so maybe we should back up with some architecture. Uh like before we even get to the notion of a connector, um, how does one run Kafka Connect? Like what is it as a program? Is it does it run on a broker? Um, is it a client library that I build into my software? I mean, what what's the the sort of operational modality of this thing?

SPEAKER_01

Yeah, so Kafka Connect um is a um a process that you run separate from your brokers. Um, and just like the whole stack, you can run it all on your laptop, but you wouldn't deploy it like that. Um so your brokers are for brokers. Um you would put your your Kafka Connect uh separate from that uh on its own node or instance or have you deploy things. Um and the great thing about Kafka Connect is that you can cluster it. So just like you can cluster your Kafka brokers, you can cluster your Kafka Connect workers, which is what they're called. So you have a Kafka Connect worker, and within that worker, it can run one or more tasks. Um so you can have multiple connectors spread across multiple machines, so you can scale out your processing, uh, and then you also have built-in um resilience. So Kafka Connect, like I said, around the whole kind of framework idea, it does all of this clever stuff for you. So if you're running, let's say you're ingesting a ton of data from a database across many different tables, you can tell Kafka Connect, I would like to scale out that processing, and it will spin up multiple tasks to pull in the data in parallel across multiple tables, um, right up until your DBA phones up and asks, what on earth are you doing? Um but that can it can scale out as much as you want it to across multiple worker machines. And if one of those goes pop, Kafka Connect will restart that work on one of the remaining ones.

SPEAKER_00

And uh that and that, by the way, all that functionality you just described is the sort of stuff that bites you when you go to write your own, because it's it's not so conceptually difficult that you can't talk through it on a podcast, but it's actually a little bit hard to write all the code. And you know, you don't want to do that. But the uh when you said worker, uh I could basically think of that as equivalent to a a JVM instance or a the a connect process running somewhere, right? Yes, that's right. Yeah. And tasks are then uh the actual little running units of connectingness where I say, you know, I have deployed a JDBC connector, and we'll dive into what a connector is in a minute, but say, okay, here's the JDBC connector, and I want to go pull this one table, and when there are new records, produce them to this other topic. Well, that connector that will now be a task that go runs on goes and runs on a worker. Yeah, that's right. I think you said if I do oh you're gonna No, go ahead.

SPEAKER_01

Sorry, I I I interrupted.

SPEAKER_00

I think you said if I tell a connector, you know, go read from these five tables and produce to this topic, because maybe those five tables have like the same kind of the same schema, they have the same entities in them, then I can even scale that out. I can have effectively a thread running on uh or one of those tasks running on multiple workers in a connect cluster and be doing that in a horizontally scaled way.

SPEAKER_01

Yes, in a nutshell. So so yeah, you can you can tell a particular a particular connector instance to uh to spin up multiple tasks uh to do its its work in parallel. Um and that could be, if we're talking about databases, multiple tables of the same entity type, but it can also be different tables as well. So you might just have a particular schema in a database with you've got um customers and you've got addresses and you've got all this other data that you want to bring in. You could just use uh a regex and say just bring in all of those tables, and that'll fire across into multiple different topics. And it's up to you whether you want to do that using a single task um with the lesser load on your database or whether you want to run that in parallel.

SPEAKER_00

Cool. Cool. So scalable and we're we're parking on the database connector because it's it's easy to understand. And I I everybody uh please don't let yourself develop the notion that Kafka Connect is just for connecting to databases, it does other things. I promise we'll talk about other connectors, but it's this is always the easiest place to do that. This is my everybody's got my background always coming through.

SPEAKER_01

I always kind of come back to a database, even though it's all about Kafka.

SPEAKER_00

Well, it's and it's this is not just you, everybody does. I mean, when I talk about Connect, I always talk about databases, and then I like begrudgingly drag myself into talking about Elastic as a as a output sync connector later. But it is just easy. Uh however, there are things like uh so now I've got this distributed system that's pulling this database. Uh and suppose one of my connect workers dies, and so the tasks on it go away, uh you know, they were at some place, they had some state. Uh well, I was about to use the word cursor, that's a bit loaded since we're talking about databases, but you know, they had some point where they were in that table. What was the last record they had read? Uh does Connect give you fault tolerance with respect to that? Like obviously it does, it's a loaded question, but how does that work if I lose a Connect node?

SPEAKER_01

Um Yeah, so Connect tracks the um the point at which it's either read records in from the source um or written records down to a target. Um and it tracks those. So something we might want to talk about later is the deployment modes of Kafka Connect called standalone and distributed. But if you're using distributed, which is the one that you would generally use, uh certainly in a production case, um, even if interestingly you're only running it on a single worker node, you can still set it up in the distributed mode. And it uses Kafka itself to persist those offsets. Um so it's pulling data in from somewhere, in from a database, in from a NQTT queue, in from wherever, and where it got to, it stores that on a Kafka topic. Um so then if it needs to restart that task because it got shut down or it got lost, um, that's offset information comes back from a Kafka topic.

SPEAKER_00

Right. So it if you know how uh how consumer groups work. So if if uh if if, dear listener, you you know enough about Kafka to understand how consumer groups scale and where off offsets get stored and everything, this smells like that. It's not identical to that, but um there's actually a lot of functionality under the hood that's shared between Kinect and consumer groups. So it's that same sort of notion of using a compacted topic in the cluster to keep track of state. Or in other words, if you need uh a distributed, replicated, fault-tolerant state store to make your Kinect cluster be fault tolerant, uh well, you have one because by definition you have a Kafka near you. And so Kinect uses that Kafka. Um so let's talk about connectors uh kind of architecturally. We've been throwing around, you know, there's a JDBC connector and there's an Elastic and an HDFS connector. So it sounds like they're pluggable modules, but just kind of tell me uh what a connector is.

SPEAKER_01

Uh so a connector is a bunch of code that's been written against the the Kafka Connect API. So Kafka Connects itself is just this framework. There's uh it's part of Apache Kafka, there's a Java doc that gives you all the APIs. So connectors are these plugins that um people write against common sources and targets, um, and which anyone can uh take and develop further against their own if they want to. So if you've come up with just a new wacky place in which you want to store data, you can write your own Kafka Connector plugin. Um but typically as a user of Kafka Connect, you simply find the plugin that you need, it'll exist already. Um it's a jar file that you deploy along with your Kafka Connect worker, and then just configure it uh using JSON.

SPEAKER_00

Nice. So yeah, that's if if you are a Java person or know uh like you know three three things about the Java ecosystem, uh the very lowest level description you can give of a connector is that it is a jar. So there is this API, you write code against the API, the API or your code talks to the other thing, so it knows how to interface with the thing you're connecting to, and then it's either a Kafka producer if you're bringing data in, or a Kafka consumer if you are. As it were, exporting data from Kafka. I guess we call those source and sync connectors, but a connector's code. A connector's jar a jar. And you deploy it to your connect workers as a jar that you put into a class loadable place. And uh boom, then you've got a connector. But you mentioned mentioned some JSON. So tell us how that works. How do I get the JSON to the Connect cluster? And you know, it's it's got the jar file there, but then I need to tell it to wake up and make that into tasks and actually run that code. So how's that whole configuration working?

SPEAKER_01

Yeah, sure. So you configure it differently in different ways. So Kafka Connect in standalone mode. Um I'm not going to address here right now, because my preferred way of doing it, whether I'm developing on a single laptop here or deploying against a 40-node cluster, is using Kafka Connect in distributed mode. So even.

SPEAKER_00

Right. I was I was surprised you even acknowledged that there was a standalone mode, folks. Robin Robin is a distributed mode only guy. And by the way, I think he's right. I used to try to do things in standalone mode for demos and things. And I'm now uh persuaded of Robin's view here. So anyway, there's there's these two modes. We don't care about one of them. Let's talk about the other one.

SPEAKER_01

There are differing views. Some of our colleagues would would advocate for standalone for different reasons. But for me, uh obviously we're going on off on a tangent here, but distributed mode, even on a single node, you're working with the same way of configuring it, which is what I'll explain in a moment, the same way of deploying it. When you want then want to take it to production, when you want to scale out, it's all the same. It's just you've now got additional workers. Whereas switching between standalone and distributed, suddenly you've got like this gear shift and it's just clunky and whatever. But there are some use cases for standalone. So my colleagues will advocate for them, and that's fine. But here we're talking distributed. So for distributed, you spin up your worker, and your worker's got a uh a REST API. And so with your REST API, you can send your JSON to it to configure a connector, to create it, you can query it to find out the status of it, to find out the status of the tasks within it. And this is yet again another fantastic thing about Kafka Connect because you now have this centralized way of configuring and monitoring and checking the status of all of your integration pipelines. So if you go your own way and brew your own solution around, let's get data in from a database using this clever thing that we're going to build ourselves, that's great. But that's for one database. And now you want to also send data down to Elastic and pin in data from a REST endpoint or whatever. And so Kafka Connect, it does all of that for you. It's it's abstracted out the configuration and the monitoring, and you just plug in the relevant technology uh connector that you need.

SPEAKER_00

Yeah, and that REST API is super cool. Uh it's as you said, it's the way that you configure Connect. You write a little blob of JSON and you post it uh to the REST endpoint on uh one of the nodes in the Connect cluster, any of the nodes, right? Just tell the cluster about the connector that you want to run and it works. But then you can get various URLs at that uh endpoint, and you'll get pretty human-readable JSON back. Uh it's a it's a nice API to just kind of browse. You can look in a browser that I mean most browsers uh will do fair fairly sensible things rendering JSON these days, but just kind of do something sensible with rendering the JSON, and if you don't want to use a browser, you can use curl and jq and you know all of all of the usual JSON suspects. But it's it's frankly uh easy to explore, like a good REST API should be, and of course then easy to automate if there's anything you have to do uh with the status of those connectors uh programmatically. It's pretty straightforward. And uh the details are not amenable to podcasty discussion, but if you want to know more about that, go to docs.confluent.io and click on the Kafka Connect link on the left, and there's uh plenty of details on the uh the REST API there. But it's pretty handy. Okay, so what do we have here? We've got uh this uh data integration framework that runs on uh uh client machines of its own. It's it doesn't run on brokers, it's its own client process with respect to the Kafka cluster, it's this external service. Um there's this ecosystem of connectors. The connectors are these little jar files that do interfacing to some external system and to Kafka, either to get data in as source connectors or to send data out as sync connectors. Uh so I can deploy those jar files to the machines that my uh or containers or whatever they are, that my connect cluster is running on, and then I have this rest endpoint that I can post uh little pieces of JSON to turn it on and actually configure it and tell it where the external resource is and what the credentials are to it and you know all the depending on what the connector is, uh how to how to talk to that thing and what to do. So that's Connect in a nutshell. Um and Connect Connect clearly has grown up, like we talked about initially, as uh a formal community-backed Kafka solution to a problem that everybody had to solve. And everybody could write their own buggy partial implementation of Kafka Kafka Connect and never quite have time to get it working properly, and it would limp along and it would sap time away from them developing features that are valuable to their customers. That would be bad. Um but there are some other things, just given the the connect that we've described so far, there it seems like there are some other things that are going to happen. Like you're bringing data in, and you know, there's this one database table, and there's some PII in it, and you don't want that to get into uh the topic. Or uh, you know, here is uh some row you're reading from a CSV file or something. I just made that up. I don't even know if that works, but um you know that it's it's it's just uh of a value, and you want to produce it to a topic with a particular key, and you want to extract that key from a field inside the the message that you're reading. So it seems like there are these usual suspects of little conversion and transformation things that are gonna happen. This gets us to single message transforms. So tell us from the top what are what are SMTs? I realize I kind of framed it, but what would be a good explanation of what SMTs are?

SPEAKER_01

So single message transforms, um, as the name uh suggests, it lets you transform single messages. So as in as each message passes through Kafka Connect, you can apply a transformation to it. And you can stack these transformations up, so you can do lots of different things. Um so some of the very common ones are you can drop fields. Um so you can, like like you suggested, you've got some sensitive data in there. You simply just don't want that to come into your topic. Um, so you're putting data in, maybe you're reading a flat file. Um so that's another common uh use with Kafka Connect, is people shouldn't, but people do use files as a mean of data interchange. Uh sometimes it can't be helped, sometimes it's just, well, it should be done better, but it isn't. And so you can use Kafka Connect to bring in data from a file, and you might say, well, in that file is this data, and we only want certain fields from it. So you could bring it into a Kafka topic and then do some processing and so on and so on, but you could also say in Kafka Connect, as you bring each record in, just drop certain fields. Don't even write it as a Kafka topic, and now it's um we don't have that data in our system. You can use it to change data types. So you can cast uh fields different to different types, you can do um processing on the topic name as well. So, particularly if you're writing downstream, um something like Elasticsearch Connector will use the Kafka topic as the name of the target index in Elasticsearch that it writes to. So you might want to modify that, you might want to add on a timestamp into it to do um time-based partitioning, you might want to do various things. So, again, you've got Synchromus Transform that lets you modify the topic name. You can use them to uh inject additional fields into your message. So you could say um as data comes into the system, you want to add a field which gives it some lineage. It's and say it came from this server, it came in this ingest process, and then that data would then propagate with that message throughout its lifecycle through your Kafka system. Um so there's tons of different things that you can do with single message transforms. The key thing is it's transforming a single message. So it's also part of the Kafka Connect API, um, and you can write your own transformations if you want to. But um there's there's a great presentation um from Yuan, one of my colleagues, around when should you actually use single message transforms? Because you could get it to do an awful lot of stuff, but you probably shouldn't. Um you shouldn't use it to try and do lookups out to other things. There's you quite quite rapidly get on to basically stream processing, which is where Kafka Streams or KCE call would come in.

SPEAKER_00

Right. Uh super good point there. And I know Yuan has made that point strongly that uh SMTs are good. And if if you think about all of the examples you just gave for all of those extremely useful things. That again, you if if you had Kafka Connect and you didn't have an S a single message transform API, you'd have to make one. There frameworks would emerge to do this because everybody is going to have to do some subset of those things that you described. So it's very good that there is a formal way to do it and uh kind of an included package of the usual suspect of suspects of the single message transforms. That said, if you think about all the things that you described, every one of them is stateless. And the name implies that they ought to be. They are single message transforms, they are not message transforms that know something about the prior message. So that warning of Ewan's, I think, is is uh uh well placed because there's this temptation once you get into this API to think, oh well, this is great. Let me just do this extra little thing, and well, I need some state now, so let me figure out how I can hack things to remember this map of and you know that's you've you've kind of failed at life once you once you go down that path. Don't don't do stream processing in SMTs. Uh it's gonna it's gonna not be fun. Um how about converters? Those are not I know they're not the same, but they're they they lurk in kind of the same part of the block diagram. What what are converters?

SPEAKER_01

So converters are part of the the internals of Kafka Connect as well, just like single message transforms. Um and they're probably the thing which chips people up the most. Um they're one of those things that you that up front seem overly complex and like why do I have to think about this? But if you actually think it through, it makes an awful lot of sense, and you can't really do without them. So what converters let you do is define your serialization and deserialization um formats that you're going to use. So we're bringing data in from somewhere, we're going to write it to a Kafka topic. Kafka topics being just bytes, you need to decide how you're going to serialize it. And then at the other side, we're reading data from a Kafka topic to write it down somewhere else. How's that data been serialized? So Kafka Connect lets you use uh Avro or JSON or whatever you want to use as a particular um defined format. So personally, I think Avro is a very, very strong choice there. Um you can also uh use JSON, you can use string if you really want to. There's also community converters for protobuf, but the converters are pluggable. So you can say um I'm going to I'm going to use this particular converter here, and you just specify that in your configuration. So rather than having to override it each time, uh you can simply set up your Kafka Connect workers and say, we always use this particular format. Unless perhaps you're reading from a topic that maybe a different team's written and they've used a different serialization format, you can override it then per converter. Sorry, per connector.

SPEAKER_00

Per connector. Yeah. Okay, nice. So yeah, I guess that that makes sense. It's not it's it's not really that complicated. Uh the stuff in the topic has to have some format. And whatever you're reading from probably didn't have that format. So it it uh gets from whatever the outside world thing was into, as you say, Avro or JSON or or Program.

SPEAKER_01

Yeah, and and the key thing about that is that you then start to think about schemas. Um and this is when I kind of like the my pet uh rants that I start going on about is because when you're pulling in data from another system, that has almost always got a schema. Um and if you say I'm gonna just chuck that data in as a string into the topic or JSON without a schema, you're throwing away that schema. Um and that's a real shame because schemas enable us to use data across systems in a much more loosely coupled way. Uh if you think about pulling in data from a database from anywhere, that schema you can actually then use when you come to use that data. So if you want to write that data down to another system, down to a database, down to Elasticsearch, down to somewhere else, or consume it within your own application, knowing what the schema is to that data is super important. So if you use something like Avro, uh, for example, you can serialize that data onto your Kafka topic with nice small bytes of just the payload, and the schema itself goes up into the confluence schema registry. When you come to use that data, whether it's in Kafka Connect or your own consumer or KSQL, wherever it is, that schema is available for you to fetch from the schema registry to deserialize it, and you've got that full rich schema uh already there. If you don't do that, then you have to manage the schema yourself, and that's where problems can creep in, particularly as the use of that data becomes more and more widespread within an organization.

SPEAKER_00

I love it. Uh that's a good point about uh discarding schema on read or you know, on import. Um it's not, it's you know, there isn't data that doesn't have a schema. Uh everything has a schema. Uh it's just either explicit somewhere or implicit and to configure a source connector to write data into a topic and not connect to the schema registry and manage that schema in the schema registry, but is in a sense to kick the can down the road. You're saying, well, uh anybody who's gonna use the data in this topic, apart from whatever immediate concern I have in my mind, um anybody who's gonna kick that can down or who's gonna use that data, uh, they're gonna have to figure it out later. Uh like maybe I put a page in the wiki that described what the schema was right now. I'm probably not gonna keep that data up to date, that page up to date. Um it's it's really uh there there is sort of a collaboration aspect here uh to schema management that you for the sake of future human beings who are going to use this data in ways you don't anticipate. And by the way, if you're using Kafka, uh that's kind of a fundamental part of the value proposition, is that once data lives in a topic and uh lives durably in a topic, other people can use that in ways that you haven't thought of. Um, you want to kind of help those people get out ahead of that and say, hey, here is what the schema is. It's centrally managed, it's in the schema registry, and you have some hope of of uh you know writing code that will be able to use that data and you know not have formatting exceptions and schema exceptions happening all the time.

SPEAKER_01

Yeah, totally. And Gwen Shapera has got this great presentation that she did at QCon, I think it was a couple of years ago now, and which she talks about schemas being the API. Um and and they're just they are just so important. And it's I can see why people don't, because it's like, ah, this is uh extra overhead, we're new to this, we're kind of like, we'll worry about that later. We'll we'll just do it in the easiest way, and the easiest way is well, we'll just put it in JSON because I can read that, I can understand what's happening. But exactly like you say, the use of Kafka then takes off, and then teams who you didn't even know existed or you didn't think they'd want to use the data, they start wanting to read the data, and now you're in that situation of very brittle pipelines that start building and and then start breaking as you try and change things. So you using the schema is super important.

unknown

Yep.

SPEAKER_00

Cannot say that enough. That's uh one of the most to me, the most interesting uh I don't know how to put it. It's like it's like Kafka exerts this influence on software architecture. I mean, you adopt it and you think I'm gonna solve this problem, and then you start getting in the habit of storing immutable logs of events, and that changes the way you think about the rest of your system. It changes what you're able to do, it affects the it makes it makes you better able to refactor to microservices and get more value out of that kind of paradigm. And like all this stuff happens, and over in the corner, winking at you, is schema management. It's it's saying, you know, hey, that that really is the API between services now and between consumers of data. Uh APIs are not synchronous things where you know there's some service network location that I have to know, and I pass data in a certain format, and I wait and I get a response in a certain format. Uh it really is just where's the data and what is it shaped like? So that's just so key. Um we kind of batted around um licensing questions. And I I know this comes up uh you and I've talked about this before. This comes up when you talk about Kinect, it comes up when I talk about Kinect. Um what are uh you know, I I said before that Kinect was an Apache, was a part of Apache Kafka, but just talk to me about the licensed landscape. How do I think about that? How does a new Kinect user think about that?

SPEAKER_01

That's a good question. And it's one that, like you say, it comes up an awful lot. Um I I post quite a lot of stuff on Stack Overflow, and there'll be people maybe completely new to the ecosystem, but they've got some data set in Kafka, and they're saying, What's I need to get it over to here. Is it free? Um and it's that's quite a it's not a simple, straightforward question because, like you say, Kafka Connect is part of Apache Kafka. It's 2.0, Apache 2.0 licensed, it's free to use and do with what you will. The connectors that you need as part of that, some of them are proprietary software, some of them are confident community licensed, some of them are from the community and licensed, however, the author has decided to license them. So you can use Kafka Connect. If you've got Apache Kafka, unless you're like on a super old version, uh, if you've got Apache Kafka, you have got Kafka Connect. So then it's a case of which connector do you want to use and how is that particular connector licensed?

SPEAKER_00

There you go. So yeah, the connect itself, the framework, the API, you know, the interface if or the base class, if I want to go write my own connector, all that stuff. 100% Apache Public License 2. Everybody knows and understands that license, take it, use it. Uh it's all very easy. Also, I think the file system connector is included with AK. That's that's a part of Kafka. Super useful. It shows up in the enterprise everywhere, but no, it's not it is included, right?

SPEAKER_01

I believe it is, but it's one of those ones which I suspect people wish wasn't always, because like I mentioned earlier, people use flat files as a means of data interchange when a lot of the time they shouldn't. They should probably just be using Kafka or something like that. But they want they they say, I've got a file here, I want to read it in, or I need to write to a file because someone has said they didn't want me to write a file. And the the file system connector is not designed for that. The file system connector is designed as an example connector that kind of gives you an idea of what you can do with Connect, and you can do a quick hello world example. Um, because it resides relies on a file being local, and Connect is a distributed system. So yeah, the file system connector is an interesting one, but I certainly I would need to double check, but I'm pretty sure it's part of Apache Kafka distribution.

SPEAKER_00

Yeah, the others um pretty much well no, not pretty much, every other connector is uh not a part of Apache Kafka. Now, some of those connectors uh just in the community are licensed under the APL as well, the Apache Public License as well. But each connector is its own program, has its own copyright holder, has is released under its own license terms. Connectors are individual. So one of the nice things about Connect Hub, if I can shamelessly plug that, even though it's a confluent thing, um it's okay. This is a interview between two confluent people, so it's totally appropriate. One of the nice things about Connect Hub is that it tells you the license of all the things. So no matter where the the connector comes from, you can get license details and just know what you're getting into. And in many, many cases, they are either an OSI approved open source license or a community license that makes them free to use, almost certainly free to use, in the whatever it is you're doing with it. Um and uh this is not a legal podcast, we're not lawyers. Even if we were lawyers, we wouldn't be your lawyers. So it would be a terrible thing to take legal advice from Robin and me. But in general, I mean the simple answer for most people who are trying to build stuff with Kafka Connect is that you don't have to pay to use most of the interesting connectors and and details vary on on each one, but most of the time to do the kinds of stuff we're talking about, those are things that most people don't pay for. Robin, your favorite thing you've ever done with Connect? What is that?

SPEAKER_01

Um I there's a there's a really good community connector which lets you interface to a rest endpoint. Um so I've always liked building demos using kind of public data sources, so um that's quite a fun one um where you can you can point Kafka connector at a rest endpoint and say like every five minutes or whatever, pull the messages in from here. Um and that then pulls in a bunch of data that you can kind of build out cool demos with around KSQL and stuff like that. So um I like using that. Obviously, database one, database background, being able to interface with a database. Um I must I have to give a shout out to Debezium connector as well, um, being able to change data capture against uh different data sources, uh different databases rather.

SPEAKER_00

Also very, very we didn't talk about um change data capture or Debesium at all today. I think they deserve their own episodes. We should get them from Debezium on as a guest. Totally sure.

unknown

All right.

SPEAKER_00

My guest today has been Robin Moffat. Robin, thanks for being a part of Streaming Audio. Thanks very much. And there you have it. I hope that was helpful to you. If you've got questions, you can ask me at TLBurgland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on any of our YouTube videos. Your question might be featured on the next episode of Streaming Audio. And feel free to subscribe to our YouTube channel and this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast and just generally helps us get the word out. We appreciate your support. See you next time.

Head Shot

Tim Berglund

Host
Head Shot

Adi Polak

Co-host
Head Shot

Viktor Gamov

Co-host