Making Apache Kafka Connectors for the Cloud ft. Magesh Nandakumar

From previously focusing on Confluent Schema Registry to now making connectors for Confluent Cloud, Magesh Nandakumar (Software Engineer, Confluent) discusses what connectors do, how they simplify data integrations, and how they enable sophisticated customer use cases. With connectors built for Confluent Cloud on Google Cloud Platform (GCP), Microsoft Azure, and Amazon Web Services (AWS), this helps users implement Apache Kafka® within their existing systems in an easy way. There’s a lot that Magesh is looking forward to when the world of connectors and the world of cloud collide.

EPISODE LINKS

• Why Kafka Connect? ft. Robin Moffatt
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.
• Get 30% off Kafka Summit London registration with the code KSL20Audio

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_01 0:00

Confluent Cloud is gaining features at a pretty rapid pace, and if you want to know what's going on with your favorite fully managed Kafka service, you'll need some updates. Today, Cloud Engineer Magesh Nandakumar joins us to tell us what's up with hosted Kafka Connect. It's all in today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the Cloud. Hello and welcome back to another episode of Streaming Audio. I am, as ever, your host, Tim Berglund, and I am joined in the virtual studio by my coworker, Magesh Nandakumar. Magesh, welcome to Streaming Audio.

SPEAKER_00 0:41

Hey Tim. It's my pleasure to be on the show again.

SPEAKER_01 0:45

Now, yes, I should say welcome back to Streaming Audio. Remind us what is it that you do here in case anybody's listening for the first time. Give us kind of a little bit of your background, what you do, how you came to do it.

SPEAKER_00 0:56

Yeah, sure. Um I've been working with Confluent Engineering for about two and a half years now. I started off working with schema registry, REST proxy, uh, did a little bit of security plugins for all of this, and then I started working on connectors, and then I did schema registry in cloud, and then the connectors in cloud. Um, so those are some of the stuff that I've been working on. This is on a very, very high level. Obviously, I can get into details of all of this. Uh each one is uh like a rabbit hole by itself.

SPEAKER_01 1:27

Certainly, certainly. What did you do before Confluent?

SPEAKER_00 1:30

Yeah. So before Confluent, I was at uh Visa. Uh I was working on their uh digital products and I was working on a fraud detection team. And as part of some of our re-architectures, we started using Kafka. Uh that was our obvious choice, obviously, because we got to do uh fraud detection, uh, which is a great use case uh for Kafka and stream processing. And uh I started learning a lot about Kafka stream processing, and uh through the course, I started learning about Confluent as well. And uh I fell in love with the product, uh, the vision of the founders, and I was like, okay, if I want to do something uh better in my career, I very well go do what Confluent does, and then I applied got in here, and the rest is history.

SPEAKER_01 2:19

The rest is history, that's right. Um cool. And you said you have been working on Kinect recently?

SPEAKER_00 2:26

Yeah, I've been working on a lot of Kinect related stuff for about uh year and a half now. Yeah. Oh wow, okay.

SPEAKER_01 2:33

And um we'll put up a link in the show notes to an episode that covers just what Kinect is by itself. Uh so like if you don't know what Kinect is, we'll refer you to an episode. We've got a number of episodes kind of over the summer of 2019 where we we covered Kinect pretty well. So we'll get that back in front of you. Um but just kind of so you know, pause now, go listen to those, get refreshed on what Kinect is. Or if you know what Kinect is, tell us, Magesh, what you've been up to.

SPEAKER_00 3:06

Cool. Um, so I started off uh uh starting with a lot of connectors that we already had in the array, like and then uh uh we did a bunch of stuff there. Uh we did some stuff with RBAC and all of that, and then I started working on the most exciting thing that I've ever done.

SPEAKER_01 3:24

Before you tell me that, I want to I want to keep everybody in suspense. You said RBAC real quick. What's RBAC?

SPEAKER_00 3:30

Um so uh there was this thing called role-based access control uh that we wanted to implement for Confluent Platform uh across all our products. Um so I did the initial discovery phase for it with uh Connect and then uh um worked with a product management team here to uh um come up with like the set of things and how it should look like for Kinect schema registry. Um and yeah, so we have it out there in uh CP5.4 as a preview. Um yeah, it's a great feature out there. It's I I think it's gonna be a pretty useful feature for everyone out there.

SPEAKER_01 4:07

Awesome. Then I interrupted you, right? And you're starting to say that you started working on the most exciting thing, and that is getting connectors on cloud.

SPEAKER_00 4:17

Yes. Yep, yep. Um, so it was a really exciting thing. Uh we started off very small. Uh we wanted to do just a S3 sync connector uh in cloud, and uh that was our first project. And while doing so, uh we had to build a lot of control plane stuff to get this in order. And we also wanted to do this with the best experience possible, uh, just so that it's super easy for users to come in, configure their connector, and get going. Um so uh connect itself is like super easy generally, uh just because like uh all you do is like uh uh declaratively get something up and running. Uh, when you can do something declaratively, then uh there's nothing better than that. Uh but connectors in cloud is much more simpler because you don't have to really manage any physical clusters of any sorts. All you think about is just the connectors and what it means declaratively uh to get them up and running.

SPEAKER_01 5:23

Right. And it's kind of funny when you know when you look at what Connect does relative to uh not having Connect, you know, it's a data integration framework, and you download a connector from the internet or buy one if it's commercial and you throw a little piece of JSON at it, it goes and pumps data from some external source into a topic or out from a topic to some external source. Like it's so much better than writing all that code yourself.

SPEAKER_00 5:48

Oh yeah.

SPEAKER_01 5:48

But then once once we're in the world of the cloud, we're like, oh man, I have to run that cluster and I have to you know figure out what where the the what the name of the thing is, where the configuration goes, and I have to go install connectors. It's like we're um you know, we're we're into complaining mode, and I feel it. I really feel it because having that just as a checkbox in a cloud UI is so much better. So where and by the way, everybody, we're recording this in early November of 2019. It will release some weeks after that, sometime in in the remainder of the year. Um and so we're gonna talk about what's in cloud right now and and when you're listening to this, whenever that is, that could be different. It's there's probably gonna be more available in cloud.

SPEAKER_00 6:35

Oh, yeah.

SPEAKER_01 6:36

But tell us, Magesh, as of right now, um, what what is Connect's footprint in Confluent Cloud?

SPEAKER_00 6:43

All right, cool. Um, so on a very broad level, uh Confluent Cloud is available in all the three major cloud providers AWS, Google Cloud, and then Ashare. Um so when it comes to Connect, we started off with S3 Sync Connector, which helps you obviously move data from uh Kafka into an S3 bucket. Uh that was our first ever connector. We launched it as a preview connector, and then it has already graduated into a production available connector. And uh we also have two other connectors in uh Google Cloud and Azure. Um so in Google Cloud, we do have a GCS sync connector, uh, which is very similar to S3. And then for Azure, uh we just enabled it a couple of days ago, uh, literally uh Azure Blob Sync connector. Uh it's also in preview.

SPEAKER_01 7:36

Okay, so that's basically S3 Sync, Google Cloud Storage Sync, Azure Blob Sync. So the the object stores of the three supported cloud providers. Yep. Those are there and and generally available, and anybody can sign up and use them and they just work.

SPEAKER_00 7:49

Uh S3Sync is uh generally available. Uh the other two are in preview today. But anyone can come in like yep.

SPEAKER_01 7:55

Yeah, and we're gonna talk about some more preview stuff. Like I know that'll come up. So if a feature is in preview in cloud, how do I get access to it?

SPEAKER_00 8:04

Uh so right now, uh, if you're a confluent cloud user, uh you get like open access, it's a public preview. Uh if you're an enterprise customer, you work with a sales uh team to like get it enabled. But uh in a few weeks from now, like it's gonna be public preview for everyone. Uh it comes with some limitations. There are limitations about number of resources or connectors that you could create. Uh, but yeah, it's a preview.

SPEAKER_01 8:32

Cool. And so by the time you're listening to this, those if they are just generally available and they don't say anything about preview, then that's why, because cloud is a rapidly evolving product and it's actually this is always a challenge. Anytime we're talking about a specific conflict cloud feature, um, you know, there's a certain publication lag, there's a a schedule of these things, and you know, we talk about them on one week, and then two weeks later, oh well, that's different. There's even more stuff available. But it's bad. It's it's it's the right kind of problem to have, you know, we're not taking features away. So how do they differ? Um, if if suppose I have experience with any of those connectors uh in the on-prem version of Connect, how do the cloud versions differ?

SPEAKER_00 9:12

Okay. So the biggest difference is uh the kind of configurations that we expose. Uh, for example, if you look at an on-premise three sync connector, there's like tons of configuration options. Uh, if you look at a partitioning strategy, you have like so many options available out there to do that to do that. Whereas when it comes to cloud, uh we think uh there's a right way to do this. Uh and uh we just expose those configurations to you uh that are super critical to get it up and running. Uh, for example, uh for an S3 sync connector, uh, it's super important that everyone gets UOS guarantees. And for you to have UOS, uh, it works only with certain partitioning strategy. But as a user, why do you even have to think about all that? Like uh we will make that the default partitioning strategy. We will give you some knobs uh to figure out like whether you want daily partitioning or hardly partitioning. Uh and that's all you have to worry about. You don't really have to worry about how do I partition, what do I partition on? We by default partition on the record time. That means uh you don't even have late arriving records. These are some of the experience that we have seen with some of the customers. Um they sometimes have late arriving records, then they go out in different partitions than what they expect. Um, so we are trying to keep it simple. Like this is the most straightforward way to sync data into S3Sync, and this is the right way to do it.

SPEAKER_01 10:44

So, in other words, you make the configuration slightly opinionated. Yep. Um and the benefit you get out of that is that they're simpler to configure.

SPEAKER_00 10:52

Yep. It's simpler to configure, it's simpler to understand. You don't have to understand 50 different configurations.

SPEAKER_01 10:58

Right, right. That's uh that makes a lot of sense. Uh because honestly, some connector configuration is uh onerous and it's nice. It's the kind of stuff that you know you see in a mature product where there's the six configuration fields you probably are usually going to use, and then there's another 30 that you know somebody asked for each one of those at some point, and and you know, two people are using each of them, you know, and that that um is typical, and it's nice that we've kind of reset that for the the cloud versions. Do you see, and I I totally don't want a rabbit trail on configuration here, it's not the most interesting thing to talk about. But I'm just curious, is it your guess that uh cloud connector configurations will also tend to sort of grow some frills over time?

SPEAKER_00 11:48

Uh it could a little bit, uh, because like I I don't think we can get the opinion right the first time. Uh we have already uh like made some uh improvements uh to the configuration based on user feedback. Uh that's the most important thing in my uh thing. There'll be a few things that we don't want to change, and there'll be a few things that we would want to like expose. Uh one good example that I could give is like initially we didn't expose the flush size, but it had a huge impact on the file size in the blob storage. Uh so we were like, okay, uh we will have a minimum flush size which will guarantee you certain kind of throughput, but we will also let you increase that from there uh just so that uh you can get bigger files if you need it. Um so we did make that adjustment and expose flush size, but we had uh uh validation around it uh with a minimum uh value that gives a certain amount of throughput. And uh there have also been uh other cases where uh uh users wanted some internal uh uh some settings like uh localization and uh internationalization. Um so those are good settings to expose because like uh if you are out there in Europe and if you want uh your partition names to be in like French, you obviously want that locale to be enabled. Um so I think those are good configs to expose. Uh but anything that uh drastically changes the behavior of your connector, those are the ones that we want to be super careful about.

SPEAKER_01 13:24

Sure, because it's difficult for the user and it's difficult to manage the uh you know to manage the running of the connector if you're getting uh you know large swings in how the thing behaves.

SPEAKER_00 13:35

Yeah, the most important thing is like you want a deterministic behavior of the connector. Uh whatever be the scenario. Uh like right.

SPEAKER_01 13:45

Right. What so you're an engineer working uh on connect. What's hard about moving connect to the cloud? Is it those weird configuration parameters? What what are the other what kind of interesting stories can you tell about uh the process of of moving it to the cloud?

SPEAKER_00 14:02

Um so um there's a lot of uh minute details when it comes to moving into the cloud, right? Like one of the things that we wanted to make it simpler for the users is like users should not really uh um care where the connectors are running. Um to them, it's just a connector. Uh it it could be in a physical cluster A or a physical cluster B. Uh like we wanted to give that notion of a serverless connector, right? Uh so that was one of the most interesting challenges uh that we had to solve for. Um so we kind of created APIs which are similar to the Apache Kafka AFKs that's available uh in the open world. Uh, but we also did not want to give control for uh every single thing. Um, for example, like uh in a typical on-prem connector world, you could go like restart specific tasks. But in a managed connector world, you should just think about the connector. You shouldn't really think about uh fine-grained things like a task. Uh today we do ask you to specify the number of tasks when you configure a connector, uh, but that doesn't necessarily mean that uh you got to be able to control the tasks. Uh, you got to be able to like monitor the connector on a high level and see if it's doing the right thing. Uh, that's all you should care about if you are paying somebody to get a connector managed. Uh so we had to take all that pain on us. Um so you got to get the observability right on things. Uh, you got to have a lot of self-healing in terms of like when tasks die, you gotta figure out whether it's dying for uh uh some bad uh bug or like bad uh worker settings, or if it is just failing because like the configuration is stale. Like uh, for example, let's say somebody configures a S3 sync connector, and for some reason, if let's say the API key uh access control is changed uh behind the scenes, uh like we do want to be able to provide that feedback back to the user uh that your API key is invalid now, and that's the reason the task is not running. Uh whereas if it is any other issue, the user shouldn't really worry why it has failed. It's sh it's a managed connector, uh, it should be handled by confluent support or whatever.

SPEAKER_01 16:35

Exactly. So that's kind of continuing in the tradition of serverless Kafka. When when we say serverless Kafka, we mean we want to kind of minimize the or strip down the footprint that you see to the barest essential thing that you need. Yes. Uh in the case of Kafka, that's topics. Yes. That's not brokers. You don't need brokers. So, you know, how many nodes are in my connect cluster? Well, you know, don't worry. Uh your connect cluster will we do ask you to set tasks, but um we'll think about nodes, we'll think about scale, uh, we'll make the connector run. You're not using a cloud service so that you can just think about your infrastructure on EC2 instances. You know, you want all that to go away.

SPEAKER_00 17:20

Yes. Yes. I mean, there's a difference between deployment automation versus a completely managed service, right? Like we want to be at a completely managed service where users just have to think about running the system. And then they obviously want some metrics about the system itself, because like, for example, as a connector, they want to know like how many messages am I moving? Uh how am I doing with lag? Uh, so we do have to give those feedback back because, like, for example, if they have to, let's say, scale the connector in terms of the number of tasks, uh, they should have some feedback about it. Um, so we do provide those kind of information back to the users. But that doesn't that just means that they have to uh think about scaling the connector itself, but not the connect cluster.

SPEAKER_01 18:08

Exactly. Uh there is uh the the question for Confluent Cloud User is what even is a connect cluster? I mean, there is one, but it's not something we manage as users. Yeah. Um what is in uh preview? What are some connectors that are gonna be coming up in preview or are currently in preview? We talked about the three blob stores, but uh what what is that you know, something that's not a blob store? Uh what's coming up?

SPEAKER_00 18:37

Okay. Um so uh pretty soon uh we are gonna have a BigQuery Sync connector that's gonna come out in preview. Uh I think that's gonna be super useful because BigQuery is widely used uh in the analytics world. Uh we see a lot of demand for it. And then we'll also have our first set of uh source connectors that will come out uh somewhere between now and uh December uh in preview. Um we are thinking about uh uh getting Postgres source, MySQL source, Oracle source, SQL Server source. Um the initial source connectors that we release are gonna be based out of uh JDBC protocol, meaning like they're gonna go talk to these tables directly. Um uh so that's those are the four sources that we will release first. Uh at some point we also want to do CDC-based uh sources for these databases, but that's not coming immediately. Uh we also are gonna get uh a Kinesis source and a Google PopSUp source. Um so these are some of the connectors that are going to uh come pretty soon uh in the next few months.

SPEAKER_01 19:51

Awesome. What else are you excited about for the future of Kinect in the cloud?

SPEAKER_00 19:56

Um so for Kinect uh in the cloud, uh at some point uh we want to make this experience really seamless and give as much feedback as possible to the user. Uh today, if you look at it, like if the metrics that we show them is just uh point-in-time metric in terms of like the number of messages, the lag, and all of that. Uh, we want to give some kind of time series information to them. Uh so we are actively doing some work around that. Uh so that's one thing. And then uh we also uh actually are looking at uh getting all the connectors that we are building, like we have about like 80 odd connectors that we have built today. Uh we want to make take as much as possible and get them all into cloud and then operationalize it. Um that I think is gonna be like super critical. And um this is uh me being an engineer here. At some point that's why you're here. At some point, right? Like uh this is a really hard problem to solve. At some point, I really want users not to even think about tasks. Yeah, yeah. That but that that's a really hard problem to solve. Uh, but I I I think we'll get there, but I just can't commit to a timeline there. What what is hard about that? Um so I think um with a sync connector, uh it's relatively not that hard because with the sync connector, you have some metrics about the lag from the Kafka cluster. Uh it's super hard for a source connector. Uh for the source connector, there's no good way to go look at the lag. Uh so that's uh biggest challenging problem. Uh we thought about this, we thought like, why don't why not just do for sync? And then we thought like it might be easier for the users to just configure tasks for both source and sync to start with, uh, and then uh take go from there based on like feedback. Um so the real challenging thing is gonna be in the source connectors, unless we have some kind of like feedback loop or if we could figure out something in the framework uh that uh will help us go find out like what the lag is, uh that's gonna be super challenging to solve. And in general, like uh what other things like as we operationalize uh Kinect, uh the one thing that we will see is like we'll also be like contributing a lot back to like Apache Kafka in in terms of framework improvements, uh, because we will get to see them uh for ourselves, and then that will help us make the Kinect framework a lot better. Uh I think uh that's gonna be exciting for everyone, irrespective if you're using cloud or not.

SPEAKER_01 22:55

Yeah, just uh heavy use of Kinect in production. Yep. Because we're you know historically uh kind of a platform vendor, you know, so we'll run tests and you know, we'll know Kinect as engineers and integrate it with parts of Conflow platform and blah, blah, blah, but that's not the same as running it at scale in the cloud uh in giant Kubernetes clusters. So yeah, hopefully that will lead to improvements in the framework that of course will be go back to the project.

SPEAKER_00 23:24

Yes, the feedback loop is just going to be much quicker with this. Excellent.

SPEAKER_01 23:28

My guest today has been Magesh Nandakumar. Magesh, thanks for being a part of Streaming Audio.

SPEAKER_00 23:33

Thanks a lot, Tim. It was great uh talking to you about uh the connectors that we have in cloud. Uh I just hope that uh we get to talk a lot more about this sometime in the future when we have more connectors.

SPEAKER_01 23:46

And there you have it. Hey, it's Kafka Summit time again, and you get another discount code for listening all the way to the end. Kafka Summit London is coming up on April 27th and 28th of 2020, and you can get 30% off your registration if you go to Kafka-summit.org and use the discount code KSL20Audio during checkout. Just enter KSL20 Audio while registering at Kafka-Summit.org, and that 30% off is all yours. I would love to see you there. And anyway, I hope this podcast was helpful to you. If you want to discuss the podcast or ask a question, you can reach out to me at at TL Burgland on Twitter. That's at T L B-E-R-G-L-U-N-D, or you can leave a comment on a YouTube video or reach out to us in Community Slack. There's a Slack sign-up link in the show notes if you want to join that group. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. If you subscribe through iTunes, be sure to leave us a review there that helps other people discover the podcast, which we think is a good thing. Thanks for your support, and we'll see you next time.