Connecting Apache Cassandra to Apache Kafka with Jeff Carpenter from DataStax

Whenever you see an Apache Cassandra™ in the wild, you probably also see an Apache Kafka®️. In this episode, Tim Berglund (Senior Director of Developer Experience, Confluent) and Jeff Carpenter (Director of Developer Advocacy, DataStax) discuss the best way to get those systems talking using the DataStax Apache Kafka Connector and build a real-time data pipeline.

EPISODE LINKS

• About the DataStax Apache Kafka Connector
• DataStax Academy: DataStax Apache Kafka Connector Course
• Join the Confluent Community Slack

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_00 0:00

Whenever you see an Apache Cassandra in the wild, you probably also see an Apache Kafka. I sat down with Jeff Carpenter, the Director of Developer Advocacy at Datastax, to talk about the best way to get those two systems talking. We go over this and more on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back, everyone. I have with me today in the virtual studio my friend and sometime colleague, Jeff Carpenter. Jeff, welcome to Streaming Audio.

SPEAKER_01 0:38

Hey, Tim, thank you. It's great to be here.

SPEAKER_00 0:41

Always great to talk to you. Um now uh you and I are in fairly similar lines of work. Uh you and I know that, but maybe nobody else knows that. What is it that you do?

SPEAKER_01 0:54

Ah, great. Uh, so I lead the developer advocate team at Data Sachs. So the title is uh Director of Developer Advocacy, which uh we we used to be called technical evangelists and succeeded in getting that renamed, but we are part of that uh developer relations world. That's our game. So we're all about developers.

SPEAKER_00 1:15

Yes, I love it. It's my game too. And um that's super important work. So basically, your team are the the front line of the people who travel and speak at meetups and speak at conferences and meet with customers and things like that, and just help people be successful in your case with Cassandra, Apache Cassandra and Data Stacks Enterprise.

SPEAKER_01 1:35

Absolutely. Yeah, and we love that term of being the zeroth user of our products. You know, we get to we get our hands on stuff and get to hammer on it before anyone else. And uh, do we always succeed in eliminating any uh UX pain, any developer experience pain? No, we do not, but uh, but we do our best to try to eliminate as much of that before it goes out the door.

SPEAKER_00 1:57

So I totally get that. And that's a part of the mission my team has. There's there's a couple of teams that that do that work. We call it developer zero, like uh patient patient zero, you know. Um and it's important because we are, you know, these teams are technical people. They're people with engineering background, they can write code, they can build stuff, they've built stuff before, but they're not the engineers who have built the product. Um, and that's that's important. The engineers who have built the product are uh, you know, the best people in the world to build it and the worst people in the world to comment on whether it's pleasant to use. That's right. Because they know it all.

SPEAKER_01 2:36

You know, oh, can I just deep dive on this for one second? So I get passionate about this one just because I know we want to get uh talked to some technical content, but uh, I just um I feel this because you you you and I are both working on technologies that are large-scale distributed systems, which everyone knows traditionally not the lowest barrier to entry, right? I mean, we've uh we've succeeded for a long time in uh making difficult, powerful technology um sound difficult. So like this is mission, this is job one in life right now, is making it easy. And we we are looking at things like where uh you know where are uppercoming up-and-developers in the world. Um it's getting younger and it's not so US specific, you know, worldwide developer community, and you know, the the the boat needs to be as large as possible, please, you know, to get people on and using this technology. They need to be able to come to speed on it quickly. So anyway, this is the passion.

SPEAKER_00 3:36

Uh, you know, sorry for the deep dive diversion, but no, no, that's that's uh just super important stuff. And uh that's why teams like ours exist. That's right. That's good stuff. Now, um so some context here for everyone. Uh Datastax is a company that makes uh a horizontally scalable uh cloud-friendly uh database that's based on Apache Cassandra. Is that a fair statement, Jeff?

SPEAKER_01 4:06

Yes, that is correct.

SPEAKER_00 4:08

Confluent, if you're a listener to this podcast, you probably know Confluent is a company that makes an event streaming platform based on Apache Kafka. So uh very much uh cousins in terms of model and space that we occupy, if not actually siblings. You know, there's a lot of similarities between our two countries, companies, based uh both based on an Apache open source project and uh um you know scalable data infrastructure. And uh I mean I think uh reasonably different kinds of data infrastructure, uh but but uh really, really similar spaces that we inhabit. And the two companies happen to be partners as well. Uh now for most of you listening, like the details of that kind of thing are probably not super important, but we are, and uh we have you know lots of the same types of people that we uh talk to and help, and like if if my team is talking to somebody or building a demo for somebody, uh very likely Jeff's team is also talking to that same group of people, that same group of developers, and helping them with something. So there's just a tremendous amount of overlap.

SPEAKER_01 5:22

Uh definitely. We definitely show up in a lot of the same places. Yes.

SPEAKER_00 5:26

And there's been uh reasonably lately an interesting development that kind of reflects that partnership and that um I need to come up with a new word because I the the word I I always want to use when you know, like you know, we show up in the same places, the word that always comes to mind is comorbidity, which is bad. That's a medical term for like you have this disease, you probably also have this other disease. And you know, Cassandra and and Kafka aren't diseases, they're like really good things. I need a new word. Dang it. I like that word.

SPEAKER_01 6:00

Well, from the standpoint of virality, I like it. Um perspective of mortality, not so much.

SPEAKER_00 6:05

Not so much, or like is this gonna you know disable you or something? No, it's not. Okay, they're good. Okay, we'll get to work on that. We need to workshop that. Let's do that after uh the podcast. But um yeah, so um they uh we show up in the same places, and people who use one often use the other. And so that brings to light the need to get data between systems. Um maybe you know you have events streaming into Kafka and you want to store them into Cassandra, or you know, streaming through Confluent Enterprise and you want to store them in Datastax Enterprise to use the the names of the commercial products, right? Which brings up the connector.

SPEAKER_01 6:46

So the connector.

SPEAKER_00 6:48

Yeah, tell me about this.

SPEAKER_01 6:50

Uh so it's actually been out since December of uh essentially free. Yeah, so it's like ahead of internet time, that's years. Yeah. Uh but it's you know, uh to um, you know, all joking aside, relatively new product. So uh DataStacks Kafka Connector, it plugs into the uh Kafka Connect framework. So uh, you know, you know, I think one of the things that's really nice about this is for customers that are using Datastax Enterprise, there is now, you know, an officially supported uh so this does work with um the various flavors of DataStacks Enterprise. So that would that would include uh the data stacks distribution of Apache Cassandra. So that's just our packaging of the open source version uh for which we provide support, etc. Um, and then also our data stacks enterprise distributions. So this is the basic distribution, which just has the our version of Cassandra and Data Stacks Enterprise, which has the full integration with uh Spark for Analytics and our graph database that we build on top of Cassandra. So that's all in that DataSecs Enterprise full version. So the connector works with all three of those distributions that we offer. Um and you know, I don't know to what extent we you want me to kind of like rehash the whole architecture of the Kafka Connect framework. I assume that your audience is fairly familiar with it.

SPEAKER_00 8:16

Fairly. So I don't want to talk about it in too much detail. I mean, I uh many listeners probably understand it, but there are always new people. Um and so without without boring the old hands, um let's let's talk about it a little bit.

SPEAKER_01 8:30

Yeah, sure. So I mean essentially uh it gets deployed on the various uh worker nodes that you have in your Kafka Connect, and its responsibility is to uh subscribe to topics, to Kafka topics. Um, it could be one or more topics, and perform transformations um according to definitions specified in a mapping file. Uh, and then it writes out from there to one or more tables in DataStacks Enterprise. So um it's you know it's not just a direct write one topic to one table. There is an end-to-end, you can get you know somewhat more complex in that. Uh, and you know, part of the reason that that is important is not only the ability to kind of join data from different sources, which is one of the major selling points of Kafka, you know, as a as a streaming platform, but also uh to be able to write out to multiple tables within Data Stacks Enterprise or within DataStacks distributions of Cassandra is uh because we support a denormalization, the kind of we we advocate for that as a good data modeling principle. Um, this is true in throughout the Cassandra world, right? Um rather than uh being a traditional relational database where you can just do whatever joins you want or put slap an index on whatever you want, uh that's really not the mode of operation. So, you know, the one of the virtues of the connector is that it is promoting a mapping that embodies best practices of denormalization that we do uh in the Cassandra world.

SPEAKER_00 10:08

Got it. And so that connector, you can actually define those denormalizations inside the connector.

SPEAKER_01 10:14

That's correct.

SPEAKER_00 10:14

Yeah. Nice.

SPEAKER_01 10:16

So um let me just And I actually know to what extent um being not as familiar in the Kafka world, I don't know the extent to which you talk about denormalization as a as a virtue or as a recommended data modeling practice.

SPEAKER_00 10:28

Uh yeah, I use that language frequently, actually. Um not many people do, and that's probably my Cassandra accent showing. Yeah, it could be right since I've been involved with the Cassandra community in the past. Um, I do use that word, and I I I wanna I want to get into that in a minute, like exactly what that means on both sides of the connector. But just to re uh to repeat what you said, or re uh so to summarize, that's the word, summarize what you said about um Connect. So Connect, you got a Kafka cluster and you have a Cassandra cluster or a Data Stacks cluster. And um we'll just say Kafka and Cassandra. And you have messages in topics in Kafka, and you want them to be in some other system. So Kinect is this uh well software. You know, it's a it's a client that you run on other hardware, and it uh it it does that reading from the topic and writing to the external system, or it can read from an external system and write to a topic. So it's it's uh a conceptually very simple framework. You know, the devil's in the details, so the complexity is all hidden inside it, but this conceptually simple framework that lets you do those integrations without writing the integration code yourself. Because it's you it's pretty low-value code, right? Connect is enormously valuable as an integration technology, but you as business software developer trying to develop features for your business, if you go write that integration code, like that's not a feature that your customers are. I want to have to go instantiate a driver, and then I have to maintain connections and do I have program retries and error recovery and you know, oh, I it needs to be able to scale out and it needs to keep track of the last offset it read, and like it just no, you don't that's not you're not about that life. So connect is and it has a a library, an extensive library of connectors. And a connector, just to make it brutally simple, is a jar file, right? It is a it is a a program of the JVM um that is written against a fairly simple API, and then you drop that connector into your connect cluster, and then there's a REST interface by which you're able to configure. You referred to all these configurations, you throw pieces of JSON at the connect you know, runtime configuration interface, and that instantiates the connectors, and then they run. And you described that as what we call a sync connector, and that's S-I-N-K uh sync connector. Reading from Kafka writing into Cassandra. Does the connector go the other way?

SPEAKER_01 13:15

Can I source uh not at this time? That's uh I don't have information uh for you on when that might be available as a feature. Um, it's certainly uh becoming a more feasible thing to do, given the fact that uh Cassandra 4.0 has uh a more fully fledged change data capture that's coming online. So, you know, we've we are uh tracking that. We're tracking the 4.0 release of of the open source Cassandra and obviously incorporating that um into our DataSecs Enterprise version. So at some let's say it's at some future point in time, I could see that being possible. Got it without guaranteeing anything.

SPEAKER_00 14:00

Current mode of the connector is uh sync. So it's that's right. In Kafka terms, it's a consumer. It's reading from a topic, and when a message shows up in the topic, it gets it into Cassandra. And uh there is so prior to this data stacks connector, there had been a community sourced Cassandra connector. Um but this one is, I think, in in my judgment, in everybody's judgment that I've talked to, um it's a richer thing, right? It's it's just a more sophisticated, like a little bit Cassandra smarter way of being a connector.

SPEAKER_01 14:37

Right. And not to be snobbish about it, but basically, if you think about it this way, the team that built the uh DataSachs Java driver is the team that implemented this connector. So these are the guys with the know-how. I mean, they know how the driver works, they're intimately familiar with the low-level details. Um it's you know, it is based on uh, you know, newer. We basically we've uh also recently, as of March 2019, released uh new versions of the Datasax Java driver, which are essentially a like a complete reworking of a number of things under the hood. So you're also building on top of that that richness as well.

SPEAKER_00 15:16

Okay, cool. And that's yeah, those are the right people to do that work. I think nobody nobody has any doubts about that. Right. Uh cool. So um, all right, so uh sync connector reads from Kafka, writes to Cassandra, and let's talk about denormalization. You mentioned that, and and yeah, so yeah, in the Cassandra world, you've got it folks, if you don't know Cassandra, uh horizontally scalable database, and it it has a tabular data model, so it it and uh a SQL-like uh query language for that, those tables. Right. And so it presents as something familiar from a data model perspective, but there are certain limitations it imposes up front on uh data model.

SPEAKER_01 16:03

Without doing like a total uh data modeling tutorial here, the you know, the key innovation of data modeling in the Cassandra world is this idea of uh a partition as a grouping of related rows. And your entire objective in creating uh a data model in Cassandra is uh designing a table so that you have partitions and rows within those partitions, and of course, columns within those rows, but you're trying to design that in such a way that you can uh a read be from one partition. That's your whole goal, which really means that you're going to uh a small, the kind of the smallest possible number of nodes within a cluster in order to service that query. Um, so that's your entire goal. So for that reason, uh the part the elements of what's called your partition key are key in your access pattern. You design you're designing your primary key in Cassandra around your access pattern so that your reads will be as fast as possible. And this is how Cassandra achieves that massive scale out that allows it to still be an operational database, an OLTP database. So that's that's your objective in that data modeling. So for that reason, if you uh have cases where you need to access your data uh in different, let's say, use cases or different access patterns with slightly different bits of information that are your key for looking up the data, um, that's when you're encouraged to denormalize, which is to create different tables which are organized around the different ways in which you want to look up the data.

SPEAKER_00 17:44

Very, very well put. So yeah, you uh that's that's the whole, you know, we're we're we're trained in relational database world that uh denormalizing is bad. Um but if you the the reality is if you want to scale out, you're gonna do it. Like it it's you need a solution for making that happen. Denormalizing is bad when you're able to maintain the the requirements of of relational orthodoxy, and and really that it that's another way of saying when you're not operating at scale, when you have a single server database, all of those simplifying assumptions can hold and they're good things, right? A normalized schema is a good thing. It's just in reality, uh for the most part, it's it's not a scaled thing, um, at least you know from a Cassandra perspective. And so Cassandra makes those changes in the data model because basically it's saying, look, you're gonna scale. Uh the only reason you use this database is because you need to go big. And uh, you know, let's just get all of your home. Go big or go home. That's right. Let's get those data model compromises out of the way at the beginning. And you get to reshape and you just describe briefly that that data modeling process where you think about how you're gonna query the data when you're designing the schema, get that done up front. Um and that way you're not disappointed later when you when you go to scale and and all these things fall apart.

SPEAKER_01 19:12

So right, that's right. And I think um that those those concepts are getting more and more uh to to be well accepted concepts. This the ideas of replication and that you're gonna have multiple copies of your data on multiple nodes and multiple data centers, you know, these are these are concepts that Cassandra and Kafka have in common. Um the ways, the specific details and ways in which um they choose to implement that and some of the controls that they give you are slightly different. But at you know, at a high level, at a 10,000-foot level, they're very similar concepts in this idea of you know partitioning and replication, and these are used as mechanisms to help us scale out to be able to do things effectively at scale.

SPEAKER_00 19:55

Exactly. And um I so you you asked if I ever, you know, if in the Kafka community, if we ever talk about denormalization, and I said, I do. I I think it's just me, but it makes a lot of sense because we we have a similar thing where key matters a lot. So in Kafka topics, messages are key value pairs. So picking the key, you know, which is going to correspond to some state of affairs in the business, like an order ID or a user ID or whatever, you know, picking that key uh determines how messages in a topic get partitioned, which in turn determines what ordering guarantee I have in a topic. So messages of the same key are always going to be in order. So uh key again surfaces as this important data modeling concern.

SPEAKER_01 20:42

And what happens it's funny that you say that.

SPEAKER_00 20:45

It is, yeah. No, it's the same.

SPEAKER_01 20:47

When I first started using Kafka, that's like my my naive um mistake zero or or or my for my first mistake with Kafka was along those lines where I actually was uh inserting only values, which is a legal thing to do apparently, but it's not it's kind of like if if you have somebody that's coming from a background of a messaging bus, or just kind of using Kafka as a messaging bus, like I'm just gonna swap that in in place of rabbit and q or something, like they might have that sort of naive assumption that I'm just throwing values in there. Uh, and that's you know, I think there's a richness to that you lose. Um, because again, yeah, I wasn't doing any partitioning, I didn't have any key that I was that you know, I kind of omitted that entire part of my data model from a Kafka perspective.

SPEAKER_00 21:35

Right. And when then later on, you know, you you need an ordering guarantee, you don't have one because you know Kafka will round Robin. If you if you partition the topic, it's still it'll still make sure things get distributed, but not not in any way that is that is uh aligned with your application. So that happens too. And so there's there's maybe somebody in the organization who got data in a topic in That uh you said naive, I was gonna say dumb way. Nice, probably nicer. But you know, it's like it could have been me. I I do I do dumb things all the time. So you've got that, and then what do you do? Well, you uh by hook or by crook, basically consume from that topic and produce to a new topic with the key of your choice. Uh so you you know you have this other topic where, oh no, I need this to be ordered by user, and so you make that the key. But you might then, with that same data, need to also order it by region or store ID or something like that, right? That just that that might be another ordering that you have some real-time analytics process that needs that ordering to be uh you know to obtain. And so um you denormalize, is what you do. You you you create duplicate data and you give you use the new key in that new topic, and that's a new topic. Uh, with so that's that's kind of Kafka denormalizing, and it's again a perfectly fine thing to do.

SPEAKER_01 23:00

That's kind of a cool pattern. I don't know if I'm reading too much into what you just said, but it sort of sounds like there's an idea of uh a pattern there where your your application is only concerned with writing to one topic that kind of reflects its concerns, and then maybe you actually configure some streams that are responsible for transformations to uh to other formats or other topics. Is that kind of the pattern?

SPEAKER_00 23:22

That is, yes, exactly. And that's an ideal uh case case right there. That's a that's a one-liner in case where you select from the initial stream um and insert into another stream with the new um, or you you can have to insert into, you can just create, you know, create a new stream as a select from another stream, uh partitioning by the key of your choice. So that you know, that function basically, that that streaming query will run forever and keep the other topics up to date. So totally. And the nice thing is that all the messages are immutable, so there's another layer of guilt removed for that denormalization. You know, you're making copies of immutable things, so it's fine. Which kind of so yes, we denormalize. Um, and we don't call it that, but we do it, it's exactly the same thing, and it's a perfectly fine thing. The only cost is, you know, we say storage is cheap, and that's kind of a uh uh I don't know, it's not even a truism because it's not true. It's a it's a canard. It's a right, it's a a not necessarily true statement we like to make. But you know, then you're provisioning your cluster and you're like, oh, I have eight copies of this data, and you know, the KSQL query needs to run, and that costs compute. So there are costs, you don't want to be stupid about it, but um there's no data integrity risk.

SPEAKER_01 24:43

Yeah, that's one of those uh that's one of those statements that kind of gotten has gotten um truncated from I think what the original thought behind it, which was that storage is less expensive than compute, let's say, for example, right? And if there's architectural trade-offs that you're making, and that somehow got condensed down into storage is cheap.

SPEAKER_00 25:02

Yeah. I think it's relatively cheap. It's relatively cheap. I can I can plot uh on a log linear graph, you know, this super impressive thing of of the cost of storage over the last 50 years, and it'll be nifty, but it you know, then there's a system and operational costs, and uh, you know, like don't be crazy. Uh but the point is both of us uh are are you know have data infrastructure technologies that encourage the practice of denormalization and um help get around the historical problems with denormalization, which were never about cost, they were always about data integrity. Now, this gets us back to the connector. So I'm going somewhere here.

SPEAKER_01 25:52

That's good because I felt like I was supposed to argue with you on that last point and I missed my cue.

SPEAKER_00 25:56

No, no, no, no. I wasn't I was not setting you up. Um I it's uh it's yeah, and and you know, when you are are are doing public developer advocacy and talking about denormalization, you still do have to sort of apologize for the fact and explain why. Yes, I know you think this is immoral, it's not, and here's why. You know, you kind of have to make that argument. But um you said something about the connector, which I actually didn't know, so let me make sure I heard you correctly. But let's say I'm consuming from one Kafka topic, and there's this Kafka topic, and I want to write that to five Cassandra tables. That that is a thing I can do?

SPEAKER_01 26:35

Yes.

SPEAKER_00 26:35

Do you know? Can you tell me about what sorts of transformations can happen in there? Hmm.

SPEAKER_01 26:42

Well, okay.

SPEAKER_00 26:43

So that's I I I'm conscious of the fact that that's rather a deep dive question that you could be forgiven for not knowing.

SPEAKER_01 26:49

So um I don't I'm not aware. Uh okay, I need to dig more, but I'm not aware that there's like filtering or uh transformation logic. Um there is a what I'm aware of is that it performs a mapping of field to field. So you name the source field and then name the destination field, for example.

SPEAKER_00 27:08

Got it. And they those could potentially be different for the different tables. And in any connector, you have a thing called single message transforms that you can define.

SPEAKER_01 27:19

Um and those so to the extent that that's a feature of the uh Kinect framework itself.

SPEAKER_00 27:27

Yeah, yeah, at least to that extent. And I I think there's a uh you know, a feature of the connector that goes beyond SMTs, but single message transforms are these little things that you can plug in, and there's a library that exists by default in Connect, and you can write custom ones, but they're just for simple things like you know, let me mask off this field, or here's a column in the table that I want to extract and turn into the key, and you know, just make little little manipulations of schema as messages come from Kafka into Cassandra or into whatever system they're going into. You can make small tweaks. Now, there's this temptation to get carried away with SMTs and like try to do ETL with them, and that's always a bad idea. Um they are single message transforms, so they're stateless and they operate on one message, and they can do just the usual suspect kind of things that you might want to do as you go from one system to the other. But here's where I'm going with this. Um this is a super cool integration because if you've got multiple schemas that you need to create, and this is totally normal, there's one some source of events out in Kafka, and you need to get the events into Cassandra, um and you explained the data model to us and you emphasized that you build tables according to access pattern. In other words, you you model for queries because you get one key, and that one key is efficient, and all of the lookups are not efficient, and so you you you build the schema around the key that you need for the query. Well, what if I had one topic and there's five queries I need to do on that data? Um, now get Kafka out of the picture, get the connector out of the picture, like you know what you have to do. And historically, this has been a question that, and I realize there are various ways to solve this within Cassandra, but it's always been a question of how do you guarantee atomicity in there, right?

SPEAKER_01 29:31

Oh, you were gonna ask it. Uh okay.

SPEAKER_00 29:34

Well, it's okay. This we have a solution now. I mean, you've got materialized views, right? Those are one way uh to guarantee atomicity. So you write as a Cassandra thing. Do you want to tell us about materialized views? Is that is that too off-topic?

SPEAKER_01 29:49

Uh it's fine, yeah. I mean, the basically the essentially is that Cassandra is maintaining you you are writing to a base table from your application, and then Cassandra on your behalf is maintaining uh these materialized views, which you specify as uh views off of a base table. And there's certain limitations um that you know, in in the syntax about what you can and can't do, that it's probably not worth getting into there. But let's say that used uh you know used well, um, they are they are something that can be an effective tool. Now, I know there's been some discussion in the Cassander community about whether or not that should be considered um an experimental feature, quote unquote, because of some scalability challenges and some edge certain edge cases. Um, I know that a lot of work has been put into making them more reliable. Um, so a lot of that um controversy, let's say, has seemed seemed to have died down. Because that was like that was one of the major uh driving features of the Cassandra 3.0 release was getting that materialized view capability in. Um, so that seems to have matured quite a bit uh in the in the more recent releases.

SPEAKER_00 31:01

Yeah, it was a I mean it was a big thing, and it like any big thing takes a while to get ready.

SPEAKER_01 31:05

Really required a rewrite of the underlying storage engine and lots and lots of low-level technical details.

SPEAKER_00 31:12

Right. And it was one way of when you know, when I need I have this this entity, but I have four different queries I need to do on that entity, therefore I need four different tables. It was one way, a possible way of defining those four tables with asterisk, you know, see footnotes, blah, blah, blah. You know, there's details that we won't get into.

SPEAKER_01 31:32

And the attractiveness of that is that you know it's writing to one table and then letting Cassandra handle the details of the organization for you. There also, of course, and has been for a long time, the batches uh that Cassandra supports, which you can use to group rights to multiple tables. Um, and that you know, again, that is not uh, let's say, full atomicity if you are an adherent uh or looking for full asset guarantees. Um sometimes they refer to them as atomic batches in Cassandra, but that's somewhat of a misnomer if you're a purist, let's say. Right, right. Um, but it but what it is saying is that you're gonna group these multiple rights together, and that if any of those rights first of all, it's gonna check like is all of this syntactically correct? Yes, okay. Then uh Cassandra is gonna say if any of those rights succeeds, it's going to commit to making sure that all of those rights succeed over time.

SPEAKER_00 32:29

So even if that means it has to store hints or etc., maybe too deep for now, but the coordinator who took that right will retry the right until it succeeds.

SPEAKER_01 32:39

So that's what I uh that's what I'm most uh accustomed to using actually in writing applications with Cassandra is doing grouping those multiple rights using batches. And then there is an additional level of guarantee that you can get uh on a single partition with something called a lightweight transaction. So sometimes people will combine those constructs, uh, group rights to multiple tables in a batch along with uh a single lightweight transaction statement as part of that batch, which is used to guarantee uniqueness with an if not exists kind of syntax. So, in other words, you know, making sure that you're not overwriting old data, um, but being able to uh have a uniqueness check that you run on one table and then group additional rights to additional peer tables, materialized or uh additional denormalized tables, let's say. And again, so the Cassandra does give you this uh toolbox of things that give you fairly fine-grained control over uh some of these coordinating rights to multiple tables.

SPEAKER_00 33:43

Yeah, yeah, because it's it's been a it's you know, it's a thorny problem, and it's always a thorny problem in a distributed system. It's not like uh it was a huge limitation in Cassandra, it's just that, well, you know, you've decided to put your data pay your database uh across many computers which don't share a single clock, and so you know, life is terrible. That's just what distributed systems are like. But I I think, and you know, argue with us on Twitter or in the comments of wherever you're getting this, if you think I'm wrong, but let's just kind of think through this. Um I I think there's uh an interesting atomicity solution with the connector. So imagine this is only going to apply in some apply in some cases, right? Um, but if you've got data that's starting, events that are starting in uh Kafka topic, and you are using the Datastax connector to uh write them into tables in Cassandra, then regardless of of how you do that configuring, if you can do the splitting inside the connector config, or you just have to deploy multiple instances of the connector, it doesn't matter. You can connect those events uh into the Cassandra tables and do the transformation. You know, since that schema mapping is a thing you can do in the connector, you get to do that uh that basically do the denormalization in the connector. So the event starts life in a topic, the connector makes the appropriate schema transformation and writes it to the Cassandra table, and it does that say five times, because I have these five different queries that I need to do on the same event. Um now um those Cassandra tables are not all going to be updated at the same time. So, system wide, this is eventually consistent. Somebody has produced an event to Kafka, and then real quick, uh, but not at the same time, those five tables get updated or inserted.

SPEAKER_01 35:46

Um then but I got system-wide eventual consistency.

SPEAKER_00 35:52

Yeah, it is it is as a system, and that's that you know that's kind of where these properties I think are most interesting. We fixate on you know the and and it's appropriate in terms of like uh you know, Cassandra's consistency model and Kafka's consistency model. You have to understand how clients and brokers and clients and nodes are interacting and everything. And that's all cool. It's a little deep divey for what most application developers need to worry about, or most architects need to worry about. As an architect, you need to think what is the behavior of the system? You know, what is the consistency of the system? What is the atomicity of this write? And so producing a message to a Kafka topic is an atomic operation. If it works, it works. If it doesn't, it doesn't. Uh, then you have this guarantee that the data stacks connector is going to consume that message and is going to write it to those five tables. If something goes wrong, like nodes fail such that you can't meet your consistency requirements on one of those tables. That would be weird. But let's just say you're you know you're you're strongly consistent and it happens. Um, that means that that connector that's writing to that table will just not do those rights for a little bit until the cluster is back up and it can start doing them. But it will remember its offset in the topic. So you're not gonna lose any of those rights. You produced the event, and that event was produced atomically. And your five tables that previously you were worried about making sure all five of those versions got written, and what if something goes wrong and it's not atomic and all that? Well, I guess it is with that connector in place there. You do have uh system-wide atomicity where you have the the guarantee eventually that each of those five denormalized versions of your entity will be written and will be queryable. So uh as they say on the internet, uh prove me wrong. Does that sound reasonable? Was it clear and does it sound like it's true?

SPEAKER_01 37:53

That is an approach that works. I think that there are probably other combinations and approaches that work for different situations, you know. Yeah, there's only some scenarios. But higher throughputs, or you know, these are the trade-offs, these are the distributed systems trade-offs that we make. Exactly.

SPEAKER_00 38:12

But that's a that's kind of taking a um what I'm hoping will become a common Kafka pattern of you know, I I've got three things I need to write to. And and those three things could be uh Data Stacks Enterprise and an S3 bucket and uh in-memory data grid or something, you know, whatever.

SPEAKER_01 38:32

I don't suppose I I just multiple tiers uh let's say of storage for different consumers.

SPEAKER_00 38:39

Yeah, yeah, yeah. And I'm like using I'm using data stacks for search and the graph recommender model, but I also need it in S3, and I also need it in my in-memory data grid because of reasons. Um you know, that that's an atomicity problem. It's the same, it's the same general form of the problem. Like I need to, I need to uh make sure all three of those rights happen or don't happen, and connect is a way to make that happen. So um also applies here. It's a way to solve that problem. With uh materialized views being Cassandra materialized views being another. Uh anyway. Yeah. I'm supposed to be interviewing you, and I just talked a lot, but you pressed a you pressed a button there, and I was all excited about it. Well, wait, you can use the connector.

SPEAKER_01 39:24

Well, sometimes we turn podcasts into design sessions. I mean, this happens. It's a good thing.

SPEAKER_00 39:29

Anyway, what else? Um what else? What else is going on in your world or the world of the connector or uh uh killer video? We need to talk about killer video. Tell our audience about killer video.

SPEAKER_01 39:42

Oh, nice. So yeah, uh, you know, you're uh you've done some work yourself on killer video back in the day, but uh this is a uh reference application that uh we use to teach people how to use uh Apache Cassandra and DSE. So it uses both features of the open source platform or a product, and then it also uses um it has a graph recommender that uses our graph product. Um, so I've been in the process over the last few months of doing a different implementation of the microservice tier. So it's set up as a microservice style application. Um I've been working on uh an uh uh re-implementation, let's say, of the microservice tier in Python. We already have implementations in Java, Node.js, C sharp. Uh so I'm doing this Python implementation, and I decided like previously, previous implementations were maybe not true uh microservice implementations in the sense that you could actually scale each microservice independently. Basically, they were all running in the same process. That's like the Java microservices were all running in one JVM, for example. Um so and we've always had ambitions of breaking those things out in separate things, but I was like, okay, I want to be able to do this with the Python uh version because eventually I want to actually be able to go serverless with this as well, breaking up the different functions within the microservices to be serverless calls. Uh so you know, for that reason, I do actually have coordination and there's some uh message passing, let's say, or message bus type capability that need to be between the different Python microservices. So I was like, um, let's do let's use Kafka. This is the right time to do this. Uh so that's that I've been through the past couple months. Um actually the Kafka integration was one of the easier parts. I kind of referred earlier to my first mistake that I made. Um, once I got over that hump, uh it was it was pretty easy to integrate. So I now have uh services that are uh killer video, the application the the purpose of this reference application is that it's a video sharing system. So you upload videos into it and then you watch them and you rate them. It's kind of like a poor man's YouTube. Uh and so basically I have services that are responsible for user management and the video catalog and for um watching videos and rating videos. And basically, these these services uh when they when things happen, they do make record of the of a new the users or the videos that are added. Uh, and they they are writing that data into uh Cassandra tables, but also they're firing off uh uh uh messages on Kafka topics, which uh I have another service that's my graph recommender. It's called the suggested video service, and it's consuming these Kafka events and doing a write into uh a data sex graph. So uh basically the the purpose of this is the the graph is the source of the recommendation. We have a graph traversal that's implemented using Gremlin, that the our query traversal language, and it's a you know, in let's say 20 steps uh of a of a graph traversal is able to produce these recommendations. Uh so I'm actually using Kafka here to help me solve um a data migration problem that I have, which is I have these Cassandra tables that are my source data, but I actually want the data to be in a graph. And yes, it's DataSecs Enterprise Graph. Uh, so it is the the backing of the graph is more Cassandra tables. Um but uh as it turns out, you know, DataSecs has been doing some work on the graph products. So the next generation graph, which we announced recently at Accelerate, can uh allow you to build graphs directly off of Cassandra tables. So I was like, long term, do I still have a need for the Kafka integration? I think I do because I have some other stuff. I want to do you from an analytics perspective to get the data into that Kafka world so that I can do some streaming analytics based on based on the that event data that I have already. So if that makes sense.

SPEAKER_00 44:13

I know I just described like three generations of my architecture to you in one massive paragraph, but no, uh but the thing about that paragraph is it was so good because that's the perfect Kafka microservices architecture, and uh it highlights that uh you know you you can make services talk through Kafka, but like you might want to do other things. You can't there's no graph database in Kafka. There's no full text search in Kafka or in Confluent Enterprise. These are just things, these are not things that we do. So like you need other things in your system, uh, you know, services that will will consume those events and make sure that they get into those other ways of reading the data. So that's a beautiful integration between uh our two technologies. I I love that killer video is going in that direction. And of course, we've had to skip a lot of the kinds of details that you'd really need to build something uh with this. So, because it's a podcast, it's hard to like read code. Where would I go, Jeff, if I wanted to learn more about the Kafka Datastax Cassandra connector? Where is there a thing? Is there a thing I can do?

SPEAKER_01 45:22

Yes. So if you go to our documentation site, docs.datastacks.com, you can get uh the full documentation for the connector. But where I recommend you start is if you go to Datastax Academy. That's academy.datastacs.com. We have a short course. You know, you know it well. Uh there's a a short course there. It's about a half an hour long on the the uh Datastax Kafka connector. So that'll walk you through downloading, uh, getting it set up, configuring the mappings from Kafka topics to DSE tables. Uh, and basically, actually, there's a great troubleshooting section at the end, which I think uh we would do well to probably have more often in training courses. Is here's what we know the things can go wrong. Uh, if you see this error message, this is probably what you did. So I think that's pretty cool, actually. Um, that's a part of that short course. So that'll get you set up and running um really fast with the connector.

SPEAKER_00 46:21

That sounds fantastic. My guest today has been Jeff Carpenter. Jeff, thanks for being a part of Streaming Audio. Thank you. Hey, you know what you get for listening to the end? A Kafka Summit discount code. Kafka Summit is coming up on September 30th and October 1st in downtown San Francisco, and you can get 30% off if you go to Kafka-summit.org and use the discount code AUDIONETIN during checkout. Just enter Audio19 while registering at Kafka-summit.org, and that 30% off is all yours. I'd love to see you there. But hey, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at TLbergland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks for your support, and we'll see you next time.