ETL and Event Streaming Explained ft. Stewart Bryson

Migrating from traditional ETL tools to an event streaming platform is a process that Stewart Bryson (CEO and founder, Red Pill Analytics), is no stranger to. In this episode, he dispels misconceptions around what “streaming ETL” means, and explains why event streaming and event-driven architectures compel us to rethink old approaches:

• Not all data is corporate data anymore
• Not all data is relational data anymore
• The cost of storing data is now negligible

Supporting modern, distributed event streaming platforms, and the shift of focus from on-premises to the cloud introduces new use cases that focus primarily on building new systems and rebuilding existing ones. 

From Kafka Connect and stack applications to the importance of tables, events, and logs, Stewart also discusses Gradle and how it’s being used at Red Pill Analytics. 

EPISODE LINKS

• Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation
• Deploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL Implementations
• Deploying Kafka Streams and KSQL with Gradle – Part 3: KSQL User-Defined Functions and Kafka Streams 
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_00 0:00

Stuart Bryson is the founder and CEO of Red Pill Analytics, and he's a guy who knows his ETL. He's been building analytics systems since the extract, transform, and load paradigm was young, and he set a front row seat to the transition from that to streaming analytics. As a result, his view into some real customer motivations for why people build these systems may surprise you. I also talked to Stuart about his work developing an open source Brigle plugin for managing large KSQL projects that have many collaborators. You can hear about it all on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. I am delighted today to be joined by my friend Stuart Bryson. Stuart, welcome to the show.

SPEAKER_01 0:55

Hey, great to great to be here, Tim. I'm a longtime listener, first-time caller. So uh really honored to be here.

SPEAKER_00 1:04

Awesome. It's great to have you. So uh you have to anybody who follows confluent things, your name could very well be familiar. You've published on the blog, you've spoken at summits. Um I certainly know of your work. Um tell us though, what do you do? Where do you work? Um what is what is your connection to this business of stream processing?

SPEAKER_01 1:28

Yeah, certainly. Uh I'm the CEO of a company called Red Pill Analytics. We're a services company founded uh about five and a half years ago. I am one of the founders. And when we sort of launched, um, we were primarily a traditional data warehouse company, right? So we were building ETL, data warehouses, dashboards, analytics. There was usually a target somewhere that was a big database of some kind. You know, in the past that was that tended to be Oracle systems. So when we founded, that's that's pretty much who we were. But you know, in the past three years, we've we've really pivoted to now where we build really more modern data architectures for our customers, either you know, net new, but more often than not, we are migrating some of those traditional customers that have come along with us for the ride. And that usually involves you know more modern distributed data platforms like Confluent platform and Kafka. And it often involves uh more entry into the cloud, less on-premises systems.

SPEAKER_00 2:39

Got it. Um, that is funny because when I get the very top-level question of, you know, why should I use Kafka? That what does Kafka do? Um, you know, what what kinds of things do you use this for? There are always two, to me at least, uh, the way I tell that story, there are two primary buckets of use cases. And one is what I guess business people call app modernization, what the rest of us call refactoring to microservices. Um, when you've got microservices, basically you kind of need uh a way to exchange messages and log those messages between the services. That's one. The other one is streaming ETL or streaming analytics. Right. Um and interestingly, those, as I like to say, those tend to kind of sneak around back and meet in the middle and end up being kind of the same thing at the end of the day, but they're separate motivations. And you are actually a company doing that second thing. It's it's what you do.

SPEAKER_01 3:36

Yeah, and it's funny, the I think the project we're probably going to talk about in just a little bit is was more of the former, actually. So where we come from is the streaming ETL. We come from traditional ETL development for data warehousing. Over the years, even before we started using Confluent Platform and Kafka, we were trying to find ways to quote unquote stream data, more, be more efficient, get data in front of customers, do the relational equivalent of events. Um, and really it's that experience that's taken us uh when we found this platform. It was like, this is exactly what we've been looking for. It's what we've been trying to do in relational databases and uh struggled at times. And then the project we're we're gonna talk about in a bit was really not even about uh data warehousing, it's more about plugging things together, not quite modern microservices, but at least it was about plugging some operational things together at the same time plugging uh some data repositories together.

SPEAKER_00 4:40

Yeah, kind of a data integration sort of problem. Exactly. I mean, that and that's that's always been um and and don't let me put words in your mouth, but that that's kind of the grunt work of traditional ETL, right? You have all these ridiculous things, and each one of them has its own uh temperamental interface, and you have to do it all.

SPEAKER_01 5:02

Right. Yeah, and the concept of decoupling, which you know is is so hot with with Kafka. Right. This I like to say disentangling because quite often it's it's more than just two things.

SPEAKER_00 5:13

There you go.

SPEAKER_01 5:15

Um, you know, just just that whole process of you know, the spaghetti architecture diagrams that you guys do, it really hits home with our customers because that's what they have, that's what they want to get off of, and that's what they've struggled with through the years.

SPEAKER_00 5:28

Disentangling is well way better. It's it's uh it is a more negative word than decoupling, right? Like we're all trained to know that coupling is bad, but entangling, well, that sounds terrible, unless it's some cool quantum thing. It's awful. So wow, I need to buy your service.

SPEAKER_01 5:41

I need to buy whatever gets me off of them.

SPEAKER_00 5:43

That's right. Sign on the dotted line. Uh I'm ready, Mr. Bryson. So good move there. But no, seriously. And that that to your uh, I guess a more um neutral point there. It's not A to B. Um the whole point of that spaghetti diagram of a typical data pipeline is that uh such an effort starts out point to point, but the uh there are many points on the left and the right by the time things are done, and it's not just match something in column A to something in column B. Those are many-to-many relationships. Exactly. Yeah, and that's uh a thing that gets simplified a lot by putting things in topics and just thinking about them as streams. Um talk to me about the net new versus refactoring. I that really stuck out when you said that. And uh just in case anybody missed that, um, I want to summarize it real quickly. Uh you said a lot of what you do is build net new things, but sometimes you are replacing an old system. So and just to just to kind of tee up at least what I'm thinking here, I think of Kafka and Confluent Platform and stream processing broadly as uh a new generation of data infrastructure technology. Okay, there was a time when relational databases were that thing, when, say, you and I were younger men or even even boys, uh, you know, that that was the new technology that was kind of elbowing mini computers and mainframes out of the way. And when you talk to people who were there building those systems, they were often, you know, no, nobody took the mainframe down and replaced it with relational databases, and those systems are still running now, you know, 30, 40 years later. But it's um the the new system that you're building, you build on the new technology. So it strikes me that, and I I kind of have this thing in this idea in my mind, that most streaming ETL systems aren't replacing the old thing, but building a new thing. So talk to me about what you see where where people are replacing old systems.

SPEAKER_01 7:52

So what I'll say first about the net new is what you tend to see is uh um a lot of startups or or at least uh new initiatives, right? And quite often those are being built from a mindset very different than than legacy systems we're built with. They're not, you know, they're they're cognizant of new paradigms, and you don't have to convince people that that paradigms need to shift. So I think you see a lot of they get the whole fact of what an event is. Um, they have already probably built microservices. So when you start talking to a company like that or or uh a group within a larger company that has plugged these kind of things together, and you start talking about streaming ETL, event-driven architectures for analytics, they get it. And there's not a lot of convincing. I think when you are refactoring a legacy system, quite often they grew up around relational databases. So all the concepts that you're talking about, you know, tearing down, quite frankly, are, you know, very relational concepts, operational data stores and staging layers, and all these kind of things that we all knew and loved when everything had to exist in a relational database. And now you have to sell the concept of event-driven architectures, you have to sell the concept of, you know, it's it's not, we're not just trying to get data uh to your dashboards faster, right? That's not what streaming ETL is. It's about really buying into the fact that data, it's that disentangling, it's that whole concept of you used to build a mapping, right? That's the term we used. You used to build a mapping, that mapping had a very distinct source and usually a single, very distinct target. And you didn't think about anything else, you know, there was no periphery. And now what we talk about with customers when we're talking about refactoring, we try to sell the value of disentangling the E, the T, and the L. We talk about being able to have organizations or teams or services use all use the same basic API to produce their data somewhere. So this team doesn't have to use this API, no one has to use different APIs. We just get everyone to kind of standardize on how we're gonna publish data. And then we get everyone to standardize about how we're gonna consume data. It's the same APIs. It's REST, if you like it, it's Java or what all the different clients that that Confluent and Kafka support. But then the real the real key here, and why I don't like to say decoupling, and why I like to say disentangling, is because we want to separate the T from it as well. And this is where you really start to get some buy-in, is when you talk about a single layer for your transformations. And that's something that most organizations that are thinking about modernizing just simply don't have. They've got the same functional logic has to get applied all over the place, multiple times, sometimes within the same tool, even. And that's when you really start to get buy-in and understanding of where the value could be.

SPEAKER_00 11:24

So the sale of the technology, if I heard you right, isn't so much the real-timeness for you. It's the decoupling and the ability to build a system where you know whatever transformation logic exists can actually exist in just one place.

SPEAKER_01 11:43

Exactly. That's it. I mean, we often go in with these legacy customers and we start talking about, you know, we're not we're not necessarily there to pitch streaming, but we're there to modernize or there's some other use case they have, and we bring streaming and event-driven architectures to the table. And quite often we hear something along the lines of, well, you know, we don't need the data in our reports any faster. Yeah, yeah, that'd be nice, but I don't see that as a major requirement. And that's when you start talking about, you know, separating the processes, being able to have a whole bunch of consumers, a whole bunch of producers. It's not one-to-one. Um, it's about um, you know, having more than just dashboards as as the consumer of data. Um, it's about having other services, it's about producing data to best of breed data stores instead of everyone saying, get it in the data warehouse, it's got to be done in SQL, it's got to be done with this tool. Now you start talking about the data really can can be produced to the data warehouse. Most of our customers are still doing that. But we don't have to square peg round whole use cases into relational data warehouses when they don't make sense. This is mobile computing and fraud detection and producing data to partners, to um customers when data is the product, right? So it's about you know making some a couple of simple investments. Uh, everybody's sort of getting on the same page about how to produce data and how to consume data. And then you have all these options for for where the data can come from and where the data can go.

SPEAKER_00 13:31

Got it. So uh the sale is more of an integration framework than something that that gets a result super fast.

SPEAKER_01 13:41

That's it. Yeah, and once they start to see that it's more than just, you know, quite often when you talk about streaming, they think you mean real time. That real time is a term that's been around for a long time. And what that used to mean was not necessarily immediate. It meant, hey, we're gonna run our batches every hour instead of every 24 hours. That and so when you initially start talking about streaming, that's what they think you mean.

SPEAKER_00 14:05

Okay. And I I wanted to ask about that. So um and you gotta remember, for for me, you know, I'm I'm a generalist. I I've read the Kimball, uh I've read Kimball uh on data warehouses, never built one in the proper sense. Um but every and apology necessary. No, right. I think that's that's uh most everybody listening, right? Um and the kind of the the origins of that framework is it's a nightly thing because your computers are all cooling off and not doing stuff at night. And so you have them do things and prepare results for people who wake up the next morning ready for you know the start of a business day. Um you know it's which which comes from a relatively less globalized economy and various technological limitations that we don't need to get into now about why you know why pre-computation and running at night and everything were all were all necessary. But over time, uh the the daily batch time has shrunk, right? And exactly. And how how far down in the conventional data warehouse architecture has anybody really gotten that? Like what's the practical lower limit? I know you're telling me that um that you know timeliness of data is not a thing that your company is not really what your customers are after, but I'm just kind of curious, where does that get to?

SPEAKER_01 15:30

Well, I mean, depends on, you know, obviously like financial data. Sometimes it doesn't make sense to deliver that midday. Um and that's how a lot of um, you know, sometimes customers will deflect with that, which is well, we can't we can't report intraday because the numbers haven't posted. And that's a very traditional way of thinking when all the sources of data were sort of financial systems, ERPs, sales-related systems, financial transactions, etc. And I'll start talking about, well, you know, when you when you run in batch, you only get the state of the source system at that time. So if a product has changed pricing five times during the day, and this was not all that uncommon with e-commerce systems, even 10 years ago, for a product price to change quite often, and you have no idea how many times that price changed before it sold. Wouldn't you? And these are the sort of the kind of just um basic use cases that you can talk about. And and what it really boiled down to, Tim, honestly, is CDC. A lot of this event or change data capture, a lot of this event-driven stuff that we were trying to do in the past before Kafka was really around trying to introduce CDC. And the and the value that that CDC and really events, because that's what CDC gives us, it gives us events and adopting an event-driven architecture. It gives us if we wanted to run uh our batch processing multiple times, let's say we wanted to run it every hour instead of every 24 hours, you pretty much had to do the same change data uh detection logic for the entire batch each and every time. So you weren't by running 24 times as opposed to one time, you were still running the same churn against the source system in most cases. So it wasn't really until the idea of events and reading events just in time and being able to run a smaller data sets because we're we're running it in micro batch and now eventually streaming, running them just as soon as they're produced in most cases. That's where you can really start to see value for uh getting things up to date. And and these are you know, it it's the it's the latency of a of a changed data um uh capture process on a on a relational database, which you know, seconds in most cases. And then also, you know, there's more and more sources of data that that aren't relational, they um they aren't corporate. And with those sources, we can stream those the the those data sets in from APIs even faster. Um, and then the final thing is really to just think about the different we call them subject areas when we talk about data warehousing, but basically the different areas you're reporting on. Some of them don't make sense in real time. And just simply don't don't update, don't, don't update your your materialized version of that data set until you want to. You're decoupled now. So uh a system like Kafka allows us to, you know, if we're gonna deliver data through something like JDBC or some sort of streaming process, we can deliver it just in time. We can purposefully delay the delivery of certain data sets or at least certain final data sets and bring the data sets that do give us value, uh the immediacy of that, as quickly as possible.

SPEAKER_00 19:15

You um you mentioned change data capture a couple times. And I just want to say as a as an aside, uh we've done a few episodes on that recently, kind of over the summer. And so if you're listening and you don't know what CDC is or what change data capture is, uh I'll have some links to those episodes in the show notes. Very briefly, it's just a way of watching, think of like watching a relational table. And when something changes in the relational table, emit a description of that change into a Kafka topic. So that's CDC in in 15 seconds. It's all you need to know, right, Stuart? That's you know, yeah.

SPEAKER_01 19:50

That's how quick it takes to enable as well, Tim.

SPEAKER_00 19:52

Um yeah, right, right. You just turn it on and it works. It's great. No, there's uh it's it is a discipline and a specialty, and there are whole projects and and um uh cons uh commercial products uh that do this. So it's it's a hard thing.

SPEAKER_01 20:07

Much easier than it used to be, though.

SPEAKER_00 20:09

Right, uh, because there are tools and there are they are tools that work. Um so um I just want to kind of backtrack through what you just said.

SPEAKER_01 20:21

It was a lot, wasn't it?

SPEAKER_00 20:22

Yeah, yeah, yeah, yeah. So uh and you mentioned data that's not relational and those being sources that have increased in number recently. Um give me some examples of what you mean there.

SPEAKER_01 20:32

You know, most systems used to be um corporate systems running on premises, um, usually in a in a commercial database of some kind with a heavy monolithic stack. And now um small companies, large companies, mid-sized companies, they're divesting their applications to a whole variety of places. Sometimes that's running still on premises, and but more often than not, the migration is toward SaaS applications, right? So primarily what I'm talking about there is SaaS applications. And these are applications that are difficult to get to in the traditional way. So you can, you know, I can't tell you how many times we've gone after Salesforce data. And in the past, using um traditional tools, it was really difficult because those tools always thought about ODBC or JDBC connections, every because everything's a database. Give me a table. Give me a table, exactly. And so, but you know, that that was and there were uh products you could buy that presented the Salesforce API as an ODBC connection, there's several, and that's what you had to do. So, but now with with solutions where we adopt a platform of which JDBC is just one element, and there's all the Different ways to connect. Kafka Connect makes this very, very simple. If there's a data source you need to connect to, there's probably a connector for it. There's probably been two connectors written while we've been talking. So there are, you know, connectors for everything. So when the first thing we do when we talk to customers, we we get a sort of an inventory of all the things they want to connect to, and we say, well, look, it's 90% non code. And that's because somebody has gone through the trouble of writing a Kafka connector for it. Other customers and and groups have used it, so it's so it's been hardened in most cases. And it's better than anything we would write with an ETL tool or a script or whatever. So I think that that idea of having a platform that isn't prejudiced in favor of a relational access, which so many of the traditional ETL tools are, even the ones that can connect to things like REST, tend to tend to try to make REST look like a table. And it's just not the right way to think about it. So I think that you know, as you know, organizations are seeing the value of cloud and and um that's a infrastructure, that's PaaS, but that's also SaaS, then more and more of the systems we pull from are outside the firewall and frankly much easier to pull from, much easier to pull from because they've been designed from the ground up to have integration be a first-class experience. Whereas some of these source systems we pulled from in the past were never meant to be connected to from outside.

SPEAKER_00 23:48

Right, right. That's that's interesting. I wonder if what we're getting at here is um is is more about the underlying data structure. Because I mean it what we'll call traditional ETL tools, uh, it's an incredibly mature ecosystem, right? You could you could even argue for as amazing as KSQL and Kafka Connect are uh traditional ETL, that that ecosystem is way more mature. There's all these graphical interfaces everywhere and point and click, and you know, you're probably thinking of cuss words right now associated with all those graphical user interfaces. But I mean the point is it's this gigantic three-decade-old ecosystem, and we're not. You know, so there's all this maturity there. And certainly there's connectors, right? I mean, everything has got a Salesforce connector. You don't write that. That that's a thing that you configure, even in traditional data warehousing. Uh I mean, they're they're certainly responding to the challenge of SaaS and to the challenge of everything is not just in a different Oracle database somewhere else on my network, uh, like in the olden days. But um I wonder if the the tension is that those things, you know, those SaaS applications, you you don't get to see the database. Okay, you're just it's never gonna happen. You don't get to connect to whatever that database backend is, uh forget it. They will expose some API and they will emit you know, what's the word? What do they emit? They emit events, right? They tell you about things. And they there may be some entity model that they expose to, but it's like fundamentally, here are these events, and I wonder if the difference isn't who can make the most connectors, because like everybody can crank those out, but putting events in tables has always been a little wrong, right? It's you can you can 100% do it, but it's just not the right data structure for that. And and that's not because tables are bad. Tables are incredibly good and incredibly important, and that's why they're in they're in Kafka streams, they're in KSQL. I mean, they they do they will not go away. Nobody wants them to go away, but the the right abstraction, kind of the right data structure to put events in is a log. And if you're trying to, through some connector or some interface, get them into what is fundamentally the wrong data structure. Um, I wonder if that's kind of a root of the friction with the the traditional model.

SPEAKER_01 26:19

I think, I think, I think you're right on. Uh, and just to take it a step further, the concept of um a subscription um that Kafka makes so easy, right? The idea that you know which records I've seen is something that we have hand-coded time and time again. There are a few tools that had some decent framework, uh reasonable frameworks for their times, but in it usually involved building. Well, I don't know if you've probably looked at Kafka Connect JDBC, right? Oh, yeah. It's all the different ways that Kafka Connect JDBC says, you know, what is my unique key? What is my timestamp key? What is my you know, all we coded that manually time and time again. And I think the idea that it is should just be built into an event, the concept of whether I've seen that and you've seen that, you you know which events I've seen and which events um my, you know, my partner services have seen. And the reason that that was always an afterthought was because there was a single source and a single target. I mean, there were multiple sources you were pulling from, but there was a final target and there was a one-to-one mapping. And nobody gave much thought to the idea of multiple consumers. And frankly, it was sort of a bad word when you would talk about, hey, we're we're gonna build another Mart over here. Because immediately the whole consolidation argument, which made sense back then, uh, storage was expensive, compute was expensive, that consolidation was a was a good thing, it was a serious thing to be considering. Now, best of breed product. Um, is Elastic better for this data use case? Do I even need a relational schema here? What is the best place to put this data for mobile um analytics and just-in-time um contextual analytics? What is the best source for that? Frankly, it's probably not the kind of data warehouses that we built in the past. So the whole concept of subscription, I know that's not what uh Kafka calls it, we call it an offset, but to sort of map it to terminology we've been using in data warehousing for years. It's the idea that you know which data I've seen. And that, you know, is the that enables you to think outside of the one data warehouse box as the single place where all analytics has to happen.

SPEAKER_00 29:01

I love the fact that you're saying this, and uh, it's because it corroborates the thing I say, and that's always nice. But when I I have a talk I give, I give a lot of talks, so I always think in those terms. But when I'm explaining Kafka from the ground up, I've got this talk where I say, like, look, here's an event. Here's this thing that happened. And at the end of the talk, we've got KSQL, you know, just kind of build it up a little layer at a time. And there's two things I always point out about Kafka that make it that are these seemingly inconsequential little architectural opinions, affectations that it has, you know, but they end up making it a completely different kind of thing from messaging systems we've seen before. And one of them is multiple consumers. So as you were talking about the subscription model, what you really were saying was uh Kafka has this ability for there to be many consumers on a topic, and each consumer has all the consumers have mutually orthogonal offsets into the topic. And like that matters so much.

SPEAKER_01 30:06

Um it's table stakes in the um in the Kafka architecture, right?

SPEAKER_00 30:10

Where dramatically affects the kinds of software systems that we build around it. And then you just you just said that in different words. So yeah. Awesome. I want to talk about Gradle. Um, but before we do, real quick, there are two terms you've used. I'm just trying to keep track, keeping track of the data e uh vocabulary, and I want you to define these before we move on. Data Mart and operational data store. Go.

SPEAKER_01 30:33

Yep. So operational data store is a is a is a piece of architecture that's really no longer required when you start looking at at modern systems. But what it was was the place where you would tend to replicate your source system data sets first, right? And it was relational because everything was relational. So imagine you've got like five or six source systems. Well, the first thing, you know, the sort of the first phase of this movement toward an ODS was well, let's turn on replication and let's make sure that we have all of those schemas uh replicated to a single place. And that will make ETL much easier.

SPEAKER_00 31:16

And it did. You you are using uh deeply oracle vocabulary here, aren't you? Yeah, I am, aren't I?

SPEAKER_01 31:21

You can see where I came from. So please go on. Yeah. So so then sort of the next step we took was where well those schemas wouldn't be exactly uh identical to the sources, we would make them insert only. So this is where events start to creep in, right? So we start to try to think about I don't care about the state of the source table. I want all the changes, I want a record for each insert, each update, each delete, right? And CDC systems can produce that. So depending on sort of what generation of operational data store you built, it was either one database with exact copies of all your sources, or it was one database with all the data from all of your sources expressed as events, at least to the best that we could in relational schemas. And so that is why the ODS is no longer really necessary if you adopt something like Kafka, because Kafka takes the place of your ODS for a lot of customers. It even takes the place of data lakes for some customers. I don't I don't want to argue, I don't want to get uh dragged down in that too much, but it can take the place. There's there's several use cases for data lakes, and one of them is being able to reload target systems, and Kafka can certainly um you know play that role, and also having a good place for your raw data, and Kafka plays that role.

SPEAKER_00 32:49

I think um we should have you back on for like Data Lake Thunderdome. You know, two data architectures enter, one data architecture leaves. Um that's definitely worthwhile. So so yeah, data understood, data lakes can't go into detail right now, but yeah.

SPEAKER_01 33:04

So that was the ODS or the operational data store, and it really served as a as a state as a staging point for a data warehouse, and that's where we built data marts. And so you can kind of think of the ODS as having sort of no opinion about the data. Um, it's just what happened. So it's the best that we could do for building raw data. And then data marts are where we apply our opinion. We transform it, we add our business logic, we um uh curate it, we we you know, we just you know imbue it with all the ways that we think about our data. But the data martyble data marts.

SPEAKER_00 33:44

The data mart still has something that looks like a normalized application schema and not a data warehouse snowflake schema, right?

SPEAKER_01 33:52

So a data mart is a data warehouse schema. So a data mart, if you yeah, if you really uh um a data mart, the data warehouse is um all the data marts taken as a whole, right? And some of them have tables that are shared across them, like customer table is used in all the different data marts.

SPEAKER_00 34:10

Let's hope.

SPEAKER_01 34:11

It's really the ODS that sort of has that normalized schema. Got it, got it.

SPEAKER_00 34:16

And ODS, so that's that's just a uh we kind of need to get this staged because we're gonna be whaling on database servers pretty hard and a lot easier if we just replicate it to here, and then we can do all of the offensive sort of queries we need to do in our extract and transform processes uh without impacting things that customers see. There you go. That's it. With that, let's talk about Gradle. Uh let's do it before you before you jump in there.

SPEAKER_01 34:48

I just want to I just want to say I've you know read both your books and uh thank you. It's they were you know um the the the literature the the Bibles at the time when you were trying to get started with Gradle.

SPEAKER_00 34:58

So I'll I'll give you that. It certainly has been a few years. I had somebody ask me a fairly meaty question about subprojects the other day, and I said, ooh, yeah. Um it's been a little bit. So uh but thank you. Thank you. They were um they were uh uh at least at some point if you wanted to write a Gradle plugin, um Gradle Beyond the Basics was where you could learn how to do that. So and you have written so Gradle for the uninitiated, if you happen to come from a not Java background, Gradle is a build tool that mostly inhabits the world of the JVM, uh not exclusively by any means, but it is uh I think mostly seen as a non-Maven, you know, alternative to Maven Java, JVM universe build tool. Um and um it seems to lean heavily on plugins. It's like a uh I've always thought of it as kind of a a platform for developing build software. Um and you've done some of that.

SPEAKER_01 35:59

That's right, yeah. So when we when we came across um when we were doing our really our first full-on big KSQL project, and it wasn't too long after uh KSQL was GA, we you know recognized that although, and and this is the same problem that SQL-based processing has had for years, so it's not nothing specific to KSQL, is how do you get your head around a whole bunch of SQL scripts taken together that express a load or a transformation or a pipeline? And you know, there there weren't any real great solutions for that. So that's why people tended, at least in the in the relational world, and at least where I come from, we tended to do that in procedural languages, right? So we needed a glue to hold SQL statements together, and that's why you'd write database procedures. Um, and that's really why they were used so often in sort of traditional tooling.

SPEAKER_00 36:56

And regular stored procedures, you mean?

SPEAKER_01 36:59

Regular stored procedures, that's it. So when we when we were uh starting to write um you know data transformations for, you know, full-on transformation between what we were doing was building all of the integration points between a new cloud SaaS application and all of the touch points that were on premises that needed to feed back and forth, it was you know bi-directional. Um, we were initially tasked from the customer with trying to do that with a traditional ETL tool. And we immediately said this is this is not the right place for this. And when we started looking at um doing KSQL, and they really liked that idea, we started uh I started trying to think of what is a you know a five or six-person team building, uh all building KSQL at the same time. What you know, what do we need to do to enable it? And the one sort of final piece that made it a little more difficult was um you know Oracle Golden Gate, which is a replication technology, one of those CDC tools that you that we talked about earlier, was delivering the data to Kafka. And it was delivering it mostly to a to a single dev environment. And so everyone needed to code against that dev environment. So it's not like we could all spin up our own containers and turn on Kafka connectors and feed data in. Um we all had to code against this environment. So what I was what you know, what we immediately found out was that the creating and delete creating and dropping of tables and streams, because of all the dependencies that existed uh between them, both within an individual pipeline and between individual pipelines, got to be a little bit messy. And you know, uh when you have a hammer, everything looks like a nail. And um, I had written a whole bunch of Gradle plugins, and it had Gradle has so much great functionality for um handling files. I mean, it's from a Java background, and I knew that it would easily um allow me to deal with a whole bunch of SQL scripts that were sitting in source control. So when we what we laid out was a complete architecture and we built a you know open source Gradle plugin for it. And we laid out you know, it handles how do you put uh how do you layer in all of your SQL scripts in a Git repository. It handles you know the developer process of just running a single command to chain all of your create table, create stream statements together, um, or just pieces of it. It really gives you the ability to execute at the directory level or or at the entire sort of uh the entire pipeline level. And also to handle things like building artifacts and publishing artifacts and pulling down the latest version of code and deploying it when it came time to deploy. And we also it also supports multiple um sort of deployment models. Uh it it supports interacting with the REST API to most of what it does is is is execute against the REST API, but it also generates a normalized version. I say normalized, uh a cleaned up, easy to follow single SQL script of all the stuff in your Git repository so that you can easily give that to the KSQL server at startup. Ah, so you've got just end-to-end handling a big team.

SPEAKER_00 40:47

Sorry about that going to uh big team writing a bunch of uh pieces of the the pipeline in KSQL, and so there's all these files in the repo, and it will give that to you all as one piece.

SPEAKER_01 41:01

Exactly. So and and it gives you and you you you sort of nest pipelines um in directories, and you're able to execute just a single end-to-end process. You can execute it any level in the directory structure you want. And if you don't pass one, then it just runs everything and it and it auto-generates the drop statements, the terminate statements, all the sort of messiness you need to rerun. What we wanted developers to be able to do is just develop. And what we found when when we first got started is they were writing in their SQL scripts, drop table or drop stream, and trying to put in terminate.

SPEAKER_00 41:40

And terminate, yep, and got to figure out the name of the thing that you're terminating. Exactly. Right.

SPEAKER_01 41:45

So what we what I definitely wanted was just create table or create stream statements. That's all I wanted our developers to write. And we wanted tooling or a framework that would manage the rest. So you just call a single gradle task, pipeline execute. Um, and then if you pass it a directory, uh then it'll just load everything recursively inside that one directory at whatever level you want. If you don't pass it, then it runs your your entire uh pipeline, or you really your entire load from start to finish and all the dependencies. And it you know, just spins it up and and and drops it again when you execute. It's got a lot of nice sort of uh command line options for you to not drop, to not um not terminate if if you want to error during instead of terminate. Um and then it also will will produce a an artifact file. Um for us, that's a zip file, uh, which includes all your SQL and that normalized structure and published. And and so when it comes time to you can put a version in your deployment script and it'll pull down either the latest or whatever version you want, which is really handy for testing if you want to test, if you want to uh you know, uh spin up an environment and test your your ETL, uh your newest version versus the the the last version or versus the version that's in production. Um, you're able to just to write um automation to do that. And we just really wanted to treat it like a Java project, honestly. Yeah.

SPEAKER_00 43:20

Um then that and that zip file is like you could put that in Artifactory or as a artifact in the GitHub repo or whatever. It it really is the you know, quote unquote compiled version of all this KSQL.

SPEAKER_01 43:33

It is for it for for as much as that makes sense, it is the artifact. And we just use Gradle's built in um integration. So you can do it to Maven, to Ivy, to S3, to Google Cloud Storage, to you know, where just file system if you're so inclined, wherever you wanted to, wherever you want to put that artifact. And because build servers know how to and understand how to pull all those different paradigms, build servers, DevOps. Servers, CI CD servers, whatever you want to call it. We were able to have developers truly just develop SQL, not worry about how all of this is going to be chained together later because they're expressing those dependencies while they code based on where they're putting things. And when it comes time to deploy, you can deploy with a REST API, and that's how developers deploy in their own environments. Or deploy with a REST API or the KSQL start script, which we generate and put that in the artifact.

SPEAKER_00 44:34

The REST API is the KSQL production, the KSQL server REST endpoint.

SPEAKER_01 44:39

Exactly.

SPEAKER_00 44:40

Exactly. This is and you talked about this at the 2019 New York Summit. Is that right?

SPEAKER_01 44:48

That's right, yeah. It was uh it was uh it was a such a fun talk. I I did the whole thing in an interactive notebook. So I don't know how familiar your users are, but I imagine quite a few of them are are familiar. Um and I had just written a nice little piece of Python magic that would call the the KSQL uh command line utility uh so that I could just walk through a notebook with you know graphing data, demonstrating the the processes, restarting, restopping, so that it was easy to follow and so that I wouldn't frankly make any uh uh mistakes while I was running the demo, but still running it live. So it really kind of puts you on rails.

SPEAKER_00 45:25

Right, and rails are nice for for live coding. Um you have blogged about this also on the confluent blog. Am I getting that right?

SPEAKER_01 45:32

I I did, absolutely. So there's a three-part series.

SPEAKER_00 45:35

Uh folks, I look summit, blog, podcast. I'm doing everything I can for you. Okay. Absolutely. Everybody, you know, Stuart has written the plug-in. We're trying. Uh it's it really sounds awesome.

SPEAKER_01 45:48

And uh, if you're not Yeah, let me just let me just put a shout out there if I could. So um I hope you'll put it in the show notes. Um yeah, what's it it's in the plugin portal, right? It's in the it's in the Gradle plugin portal, but then also it's just a uh a public GitHub repo.

SPEAKER_00 46:02

We will absolutely make sure I have both those links. They will be included. What's it called in the plugin repo or the plugin?

SPEAKER_01 46:07

It's called Gradle Confluent. And it uses uh it's not Gradle Kafka, it's Gradle Confluent because this KSQL, exactly. But it has a lot of Kafka functionality in there as well. So it's got it's got some functionality that helps you with writing Kafka streams, uh, it's got some functionality that helps you in writing uh KSQL user-defined functions. And I'd really love some people to get involved. Right now it's it's 100% me that's written it. Um open uh an issue. I know that there's some users of it now. Uh open an issue, open a pull request. Uh I would love for for there to be some um community involvement. And frankly, I think a lot of what it does, hopefully, as KSQL matures, it will be less and less necessary. Uh, some of what it does, like terminating and all that, I think you guys will probably handle at a certain point.

SPEAKER_00 47:01

Yeah, no, this is the kind of thing that has the potential to wither on the vine, but not yet. Yeah, not yet. No, it's so useful. Uh and yeah, hey, Stuart is all alone, everybody. He's all alone on this project. He made this for you. You could contribute. If you want to contribute, you should get in there. Uh we have helps you. Exactly. Help him help you. If you if you've been listening with us through the summer, we had uh Hans-Peter Grossel on talking about his um MongoDB connector. And we spent a while talking about sort of the psychology of being a sole maintainer on a on a popular open source project. And it it is a grind. So get in there with Stuart and help him out. This is a great, great project. Be awesome. All right. My guest today has been Stuart Bryson. Stuart, thanks for being a part of Streaming Audio.

SPEAKER_01 47:53

Thanks so much for having me. It was it was a it was a big uh big bunch of fun.

SPEAKER_00 47:58

And there you have it. Before I go, I want to tell you that we have a pretty cool new offer to help you get started with Confluent Cloud without you having to pay for anything. If you're a new user and you go through the regular signup process and start using Confluent Cloud, your first $50 of usage per month are free. This will last for the first three months after you sign up. So that's $50 per month of serverless Kafka for three months at no cost to you. So go to the sign-up link in the show notes. I don't want to read you the URL, and sign up now. I think the only thing I could really do more is write your code for you. And I think we can both agree that's too much to ask. So check it out and hey, let us know how you like it. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can reach out to us on Twitter at Confluent Inc. or reach out to me at TLberglund. That's T-L-B-E-R-G-L-U-N-D. Or you can hit us up in Community Slack. There's a sign-up link for that in the show notes as well. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks a lot for your support, and we'll see you next time.