Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Hi, we’re Tim Berglund, Adi Polak, and Viktor Gamov and we’re excited to bring you the Confluent Developer podcast (formerly “Streaming Audio.”) Our hand-crafted weekly episodes feature in-depth interviews with our community of software developers (actual human beings - not AI) talking about some of the most interesting challenges they’ve faced in their careers. We aim to explore the conditions that gave rise to each person’s technical hurdles, as well as how their experiences transformed their understanding and approach to building systems.
Whether you’re a seasoned open source data streaming engineer, or just someone who’s interested in learning more about Apache Kafka®, Apache Flink® and real-time data, we hope you’ll appreciate the stories, the discussion, and our effort to bring you a high-quality show worth your time.
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Making Abstract Algebra Count in the World of Event Streaming ft. Sam Ritchie
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
During his time at Twitter, Sam Ritchie (Staff Research Engineer, Google) led the development of Summingbird, a project that helped Twitter ingest and process massive amounts of data. It relieved some key pain points, saving developers at Twitter from doing work twice, as was a natural consequence of the then-current Lambda Architecture. In this episode, Sam dives teaches us some abstract algebra and explains how it has informed his attempts to make stream processing programs easy to write in a more general way.
EPISODE LINKS
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
What happens when you take the numbers out of arithmetic? And what does that have to do with stream processing? Well, a lot more than you might think. Sam Ritchie explains it all on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. I'm your host, Tim Berglund, and I am joined in the virtual studio today by Sam Ritchie. Sam is a Googler who has done honestly a lot of interesting things and is currently doing some work with machine learning. Sam, welcome to the show.
SPEAKER_00Hey Tim, thanks for having me on.
SPEAKER_02You got it. Tell us a little bit about yourself. What uh I I know this very vague introduction I gave. Uh tell us what you're doing now, how you got there.
SPEAKER_00Yeah, uh, well, I'll keep this short. It's a it's an odd road. Um so now, uh to give you the end of the story, I'm working at this kind of side lab at Google X. Um, I think relevant to this topic, uh, the slight sketch of my background is I spent a few years at Twitter uh building the kind of main product that came out of that was this project called Summingbird and many little component libraries that were part of that. Um this was, I would say, the first kind of big implementation that was public of this Lambda architecture idea that we can talk about. Um, this is kind of a generalization of the MapReduce idea that Hadoop uh is all about. So I spent a few years there, we'll talk about that. Um, and then moved on and spent a few years at Stripe building tools and infrastructure for machine learning engineers who were at Stripe uh doing fraud detection and then a number of other machine learning applications. Um that's where we really kind of leveraged uh Kafka's ability to run applications online and run applications in batch mode and just sort of remove our you know data persistence long-term problems. Um so Twitter, Stripe, a few years off in the wilds of Colorado doing my own thing. And uh yeah, here we are.
SPEAKER_02All right. Hey, that few years off in the wilds actually sounds interesting. Yeah, unfortunately off topic, but it would be great to explore sometime. Um yeah, so uh uh Twitter, Stripe, and and now uh Google. Uh tell us about Summingbird. I mean, that was a few years ago now, but what was the what were the motivations for that? And really, what kind of a thing was it?
SPEAKER_00Yeah, so Summingbird was uh an effort that I did with a colleague named Oscar Boykin. And um this was a project that okay, the motivation for this project was we both worked on the revenue team at Twitter and uh found ourselves writing many of these online, uh what I'll call like stream processing, well, stream processing jobs that uh had the shape that I think we're all familiar with in this field of some events come in at Twitter, their tweets, their clicks, their impressions, they're just sort of any event you can generate, things that you'd go store in a log. Um and so many of these little projects we would take on at work were you know, take the thing, break it into key value pairs, do the basically what MapReduce is good at. Get some sort of key and a value that's often just a number, uh, and feed it into a MapReduce system like Hadoop or Storm. And I'm sensitive to dropping these uh these buzzwords, so we'll we'll sort of unpack this as we go. Sure. But create some sort of key, have a value like a number that you care about. So this might be a uh a tweet from a particular person on a particular day is the compound key, and then the value is a count. And you feed it into a system that will add all those numbers up and let you create the backing store for a dashboard that will tell you, you know, for every hour how many tweets has this user kicked out, or for this ad, how many people have seen it per hour. Things like this. So many, many things that all had this shape of group by a key, uh extract some number, add them all up, and then present them to a database, uh, or present them to some sort of database that could inform a dashboard. Uh so Hadoop is great at this. Um, there's a lot of batch jobs that were running at Twitter that would calculate queries that looked like this every day. There's a character who comes into the story named Nathan Mars. Uh, he's an engineer who had a lot of experience from the closure world building these big batch jobs. Um, and he created a system called Storm, which lets you do effectively that same style of job, but in real time. So you could feed in one event at a time, and some sort of database that could accept real-time updates would create uh you know key value pairs for you, um, which you could then have a real-time dashboard uh you know query and present something to you. Okay, so um what Summingbird is, what Summingbird came out of was a motivation to like not have to go write the same thing twice. We just had dozens of jobs we were writing where you'd write the logic in Java to run on Hadoop, you'd write the logic in Clojure to run on Storm, and then you'd have to figure out how to make a dashboard that could look at one or the other or fuse the values together. And it was just a total mess and like honestly, just a huge waste of time.
SPEAKER_02Like really, which was kind of kind of the the classical criticism of the Lambda architecture. Like it was it was responding, I think to give it its due, let us let us honor of our our elder here. That's right. It was responding to legitimate engineering constraints at the time, that it was very difficult to build one system that did batch well and real-time well. And you know, we had the tools we had and say, okay, let's let's fine, let's do everything twice, and then people started doing everything twice, and then we have guys like you rage coding solutions that that respond to that because it really hurt.
SPEAKER_00It did hurt. It did hurt. And um, you know, I I uh well, so Summingbird was our way of like reducing the pain a little bit, uh, which you know it sounds like you did too. Like we all felt we wrote these things multiple times. Not only is it a pain because you're writing things multiple times, but you just can't do anything terribly interesting because all your mental bandwidth is consumed with uh, you know, orchestrating when the jobs are running, can the data actually match? So Oscar and I, faced with yet another one of these things, uh, sat down and and Oscar's sort of enormous uh quantum information theorist brain uh began to pulse and quiver. This was very much a sort of mentee-mentor relationship with any of these.
SPEAKER_02Him too, yeah. So strange. Seems like everybody I know fits that description.
SPEAKER_00Yeah, that's right, that's right. Um, more uh more tangents we could we could go off to. Um so okay, so what Summingbird was was just an attempt to remove uh the rage coding from our life. So you have the Lambda architecture. We were working alongside Nathan. Um the Summingbird was it ended up being um, let's see, how do I frame this well? Um give me a sec to collect this. So Summingbird is a library that we developed in Scala over a number of months that lets you write map reduced logic. So you can take a representation of some source of data. So you say, here's some type, uh, here's some um object that represents just an abstract source of tweets or impressions or something like this. And so the type system knows what this abstract thing is going to produce. You can then write map calls, filters, all the operations that really are available to the Scala sequence API or any sort of functional programming if you're familiar with these functional programming ideas of you know, map, filter, flat map, reduce. You can chain up a number of steps of functions here, and what Sunningbird does is it builds you a graph behind the scenes. So it's it's doing this sort of first compilation phase where you get a big object that represents the thing you want to compute, and the thing you want to compute, you'd like to be able to run in Lambda architecture style on both systems. Um, Scala gives you this beautiful way to sort of declare things about your types that the type system can pull in and validate as you're compiling your code. So an example of this would be okay, I can't save anything to disk that's not serializable. So I don't have to worry about that as the code author, but the compiler will just tell me, hey, you know, you've ended with a type that we don't know how to store to disk. And then somebody else can go build a library of serializations. So we had that component. We knew that we had to run in multiple places, we only wanted to write the code once. Now, the trick with the Lambda architecture that it sounds like, again, you've experienced Tim is you know, what if I fire this thing up on two systems? How do I know that I can go query one versus the other and not have any double counting or overlap? Um, so Summingbird, in addition to being this library that lets you write the logic, you can take that logic, you can execute it on Hadoop, you can execute it on Storm or on Kafka Streaming or really anything that can know, can speak this MapReduce language. Um thanks to some type guarantees that we'll talk about, when you do that and run on both systems, you get a reader that can pull data from both uh databases that you're writing to, fuse the counts together in a way where there's guaranteed no overlap between them. And so that boundary stays completely clear as you move forward. What would happen, what this, what uh the result of this at Twitter was is we'd have these jobs running in um you know, in batch mode on Hadoop on multiple years of data. Uh, and then only the last two or three hours would really exist in a real-time store. Um, so our final dashboards had, you know, as the Lambda architecture pushes, the resilience of uh Hadoop's basically you won't get an error even on years of computation, and then this real-time extra little bump that you get from a system like Storm.
SPEAKER_02And that that being um much more recent data.
SPEAKER_00Much more recent data.
SPEAKER_02Yeah. Okay.
SPEAKER_00So it uh and the special sauce is in how all of that is able to happen without you having to really know about it at all.
SPEAKER_02Um Right. Sounds like there's some interesting things going on under the covers uh to make that work.
SPEAKER_00Yes.
SPEAKER_02Okay, so in terms of uh I in a sense this is potentially a naive question, but um sometimes I ask those. It's an API that uh tries to fix the you know the fundamental designed in flaw of the Lambda architecture, which is that you have to do everything two times. Um in what sense it is is it like or unlike Beam?
SPEAKER_00Um, this will be a naive answer because I'm not I uh I don't have experience with Beam, so I don't know.
SPEAKER_01There you go. Okay. So it might be. Anyone wants to comment on uh you know, tweet us, tell us tell us things we should have known.
SPEAKER_00Yeah, tell us things we should have known. Now, this may be a problem, uh, you know, there there almost certainly are other attacks at this uh solution. The one what Summingbird is, and you know, my presentation here is a little wandering, but what Summingbird is is a very kind of odd like functional programmers' attempt at solving this by leaning as hard as possible on the type system to solve these problems for you. So let's see, I've been trying to walk up to this without without freaking anybody out. Um it's okay.
SPEAKER_01It's time. We're 13 minutes in, or maybe 10 minutes in. I I think I think it's safe. Okay. Yeah.
SPEAKER_00Um, so the question is with the Lambda architecture is like one, how do you keep the boundaries separate? Uh so the trick there uh is behind the scenes, what we do is we uh basically partition everything into uh clear batches. So every event that comes in has to have a timestamp, and we keep those things. The timestamp follows events through the system uh as you process them and split them into key value pairs and join things. And so behind the scenes, what's happening, this is sort of one of two tricks. Behind the scenes, we're being very careful about accumulating things only into clear batches of data that might be one hour wide, say, uh, where the real-time system is sort of transparently to the user storing uh key, comma, hour of aggregation. And the real-time system has a marker on every single key value pair about how far up the aggregation has happened. So when you go to query, query the offline system, you get a marker that tells you, given that in the current timestamp, what you have to fetch from the real-time store. Okay, that's sort of interesting, but not that interesting. The real sauce, the real special sauce with Summingbird was we started thinking about, okay, what restrictions do you have on what you can actually aggregate? Like numbers are fine, but numbers are boring, right? Like counting averages are a little less boring. Exponentially decaying averages are something you might want. Like that's a little less boring. And we're talking about, for some key, aggregating, for example, the exponentially weighted, exponentially decaying average of how many tweets someone's done just in the past. Lists are interesting. You might want to know, you know, what someone's entire, I mean, lists are what Twitter's all about. You know, what tweets have I put out? Uh sets are interesting. So there's all these things that you might want to aggregate for a particular key. And they all look sort of the same. They have this property that if I have two of them, I can sort of quote add them together and I get back the same thing, right? If I take two sets, I can take the union. Uh, if I have numbers, I can add them. If I have an average, well, how would you how would you do that? You'd say, okay, well, I I could track a count and I could track a sum of, say, the you know, amount of money I've seen for a particular advertiser. Um, and if I had both those things, I could just keep adding the pair up. And then when I want to finally read, I can turn that into an average. So that's fine. Um, so this the spidey sense kind of began to tingle as we were designing. And we realized that what we'd done here, what we'd found, is a system that actually wasn't about numbers at all. It was about things that had this property that if I have two of them, I can add them together and get the same thing back. And a little bit beyond that, I'd like to be able to put kind of the parentheses of that addition wherever I wanted. So if I have three things, I'd like to be able to add, say, two of them and store them in the offline database and then add the third one later. Or take the old one and then sort of add, you know, all of the real-time things together and merge those two. So does that make sense? This kind of multiple things. I want to be able to add them up, like not shuffle them, but just add them in different groups, sort of put the parentheses wherever I want.
SPEAKER_02Right. And I um so I I don't I don't want to uh I have some idea of all of the things we want to talk about today, but just listening to you, even if I didn't have some idea of what we wanted to talk about, I would say it sounds like you're kind of talking about abstract algebra. Like, you know, you said the numbers don't matter. What would I do if I took the numbers away and I have these things that are commutative or associative, you know, what so is that is that where you went?
SPEAKER_00That yeah, that that thank you for the lead. That's that's exactly where I'm going. We have this this what it look this looks like is uh the associative property. If you've got things that can add associatively, and add just means combine in this case. Actually, that's all we were leaning on, it turns out. You know, we're storing stuff in multiple databases. Uh, we want to add them together and get the same thing back out so we can rely on it on our dashboards we're building. That's not like a big interface to have to hold to. Um, so you know, software engineer mind that I have would say, okay, just well, code to the interface, right? Um, but the the extra step with this library is we said actually this interface is a particular thing called, and this was a math term that you know, math is these kind of abstract algebra, is this field of math that has these kind of freaky terms, but like everybody relax, it's gonna be fine. This is a thing called a semi-group. Like a semi-group is just a data type that has those properties.
SPEAKER_02And so why remind us of those properties again?
SPEAKER_00So the properties are if I have two things that are of some type, I can have some operation, I can basically write an interface with a plus method, and the plus method takes two of those things and returns the same type of thing. So A plus A gives you back an A.
SPEAKER_02And if those things are, let's say, um real numbers, everybody has some intuitive sense exactly, just through training as children, of what plus does. But this this plus is not, this isn't arithmetic. We're taking the numbers out and saying more generally this is a a plus thing and it does things with groups of those things at the same type.
SPEAKER_00Yeah, and you probably have an intuitive sense, honestly, for a lot more than just numbers. Like, for example, lists like we talked about. If I have two lists, I know that I can append them to each other and I'll get a new list back out. And if I append in, you know, if I keep lists in the same order and I append them in, I can append them like groups of two in parallel and append the whole thing together, that's fine. I'll get the same list out no matter what order I choose to put the parentheses. So there's a lot of things that fit this. The the sort of odd move is to tr squint and realize that these are all the same uh these all can be coded to the same interface that just has a plus method and also as a test associated with it that if I give you things, if I give you three things, I should be able to say A plus B, parentheses plus C is the same as A plus parentheses B plus C.
unknownRight.
SPEAKER_00So does that on some level this might be confusing, but on a different level, like actually it's so simple that it might just seem why are we even talking about this at all?
SPEAKER_02It that's that's what I want to encourage everybody as they're listening to your your abstract algebra tutorial here. Um there isn't a catch. There's no catch. There's no catch to what Sam just said. It is it is that thing. You know that, you learned that when you were a little kid. They taught you in school the associative property, and I remember thinking, this sounds dumb. Why are you teaching me? And I'd love math. Um maybe thinking it now. Completely pointless. No, I'm not thinking it now. But uh, you know, when when those properties are a part of like middle school math curriculum or elementary school math curriculum, um the point is not obvious at the time. But the point is about to be somewhat more useful here.
SPEAKER_00Yeah, so the point the point is not obvious at all in uh you know back in elementary school. The point here is that once you realize that you found an interface that actually people have thought about for you know 150 years, you can you can think about how you build applications in a very different way. So Summonbird can accept any system that gives you, at the end of the day, a key value pair where the value is a type that conforms to this interface. So now what you can do is you can turn around and go to uh you know, you can go to GitHub, you can go to computer science literature, you can just start mining the internet for data structures where you can code up an implementation of this interface for them. And so we have the basic ones we talked about. Um what it turns out is that there's this whole zoo of data structures that, you know, I personally would never have thought to try to use at a place like Twitter, just doing revenue dashboards, but that are just unbelievably fascinating. And okay, one example is this class of uh let's call them like sketching data structures. So these are data structures where they kind of look like the things you're familiar with. There's one called a hyperlog log, for example. Yes. Um so this is a set, sort of. It's a set that I can put things into and I can ask it a question, how many things have you seen? How many unique things have you seen before?
SPEAKER_02Have you probably seen?
SPEAKER_00Have you probably seen?
SPEAKER_02It's not straightforward to add one hyperlog log structure to another, right?
SPEAKER_00It's absolutely not straightforward. Um and it's so unstraightforward that you might not think uh if if you were trying to build an application like, for example, yeah, roughly how many people have seen uh this ad, or roughly how many followers, roughly how many people have seen this particular tweet. You might not reach for a thing like this. Um but if you're just on the hunt for things where you can write this interface, you probably would find hyperlog logs separately. You realize, okay, I can, oh, so yeah, what this thing is is you add you can add stuff to it, you can ask it approximately how many things have you seen, but it doesn't get any bigger, really, as you add more things to it. So you can set up something that's roughly, you know, uh you know, a megabyte or I forget the error tolerance you get from that, but it's something that's so close to the exact number that it's absurd.
SPEAKER_02You get you get some relationship between size of the structure, uh cardinality of the things you're probably counting, and error tolerance. And so yeah. That's right.
SPEAKER_00And you'll just never you'll just never hit the limit. If you store a set, you're gonna have to store every element to get this uh this problem.
SPEAKER_02The theory is you you you don't get to do that.
SPEAKER_00That's right.
SPEAKER_02You don't have enough places to put all the things in the set.
SPEAKER_00It's not gonna happen. Um so you go out, you mine the literature, you find the hyperlog log, you put it in the zoo, which is this other library we built called Algebird, you add an implementation of the semi group interface, and you just move on and you keep mining and mining and mining. Then you put a different Hat on, you come back and you say, okay, it's time to build stream processing applications. What could I use this for? What could I use that for? And you start to build systems that you probably never would have thought of before, and maybe you're not sure will work, except for the fact that you're leaning on this interface that's so simple, but that you've built this entire massive engine around of Summingbird. And you could take those ideas and go implement them in some other system afterward. But I think the point here is really about the kind of creative boost you get from dissociating one, as we all know, like code to interfaces, don't code to concrete data types. But two, the interfaces you pick should be ones that other people in a different field have thought about for a long, long time. And that kind of gives you this secret, this, you know, this secret weapon. Moving across fields is always a secret weapon. And pulling from math is just like a totally winning thing to do. It's awesome.
SPEAKER_02It's it's the the the best of the secret weapons because the tools tend to be powerful. And to be uh to remind me, semigroup is that interface you're talking about?
SPEAKER_00Semigroup is the interface.
SPEAKER_02It's okay.
SPEAKER_00I mean it's like it's almost designed to be confusing. It should be called like an addable.
SPEAKER_02If yes, if it if it were a Java interface, it would most certainly be called addable, but uh That's right.
SPEAKER_00That's right.
SPEAKER_02It's I don't honestly, I think for Scala code, semigroup is probably a better name, better name.
SPEAKER_00Yeah, it'll get you more, it's more it's more hipster for sure.
SPEAKER_02Um definitely.
SPEAKER_00Yeah. So okay, so we can go a few places. I can take you a little deeper down this rabbit hole. I think we've hit the we've hit the ab the gold mine of abstract algebra.
SPEAKER_02Yes, and you can go actually go a little deeper if you'd like. This is uh if there's more you want to say, I'm I'm with you in enjoying this, so don't stop on my account.
SPEAKER_00Okay, perfect. So we've set up the kind of pedestrian, you know, you're you've got you know, you you're living the matrix life, you're just coding applications that add stuff. That's fine. That's a it's a good way to make a living. And then, you know, NEO calls and you see the semi-group idea. And so you have to take the red pill. I think that's the one that takes you into the matrix.
SPEAKER_02That's the one that takes you all the way down, yeah. See how far down the radical this really goes.
SPEAKER_00So then you go deeper and you say, okay, well, what what else is there in this field? Uh and again, what this field is, what abstract algebra is, is the idea, it's this field that came out of the observation that, okay, we do there's a lot of properties we rely on when we add numbers together. What if we just broke that apart into all the different properties, again, that you probably learned in elementary school, and just proved theorems, or in the computing case, like wrote programs that only relied on one or two of those ideas, or three, and we just built up. Just don't take the whole grab bag. Okay, so semigroup gives you associativity. Associativity lets you store stuff in different databases. So lambda architecture is only possible for associative data structures. Um the next thing on the abstract algebra train is something called a monoid, um, not a monad. Uh so you know again, again, we're in a zoo of sort of scary uh buzzwords, but a monoid is a semi-group. So again, a thing you can where you can add two together and get one back out, you can move the parentheses, uh, with a zero element, so with an identity. So this just means if you have a type where there's something you can add and it won't do anything. In numbers, it's like literally the zero. Uh in lists, it's an empty list. If I append an empty list onto something, nothing happens. I get the same list back home. You know, if you have that, then you can talk about missing data. So then it's fine to say, you know, what happens if something's not in the offline store if I haven't done a backfill yet? Or say I want to query, you know, five different databases for the same value and sort of add them all together. I've been storing my things in like multiple real-time stores. What if I'm missing data from something? Well, return the zero, and you know that when I do that add thing I've implemented in the interface, nothing bad will happen. It'll just be identity. So there's this eerie thing we started to notice was there's this correspondence between these ideas from abstract algebra and some sort of system optimization that you can do. So associativity gives you lambda architecture. Missing values comes from adding the zero, which gets you the interface, then is called a monoid. And okay, so at this point, this gets you.
SPEAKER_02So a semigroup or a monoid is a semigroup with an identity operator or property.
SPEAKER_00Yes, and you'd implement this by saying, okay, I have the addable or semigroup interface. I'm just going to extend it and I'm going to add a new method. And the method is, you know, in Scala def0 or def identity, and it just returns something of the type that you're talking about.
SPEAKER_03There you go.
SPEAKER_00Um, so super simple. Again, like uh if you read the code and didn't see the word, embarrassingly easy. It's more just that you've chosen the easy thing from an ancient field full of spells. Um just like good, you know, you pick up like the ring, it's just a ring, but you put it on and you know and wow, you know. Wow. Um don't put the ring on, by the way. In the Lord of the Rings example. Super bad plan. Super bad plan. Um okay, so just a just another example here. And uh, you know, you've got uh so if you if you know that your data structure can commutatively combine, if you know that, for example, I can change the order of things, like I can do with numbers. One plus two equals two plus one. Well, at this point, it's like we we sort of suspect that this is gonna give you something. So we noodled about this. We thought, okay, we can write the interface, we know that some of our data structures can add commutatively. You know, we know we can shuffle numbers. We can't shuffle lists, we have to keep the order. We can put sets out of order because they don't care. They're not ordered. Okay, so what does this give you? Well, in MapReduce world, what this gives you is the ability to start kind of pre-aggregating things as they appear. So there's this idea of kind of map side aggregation. But you can think if I'm if I'm buffering something on a particular server, I'm getting my key-value pairs. I haven't sent them off to the thing that's going to do the final aggregation yet. If I know that the the add is commutative, I can just start adding them whenever they show up. No problem. So, what this does for you is if you have a system with a massively skewed distribution, say you've got, God, I think it was Justin Bieber when I worked at Twitter. Like every time Bieber tweeted, it was just total chaos.
SPEAKER_03There you go.
SPEAKER_00Yeah, so if you try to aggregate with his user ID as a key, your system's just gonna fall apart. But if you aggregate, you know, Justin Bieber key value pairs and you know you have a commutative data structure in your value type, then no problem. Those, whatever node is getting hammered goes ahead and just pre-aggregates and it's just sending out instead of counts that look like one or two every time something happens with this key, you send out you know, a thousand or two thousand. Or if you're using sort of a different data structure and you're adding that together, you get the equivalent boost that you can imagine, where just that that really heavy, uh, that really hot key just gets kind of calmed down a little bit. Um so anyway, we you know, we went down this rabbit hole, we built the whole library that is just like a whole zoo of these data structures and a bunch of interfaces that you can code to. And you know, I spent a good happy uh couple of years just mining computer science literature and finding things where you know you'd kind of go, I don't know if that's useful, but I'm gonna, I know I can write this operation. And we'd add it to the zoo, and invariably somebody would find something just bizarre and wild and amazing uh to do with it. So I'm I'm so hooked on stealing good ideas from people that sort of left the door unlocked. It's right, it's it's wild.
SPEAKER_02Um this was all a scholar library, and I I just I um I don't know if it's me adding this to what you're saying or if you sort of speak with that accent, but I hear the it's like the type system is kind of winking at us from the back of the conversation here.
SPEAKER_00Yes, that's right.
SPEAKER_02Yeah. So that that seems like uh that was probably a good language to be working in.
SPEAKER_00Yeah, the the thing, yeah, that that's a really good point. This kind of thing where you're saying, oh, you know, I'm I'm I'm trying to code uh against types that I want to have certain guarantees. Scala is a language where you know you can code in Scala like it's Java, and you can have the type system be your, you know, effect like sort of watching over your shoulder and being the bumper lanes in the balling alley. Or you can take this sort of wizard demon thing and harness it and make it do work for you. And so, you know, you can say, one, I need the I need to make sure that these properties apply, or don't submit the job. And it'll go do that. I need to go get these implementations of how to go add together. And they're written in some other library, I don't know where they are, I didn't import them. Please go find them. Compiler will do that for you. You might have something where you've got, you know, a tuple with 10 items in it, and each item is like a dictionary of a string, and then the value is another list, and the list has hyperlog logs in it. And all of these things are you're able to build up, you know, semi-group and monoid instances for these, but it's just a bunch of work. And you have to say, well, I've got a tuple of 10 things, I can make a monoid for that by getting 10 other monoids, and then I'll just sort of go down and element-wise add them. So there's a lot of busy work that is just kind of a pain and makes you not reach for these tools. But you can get the compiler to build these things for you at compile time without you writing anything at all. Um, and we use that trick in Summingbird all through. We use that trick for a similar library called um with another scary name called bijection that was able to build these like really complicated serializers for you, for your data types, without you really having to do anything at all. So, yeah, totally agree. Like, use the compiler, lean on the compiler. It's not just something there to catch errors, it's there to be like harnessed and and really uh bent to your will.
SPEAKER_02Very nice. Now, this you um the the I'll call them backing stores. I that word just came to mind. I don't like it. But the uh the systems this was built on top of at Twitter were um you know the contemporary systems at the time you were building it. You've mentioned Hadoop and Storm. Um later on at Stripe you took this and did some work with Kafka. Yeah, how does it change there?
SPEAKER_00This is like a this is like a no-brainer plug. Uh I mean the way it changed, and I'm not just saying it because we're friends, I'm not just saying it because I'm on the podcast. I mean, things just got things just got much, much simpler. Um, you know, a lot of the problems with well, another problem with a system like the Lambda architecture is okay, say you solve the problem of writing the code twice. That's that gets you far. Good. Uh but what about your data? You know, your data's coming in, having to go replicate your data into something that can feed into an offline, powerful, error-free system. That's one engineering task. And then getting these things to go fly across the wire one at a time is another one. And in a sense, you just push the problem back. You know, if you can write these jobs that can go run in multiple systems, now you have a lot of jobs running. It's like you have more pain than you did before, actually. You've got and you know you have good ideas that if you could just solve this, um, you know, they'd be exciting and wonderful.
SPEAKER_02Right. So and and it's an engineering problem that feels like it's in the way of this beautiful and powerful abstraction you're trying to unleash on the world.
SPEAKER_00That's right. Um, so Kafka was just, I mean, just solve this. Like I just frankly didn't at Stripe we I think at Twitter we we did adopt Kafka. Um I'm just gonna forget the details of what we used for the online streaming system. Um I'm sure that the move was made at some point, but very clearly at Stripe uh it was Kafka from the beginning, and it was just sort of a no-brainer. Events would come in, they go into Kafka, they get uh you know put into these wonderful sort of long logs that, you know, by topic that we're able to read out anytime we want. And offline Hadoop would consume these hourly batches of data that we stored in our logs and do its thing. Uh our real-time system, which was a custom, sort of a custom machine learning-ish version of what Storm did, um, just had a pointer into the same topic and would scan along. And the architecture of Summingbird there worked, I think, very, very well with Kafka, because frankly, it didn't matter if we double process data. We didn't have to worry about it at all, uh, because that was that implementation detail where everything is associated with a batch with a very hard boundary. And the boundary is derived from the event itself, not from when the processing happens. So if you if you want just, you know, your real-time system dies, you just pick a couple days ago, sometime before you think your offline systems process up to, and just start churning away. And your pointer will catch up and you'll be fine. Um, so I mean it it really uh I think is a nice example of how if you get the right abstraction and lean really hard on it, um classes of problems just go away. There's no more kind of you don't get the you know the nice feeling of like solving hard problems, um but they're just gone.
SPEAKER_02Given all of this, uh it seems like it would be there's there's elements of what you're saying uh in summingbird and the way you're describing summingbird, elements that sound familiar with respect to uh you know what I'll call a mainstream stream processing language like Kafka streams, but then there's things that Kafka Streams hasn't even come close to doing that Summering Bird does, um not just with type abstractions, but with that catching up sort of uh functionality you were just talking about that you did at Stripe. Um where do you see uh things going? I I realize it's a terribly broad question, but uh uh given that you've built this stream processing thing, uh you I'm assuming you sort of know the state of the mainstream APIs now. Um where do you think we're gonna be in a few years with the way we we think about uh computations over unbounded data sets?
SPEAKER_00Oh man. Um I mean, good question.
SPEAKER_02Uh who knows, right?
SPEAKER_00Yeah, so let's see. I think that the trend that I think is well one, let me just address something you just said. So you're talking about sort of you frame summingbird versus Kafka streaming. This is a um I would say that the the um well one, just concretely, what the library is meant to be is not something to really compete with something like Kafka streaming. It's really just a little DSL, a domain-specific language, that just acknowledges and exposes the fact that, okay, Kafka streaming is plenty powerful enough to be the real-time part of a system like the Lambda architecture. But you have to write stuff multiple times. So we'll just code generate for you the what would be this finicky, kind of delicate job where you're having to go figure out where the boundaries are and do things like this. We'll just take good patterns and have the compiler do them for you instead of you having to do it. So, Kafka streaming, because of the level it sits at, one is a lot more accessible and usable than Summingbird, which if you picked up now, you'd hit this kind of abstraction wall probably. And you'd wonder where you're sitting. Kafka streaming, you're able to get started right away. Um, so in one sense, they're a little different. Um, but I think this difference gets at maybe my answer here, which is that um you know, the early systems really really left you down in the nuts and bolts of things. You really were specifying a lot of how things happened and how to serialize your data and how shuffles have, you know, right.
SPEAKER_02You're you had key value pairs and that's right.
SPEAKER_00That's right.
SPEAKER_02Not well, the early systems gave you map and reduce uh functions, and you know, then then they got a little bit more sophisticated.
SPEAKER_00But yeah, it was it was hard. It was hard. It was really hard. And and because it was hard, people didn't do anything beyond a single map reduce job. Um and I think the movement I would love to see, and I think is happening, is toward more of kind of a declarative style of programming, where you know this was our attempt at something where you say what you want to happen. You're still saying, you know, I want you to take this event and make key value pairs. So it's not quite there. You know, but you're saying I want, I want this computation to happen. This is a computation I could have written in sort of the Scala sequence API. I don't care really how it happens on Hadoop or on Storm or on both. In fact, the data structures themselves are gonna tell you everything you need to know about how to work with them. You want to know how to store these things? The data structure is gonna tell you, and the compiler is gonna go ask the data structure for that. So, really, like this impulse toward uh declarative programming. This is what I want. I don't really care how it happens. You know, I've seen so many attempts at that that it really just seems like this core uh desire for software engineers. Um I I maintained another project for a number of years called Caskalog, which uh was like Yeah, that was uh like a DSL on top of uh MapReduce, right?
SPEAKER_02That's right. It was enclosed. Only ever wrote one job, and I was thinking of Caskalog as the thing that made it so that you could not.
SPEAKER_00That's right, that's right. Kaskalog, you could make, you know, really like a you could say the it it was this little data log implementation where you could just say things you wanted to be true or facts. So it was this logic programming language, and it would go make something where the number of map produced jobs often was very surprising. You could express something logically, sometimes that would take hundreds of jobs to compute. And you know, that's got its problems too, but the uh the fact that your creativity is not bounded by the uh limitations of a particular system, I think is a very, very powerful uh primitive. So I'd like to go to a world where the information that really just slams everybody, like how do I store my data, how do I manage data sets, how do I know I can access this down the road, how do I do this efficiently, where that's all just part of the data sets themselves, the data structures themselves. And you know, the tools we build are able to kind of communicate behind the scenes and just not just not bother us with that stuff. I think it's it's been too long, Tim. You spent too much time, it's been too much rage coding. Um, I I'd be curious to hear what you think. But I think this this, you know, in a hundred years, are we going to be specifying sort of, you know, beyond just responding to a cost estimate from a compiler? Like you've got to think with many of these, many of these jobs, um, you'll be kind of finding very interesting data structures, taking cues from them about what sorts of things they could tell you about the world, and then asking your system if if it's possible to build this stuff. And often it will be because you're leaning on these sort of tried and true amazing gems that we just haven't really plugged into our incredible computing engine yet.
SPEAKER_02I appreciate you asking me. I I I'm normally the one asking the questions here. Um and I actually don't have a solid answer to that, but I know I know what has happened since we've had computers and since we've had programming languages just over the last like you know three generations of of human lives. And that's that that uh things have become more abstract, right? We have we have added layers of abstraction. That's Grady Booch's observation that that's the history of computing. Um we don't do any less, we don't think any less. Uh you know, uh in the uh mid-20th century, uh John Maynard Keynes uh predicted that by now people would work, he was an economist, people would work about 15 hours a week. He was just sort of extrapolating productivity gains. And he said, Well, you'd only need to work 15 hours a week to provide for your needs. And we're like, that's that's cute, John. That's not the case. Uh we work a lot, we just have more. And so, you know, we never work less, we never think less as as developers, but our tools do get more abstract and therefore more powerful. So what you're saying is is credible in terms of that framework that I use to think about the future. Um because um you know, where is my data getting stored? And hey, when you write to the data store, uh should you retry if it doesn't go through the first time? Um, you know, great. That's give me a break. Why why am I thinking about a thing like that? Um that's that's way down in the weeds. And the first the first generation of anything we build has to be like that because nobody's smart enough to to build the more abstract thing. But then once you use that, you get some experience with it, you get thinking like what you've provided here.
SPEAKER_00Well, I think Yeah, I think the the um and what you're saying, what you're saying is really interesting, right? There's this the the the going up the abstraction ladder, I think, does not it's not obvious that that is gonna be that that's a great thing, that that equals more power. I mean, I know a number of developers worry about this sort of future when we're just really plugging components together and the creativity has been pulled out of the enterprise. And I think that's that's a valid fear with certain types of abstractions. You know, a lot of people love their Emacs because they can really get in and sort of open up, uh open up the editor and do anything they want. Um, so I think what we're after is, and I these are the abstractions I think you were talking about. You know, we don't want abstractions that box us in and say, okay, here's something where you don't know what's going on inside, but there's a limited number of ways to put it together and just do that. You know, we want to find these abstractions that are really like portals to another entire field. And I think that that's that's what the attempt was here. And that's not something at all that I knew in advance. It was sometimes you find an idea that just tidies code for you and maybe solves a problem so you don't have to think about it again, like this sort of retry dance that you're mentioning. Um but sometimes you find something that almost zooms you in to an entirely new, like, you know, it's just as creative and open as when you started. You just sort of zoomed in on the Mandelbrot fractal. And I think that, you know, if I were to really push and evangelize for people to go, like if you're gonna do a concrete thing, like look at the systems you have and just see if there's something where if you kind of cocked your head and squinted, there's some simple idea that you could actually spend years on just sort of plugging in different pieces to that. It's maybe not the best way of framing it, but the the the dive into something like the field of abstract algebra and realizing that system optimizations actually seem one-to-one with these different properties, just I mean, you become like an explorer in a land that you know no one's seen before. Um, so those are the abstractions we want uh to hunt down, I think.
SPEAKER_02My guest today has been Sam Ritchie. Sam, thanks for being a part of Streaming Audio.
SPEAKER_00Tim, thank you so much. I really appreciate the time.
SPEAKER_02And there you have it. I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at at TL Burgland on Twitter. That's at T L B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in Community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which we think is a good thing. So, thanks for your support and we'll see you next time.
