From Git Blame to Principal Engineer with Sage Pierce | Ep. 22

Adi Polak talks to Sage Pierce (Indeed) about his career in software engineering and event-driven architectures. Sage’s first job: Java Swing development at a Department of Defense–affiliated research lab. His challenge: working at Indeed on event-driven views and IMI to join data across domains in a polyglot microservices world.

Sage's Atleon project: https://github.com/atleon 

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_02 0:00

A lot of the hardest lessons that I had to learn had to deal with quote unquote scaling myself up. You have to learn to stop doing things yourself. Even though it's hard and I'm saying it's hard, you shouldn't be afraid of it.

SPEAKER_00 0:11

Hello there everyone. I'm Andipalak and welcome to Confluent Developer, where we explored the wild, weird, and wonderful journeys of software developers building systems that hopefully don't wake up at 3 a.m. This episode I'm interviewing Sage Pierce from hacking on Java Swing apps inside a top secret defense lab. Yes, swing, it still exists, to becoming a principal engineer obsessed with domain-driven design. Sage has seen things, monolith things, microservices things, page and go burr things. His journey takes us from calling sidegs and governance security clearance all the way to event-driven architecture, scaling through people, and even inventing a new streaming pattern like IMI, all while discovering the universal truth of engineering. If your name is in Gitplame, you're on call. We talk growth, leadership, streaming data, and how to avoid building yet another distributed header ball. Let's get to it.

SPEAKER_02 1:31

Hey Addie. Yeah, excited to talk talk tech and talk talk streaming.

SPEAKER_00 1:35

I know you've been recently promoted, so congratulations. Kind of like crossing the chasm in between uh staff engineer to principal engineer. It's it's a big deal. Um I don't know if engineers know it at home, but it's kind of like when you reach a senior level, it could it could be like you know, a finite level for some people, for some folks. And getting anything beyond that, it's definitely um um very, very, you know, it shows that one's is really eager to learn and grow and uh do what needs to be done in order to continue pushing forward. So I'm curious if you have kind of you know learnings or things you had to learn or you know, de-learn also forget, put it that way.

SPEAKER_02 2:17

Yeah, uh honestly, all of the above. I think every person's uh every every engineer's journey to the levels of staff engineer and principal and then technical fellow after that. I know there are different terminologies at companies. But every every engineer's journey to those levels, I think, is is unique. Everybody has their own experiences and a unique story to tell about that. Um so what I what I can tell you is every everything everything about my journey is very uh unique to me, I think. Um I can tell you that I I've been at I've I've been at this job of software engineering. I've I've I've been doing it for about 15 years. I got my uh start um in 2010. Um and and since then, you know, the similar to all engineers, it's just been a long series of of lessons and learning and growing. And as far as the transition from uh sort of a junior or senior level into staff and and principal goes, uh for me, the the lessons, a lot of the hardest lessons that I had to learn had to deal with uh quote unquote scaling myself up. Um and that's that's a quote from uh one of my one of my managers that um I still work with. Um he's not not currently my manager, but he's somebody I look up to a lot. And um one of the first things he told me as I was getting further on into my career was you have to learn to stop doing things yourself. I am I am an engineer that I I really, really enjoy getting into the weeds, solving the problem, not just not just saying what the problem is and then saying what a good solution is. I get really focused on if I if I know what the right solution is, I'm it it takes it it takes a lot of willpower to not just go and do that thing. Um but as I found out over time, as you as you build more and more things, people start looking to you as the person of support. You become the whether you like it or not, empirically, you become the first person that gets contacted when something you build built breaks, even if you've tried to hand that thing off to another team. Um you you are if if your name is is in the get blame, you're the you're the person that gets that gets tagged first. Uh so one of the hardest lessons I've learned in the transition from senior engineer to staff engineer to principal engineer is is focusing less on doing the work, doing what I what I used to consider the work, which is writing the code, and more focusing on how can I teach the engineers around me uh to do that thing and and help coach them up when they do something different that uh differently than I would have done it. Um and if I think they're that the way I would have done it is better, I try to explain why that is. And sometimes I'm not always right, and that's even another opportunity for me to grow when there are disagreements and I learn that you know the way that I would have done it wasn't the best way to do it, and that happens as well. So it's a it's it has a lot to do with learning from others and teaching others um as as you move up rather than just doing the work yourself.

SPEAKER_00 5:38

Yeah, it's um you know, this is gonna stick with me though for a while. It's like scaling through others and learning to scale yourself in in the organization, knowing that um our knowledge is finite. We only have 24 hours a day, no matter what we do. Uh, you know, we can have other resources, infinite other resources, but time um is really as finite in uh yeah, so this is a this is a really interesting learning. I wonder if you had like something distinctive between uh staff and principal that you can think of.

SPEAKER_02 6:14

Yeah, for sure. I and so I obviously you mentioned I I was promoted last year and that promotion happened while I've while I've been at Indeed. Um so I've been at Indeed going on five years now. Um the the biggest difference to me between staff and principal generically speaking is is this transition from working, being a tech lead for a group of engineers, and instead being a tech lead that knows how to coordinate amongst different groups of engineers in different organizations. So I'll give I'll give you an example. This ties a little bit into uh the talk I'm giving uh next week. Um I'm I'm a I'm sort of a domain-driven design junkie. Um you can see some of my domain-driven design books uh behind me on my shelf. Um and when I I use domain-driven design and domains in particular when talking about different teams and what they're responsible for and how those domains and those teams talk to each other. At indeed, uh, the current team that I'm a principal engineer for has to deal with managing the ingestion and management of uh intents to hire. So we we maintain the data storage and data flow of how employers say I'm I'm looking for a job to hire, or looking for a person to hire for a job. And then there are there are other domains that are responsible for some data that is completely orthogonal to the hiring intent. Um, that might be information about the companies themselves. It might be other domains that are responsible for deriving data from other data that employers give us. Um moving into the position of principal engineer requires much more of not just being a tech lead for all of the services that the subset of teams that are in a particular domain are working on, but then understanding how that domain relates to other domains that other teams are responsible for. Um and not just uh internalizing the communication or the necessities of communication between those domains, but also working with the members on either team and either organization that are that are designing and building out those domains.

SPEAKER_00 8:32

Cool. And also I'm guessing like finding the dependencies and um trying to unblock. Yeah, there's it's fascinating uh going from staff to principal. I'm sure it's a lot of work in uh Code of Steel for finding the ways to uh to do it uh effectively efficiently.

SPEAKER_02 8:49

Yeah, I I do I do feel compelled to tell you a part of my story in particular. Um I did not want to be promoted to principal engineer for for quite a while. Um my manager at the time, uh I think we we first we first broached the subject of being promoted from staff to principal uh about two years before it actually happened for me. Um and the I'll be honest, the main reason I didn't want to be promoted was that I was I I knew then, as I know now, that I very much like doing the coding work, right? Um but I I uh I saw my my colleagues that were already principals at that level, they spent much more of their time in meetings. And I I was already spending a significant percentage of my time in meetings, and when I would see the calendars of my principal engineer colleagues, and I would see 50, 60, 70 percent of their week blocked off for meetings, that just didn't appeal to me. So growing into principal, um, I had to really get my manager and uh uh the leadership apparatus to agree that if I if this was going to happen for me, I wanted to do it at my own pace. I wanted to uh maintain a connection to the actual code being delivered, uh, as well as taking on those extra responsibilities of uh scaling myself out through others.

SPEAKER_00 10:15

Yeah, you know, it's it's amazing to me how um we have more authority to define the way if we're willing to uh first of all identify what are the things that we we enjoy and we want to do, and second thing be able to communicate it in a way that um other sides can relate and understand. And um it's um yeah, I think you did that because you were able to kind of shape uh your role in the way that uh where you can continue to enjoy it um and also satisfy the uh the rest of the leadership and company needs, uh put it that way. So congrats, that's super exciting. Um I you know, we're here to talk about challenges, we're here to talk about work. Uh but I was curious, do you remember your first ever job?

SPEAKER_02 11:07

Oh yeah, absolutely. Um yeah, so I mentioned I got I got started about 15 years ago. So 15 years ago, um I I got a job while I was in college. Um I was at the University of Texas. Um go Longhorns. I'm a big Longhorn fan. You can see my Gilder Bear has got a longhorn on it. Um so I I had I knew when I was in college that I didn't really know for certain in college that I was gonna be a software engineer when I got out. Um my major was electrical and computer engineering. I was really into renewable energy, um, but also microprocessing. Um so the programming I was doing was really kind of systems level, uh, embedded processing level uh in school. So I didn't really uh I didn't know that I would be real into high-level programming, kind of what I'm doing now with JV and languages and whatnot. Um but I I uh knew I needed to get a job because I was I was low on cash and I was a poor college student. Um and I applied for an open position at a research lab on a research campus affiliated with UT. And the name of that research lab was called Applied Research Laboratories. And they they hired me as uh initially as a software technician. Um that was just a fancy way of saying I was a uh a student uh that was being paid part-time to uh work at the research lab. Um and this research lab had federal contracts uh with the Department of Defense. So there's a little bit about my work that I can't that I backed in that I can't, I'm not supposed to talk about, even though it was like more than 10 years ago. And I'm pretty sure I don't I don't think I was really working on anything that was classified, but I had to go get a they made me get a government clearance and whatnot. Um, but that was just because I was around classified things. Um, but that job uh was I didn't know it at the time, um, but that job lacked every single bell and whistle related to software engineering that you could imagine. It being a research lab that was affiliated with uh or that had federal contracts and needed to be security compliant meant that they were very, very picky about the the tools and technologies that they allowed you to use. Um so my my first development environment was a bare bones Linux uh laptop, or not laptop, it was a it was a tower actually at first, and they gave me a laptop. Um and that uh computer, I wasn't I I wasn't allowed to take that computer home. I had to work on it, I had to stay at the office. Um and my coding environment was there was no IDE, it was a a shell script and a uh graphical text editor called GEdit. And I had to learn, I I had to learn a lot of things that I take for granted now, like how how do I compile Java locally, um, how do I run that application? And the applications I was working on, and I don't know if this isn't sensitive or anything, but I was working on applications that helped uh helped uh defense contractors monitor uh the state of our GPS satellites. Um so they were they were Java applications written with swing. Um and I believe swing is still available as uh an API in Java. I don't think anybody uses it. I don't think they should. But everything I was working on at the time had to do with Java Swing development.

SPEAKER_00 14:48

Yeah, maybe those you know, Java jobs that you created are still around and running somewhere, monitoring everything. I can only use it. If it works, you know, don't change it unless there are new requirements. Um but that that's that's really super cool how you did the transition from uh from that world into software and kind of working on uh yeah, working um also working at the governance. It's always interesting because there's so many cool projects that unfortunately people cannot share. Uh but a lot of very cool things.

SPEAKER_01 15:22

Now a quick word from our sponsor. Confluent developer the podcast is brought to you by Confluent Developer the website, which has everything you need as a developer of data streaming systems. And it's completely free. We've got curriculum, hands-on exercises, executable tutorials, the online data streaming engineer certification, also free, a way to find a meetup near you, those are free. Everything is there. I really want you to be successful in your journey as a data streaming engineer, and this is the site that has what you need. Check it out at developer.confluent.io. That's developer.confluent.io. Now back to the show.

SPEAKER_00 16:01

Okay, let's do a jump forward. There's a lot of, you know, challenges, software challenges, architecture challenges, um, or as you mentioned, like domain, how do you even define like owners? How do you split it? How do you how do you create the right interface and so on? Um maybe you want to share some recent challenge that you worked on or you're still working on and some learnings around that.

SPEAKER_02 16:23

Yeah, so the it's this is I I I'm gonna take this question, I'm gonna uh move it, bleed it into kind of what this this overarching problem that has permeated the last 10 or so years of my professional life. And that challenge, I I refer to this challenge as event-driven views. And the the the setup for this challenge is is pretty I I my guess is it's pretty um uh common to a lot of people that have been in tech for any significant amount of time, especially any anybody that's worked at companies um that started with large monolithic structures and found out that it doesn't scale well. Um so uh the the problem statement starts is that starts like this. Most really successful tech products, software products, at least prior to five years ago and in the previous couple of decades, they usually started really quickly with somebody creating a database with a bit uh a nice schema and then setting up a service in front of it and then demonstrating value and then launching that to production. And then as they wanted to add new things, they would just add new things to that service, add more data to the database, and then as the as the product might have grown, they would have had people, other internal uses of consuming that database to run things like analytics. Maybe they attach some change data capture to the database so that they get real-time uh data propagation downstream. By and large, what you find out is if the company has a long life, has a has a longer than non-trivial life and they keep building on that initial structure, what they end up with is a giant mess of a database and and monolithic service and monolithic architecture that becomes difficult to work on, right? We've we've heard uh monolith might as well be a four-letter word in the world of uh software engineering and software architecture. Um I think everybody's familiar with with the pain that those kind of monoliths can solve or can uh induce. Um so what ends what ends up happening by and large, and this was really popular throughout the 2010s while I was uh progressing from junior, senior, uh, and a staff software engineer, uh, was the approach to what the the approach for a solution for monolithic architectures is start breaking it down into microservices. So microservices, in my view, should imply uh at least a little bit of knowledge and background on domain-driven design, because you can't have microservices kind of spawn from the world of domain-driven design. Um so uh uh having having an understanding of domain-driven design and uh leading that leading into creating microservices was pretty important for me. Um and I was fortunate at my at my second job that I had, uh they were uh pretty adamant that they spend a good amount of resources on up leveling their engineers. And one of the topics that we would talk about uh was domain-driven design. So I had a I had a pretty good uh good academic start on learning about why why you break up monoliths into microservices and how you use domain-driven design to do that. Um and it wasn't too long after that that I stumbled into this problem that kept on coming up of okay, we've broken that we've broken this monolith into microservices using domain-driven design. And now the problem is we have use cases like analytics, or we have use cases that are domains that are span other domains that we've identified. We need we need a domain that aggregates data from domain A and domain B, and we need to keep an entity uh up to date in real time based on changes to these other domains. It used to be the case if you had a monolithic architecture, all that data was in one place. You could just, it was easy, create create that view off of the data that spans multiple bounded contexts, it's all munged together in the same database anyway. Seems like a pretty straightforward thing to do. Obviously, you can't really do that with uh with uh microservice architecture. And and even doing that even in a microservice architecture implies you have some form of an event-driven, uh an event-driven design baked into your architecture as well. Because how can you, how does any downstream domain know about data changes in an upstream domain if that domain isn't producing events?

SPEAKER_00 21:12

Right.

SPEAKER_02 21:13

So we so we've already talked about like three three pretty large concepts that have entire books dedicated to them domain-driven design, microservice architecture, and event-driven, event-driven design. Um and the those all those all lead to this, are all a component of this problem that kept coming up of we need to be able to do something interesting with a view of data that spans multiple bounded contexts. That could be analytics, it could be indexing into a search engine for multivariable search, it could be its own domain that other people want to incrementally operate on, but keep that data replicated from the other, keep certain data replicated from other domains. There's a whole wide gamut of use cases.

SPEAKER_00 22:03

Let me just see if I can I can I f I follow up everything you said. So essentially you you have uh you broke into Monolith, right? Uh you turn it into microservices.

SPEAKER_02 22:12

Each one of them has their own you know streaming data from place to place and um uh serving uh whatever requirements uh they are responsible and and owns um and suddenly you get more requirements that are now more kind of like data focused like you have analytics you have different services that needs to aggregate uh information right and this is where kind of like new challenges uh appears in in the system okay cool that's exactly right and the the extra compounding factors that make it more that make it more difficult is you find out that if when you one of the benefits of microservices and domain driven design is you can separate things but as those things become owned by different teams those teams have different preferences one team might want to use a relational database or it might make sense for their their domain architecture to use a non-relational database. Another team might be using Kafka to produce their events but they might be using an AWS native solution so they might be producing events to SQS. Or and one of the things that I I find really common is that given that Kafka is rel is is relatively young compared to message buses in general the some of the older companies and tech stacks will use something like RabbitMQ or some uh any any older uh message bus technologies and what the the compounding factor there is well any solution for an event driven view that you need has to be able to account for this polyglot infrastructure different databases different message buses and whatnot is different right so the tech stack is different so what what what's what do we do? Yeah absolutely so um this the the the nature of a polyglot architecture and then the the nature of the eventual consistency of an event of an event driven system those all lead to a unique set of problems if you're trying to solve this this come up with a solution for this abstract need to come up to see a view an up-to-date real-time view um of the data that you care about that needs to be joined across domains um one of the things I touch on and I'll touch on in my talk next week is this sounds really similar to like GraphQL right to like query federation I have data across multiple domains that I need to join on demand and and be able to view that data. Where event driven views differs from something like query federation or GraphQL is that event driven views is much more backend oriented. You want to you you know the exact services you need to hit so it's not it's not it's not exactly federated and there's a lot of throughput and latency constraints that you'd like to account for that aren't necessarily addressed in GraphQL. And in addition not just querying the data but notifying your downstream consumers when that view that you're keeping has changed becomes not just desirable but a requirement in a lot of cases because a lot of domains will care about when the join data changes not just the independent data from the upstream domains change. So that is it's a long-winded way to explain the the entire domain of this uh this problem of event driven views um which is what I've been working on coming up with I've come up with helped come up with several solutions um in the in the past uh decade or so um and the one I'm presenting on next week has to do with this uh stream topology that I that I have termed ingest materialize index um so that that is the that is in the name of my presentation next week which um I shortened that to IMI um and IMI is all about coming up with a simple a maximally simple solution attempting to be maximally uh simple uh solution uh for event driven views cool can you give us like a brief uh you know this is how we build it absolutely yeah so uh uh IMI ingest materialize index it's I name it that because it's based off of these three uh three stream stage each each of those words maps to a stream stage so the way IMI starts off is you ingest events from any any uh producers of events and those those events should indicate um typically ideally they should be domain events but not everybody has those sometimes they're just raw CDC events but these events should indicate when data that you might care about viewing has changed so that ingest stage is responsible for consuming whatever the raw schema is upstream that you don't control because the producers control that schema consuming that those events and then using the data in those events to relate the foreign data the foreign keys in that data to the view keys that need to be reprocessed or rematerialized. So in ingestion you have this responsibility of correlation and then ultimately once that correlation is done you produce commands. So this is I'm CQRS command query responsibility segregation is another concept that's baked into this not just the problem but also the solution. So ingestion is responsible for correlating the changes in foreign domains to the domain of your view and then generating commands to reprocess that view. That feeds into the second stage of IMI materialization and in materialization what happens is you take these commands and in our in all of our cases we apply micro batching um which is take in elements or take t duration of commands whichever one comes first and then you take that batch of commands and you dereference as usually the word that we call we dereference all of the data that's uh related to the view that you want to generate and then we feed that blob that aggregated blob of raw data to a generation phase and then that generation phase produces all the views that need the current view the current state of all the views that need to be indexed and those get put onto another Kafka topic and that feeds into the the third stage which is index so each of those views can be indexed to any number of destinations including a database where you can look up the data on demand or you can have CDC from that database which we also do which lets you let your downstream clients know when that data is changed.

SPEAKER_00 29:12

Cool.

SPEAKER_02 29:13

So I'm guessing like you know thinking through there are some requirements that this system has to comply to like I'm thinking maybe um exactly once right if anything changes you know there's some command or operator something happened to the data and we are sending those events as changes we need to make sure it actually arrives at the destination right yeah so it it ends up being the case uh with all of our use cases that we don't care about exactly once we just care about at least once um so that that is our minimum level of guarantee you will get you will get these changes will flow through the system at least once um the the other thing I wanted to call out um before we move on to uh before we move on to the next thing is the the architecture itself I'm I'm one of the things I'm happiest about with it it's number one it's abstract so it can be applied to several different uh streaming methodologies um but it requires a very minimum set of resources which is one of our requirements we don't want to we would like to not have to replicate our data all over the place just to make it compatible with a given tool um and the solution the the bare solution that we that uh IMI proposes is a single application with at least three stream processes in it and then two Kafka topics so that absolutely minimizes the number of resources you get we we get at least once processing especially using Kafka as uh the intermediary between the stream stages um and we get robustness um because we know we won't uh we have we have uh a library that we use it then this is also a part of my presentation um a bit of a passion project that I've been working on called Atleon um and that's a it's a lightweight stream processing framework that gives you the guarantee of at least once processing regardless of what infrastructure you're consuming from or producing to that's really cool well I'm very I'm very excited for your talk and if anyone is watching it afterwards it's gonna be available on YouTube as well so we can always catch up with with that um I'm curious is the is this library is ever going to be open sourced or do you think about that? Yeah it is open source oh it is open source yep you can check it out at atleon.io very very cool and the the name the name is uh comes from at least once at Leon um so that's uh I've been it's it's been a project that I've been working on for going on through might be four years now wow okay that's uh I mean some of the best projects are the ones that we work on for a long time in my opinion so very cool all right any like learning so far that you want to share before we're wrapping things up from you know thinking about the system designing architecture pitching uh the system also getting buy-ins from some folks as well I'm sure you've been working you know across the company with multiple teams I think my biggest lesson with with with all all of this stuff concerning uh not just event driven views but but stream processing in general it's super super powerful but there there there is a pretty big learning curve when you're going every everybody starts out their their first project is uh or their first like business project is usually create a a rest endpoint or maybe a grpc endpoint and it it's really intuitive to think request response right moving into the world and and this is the challenge that I I face a lot when designing asynchronous systems and teaching other engineers about asynchronous uh systems and stream processing is you don't really know what's gonna happen after you fire an event off. You know you you agree to a contract and you produce it and then the things that happen after that not only not only are you not really sure anybody's gonna do anything after you fire that event off but it some of the things that that happen might be non-intuitive. So it it's it's definitely I I find asynchronous processing to be extremely powerful a very fun problem space um but I but a pretty a pretty high learning curve but a learning curve that's definitely worth it because it it definitely moves you to the the next level of being able to think and design systems that scale and even though even though it's hard and I'm saying it's hard shouldn't be afraid of it. Dive in and uh learn learn what it's all about.

SPEAKER_00 34:00

Yeah just do it it's the future cool sage thank you so much for joining me today and sharing all your expertise and again congrats on the promotion and uh you know if this uh people watching after current will share the link uh and as well to the YouTube uh video that will be uploaded after the event so everyone can uh you know go back and and listen and learn uh from the recordings.

SPEAKER_02 34:26

Sounds good.

SPEAKER_00 34:27

And of course it was a pleasure