Killing Clusters & Orchestrating Chaos with Colt McNealy | Ep. 20

Tim Berglund talks to Colt McNealy (LittleHorse Enterprises) about his career in distributed systems. Colt’s first job: software engineer at a real estate company. His challenge: working in a complex microservices environment and turning that pain into Little Horse.

Colt's Current 2024 talk: https://current.confluent.io/2024-sessions/kafka-streams-as-a-data-store-for-a-workflow-engine
Gunnar Morling's blog: https://www.morling.dev/blog/
Jack Vanlightly's blog: https://jack-vanlightly.com/

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_02 0:00

Today, from killing a production Kubernetes cluster right into a process orchestration startup. This is Confluent Developer.

SPEAKER_01 0:08

A very loud group of people who think Rust is the answer to everything, including Cancer. And that's not the only hard part. The hard part is there's other systems outside that you don't control that have their own database. Like there's other ways to do it. Is that the best way to do it? Not really, right? But that's not really a business thing. That's literally like a it's checking the validity of a token.

SPEAKER_02 0:27

Hey there, everybody. I'm Tim Berglund, and welcome to Confluent Developer, the podcast where we explore the fascinating journeys of software developers tackling complex problems. In this episode, I'm interviewing Colt McNeely, the founder and CEO of Littlehorse, a Kafka-based platform to, and I quote, create, compose, and introspect your microservices workflows and AI agents. We talk about the rude surprise going from debugging undergraduate operating systems projects in GDB and VS Code, which is like X-ray vision, to the ordinary world of enterprise distributed systems where you can't see anything. And what Colt wants to do about that gap. Let's let's go. Hey Colt, welcome to the show.

SPEAKER_01 1:15

Thanks for having me, Tim. It's kind of an honor.

SPEAKER_02 1:18

Awesome. Good to see you again. And um Hey, tell us a little bit about you. Uh who are you? What do you do?

SPEAKER_01 1:27

Uh so if you've been hanging around the confluence Slack and the Kafka Streams channel, I'm the supplier of about 50% of the questions. So that's one thing I do. Um but uh It's good work if you can get it. What was that?

SPEAKER_02 1:41

It's good work if you can get it.

SPEAKER_01 1:43

Oh. Yeah, so that that's one thing I do. But what what I actually do is I'm the founder and CEO of uh a startup called uh Little Horse Enterprises, and we are in the distributed application management space. So we make it easier to build applications, workflows, integrations, AI agents in the modern world, which is distributed uh sometimes in the cloud, sometimes hybrid cloud, sometimes on-premise, and um with complicated business processing. So we we build basically data infrastructure software.

SPEAKER_02 2:16

There you go. Uh is it appropriate for me to think of that as a durable execution engine?

SPEAKER_01 2:20

Uh durable execution is a smaller part of it.

SPEAKER_02 2:24

Okay.

SPEAKER_01 2:25

Um in a way. So we we do workflow, um, and workflow and durable execution are very intimately related, but they're not exactly the same. So uh I don't know how much time we have to go into that, but uh happy to if you want.

SPEAKER_02 2:39

Maybe what may I I suspect we're gonna get into it in a minute. So before we get there, tell us what was the first job you ever had?

SPEAKER_01 2:48

Uh well, it's not as cool as my dad's first job. My dad was uh, I think he parked cars and swept floors for $1.75 an hour. Unfortunately, my first job was in tech, so I can't say anything cool like that. I was a uh I uh my my first summer of college, I got a job as a software engineer at a real estate company, and then I ended up working during school for um my last two years of school. I I I got out in three years. Um but I was working during school, and then in that job, it motivated me to uh create Little Horse. So this is basically my second job.

SPEAKER_02 3:23

I love it. I love it. Uh I also uh had a job working uh writing software in college, and that's a great way to do it. Um you learn a lot more writing code than you will learn in software classes, yes, uh in whatever major uh it is.

SPEAKER_00 3:39

So I don't know if you can see that there, but the the O'Reilly books are great too.

SPEAKER_02 3:44

Uh yeah, yeah, they're not bad at all. There's some good ones.

SPEAKER_00 3:47

Yes.

SPEAKER_02 3:47

Um awesome. All right. Oh man, uh it's gotta be a separate podcast. I'd I'd love to talk some more about that, but we'll we'll we'll do that. We'll do that some other time. Uh what is uh you you you know you've said you're kind of new at this, but tell us about uh the most interesting problem you've ever solved or or working on solving.

SPEAKER_01 4:06

So uh I I yeah, I think working on solving is a better way of doing it because this it's a it was a very humbling experience realizing how how tricky everything about starting a business is, um, including the technical difficulties of it, uh the people part of it, the business part of it, trying to make uh trying to translate the cool technology into actual business value is is another challenge. Uh but if we're gonna focus on the technical challenge I'm solving. When I was in school, you know, we were doing code in C, C, I did a lot of operating system stuff. Uh, it was great because you could just you know use GDB and you could uh stop your program in the middle of its execution and inspect where everything is. You could see all of the code that's running in a single um VS Code window, and everything ran on your laptop in one uh environment. And you know, distributed meant multiple multiple cores, and you had like a cache coherence problem.

SPEAKER_02 5:05

But you could still see like Superman with X-ray vision. You can see everything. Yeah, it's amazing.

SPEAKER_01 5:11

And then here I am, I think I'm bulletproof. I jump into the real world and I get a job, and I'm doing something. Uh what I was working on was making a uh widget in a portal that allows scheduling uh open houses, showings, home inspections. Um, and I think you know, there's another thing. It was like four or five different things we're scheduling for that real estate company. Um, and and there's a lot of different parties involved and a lot of external systems. There are, you know, some real estate specific uh systems, there's uh you know, Salesforce, there's our own internal APIs, our our mobile app for our agents, mobile app for the customers, uh, some really legacy stuff we had in our website. And you had to coordinate all of those pieces together. And I hadn't seen the code for about 90% of what I was calling. It was just a bunch of API calls over the internet, flaky connections, you know, interns like me would deploy things and microservices would be buggy, so then you'd get 500 for three hours out of two.

SPEAKER_02 6:05

You know how those interns are, you know.

SPEAKER_01 6:08

Yeah, I I I took down the site a couple times. I'll tell you about that later.

SPEAKER_02 6:12

That's yeah, it's you that's that's good to already have that under your belt. I mean, that's that's that's an important experience.

SPEAKER_01 6:17

I I thought I was pointed once at at localhost or my local Kubernetes cluster, and I uh deleted every single deployment on prom.

SPEAKER_02 6:24

Yep, yep. I dropped a production database once, so it's it's okay.

SPEAKER_01 6:29

That's not good. No, that's that's I I I at least only killed stateless services.

SPEAKER_02 6:37

I destroyed state. I mean, I was able to restore it. I had it was you know, we had a battery ended okay. But it's it's good to to know the the sort of primal panic when you realize the error.

SPEAKER_01 6:47

I also I think I got our QA database hacked um with SQL injection, but that's another thing. I haven't confessed that publicly yet.

SPEAKER_02 6:55

Okay. That'll be the podcast series on what's the worst security vulnerability you've enabled. Oh gosh. Yeah. I mean, comes with we we offer legal counsel to come with you to tell you what questions to answer and what questions not to. That'll be a separate series.

SPEAKER_01 7:08

I would not trust someone who tell looks you in the eye and says, I have never caused a security vulnerability, because that means they haven't learned anything.

SPEAKER_02 7:14

That's right. They just don't know. Right.

SPEAKER_01 7:17

Yes. So, well, back to the story.

SPEAKER_02 7:20

So you're you're you're this intern and you're you're being an intern. You know, you're building code, you're learning, occasionally making mistakes, but you're seeing uh the comparative sort of uh deity-like view of the world that GDB and VS Code gives you compared to these containers. Yeah.

SPEAKER_01 7:40

Well, there's containers, and that's not the only hard part. The hard part is there's other systems outside that you don't control that have their own database. Like Salesforce has its own databases, and I can't run a transaction in a Salesforce database that also you know touches our you know mobile apps backing database and our agent portal backing database in that microservice, and also dot loop and also SAP, right? Yeah, so orchestrating transactions across all those things, it I just really wanted a place where I could write code on one screen that involves steps running in all of these different things with rollbacks. Um, and that was the genesis of something that was very similar to uh durable execution. And and I looked at some durable execution players, I won't name them, not to turn this into a hit piece. Uh kind of hard to use. And also the the thing I realized was, well, this is also workflow. Um and workflow has predefined sets of steps, and there's an orchestrator that dispatches work to task workers that do those things. And if the workflow engine knows what's happening and what's gonna happen, you can get really, really rich data out of it, and and you can get better sort of control of visibility um into the process. It's not necessarily as dynamic, but we do have durable execution within a task. So durable execution is basically let's cache side effects. And if there's a retry, we don't redo the side effect, we just take the result of it. That's durable execution. So it's basically retries with caching. Um and workflow is a predefined set of steps, go through them, and there's you know, branching, loops, exception handling, multiple threads, and such. So workflow is is can be turned complete, and in fact, little horse is. Um, but it's very different from dubal execution, and we have dubal execution within little horse at the task level. Um, anyways, that is where little horse was was motivated. It's this problem, and there's so many other things that have come with it, which is you know, the little horse was based on Kafka. Um, it's it's actually uh Kafka is the commit log, and uh Kafka Streams is kind of the data store that we use. Um and there's many reasons why we did that. I I did a talk on current about that at current uh 2024. Um if if you're interested. I don't have time to do that.

SPEAKER_02 9:55

Maybe we'll put a link to that in the show notes. Uh the audience will absolutely be interested in that talk.

SPEAKER_01 10:00

Yes, and if you haven't ever seen Kafka Streams, don't know what Kafka Streams is, that's actually a really great talk. Because I spent the first probably 70% of the talk talking about Kafka Streams.

SPEAKER_02 10:09

Oh, nice. Okay.

SPEAKER_01 10:10

Yeah.

SPEAKER_02 10:10

Uh and let me just say, if because uh this isn't necessarily a show for data streaming people. So if you don't know what Kafka is, uh that's check that out. You know, it's this durable log. You probably know what Kafka is if you're listening to this. If you don't know what Kafka Streams is, it's a Java API uh that's you know a part of the open source project that lets you do stream processing on top of Kafka topics, all the usual kinds of stuff that gets difficult that you want to do with consuming messages from a log, you do with Kafka streams, right?

SPEAKER_01 10:39

So at the risk of getting toasted on Twitter, Kafka Streams helps you turn Kafka into a database.

SPEAKER_02 10:45

Nice.

SPEAKER_01 10:46

Well, it it it makes it possible to. That's not what most people use it for. Most people use it for well, I have a bunch of data in Kafka, I have a bunch of events happening, and I want to quickly derive some insights that involve joining a couple streams together, counting things over time windows, aggregating things, um and putting them into a key value store, which you might also refer to as a database.

SPEAKER_02 11:09

I yeah, it's uncontroll uncontroversial. Who who could toast you for that? Of course. Somebody will.

SPEAKER_01 11:30

And there's another really loud group of people who think um AI agents are already smarter than us and we don't need programmers. And then there's another group that says we need Postgres and no Kafka, and and Postgres can do everything that Kafka can do.

SPEAKER_02 11:43

I yeah, I I'm I'm we're not going to link to that in the show notes, but uh I I just I will say it is early November, November 3rd, 2025, as we're recording this. Very timely reference, Mr. McNeely. Um yeah, that that that Postgres instead of everything.

SPEAKER_01 12:00

You can build a queue on Postgres for small scale. Um, but you know, there's a difference between a cue and a log. And in fact, you know, many of the use cases for Kafka are not cues, but rather logs. I think um uh Gunnar Morling and Jack Van Lightly both wrote fantastic pieces on this recently, uh, I think in their own personal blogs. Uh both of the code. Yes, oh we will try to get links to Wilson Shadows.

SPEAKER_02 12:22

Gunnar and Jack are you just you need to read them.

SPEAKER_01 12:25

I read uh Jack Van Lightly religiously. He's he's fantastic. But uh, anyways, so well, we were talking about Little Horse being built with Kafka as the commit log. And the core scheduler of Little Horse is a Kafka Streams app. And we don't send tasks on a Kafka topic, actually. There's a GRPC API, and all the Little Horse clients interact with the GRPC API. Kafka's hidden as of a year ago, hidden from the customers uh or the users of Little Horse. Um, but because it's a Kafka Streams app, every time something interesting happens, we can very easily throw it into Kafka. So then we started throwing stuff into Kafka as things happen. So we throw a stream of updates to workflows, tasks, nodes, and such, very low-level business or sorry, not business, very low-level technical things happening. Like this workflow reaches this next step or this this variable was changed, its value, you know, the the variable order status changed from shipping to delivered in in this particular workflow. So now we have that data in Kafka, right? Imagine what you can do with that. So that might give you some insight into what we've been working on the last two or three months and is gonna be coming out pretty soon. Um and the other thing that we've been doing with regards to the Kafka ecosystem um is well, workflows are great at responding to things happening. So you'll have a bunch of events in Kafka, and you're not gonna run a workflow for you know a million events a second, but you could derive anomalies from that using Flink, uh, okay, SQL, Kafka streams, some, or you know, even to be vendor neutral here, you could even do something like what's it, click house, an enrich thing, and to detect anomalies, and then throw something into a Kafka topic that then triggers a workflow to handle it. So, for example, fraud detection, right? Uh, one of our uh more common use cases is well, there's some cybersecurity violation or some anomaly, some SLA is missed, something like that. We uh create an enriched event and then react to that in a workflow uh with Littlehorse. So we have a two-way connectivity between the event platform Kafka and Littlehorse. The events trigger workflows, workflows produce events.

SPEAKER_02 14:36

Can I clarify there? So there's there's two kinds of events. Uh well, I'm gonna divide the world into two kinds of events. Okay. One is the business events that my application uh creates. Somebody buys something, uh, payment clears, uh, thing ships, you know, that kind of stuff. Microservice emits event to you know, record completed work. The other kind of event is the workflow engine or durable execution engine has made progress somewhere.

SPEAKER_01 15:04

Correct.

SPEAKER_02 15:06

Are you talking about anomaly detection on that second category of events?

SPEAKER_01 15:10

Well, uh, I you're correct that those are two two way, two places that events can come from. Um, but I wouldn't necessarily say they're different types of events, right?

SPEAKER_02 15:19

Okay, that that kind of answers my question. You're viewing this as there's a world and I think there's low-level events and there's high-level events.

SPEAKER_01 15:27

Like the a click is not really that high level of an event, right? And and uh, you know, people throw um what is it, they throw logs and and OTEL traces into Kafka. That's a very low-level event, right? But you can process that, detect that there's a lot of errors, and then you throw, oh shoot, you know, there's there's some outage of some sort. Let me page the engineering team. That's a higher level event. And um, you know, Confluent makes the very eloquently points out that Kafka doesn't really care between high-level and low-level events. You need a data streaming platform to give you that difference. You need stream governance and you need to understand the schemas of these things and you need a way of differentiating them, right? Um, but they're still just events, you know, big events, little events, you know, biggie, little y doesn't really matter. And and the idea is that the workflow engine can react to events. We have Kafka connectors actually just got certified on Confluent Hub, which is really exciting. Um that read from Kafka and trigger workflows in Littlehorse, and then Littlehorse produces more events back into Kafka. Um, and that's all open source. The connectors open source, the the workflow engine open source. What we're working on in our enterprise product, because I gotta make money, uh, is uh uh a thing that will take the little events coming out of the server and the the workflow engine and turn them into big events that are business meaningful.

SPEAKER_02 16:49

Okay. Okay. Um I've got some uh more questions about the um about both the workflow engine and the the durable execution stuff that that are I'd like to dig into. I don't want to uh rabbit trail you yet. Was there more uh am I am I cutting you off? You want to go forward on some more of that stuff, or shall I just dive in?

SPEAKER_01 17:11

Um I think we've uh yeah, I think we can go forward.

SPEAKER_02 17:17

Now a quick word from our sponsor. Confluent developer the podcast is brought to you by Confluent Developer, the website, which has everything you need as a developer of data terming systems, and it's completely free. We've got curriculum, hands-on exercises, tutorials, online data training on our architecture, also free, a way to find our web arrow, also free, or everything. I really want you to our journal as a data term as a site that has to check out our developer.confluent.io. That's developer.confluent.io. Now back to the show. There are some things to solving a problem like this that are uh uh thorny. Now the durable execution side of things um that's just uh depending on the the granularity that you you go for, it's either you know kind of a uh uh not too hard of a problem or a moderately technically difficult problem. So it's just kind of hard to do. The workflow engine is it's uh it's software, it's all hard. There's all lots of corner cases, and none of this is easy. But you also have a problem of how to express the workflow. Um because uh to some degree writing business software uh in you know language, um whatever, like you know, it's it's uh it's uh Rust if uh you know you believe Twitter, uh that corner of Twitter, or for most business software, it's job. Um and and I'm fine with that, that's my own background. But you're you're that's a workflow. You're describing a workflow, you're just doing it uh in a programming language. So where uh uh how did you figure out where do you figure fit in that curriculum or uh continuum before uh 20 years ago there was uh well probably 25 years ago now, a flavor of XML um uh that that was that you could use to describe uh uh business workflows, and it was just as bad as that sounds. So uh what do you do? How do you how does Little Horse describe workflows? How did you think about solving that problem? Uh how do you interact with your users and community and everything? Just talk to me about that.

SPEAKER_01 19:29

Uh well, users and communities and like what's our Beverel posture and such?

SPEAKER_02 19:34

I mean, well, no, like whatever, however it is you interact with customers and and users around that that language.

SPEAKER_01 19:40

I think this gets solution. This gets quite philosophical, actually. And we have a lot of discussions inside Little Horse itself about it. And then, you know, the what we decided at not decided, but what we thought, our our customers actually told us something a little bit different, which was at first we thought that the workflow language we built, it is turning complete. It's got variables, it's got conditionals, loops, it's got exception handlers, error handlers, interrupts, child threads, all of those things that you would expect in a programming language. We have so we thought, oh, people would use it like a programming language. But it turns out uh most of our users write, they do something with little horse that you can't do with durable execution. What they do is they write code that mirrors a business process, kind of business's code, not infrastructure as code, business as code. Uh and and then they get a good, you know, there's better alignment between the engineering teams and the product teams with, you know, business as code. Um and certain things that, you know, they uh certain things that are pretty technical and and kind of you know, like if-then logic that is not really mirrored at the business level was was kind of creeping in there. Um and people were like, yeah, it's it's a little messy, but overall you can kind of see the business process in this, what we call a workflow spec, um, this drop by DSL. But there's a little bit of, you know, that looks like data massaging. So what we've done and what we're working on with some of our users is we developed what we call check-bounted tasks, which are literally just durable execution. So it's it's basically what temporal does. Um, and that is available inside a task in Little Horse. So now what we're doing is we're we're recommending that we take, you know, the workflow spec, the WS spec represents a business process and the technical details are done by the tasks, um, which uh is is pretty nice because now you you have what I was complaining about not having when I was an intern, which is a picture of what's happening and where in the business process at a higher level.

SPEAKER_02 21:34

And the tasks are always code. I have to write those in a JVM language.

SPEAKER_01 21:39

Uh Java, Python, Go, C sharp supported JavaScript. If we get, you know, if I every time I open Twitter, I think we need JavaScript, but it you don't see many people writing enterprise backends in JavaScript in the wild.

SPEAKER_02 21:52

Uh it will be uh a minority player in that in that pie chart.

unknown 21:56

Yeah.

SPEAKER_01 21:57

So the workflows can be expressed in code. We're also working On a visual workflow builder because we have a language that it, you know, it's like uh workflow.execute this, you know, workflow.declare variable, and then you have variables that you can pass into tasks, you can take the output of one task, pass it in the other. So it's it's very similar to writing code with like, you know, a durable execution framework. Um and it's almost exactly the same writing writing the workflow. Um, but under the hood, the the thing that is different architecturally from dubal execution is that that compiles down to a directed graph. Directed, it's not acyclic, it's just a directed graph, which is just nodes and edges. Boxes.

SPEAKER_02 22:34

Oh yeah, fair enough. If it's Turing complete, it's it's not acyclic.

SPEAKER_01 22:37

Correct. I like the live fact check. But because it is a graph, right? And and these are, you know, now with this whole the durable execution inside the tasks itself, what we can do is, well, it's it's a high enough level thing that it might actually make sense to just create a lightweight UI that lets you build workflows without writing code and see if it's usable, right? Or see if people people like it. You know, the our current users are very um, you know, generally we we've got some people who um have have migrated off of uh Comunda um because they they went the they you know we started out server-side public license and then went to fully open source HPL and they kind of went the other way. And they're you know, we're catching some of that. And then you know, some people frustrated with MuleSoft. So they are used to clicking and dragging, but they actually really like the the SDK, uh, which was surprising because they're moving off of the click and drag thing. But you know, we might as well just try and see who uses it because we can.

SPEAKER_02 23:37

Uh yeah, yeah, yeah. Um you said uh at one point this becomes philosophical. Could you expand on that a little bit? I'm I'm intrigued by that statement. Uh I wonder if you mean what I think you mean.

SPEAKER_01 23:51

Okay, it so the it becomes philosophical. Uh you're roughly asking, what is the um, you know, do we write workflows in XML? Do we write workflows in code? What I was saying is a philosophical discussion is um where is the delineation between what's a task, what's one workflow, and what's the interaction between multiple workflows together, or workflow specification to be in very precise and little or strength. What's what's a task run, what's a workflow run or workflow spec, and what are multiple workflow runs composed together? Uh that is a philosophical discussion. So um you can, and we have seen live in in in you know one of our users, a workflow that is a loop that like every certain amount of time runs a task to check for some uh condition in the world, and and then if it is you know out of line, it it runs some compensation thing. So it's like a loop that just kind of runs forever, right?

SPEAKER_00 24:48

Yeah, okay.

SPEAKER_01 24:49

There's other ways to do it. Is that the best way to do it? Not really, right? But that's not really a business thing. That's literally like a it's checking the validity of a token, uh, which is not a business process, that's a technical process, right?

SPEAKER_02 25:01

That's yeah. And you may not not nothing's going to produce an event when the token expires, and so you know, you pull.

SPEAKER_01 25:07

Correct. Uh that's very, very much durable, you know, very much a use case that you could do with durable execution, or you could have a cron job that that does that and and you know. Um and well, we we do have a cron run workflow capability, which probably would be a better way to do it than have a loop running forever. Um but you know, that that works, and and and we're making the server support that a little bit better. Um but on the other end of the extreme, we've we've got someone who's building a work order management system where there's one workflow for managing a work order, right? That's very different than a workflow that checks the validity data token. Um, and philosophically, on one extreme, well, oh, you know, I'm just running a bunch of tasks on different computers, right? Another one is, well, hey, this workflow models a very core business process. Um and there's steps including human in the loop steps, there's you know exception handling uh and error handling. It could we call exception handling something the business going wrong, like a credit card being out of funds. Uh error handling is something like you know, 500 or API is down, right? And we handle those errors differently.

SPEAKER_02 26:13

Um okay. An exception is a is a business process exception. An error is something broke.

SPEAKER_01 26:20

Something broke, you know.

SPEAKER_02 26:21

A computer broke, basically.

SPEAKER_01 26:22

Correct. Or or you had a bug in your workflow, like you passed an int when you need a you know string or something like that. Actually, that that would work, but uh, you pass a string when you need an int.

SPEAKER_02 26:32

There you go. May and it may not work, right? Yeah. Or yeah, it would not work if yeah. I'll I'll just I'll throw something out. Tell me if uh if this resonates or if you thought this way. In in my mind, uh business software and and and business workflow engines get at the absolute heart of uh business software. Um because they take uh a process that a business does and they they try to encode it uh literally. Um and I'll put that a different way, uh a somewhat more philosophical way. Um if you could imagine a a pre-automation business, some company in 1950. Um plenty of businesses don't uh do a lot with computers, uh, but just just any any company doing stuff. You got an office full of people and um you know they pass information around, they have forms, they have memos, there's files, and uh they do stuff. They interact with customers and with each other, and the business process is mental. Uh it's stuff that they learn and do um in their mind. And uh uh you yeah, you have to learn, but you get good at it and and you do it, and uh there can be exceptions like, oh, this person didn't fill out that part of the form, but it's okay, I know what they mean, or they didn't uh fill this out, but they've had this, and I can kind of guess. And all the kind of messiness of uh doing stuff as a person or like you know using language is is the same way. There's uh there's all kinds of stuff that's vague and and and unclear and implied, but we get by. Uh so business is this mental world.

unknown 28:08

Uh uh.

SPEAKER_02 28:08

And then when we take that and we and you know, this has been going on uh at a rapid pace in the last 30 years, um uh you take those processes and you try to turn them into uh machine instructions, basically, in whatever language you want. That's really hard. Um it's like it's like a kind of alchemy. You we as developers, we do this magic where we take these business processes and turn them into uh machine instructions or or formal language, uh expressions in a in a formal language. Um doing this with uh uh so I'll say it again. That's super difficult. It's uh not everybody can do that. Uh and the people who gain the ability to do that, well we call them software developers in the world of uh uh enterprise software, uh and we use these uh extremely open-ended, general-purpose turn compute turn complete programming languages for that. Uh then you have the early attempts like BPM of of doing it in XML, and you pretty much just end up reinventing a uh turn compute turn complete language that's just really ugly to look at.

SPEAKER_01 29:15

Um and also single point of failure, slow uh uh well of course, yeah. There were architectural problems with the engines that the 1990s workflow, very different than you know, like the Communda V8 and Little Horse and Temporal.

SPEAKER_02 29:32

And you know, we uh we're doing the best we could with the tools that we have. Um so uh uh not maybe not last question, but among my last questions. Uh that uh workflow language you called, I forget what you call it, you what you what little horse calls it, but that way you express the Java DSL. Yeah um are you successful in keeping that? Do you think you can be successful in keeping that from just becoming another programming language? Can you can you can you can you do it?

SPEAKER_01 30:09

You know, we're we're joking about uh a project called Hoof internally. Uh there's a lot of like we're about to release a product called Saddle, and and uh we we're working on renaming something Pony ID.

SPEAKER_02 30:21

I like this kind of Western theme.

SPEAKER_01 30:23

This is our serverless cloud is gonna be called Pony Express. Uh so we're we're thinking about Hoof as a as a um you know, as a language that there's a compiler that takes that language and compiles it directly and runs it on the little horse server, just like Java is compiled by Java, Java C and then run on the JVM, right? It'd be exactly the same thing. So why are why are we you know trying to keep it? Well, because the Python, Java, C or C sharp and and uh Go DSLs are working just fine. And people have uh they've given us feedback. We've been able to, you know, most of the feedback we've got, almost all the feedback we've gotten is minor bells and whistles, not an architectural fundamental complaint. So we just haven't built the actual uh you know programming language itself. But you know, the the key thing you said is hit the nail on the hammer, which is you know, there's this kind of emergent business process, and there's a lot of manual steps, and there's data that is moving, data that's getting lost, especially when there's not automation or there's stuff getting thrown into legacy systems or even modern third-party SaaS systems that don't have great APIs. There's data that you're not capturing and unable to put into your analytics engine and and such. So to go map that process into something technical is very valuable for many reasons, because you can, you know, uh do things faster, you can get more analytics out of it, you can automate things. But um that process is is difficult because there's a lot of asynchronous passing things between between different nodes, where a node is a person, it could be now an agent, it could be an external API, like you throw something to uh Manhattan Associates to do your uh order processing, like you call it Active Omni, right? Then some days later they'll throw back an event to you to say that this order has been shipped, right? Then you go take that information and then you throw it at this a system called uh you know XCC, which will calculate some payment information and you know, and then you send it over to Oris, and then you know later on, Ouris tells you that the payment is done. So you're you're taking data and you're chucking it somewhere, waiting for it to come back, right? And if you if you try to write this process in in normal Java, um you're gonna end up with a bunch of spring boot endpoints, either on one server or on different server that accept a webhook from something or read an event from somewhere, do something, and then throw the event somewhere else. So you end up with little disjoint pieces of this logic spread over a bunch of different endpoints. And those endpoints are often now spread over a bunch of different microservices. There's nothing that allows you to durably write something that says, okay, now what I'm gonna do, I'm going to initiate the order process in Active Omni and I'm gonna wait for Active Omni to send something back. If it doesn't send something back in a certain amount of time, I'm gonna go ask the customer, hey, you know, do you still want me to wait because you know they're perhaps the item is out of stock or do you want to cancel your order? Right. And then we wait, and depending on that, then we'll go to Kobe, uh, you know, uh calculate the uh the price given loyalty points and such, then you know, send it to orders for payments and such. And and all of that lives in one place and then is orchestrated by one thing, the server. And the work actually doesn't run on well, the server is what we call uh little horse. It's an internal word, I guess we might make it uh there's many things called the server, but naming naming things is hard, you know, the servers in it as well.

SPEAKER_02 33:48

Like, oh server, I see.

SPEAKER_01 33:50

Name it. Um but in contrast to sorry, all that's orchestrated by the the workflow orchestrator in little horse, and you get to write code that that um corrals that process and handles handles things, make takes care of retries for you, takes care of making it really easy to handle exceptions, and it it runs in a non-blocking manner across many different computers. So you you could have uh complex processing that requires a GPU as one of the tasks, and the task would be sent to a worker on that physical infrastructure. And you could have just you know plain old just fetch something from a database or call this API uh running on other task workers, other places.

SPEAKER_02 34:29

Got it. This brings us back to your original insight, which was you know, you could debug operating systems projects in school with GDB and VS Code and see it all. And um, in a modern, you know, just somewhat complex, not hello world enterprise environment, there's all those services and and you're getting. So yeah, maybe it does become just a programming language, but you've got this one view over the whole thing.

SPEAKER_01 34:58

Exactly.

SPEAKER_02 34:59

And it is. I mean, you've you started by saying it was Turing complete, but uh it's a simplified language. Uh yeah.

SPEAKER_01 35:07

Our website used to say a distributed operating system for the enterprise. Distributed as in it an operating system, basically you tell it to run a program and it will assign that program to run on a certain cores, right? And yes, uh with a single operating system program, you can debug it and you can see exactly what's going on. Um the idea of calling little horse a distributed marketing operating system, which is very much a marketing term, um, is that well, this program runs on little horse, but it runs on many different servers that are distributed. And in fact, certain steps of it can can involve human input as well.

SPEAKER_02 35:44

They and they could have nothing to do with little horse resources. They could be Salesforce, it could be uh, you know, uh Pinterest or whatever, you know.

SPEAKER_01 35:54

Yes.

SPEAKER_02 35:54

Yeah.

SPEAKER_01 35:55

Exactly like if it's something you can interact with through code, uh, you can have it as a task in Little Horse.

SPEAKER_02 36:02

There you go. To close, uh, you're a startup, but you've talked about customers. What kind of impact is this having having? What are you seeing this uh unlock? You know, you're still building it. There's a vision I sense that you haven't achieved yet. Correct. Uh but you've got product and what change is it bringing about in the world?

SPEAKER_01 36:22

Um I think you know, uh the the goal is if I ask anyone on the street, well, not on the street, if if I go to current and I ask anyone, what do you think of when you think of relational database? They'll say Postgres, right? Um and then I ask, what do you think of when you think of cloud storage? You'll almost always get S3. You might get um you might get like a GCS or something, right? Sure.

SPEAKER_02 36:47

And somebody it's a database, you might somebody might say Oracle, and you'll be like, nerd. Well, I really But those are those are the associations we have.

SPEAKER_01 36:57

Exactly, yeah. And I think what do you I ask you, what do you think of for cache, right? Everyone thinks Redis, right? Yeah. Um, or um for a search index, you'll think of Lucene or Elasticsearch. Now I want to be able to go to current and say, well, what do you think of when you think of distributed app management? Or what do you think of as a solution to writing applications in a distributed environment? I want people to say little horse, right? People would say, what do you mean managing distributed applications? They just accept that it's so hard to do this. They don't really think about uh the fact that there might be a better way to do it. And I want LittleHorse to become so mainstream that uh you know the open source project is used ubiquitously.

SPEAKER_02 37:34

My guest today has been Colt McNeely. Colt, thanks for being a part of the ConfLM Developer Podcast.

SPEAKER_01 37:39

Thanks, Dan.