Streaming Audio: Apache Kafka® & Real-Time Data

Event-Driven Systems and Agile Operations

• Confluent, founded by the original creators of Apache Kafka® • Season 1 • Episode 225

Share episode

How do the principles of chaotic, agile operations in the military apply to software development and event-driven systems? As a former Royal Marine, Ben Ford (Founder and CEO, Commando Development) is also a software developer, with many years of experience building event streaming architectures across financial services and startups. He shares principles that the military employs in chaotic conditions as well as how these can be applied to event-streaming and agile development.

According to Ben, the operational side of the military is very emergent and reactive based on situations, like real-time, event-driven systems. Having spent the last five years researching, adapting, and applying these principles to technology leadership, he identifies a parallel in these concepts and operations ranging from DevOps to organizational architecture, and even when developing data streaming applications.

One of the concepts Ben and Kris talk through is Colonel John Boyd’s OODA loop, which includes four cycles:  

  1. Observe: the observation of the incoming events and information
  2. Orient: the orientation stage involves reflecting on the events and how they are applied to your current situation 
  3. Decide: the decision on what is the expected path to take. Then test and identify the potential outcomes
  4. Act: the action based on the decision, while also involves testing in generating further observations

This concept of feedback loop helps to put in context and quickly make the most appropriate decision while understanding that changes can be made as more data becomes available. 

Ben and Kris also chat through their experience of building an event system together during the early days before the release of Apache Kafka® and more. 

EPISODE LINKS

Kris Jenkins: (00:00)
Hello, you're listening to the Streaming Audio podcast. And on today's episode, we have an old friend and colleague of mine, Ben Ford. Ben's a programmer, but he's also something of a systems design thinker. And during this chat, we managed to bounce around a lot of different topics, drawing parallels between event systems design and military organization, agile, both the good and the bad kinds of agile, psychology, how we interpret the world, music composition. It's is a real smorgasbord, this episode, and it all ties back to our world of event streaming. I'm really glad we got Ben on the show at last, because I always love talking to him. He's always interesting to me. I hope it'll be interesting to you. Before we get started, let me tell you that Streaming Audio is brought to you by Confluent Developer, which is our site that's jam-packed with information on how to build successful systems with event streaming and Apache Kafka.

Kris Jenkins: (00:58)
Whether you want to get started or dive deeply into the internals of how Kafka works, there's information there to help you. And when you need to get Kafka up and running, take a look at our cloud service, Confluent Cloud. You can get a free account and a Kafka cluster up and running in minutes, and it will scale all the way up to ludicrous sizes. And once you've got an account, add the promo code PODCAST100 on the billing page and you'll get $100 of extra free credit. And with that, I think it's time for Ben Ford to tell us how he got started as a programmer. My guest today is Ben Ford. Ben, welcome to Streaming Audio.

Ben Ford: (01:42)
Thanks, Kris. I've been really looking forward to this one.

Kris Jenkins: (01:44)
It's good to have you. We haven't spoken in far too long because we used to work together, right?

Ben Ford: (01:48)
Yeah. Twice actually. It has been ages, hasn't it? Apart from the odd sort of WhatsApp, it's been yeah, God, an uncomfortably large amount of years.

Kris Jenkins: (01:56)
Yeah. But we met even longer ago than before all that. And to frame this, where we're going, let's start here. I learned programming by reading a book on how to program in my bedroom, which in a way is similar to you in a way really, isn't it? Tell people how you got started with programming.

Ben Ford: (02:18)
So back in 2003, I was serving the Royal Marines on my way to the invasion of Iraq, called Optellic. I was rather bored on board a ship called HMS Ocean. So I had two books delivered. One was on Linux, one was on Python and I basically sat below decks all the way through the Meds and down past Saudi Arabia and banged my head against my laptop until I figured out the basics of Python.

Kris Jenkins: (02:47)
Banging your head against the keyboard as a way of relaxing from the day to day.

Ben Ford: (02:51)
Pretty much. I mean, I don't think my programming journey's ever really been any different than banging my head against the keyboard. I guess it's the same for everyone else, right?

Kris Jenkins: (02:59)
It's a very familiar experience.

Ben Ford: (03:00)
Yeah. At least most people have internet to go and like check what the problem was, and I didn't, I just had the book, so I literally had no option other than banging my head.

Kris Jenkins: (03:09)
Oh, God. Yeah. There's not much support slightly underwater in the Mediterranean, is there?

Ben Ford: (03:13)
Not really, no.

Kris Jenkins: (03:16)
So how long did you spend between there and actually becoming a professional programmer?

Ben Ford: (03:24)
So that was 2003. I left the Marines in 2004 and I just, I knew the basics of programming, but not enough to be a professional software engineer by any means. So I just worked little bits of scripting into different jobs. I found myself in New Zealand working for a telecoms company and I was one of the field engineers and we had no maps of how to get to anything. So I taught myself a little bit of Jengo and built a little web app for the field engineers to find their way around. Then I did a couple of other odd zones. And by the time I'd moved back to the UK in 2008, I was good enough for somebody to take a punt on me as a full-time engineer.

Kris Jenkins: (04:07)
Is that how you got into the banking world?

Ben Ford: (04:10)
So that was interesting actually. So I definitely wasn't qualified for banking either. So after a year or two of working for that company up in Oxford, I wanted to get into financing to get into the city. One of the things I'd done with Jengo was basically pull it apart and figure out how to make it work with multiple databases, way back when I was doing this telecom stuff. And there was a hedge fund in London that had, they chosen Jengo to replace an internal tool and they needed it to talk with multiple databases and it didn't do that at the moment. So there was a bit of direct experience that kind of made up for the lack of any kind of math skill. Yeah. So I kind of blanked my way into that. And then once you've got your first sort of foot in the door in the finance world, especially with a big name like that, the rest of them become a lot easier.

Kris Jenkins: (05:04)
Yeah. Yeah. And I don't know about programming for a bank. Sure, you need some maths, but the ability to wrangle the impossible seems far more about the day to day, right?

Ben Ford: (05:14)
Well, yeah. I mean, we can certainly get onto technical problems not turning out to be technical at all. I'm sure that's what we'll end up spending most of our time talking about.

Kris Jenkins: (05:22)
Yeah. So from banking, at some point, I know you got into things like functional programming and event streaming. Tell me how that came up.

Ben Ford: (05:33)
So after that first hedge fund, that was an algorithmic hedge fund. So I went to work. I had a short stint at an investment bank, and then I went to work for a startup hedge fund, which was also an algorithmic hedge fund, but very, very purist in how they thought of trading signals and things. Along the way somehow my, I'd started being maybe less satisfied with the object oriented way of programming and just becoming more and more functional. Python has the basics of functional programming built in. And as that kind of style had developed, I'd got some advice from people that there's a whole sea of stuff here that Python is a very thin slice of what functional programming actually is, and if you want to learn it properly, you should look at Haskell. So I did. Completely alien, weird syntax, but quite captivating because I've already been sipping the Kool-Aid a little bit. So I wasn't allowed to use Haskell at my hedge fund, but I was sort of teaching myself out of work hours. And I did a little play around with some time series analysis

Ben Ford: (06:50)
The mind blowing moment was having very little experience in this language, wanting to read some stuff from a database, display it as a time series, banging my head against it for ages, probably two days on and off, just figuring out how the types work together, then finally get it to compile. And this is my first experience in a compiled language, and obviously from Python background, expecting now, okay, now I just need to work through a long list of bugs and make it actually do the thing I want it to do. And when I ran it, it did the thing I wanted it to do immediately because I'd made the types line up and it was just like, pew. So that was it. I was sold from that point.

Kris Jenkins: (07:32)
Yeah. Yeah. I think that's one of the things that's always appealed to me about Haskell is like you get all your pain upfront, but at least you deal with it early and you don't deal with it so much later, right?

Ben Ford: (07:43)
Yeah. And it's very principle pain. You only get pain when you don't adhere to the principles that the laws of nature and mathematics tell you, you have to adhere to. So it's kind of like [inaudible 00:07:52] in a way.

Kris Jenkins: (07:54)
Yeah. It sort of forces you to think in a particular way, but once you get there, you start agreeing with it. I don't know...

Ben Ford: (07:58)
Yeah, or we both completely indoctrinate it, one of the two.

Kris Jenkins: (08:01)
Yeah. Yeah. It's either doing things the right way or it induces Stockholm syndrome.

Ben Ford: (08:07)
Or both.

Kris Jenkins: (08:09)
But you've had those couple of years half underwater, banging your head against the keyboard. So you're ripe for learning Haskell.

Ben Ford: (08:16)
Oh, absolutely. Yeah. I was conditioned almost.

Kris Jenkins: (08:22)
Yeah. So that takes us to kind of about the time you met. Right. Sorry, you and I met.

Ben Ford: (08:29)
Yeah.

Kris Jenkins: (08:29)
Because, well, I remember first meeting you at a company called Finder. I think we're allowed to mention the name.

Ben Ford: (08:35)
That's right.

Kris Jenkins: (08:36)
And I'd been doing a lot of functional programming in those years, but this was the first time I'd seen event systems. So tell us how you got into doing that and why you did it that way.

Ben Ford: (08:48)
So Finder was a, it had a grand vision of being like an Uber for time slots, right? So rich executive moves to a different city, wants to do a Pilates class or something and has an app that they can go and find it all, hence the name. One of the things that I'd learned in the process of building this algorithmic trading system is if you separate things into the things that have concretely happened, and you touched on this in one of your earlier episodes, which I really enjoyed, and the thing that you said there was, "Don't store the conclusions, store the facts," and functional programming makes you very much think of immutable facts and it's a fold in our language, right?

Ben Ford: (09:36)
So I'd already been thinking along these lines. And then when the founder of Finder, sounds a bit weird, but when the founder of Finder explained the vision, I was like, "Well, you really want to think about this, that the architecture that you need for this is to do it this way so that you can ask the questions that you don't know later on and you can much more appreciate the properties of the system. I think if I was to build a team of Haskell engineers, that we would find really incredible people and it would make... It's the right tool for the job." If you remember, the team that we had there, that was absolutely the case. Haskell is a massive talent magnet. It still is now, but back then, it was even more the case, right?

Kris Jenkins: (10:25)
Yeah. You could get the top of the line Haskell programmers who would work at a discount just to be able to use it as their day job, right?

Ben Ford: (10:32)
Exactly. Yeah.

Kris Jenkins: (10:35)
Yeah. Looking back on that gig, I should have asked for slightly more money, but yeah, we got into that thing of just record the facts and then fold, or in Kafkry language, do a transformation on that stream of events and gradually build up the picture you actually need to run the business.

Ben Ford: (10:55)
Yep.

Kris Jenkins: (10:56)
And it was actually, you are the starting part of the causal chain that leads me to be working for Confluent these days because that was the first time I'd seen event systems.

Ben Ford: (11:08)
[inaudible 00:11:08].

Kris Jenkins: (11:08)
Yeah. You set in motion the first domino.

Ben Ford: (11:12)
Yeah. Well, so what was really, really interesting about that was that we had that in the back end and if you remember, you took over from a guy that had built a front end in Angular.

Kris Jenkins: (11:22)
Yeah. And this was Angular one, right?

Ben Ford: (11:24)
Ah, I can't remember.

Kris Jenkins: (11:25)
I'm sure it was. It was the version of Angular that they gave up on.

Ben Ford: (11:30)
Right, but I think that the more important thing than the technology was just the difference in understanding of the possibilities of that architecture. Because I remember talking to you when we first met, literally the interview, and the moment I thought this is going to work is when you were like, "Ah, that's a really interesting architecture." Then I knew that you would want to kind of take that into the front end as well. And we ended up with obviously a completely realtime, not hard realtime, but a realtime front end where the state of the world was always consistent, no matter what device you were looking at. And that was all the function of picking the right abstraction, really.

Kris Jenkins: (12:12)
Yeah. Yeah. It was the first time I'd seen an architecture like that was like, we send facts in, they get transformed, they get streamed out in realtime over web sockets into various front ends. I've actually written a blog post about this like reflect [inaudible 00:12:26]

Ben Ford: (12:25)
I read it. It's great.

Kris Jenkins: (12:27)
We'll put a link in the show notes. But it was the first time I'd seen the potential of that kind of live system. Because we talk about a lot on this show about the difference between soft realtime and batch processing. But looking back, this is the first time I'd seen a website that wasn't batch processing.

Ben Ford: (12:48)
Yeah.

Kris Jenkins: (12:48)
It was a living reacting to fact event system.

Ben Ford: (12:53)
Yeah. And I mean, the nice thing about building things in an event system is that you can always batch because these things have very well understood semantics about composition. So you can always go from an event driven system to a batch system because you just chunk up your inputs and run, it's a one line of code once you've got the underlying event system written, but trying to go the other way from a batch system or from a world where you've thrown away facts and stored conclusions, you've lost the ability to do that backwards transformation.

Kris Jenkins: (13:27)
Yeah. Yeah. I know a lot of stuff we do internally at Confluent is about trying to gradually... You can't bolt on realtime onto a back system and you don't want to do a big bang rewrite. So a lot of the work we do ends up being finding ways to gradually bring in a realtime system to an existing back system and make them work in harmony.

Ben Ford: (13:51)
Right.

Kris Jenkins: (13:51)
Which is...

Ben Ford: (13:53)
I don't know if you remember the bit of work we did at Finder to extract... Once we got the system built, we wrote a thing to extract data from an old system and transform that into an event stream that would be applied to the new system. Do you remember that?

Kris Jenkins: (14:08)
Yeah, I do. I do. We were trying...

Ben Ford: (14:10)
We were. So that was another...

Kris Jenkins: (14:12)
... to migrate from the old.

Ben Ford: (14:12)
Yeah. That was another really interesting property of having those clearly defined semantics in the new system is that you could then write a script that turned the old system into an event stream from the data that you had, so a database that had timestamps of stuff, and you could catch all of the logic errors that you had in the old system, like an instructor that was instructing two classes at the same time.

Kris Jenkins: (14:37)
Yeah.

Ben Ford: (14:38)
So you could basically find out where things didn't clearly apply, but that was super interesting as well because we were able to use the core semantics almost like a really rough and dirty scripting thing around the outside that quality didn't really matter because the quality of the code didn't matter so much because the quality of the data was never in doubt because it had to pass all of these constraints before it was allowed to write into the new system.

Kris Jenkins: (15:07)
Yeah. Yeah. Once we captured those facts, you could always reconstruct the state you were looking for experimentally until you got it right.

Ben Ford: (15:16)
Yes.

Kris Jenkins: (15:18)
Yeah. It was a very interesting system. In a way, I think it was.... It's tempting to say it was ahead of its time, but looking... At the same time, you've got Kafka being built because this is around 2014, right?

Ben Ford: (15:32)
Yep.

Kris Jenkins: (15:32)
Unbeknownst to us over at LinkedIn, they were building Kafka, spinning out to Confluent.

Ben Ford: (15:37)
Actually, I think Kafka, there was an early version of it available. The reason that we ended up going with Postgres was because we used the constraints in Postgres as part of the checks when you want to apply the facts. So if you remember, we had commands that would come in via one channel. Those commands would generate a series of events based off what the current state of the world was. So I want to book a class, but I don't have enough credits means that you would get a certain event stream. If I do have enough credits, I would get a different event stream, which included a class booking. We used the constraints in a [inaudible 00:16:25] normalized Postgres schema as part of the semantics of whether the events could be applied. And then obviously, we also had for the web socket side of things, we had the ability to subscribe to Postgres events. So it wasn't, I don't think... If you look at what's available in today's world, I don't think you would consider that a full kind of streaming system. Event sourcing, yes, but you'd [inaudible 00:16:54].

Kris Jenkins: (16:54)
Event sourcing on top of relational, right?

Ben Ford: (16:56)
Yeah.

Kris Jenkins: (16:57)
Yeah.

Ben Ford: (16:58)
And we could fold the events into anything else. Like I think I... What did I do? I folded it into a couple of systems, I think, just to do analytics, but it's less of a... The event stream was the database, whereas I think you'd probably consider that slightly different than building a Kafka from the ground up system, from what I understand of Kafka.

Kris Jenkins: (17:25)
Yeah. I think I've always thought it would be interesting to rewrite that system with today's technology and see how it played out. Maybe somehow I get to do a prototype of that one day, but moving on...

Ben Ford: (17:38)
That would be cool.

Kris Jenkins: (17:39)
Yeah. It would. There's a lot more support for building that kind of system today. Yeah. But moving on from that. So then finder for reasons, I don't think that were technological finder didn't work out and that's a story for another day, but you then moved on into some more banking gigs. I know a couple of government gigs, right?

Ben Ford: (18:03)
Yep. Yeah. So the banking, the banking gigs came, came first, went back into banking. And then I went into, you know, after a year or so after finder, I moved into a bank that was heavily functional programming, you know, into a heavy functional programming team. And yeah, that was a massive eyeopener as well, which we probably also don't have time to dig into. But then obviously we had our, after a year of that, I took over the kind of DevOps side of the front office there. And that's where you and I had our second stint working together. Actually, no, we've had three stints working together. Haven't we? Because then we did the blockchain stuff after that.

Kris Jenkins: (18:44)
Oh God. Yeah, yeah. We could really get into a rabbit hole with that.

Ben Ford: (18:51)
Yes we could. Let's not.

Kris Jenkins: (18:52)
Yeah. But the banking one again, it was like, how can we apply event systems to the world of DevOps and continuous integration?

Ben Ford: (19:01)
Yep.

Kris Jenkins: (19:02)
And the architecture ended up being very similar in a way.

Ben Ford: (19:05)
It did. It wasn't quite as rigorous, was it? But it was very much a, we have a bunch of stuff that's happening in the real world and we fold over that to come up with the state of the world and it absolutely suits. I mean, obviously in that context, we also had the paralyzation of all the test suite that we had to do as well. So that's almost like, well, yeah, I guess it's kind of unfolding, I guess. So I mean, everything in functional program is either a fold or an unfold. I learned that from Ollie at Finder and yeah, we use both in the DevOps thing.

Kris Jenkins: (19:40)
Yeah. You either roll up facts or you split the facts into groups and roll those up.

Ben Ford: (19:45)
Exactly. Yep.

Kris Jenkins: (19:45)
Yeah. [inaudible 00:19:49]. It's funny how you get all these parallels between systems that seem disparate. Like you think that fold, that you think that the world of functional programming is a completely different thing to the world of building event systems, but it's really not. And there's parallels in DevOps and the way we do front end and all that stuff.

Ben Ford: (20:14)
Yeah. Obviously, at the time in the bank, another sort of gig that didn't really end particularly well, I'd also been starting to dive into up until, I don't know, probably 10 years after I left the Marines, I thought, "It's a cool line item to have on my CV. Gets people attention every now and again.' And it was a fun thing to do when I was in my twenties. But after those two gigs, I really started to dive into... I'd started to make some early equivalents between how things are done in the military and how things could be done differently in tech companies. For the last, I don't know, five years, I've really, really been diving into that. And there's an awful lot of military doctrine and things like that, that also look pretty decent with a functional programming slash event sourcing lens applied to them.

Kris Jenkins: (21:12)
Now, take me through that because I know by this stage, you are a seasoned programmer and you've been doing a lot of project management stuff. Here you are. You've got definitely a functional programming, definitely an event systems mindset, draw the line from there to military operations.

Ben Ford: (21:31)
Yeah. Okay. I might have to unpack a few things first, please.

Kris Jenkins: (21:34)
Yeah. Take your time. It's a big bridge to build.

Ben Ford: (21:37)
Yeah. So the stereotype of military operations is what people say is planned with military precision and you think that you have this kind of very top down hierarchical, that the general gives the orders and the people underneath follow the orders and things like that. And that's true to a point, but it's also the case that the operational side of the military, i.e., when people are going into dangerous places, doing dangerous things is very emergent and very reactive. So you have this kind of dichotomy of a plan and a purpose and a mission meeting ground reality and having to adapt. And that's the thing, it's not that we should blindly apply... Well, you should never blindly apply anything from anywhere, but, I see a lot of stuff, books that have been written by Navy SEALS or generals or whatever, and they're brilliant books. The principles are really interesting and very applicable. I feel like what is missing from some of those books is they try to apply them too literally, rather than extract the principles and let those principles play out in the context of the work that we do.

Kris Jenkins: (22:59)
That sounds like people getting too obsessed with programming patterns.

Ben Ford: (23:06)
Yeah. Programming and social patterns. I mean, when's the last time... Well, probably not that recently, but people doing Cargo Cult Agile, or in bigger organizations, having things like Safe thrust upon them by a big top down team of consultants.

Kris Jenkins: (23:24)
Yeah. We're going to do Agile, which means here's the process.

Ben Ford: (23:27)
Yeah, exactly. Here's the process. If it's not working, you're not doing it hard enough. Oh, and by the way, now we need to think about OKRs because they also are now this kind of concrete pattern that seems to have emerged. And it just doesn't. I mean, it flat out doesn't work. The principles are universal, but practice is emergent and context specific.

Kris Jenkins: (23:52)
So what are the principles that you think that matter? What are the headline ones?

Ben Ford: (23:58)
So principles. So this is not really a principle. It's more of a, I guess I kind of view it as almost kind of axiomatic about how the world works, but there's a concept called the OODA loop, which basically encapsulates pretty much all of this stuff.

Kris Jenkins: (24:17)
The OODA loop.

Ben Ford: (24:19)
The OODA loop, O-O-D-A. Stands for observe, orient, decide, act.

Kris Jenkins: (24:25)
Okay. [inaudible 00:24:26].

Ben Ford: (24:26)
Now, this... Yeah, me too. When you just unpack those individual things, so observation is events that are coming in, orientation is what those events mean when they're applied to your current situation, decision is [inaudible 00:24:42].

Kris Jenkins: (24:42)
Folding up a state.

Ben Ford: (24:43)
Sorry?

Kris Jenkins: (24:44)
Folding up a state from the extreme of events.

Ben Ford: (24:46)
Exactly.

Kris Jenkins: (24:46)
I like it.

Ben Ford: (24:47)
Yeah, exactly. Decision is what does the mismatch between what we now understand from doing that versus where we want to be leaders to want to do? So that's a decision. And then we take an action. Now, that's a really... We take an action, then actions generate further observations. Now, that is a very, very quick run over what people commonly take away from the OODA loop. So you see this diagram of a linear circle, sorry, a linear process of observe to orient, to decide, to act, and then back to observe. Now, the truth is that Colonel Boyd, who sort of came up with this system, he only wrote the diagram down, which is a much more in depth version of the circle. He never drew the circle. This was a combination of 40 years of his work from fighter aviation in Korea, through military strategy, through studying Musashi, Sun Tzu, natural sciences, Heisenberg, [inaudible 00:25:52] completeness theorem. I mean, he covered a lot. So reading up on OODA is basically a compression algorithm on all of that stuff that went before.

Kris Jenkins: (26:03)
Right.

Ben Ford: (26:04)
And it just mapped really, really well to dealing with uncertainty and being reactive because the observation that I've had over the last few years is that technologically driven change is accelerating and even things that would've been good practice a few years ago are now slightly behind the times. For example, if you wanted to build the type of thing that we built at Finder, which we rented some servers, we used Nicks and we had some DevOps pipelines and some tests and things like that. Well, nowadays, for the front end, you'd use something like Versell, if you wanted to get moving really quickly, you'd use cloud, cloud resources like Kafka, maybe for the event stream, you'd certainly use elastic cloud resources for almost all of the other parts of the stack.

Kris Jenkins: (27:00)
Yeah. That's become the default these days.

Ben Ford: (27:02)
Yeah. And maybe you'd use GitHub actions to do some light testing and you'd be shipping, I mean, we were shipping multiple times a day on demand, but we weren't properly, properly in production. But the affordances that technology offers you for building stuff now means that the practices that you've built up that would've been a good fit five years ago probably aren't that good a fit now, right?

Kris Jenkins: (27:24)
Yeah.

Ben Ford: (27:24)
So things like sprints, daily standups, if you are shipping to customers and getting feedback daily, what's the point in doing it in two week sprints?

Kris Jenkins: (27:38)
Yeah.

Ben Ford: (27:39)
But we still kind of... We don't think enough from first principles when it comes to structuring teams and we fall back on practices that we've seen work in the past. But the problem is, when reality is changing, this is one of the things that the military's also quite good at on a tactical level, you have to adapt your tactics to the ground that you find. You have to adapt your tactics to the situation that you find yourself in, not the other way around. The situation doesn't adapt to your tactics. You just don't succeed if you do that.

Kris Jenkins: (28:17)
Yeah. So I see the parallels there to event streaming, right? Because I'm always trying to bring this back to event streaming. That's the root from which our branches grow here at Streaming Audio, but there's a definite parallel to this way of thinking, which is events come in, you need to deal with in realtime, you need to build up a state of the world that leads to decisions that go on and react in the world immediately. It's no use waiting overnight, or at the end of the month. We're all about deep building reactive realtime systems. There's also a parallel there to proper agile, like agile with a small A maybe. Where we think about people, not processes, because processes are static, but people are living, breathing, human beings we interact with.

Ben Ford: (29:14)
Absolutely. Boyd had a saying, "People, ideas, technology," in that order always. I sometimes feel like many kind of technical companies get that either the wrong way around, or they get the wrong ordering. Either they put the technology first and they force people to jump through the hoops that the technology forces them to do, or they put the idea first and they get completely fixated on an idea and they forget that it's always people doing the thing, right? The other same boy head is that, "Wars are fought by people and they use their minds." They certainly are afforded by technology. But look at what's happening in Ukraine currently for a, just completely black and white view of reactive, smaller, fast learning, force applied to a non-learning, hierarchical, top down much bigger force.

Kris Jenkins: (30:17)
Take me through that because I know nothing of the military strategy on the ground. So how is that playing out?

Ben Ford: (30:24)
So my understanding of what's happening in Ukraine is that... So there's a concept from military doctrine, which again is very, very prevalent in tech, but not really understood. It's what all these military books really write about. It's the concept of mission command and mission command is basically decentralized decision making and decentralized empowerment of people. So the idea with mission command is that you have a strong kind of alignment with a mission, but how you achieve that mission is completely the person doing the works remit to decide within certain constraints. So how that's playing out in Ukraine is that you've got the very, very top of the Russian military is saying, "I want this to happen," and completely ignoring the fact that the kit doesn't work, the conscripts aren't motivated, they've got none of the conditions necessary for ultimate success other than overwhelming force. And the problem there is that because of the command system that's built up in the Russian military, nobody's able to tell the level above, "No, this isn't going to work," or, "No, this isn't going well."

Kris Jenkins: (31:42)
Yeah, no feedback loop.

Ben Ford: (31:44)
No feedback loop, right? No link between bottom-up sensing of your environment and top-down how we're going to achieve what we want to achieve. Whereas the Ukrainians are, they've done a lot of work over the last few years to try and re-engineer themselves into a more Western kind of way of fighting wars with lots of NATO advisors and things going over there. So what's happening with the Ukrainians is that they have obviously a massively aligned mission in, "We want our country to still exist and we want our families to be safe." Within that, the units are being given all of these new and shiny weapons and basically figuring out on the fly, how to use them. And then they are taking that knowledge of like this is how you use this new weapon against this Russian formation or this Russian tactic. Then that is being spread around. So it's a complete bottom-up process and where something works, it's transplanted new somewhere else.

Ben Ford: (32:48)
So you've got this... I wrote a post on LinkedIn a while ago, likening this to the surface area of objects floating in a fluid. If you've got a big block of ice, versus the same mass of ice broken up into multiple different little pieces, the ice that's got the bigger surface area melts quicker or becomes in, is in harmony with its environment quicker is another way to look at it. So the Ukrainians are transmitting what they learn around much faster than the Russians because the Russians clearly aren't learning at all. They're getting hammered with the same tactics that has been happening all along. So again, back to event sourcing, if this was a technical system of some sort, the Ukrainians have a much better event sensing and a better fold algorithm than the Russians do.

Kris Jenkins: (33:48)
Yeah. Yeah. I can see that. I mean, I hesitate to take a such a serious event as that and then talk about our world of programming, which is a sweet gig, let's face it. But you can definitely see parallels in the system design. Where I've worked for companies that operate like a top down, no feedback system, we're going to do this. And two years later, they find that the project didn't deliver. I've worked for other companies where people say, "Here's what we need to get done, but you figure out the best way to do it." And that works, that kind of trusting people and letting them feedback what's working thing is the essence of Agile.

Ben Ford: (34:30)
Exactly. Exactly. Yeah. Funnily enough, the more top down that those businesses are, the longer it takes for them to find out that it's not going to work. So the more top down it is, the closer to the supposed delivery date you find out that actually isn't going to deliver.

Kris Jenkins: (34:47)
Yeah. Yeah. Or usually long after the official delivery date.

Ben Ford: (34:52)
Yeah, yeah. Yeah. I mean, and then you've got all the kind of human biases involved, but it comes down to not having an effective feedback loop, which in the context of this conversation means that you don't have an effective event sensing system.

Kris Jenkins: (35:08)
Yeah. Yeah. It's also why I really favor for businesses that launch early, because until you've got the customer in that feedback loop, then you haven't really got feedback, a real feedback loop at all. You're just talking to yourselves internally.

Ben Ford: (35:23)
Yeah. That's exactly it. So I mean, one of the other observations from the OODA loop, Boyd was very heavily inspired by the physical sciences, as I said. So if you view OODA as an outside to inside flow and an inside to outside flow, so observations to orientation is out to in, decisions to action is in to out, and if you don't have effective observations, that's your inward loop broken. If you don't have effective actions, that's your outward loop broken. And if you don't have any contact with the environment, you have a closed system, which inevitably collapses into entropy.

Kris Jenkins: (36:06)
Yeah. I am...

Ben Ford: (36:08)
Go on, sorry.

Kris Jenkins: (36:10)
No, yeah. It's just reminding me of something I read in a psychology book years ago, which is that people have internal strategies for things. They have strategies for decision making, for example, and you think, "Okay, well, some people's strategy for decision making is they think of a thing they want and then they think of three different variations, like three things on a menu, and they imagine what it might taste like to eat those things in a menu and then they pick the thing in their head that tastes nice, and that's what they end up ordering." We all have different strategies playing out in our heads. And one of the principles in this psychology book was all these different strategies, at some point, they go out into the real world and they test something in the real world. And those generally are much, much healthier strategies than the ones where you spin inside your own head and never get some external component. It was the theory of this book that most bad psychological patterns have an inner loop that doesn't go for external feedback.

Ben Ford: (37:11)
That's very interesting. I'm actually reading, Being You at the moment by Anne L Seth.

Kris Jenkins: (37:17)
Okay.

Ben Ford: (37:18)
Which is an amazing book on what it means to be conscious. The prevailing wisdom, probably up until quite recently, was that you've got this kind of bottom-up process of sensing different features. And then those being glonned up into bigger and bigger things until you get meaning. And actually, what the kind of cutting edge cognitive science, which I think applies to organizations as well if you squint a bit, says that what you're actually doing is you have this kind of preconceived model of the world. What you're actually doing is looking for error minimization. So you're actually predicting first, and your bottom-up process is actually error minimization. So you are checking for errors in your prediction. So in his book, he calls it controlled hallucination. So you are imagining, you're operating, your internal model is imagining the world, and then your senses and what you come into contact is telling you what is wrong with your imagination rather than this picture being built from your senses inwards.

Kris Jenkins: (38:32)
Okay. That makes... We're bouncing around topics, but that really makes me think of a principle in music composition, which is if I can really summarize it quickly, you want to establish a pattern and then break it. You play a piece of melody and then you play a slight variation, so people start to build up this prediction of what the third repetition's going to be. Then you change it just enough that there's a pleasant surprise between the internal, what you thought was about to happen, and what actually happens. We have this thing in our pattern-matching brains, don't we, where we like to establish patterns, but then we get bored with them. It's about introducing just the right amount of novelty to keep things exciting.

Ben Ford: (39:20)
Yeah. There's loads of principles from the physical sciences. Like there's Ashby's Law of Requisite Variety, and lots of, lots of things that say that. I guess we could go onto Western management principles now, but western management principles is all about driving out variation. So the whole Agile really started, I mean, at big company, Agile, not the kind of lean startup, that kind of thing. But...

Kris Jenkins: (39:50)
The kind of Safe [inaudible 00:39:52].

Ben Ford: (39:53)
It's really designed... Yeah. It was really designed to minimize variation, but you don't minimize variation in the natural system. Like minimizing variation means ignoring the outside world because the outside world is always changing.

Kris Jenkins: (40:06)
Yeah. Because a lot of our classic management thinking comes from like Henry Ford building his car factory, trying to nail things exactly down. So it's just a repeatable process.

Ben Ford: (40:18)
Yep.

Kris Jenkins: (40:19)
But that doesn't quite work when... You can make that work when you have a repeatable output of 20,000 identical cars.

Ben Ford: (40:28)
Yeah.

Kris Jenkins: (40:29)
You can't quite make that work in the... It doesn't service in the world we're dealing with today, reacting to customers that want different things all the time and different sources of data coming in all the time.

Ben Ford: (40:41)
No, doesn't [inaudible 00:40:42].

Kris Jenkins: (40:42)
Need a new way of managing things.

Ben Ford: (40:44)
Yeah. It works when you are the only car factory in the world. It doesn't work when you're competing with someone like Elon Musk building car factories.

Kris Jenkins: (40:51)
Yeah.

Ben Ford: (40:54)
The world is getting more and more fragmented in that sense because all of the technology that... I mean, look at Kafka itself, right? That was born out of LinkedIn needing to develop a piece of technology, people within that business developing that technology and then taking out and turning it into their own business. I mean, you can't move in the tech scene for new technologies that have started in that way. Like I mentioned, Versell earlier. What would we have had to do five years ago to get an SSL protected front end up and running, or 10 years ago, or 15 years ago. Now, 15 years ago, you would've had to go and buy a box from somebody, get it shipped to a data center. Maybe not 15, maybe 20.

Kris Jenkins: (41:45)
20 years ago, I did exactly this, bought a box from Dell, had it shipped to a data center, wired it up.

Ben Ford: (41:51)
Yeah, exactly. Now, you literally pushed to a GitHub repo and you've got a live website up and running, takes seconds. So things that were gating factors before have been mitigated by people inventing technology and that constant process of evolution means that now the bits of technology you use in your day to day job are smaller, back to the kind of Unix philosophy, smaller, and they do one thing well, which means that your average business now has that sense making part where they come into the outside world, they come into contact with the outside world is now mediated through probably 50 to 100 different pieces of somebody else's technology. So how you make sense of the world now is completely different to what it was five, 10 years ago. And now, you've got this massive data aggregation problem of what actually is all this stuff telling us because it's coming from all over the place.

Kris Jenkins: (42:57)
Yeah. We've got sharper and tighter feedback loops over the years, but we've now got the management problem of all this data that we've got coming in realtime.

Ben Ford: (43:07)
Yep. Exactly.

Kris Jenkins: (43:09)
So do you have any concrete suggestions on how we manage this world of, we need to be paying attention to data in realtime, but there's the risk of not doing it. There's also the risk of overwhelm.

Ben Ford: (43:25)
Yep.

Kris Jenkins: (43:25)
Where do we go, Ben?

Ben Ford: (43:27)
Well, so funny you should mention that. So my exploration of military stuff sort of took me into a bit of consulting, a bit of leadership type consulting, and working with other leadership coaches and things like that. And the thing that's become really apparent over the last couple of years is that you have to have some capacity in the system for any of that stuff to land. All of the stuff about building better OODA loops and having time to think. Well, if everyone's completely maxed out, if all your calls are running at 100%, you don't have space for another job. So over the last couple of years, I've actually taken some open source technologies and used them to build internal systems with a view of giving people back that capacity first, because with all of these operational, let's call it operational fragmentation.

Ben Ford: (44:22)
You got a bit here for doing a very specific job, a bit over here. They'll be spread over different departments in the company like marketing and ops and customer success. Your aggregate picture of the world as a business owner is now fragmented across all these different systems. Very much like that picture of cognition from being here. You've got your predictions of the world and now meeting fragmented data coming in. And in most businesses, that defragmentation process is done by either it's the middle managers who are spending inordinate amounts of time preparing reports by pulling data from all these different systems. So you've got your most expensive people that should be leading and doing the human side of sense making stuck in Excel for...

Kris Jenkins: (45:14)
Yeah. Compiling spreadsheets.

Ben Ford: (45:16)
Yeah. A large portion of their day, or you've got those sort of more back office systems being built by actually probably fairly junior admin type folks. So you've got people doing the job of machines. We both know. We've just spent 40 minutes talking about how algorithms that do this stuff have been developed. So the approach that I've taken over the last couple of years of consulting is, well, let's just take a bunch of that awesome technology that's been made available to us, turn it into a stack for building internal systems, build lightweight infrastructure to aggregate all the data from all those different systems into your model of the world, and then put BI systems over the top from that single consolidated source of truth.

Ben Ford: (46:07)
It's a dead simple approach, but the application of five or six different open source technologies to build that internal system, which has to be unique for every business because the combinatorial nature of all these different operational systems means that whatever sits in the middle is going to be pretty much unique, probably some patterns and some similarities, but that thing is your orientation from the OODA loop, your orientation is the thing that sits in the middle and lets you make decisions.

Kris Jenkins: (46:42)
Yeah. Yeah, I think there's a big question of how we can get a kind of reusable structure that does that bespoke reaction that we need.

Ben Ford: (46:51)
Yeah. The conclusion that I've come to is that you can't have that as a product. You can have a bunch of products that you join together, but the actual, the wiring of the neurons of that thing is specific to every business, which is why I'm not building a product for this, I'm building a service for this where the product kind of emerges over time internally in response to the needs of the business. One of the biggest needs is don't run any of your operations on Excel, for God's sake.

Kris Jenkins: (47:31)
Yeah. Excel's good for prototyping, I think, because it's fast, but...

Ben Ford: (47:37)
Even then, I think I would've agreed with you maybe six months ago, but I've come across a product called Causal these days.

Kris Jenkins: (47:43)
Causal. You're going to have to give us a list of links for the show notes

Ben Ford: (47:47)
Yeah, absolutely. Yeah. So Causal, causal.app, it's a spreadsheet replacement online, but you build a... Pretty much like, you remember the system that we used to have in the bank, which was the Dag, the trading strategies would be arranged in a Dag where it's very much like an Excel Dag in a spreadsheet. So they have something similar with a little lightweight programming language. But the difference is that in each of the cells, you can put a probability distribution. So then for building your model, you end up with a probabilistic model and they run [inaudible 00:48:29] simulation for you. So you build your model kind of bottom up from the unit economics of your business and then you play forward what does that look like played out over time? And then you tweak variables and it reruns a simulation for you.

Kris Jenkins: (48:43)
Well, so you're putting in a probability of someone purchasing a particular item and that eventually leads to what your month sales figures. Is that the kind of system you're talking about?

Ben Ford: (48:53)
Yeah. So you could do, I'm going to charge between X and Y for this product, see what that does. Okay. If I charge X and it doesn't work, and it's not profitable, much better to have done that in some sort of simulation of the world rather than actually play it out in real time and find out that you weren't profitable six months after you needed to know it.

Kris Jenkins: (49:15)
Okay. That's a curious system. We're going to have a look at that.

Ben Ford: (49:17)
It's really cool. Really, really cool. It also, because it's a modern piece of software, it integrates well with lots of different source data. So you can actually feed in your real results to it as well. So you build a model, you feed the model with real data, and then you watch your predicted versus actuals play out, and it's always up to date. So you've got that part of the, that predictive part of the business about what the business model looks like and whether we've got cash in the bank in six months is kind of updating in realtime versus, again, somebody having to spend time cutting and posting new CSVs into an Excel spreadsheet and making sure you haven't overwritten a formula and stuff.

Kris Jenkins: (50:06)
Oh, interesting. Okay. Well, that makes me think maybe we should begin to wrap this up with... Because I know you're always researching and watching the skies, give me a technology or a book recommendation I should look into beyond Being You and Causal.

Ben Ford: (50:20)
Okay. So Boyd, The Fighter Pilot That Changed the Face of War is a biography of Boyd and his work and is much more in depth than trying to [inaudible 00:50:34] a load of stuff from a diagram that he wrote towards the end of his life.

Kris Jenkins: (50:37)
Well, that's how we can break down the OODA loop, right?

Ben Ford: (50:40)
Yeah. Yeah. It's much more of a kind of base understanding of what his motivations were. Another book that I've really, really enjoyed and has shaped a lot of my thinking is The Origin of Wealth by Eric Beinhocker, which it looks at the...

Kris Jenkins: (50:58)
What's that about?

Ben Ford: (51:00)
It looks at modern economics and this kind of system that we all work within from through the lens of evolution and natural selection.

Kris Jenkins: (51:10)
Okay.

Ben Ford: (51:11)
I mean, when you think about all of the evolution of technology that we've just been talking about, it's a book about the underlying mechanics of how that works.

Kris Jenkins: (51:22)
Okay. I should get onto Amazon.

Ben Ford: (51:25)
Yeah. Sorry. I always do this. I'm a [inaudible 00:51:28] of service attack on people's book lists all the time.

Kris Jenkins: (51:30)
You are. Every time I talk to you, my reading list grows. Ben, it's been an absolute pleasure talking to you.

Ben Ford: (51:38)
Yes, you too, Kris. It's always a pleasure.

Kris Jenkins: (51:41)
Yeah. I'll catch you again soon. Thanks for being on the show.

Ben Ford: (51:44)
Indeed. All right. Cheers, Kris.

Kris Jenkins: (51:45)
Cheers. Bye.

Ben Ford: (51:46)
Bye.

Kris Jenkins: (51:46)
That was Ben Ford. I've got to tell you, he's exactly like that in real life. Every time I talk to him, I go away with a new list of links I need to check out and a handful of books I need to append my stack. I've got homework. We'll put links to all of that in the show notes if you want to follow up with any of it. Among the show notes, you'll also find a link to Confluent Developer, which is our site that teaches you everything we know about Kafka. So if you need a backbone for your real-time data systems, that's the great place to get started. We've got everything from getting started guides to full system walkthroughs, to videos about how people are using Kafka in the real world. So check it out at developer.confluent.io.

Kris Jenkins: (52:28)
If you need to get Kafka running quickly or keep it running easily, then head to Confluent Cloud, which is our fully managed Apache Kafka service. You can get started in minutes. There's no credit card required, but if you add the promo code, PODCAST100, on the billing page, you'll get $100 of extra free credit to run with. With that, it just remains for me to thank Ben Ford for joining us and you for listening. I've been your host, Kris Jenkins, and I will catch you next time.