Streaming Audio: Apache Kafka® & Real-Time Data

Automated Event-Driven Architectures and Microservices with Apache Kafka and SmartBear

• Confluent, original creators of Apache Kafka® • Season 1 • Episode 166

Share episode

Is it possible to have automated adoption of your event-driven architectures and microservices? The answer is yes! Alianna Inzana, product leader for API testing and virtualization at SmartBear, uses this evolutionary model to make event services reusable, functional, and strategic for both in-house needs and clients. SmartBear relies on Apache Kafka® to drive its automated microservices solutions forward through scaled architecture and adaptive workflows. 

Although the path to adoption may be different across use case and client requirements, it is all about maturity and API lifecycle management. As your services mature and grow, so should your event streaming architecture. The data your organization collects is no longer in a silo—rather, it has to be accessible across several events. The best architecture can handle these fluctuations. Alianna explains that although the result of these requirements is an architectural pattern, it doesn’t start that way. Instead, these automation processes begin as coding patterns on isolated platforms. 

You cannot rush code development at the coding stage because you never truly know how it will work for the end system. Testing must be done at each step of the implementation to ensure that event-driven architectures work for each step and variation of the service. The code will be altered as needed throughout the trial phase. Next, the product and development teams compare what architecture you currently have to where you’d like it to be. It is all about the product and how you’d like to scale it. The tricky part comes in the trial and error of bringing on each product and service one by one. However, once your offerings and architecture are synced, you’re saving time and effort not building something new for every microservice. 

As a result of event-driven architectures, you can minimize duplicate efforts and adapt your business offerings to them as the need arises. This is a strategic step for any organization looking to have a cohesive product offering. Architecture automation allows for flexible features that scale with your event services. Once these are in place, a company can use and grow them as needed. With innovative and adaptable event-driven architectures, organizations can grow with clients and scale the backend system as required. 

EPISODE LINKS

Tim Berglund:
Okay. Here's an idea. Automated behavioral-driven testing of your event-driven microservices. Sounds great, right? But how do you do it? Well, I talked to SmartBear's Alianna Inzana today about that and some tooling that SmartBear makes that supports things like this. They actually are responsible for a lot of products that you're probably familiar with. And she's got some great thoughts on the topic. I'd love you to listen in to us talking about it. On today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud.

Tim Berglund:
Hello, and welcome to another episode of Streaming Audio. I am. Once again, your host, Tim Berglund, it seems like I always am Tim Berglund it's inescapable and I'm joined today in the virtual studio by Alianna Inzana. Ali, she's a product lead for API and virtualization, am I getting that right? API testing and virtualization at a company called SmartBear. And if you don't know who SmartBear is, you probably use some things they're responsible for. Ali, welcome to the show.

Alianna Inzana:
Thank you so much, Tim.

Tim Berglund:
I want to hear about SmartBear and some of the projects and technologies that are associated with you. And really what I want to talk to you today about is testing event-driven systems and how you are seeing the maturing of testing and just the maturing of a vendor of an architecture is in general because you are close to a lot of this stuff, but tell us a little bit about yourself. What does a product lead do there and how did you get to do that?

Alianna Inzana:
Sure. So I have been doing various types of platform product management for longer than I care to admit, I think. And I was a customer of SmartBear before I came here to work on virtualization and then API testing. I was a customer of SmartBear through Swagger. So Swagger UI and SwaggerHub, the specification formerly known as Swagger, OpenAPI specification.

Alianna Inzana:
So as a platform person, I was always responsible for delivering my company's information and insight. And as it turned out, a lot of the SmartBear products were how we did that, where at least were the backbone of how we did that. In addition to Swagger, we also have products like ReadyAPI for API testing and virtualization and performance testing. And outside of the API business line, we also have test management and functional testing.

Alianna Inzana:
And so there, the brand that people are probably most familiar with is Cucumber. So behavior-driven development, BDD, the Cucumber folks are also smart bears. So, yes, we are a pretty big tech company that you may not have heard of, but we are really happy to be here today and to talk about event-driven architecture, which is one of my personal favorite topics in the API space.

Tim Berglund:
Absolutely. And yeah, it was the Cucumber connection that got me when I was introduced to you guys. I'm like, "Wait. Oh, that's oh, well, okay." And I know that you probably get that a lot, but also I like that you have a name for people who work for the company and it's so obvious. I don't know why I didn't think of it, but smart bears. So at Confluent we occasionally talk about that.

Tim Berglund:
Where we conflict tiers con fluid tutors, there isn't anything obvious. Also I, feel like I need to say the bear is just there. Generally, it's got nothing. This is not a plant for today. The bear is just a part of my office. This is when we kicked off the Confluent community catalyst program in, I guess it was September of 2019 in San Francisco at the last in-person Kafka summit, before the pandemic, the catalysts got together at that Build-A-Bear down on the pier in San Francisco.

Alianna Inzana:
Aww.

Tim Berglund:
It was the first Build-A-Bear bear I'd ever made. I've spent maybe four figures on Build-A-Bear through the raising three children. And this is-

Alianna Inzana:
I've got a seven-year-old. I am right there with you.

Tim Berglund:
That's a great idea.

Alianna Inzana:
Yes, SmartBear are... I was going to say my brand people will probably come from me for this, but we used to have a bear. His name was Eugene. And now we have, if you look at our logo, it's the gear bear, which I actually like, because we're builders, we're building bears. So, but yeah, brand, don't come for me.

Tim Berglund:
Please, don't. And not a paid endorser of Build-A-Bear, no association with that at all. Just who doesn't like stuffed animals? I guess that's all I'm here to say. Anyway, more about a vendor of an architecture is probably a good idea. So there's a technology that has been on the radar and this always happens. There are things that start popping up that the first time you see it you're like, "What's that?"

Tim Berglund:
And then all of a sudden everybody's talking about it, ReadyAPI is a thing that I can say in the Kafka summit call for proposals and program committee process for this year or every three Kafka summits this year, which is a lot, but I've seen a lot of ReadyAPI stuff. So could I start by asking, what is that and where does it fit into the world?

Alianna Inzana:
Sure. So ReadyAPI, it's our platform for API testing. And there's a lot of different ways that I think people talk about API testing. There's simple validation checking is this endpoint supposed to give me back a phone number? Does it success? There is validation against specifications. So when you have a specification like the OpenAPI specification that tells you a little bit more about the architecture of the service.

Alianna Inzana:
Things like, I don't know how many different response codes are. There's some example interactions that the specification gives you an indicator for. So the next level up is making sure that those things, the build meets the spec the same way that an inspector that was looking at a build of a house would to make sure that you haven't moved a load-bearing wall or anything.

Alianna Inzana:
And so we've seen this really evolve very strongly for RESTful architecture, but what we've noticed at SmartBear and what ReadyAPI has always really supported is multiprotocol. In fact, ReadyAPI started its life as SOAP UI, and it was called SOAP UI because it put a user interface on SOAP protocol APIs. And so we built out from that. And so within the platform, you can not only do simple validation tests for REST and SOAP GraphQL, and other event-driven protocols like MQTT, JMS, and MQP, and now Apache Kafka.

Alianna Inzana:
So not only can you validate payloads, make sure that all of those things look good, but in addition, we believe very strongly in testing from the spec on down and testing the capabilities. When you're thinking about a microservices architecture, if you only look at what a single end-point does, and you don't look at the integration steps, you do wind up missing out on really important data in terms of how people are consuming your service and what are some of the patterns of consumption.

Alianna Inzana:
And there's a lot of ways that are contained within a spec that you can test. And then there's a lot of things that happen outside of the spec that you could change, like rate-limiting and so ReadyAPI, not only validates what's there and how an individual service works but it also allows you to create very simple chains of behavior that are not very brittle because they can be derived and refactored from specification. And when I say that, I mean, specification schema for GraphQL.

Alianna Inzana:
All of those are brought into the tool and you can build tests from the components in those descriptions, which helps us make the test less brittle. And it also helps us to understand a true end-to-end or journey scenario as our BDD folks would call it and use that to validate and then take the same test, use it to performance test and take the specifications and create virtual services. So you can isolate a system under test or virtualize a third-party dependency, so that you're really testing the right thing and bringing it all together in your CI/CD pipeline, for example.

Tim Berglund:
Right. So third party dependency is when we're at the level of integration tests, sometimes one might have things in the world that you want to, can I say, mock, is it okay to say mock still? You want to mock those things?

Alianna Inzana:
Yes, you absolutely can say mock.

Tim Berglund:
Okay. All right. Great. [crosstalk 00:08:58].

Alianna Inzana:
Well, there's one thing I will say about virtualization, which is mocking, especially in REST, mocking is often about an endpoint and it is often dumb. It's just like, "I'll give you response one. Then two, then four." What we allow with virtualization is routing based on headers and different things. It really creates a much more like a synthetic experience of the API, which is useful because when you're building with something that's a little bit closer to real life, you can validate a lot more.

Alianna Inzana:
In fact, we have a lot of customers who they build a lot of APIs for their partners. And so they don't break their partners' implementations. They'll create a virtual service, hosted up on a virt server and put it behind a gateway, and say, "Hit it. This is what we're thinking for the next iteration of our product. Here's the wireframe, essentially. Tell us if this works for you."

Alianna Inzana:
And they get a lot of valuable feedback before they commit to coding something, which if you think about it, we don't really have a great way of prototyping services and service virtualization for us is very much about that. In addition to third-party dependencies, things like not having your ops manager come for you because you accidentally did a performance test against Google maps or Google's geocoder things like that.

Tim Berglund:
Which might cost money, right?

Alianna Inzana:
I'm from not speaking from painful personal experience at all on that one.

Tim Berglund:
You're just saying one might do a thing like that. And that would be bad probably.

Alianna Inzana:
For example.

Tim Berglund:
That's amazing. So the behavior-driven testing concepts, it seems like have very much seeped into ReadyAPI that's part of the DNA from which it comes. Well, also SOAP UI. I think maybe you told me that we had a call a month ago. Somebody might've said that it sounds vaguely familiar, but is again surprising because the little guy you used to give nuggies to is all grown up now, that's really cool. And the whole importance of this kind of testing, you've hinted at this, and there's a meme for the video version.

Tim Berglund:
I should have had this prepared and printed out, but just a family of integration testing memes that show things like extremely awkwardly configured bathrooms with doors that open into the toilet and where the door got cut out. So it could move past the toilet. And it says something like, "All unit test pass." There's that whole reason why behavior-driven and I guess more, slightly more generally integration testing matters. And that's what ReadyAPI does. It's super cool. And it includes Kafka now.

Alianna Inzana:
Well, and I think this is really the thing that we've been seeing. At one point in time, if you looked at the types of integration tests that our customers were creating, it was very much about RESTful endpoint to RESTful endpoint, right? And that's fine. And we allow you to create that as a model and then data drive that scenario with happy path data, sad path, data, wacky path data.

Alianna Inzana:
But you do security testing for example, but what I've noticed, especially lately and hearing it from our customers is really in almost every organization, there is much more of polyglot services people sort of saying, "Hey, we really understand our data model well. For our microservices, we're going to use GraphQL backend for front end type of pattern, help us get stuff out there more quickly for people to consume or using GRPC for speed and for the polyglot architecture on the receiving side." With Kafka, a lot of the architectural patterns, I think of any microservices, any domain-driven design, something needs to diverse the boundary of that domain.

Alianna Inzana:
I read a statistic once at Netflix, every call to their API fans out to at least six other services. So it's like you do something in your Netflix queue and rating service, all of the things that potentially could update from that. And you can't achieve that without event-driven architecture. But if you want to truly test the full workflow, you need to actually bring in the event-driven services and services like Kafka, which behave a little bit differently than REST surprise.

Alianna Inzana:
And so I think that's really a challenge of any testing tool. You can proxy anything, but how do you get something that can deal with things in terms of how natively it behaves and what your expectations are for a given service or how it should integrate with other services. So chasing that train of behavior and being able to evaluate it, and then turn that same thing into a performance test is a pretty powerful thing to be able to say, as you extend your architecture.

Tim Berglund:
I'm buying it just as you're talking, that makes a lot of sense.

Alianna Inzana:
I have a very soothing tone of voice.

Tim Berglund:
Right? It's those of us since line of work, it's just natural. But now I get it. I guess I already got why automated testing that was a level or two up from unit testing was a good idea you didn't need to persuade me of that, but in particular with microservices architectures, why a framework to help you with that matters a lot. And you said something else in there that makes me want to ask you about something because it strikes me that you are probably uniquely positioned to talk about this. You've got the synchronous world and maybe the second generation of microservices architectures were and still are synchronous with REST and GRPC. And we need frameworks to help test them out. And that's all a big pain and ReadyAPI you're my only hope or one of the small number of hopes, one of which is me building a bunch of bespoke stuff.

Tim Berglund:
But now that you have event-driven things and you've got Kafka in the loop and that's supported by the product, you have to work with people like you, in particular, Ali, you need to work with your customers who are doing one, the synchronous thing, and two, the event-driven thing what's happening in the world? What do you see happening in the world there? Because I know people I talked to, but they're all fairly explicitly Kafka people. I have this massive selection bias, they're already into it, but you're talking to people who might not be, or who are on the fence. So just, I don't know, riff on what's going on in the world of event-driven architecture from your perspective.

Alianna Inzana:
Well, I think the, I can't really say historian. I have a degree in biology, but the history buff in me says if you want to understand some of the patterns that we're seeing for the future of these things, look at some of the patterns that have already happened in similar situations. So I look at REST, which though synchronous, I think what we really saw happening in terms of how RESTful architecture has evolved is first someone was building an API to deliver on a thing and it was like they were doing it.

Alianna Inzana:
It was theirs. And then the first thing that forced them to scale is other people found their API and was like, "Oh, this is really useful. I don't have to build customers, the API I'll use bobs." And then since he didn't want to have to explain it every single time somebody wanted to use it he started to do documentation. So from code first documentation people started saying, "All right, first thing we did was we scaled for the use of the service scaling."

Alianna Inzana:
But then the next thing that happens is you start to see patterns or things that you want to surface to the larger organization. And so you have to start to scale in a slightly different way or with a slightly different, I guess you could call it an objective. And that objective would be more about scaling for how users are going to interact with your services, right? And understanding automation and making sure that the scenarios that you're trying to deliver do that.

Alianna Inzana:
And then as your organization scales, and maybe you started to embrace things like design first governance comes in, and I think this is something that's really overlooked because when you have really large organizations and you're telling the devs, 'Move faster, deliver more, tests more don't ever be a bottleneck." There are things that you just can't do unless you have a certain level of governance because governance implies that assets are there, objects are there a way that you approach it, it gives brand and structure to your services.

Alianna Inzana:
But in addition to that, it also means that you have more to draw on so that you don't have to create everything brand new and the number of organizations that I've talked to you that have 20 different versions of the customer API or a million permutations of the customer object and their Jason schema or something like that, it's a lot. And then the final level is really developer experience for me, at least.

Alianna Inzana:
This is like a new thing we're talking about like, "Oh, developers deserve nice things film at 11." But it's really important because if you want people to use the same things, if you want them to know what's out there to be able to reuse objects, to be able to reuse things so that they can go faster so that they can deliver more and you don't provide a simple, intuitive, helpful way of having them access it, it's going to be really difficult.

Alianna Inzana:
Right. So this is how I've seen things evolve on the RESTful side. And I've seen a similar pattern beginning to happen on the event-driven architecture side. Initially, it's a coding pattern. It'- not an architectural pattern really, but as your approach to event-driven architecture evolves, and as the needs of your organization evolves, you're going to start to see the same thing.

Alianna Inzana:
And I think personally, one of the reasons why you're not seeing people talking about event-driven governance, is there certain platforms, isolated platforms that you have to sort of be all-in and it gives you a certain level of that. But overall, the toolchain has not expanded enough to embrace those things. And I think we're seeing that right now, we're seeing it with specifications like AsynchAPI that don't just talk about the payload, but they talk about the relationship between pub and subscribe.

Alianna Inzana:
And we're seeing it with companies that are providing developer portals and design systems and tools that help you think first and then execute on your event-driven architecture. So if you look back to the REST world and see how that's changed and things that you think of as, "Ah table stakes, we need to make sure everything is camel case or whatever." Those things are coming in event-driven systems as well. And it will help us move faster and deliver better and more efficiently, I think. Because in the end, I don't know that certain decisions that are, you know, for example, governance decisions are things that people feel all that impassioned about changing like camel case. Great. I'll just use that and we won't have that discussion. You can save your creativity for actually delivering the value to your consumers

Tim Berglund:
Now. Well, that would be a great place to end, but I just a little bit more, I want to talk about that. That was amazing. Okay. When you talk about governance-

Alianna Inzana:
I don't think I have you ever given a short answer to anything, I apologize.

Tim Berglund:
Again, occupational hazard, right? It's just the line of work. Some of us are in we're rewarded for good solid answers. You talked about governance and maturity. And I think this is just the thing I've been thinking about a lot in the last year or so of just developmental stages that people and organizations and products go through. And when you look at governance features in a mature relational database, let's say, it's all there, right? Because it's been decades. It's actually been decades of these things being deployed in contexts where governance was a non-negotiable features getting built in some of them clearly features that really big customers twisted arms for when you look at the documentation.

Tim Berglund:
So there's a lot of maturity there in all the good and bad ways, bad senses of that term. And it's not present in a 10-year-old or 11-year-old system like Kafka. Now, when you get into the commercial side of things, you start to see more of that in like conflict platform, conflict cloud, but even so it feels early and I hope my product people don't get mad at that there's lots of cool stuff that we do on the governance front.

Tim Berglund:
But it's like you're 13 and you just went out for your first track and field competition you're running. And that's really cool. And you used to not be able to run that fast, but you're also not competitive at the college level. So we're early days in the event-driven world for things like that. What does good look like in your mind? What would the feature set look like for something that felt grown-up and completed?

Alianna Inzana:
It's really interesting. As a product person, I'm a very strong proponent of when the pupil is ready, the teacher will appear. And I think that goes as well for products and features and things like that because if you don't go through those evolutionary phases in the beginning, you're not going to know how you want that to work in the end system.

Alianna Inzana:
So if you don't go through the phase of understanding and using event-driven architectures as an architectural pattern and not just a one-off in your organization, what will you put in an event portal, for example. And I think from an event-driven architecture standpoint, I think one of the things that is a big driver of that as well is also how much importance organizations are attaching to their event-driven architecture.

Alianna Inzana:
We hear a lot more about real-time. And you think about things like machine learning without a broker like Kafka, how hard would that be because you want to replay history, again and again, to teach the thing? If that didn't exist? What would be the motivation to build something like Kafka? So I think there are going to be items that, on that list that like you and I maybe don't know, or can't see at this point in time because of how systems and needs and user needs evolve.

Alianna Inzana:
But I think that there are a few things that we can call out because to me, it's one of the central struggles of how does engineering explain to the rest of the business, what they're doing and the business needs drive what engineering builds and how engineering builds it. But if you're refactoring or rebuilding an architecture to embrace Kafka or just event-driven architecture in general, that means it's time that you're not spending building something new or something like that.

Alianna Inzana:
So you have to understand, well, what is the importance of it? So when I look at where we want to get to, there's a few things. One is, I think governance becomes a part of it immediately when you start thinking strategically about your event-driven services. And I want to give credit where credit is due because in the way that we've built a maturity model at SmartBear for in general, how do you approach your APIs and services? Some of the analysts at Forrester have done a great job. And David Mooter in particular thinking about event-driven maturity and what they were seeing with their customers.

Alianna Inzana:
And they too believe that we are not where we want to get to is the point where events are governed, where events can be presented to a consumer-like an event portal, that type of thing, where the data is unsiloed, but the access is orchestrated and choreographed. And I think that's another really important aspect of it. And the organizations are actually beginning to see the importance of their event-driven architecture.

Alianna Inzana:
So moving towards thinking of it as part of business enablement and not just part of the technical or even architectural patterns, how does being real-time help you deliver on business value? And that's a really important... Especially if you think about the last year, we've all been at home, working with our streaming services to entertain ourselves.

Alianna Inzana:
When you think back even five years, the idea of being notified in real-time about fraud at your bank, like that sounded crazy, didn't it? They maybe could tell if you went to a foreign country that they weren't expecting, but that was about it so much has changed because of the availability of data, the prevalence of machine-learning and the event-driven architectures that underpin it.

Alianna Inzana:
But I think it's that business need that really is going to drive it forward. And that's why when I think about maturity and what elements that maturity model might have, it's not just technical, it's actually bringing the business in because they to understand the strategic importance of events in delivering the commercial vision. And we certainly have seen that with APIs.

Alianna Inzana:
I mean, there's companies with billion-dollar valuations, like Stripe that are API companies, that's what the product is. And so I anticipate we will see something similar with event-driven architectures and with externalizing events, for consumption, for IOT devices for all of these different things, and bringing them together and presenting them cohesively is going to be a huge part of how we do that.

Tim Berglund:
My guest today has been Alianna Inzana. Ali, thanks for being a part of Streaming Audio.

Alianna Inzana:
Thank you so much, Tim. I really appreciated it.

Tim Berglund:
And there you have it. Hey, you know what you get for listening to the end? Some free Confluent Cloud. Use the promo code 60PDCAST—that's 60PDCAST—to get an additional $60 of free Confluent Cloud usage. Be sure to activate it by December 31st, 2021, and use it within 90 days after activation. Any unused promo value after the expiration date is forfeit and there are a limited number of codes available. So don't miss out. Anyway, as always, I hope this podcast was useful to you. If you want to discuss it or ask a question, you can always reach out to me on Twitter @tlberglund, that's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out on Community Slack or on the Community Forum. There are sign-up links for those things in the show notes. If you'd like to sign up and while you're at it, please subscribe to our YouTube channel and to this podcast, wherever fine podcasts are sold. And if you subscribe through Apple podcasts, be sure to leave us a review there that helps other people discover it, especially if it's a five-star review. And we think that's a good thing. So thanks for your support, and we'll see you next time.