Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

Introducing JSON and Protobuf Support ft. David Araujo and Tushar Thole

• Confluent, original creators of Apache Kafka® • Season 1 • Episode 103

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 40:00

Confluent Platform 5.5 introduces long-awaited JSON Schema and Protobuf support in Confluent Schema Registry and across other platform components. 

Support for Protobuf and JSON Schema in Schema Registry provides the same assurances of data compatibility and consistency we already had with Avro, while opening up Kafka to more businesses, applications, and use cases that are built upon those data serialization formats. 

Tushar Thole (Engineering Leader, Confluent) and David Araujo (Product Manager, Confluent) share about these new improvements to Confluent Schema Registry, the differences between Apache Avro™, Protobuf, and JSON Schemas, how to treat optional fields, some of the arguments between Avro and Protobuf, and why it took some time for Schema Registry to support JSON Schemas and Protobuf.

Later, they talk about custom plugins, adding another layer of safety in Confluent Platform 5.5, and their vision for data governance.

EPISODE LINKS

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

  •  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
  • ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
  • 👍 If you enjoyed this, please leave us a rating. 
  • 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
SPEAKER_03

Have you ever looked Confluent Schema Registry in the eye and said, I wish you supported Protobuff? Well, now it does. Today I talked to the product manager behind the changes and the engineering manager of the team who built them. We talk about schema registry and protobuf and avro and more on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. I am your host, Tim Berglund. I'm joined in the studio today by two guests, David Luis and Touchar Tole. David and Touchard, welcome to the show.

SPEAKER_00

Hi Tim. Thank you very much, Tim.

SPEAKER_03

Now, uh, we're going to talk today about um the newly released support in Confluent Schema Registry for protocol buffers and JSON schema. Uh there's a little bit more to the story than even just those two, but I think this is incredibly exciting news because personally I get asked a lot uh by developers who are building things with uh Apache Kafka with Confluent Platform. You know, hey, when are you going to support Protobuff? And it's so nice to be able to say, well, we do. So anyway, David, you are a product manager uh for Schema Registry. Too sure you're an engineering manager uh on that project. And as always, I I just love to get a tiny story from both of you on how you came to do what you do. So if you're listening and you think, wow, I'd like to be a product manager someday, uh, or I want to be an engineering manager someday, if you're not in that role right now, you'll just have a little story from our guests on how to do that. So, David, how'd you get to do what you do today?

SPEAKER_01

Um so uh uh my career started as a software engineer. So I had 10 solid years of software engineering, working with cryptography um systems and payment gateways. Uh after that I went into consultancy. Um and so spent a lot of time working with telcos in the APEC region, um, mainly doing reconciliation of events for them. And um after that I uh joined a company in the Silicon Valley in the ed tech space, so advertising technology, and uh I uh started working as a product manager for uh the data platform. And uh and and now I'm here at Confluent. I am the product manager for schema registry and data governance.

SPEAKER_03

Awesome. Tushar, how about you?

SPEAKER_02

Yeah, hello everyone. I'm Tushar Thule. I am an engineering leader at Confluent. And I can't believe I just completed a year at Confluent and I'm still as excited about Confluent and event streaming as I was on day one, primarily because I get to work with really smart people and work on really hard disputed systems problems. And uh before Confluent, I spent more than a decade at VMware, uh, where along with my team I solved hard-disputed system problems in infrastructure software space.

SPEAKER_03

Wonderful. Now, like I said, this uh the topic we have to talk about today, super exciting thing, long awaited. Um, I think it's gonna unlock the ability for a lot of people to use schema registry that haven't in the past. But um I want to start. Oh, and by the way, since there, everybody, since there are uh two guests today, every ball is a jump ball. David and Touchard, I'm gonna ask questions and you get to pick who answers them. So just do it and this will work fine. Now, um, before we dive into the product and the specifics of it, because there are a lot of interesting things to talk about there. Um let's talk about how protobuf and avro are different.

SPEAKER_01

Okay, I can take that one. Um so there's a few key aspects uh where they are different. Um, first of all, they're both great cross-platform binary data serialization systems. Um the first thing is that their origin is very different. So protocol buffers started at Google in 2001, and and then uh they were open sourced in in 2008, I believe. And uh uh Avro uh was born in the Hadoop ecosystem, so as a sub-project of uh uh Hadoop, and it became the de facto serialization format for Hadoop. Um so both are built around the concept of schemas uh that describe the structure of the data. Uh however, the schema language uh that describes uh uh the data it's it's it's different in both systems. Uh so in Protobuf they use the Protobuf IDL, um that is an IDL interface uh description language, where in Avro uh they support both an IDL version of the schema and a JSON format of the schema. The JSON being the mostly used, I believe, and the one that we support in schema registry.

SPEAKER_03

It's funny, I can't remember the last time I've actually seen Avro IDL in the wild, I just see AVSC JSON files.

SPEAKER_01

Yeah, yeah, that's correct. Um and then there's some specificities around uh how they treat optional fields, and that's a very important concept for uh schema evolution and making changes to schemas. I personally think that uh protobuf language is it's uh cleaner, easier to use. Um so, for example, in in protobuf fields or in specifically in protobuf tree, uh fields are optional by default. So they are backwards and forward compatible by default if you don't reuse the tag number. So that's very nice. I think it's it's uh makes it extremely easy to um to write uh uh proto IDL. Um but then on the other side, for example, schema evolution and this idea of evolving schemas and making changes and not breaking things, I believe it's more native to Avro. Um so in Avro you always need to have the writers and the writer and the reader schema with the data to read messages, uh, while in uh in uh Protobuff that's not necessarily the case. Uh Protobuf was created at a time where uh the language of choice at Google was C, so strongly typed type of uh uh language. And uh and that you can see that in the implementation of Protobuf, where code generation is a necessary step for you to use Protobuf. While Evro uh they they took a different approach, so they they have uh better support for dynamically typed languages, the code generation step and the compile uh uh uh phase is is optional in um uh in Evro. Uh the encoding also is a little bit different, the way that both encode data and the bit sequence is a little bit different. Uh because uh obviously both like they don't encode the field names, like in JSON, and uh and they're both extremely compact. Uh one main difference I would say in uh in uh in protobuf, you need to encode the tag number and the data types while in Avro you don't need, since you have the schemas always available to you. Um so the result of that is that we've seen and we found Avro to be slightly smaller uh than protobuf. And also as a result, serialization and deserialization time is also a little bit smaller in Protobuf. Language in terms of programming language support, uh I think they're both on pair, uh to be honest. They're well supported. And then the community support, uh, I would say I have to give the notch to Protobuf. Uh they have a bigger community uh than Avro. Um, so yeah, I would say these were the main differences between both serialization systems.

SPEAKER_03

Uh have you been much a part of the debate between fans of either camp?

SPEAKER_01

I I think there are like pros and cons about both. I I personally prefer protobuf just because the language, the schema language, it feels more natural to me. So if you're using to if you're used to Java or C, like defining the schema in an ideal format, it it feels more natural.

SPEAKER_03

Got it. And I didn't I should have been more clear. I didn't mean to corner you with like a political question there. Uh it doesn't it doesn't mean so much, it's good that to know that you have a preference. Doesn't so much matter to me what it is. Where I was gonna go with that was um when the debate happens, because it happens, right? Uh there are there are fans of this, this is a technology, sort of a binary technology choice that developers can make. So there are well-defined camps. Of all those pluses and minuses, what are the issues that normally get trotted out? Like if I'm gonna go get in a fight on Twitter about protobuf and Avro, what am I most likely to talk about if I'm arguing for protobuf? And what am I most likely to say if I'm arguing for Avro?

SPEAKER_01

Uh I think like the dynamic the dynamic support for uh Avro, people might say, hey, it's better to use if you're using uh dynamically typed languages, then Avro is the way to go. Uh Protobuf, it's very um uh it's very statically uh uh type uh language uh suitable. Um I the what I mentioned before around the uh uh optional fields and the way that you define optional fields, uh it's much cleaner.

SPEAKER_03

Sure, it's sort of evolvable by default since everything's optional.

SPEAKER_01

Yeah, exactly. Um but then on the other side, um if you really want the fastest and the smallest, then then yeah, Avro is slightly better than Perubuff. Um but again, they're both, I think, great modern data serialization formats, much better than the old days of ASN1 or some other uh more complicated and unnatural serialization formats.

SPEAKER_03

And folks, if you're listening with the kids around, I'm sorry we didn't give you a warning that David was gonna say ASN1. Uh that's sort of that sort of language on a family podcast.

SPEAKER_01

I I had my fights with ASN1 in the past, so uh I work extensively with telcos where ASN that's where you do it. Yeah, exactly.

SPEAKER_03

I um I actually never wrote any code against it. Uh like came close once and sort of stared into the abyss, and you know, a friend pulled me out and I was like, you know, shaking and took a few days before I could write code again. So um totally. And I I'll come back to that later. Actually, I want to come back to, and we'll try to give you a warning if you're listening with the kids before we talk about it again. But um the that there are some other features that are happening in schema registry that could potentially relate to other serialization formats. So we may we may say it again. Um okay, thank you. That's a that's uh I think a really helpful uh trade-off between their you know uh account of those two formats. Um and like I say, and I know this has been long awaited, so it's great. There are plenty of people who use Avro and build very successful systems with it. It's it's not and I I think you made the point that uh community-wise, protobuf seems to have the momentum. Like Avro hasn't doesn't change at a very high rate these days. Um which uh you know, I always want to be careful with that, because it's possible for things to become more or less complete, but still, community momentum kind of on the Avro side or the Protobuff side of things, but um you know, we don't mean in discussing uh confluence schema registry, we we don't mean to really take a position. Now you have options, and that's great. And you've got this other option, which is JSON schema, uh, that's also supported in schema registry 5.5. Tell us about JSON schema.

SPEAKER_01

Uh sure. So so by the way, the the choice of the languages, protobuf and JSON schema. Uh when I started Convolent and I started looking at what the community was asking, I started looking at what customers were asking. This was probably one of the easiest decisions that I had to make as a product manager. It was crystal clear that there was a need here. And that need was mainly in protocol buffers and JSON, and that's why we implemented those uh out of the box uh in schema registry. Um so JSON, I think the reason for JSON is just because it's everywhere, right? It's the language of the web, it's ubiquitous. Any technology stack out there can parse, read, do everything that they want with with JSON. Uh so I I could even say that it's the best default choice for you, right? So um so the first big advantage is it's it's a textual format, so it's human readable. And uh so it's quite convenient to to work with JSON uh overall. And uh and maybe you don't need the smallest and the fastest, and convenience wins over efficiency. And and JSON it's it's it's a great uh uh it's a great format for that. So historically, JSON didn't have the need for schema, but with JSON schemas, now we have a way to structure uh JSON uh documents, and that's great. Uh JSON schema it's it's quite powerful, it's still in draft, so it's still not a standard, and we implemented the the draft 7 of the spec, uh, but has some very nice features, I would say. Um uh one of those being uh the ability to perform more elaborated validation rules through the schema. I'll give you an example. Uh so with the JSON schema, uh you can include regular expressions on strings. So you can say, look, I'm only going to accept uh a string that matches this regular expression, or I'm only going to accept this integer if it falls under uh these lower and upper bound. So this type of extra, more semantic type validation rules, I think are very nice features for JSON. It's obviously not going to be as efficient as uh Avro and ProBuff for sure.

SPEAKER_03

Right. But like you said, if you're using JSON, uh you have already decided that serialization efficiency is not your driving force.

SPEAKER_00

Yeah, yeah.

SPEAKER_03

Now uh to be clear, this is JSON schema, which is not just vanilla JSON, right? This is there's extra kind of metadata or schema payload in each message. Is that correct?

SPEAKER_01

Yeah, yeah, that's correct.

SPEAKER_03

Yeah. Right. And we will put a link in the show notes, by the way, to the JSON schema spec if that is a thing you want to read more about. And frankly, I think uh it is a thing you should want to read more about if you're listening to this podcast. It's probably helpful stuff. So and David, you said this is the easiest choice that you've had to make as a PM, and I can absolutely believe that. If you're in charge of schema registry, like this is the thing that everybody wants and everybody needs. And it's nice, by the way. So if you haven't taken a moment to just acknowledge your gratitude for this moment in your product management career, you totally should, because the what the essence of being a PM is that every decision that you make is wrong, right? Somebody hates it no matter what. Um, and that's just a personality trait that which is one of the reasons why I'll never be a PM, because I would die within three months. Um, but uh that's that's the nature of the work, and this is awesome because pretty much nobody is going to disagree with you.

SPEAKER_00

So yeah, yeah.

SPEAKER_03

It's a good one.

SPEAKER_00

For at least once in my life, I think that everyone is happy around me.

SPEAKER_03

So right, right. And that's it. That's maybe the only time that you get. And you can look back on this uh late in your career and say, I remember that one time everybody loved me for like five minutes and then it stopped. Uh yeah, so that really is the reason we're rolling this out, right? Because there was just kind of uh broad-based uh community interest. Is there anything else going on behind this?

SPEAKER_01

Yeah, so so schema registry is a very important piece of our data quality story, and we are making some investments in data governance, the area of data governance in general. And so data quality and integrity of the data, it's an extremely important uh piece of that story. And so you will see more features that are uh related with data governance coming out in the next few quarters.

SPEAKER_03

Which uh gets me to another question I wanted to ask. There have been plenty of people, like I've talked to people who have said, well, no, we're committed to protobuf. That's just what we use in our organization. And you know, look, there are these other parts of the system that assume it's there and the cost of switching is just too high. We're protobuf all the way, and so we don't use schema registry. So um that, and we've talked about conflict schema registry on the broadcast or on the podcast before. Um, but as a refresher for people who haven't heard that episode, uh, what good does schema registry do me? Like, why couldn't I just have gone off campus and done all my serializing and deserializing with with protobuf and gone my merry way? Why did I have to, as it were, wait for schema registry support?

SPEAKER_02

Yeah, maybe uh yeah, I can take that. So conference schema registry uh provides a restful interface uh to you uh that developers in your organization can use to define schemas uh for their events. And once the schema is defined, they can actually share it across the organization and it allows you to do certain things, such as people will not step over each other's doors because they can evolve schemas in a way uh that is backward compatible. Uh, it can also be future-proof depending on the compatibility rules that you choose. And moreover, you can run this schema management functionality separately from your broker, which won't uh impact the performance of your brokers.

SPEAKER_03

Got it. So, in general, it's helping uh producers and consumers make decisions about whether version changes will be compatible, right? You if as as producering producing and consuming applications evolve uh and schemas evolve, which they do no matter what, because the world changes, um, this helps producers and consumers make sure that they're not going to either produce a message that's going to break things, or if they're about to consume a message that might break them, they'll know. Does that sound fair?

SPEAKER_01

Yes. I like I like to I like to add that uh if you have data, uh you should have a schema, period. And uh uh consumers work best when they know and understand the data, right? So uh I would say that schema registry and a schema-driven architecture is mandatory in today's world.

SPEAKER_03

I agree, uh David, I really do. And I that's a little um maybe um partisan of me to say that because it's you know it's a confluent thing, but you can use it for free, schema registry. Um, but I absolutely agree. I mean, this is uh at least to me a 10-year-old debate um in the the the heady days of all the NoSQL databases, half of which were were schemaless, quote unquote, uh in you know circa 2010. Uh you know, the the the the truth quickly emerged that there's always a schema. There's always a schema. The schema can be implicit. You know, your data storage uh infrastructure may not help you enforce it or make assertions about it, but there's a schema. And if you do have a quote-unquote schemaless system or no managed schema, that just means you manage it on read. So your your consuming application now has to do the work and the frankly complex exception management work of dealing with old versions of things and what to do when things won't work out. And so the the piper gets paid. Uh it's just then probably going to be the application developer who's writing the consumer who pays that invoice, and that will make that person sad. And we don't want to make that person sad.

SPEAKER_02

That's very true. And uh one thing I would like to add to you know, uh going back to what David was saying about data governance. So maybe you know, initially when you start out, you re you feel that you may not need uh any need for schemas or schema registry. But as the usage of Kafka or event streaming evolves in your organization, you would realize that you know in order to mandate or uh provide the guarantee about data quality or integrity, you do need this. Uh so it's better to uh start using it earlier than later.

SPEAKER_01

Yeah, simple simple, let's say simple data models evolve into complex ones very quickly.

SPEAKER_03

They do. They do. And anyone has writ who has written, let's say, built one system in the past knows that. Uh, so it's a good idea to get a little bit of infrastructure behind it. And the the trick is just the right amount of support, right? You don't want to make schemas incredibly brittle and difficult to change, because as I said, the world will change, your schemas will change, but you also don't want the Wild West where there is you know there's no assertions of of typing at all. Yeah, it's kind of funny. Uh, I think David, when you were talking about uh uh who likes who likes Avro versus who likes schema, seems like the dynamic language people tend to like Avro more, and the static typing people tend to like Like protobufmore, and and that's another um you know fairly venerable debate that we will not enter into here, but it certainly gets touched on. How about so we uh you mentioned ASN1 before, and again, I'll put a link in the show notes to ASN1 if if you don't know what that is, and I was joking about it being complex and difficult to work with. And that's true, but really uh what it is is just another way of taking the type system that you use in your programming language, or maybe that you use in your mind to think about the structure of data and reducing that to bytes, and then taking those bytes and turning it back into the type system that you're using in the abstract sense. It's it's a serialization format. Um and imagine I wanted to support that, or just something that isn't Avro, isn't protobuf, isn't JSON schema. We also have added pluggability into uh schema registry 5.5. So talk to me about custom plugins.

SPEAKER_02

Yeah, so uh schema registry, as you mentioned in the beginning, uh, is a source available uh project, right? You can go to Git uh Confluence, GitHub, and you can check out schema registry where you can see the code, you can see the changes that we are making. And to uh answer your question, I'll take it in two ways, right? One is that in first of all, what changes we had to make to make schema registry pluggable. That's one that is something that we did under the hood. And then now, if you are interested in writing uh a custom plugin for your favorite data format, such as say ASN1 in Tim's case, uh, what does he have to do? Right? So, and and by the way, as I mentioned, you can you know all the changes that we made are uh source available, so you can check it out. So the first part is you know, what did we do to make uh schema registry pluggable? So I'll keep it at a very high level, but there are two uh changes that we made. First one is that uh we introduced a notion of schema type uh and after JSON and protobuf uh from JSON schema and protobuf uh support was added, uh as of today schema registry supports three types of schema types, right? It is Avro, JSON schema, and protobuf. And second support that we added uh in the architecture was a notion of schema references, uh, and this is something uh that we plan to support for all three uh schema types. And uh so this is what we had to do to make schema registry pluggable. Now, let's say if you're writing a plugin of your own, then there are two main extension points that schema registry exposes. Uh, first one is a schema plugin that you would write for your schema format uh or data format, and second is a rest extension because as I mentioned in the beginning, uh schema registry exposes a restful interface to do cloud operations on your schemas. So rest extension is a mechanism to do that, and uh so there are a couple of uh great resources that I would mention uh that people can take a look at. So, first one is a blog that Robert Yokota wrote, uh, which talks about what we did for uh Confluent uh platform 5.5, uh which is on Confluent.io. Uh and second shout out that I want to give to Robert is uh he talks about how we can extend schema registry to write a plugin for something uh of your choice, and he gave a really fun example of how do you write a chess plugin uh to play with schema registry, and that is on his blog yokota.blog.

SPEAKER_03

We will uh definitely link to that blog post. You said a chess plugin.

SPEAKER_02

Yeah, so he so yeah, he talks about how to play chess with conference schema registry. So there is a really neat example about now you want to write a plugin for playing chess and using the two extension points that is a rest plugin and that is a restful uh API and uh plugin in schema registry, how you can you know extend it to do whatever you uh wish to do.

SPEAKER_03

Okay, that sounds kind of fun. Uh there'll be a link to that, and folks, you can read about that if you'd like. Now, one of the things with schema registry, uh we've agreed amongst uh the three of us uh partisans on this topic that it's essential if you have any complex system uh with you know schemas more advanced than what Hello World would require. And uh consuming and producing applications are going to change over time. Now that's the case with all of them. For any system that's not being decommissioned, schemas are going to change. So like we agree that schema registry is essential. Uh one limiting factor there is that, though, that is that that community of producing and consuming applications need to play ball. Uh so the interface is at the time of producing and at the time of consuming. The the place your application actually talks to schema registry and asks, is my schema compatible with this topic, or is this schema I'm consuming compatible with what I expect it to be? Um that happens during, well, basically, for simplif for purposes of simplicity, we'll say during serialization and during deserialization at the producer stage. So producers and consumers, applications are responsible for configuring themselves to use the schema registry. And if they do, if all of the boys and girls in the producing and consuming community are well behaved, then you're fine. Um but if someone is a rogue and writes an application that does not use the schema registry, that leaves out those config parameters, doesn't define a schema, and just produces some bytes, we can still do bad things. So, what has happened in Confluent Platform 5.5 to provide an additional layer of safety there?

SPEAKER_02

That's a great question. So far we have focused on enhancing confluence schema registry to add support for these two new schema types. But in reality, Tim, as you mentioned, uh your application might be a KSQ application or it can be a connector, which needs to use this new format, right? So uh if you're trying to realize your vision of putting event streaming at the heart of your organization, you need conference entire conference platform to support these schema types. And that's pretty much what we did as part of uh conference platform 5.5. So, as part of that, adding support, we added support for these two schema types across the platform. That means uh we added uh that means we added support, and I'll take example of just protobuf uh and we did similar changes for JSON schema as well. So, for example, for Kafka Connect, we added a protobuf converter uh so that it so that connectors can start supporting protobuf schema. Uh in case of Kafka streams, uh we have added a serializer deserializer for protobuf. Um if you take example of REST Proxy, which is another source available uh project on GitHub from Confluent, we added a new protobuf embedded format uh that is uh associated with REST Proxy. Uh if you take example of Java uh the Kafka consumers and producers in Java, we added Protobuff Serializer and Protobuff Deserializer. Similarly, if you take, and we already talked about schema registry, and in addition to that, there are a few proprietary components that we have, such as uh Conference Control Center or Confront Cloud UI. Uh in those areas also we have enhanced those components to support these new uh schema types.

SPEAKER_03

Awesome. So it sounds like kind of platform-wide there is support for the new formats.

SPEAKER_01

Yes.

SPEAKER_03

Cool.

SPEAKER_01

But I I would have I have to add uh there's something else. And and that is if the producers and consumers are not behaving right, there is still an extra level of enforcement that we build on the Confluent platform to avoid that polluted or bad data lands and uh and it's stored in a topic. And and that is what we call broker side schema validation, where we are now validating all messages, uh all incoming messages into a topic uh against a schema. And if if the clients are not behaving, as you as you mentioned, team, uh then um uh we at the broker side will reject those messages.

SPEAKER_03

Nice. So the broker, in other words, knows about the intended schema of the topic and actually communicates with schema registry uh regardless of what you're doing uh in in the application.

SPEAKER_01

Yes, now broker has a direct line of a direct line with schema registry, basically.

SPEAKER_03

Very nice. And that is the so-called confluence server. So that's the confluent platform uh broker software, so that's a uh confluent platform extension to the Apache Kafka broker functionality. Exactly. Right. And that's huge. Now it's it's this doesn't make schema registry, in my mind, go away as a useful component because you mentioned early on, David, you were talking about the various IDLs that people can use to define types. Um and there's the whole and we're focused on basically, you know, if I could summarize uh this episode in two words, it's uh yay, protobuf. But kind of the broader uh schema registry uh concerns, just because there's schema validation in Confluence Server at the broker level, like and that's huge that there is, if you're serious about data governance, that's a component of data governance. Uh, but that does not make schema registry useless, because the whole practice of defining your types in an IDL is potentially a very important part of the way a development team collaborates around schema evolution. Because, you know, by virtue of you using an IDL, uh to put it in very concrete terms, that means there's a text file that programmatically defines your type. And tooling can code generate from that and give you the actual objects that you use. And now that pushes all collaboration to that IDL file. So, oh, I want to change a type. Well, like, you know, we have ways of doing that. We have branching and we have pull requests and things like that. Everybody who makes a living writing computer programs, basically these days, kind of knows how to do that sort of thing. And what what is otherwise a difficult problem of where do we put that schema information and how do we negotiate changes to it? We get to bring all of our rather wonderful remote collaboration tools that we use to negotiate changes to text files, all those software tools and you know cultural patterns and everything that we use to do pull requests, really. We can now do that around schema. So broker broker level support for schema validation is incredibly important and a step forward towards some stuff I want to ask you about, but it also does not obviate schema registry. You still need schema registry in your life, it's still very useful.

SPEAKER_01

Correct. Yep. Yeah, when I when I think actually when I think about schemas and the need for schemas, I always think about these five things. Uh, so the first one is uh uh the description of the structure of the message. The second thing is the validation of data against uh that structure. The third thing that I think is very important, and you kind of like touched on that, is it serves as documentation. It serves as a data catalog for your data. So everyone in the organization can find what kind of data exists. Um the fourth thing is uh the idea of reusing schemas, right? Data structures that are used across different applications. So the idea that they can reuse these data structures in a schema, uh I think it's it's incredibly valuable. And the third and the last thing is evolvability, right? So data change without breaking things. So it really helps across these four uh these four variables, I think.

SPEAKER_03

Yeah, could not have said that better myself. That was great.

SPEAKER_02

Um, I have uh one comment to add to the broker side schema validation. So uh as David said, right, one of the pillars that he mentioned is making sure the data that enters into the broker is valid. And uh one clarification is that that uh support is entirely dependent on schema registry. So it doesn't really make schema registry, it it actually makes schema registry more important because your broker is now going to communicate with schema registry to make sure to validate the schema, uh whether the data uh in your message is valid or not. So it makes schema registry all the more important.

unknown

Yeah.

SPEAKER_03

Good point. Much more important, not less less important.

SPEAKER_02

Yeah.

SPEAKER_03

Uh I want to ask a little bit of a forward-looking question. And with an engineering manager and product manager for schema registry on the podcast, I understand that there are certainly limitations about talking about roadmap and things like that. That's not what we're gonna do here, but uh maybe more broadly and with the explicit acknowledgement that this is not this this is not a roadmap discussion. That's hashtag Safe Harbor there. Uh, I kind of threw out the term data governance and said that that broker side schema validation is an important component of data governance, um, almost like a minimum baseline for it. What is and uh again, jump ball, either of you can answer this. What is a more comprehensive vision for that at the far end of the enterprise scale in enterprises with demanding data governance requirements? Where do they have to go? What does that have to look like in its end state?

SPEAKER_01

Our vision for data governance uh at Confluent, it's really to allow our customers to fully adopt uh Kafka and event streaming. And for that, uh we realize that there's key components that uh need to live on top of Kafka to make that happen. And so data quality is a very important piece of our data governance vision, right? So the ability to say that streams are fit for consumption and that the data that is stored and travels through Kafka, it's safe data, it's it's clean data. Uh the second the second piece uh it's around data discoverability. So very quickly you see a rapid expansion of Kafka in the organizations, and you can go from 10 topics to thousands of topics. So the ability to understand and discover the data that you need, I think it's really key to unlock expansion uh across the entire organization for Kafka. Uh and then the third thing it's around understanding how data travels, so understanding the origin of the data uh uh in Kafka. So the idea around data lineage and being able to understand the lifecycle of data from the moment it enters to the moment it exits Kafka. I think those ones are three very uh important uh aspects of our data governance vision.

SPEAKER_03

If people wanted to try this stuff out, if they had questions they wanted to ask about it, what's uh what do you recommend they do?

SPEAKER_02

Yeah, so uh I mean this is music to my ears. We would like to know, we would like uh listeners to give it give these features a try. So there are a couple of ways I can think of. The first one is you know, if you're feeling lazy, then you know maybe uh try manage service on Confront Cloud. Uh there are three different tiers, and you can pick the one that is right for you. And uh these two schema formats are supported across all tiers. That's option one. Option uh two is you know, if you're if you want, if you want, you can download and run Conference Platform, and all you have to do is go to conference.io slash download and you can get the latest uh release and uh that is conference platform 5.5 and uh try these features out.

SPEAKER_03

My guests today have been David Luiz and Touchar Tully. David and Tushar, thanks for being a part of Streaming Audio.

SPEAKER_02

Thank you, Tim. It was a pleasure. Thank you, Tim. Yeah, I have one request to uh to your listeners is that we would like to know what else you would like to see in the domain of data governance. Uh yeah, looking forward to hearing from you. Thank you.

SPEAKER_03

And there you have it. I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at TL Burgland on Twitter. That's at T L B E R G L U N D. Or you can leave a comment on a YouTube video or reach out in Community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which we think is a good thing. So, thanks for your support, and we'll see you next.

Head Shot

Tim Berglund

Host
Head Shot

Adi Polak

Co-host
Head Shot

Viktor Gamov

Co-host