How AI Is Changing Apache Iceberg with Russell Spitzer | Ep. 30

Adi Polak talks to Russell Spitzer (Snowflake) about his career in open source data infrastructure. Russell’s first job: software engineer in test at DataStax. His challenge: making Apache Iceberg ready for AI and streaming.

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

•  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
• ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
• 👍 If you enjoyed this, please leave us a rating. 
• 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.

Transcript

SPEAKER_01 0:00

What will the future even be of having all these different coding languages in a world where the LOM can write all of them pretty much equivalently well? It's fine if a vibe code project adds like 10 extra classes when only one was required. For something like iceberg, we need to keep it tight. We can't turn it into a hairball.

SPEAKER_02 0:17

Hello, I am Adip Polak with Confluent Developer, and I invite you to join us today to my conversation with Rysel Speitzer. Rysel is a principal software engineer working for Snowflake, and he is also a thought leader in the industry and supported and contributed to many open source projects. So sit down, grab your favorite drink, and join us for a fascinating conversation on how AI shapes our distributed data infrastructure, and what are some of exciting features that are coming out specifically for iceberg? It can make your life a lot more easier when thinking about unstructured data, referencing other file formats from your catalog or from your tables, and generally making AI workloads accessible, available, and fast. Yes. Also fast because we need to take care of streaming architecture. Hey Russell.

SPEAKER_01 1:19

Hey Addy.

SPEAKER_02 1:21

So good to see you. It's been a while, and I know you've been busy recently. So what's what's happening with iceberg?

SPEAKER_01 1:31

Yeah, I mean iceberg is going through a huge transition time right now, as I'm I'm sure everyone is, right? We we've come to this new world where this LLM-based AI stuff and agentic things is everywhere. It's part of everything, uh, part of coding, part of reviewing, part of product management. And iceberg is changing to kind of meet that need. Uh one of the cool things that we're working on right now is Iceberg format version v4. And the two big targets for V4 of Iceberg are to make it a great solution for AI use cases and for streaming use cases. And this is really showing the history of the project, right? It started as this format that was really oriented towards this kind of hive replacement, treno replacement, data warehouse replacement. And that has a really different characteristic from streaming workloads, as I know you're extremely familiar with, and these new AI workloads, which are coming in with some multimodal things, we're seeing vector stores, we have all kinds of different uh challenges that we need to kind of address. And one of the coolest things about working in open source, especially on something like iceberg, is we can take ideas from all over the community, all over the industry, and integrate them into something that hopefully ends up working for everybody. So I'm very excited right now about Iceberg V4.

SPEAKER_02 2:49

Cool. Yeah, it's an exciting time to be working on the infrastructure for sure. I think it's kind of coming back now with AI. We're realizing that, hey, data infrastructure does matter in the age of AI and we do need uh specific, you have specific, we have now specific requirements from our infrastructure, whether it be analytics or a streaming um space, but we do need other stuff. So what are the new things in in v4 that enables us to run AI and also whether some of the things you know we should have in it?

SPEAKER_01 3:28

Yeah, I mean, so the things that we have pretty much consensus on now for v4. And since we're a community project and we don't we don't actually have a v4 until the whole community gets together and votes and says, that's what v4 is. Everything I'm saying is pretty much what is the current state of the project. But the biggest things we're doing for AI enablement here is we are adopting a method of basically going uh vertical with your changes to a table. So previously, the way we always like to think about tables is you know, we've basically got this relational table, and every time you add a new row, you're just adding things onto the bottom, right? And then iceberg later added the ability to do acquit uh delete files and uh delete vectors, and that'll just let you knock out rows. But again, these are horizontal cuts to your table. What we didn't have is the ability to divide your table up vertically. So basically saying these columns are in one file, these columns are in another file, and then the ability to replace a column within a file without deleting it. So basically, previously, every time you wanted to get rid of a column, you had to rewrite the entire data file and all of the other columns that were adjacent to it. Uh, this is really bad for a lot of AI use cases where you're essentially maybe adding a new feature every day. And that feature is essentially just an extremely long uh set of vectors or set of doubles or floats or whatever. It's it's usually this maybe sparse, maybe dense, but columnar addition to your table. So basically, instead of our old tables where we added at the bottom, we are now adding sideways. So as we go sideways, we need to adjust the table format to handle that sort of thing. And this is, you know, uh an approach that's been taken by a few other projects, and we're excited to kind of make it into a first-class feature within Apache Iceberg. Um, the other big discussion that we keep having is how are we going to better deal with the multimodal aspect of a lot of these table formats? And there's there's two big approaches that I think are kind of going being being worked on in conjunction between the Apache Parquet community, the Apache Iceberg community, and a lot of other communities, where we're kind of trying to figure out is this a file format responsibility or is this a table format responsibility? Because there's there's two modes, at least I think, of going about handling that. And one is we embed our multimodal, our non-structured data directly in the same kind of data files that we are storing, everything else. So this is an approach that I think like Lance takes, and uh I think some other formats are also trying to do this this kind of approach. Um and that's one thing you can do. We could do the same thing with parquet, and we're kind of looking at parquet about does it make sense to improve the way we're storing things so that we can do something similar, and then we can use again within the table format the ability to store columns separately to put larger objects inside of completely different files or something like that. The other approach, which I think is really interesting, is actually just using a reference within your relational table to uh unstructured data that is not a part of the row structure itself. So this is basically saying, why are we even putting this data, which we know is actually very compressible and very, you know, good in its current formats? Like a PNG is pretty great as a PNG file. You can zip a thousand of them together and get pretty good behavior if they're all pretty similar in structure. Why do we want to move that into a structured file that has a lot of other things in it? So there's a question of whether we could just store references to those files within a relational table and then have the table format be responsible for the lifecycle management. So, like whether or not you actually have credentials and access to get to those external files or when to remove them because there's no longer any references to them in the table and things like that. So there's two directions I think that Iceberg is going there. So we're we're obviously working very closely with the people in the parquet community to try to make sure that we have support for storing this kind of data. And also we're approaching that external reference idea as well to kind of see if that is the right way to go uh at the same time. Because, you know, I think there's there's different, there might be different fits for different types of multimodal data.

SPEAKER_02 7:48

Yeah, no, it's um it makes it really interesting. It makes a really interesting problem. I remember back in the days, and and maybe that's a little bit old, but if you know, we held references inside a table for for years, right? And the always the challenge was who manages that life cycle to the way you that you mentioned, because it's um there is a question of transaction and atomicity of that transaction that we need to think about. Like now the file got changed. How long does it take to manifest inside the table that holds the reference that that file now got changed? And which side is responsible uh for that atomicity to happen? And how do we pair these two systems and perhaps these even two tables together in order to make validate that that actually the case? Um Yeah.

SPEAKER_01 8:44

I mean that's why I think it's really like a table format level problem. Like if we can define that at this multi-file level, then I think we we can actually go forward pretty confidently and say like these are expected behaviors, and then we can have engines and everyone kind of work within that framework. Because I I know that we we did the same thing back in the day, and a lot of jobs I've had, and like if everyone's doing it on their own, that's the problem, right? Because we end up with a ton of divergent things and none of them work together because the way you do it doesn't work the way I'm doing it, and you end up with atomicity and consistency issues and broken guarantees.

SPEAKER_02 9:20

So yeah. And from my experience working with iceberg, and I also reviewed a couple of really really great books from from you know, great authors from our community. I had the opportunity to contribute there as well, kind of like more behind the scenes as more of a technical reviewer. Um, there is the notion of branches in iceberg today.

SPEAKER_01 9:43

Yeah, I mean, iceberg already has a lot of those components we would even want to do this sort of thing, where we're where you want to just build a completely separate branch, do all of your testing there, and maybe do your management of files, and then only merge it back in when you're like, I'm sure the state here is good. I mean, I haven't written up any ideas on how we're actually going to do that for these reference types. Because right now, you know, the question is what should be in a reference type, where should we define that? Um, we've had some proposals that maybe we should even put the base definition of what that structure looks like in parquet, just so that we have a kind of global definition that isn't as table-biased, and then we can put our rules around it inside of iceberg.

SPEAKER_02 10:28

Right, right. Well, you know, and if I look at Kubernetes world, if I look at uh things like Kafka, uh RocksDB as a source of truth is always uh you know a solid idea and direction. But you know, I I'd be curious to follow along and see how that works.

SPEAKER_00 10:47

Now a quick word from our sponsor. Confluent developer the podcast is brought to you by Confluent Developer the Website, which has everything you need as a developer of data streaming systems. And it's completely free. We've got curriculum, hands-on exercises, executable tutorials, the online data streaming engineer certification, also free. A way to find a meetup near you, those are free. Everything is there. I really want you to be successful in your journey as a data streaming engineer, and this is the site that has what you need. Check it out at developer.confluent.io. That's developer.confluent.io. Now back to the show.

SPEAKER_02 11:27

Yeah, and also that there is the question, uh I think you touched about it a little bit, of streaming, because adding another layer of atomicity and adding in perhaps another source of truth database that is strongly consistent um adds latency.

SPEAKER_01 11:48

Yeah. I'm really hoping we don't have to do a second system to make this work. I think we we may design it with limitations that make it so that you we just don't need that. Um, you know, do immutability and and other stuff like that, where you know, there is one state of the table, the files have to be in the same state for their entire lifetime. You may not change the state of a file, offsets should always be true, you know, that that kind of stuff. For streaming, though, within iceberg v4, the things that we're excited about is we're really trying to break down the amount of time it takes to do a commit. And that's one of the other big v4 changes, is we're changing the structure of the metadata tree. So it uses almost all the same ideas, but pretty much anywhere where we couldn't cache something before because we were changing it so often, we're we're breaking that and making it more immutable, more lasting. So basically, we looked at all of these ideas that we've been applying to our data files, like the ability to knock out a row in a file without deleting it, and the ability now to replace a column in a file without deleting that whole file. We're like, well, why don't we take those approaches and apply them to the metadata layer as well? So when you do that, suddenly we're we're basically gonna set it up, and and I don't want to go too deep in the technical details of this, but there'll be one file that's basically your entrance into the rest of the metadata tree that will be completely overwritten on every commit. And we're gonna keep that very small. So something like 400 kilobytes is what I keep throwing around, because on our tests we saw that you know, doing IO to S3, you basically have no difference between like four kilobytes and 400 kilobytes. I mean, there's a slight difference as you as you go up and up, but you know, if if you're going to open a handle to S3, you might as well uh give read that many bytes. Anyway, so you you read that many bytes back and you're gonna replace that every time. And when you add new files, you'll add them directly into that layer. So the way you used to have to do it is if you wanted to add a new file, you had to also write another file that a manifest file that it would refer to. And then the data file would be underneath that. And that meant that when you were creating a snapshot, you had this serial process where first you made this file that had a listing of the data file, and then you made another file that referenced that file. And because that's you know a serial process, you basically had to pause each time to actually do a commit. Now you'll basically say, I'm only overriding that top-level file, which hopefully you have cached in a memory or like a you know something like that. And you quickly change it, you write that new file, and you're done. So for a streaming use case now, you should have only one IO operation required to do your commit each time, as opposed to two that have to happen in serial. So hopefully that helps a lot. And then the other thing we're doing, right, is that because this will always be in this tiny, tiny layer up at the top of this metadata tree, that should also be a pretty low latency operation, as opposed to having to do some more drastic restructuring. One good example of this is if you wanted to ever delete something before out of an iceberg table, you basically had a manifest that had an entry that said this file exists. To remove that entry, you had to delete that manifest file. You had to completely rewrite it. Because if it still existed, you would always have this little entry that says that file exists. So we would have to delete it and write a new version of it. Now, since we're using those abilities that we had on data files at the metadata layer, I can just say I'm taking that delete and I'm applying it to a manifest file and just have a little marker. So basically a delete vector that says that row doesn't exist. So we can basically, without disrupting your cache, that file is still cacheable, you can still keep it in memory. You can mark things as deleted. So suddenly deletes are way, way faster than they used to be at this metadata commit layer. So those are the kind of things we're pulling together. There's a lot more details on how we're kind of structuring this, but the end goal is to have that commit process be fast enough that you can start doing very rapid commits to an iceberg table if you really want to.

SPEAKER_02 15:48

That's awesome. I love the idea of caching, and it uh reminds me a little bit of uh Hadoop world back in the days and some of the latency challenges uh we had with uh with HDFS and uh and yarn and it's the rest of the zoo.

SPEAKER_01 16:02

Uh yeah, I mean we don't want to do I. That's the story of all of these technologies, right? Is we're just trying to avoid I.O. all the time. It's what can you not read? What can you not write? What can you cache and not have to access again?

SPEAKER_02 16:20

Right. We've learned as as an industry, as an open source community for sure. And I think AI pushes us to for that as well. I mean, latency is waiting for solutions like cloud code or you know uh other other solutions like vertex AI and uh Gemini and and so on, waiting for them to actually uh give us back a response starts to be you know frustrating. It's like I gave you a prompt, can you can you make it happen?

SPEAKER_01 16:57

This is this is actually one of the reasons that I of all of the technologies that I think people are working on right now, I feel like table formats will probably be the least changed because they're at this layer where you need to interopt with them. And if you have to, like, I mean, right now we're in a great time, right? You can basically say, please write me an interop layer between these two type file formats, or write me something that converts this query into another type of query. But the problem is the underlying data structure is still held by whatever its original limitations are, and all those layers that you build in between just make it worse. So we need to have these interoperable formats at the bottom of everything, because otherwise, regardless of how fast your models are or how smart they are, they still will have that translation cost or that modification cost. And the only way to make it better is to fix the I.O. at the at the very bottom layer of it. So I think it's definitely interesting time.

SPEAKER_02 17:54

Yeah, and and that requires, you know, you know, solid engineering, a lot of the work that you do with the open source community and uh bringing different people from different organizations together. I believe this is where when people ask me, like, what's why community, right, why open source? I always say this is this is the power. It's like you um it's a hive of minds that comes from across uh you know the industry and make decisions and discuss the challenges and uh go very deep into the technology and the nuances of you know the exact format, the solutions, what do we need, and what's gonna propel us forward and um be a solution that it's kind of solves the problem rather than a company specific that sometimes can be you know very siloed.

SPEAKER_01 18:45

Yeah, I mean that's that's the danger with working on you know a company specific project is even even if you have the best intentioned engineers, there's so much pressure from you know the the folks with the dollar signs and the C titles, right? We're like, you know what, but couldn't we just delay this? Or you know, shouldn't that be only available to our users? Or oh, if we didn't do that, wouldn't it make it harder for our competitors to use the same thing? And when you're in a true open source governance, and I mean that's one of the things I love about the iceberg community, is you just can't do that. Because even if one person says, hey, I'd really love to do this thing that would make it harder for everyone else, you'll get nine people who will say, No, that will make it harder for me in my job. So I'm not gonna support you in doing that. You better have a really good technical reason why we should all take on that burden. Um, and it the reason can't be it'll make you more money in Q1.

SPEAKER_02 19:36

So Yeah, it's I think as engineers you want to solve the problem. Um and sometimes organizations problem across the board, it's like, hey, we we want to make money. And then the question is like, how do we how do we support that motion in the uh while still solving the real the real engineering problems?

SPEAKER_01 19:57

Yeah, I mean I like again, that's another thing I kind of like about the formats, is that the only the real value of iceberg at its core is this interoperability and this open governance layer. So all the people in it keep that as like a very strong focus because there's really no point in supporting iceberg if you're the only one who supports iceberg. We we want to support iceberg because everyone supports iceberg, which means that you know the the pitch I do all the time internally is I'm like, every person who's using iceberg is a zero friction customer to Snowflake, potentially, because they are using the four method. We can integrate directly. There's no ETL, there's no convergence, whatever tools they have can keep working while they try POCs and things like that. And every other vendor, I think, sees it kind of similarly. This is the open playing field, which means that everyone can try whatever they want. And of course, from an end user perspective, I really love it because that makes everyone compete to make sure that this layer really works well and is really compatible. And I think the the general industry benefits a ton from having this like community project, similar to, you know, like HTTP standard and things like that, where like just having an industry standard where people can all agree benefits end users way more than it benefits the end the companies, but they can also take advantage of that because you know everyone who's compatible can build up whatever their offering is on top of it.

SPEAKER_02 21:23

Yeah, yeah, I love that. And it's so true. It's faster adoption as well as faster POC. Imagine if a customer now wants to see it's you know, if they they want to buy a new software solution, right? We do it all the time, we evaluate software. We want to make sure we can actually build that POC relatively fast and see the value fast, if you know, if we can. And and if that's not the case and it's too challenging, then we flag it. Hey, there's gonna be a steep learning curves, here are the integration challenges. This is something that we expect you know the software company to solve. uh for us to integrate with that with them. Um and then it's it becomes kind of like a question of back and forth and if it's the best solution for us. And of course there are other comp competitions, right? The competitors on the market that might have a better, simpler to integrate, simpler to run a POC solution. It's gonna take us as a customer and this is uh it's a huge huge power for for open source because um if we can enable that then we have a faster path to win. So that's true. Cool. So we talked a little bit about AI and some of the AI workloads and some of the requirements especially around referencing when discussing you know unstructured data and then we need to dive and kind of bring uh unstructured data um that is compressed and that you know part of the table is kind of like a binary nasty binary uh format um and we talked about a little bit transactionability and optimicity and so on and a little bit about streaming. I know you know you mentioned there is um icebergs of support bringing in your own catalog as well right there's a dedicated um API and there's a dedicated spec that you can work against uh as a developer to bring your catalog and recently I've been seeing more and more uh about some of the catalogs that you've been working on uh and I'd love I'd love to learn more and also you know how you see that connects with maybe the new format that you know you're developing.

SPEAKER_01 23:42

Well I mean the the iceberg rest specification is also growing a ton right now and as you mentioned it's basically the the lingua franca we've made for integrating with an iceberg catalog which is a completely separate service we basically define this is how you will talk to a catalog for an iceberg table this is how you will get back metadata this is how you will propose new table updates all the all that kind of stuff and what's really great about having this API so well defined is that now it's just really easy to move your catalogs uh your your tables from one catalog to another so in the old days before we had this you basically were like I'll pick hive and you pick hive and you configure everything specifically for hive all of your clients use hive specific code hive specific version things and what if you had a an issue you really had no way to control anything because everyone is talking a hive specific language to your catalog and your Hive Metastore doesn't really know it's dealing with iceberg tables. So there's no way to differentiate what's happening or tell the client like oh no I think you're using a bad version of this iceberg library because you're using a Hive library and all you have is this thrift request. So moving to the rest suddenly allowed everyone to to put more information in their in their requests and have all of these catalogs have this front end that says okay I'm dealing with an iceberg client oh this is a bad version I'm just not going to accept requests from you. I'm gonna send back an error message that you need to upgrade. Or you can take requests and you can say well I have all my commit logic now internally because in the old days basically the the the client would stage the commit using the catalog's own specific API. So for example for Hive you would say Hive I need you to lock this table reference from the client and then on the client you would check to see whether you had a valid commit and then you would then swap the table metadata using an alter command and then unlock the table with a direct hive API again. So it's all very hive specific and it's all done on this client. So the clients can propagate weird errors if your client doesn't know how to interact with the underlying storage. So this is why I really love the REST API, because it puts a layer in between the client and the actual persistence for your table entities which means that it's it's much easier to centralize fixes for this side of thing or stop badly behaved clients from doing something and disrupting your whole setup. It also of course completely detaches authentication from all of the backing storages so you can have whatever auth you want. You know if you're using HMS before you basically had to use Kerberos right like that was your that was going to be your security system sorry that's just it but if you talk with a REST client right it's whatever that catalog decides to expose. So you know we have a lot of OAuth things so I work on Apache Polaris which is a project that uh uh a while back was was started by Snowflake and they they basically as I was hired and started working on iceberg there were like we need to be more open source we can't control these things we need to have an open community that's behind them so they took the Polaris project they donated to Apache and now we're just one of many contributors to this project which you know because it has the REST API is compatible with iceberg clients all over the place and provides all kinds of awesome features and integrations with all kinds of identity providers and offlayers. So you know it's it's really great to kind of be in this space now where if you want to you can use Polaris as your catalog. Or if you want to switch you can go find another producer of an Apache iceberg catalog that services the REST API and your switch is actually pretty easy because the client is exactly the same. It's an iceberg REST client. So again a lot of cool benefits that come to any random user and it's nice to just be able to keep working on open source even at that uh part of the stack.

SPEAKER_02 27:41

Yeah it's it's pretty amazing catalogs has been I you know forever kind of like the war of catalogs and who builds a better catalog and uh I'm I'm curious because you did mention Hive and HMS. How does Polaris and you know we all had our fair share of uh of working with Hive and fixing tables and and the references and the and and so on. How would you say Polaris is different today?

SPEAKER_01 28:12

So I mean Polaris was built kind of for this new iceberg Rest world as like a first class citizen. I I do want to say Hive Metastore also is a has a first class iceberg REST API right now. So you don't even have to talk to Hive using the Hive client. You can actually talk to it with the REST API. The biggest differences are the way Polaris is built is basically structured around iceberg metadata as a first class construct. You know it basically assumes that you're going to talk to it, you're gonna talk to this REST API and your security, your auth and all of these are basically pluggable components into that system. So you can decide whether you're going to use you know OAuth 2 or something or some other layer of security as opposed to in the Hive world where it's very strictly tied into the Hadoop ecosystem. So you know what's what's supported is what's in Hadoop. And because of that you know like I said Polaris can support Ved did credentials and support various cloud providers with pretty pretty easily and um one of the other cool things is we also are able to because we didn't assume right that the catalog actually understands how to communicate with these tables right you basically surface up this iceberg metadata whereas in the Hive Metastore we have input formats that are like bound to every single table. So there's actually like literal code you are supposed to read a table with stored into the catalog and you have to know that you can load that class and everything. But we've got support for a whole bunch of other table formats now in Polaris. So you can actually use Lance tables in Polaris you can use hoodie tables. I'm forgetting what we just added but um you know there's all that cool stuff. And then the pluggability also exists on the backend. So Polaris itself has a persistence layer and we made that persistence layer pluggable as well. So you can choose any relational database you want to be the persistence layer or you can use something like Mongo or you can use BigQuery like you can now hook that up to NoSQL things as well and have that be the backend for your your system. So it's it's really nice there. So I I'd say the biggest differences there are those kind of pluggability layers that didn't really exist within the original Hive code and of course almost no connection to the Hadoop ecosystem so no dependencies on Kerberos or or strong ties to Java.

SPEAKER_02 30:32

That's really cool. No strong ties to Java it's uh that's a really good way to uh to segment that beyond you know the the the fact that we have a REST API actually now that you mention it I have two questions uh I hope you're ready let's start with the strong ties to Java um a lot of the data community and data practitioners right whether they're your data engineers a data science a data analyst uh analytics and there's so many titles today um kind of starting using more Python right in a lot of resolutions in the data space are actually were were built with you know JVM in mind and and and Java and so that uh interoperability between those two frameworks especially around performance right like how do we build that um was always uh a little bit of tricky it's like you don't want to rewrite your pipelines in Java and yet again you do want to get the highest performance possible um and the fact that you mentioned that I think that's that's a really good val added value to Polaris as a catalog because uh uh there's a lot of friction there.

SPEAKER_01 31:49

Um I wonder you know if there's more things that you know made the community to make this decision beyond what I mean I doubt that was one of the main reasons behind the REST API in the first place right like there's there was almost no way to get a Python client with security to work properly with a Hive purely Hive interface right because you you had to get the basically you have to pull in all of Kerberos to make that work properly. And that just didn't exist. And the moment we have the REST API up suddenly you could write a Go client a Python client a C client a Rust client a Java client and they don't have to know anything about Java to work properly with the catalog. So I mean you'll see right now the Apache project for Iceberg has all of those implementations currently being worked on. And it's really exciting to me it and one thing I've seen recently too and I don't know what if we're we're we aren't going to see this for a while but a lot of folks have told me and I've seen some benchmarks that prove this that the the LLMs and all the coding agents are actually really good at writing Go because of how opinionated Go is. So the the difference there is that when you write Python right there's many ways to write something right there's there's many many right ways to do a lot of things in Python. You've got list comprehensions and map functions and all of these like different structures. But Go does not have that same flexibility. So some rank rankings I've seen have shown that these agents are are way better at writing Go than they are at writing anything else. So I'm wondering if we'll actually see a big push for more Go libraries. I mean it also doesn't really matter because it's it's really a difference in token cost, right? On how long you let it churn and like write things and have to fix its own mistakes. But once we get to the point where LLMs are are commoditized and we actually are playing the true true cost for all of this I'm sure that's the we'll we'll get a big push into to working with more efficient languages for LLMs even if they are not the favorite languages of humans.

SPEAKER_02 33:52

Yeah that's a really interesting observation that I must agree with it's the cost of tokens as uh you know like leaders like Jensen said it's you know engineers we're gonna start negotiating that in our contracts like how many tokens am I gonna get uh as part of my compensation package so I can actually you know perform at the level I want to perform for my boss right yeah I think I think almost no one is really is really leaning hard on what that cost is going to be because I again I do think we're at a we're at a point now where uh VCs are like paying a lot of our token costs right now.

SPEAKER_01 34:29

Like your our tokens I think are very subsidized. And we will get to a point where that will not be the case and we'll probably get this huge pushback to be like why are you using why are you using your your engine to write C code when you could be writing Go code or or Rust code which we know the engine we know the LLM is better at more efficient at and can do for a cheaper token cost. It'll be much harder to justify language decisions. I mean I I've wondered a lot on what will the future even be of of having all these different coding languages in a world where the LLM can write all of them pretty much equivalently well with these slight cost differences. You know it it'll it'll be an interesting world we'll we'll be in because like we could have LLMs just write assembly like directly if we wanted them to but we obviously that's probably not the most cost efficient thing and I think that's what will will end up being the the final trade-off is what language is the cheapest to write efficient code in.

SPEAKER_02 35:27

Right token economy for engineers maybe should you know maybe you should write a book about that could be interesting.

SPEAKER_01 35:35

Everything is churning so much right now that it's hard for me to say I mean but I I definitely think that that it will come up in the next next few years people will start I mean I I couldn't I wouldn't be surprised if someone writes even a a language specifically for LLMs to write that they can do more efficiently or has the right kind of you know built-in checks that let it do one shots better right like that's that's really the goal. You want it to be able to do this with the smallest token cost. And anytime it writes something where there's a possible syntax error like that's that's a bad language for an LLM to write code in.

SPEAKER_02 36:11

Right and and that happens a lot. I had another question. So we has we have the REST API for uh for working with iceberg and you know mcp uh is kind of on the rise I wonder if this is something the community looked into it's like having a dedicated mcp server instead of uh you know writing dedicated you know working directly with the Rust API.

SPEAKER_01 36:37

So the Apache Polaris community actually already did and built a MCP server to go along with Polaris. So if you want to I believe it's in our tools repository now so you can basically stand it up at the same time and have your Polaris catalog directly MCP enabled out of the box if you want. So it's definitely something we're thinking about from the server perspective. As like an iceberg project itself there you know we don't really have any server components there. So there's nothing really there. But we actually we just merged our our first agents md file so I was kind of excited about that because I think we were a little slow to the game to to adding that we we've been seeing the AI PRs, we've been seeing the AI issues and uh you know we're entering a world where I think people do realize have to realize that they need still need a human in the loop. I mean I all the buzz is great, right? But there's a lot of times when AI will do things in ways that don't make sense to a human reviewer and you need a human to check a lot of these things especially on a code base where we can't have it grow indefinitely in the way that I think a lot of vibe coded projects can grow. You know like that it's fine if a vibe coded project adds like 10 extra classes when only one was required. But for something like iceberg we need to keep it tight not only so that we can keep working on it and understand it but so that LLMs can continue working on it understand it. We can't turn it had turn it into a hairball which makes it impossible to work on. So you know we're trying to trying to work on that.

SPEAKER_02 38:08

That's that's a solid point and I see it across the open source community with uh specifically Kafka and Spark and Flink and so um and thinking about this token economy at the end of the day it's it's gonna cost us now it's relatively accessible I would say um but you know who knows what will happen a couple of days months years down the road yeah I can't make economic predictions but you know people are spending a lot of money right now and they need to make a lot more money back.

SPEAKER_01 38:41

So that money's got to come from somebody and it's probably gonna be us.

SPEAKER_02 38:45

Yes 100% all right thank you so much um is there anything you know we have a lot of people listening in that thinking about their careers that they're in the data REI space in the open source space and so on. If you would to look back and give them you know one or two pieces of career advice what what would that be?

SPEAKER_01 39:09

I mean this this is a hard time to give career advice because I feel like everything is is shifting again. But the thing that has served me well and and again this is a big your mileage may vary sort of thing is I've I've always tried to take whatever job or whatever opportunity will get me closer to the thing that I actually want to do. So if you look at my LinkedIn you'll see I started out in as a software engineer and test at Data Stacks because that was the first position the first interview I got at a company that was doing distributed computing that would talk to me, would even let me in the front door. And I did not care. As long as someone would get me a little bit closer to distributed computing I would take that job and I applied all over the place and that was what what got me in the door and that's my my same advice I think to to almost everyone I talk to who's entering computer science is you you got to just take whatever opportunity you can that gets you one step closer. Don't be afraid of not going all the way to what you want to do in your very first job or your very next project. Just make sure that it's it's one step closer to the thing you want to do.

SPEAKER_02 40:18

I love it. I love it. Russell thank you so much always a pleasure speaking with you I always you know get out of these conversations with tons of insights and of course more questions uh uh and and learning so really appreciate you taking the time to uh jump on this call of course it's been great thanks so much for time