Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

Connecting to Apache Kafka with Neo4j

• Confluent, original creators of Apache Kafka® • Season 1 • Episode 52

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 54:29

What’s a graph? How does Cypher work? In today's episode of Streaming Audio, Tim Berglund sits down with Michael Hunger (Lead of Neo4j Labs) and David Allen (Partner Solution Architect, Neo4j) to discuss Neo4j basics and get the scoop on major features introduced in Neo4j 3.4 and 3.5. Among these are geospatial and temporal types, but there’s also more to come in 4.0: a multi-database feature, fine-grained security, and reactive drivers/Spring Data Neo4j RX. 

In addition to sharing a little bit about the history of the integration and features in relation to Apache Kafka®, they also discuss change data capture (CDC), using Neo4j to put graph operations into an event streaming application, and how GraphQL fits in with event streaming and GRANDstack. The goal is to add graph abilities to help any distributed application become more successful.

EPISODE LINKS

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

  •  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
  • ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
  • 👍 If you enjoyed this, please leave us a rating. 
  • 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
SPEAKER_03

When I say database, you probably think relational database, SQL and tables and the rest. But we all know that's not the only game in town. Neo4j is a popular graph database that's been around in helping people solve interesting graph problems for a long time. And they've recently improved their integration with Kafka. I talked to their own Michael Hunger and David Allen about how graph databases work together with streaming on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hello and welcome back to another episode of Streaming Audio. Very pleased to have in the studio today two guests from Neo4j, Michael Hunger and David Allen. Michael and David, welcome to the show.

SPEAKER_01

Thanks for having us. Thanks for having us, Tim.

SPEAKER_03

You got it. I'd love to hear a little bit about what you do. Michael, you first. What uh what's your role?

SPEAKER_01

So my role at Neo4j right now is I'm leading the Neo4j Labs effort, which is part of developer relations at Neo4j. I've been with the company actually for a long time. I started uh in 2010 to work with Neo4j. Uh so it's been almost 10 years. And it's really great to kind of see in all the time uh the product growing up like a child to an adult, growing the company being growing, and the community uh growing as well, and like having all these amazing people to work with. So and uh as part of Ne4J Labs, we build integrations of Neo4j with other technologies. We build uh like bleeding edge technology, um, stack integrations and things like that. And that's what we do, and then talk about them, show them and have people use them and good time uh using Neo4j with those.

SPEAKER_03

Awesome. And yeah, right, your path and mine have been crossing since about that time. Exactly. So we're capacity as developer relations people, even back before you called it developer relations.

SPEAKER_01

Yeah, exactly. So it's been almost also like 10 years almost, right, Tim? Yeah. So it's incredible. Time flies.

SPEAKER_03

David, uh tell us tell us about yourself.

SPEAKER_02

Um my name's David Allen. I'm I'm a partner solution architect with Neo4j, and day to day, what I what I mostly do is work with Neo4j's strategic partners and also look for spots where we can improve the product by developing uh integrations and interoperability with other systems like Kafka, and that's really what drew me into the Kafka world. Um I've been with Neo4j for about two years, and I was a user out in the community trying to build real systems and running into graph problems. That's kind of what brought me to it. It was only a couple years after that that I ended up working for the company.

SPEAKER_01

Well I tried really hard to get him in. So I'm really happy that it worked out.

SPEAKER_03

And you and you did. That's excellent. Um so um that's uh David, we may have to may have to call you Segway Man because that's an excellent Segway uh that you just gave us there. Thank you. Um because that's really what we're talking about today is uh there is a new Neo4j and Kafka connector. Before we get there though, uh I kind of think uh just in case anyone is uninitiated, I sort of think we should talk about what Neo4j is. So, and by the way, all these questions, these are just gonna be jump balls between you guys. So uh whoever wants to take them, take them. But tell us what Neo4j is.

SPEAKER_01

Yeah, I'll I'll take this one. Uh so Neo4j is a graph database, it's an open source graph database. And graph databases are databases like others, but they actually were built to handle those gazillion of joint tables that you usually have in a relational system, which kind of cause a lot of pain and a lot of confusion for people. And so graph databases evolved as a way of dealing with these complex connections between your business entities and allowing you to have uh very uh involved um queries that traverse and include a lot of these entities in a single query. And they do that by using different data structures. Within the database, there's actually a real data structure to handle relationships between entities, which is really cool. And by following these relationships, like chasing pointers in memory, they are allowing you to have uh queries running much faster that have a lot of joints. And so graph databases have been around for quite some time. As I mentioned, Neo4j, I joined Neo4j in 2010. Uh, but Neo4j started much earlier as actually a part of a real system and was then extracted as a separate project, uh, open source project and and uh product. And uh what's really cool about graphs is that's a very natural way of modeling data. So you might have seen graphs all over the place, right? Every marketing material that you look at have has bubbles that are connected by by arrows or lines.

SPEAKER_03

So that's what you've got a very trendy piece of kind of background graphic art. Exactly. I noticed the other day the default LinkedIn banner right now. It's a graph. You know, so exactly.

SPEAKER_01

It's it's kind of really cool to see that. But you also know that when you're, for instance, discussing with someone in domain problem, you oftentimes grab the whiteboard, you draw circles and arrows for the different entities that you're talking about. So it's basically you're drawing a graph on whiteboard. Exactly.

SPEAKER_03

It's it's the native, it's like everybody's native visual language is a graph.

SPEAKER_01

Yeah, and that might make it also really natural for non-developers to kind of carry on discussing the data model while it's already in the database, because the database model is exact exactly this bubbles and arrows, so entities and relationships model. And that makes it so powerful because it's really easy to take real-world concepts and connections and put them into this database. Whoever tried to do that with relational databases, I I've done that in the past and it was not such a fun experience. It's kind of putting really complex domain uh models in relational databases, you know that you end up with a lot of join tables, a lot of joints, and it's not fun and it's also not very performant. And so with a graph database, this natural model keeps ongoing, but it's also kind of faster to work with that, which is kind of really cool. Also, if you have trees or long paths or you do routing or some things like that, that makes it also quite quite natural to do that as well. And so NIFJ is actually sorry, go ahead.

SPEAKER_02

Um I I'd wanted to give an example of this is kind of the example of how I came to graphs. I had this real-world business problem. I had to come, I had to start using graphs because I couldn't get it really done any other way. Uh I was working for a company in the DC area that was serving the federal government, and we were building this provenance solution where it basically was a database of all these reports that they had and where those reports were derived from. You know, what kind of source data did they use, how trustworthy was it, that sort of thing. And so you ended up with this graph of reports where you know this report cites these other reports, and that in turn is based on something else, and very quickly that adds up to this really dense, complicated graph. And so when you're doing this in relational databases, typically what most people are gonna do is they're gonna have a table for reports and it's gonna have a foreign key on it that says, you know, who is my parent, or you know, maybe some other join table where you're gonna say, what am I based off of and then join to another table. But when the business user comes to you and says, I want to get a list of reports that are about this topic that contain data sourced from this particular federal agency. I don't care if it was sourced from them uh in the immediate preceding report or six hops ago, uh it's really, really difficult to express that in SQL because it's not only a join of two tables, but it's a recursive join and you don't know how deep you have to go. I mean, really you're talking about a breadth-first search, uh, but with SQL joins. And um I kind of went up the mountain and consulted with the SQL Wizards and they they showed me how to do that. And while it's possible to get an approximation of that, it's quite difficult with relations. When you then take that same data and you model it as a graph, the answer turns out to be two or three lines of cipher, uh, which is the query language that we use with Neo4j, and it's quite intuitive to understand and works really, really well. And so, you know, the moral of the story is that if the data in your business domain is fundamentally graphy, or if the way that you think of it is most easily expressed as a graph, uh you might be in a good space for a graph database.

SPEAKER_03

Aaron Powell Especially when you get to those kinds of things, like you said, you know, uh was is there data in this report that at any time came from uh something labeled uh according to some federal agency or you know, some some source, no matter how many hops ago. Yeah. Uh exactly. You know, another case Oh go ahead.

SPEAKER_02

Another case might be, for example, everybody knows the Kevin Bacon game. How many uh hops or how many degrees of separation are you away from Kevin Bacon if you ever were in a movie? That's a very graphy query too, where you're basically asking for a shortest path through a graph. I acted in this movie along with this other person, and then there's some path that leads back to Kevin Bacon. You know, if if folks want to know what the the value of a graph database is, I would just challenge them in their own time to try to write the Kevin Bacon score query in SQL. Right.

SPEAKER_03

Like it can be done, but um it's not pleasant, and performance relational databases are not optimized uh to do that in a performant way, period. Exactly. Yeah, so when you have those the the the deep joins or recursive joins, so just just just don't. You know, we have databases like Neo4j for that kind of thing.

SPEAKER_01

Yeah, and actually it's interesting, there's not just Neo4j, there's a family of graph databases, so lots of different databases from different uh vendors, and you have also some graph uh background, Tim, as as you know, as I know, and you know. Uh so it's it's a lot of kind of different um companies are looking at this and say, okay, we want to provide people with a better solution in this area. And um so it's been uh a long time since Nevo J has tried that, but now we are joined by lots of other companies as well who also provide databases in this space, so which is really cool. Uh, something I wanted to mention about Cypher, because why do you need another query language for graphs? Why can't you use just SQL?

SPEAKER_03

Right.

SPEAKER_01

It's actually quite interesting. So, besides what David said, is also what we try to do with Cypher is actually to take this visual diagram that you would draw on a on a whiteboard or on a piece of paper or in your diagramming app and turn this into a query language so that you basically visually can represent your entities and their connections with ASCII art. So you take round parentheses around your entity that represent like the circles on your whiteboard, and then dash dash greater than signs as directional arrows that you can express. Okay, this person acted in this movie, or this report was from this agency, or um, this uh flight departs from this airport or something like that. So it's just an ASCII art representation, which is really cool. So it's easy to write and easy to read, and especially for non-developers, also easy to read. So you have all the business folks that know more about the domain than the developer can still follow these queries and understand them and correct them and comment on them. And so that makes it very intuitive not to take uh you know this data and the domain model and throw it in a database, and suddenly all this richness of the domain is gone, but you keep it on both in the database and also in the query language, which is I find always, you know, like an epiphany going up in people's eyes when you talk to them about that. And when you say, okay, now we can take what you just throw on a whiteboard and take it in the database, you show it to them again, and they say, Oh, it looks the same like it looked before. And that's makes me always happy to see people um excited about that um opportunity.

SPEAKER_03

Yeah, it seems like the value of the graph database is well, there's there's two things. Number one, like David was underscoring, uh, there's simply performance on some kinds of problems, uh, where it's optimized to do them in a reasonably performant way. And if you try to bend a relational database or a key value store or something into doing that, it can all be done, it's just you're not gonna like it, it's not gonna run well. Um, and the other one is in the mind of the developer, and this is often where I am and where I'm trying to see the value in things, is how are we optimizing for uh you know developer cycles? And for graphy, very graphy problems that we're thinking of, and like you said, you go to the whiteboard, you draw a dang graph, you know, that's what you do, and that's what's in your mind. And now, well, that's the abstraction that the database presents to you. Uh, you can kind of see it with arrows and things in the query language, and it's almost like the edges are labeled in cipher and everything. So you just get a lower cognitive load of translating your very graphy problem into um, you know, the actual data infrastructure that's managing the graph.

SPEAKER_02

I think that's right. Um there's so many different databases out there. You know, it's funny, from my perspective, maybe it was about 10 years ago, the database market completely fractured, and you ended up, instead of everybody having the one database to rule them all, you started to have these many different families of specialized databases that were really, really good at a narrower subset of things. So that's when you started to have the document stores, the key value stores, the graph databases, the relational databases, and then relational even subcategorized out into row stores versus column stores and on and on and on. And so that entire market kind of fractured. And so for the developer, really it's a right tool for the job type of a story. Um it's not that one database is ever going to be better than all of the rest of the databases, but you know, sometimes you need a hammer and sometimes you need a saw. And I would compare the different database families to that kind of yeah, that at least at least the tools are available.

SPEAKER_03

And that's what I was saying nine years ago, you know, when I was giving NoSQL survey talks, including talks about Neo4j.

SPEAKER_01

Um Yeah, I know I found your slides on the internet while I was looking for something else, which was really cool to see. There they are.

SPEAKER_03

I probably don't want to look at those slides from nine years ago, but I'm glad you like them.

SPEAKER_01

Yeah, but uh I have to say they were already great back then. So it was uh no after us con I was looking for something and I came across your old slide and said, Oh, cool. Tim already did really good slides back then, not just now.

SPEAKER_03

So it's well thank you. Um But it's um uh it's it's just good that the toolbox has those tools in it now. Uh and it it used to be sort of one and now it's a lot.

SPEAKER_01

Yeah, what's interesting about this is of course that you have to solve the integration story, right? It's similar to the monolith to microservice problem where you move one you go away from one problem basically from the maintenance of the monolith to like maintaining smaller microservices, but then you get an orchestration problem. And it's similar, I guess, with applications and NoSQL databases that you still have the orchestration problem, right? And that's actually something where where Kafka comes in as a really good solution for you know connecting streams of data between databases or being the backbone of such a data architecture as well, right? So and that's right.

SPEAKER_02

If you look at Kafka as sort of the glue between some of those components, though, one of the things that it does is it frees you up to not have to choose the one database to rule them all and then deal with its uh shortcomings, but rather you can pick different solutions in different places and wire them all together with a common messaging bus and so on. And I think you know that's one of the things we wanted to talk about today was this idea of adding graph capabilities to a streaming architecture. It's not about making sure that Neo4j is the primary or only database in an architecture. I don't, you know, there are a lot of very large-scale applications for which you know it would be crazy to think of any one database as the database. It's about adding a set of new capabilities to a distributed application that may be strung together exactly.

SPEAKER_03

Uh once again, David Allen playing the role of Segway Man. Thank you, David. Because I and that's I I was thinking I because I clearly remember ten years ago, the the summer of NoSQL and all of the terrible and just misguided posts on Hacker News about the CAP theorem. It was like there was one every week, and you know, you're trying to read the news in the morning and these posts are assaulting you. It was kind of a dark time in that sense, but you know, overall, a really good time in that you know, now's this there's this flowering of databases, and the paradigm shift that we see, at least that you know, that I mostly focus on now, is uh this shift to streaming. And Michael mentioned microservices integration, and you've got you know that I think they're architectural mega trends, uh they have separate origins, but uh here they are meeting in the middle, and we're finding that, oh hey, we broke our programs up into a bunch of little pieces and ran them on separate computers. Uh looks like the only way for them to communicate that really works is through messaging through something like Kafka.

SPEAKER_02

And then you know, Tim, the very first time I used Kafka, I I I have to admit, like my very first reaction to it was this feels like the ESBs I was using in the 1990s, except it's way less painful.

SPEAKER_03

So we're we're always careful. I have there's this one slide I have in this one presentation, and I always get this little tinge of guilt, like, oh boy, this looks like I'm I'm I'm making the ESB pitch from 20 years ago. And we're absolutely not, you know, there are some very important architectural differences that make Kafka not a dang USB. Um and and you know, there's much less XML, which is kind of you know, just sort of uh uh incidental thing, but there isn't routing that goes inside it, right? There's topics, and topics are just logs, they're just there. If you want to route, you write a program for that. You know, that's that's your application that does that. And that rather than locking that routing and transformation and more and more sophisticated rules that some specialist is defining with big complex XML files, um, you know, rather than locking that inside a temperamental team that doesn't want to help you, um it's always in the hands of the developer in Kafka, right? You write software and your software uh logs messages back and forth and you know, then you win.

SPEAKER_02

Yeah. Well, to you to your point about the architectural differences, uh, you know, maybe 15 years ago people were mostly passing big heavyweight messages between gigantic stateful monoliths, and uh the idea of you know message processing functions being these tiny little pieces of stateless logic that map one stream onto another, I think is a a pretty major development. And then that itself is only one component of how you build a larger application, which kind of even further moves this kind of an architecture away from what people were doing with the ESBs before, and you you start to start to think of it more as an enabler for a fundamentally new way of writing applications and less of, oh hey, we're moving messages across the wire.

SPEAKER_01

Yeah, it's a lot of like CQS architecture support as well, right? So where you have kind of the separation of where's the data stored and uh where's it actually used, and um so kind of the separation, clean separation of responsibilities as well, and having a reliable system that underpins that helps a lot in making making those decisions so you don't have to reinvent the w as a developer to build it all the stuff yourself. No, exactly.

SPEAKER_03

I couldn't have said it better myself. So with that, cool, let's let's talk about that. And Michael, you mentioned CQRS, uh, which is for the uninitiated, you can Google it. Uh it stands for command query responsibility segregation. And that's the idea where uh an application takes inputs or commands, you know, hey, go do this thing, and then later on you want to ask that application, what's the state of the world? What what's the result of those commands that I issued in the past? What happened and how what are things? Um and the way that you process the inputs is probably different than the way you process those requests. Like inputs could be whatever they are, but what if you have some very graph-shaped query problem? Well, you want to get that data that's coursing through these Kafka topics, you want to get that data into a Neo4j database, for example, in that case.

SPEAKER_00

Yeah.

SPEAKER_03

Which gets us to the whole reason that we wanted to talk today, which is that you have a new Kafka Connect connector. That's true. So um how did that connector come to be?

SPEAKER_01

That's a really uh really cool point. Actually, uh, what I want to mention initially is that it's not been even 12 months since we started with the first line of code of the connector. And now we're already talking to you on the uh streaming audio podcast, which is really, really cool. So there's been a lot of really quick development on all sides of uh kind of working on the connector code, partnering with Conflict, all our users and customers that have been using that and giving us feedback. So it's been a really energizing and cool place to be. So actually, it's it started with um, you know, many Neo4j customers and users have already been using Kafka to uh connect Neo4j to other databases or other data sources or things, right? So getting data out of Ne4J into other places or the other way around and ingesting data from as we are using Kafka into Neo4j. But we got kind of this more and more uh frequent requests isn't there something out of the box that we could use uh with Neo4j so we don't have to reinvent the real uh for the end time. And so we um looked at uh some of the existing uh implementations, some of the offerings. There was also a proof-of-concept connector on on um um on connect on a conflict hub, and we decided to to partner with one of our um technology partners to build something uh from scratch that actually serves uh all these these needs of the users. So it was uh meant to be um both as an um Neo4j extension, so that means you would install it into Neo4j, uh, but also as a Kafka Connect plugin. So depending on your deployment needs, you could uh choose either or uh to run this. And we implemented it in Kotlin because I really love Kotlin and it's a really great language. Uh so I that was also one thing.

SPEAKER_03

You implemented it in Kotlin because you're cool.

SPEAKER_01

No, but I I love Kotlin, so I've been using Kotlin for since 2014 or something like that.

SPEAKER_03

And uh Oh wow, oh you've been at it since the beginning.

SPEAKER_01

Yeah, it was really, really early, and I I just like this language, and so I said if you want to build something new, I want to build it in Kotlin. Um, so um the uh As I said, the needs were different, right? So one of the needs is, of course, uh capturing changes in Neo4j and exposing them to other systems, be it for auditing reasons, be it for uh aggregating changes lower, uh, and uh visualizing them, for instance, or having other databases or other systems being updated from changes from Neo4j. So that is the whole CDC topic as change data capture. And I had really good conversations with Gunnar Morling from uh Debisium, whom you also had on the on the show, I think, right? So and Gunnar and I had really good discussions about how Debisium approaches things. And um originally we wanted to build something directly in Debisium, but unfortunately Neo4j does not have an external API yet to uh capture or extract uh CDZ information directly. So we had to build this uh Neo4j extension, which we then kind of built in a way that it's compatible to the Debisium APIs and message formats. Uh, but it would you have to install it in Neo4j. So that's something that's planned for the future, that we have an kind of database independent uh Kafka connector, and then the question is where we put it. Um so you can capture change events uh from Neo4j into uh into arbitrary Kafka topics. So you can basically in your configuration describe I want to have these entities or this relationship uh with these labels or or names uh captured on change and export these properties of those uh entities into Kafka messages.

SPEAKER_03

And to be clear, this this is a feature of the current version.

SPEAKER_01

Exactly. That's a feature of the current version. So I'm talking a little bit about the features of the current version. Yeah. So the um the other feature uh that we implemented is an is a sync. So that means events from Kafka can be ingested into Neo4j. And the sync has many or three different ways of operating. So one is where it can actually ingest these CDC events uh from another source that has uh like the DBZM format or a Neo4j source. Um, one-to-one where you don't have to do anything about it, just say this is I want to ingest uh from this topic, and it takes care of that. Another mode is where you basically say I want to turn events from Kafka one-to-one into nodes in Neo4j. So it's just a one-to-one ingestion of uh event structures. But the third one is one that I'm really personally happy and proud about, which is you know, usually uh a graph structure is not a trivial data structure. So it's not just like one table or one document or something like that, but it could be something where one message actually triggers an update in this place in the graph, then adding a relationship in this other place, removing a relationship in a third place, and adding a new property to another node. So it's like you might have with one single message, you might have to change things in seven different places in your graph to represent the essence of that message. And so we kind of discussed back and forth on how we want to kind of represent this mapping. And I said actually, we have a really powerful query language which can already do all of that. Uh so we allow people to map an Kafka topic to a cipher statement. Uh so the events that come from Kafka are batched into uh regular size batches, so 5,000 to 50,000 uh batches or so. And then it actually executes that cipher statement. And so your single Kafka event can be turned into an arbitrary number of changes in the graph. It can create new subgraphs, it can update parts in the graph, it can remove parts of the graph depending on what you want to want to do. And that's a really powerful mapping where you take a single document model, basically, what an event is, and turn it into a much more um advanced data structure.

SPEAKER_03

That is super cool. I did not know the connector did that. So just to read just to cap there, it's it's proper to say that it does full new4j C D C so I can use it as a source connector.

SPEAKER_00

Yeah.

SPEAKER_03

And there's the trivial sync connector, which is I have a message, go make it a node. Um and we don't know what happens with edges in that case. It's just here's a new node.

SPEAKER_00

Then it just creates nodes, yeah. So it basically turns it directly one-to-one into another.

SPEAKER_03

But the thing that you probably want is run the cipher statement taking the event as input and mutating the destination graph, the sync graph, however the cipher statement says. Super cool.

SPEAKER_02

So I I a thing we've got coming in the connector too. This isn't real today, but it's actually in PR stage right now. It's getting there. Um so Michael gave you two options. Uh to use the cipher statement or to just put a regular node in there. Um there's a there's going to be a third option. Now, if you were to write a big complex cipher statement, it would almost be like you're doing the message transformation yourself in cipher, if that makes sense. If you were to just create a node, it's just a flat node, and then maybe you do something else with it later, maybe not. Um the third option is to use something like KSEQL to do the transformation for you, and then to have KSQL output a separate stream of messages that are basically like create node and create relationship commands. Okay, so imagine I took a big JSON object and I destructured it and I and I know that the implication of this JSON object is that I need to create three nodes and a relationship between them. Well, you could use KSQL to do that transform and inject onto a different topic a really simple format that the connector is in the future going to support that basically says create a node like this, create a relationship between these two nodes and so on. And that way you can delegate the transformation out of the cipher layer to KSQL if that's what you prefer.

SPEAKER_01

Yeah, that's that's a really good point. Thank you, David.

SPEAKER_03

So uh ladies and gentlemen, uh this is the voice of a gentleman who understands how to use KSQL. So David, yeah, that was great.

SPEAKER_02

You know, I've been I have had the privilege to um have some early access to the KSQL preview on Confluent Cloud, and I've been really digging it.

SPEAKER_03

Nice, nice. So yeah, I mean there's a lot of things you can do with KSQL, it's not just this, but when you're in this problem of I have you know maybe several event streams or several topics, and what I really want is this row to get inserted into this database or this node to get created in a graph. Um well, you know, you can turn those several things into one thing. That's kind of what KSQL does. You can join and transform in there, and so that is absolutely brilliant. Uh just have that query running, and then you you run, you configure the connector to run on the topic that's the output of the KSQL query.

SPEAKER_02

Well, right. And if if you have an abstract JSON format that is essentially like a command to create, update, or delete a node or edge, the the interesting possibility for me with that is that because Neo4j is schema flexible, you can change your data model from the outside by modifying how you do that transformation without touching the connector config, if that makes sense. Right.

SPEAKER_03

Right. Um now the just a technical question to satisfy my curiosity. In the case when you are doing this with Cypher, um and my guess is a lot of people do that because it seems super flexible. Yes, that the only place in the system computationally that knows how to do that is the Neo4j server. So that's uh like a rest call back to Neo4j from Connect in that case, right?

SPEAKER_02

Well, in in this particular case, so Neo4j has a binary protocol called Bolt, and you can think of that as serving the same purpose as JDBC does for relational databases. Got it. So effectively you issue cipher commands over the bolt protocol, and when you use the connect worker, uh it's making uh a bolt connection to the server and um basically turning those events coming from Kafka into a stream of cipher queries. Or at least that's one way to use it. Right.

SPEAKER_03

And that's any just like any just like you're using a database.

SPEAKER_02

Well, in fact you are. So exactly like using a database.

SPEAKER_01

And what's also cool there is uh it also takes care of batching there for you. So it does not send actually one cipher query per event, but it can also batch them together and send just a single query and a and a pack of uh events batched together as a parameter. And then it takes care of um you know applying them all in one go so you have much more uh uh throughput uh on that on that side as well. Yeah, and that's kind of exactly what's kind of running in the in uh in Kafka Connect is uh Ne4j uh the Neo4j Sync plugin, uh which you then can connect to any arbitrary Neo4j uh instance or cluster, and it will take care of things like routing and retries and and all the other things that you get from from the uh bold protocol out of the box. Uh so that's something you don't have to worry about um at all because it's just like a client to the database.

SPEAKER_03

Is that uh well, let me just ask. Uh what was the hardest thing about the connector? I know you guys weren't the original author, but you had your hands in it. What's hard about doing this? My guess is this cipher integration, but uh yeah, actually that was not so hard.

SPEAKER_01

Uh I think uh there are three things to come to mind for me uh that were not so trivial. And uh so one is um like the configuration, especially around batching, uh, was not so trivial because you know have you have batching on the on the Kafka side on in terms of like um batch sizes and and um message sizes and things like that. So kind of how to control how big a batch is is not so obvious, always that you get it in one go. Uh the second thing was um error handling and monitoring. So it seems that uh this is a little bit of a challenge where you have to decide how do I do that, right? So do I just lock errors and proceed? Do I use a dead letter queue? Um how do I do it in Connect, for instance, and and things like that? Do I throw errors or not? Um so that was something that um is uh a little bit challenging. And the third one is SNEA4J is a schema free database, uh, kind of integrating with schema registry is something that's also something that is not so trivial because it's similar to when you have, let's see, CDC from something like Mongo, where you also don't have a schema. So you basically have to also send along the schema with each CDC message, for instance, right? So which is then harder to do with a schema registry because it you can't really guarantee that like the schema is uh the same for one node versus the other, because every node and every relationship in a graph can have arbitrary number of different properties depending on how much data you have or how much information you have available for for that entity. And so that's uh that's the uh third uh bit that was a little bit or is a little bit challenging. So we're still working on making that.

SPEAKER_02

I've got two more that I that I might have. So many hard things. Well, right. Uh when you're moving.

SPEAKER_01

Databases are hard, right?

SPEAKER_02

So databases are very, very hard. I mean, which is why so many people in the architecture world are trying to move to stateless components right now, because you get rid of a lot of the headache if you get rid of the state. Right. Um what are databases about? They're fundamentally about managing and tracking the state. So it's uh a little bit hard to make those stateless. Um uh one thing that is is tricky is balancing uh throughput and sequencing. So for example, let's say that I read a hundred messages off of Kafka in a batch. Um now the database tends to be optimized for doing lots of transactions in parallel. So I would definitely maximize throughput if I would do lots and lots of those batches, and at any given time I'd be executing 20, 30, 50, or more transactions against Neo4j, Neo4j wouldn't even break a sweat and you'd have great throughput. Um on the downside, if we're using this for a strict data replication CDC type use case, the order of evaluation matters quite a lot. And so uh, you know, if these commands that are creating, updating, and deleting things to you know mirror Neo4j along with some other source are executed out of order because the transactions succeeded uh in a different order, you could be in trouble. So uh, you know, obviously we can make it strictly ordered, but then we might have to compromise throughput, like how to balance those kinds of factors is one. And then the last is maybe some differences in model semantics. So for example, in Neo4J, when you create a node, let's say that you hang five relationships off of it that go to other nodes. And then let's say somebody sends a tombstone record or says, I want to delete that node. Okay, in Neo4j, if you say I want to delete that node, by default, this operation would fail because it would say relationships have to be connected to something. You can't just delete the node. You need to delete the node and also detach it from its relationships, which will delete those relationships as well, because we can't have dangling relationships that point into the void. No floor for the That's right. So when a person says to delete a node, there's uh this little question of differing semantics. The user probably means to detach the node, delete both the node and the relationships, and yet it might be uh surprising to them that when they say delete a node, they're also deleting 15 relationships. Right. Right. Because keeping in mind that the relationships can be separate property holders and might have any number of other pieces of metadata on them. And so thinking through some of those things and making the most common use match what the user expects, I think is a tricky part because uh of the difference of how graph data model semantics work.

SPEAKER_01

Aaron Powell That's also a good point in general. I think uh that we only started so far, but we need to do more of is uh to talk to more different users and use cases and see which of those have we already captured and which and what we offer and which ones have we not captured as well yet. Right. So some of those come up with where people put up issues and say, okay, this didn't work as we expected it to work. But sometimes we need to reach out more and see did we capture like the majority of the common uh use cases that people want to do between Kafka and A4J? Yeah. So that's something that uh is uh an ongoing process for us.

SPEAKER_02

Um so I I should say generally about the connector, one of the as a as a person who works with our strategic partners and customer base, a reason that we wanted to have this connector in the first place is that we see Kafka being used in our customer base quite a lot. And so when we think about how we evolve the connector over time, a lot of that um, or at least part of it, is going to be driven by community open source requests and also what the commercial customer base says, you know, what do you need? How are you using this? And what then does the connector need to do in order to best support uh getting value out of this?

SPEAKER_03

Right. So all of that discussion um was fantastic. I mean, it started off uh, you know, Michael, you said error handling and batching and things like that, which is like software adulthood, you know, actually making it a thing that is mature. Um and then all the way to the discussion of the different model semantics is such a great illustration of why Kafka Connect matters. Now, it's fairly rare, but every once in a while, when I'm talking in public about Connect, it'll usually be like a part of a introduction to the platform sort of talk, you know. Every once in a while I'll get a question like, Well, couldn't I just write that myself? And you know, in in my inner voice is the answer is, Oh, child. Yes, yes, you could. And maybe you should. You know what? I think maybe you should just once, because you're not you're clearly not gonna listen to me. But you know, we all are I think are formed by by hardship, and maybe you need some in your life. You haven't had enough. But so this is all just a super good example of the fact that it's it's conceptually trivial. Thing happened in graph database, put thing in topic, you know, message is in topic, yeah, put message into graph database. And it's even a little weird putting it into a graph database. Okay, transform it through a cipher query, I get it. But the reality is um it is kind of a grind. Uh it's it's conceptually very easy, but uh connect exists because this is a non-trivial problem. And and Michael, you said it's only been being worked on for a year and we're already on on the podcast, you know. Well, yeah, it's it takes that long to do this right. It's it's actually hard. So uh I just it's a good thing to underscore. It it definitely is.

SPEAKER_02

I I think every piece of software has a long tail that when you're looking at it from a high-level conceptual point of view, you always miss. And so you know, it's it's valid to describe a connector as, well, okay, you take a message off of a topic and you put it into a database, and from that conceptual perspective, everything looks dead simple. Um but uh uh you know, you have some 20% of features that get 80% of the use, and then you have this really, really long tail of very specific yet still critical needs, and that's where that grind comes from. And you know, part partly, you know, Tim, as you're as you were saying that, the reason I'm laughing is because I'm just reflecting on the the difference between old age and experience and youth and vigor. Right. Um I I've been guilty of that many times where where I think to myself, wow, that that looks really simple. Why don't I just knock this out in an afternoon or in a weekend or something? And yeah, you kind of sorta could do that, but uh it's it's it's kind of like the month after that when the pain would start.

SPEAKER_03

That's it. You could knock out something in a weekend. It's just nobody's gonna use it. Yes. Because now it needs six months of edge cases.

SPEAKER_02

Oh, even worse, somebody might start to use it and then they would look to you for support, and then you would, through a long, iterative, painful process, discover why it wasn't actually that easy. I think uh Linus Torvalds did that with operating systems once upon a time. Thirty years later. That's right. I mean, look at that guy. I mean, he really was that smart and kind of did knock out an operating system in a relatively compressed period of time. Thirty years later, he's still on that mailing list.

SPEAKER_01

I guess it's uh it's actually a good uh um illustration of two aspects. So one is of course you can knock it out over the weekend, but then um if you don't have users, you don't see all the long tail basically. So that's where open source comes in, and it's really cool to get people using it and reporting back or even contributing back, fixing issues. And so that's one aspect. But then, of course, um you also have like people who want to put this in production and want to have support for it. So there's uh where the partnerships and the you know providing this higher level of um guarantees uh for something like that. So and having both in in such a project is really helpful, right? So you can do on one side you can iterate quickly, you have open source, you get all your contributions and lots of feedback. And on the other side, you can provide uh a sound solution uh through the partnership and through the efforts on both sides uh that customers can and users can rely on. And I think that's a really good combination of both.

SPEAKER_03

Awesome. How tell me about GraphQL. Now I know Cypher is uh the native language of um of Neo4j, and it's a it's a you know it's kind of a an invention of Neo4j. It's grown up out of Neo4j. So where does GraphQL fit into all this?

SPEAKER_02

Uh Michael, you want me to take it?

SPEAKER_01

I can shot it. Yeah, I started and then you finished it. Okay. Uh so GraphQL is actually not a Graph database query language, it's an API uh language. So it's a language to express uh APIs as a domain schema to say so. So you have entities in a relation in a in a in a schema, which is pretty strict actually in the GraphQL uh case. It's out of Facebook, so they open sourced it in uh 2015. Uh sorry, in 20 yeah, 2020 something. And um the uh but what's really cool about GraphQL is it lends itself really naturally and very closely to the graph or domain model in general, right? So to the object model or graph model. So what we did for Ne4J and GraphQL was uh basically we uh built an integration that allows you then to uh take the data that's in the Graph and expose it as a GraphQL API and put your own uh business code as well in into that, and we call this whole thing GrandStack. And so you can actually um combine all three of those and say, okay, I have a stream of events coming in. Um they're getting aggregated into a domain model in NeoVerj, and then the GraphQL API in front of that can power your application uh or your mobile application uh as such.

SPEAKER_02

So there are a lot of different uh plugins out there for different databases that will let you expose any database as uh you know GraphQL layer, uh, because in GraphQL you have this separation between the API that you're presenting and a layer that they call the resolver, which is how you're providing the data implied by that graphic that graphy kind of query. Um and so if you were to put GraphQL on top of a relational database, typically behind the scenes, the resolver would be doing a lot of joins in order to uh traverse the graph and give the user what they want. I think using Neo4j and GraphQL together is particularly powerful because you're you're querying a graph model, and then what Neo4j is storing under the hood is literally a one-to-one mapping with what you're querying. And so if you've ever heard of like you know, impedance mismatches in systems, there kind of isn't one there, and that makes the API really performant and quite easy to use. Um with the can you know, kind of tying this back to Kafka and the connector too. So if you if you zoom out and think architecturally about this, if Kafka lets you do streaming applications and using the uh Neo4j connector, you can get any stream into a graph, and then using Grand. Stack, you can put a GraphQL API on top of any graph. Essentially, this GrandStack stuff that we're talking about, which you can find more about at uh Grandstack.io. You can use these pieces together and put a GraphQL API on anything moving in or through Kafka. So that's that's pretty cool and pretty useful. We have a lot of customers who are using that to build front-end applications with React on top of Neo4j databases to do just any kind of thing that you would want to do with a web application.

SPEAKER_03

Fantastic. And Grand Stack for the uninitiated is GraphQL, React, Apollo, and Neo4j database. Otherwise it would be GRAN.

SPEAKER_01

Yeah, you got it. Well done.

SPEAKER_03

That's right.

SPEAKER_01

Yeah. One other thing that is really uh cool about kind of combining Kafka with Neo4j is also that you can run something on your data that you usually wouldn't run on a transactional database, which is Graph algorithms. So which kind of brings us in the whole area of machine learning analytics and and so on. Basically, it uses the uh topological structure of your data to get your new insights. So just by looking who's connected to whom, not just by one hop, but by n hops. So remember page rank for Google Search, for instance, uh, allows you to get new insights from your data. And so you can take your Kafka stream, or ingest a certain portion of that into, or number of streams, ingest a certain portion of that into a graph, then run graph algorithms and have the results of these graph algorithms either stream back again uh to Kafka to consume some else, or enrich your actual graph to uh serve other kind of um use cases as well. So you can do lots of really cool uh stuff putting all these kind of building blocks. It's almost like Lego um putting that all together.

SPEAKER_02

Michael, can I can I tell a story about the NBC thing to kind of make that concrete? Yep. Um so uh by far one of the most fun things I ever got to do at Neo4j since I've worked here was part of what this data journalism program that we have, where we sometimes work with journalists who have data sets, we get insights out of them, and then they write articles about it. Some journalists from NBC approached us a while ago. Uh they had this dump of Twitter data that was evidence of Russian manipulation of the 2016 election. And so Twitter being a social network is a really great graphy data set because it's just a bunch of tweets and who's talking to who and uh you know who follows who and so on and so forth. And so we took all of these tweets and then put them into a Neo4j database and then used a set of graph algorithm techniques called community detection. And effectively we were looking at who is replying to who and who is retweeting who. And you can think of this graph as being densely connected in areas where there are a bunch of friends who hang together and sparsely connected between areas that are not talking or that are not connected to one another. And so by using these graph algos, we can actually discover uh sort of subclusters of Russian trolls and then talk about what topics they were pushing and how that might have had an effect on the election on the basis of how they clustered into these communities. Like maybe some were pushing this particular piece of divisive rhetoric while others were focusing on something else. Um using Kafka, you can imagine setting up an architecture where tweets are streaming in real time through Kafka and into Neo4j, and then you're running these types of graph algorithms on an ongoing basis that's providing you insight about how an entire social community is communicating. Um it's not about the individual tweets or the individual users, it's about the patterns of interactions. And so that's where I would sort of say sometimes you're going to use a graph database is when the relationships between data items are more important than the data items themselves. And um we have this entire package of different graph algorithms, all the way from the really standard ones like PageRank that have, you know, were were created by Google way back when, um, all the way up to some more exploratory and new ones. And those are just functions that you can call inside of the database. Once you then use those graph algorithms, they produce extra pieces of metadata or extra properties, if you will, which you can then re-inject into a Kafka topic. So with adding that little loop to an architecture, you can, I mean, effectively the bottom line here is that you can add graph superpowers to an existing application.

SPEAKER_03

Easily. And if if into a streaming application, if you've got your data in Kafka where it belongs, it's not straightforward to get it into this graph engine. And it's really cool to see how those algorithms have kind of uh been pulled up into the cipher stack, right? And and that's the things I can do now, you know, because I I know you used to have to write that in cipher, but why would you write that? You know, what what what's the value in you writing that when you could just make it a function call?

SPEAKER_01

Exactly.

SPEAKER_03

That's that's right. How about um you guys have a Kafka Summit talk uh coming up? Kafka Summit is coming up soon.

SPEAKER_02

Well, see Tim, now your segue guy. Um the the thing is, yes, we do have a uh Kafka Summit talk coming up. Uh I think that's September 30th and October 1st in San Francisco. Is that right, Tim?

SPEAKER_03

Nailed it.

SPEAKER_02

That's right.

SPEAKER_03

Well, um in the Hilton uh Hilton Union Square in San Francisco and listen through the end of the show for a discount code. Anyway, go on.

SPEAKER_02

Uh I will be there um co-presenting with my colleague Will Lyon. And so as we've been talking about this Grand Stack stuff, Will Lyon is the mastermind behind the grand stack stuff and is part of our developer relations group and an all-around awesome guy. Yep. We're the topic of our talk is extending the stream table duality into a trinity with graphs. And some of the topics that we've been talking about on this podcast are going to be covered there in terms of how to add graph analytics into an application, but we're gonna go deeper and there's going to be some technical configuration gore, and we're going to pair it with a real-world use case from a customer that's a large financial institution.

SPEAKER_03

That sounds amazing. And uh there's four tracks, so everybody has choices to make, and I don't recall off the top of my head what you're uh up against in that slot, but that really sounds like a great talk. So as a reminder, everybody, just listen a few more minutes, you'll get a discount code.

SPEAKER_02

Oh, um also we have uh Neo4j is going to have a small booth at Kafka Summit, and we have some colleagues who I think the world of who are showing up just to geek out about graphs and uh streams. And so if for some reason you can't make the talk, but you're still at the event, come say hello, get a sticker, and uh um look for some of our folks who can talk about the the details.

SPEAKER_03

Yep. And folks must be present to win. So please be there.

SPEAKER_02

We we have this uh you if you guys haven't met Corrine Wallach, you may meet her at this event. She's amazing. At the last conference that I went to with her, she she systematically located every single David at the conference and had a group photo of all Davids at the conference at our booth on the last day. That is I was one of the Davids. I mean David's okay.

SPEAKER_03

If we do that again, I'm also gonna need a photo of all of the Karen's. Uh so that's the thing that we're gonna need to do. So um that's great. Fair enough. Yeah. I'll I'll I'll I'll watch the David photo. Obviously, I can't be in it.

SPEAKER_02

Well, that you know, she she got inspired in the moment. I don't know if we're gonna do David's. I think we could do Karen's or Kareens, that would be a great idea for this one.

SPEAKER_03

So yeah, Kareen um is it's it's I know it's spelled Karen, I know it's pronounced Kareen, I know her. Um, but I she would have to, I think, broaden that to the Karen. That's true.

SPEAKER_02

There might not be enough Kareens around.

SPEAKER_03

There's not gonna be. There's not gonna be a lot. It's only like gonna be like a couple thousand people, and you might have a couple of Kareens in that crowd, but there should be a substantial number of Karen's and Kareen, if you don't mind, since I know you get called Karen all the time anyway, um, if you could please just broaden that and do the David picture and do the Karen picture, tweet them and at me at Teal Berglund in that tweet, and I think everybody will be okay.

SPEAKER_02

Yeah. Well, she's fantastic at creating new relationships between nodes at a conference, and I expect she's gonna be there doing that.

SPEAKER_03

I have been a personal beneficiary of that very activity uh by that very person. She really is fantastic. That's true. Well, my guests today have been uh Michael Hunger and David Allen of Neo4j. Michael and David, thanks for being a part of Streaming Audio. Thank you, Tim.

SPEAKER_01

Thank you, Tim. And uh just a final uh note uh we would love you to try out our connector to Neo4j and give us feedback, and uh having that would be really valuable to us. Um you find it under um Neo4jcom slash labs slash Kafka, and there's everything that you need to know about a connector.

SPEAKER_03

Hey, you know what you get for listening to the end? A Kafka Summit discount code. Kafka Summit is coming up on September 30th and October 1st in downtown San Francisco, and you can get 30% off if you go to Kafka-summit.org and use the discount code AUDIONETI during checkout. Just enter Audio19 while registering at Kafka-summit.org, and that 30% off is all yours. I'd love to see you there. But hey, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can always reach out to me at TL Burgland on Twitter. That's T-L-B-E-R-G-L-U-N-D. Or you can leave a comment on a YouTube video or reach out in our community Slack. There's a Slack signup link in the show notes if you want to register there. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks for your support, and we'll see you next time. Thanks for your support, and we'll see you next time.

Head Shot

Tim Berglund

Host
Head Shot

Adi Polak

Co-host
Head Shot

Viktor Gamov

Co-host