Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov

Securing the Cloud with VPC Peering ft. Daniel LaMotte

Confluent, original creators of Apache Kafka® Season 1 Episode 67

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 31:56

Everything is moving to the cloud, which makes it increasingly important to secure your cloud infrastructure and minimize the threat of potential attackers. With a virtual private cloud (VPC)—your own private network in the cloud that you can launch your own instances into—this can be done with VPC Peering, connecting VPCs together to create a path between them to keep your data safe and accessible to you alone. Although typically performed in a single cloud provider, it is possible to do in more than one—think of it as your cloud router

Daniel LaMotte (Site Reliability Engineer, Confluent) walks through the details of cloud networking and VPC peering: what it is, what it does, and how to launch a VPC in the cloud, plus the difference between AWS PrivateLink and AWS Transit Gateway, CIDR, and its accessibility across cloud providers.  

EPISODE LINKS

SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites 
Artwork by Phil Vo 

  •  🎧 Subscribe to Confluent Developer wherever you listen to podcasts. 
  • ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
  • 👍 If you enjoyed this, please leave us a rating. 
  • 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
SPEAKER_00

Running your infrastructure in the cloud frees you from any number of operational concerns. That's the whole point of running in the cloud to begin with. But just like they say that the cloud is somebody else's computer, it's also somebody else's computer network. And all networking concerns don't just suddenly evaporate when you're in the cloud. Even if they do get a lot easier, you still need to know some things. Daniel Lamont is a site reliability engineer with Confluent Cloud, and he wants to teach us some of those things. He joins us in the studio today to talk about VPC Peering, or really cloud networking 101. Check it out on today's episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. Hey. And you know what? I didn't ask you beforehand. Am I saying your name correctly? Lamotte is that pretty soft?

SPEAKER_01

Yeah, that's that's it. That's how you say it.

SPEAKER_00

Okay, good. All right, good. Uh just occurred to me I could be getting that wrong, and thousands of people are hearing it, and thousands of people should hear me eat my hat if that's the case, but I'm glad I don't have to anyway. Dan, what do you uh what do you do? What is your what is your role here?

SPEAKER_01

Um I'm a site reliability engineer on Confluent Cloud, uh working on a lot of the networking topology changes that we've been making to it.

SPEAKER_00

So cool. Um so as an SRE, are you mostly concerned with um monitoring and measuring? Uh are you concerned with developing new code that that helps drive these things or configuration of network infrastructure? Like where where does your attention go there?

SPEAKER_01

It's a little bit of all of that. Uh we do a lot of monitoring, uh, but my specific focus has been on the network connectivity with the clouds. So we're more of an infrastructure SRE team, and we focus on how we're configuring all of our infrastructure for connectivity to Kafka. So yeah.

SPEAKER_00

Yeah, no, that that makes sense. So we don't have um this is a podcast, and so we don't have diagrams, and uh, even so I don't have a diagram handy of the architecture of Confluent Cloud, but um I think you know we've we've said publicly at least this much. Imagine there are some Kubernetes clusters in the various public clouds, in this case Amazon, Azure, and uh Google. There are these Kubernetes clusters out there, and then there is uh a set of uh infrastructure separate from those in uh another one of the public one of the public cloud providers that actually like runs the code that is Confluent Cloud, you know, that that does the monitoring, and when you're looking at the UI, that is all some application that's that's kind of orchestrating what's going on in those Kubernetes clusters. All of these things need to talk to each other, and everybody's applications, which are not Confluent Cloud, but they're people's applications running in Kafka, need to talk to these Kafka clusters. And that connectivity is your work, is that right? Correct.

SPEAKER_01

Yes.

SPEAKER_00

So specifically, the thing I wanted to have you on the show to talk about is uh VPC peering. And this is interesting, not just because it's a conflict cloud feature, but I think to dig into it uh and and talk about it thoroughly, we get to learn a little bit of network engineering and and just some elements of uh uh frankly how IP networks work that uh most of us who aren't connectivity SREs almost never have to touch, right? If you've never, if you're just if you're a software developer living out a peaceful life, uh you know, writing business applications in Java like you should, and I honestly I'm not saying that as a joke. I think like that's a really good thing to do. You're never gonna come across this, but you know the word, it's out there, and so I wanted the opportunity to just dig into it a little bit. So the first question for you is what's a VPC?

SPEAKER_01

So a VPC stands for virtual private cloud, I believe. Uh they're called different things in some clouds, like in Azure it's a VNet, but effectively it's your own private network in the cloud that you launch your instances into. And that's pretty much it.

SPEAKER_00

Uh there you go. So it it acts like it's like all your infrastructure in one secure environment.

SPEAKER_01

Correct.

SPEAKER_00

Is there any relationship uh on the back end or under the covers between a VPC and a VPN?

SPEAKER_01

So a VPN is just a form of connectivity into a VPC if you so choose to connect it that way. So in much the same way that we're about to talk about peering, you could use a VPN as well. Um but typically you only use VPNs when you're connecting from either another cloud, uh, another region in another cloud, or from like an on-premise data center to your VPC. So gotcha.

SPEAKER_00

Or from one on-premise data center to another, or it's yeah. It am I right in when I say VPN, um it that is beginning to feel a little bit old-fashioned, like that's sort of a pre-cloud thing that doesn't happen much anymore, or am I just disconnected enough from network engineering that I'm off basis?

SPEAKER_01

Maybe, maybe not. I I'm not like if if you're a hundred percent in the cloud, yeah, it probably nobody uses it. But if if you do have on-prem data centers, it's common. I've I've used them in my time. So cool.

SPEAKER_00

So um a VPC is a virtual private cloud, and this is a way of making cloud resources look like they're uh that that like I have them exclusively, and I can talk securely to them, and nothing else can see them except me, and they can't see other things except what I allow. Is that kind of right?

SPEAKER_01

Yeah.

SPEAKER_00

Okay. Which it's funny, the VPN analogy just occurred to me. Uh it's, I guess, analogically similar because a VPN is a way of saying, well, in fact, uh there is this massive public network where you know lots of people could see lots of things, but we're gonna make a connection look like it's a cable in between me and you, and nobody else can see it. Yeah. Uh and this is gonna make it look like, okay, now this is a cage in a data center that is only networked to me and not to anybody else. That's sort of what a VPC is. You think of it like virtually fencing off some set of physical infrastructure inside the cloud data center and saying this is only me. Exactly. So um probably if you're listening and you don't know networking well, and by the way, this is one of those episodes, everybody, where um I probably stand to learn things. I don't I don't know network engineering terribly well. So um, you know, some of my questions will be uh contrived to make sure everything's clear, and some of them will be asked out of rank ignorance, and I'm not gonna tell you which is which. But uh tell us, Dan, what is what is VPC peering?

SPEAKER_01

So um VPC peering is simply kind of connecting two VPCs together so that there's a path between them privately that is not over the public internet.

SPEAKER_00

Okay. So it's not over the public internet. It's uh uh so a VP VPC peering would be within a single cloud provider, or can I go between cloud providers?

SPEAKER_01

It's typically just within a single cloud provider. Uh there's no good connections between cloud providers. That's just the public internet. Yeah, public internet or VPN, like you were saying earlier. But typically what we see is we try to be co-located in the same region with the customers. And so they'll have a VPC where we're at, and they just need to peer to that in the same region. So Got it.

SPEAKER_00

And VPCs are, well, like any other cloud resource, they're region-specific things. Correct. Gotcha. And I'm assuming you just said you try to be in the same region. Um that is to say, what you mean is Confluent Cloud tries to have Kafka cluster resources in uh many regions so that you have low latency access to your Kafka cluster from whatever it is, whatever your application is.

SPEAKER_01

Yes, exactly. Got it. Okay.

SPEAKER_00

Okay, so making sure I'm getting the pieces on the table properly here. VPC peering is a way of creating a connection between two VPCs. The whole idea of a VPC is that otherwise nothing can see it. But peering is a way of securely and intentionally poking a hole in that, saying, no, this one virtual private cloud is going to talk to this other virtual private cloud, and they will see each other and be able to route securely between them like uh they were in the same VPC. Exactly. Got it. Okay. And how do you normally do that? So pretend there's no confluent cloud. Like what uh walk through that with one or two cloud providers. What is it that you do to make that happen?

SPEAKER_01

Sure. It it actually differs quite a bit between the three. Yeah. We'll go through we'll go through AWS. That's simple enough. So depending, usually I guess a simple case is you have two VPCs within your own account and you want to connect them. So maybe one VPC is for your finance department and another VPC is for engineering. Um you want a connection between them. And so you'd have these VPCs there, and you simply need to create a peering connection between them. Um and since they're both in the same account, you can actually request it and accept the peering all within the same account. Once you start doing it across accounts, you create the peering connection between the VPCs, and one side creates the request, and the other side kind of accepts the request and allows the peering to kind of enable or connect.

SPEAKER_00

Is there a key exchange that happens in the interaccount case?

SPEAKER_01

Uh the VPC peering is no it's not encrypted, so there's no key exchange, but there is an exchange in terms of uh both sides have to agree to the connection. So um you you can't connect two things without both parties saying this is okay. Uh right.

SPEAKER_00

That's that's uh well you know you can on Twitter, but you can't on Facebook. That would be bad in networking terms. So is it um and I I don't mean to dive too much into implementation, but I think it's interesting just so everybody can learn a little bit about networking. Is it basically like adding a route? Is that kind of what's happening?

SPEAKER_01

So it it actually ends up working that way. Uh but you need the route. And so effectively the peering is the cable. And so if you don't have a cable between the two, let's say switches in this case, you can't create a route over that you know particular cable because that's not there. So the peering is simply the actual connectivity. And then once the connectivity is established, you create routes over that wire to say this is how you get to Confluent, or this is how you get to the customer.

SPEAKER_00

Okay. And that uh adding of the routes is a separate operation within the user interface of or command line tools or whatever of the cloud provider.

SPEAKER_01

Correct. Yeah. Got it. Okay.

SPEAKER_00

That sounds ra all rather manual. Um hopefully you're gonna give us a story where this can be automated a little bit. Um all right, so uh I guess just to make sure that we're we're clear on this. Um the I mean let's I guess let's bring Confluent Cloud back into the story. Uh there is uh this Confluent Cloud uh Kubernetes cluster in some region. Effectively, as far as I know, uh I have what I call a Kafka cluster in Confluent Cloud. Now uh quick footnote when we say cluster in Confluent Cloud, we abstract away the brokers. You know, you don't really think about the brokers, you're just saying, well, here's a cluster really becomes a namespace for a collection of topics and number of brokers and scaling and all that. That's that's what Confluent Cloud does. It figures that out for you. So when I say cluster, I mean group of topics. So you have a Confluent Cloud cluster. Um the reason, just to be clear that I know this, the reason I would want to pair uh or peer with it is so that because I have some application resources, uh, say just running on some EC2 servers, I've got some application running somewhere, uh again in the Amazon case, uh, and I want that application to be able to talk to uh Confluent Cloud, and VPC peering is the way that I do that. Is that right? Yeah. Is there any other way to do it without VPC peering? What are my options?

SPEAKER_01

So there are some other ways. Some are not quite there yet, but uh transit gateway is another option, and uh private link is a popular option that we're adding support for in the near future.

SPEAKER_00

What is transit gateway?

SPEAKER_01

Transit Gateway is similar to peering in the connectivity, but it gives customers more flexibility in the topology of their network. So it gives them the ability to connect more than one of their VPCs to the same Confluent Cloud cluster without involving Confluent at all in the kind of handsake handshake process. So it effectively is like a hub in the hub and spoke model, and we're simply our Confluent Cloud VPC is simply a spoke off of that, and the customer can connect you know one or ten or a hundred VPCs to that transit gateway to create connectivity to their Confluent Cloud cluster.

SPEAKER_00

Got it. If and transit gateway, by the way, is an AWS only thing, right?

SPEAKER_01

Yes.

SPEAKER_00

Is there an equivalent in that in Azure and uh Google that you know of?

SPEAKER_01

In Google, I don't believe there's an equivalent. In Azure, we're investigating something that looks like an equivalent, but nothing yet.

SPEAKER_00

As of this recording in mid-October 2019. Uh maybe. Yeah. So what um if if you were gonna make an analogy, I think we we tried to analogi peering to traditional kind of physical network infrastructure. If you were to do that same thing with transit gateway, what is transit gateway in terms of traditional infrastructure components?

SPEAKER_01

It feels like it's a few things together, but it's it's literally instead of if VPC peering is just a wire, then transit gateway is your router, and you're connecting many wires into your transit gateway. So one of those wires would be our VPC into that router, but then you're allowed to kind of connect as many other wires into that router as a customer owning the router as you'd like. Got it. So it is a it's a cloud router, basically. Effectively, yes. Okay.

SPEAKER_00

Okay. Um and when would I do that? Like when would I want to do that rather than peer?

SPEAKER_01

I think it it for sure depends on the company and the network topology that exists or that you that the company wants to exist. So if you have a lot of VPCs that need connectivity, um it seems like Transit Gateway might be the solution for you. But there is a situation with just regular peering where if all of your VPCs are in the same region, we can peer multiple of your VPCs to the same cluster. But the problem is every time you want to peer to our VPC, we need to be involved. And so if you're constantly adding and removing VPCs in your use case, uh Transit Gateway will give you the flexibility to do that without involving Confluent, like a support request, every time you want to do that. Um, but regular peering won't. Like you'll have to involve us for every single peer that you'd like to create.

SPEAKER_00

Got it. Got it. That's not automated. Is that um, and like anytime we talk about Confluent E things on here, I'm always very careful to say, folks, we're not talking about a roadmap because that's not what I do ever. That is my hashtag safe harbor hashtag safe harbor statement. So given that, um, is that the fact that it's manual now, is that the kind of thing that it is conceivable to up to automate in the future and we just haven't, or there are good reasons why that can't be automated and and we shouldn't look for that.

SPEAKER_01

Automation is coming. It's just not been on the immediate roadmap. There you go.

SPEAKER_00

Whatever that roadmap is, we're not saying, but so it's it's conceivable that one day uh there's a possible world in which there's a future version of Confluent Cloud where that uh VPC peering is something I can accomplish through UI. Exactly. Yes. Nice. But for now, in AWS, if you're in AWS, Transit Gateway is the money there. And if you're in Azure and Google, that that's uh there isn't any currently supported infrastructure that does that. You mentioned private link. What is private link?

SPEAKER_01

So private link is a new style of connectivity that AWS release. It's effectively double NAT as a service, which is probably pretty jargony, but you know, it allows us to kind of it it allows us to connect networks that could possibly overlap in IP space without having to worry about that. So one of the issues with peering is our IP space has to be joined with the customer's IP space. So we have to agree on what ciders we're gonna use as well as what ciders the customers are gonna use. And um, sometimes that's actually complicated and hard. With Private Link, you just connect them and we don't actually worry about those kinds of conflicts ever. And so we can easily, and it's meant to be automated in a way that we can just drop those connections straight into the customer account and give you access very quickly to your cluster with very little kind of uh uh back and forth between us and the customer.

SPEAKER_00

So Got it. Because these uh these are all private IPs that uh the public clouds use when you when you get assigned an internal IP. That's a non-publicly routable private IP.

SPEAKER_01

Yeah. Effectively what it what it'll look like is instead of a route out of your VPC to go to our VPC, it'll just be new IPs that show up in your VPC that you connect to that has confluent cloud behind them. So it's it feels like just more instances in your VPC.

SPEAKER_00

And there's a layer of networking infrastructure that is somewhere along that VPC peering pipe that does that IP translation.

SPEAKER_01

Yeah, the private link, the private link IPs. Yeah, exactly. Yeah, okay. Yes, yes, that is in fact private link.

SPEAKER_00

You said cider, and some people know what you mean. Some people are thinking of like a fermented fruit adult beverage, uh, and I think you should set them straight.

SPEAKER_01

You could have meant either one, but yes, so cider is uh simply the the class of networks. So if you have uh a certain subnet that you would uh give to uh a customer or something. So for instance, a lot of people talk about slash 24 networks, or it's it's it's a grouping of 2025 contiguous IPs that you are willing to lend to somebody. So uh you know a classic example is like 10.0.0.0.0 slash eight, and that is a CIDR CIDR.

SPEAKER_00

Gotcha. It stands for classless interdomain routing. So that 10.0.0.0 slash eight, that means the most significant eight bits are the thing you should route on, and the other 24 bits are stuff inside this network, and don't worry about them. Exactly. Got it. Okay, and like so uh 192.168 or 192 uh you you know the rest of it. Uh and I'm and I'm blanking on it. You know that one. Yes, that one the 24-bit one. Uh uh that that would be uh like you see that in a lot of home networks. Um your IP that you may know at home, um which is 192, 168, something, something. Uh so yeah. Yes. And so reconnect that. You said that, uh, and if you could just go over that part of what you said with respect to private link, because I want to make sure everybody got that.

SPEAKER_01

So with with private link, you don't have to agree on which networks we're both allowed to use. So private link shows conflict cloud service as IPs in the network that you already have. Whereas with VPC peering, we have to agree to not overlap. So an example might be um with your home network range, 198.1 or 192.168.001. Slash 24, we can't use that same range. We need to use something like 198, uh 192.168 uh 1.0 slash 24. They have to be separate. And that has to be for the whole network, right? So if you have a topology of hundreds of EPCs, all of those networks cannot overlap with our network if you want them all to be routable. Um and so that's kind of a problem when it comes to peering, is you have to kind of, you know, it's called IPAM or IP address management to kind of know that these uh citers are not in use anywhere. And so with private link, we lift all that, and there's no kind of cider agreement anymore. You just create the Confluent Cloud private link endpoint and away you go.

SPEAKER_00

Nice. It'll just get mapped away for you. Yeah. Uh and is that again, private link is the name of, in this case, an AWS service. Correct. So if you're running an AWS, uh we'll at some point be able to employ that to do this uh IP address mapping.

SPEAKER_01

And Azure is bringing it. Oh, Azure as well. I believe it's in preview right now for Azure.

SPEAKER_00

So excellent. Um you've been talking about this as uh an intra-region thing. So I've got a single uh region or availability zone. Um and uh my Kafka conflict cloud is running in that region, and my application is deployed to that region. Can I do this across regions?

SPEAKER_01

The answer is yes. It depends on the cloud provider's support of inter-region peering. So in AWS, peering now can work across regions that didn't used to. And so we can support it in AWS. In Azure, I believe we do not support it. And Google, I I'm unsure actually. So got it.

SPEAKER_00

Those are those are things uh number one, things we can check, and number two, things that tend to change over time. So if you're listening to this significantly after uh, you know, say November of 2019, that could be a different story. Um is it so that that seems like how I might do DR or like you know, have resources in multiple geographic availability zones so that uh if something very bad befalls one data center or network connectivity in one region, um my application can still survive elsewhere. Is there a way to do disaster recovery without that? Like how would you do that?

SPEAKER_01

It's not as it's not as straightforward, I guess. Like I think the the optimal example is having two confluent cloud regions where you have Kafka running, and then two regions where the customer's application is running that kind of match each other, and then you crisscross in the middle. So if any region were to be inaccessible or down for some reason, you're constantly kind of you know using Kafka to replicate that data between the clusters. And so if you don't have that, uh I mean it it's it's hard. I'm not exactly sure how you'd do it. Right. So got it.

SPEAKER_00

Um what um I feel like I have a good I I feel like I have a good picture of how this works now, and I hope if you're listening, you do too. Is there anything else we should know about this stuff? Or uh you know, if you find like when customers are working with this and there's networking knowledge that they don't have that gets in the way, what what do you want people to know about uh cloud networking? Uh that you know you have an opportunity to tell them right now. So what would you tell them?

SPEAKER_01

Uh it's more complicated than it looks sometimes, but we're trying to make it simpler and easier all the time. Uh there's a lot of different ways people want to connect to conflict cloud, and uh we want to enable those because uh everybody's infrastructure is set up different. So if we can enable it, then uh you know you can use Kafka better, easier, faster.

SPEAKER_00

Um and you know, maybe I should have asked you this first, but I'm kind of curious. Uh what is your tell us about your background? Like how did you um you know, you're a you're a cloud SRE now, but uh how do you get there? What was the the journey you took?

SPEAKER_01

Yeah, I I mean I guess it was a it was a interesting journey. Uh I started as a developer, but uh, you know, eventually grew into the Kubernetes space and learning about how to deploy that. And a lot of the networking aspects of it were kind of problematic as we did it on premises. And so I learned about a lot of the on-premises networking through that and eventually um you know got to a position where I needed to figure that out in the cloud and how they kind of mapped the how we do this on premises in the cloud. And so uh it just kind of naturally fell into these details and uh figuring out how we can connect customers to Confluent Cloud with different networking options. So nice.

SPEAKER_00

I'd say the uh sort of pure DevOps route, really. You were a developer, you started working with infrastructure, writing code for infrastructure, and now here you are. Exactly, exactly. I want in my uh own exposure to networking things, uh, this was uh more than 10 years ago, but you know, I was responsible for some on-prem uh infrastructure. And I was uh in a recent episode just talking about this, where there was real sheet metal that you could really cut your fingers on, and a real cold, noisy data center that I spent time in, and like everything that is not how we do things these days. And there was a bunch of networking we had to do. There was a uh customer that wanted to drop EDI files on us nightly, which gives you some idea of the um the vintage of some of the technology involved. Um legit EGI files, EDI files that they were doing file drops over a VPN, and it was super squirrely, and it was just really hard to get up and running and to keep the connection up and you know, all this. It was just site-to-site VPN between our rack and and some stuff at the the customer site. And I was talking to the uh we didn't say SREs back then, but the SREs at the data center, you know, that supported all this stuff and helped us. And I I um was just making a joke. I said, man, this IPsec stuff is just because it was an IPsec site-to-site VPN. This IPsec stuff is just too hard. I think we should switch to triple sec, which like I'm super impressed. That's a pretty solid dad joke, right? I thought it was funny. That's good, but that's good. I think it's funny. It's great, but such such was my demonstrated knowledge of networking technology that the lady I said that to, who was the SRE there, uh, you know, she was super sharp, and she didn't laugh right away because I think she wasn't sure whether I knew. Right? Like, does he know he's making a joke or is he just that confused? So this is uh ladies and gentlemen, listening audience, this is how you do not become an SRE. Demonstrate your confidence to people such that when you're making a joke that you think is funny, they actually can't tell if you're joking or not. So, Dan, that was clearly not you. I got it. Anyway, good. I'm glad I'm glad you got it. And you laughed, you thought it was funny. You like knew it. Yeah, yeah, it's good.

SPEAKER_01

I like those dad jokes. Those are my favorite kind of jokes.

SPEAKER_00

They're the best kind because you just don't care whether anybody thinks they're funny. That's the whole thing. Anyway, um you want us to know about VPC peering, cloud networking, anything? Final uh final words.

SPEAKER_01

I think that's pretty much it.

SPEAKER_00

All right. My guest today has been Dan Lamont. Dan, thanks for being a part of Streaming Audio.

SPEAKER_01

Thank you.

SPEAKER_00

And there you have it. Before I go, I want to tell you that we have a pretty cool new offer to help you get started with Confluent Cloud without you having to pay for anything. If you're a new user and you go through the regular sign-up process and start using Confluent Cloud, your first $50 of usage per month are free. This will last for the first three months after you sign up. So that's $50 per month of serverless Kafka for three months at no cost to you. So go to the sign-up link in the show notes. I don't want to read you the URL, and sign up now. I think the only thing I could really do more is write your code for you. And I think we can both agree that's too much to ask. So check it out and hey, let us know how you like it. Anyway, as always, I hope this podcast was helpful to you. If you want to discuss it or ask a question, you can reach out to us on Twitter at Confluent Inc. or reach out to me at TLberglund. That's T L B-E-R-G-L-U-N-D. Or you can hit us up in Community Slack. There's a sign-up link for that in the show notes as well. And while you're at it, please subscribe to our YouTube channel and to this podcast wherever fine podcasts are sold. And if you subscribe through iTunes, be sure to leave us a review there. That helps other people discover the podcast, which is a good thing. Thanks a lot for your support, and we'll see you next time.