Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Hi, we’re Tim Berglund, Adi Polak, and Viktor Gamov and we’re excited to bring you the Confluent Developer podcast (formerly “Streaming Audio.”) Our hand-crafted weekly episodes feature in-depth interviews with our community of software developers (actual human beings - not AI) talking about some of the most interesting challenges they’ve faced in their careers. We aim to explore the conditions that gave rise to each person’s technical hurdles, as well as how their experiences transformed their understanding and approach to building systems.
Whether you’re a seasoned open source data streaming engineer, or just someone who’s interested in learning more about Apache Kafka®, Apache Flink® and real-time data, we hope you’ll appreciate the stories, the discussion, and our effort to bring you a high-quality show worth your time.
Confluent Developer ft. Tim Berglund, Adi Polak & Viktor Gamov
Using Terraform and Confluent Cloud with Ricardo Ferreira
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Tim Berglund hosts Developer Advocate Ricardo Ferreira to discuss the concept of infrastructure as code, as well as the differences between Terraform, Ansible, Puppet and Chef. They also chat about why Terraform is such a big deal, some of the challenges involved with learning it and how Confluent leverages Terraform to achieve multi-cloud support for Confluent Cloud and tools for Confluent Platform.
EPISODE LINKS
SEASON 2
Hosted by Tim Berglund, Adi Polak and Viktor Gamov
Produced and Edited by Noelle Gallagher, Peter Furia and Nurie Mohamed
Music by Coastal Kites
Artwork by Phil Vo
- 🎧 Subscribe to Confluent Developer wherever you listen to podcasts.
- ▶️ Subscribe on YouTube, and hit the 🔔 to catch new episodes.
- 👍 If you enjoyed this, please leave us a rating.
- 🎧 Confluent also has a podcast for tech leaders: "Life Is But A Stream" hosted by our friend, Joseph Morais.
Chef, Puppet, Ansible, and now Terraform? What's that all about? Well, we'll find out on today's Streaming Audio, a podcast about Kafka, Confluent, and the Cloud. Welcome back, everyone. I'm your host, Tim Berglund. Today I have with me in the virtual studio my friend and co-worker and Confluent developer advocate, Mr. Ricardo Ferreira. Ricardo.
SPEAKER_00Hey Tim, thanks for having me. Very excited with the opportunity to be here.
SPEAKER_01You got it. You got it. Glad to have you on the podcast. Tell us a little bit about what you do here before we get into our subject today.
SPEAKER_00Sure. Well, I work at Confluent. Uh, as you may know, the conf the company that was created by the inventors of Apache Kafka. And my role in the company is working as a developer advocate. Which means that I spend my time talking to developers and making sure their lives are less miserable. So I think that's summarized what I do.
SPEAKER_01That's I like I like the way you put it. I tend to word things, frame things in that same way. Some people might say you try to make their lives more happy, but like we know, we who have done this for a living, we're trying just trying to make your life less miserable. That's all. Anyway, you've got a lot of experience with uh cloud technologies. That's the thing you focus on, on the team. Uh everybody in the interest of full disclosure, uh Ricardo is on my team in uh the developer experience team here at Confluent. Uh, and you got a lot of cloud background. And when you came here a few months, you brought uh a lot of enthusiasm for Terraform now. Uh if you haven't heard of Terraform, um it's a new-ish tool, it's kind of increasing in profile recently uh in the tradition of the uh DevOps automation tools of your like Chef and Puppet and Ansible and things like that. But I don't wanna so if if you haven't heard of it, that's what it is. Um I don't want to say anything else about it because we have an expert on the show today who will tell us, Ricardo, what is Terraform?
SPEAKER_00All right. Um I think the best way to describe Terraform is by explaining uh what is the motivation for it. I mean, I think every software technology has this behind motivation, so I think we can start from there. Um when Terraform was created, uh it was in a time where cloud provider was kind of uh catching up. And uh as any other, as like any other technology, I think developers want to avoid being locketed into a specific technology or provider. So the main the main goal of Terraform is to effectively address how developers can abstract infrastructure in such a way that they can deploy their infrastructure in any cloud provider. I think that's the first reason why Terraform exists. And secondly, and probably most importantly, uh Terraform is an instantiation of the concept of infrastructure as code. So for those of you that are familiar with the concept of having your entire infrastructure defined it as a code, and therefore you can execute that call as many times as you want it, I think that simplifies the understanding of Terraform. So, in other words, if you want to define an architecture, I'm sorry, um, an infrastructure, and you wanna you want to have this infrastructure as controlled by a software version software, or maybe you want to run tasks, you want to perform regression regression tasks on top of that code, you can manage as if you were part of a larger code base for a specific project. So that gives you the ability of managing the infrastructure as part of your entire your remaining business logic with the other code that you your project may contain. But as I said, I mean Terraform does a nice job in uh abstracting the infrastructure details for you, and that gives you the ability to uh deploy the infrastructure in any cloud provider. And also it gives you the ability to deploy even on-prem. So you have uh plugins for um OpenStax, for example. So you don't necessarily need to rely on Terraform just for the cloud, so you can have hybrid deployments where infrastructure is both defined for the the two layers.
SPEAKER_01Okay, pretty cool. So um it's it's kind of grown up in the cloud era, and it seems to have the assumption that this is for automating cloud deployments. But like you said, if you've got at least OpenStack kind of private cloud sorts of infrastructure uh on premises, then you can use the same infrastructure definition to deploy to both places.
SPEAKER_00Right, right, exactly. So that gives a team's ability to uh to observe infrastructure as a code, as at heart. And no matter if you are dealing with on-prem or cloud, I think what's important is that you can rely on Terraform to use as part of your team's future. And um therefore it's it's gonna be part of every project that a team is going to implant. So I think that's exciting.
SPEAKER_01So briefly, just in case for the for the very uninitiated, um you the the term infrastructure as code is not especially new, but you might be new to uh this world. And if you're listening to this podcast and you aren't familiar with that phrase, briefly, what it means is uh you write a text file that describes in some language, um, you know, some kind of domain-specific language, uh, what form you want the infrastructure to take. Like I want this kind of server, and I want it to have this Linux distribution and I want it to have Confluent Platform 5.1 installed on it and Tomcat 3.3, because I want to remember the good old days when it was everything was simpler or something like that, right? You just uh you you define components and maybe uh this file needs to go in this directory, and here are some secrets you should have access to, you know, all that kind of stuff. Uh you define that in a source file, and then the tool takes that source file and goes and makes it happen in the cloud or in your OpenStack or whatever. So that's what if if you're not by some chance you're not familiar with infrastructure as a code, I'm glad you're listening. And that's uh basically what that means. The nice thing is, then it's code. You check it into GitHub and you can collaborate on it through pull requests like anything else, as opposed to, well, I ran some commands at a terminal and forgot to write them down. You know, that's kind of the very bad old days of operations. And this is this is really modern operations how we maintain infrastructure these days. So with that, Ricardo, why? Uh so we had Chef, we had Puppet, we had Ansible. Um that debate or I don't say debate, that landscape continues to shift, like who has the upper hand there. So uh compare and contrast. Why Terraform given those three things? Is it the same? Is it different? How does it differ? What do you think?
SPEAKER_00That's a very good question, Tim. Um and I've heard a lot discussions uh as well about who is gonna be the next big thing in the area of infrastructure as code, particularly uh comparing Terraform and Ansible, which is two great technologies. But I'm gonna use something that you just said as an example to explain what are the differences. So while you were describing infrastructure as code, you gave the example about oh, I'm going to define my uh VM, and this VM is gonna have the specific hardware, and I'm also gonna put Tomcat version 3.3 on the top of this VM, etc. etc. So uh when you start saying things like I'm gonna use Tomcat in specific versions and software running in top of the infrastructure, that's what we call as bootstrapping, which is the ability to mutate a specific hardware resource and put some behavior or logic or a specific state that you want to leverage as part of your architecture. So in the area of bootstrapping, that's what technologies like Ansible, Chef, and Puppet do at its best. I mean, they are very good in bootstrapping uh pre-existing infrastructure and make sure that that infrastructure is going to look like to a specific golden image. In other words, you make sure that infrastructure is going to behave and have the appropriate software that it should have. Whereas Terraform is more about defining and creating infrastructure. So uh what that means is that you usually use Terraform to define, manage, and create or destroy your infrastructure. But Terraform is not necessarily the best tool to perform bootstrapping. So, as I said, bootstrapping is more the target goal uh of Ansible, chaff, and Puppet, and Terraform is more about uh defining and describing. So, my opinion about these two technologies is that they can literally can be used all together as and in conjunction, you're gonna you're gonna end up with a very powerful way to handle all things related to your infrastructure and the behavior on top of this infrastructure. So I conclude this um comparison by saying I wouldn't necessarily compare them because if you think about it, one is more dedicated to uh provisioning, which is Terraform, and the other one is more dedicated to bootstrapping, which is Ansible chat and pop.
SPEAKER_01Okay. Now, and make that let me drill down on that a little bit. When you say bootstrapping versus provisioning, go over that with us one more time. What do you really mean uh by the difference between those things?
SPEAKER_00Sure, no, no worries. Um, if you look to uh um a Terraform code, you're gonna see that all the components, for example, they are described as uh what they are, the nature of their R. Like, for example, I'm gonna create a load balancer, right? And you want to define uh what are going to be the characteristics of this load balancer. For example, this load balancer are gonna have one listener, and this listener is gonna be for handling traffic over port 80, for example, right? And this is what we call defining your architecture, sorry, defining your infrastructure. But it doesn't give you the details about how that listener is going to be executed on that particular VM. So when you start digging it into uh how the software is going to be installed, configured, oh the uh we're gonna use Apache web server or any jinx or any kind of a web server technology, those are details that are usually covered and when we do the butt strapping part, right? So um usually when you start dealing with the how do you provision things, you end up looking to a code that looks like a script uh using uh some sort of imperative programming language. Like you execute the code top-down, and you're gonna end up with instructions that is going to install things, set up things, or patch things, and making sure the software is going to be the specific behavior, right? And their form is more about defining how the infrastructure is going to look like, and eventually that infrastructure is going to be instantiated. But it's instantiated in a raw form. It's not usable. I mean, you cannot you cannot use a simple VM if there is nothing, no software installed on that VM, right? So installing software and making sure the software is well configured is the area of bootstrapping, which is covered by Ansible and ShafiPuff, or something like that.
SPEAKER_01Okay, okay. So I, you know, maybe I have uh five M4 larges and a uh load balancer and uh you know some sort of data infrastructure somewhere, like maybe a Kafka cluster. But they're just the M4 larges in the load balancer. Now I need to put things on them. I need to go install my Tomcat 3.3 on them. Yep. And that would probably, in your mind, best be done in Ansible.
SPEAKER_00Exactly, exactly.
SPEAKER_01Okay.
SPEAKER_00There's one uh very good uh difference between them as well that I like to emphasize when I explain Terraform, which is about the dependency management. So if you are a developer, a programmer that comes from the uh scripting board, how do you define dependencies between models? So usually you have this main program that once executed, start executing sub uh sub-routines or sub-programs in a sequence, right? And that's how you ensure that the dependencies look first create A and then create B and then create C, right?
SPEAKER_01Yeah, in an imperative model, the order of operations, when you're talking about dependencies not in a build sense, but in infrastructure, uh you've got an imperative script and you get to define the sequence of the operations, and so you know get that sequence right. So your dependencies are satisfied.
SPEAKER_00Exactly. Exactly. And uh the the way Terraform handles this is a little different because you don't have sequence in the Terraform code. What you have is implicit and explicit dependency. So it's more high-level. So for example, you as a developer, you end up creating architectures, right? So you might say that your architecture is comprised by a public load balancer, which is going to uh handle the inbound traffic of requests, and behind this load balancer, you're gonna have one or more VMs that runs in a private subnet, and that private subnet is for your application server layer, which in turn is going to uh talk with another private subnet that runs your databases or Apache Kafka or any other persistence store that you might have. So this home architecture and its dependencies is more easily defined and described using Terraform because you don't necessarily have to write a logic to create this uh to create this sequence, right? So uh I think dependence management is another powerful constant from Terraform that if you start struggling more with uh that when you are defining your um provisioning tool, that's probably a very good indication that you are using the wrong tool for the run for the right job. So, for example, if you are struggling to define dependencies, it means that they may be Ansible might be not your answer. You shouldn't look for Terraform.
SPEAKER_01Okay, okay, that makes a lot of sense. And um another thing, uh and I and I just want to say uh if you're if you're listening, the questions I'm asking are not affected ignorance about Terraform. I actually don't know it well at all, uh, and Ricardo does, so I'm not as often as not I'm asking a question that I actually want to know the answer is. Having said that, having looked at some Terraform code and being familiar with the other tools, it feels to me like functional programming people made it. Like I look at that and it looks like it is following the functional paradigm, whereas certainly Chef and Puppet feel like DSLs in an imperative scripting language where there's structure and everything, but you're you're writing code. You're writing, you know, this step, then this step, then this step. And they they attempt to be moderately declarative, but you can always break out into uh imperative mode easily, whereas terraform looks like a functional language to me. Do you think, Ricardo, is that am I onto something there?
SPEAKER_00No, I think you were right, uh Tim. I mean, uh in a matter of fact, you nail it that the main differences between uh uh Terraform code and uh chef code is exactly the behavior, how you define it, the the logic you well write. So definitely uh chef is more imperative, as you said, and uh Terraform is more uh functional. So that also explains why some people might struggle with Terraform. If you are a Shell script developer that has been spent the last 10 years creating scripts for Linux servers, you're definitely going to be more uh comfortable writing Shell scripts. Whereas if you have more less than uh probably 25 years old, you're born in a functional programming era, so you're likely to be more comfortable writing Terraform script. But I think that uh kind of uh influences the type of uh technology they're gonna use for your um infrastructure escal project.
SPEAKER_01Yeah, got you. So uh this is not your parents' C Shell script. Um it's it's a little different. But does that I was I was gonna ask you how you learned it and is it difficult? And like what we have listeners who are new to Terraform, they want to pick it up, they want to do stuff with it. What sort of challenges should they be ready for? And what was your journey through adopting and learning it?
SPEAKER_00Um yes, uh, I think I would say that Terraform Code, which is basically a high-level DSL, uh, is it's from a programming language called HSL, which is the acronym for HashiCorp configuration language. And it's very simple to understand in a sense that if you read the code, you definitely if you are a software developer, of course, right, if you are familiar with uh JSON payloads, you are definitely going to be very, very comfortable in reading the code, right? Because in the end of the day, it looks like lock with JSON and the semantics of a JSON payload. However, and I might say by personal experience, that it's not that easy to express some logics in Terraform code exactly because of the reasons we've discussed them before. Uh again, Terraform is not an imperative programming language. So I when I was creating an open source project for Confluent, I had to deal with some situations where I had to define execute things multiple times. So for any decent developer, they would express this using a looping, some sort of a looping uh conditioning, right? So for looping, a while looping, a do repeat looping. And those constructs are not present in HSL. So that's uh that's a uh a very good indication uh how much you are going to struggle if you think imperative uh when you are dealing with turbo. And uh so it doesn't necessarily mean that there's no answer, there's no way to do it. It's more like uh you have to research and find different ways, functional ways, to express the same behavior. So yeah.
SPEAKER_01Which would be like you know, if you have a Kafka Streams application that you're deploying, and you know, it's it's an Uber jar and you want it to run on you know five of some instance type in whatever cloud you're running in, you're gonna define that. Here's a node in my Kafka Streams app, and then elsewhere you're gonna say, Oh, I want five of these. Is it like that?
SPEAKER_00Yep, exactly, exactly. So uh in order to handle this repetition, you have to come up with different constructions other than a looping.
SPEAKER_01So Yep, yep, yes, you do. And if you have only ever been an imperative programmer, the first time you do that, I can say from personal experience, is weird, right? Because I I grew up as uh, well, very early on, a basic programmer, then a C programmer, then a Java programmer, and what do these languages all have common, but they're all imperative. And then for the first time I try to write foreclosure, it was it was really hard. Right? It's just a totally different way to think, but it's rewarding. And so if if you're new to this kind of thing, I would say far from letting that scare you off, it should pull you in. Um you you do need to give yourself an opportunity to go through the exercise of learning the functional paradigm at some point. Um so you know, whether Terraform is that or not, I I would I would encourage you to look at that, look at that as an attractive attribute that uh will just help you level up. Yep. How do uh how do we use this here at Confluent? And uh hey everybody, I'll give you a hint. Uh I kind of know the answer to this question, and all of it is because of Ricardo. So, Ricardo, how do we use this here at Confluent?
SPEAKER_00Um well, we we use it in a variety of forms, and uh, but just to give you some concrete examples, um Confluent provides a Apache Kafka as a service product called Confluent Cloud. And uh this product is somehow offered to users as a SaaS product, which means that there are no exposure whatsoever to the underlying infrastructure. So you don't necessarily need to know where your cluster, Kafka cluster, is ready.
SPEAKER_01Uh but more of you get a you get a domain name and a port and a key and a secret and a little bit of config to put in your client, and we do the rest.
SPEAKER_00Exactly, exactly. So as a developer, you all you're basically entitled to how do I connect to that thing that's running somewhere. And one of the great things about Confluent Cloud, it allows you to run in multiple cloud providers, such as uh AWS and GCP, Google Cloud Platform. And this way we the way we've could we've built that uh it gave us a lot of uh a lot of challenges when we what we were building this but in the end it gave a very great result which is the ability to provide portability between cloud providers which is something that we offer to our customers and the way we abstracted the infrastructure part is guess what using Terraform so uh of course this is not very easy to understand uh by the end user because as I said the product is offered as a SaaS experience so you don't necessarily have to worry about that but for the sake of understanding that's what we use uh under one in the underlying architecture of our provision process and on top of the Terraform you you have the first layer of abstraction which is the infrastructure that is handled by Terraform and have a second layer of abstraction which is handled by hey Kubernetes which is another great technology that our friend Victor is the expert on the team. So we have a first layer of abstraction which gives away the infrastructure part and we also have a second layer of abstraction which make the application as freed of concerns about clustering, scaling in, scaling out etc etc so that's how one of the examples how we use in Confluent. And there is another project that we offer as a kind of a tool for for our users that wants to run confluent platform tools in any cloud provider. And this open source project allows developers to uh host tools like a schema registry, a REST proxy, Kafka Connect, KSQ servers and Control Center and multiple cloud providers and we also rely on Terraform to virtualize the infrastructure.
SPEAKER_01There you go. And I know you've written some um pretty cool demos uh that use Terraform to kind of get all of the infrastructure set up and the thing is like regardless of the whatever future version of Confluent cloud you know it it's in it's completed future form all elements of the Confluent platform will be present with a click and a slider and you know you want this much KSQ capacity and this many connect nodes and all that kind of stuff like uh regardless of the uh what Confluent cloud can deliver the whole thing about building applications on Kafka is that you're building applications and you're gonna write programs, you're gonna write producers and consumers and Kafka streams apps and things like that that that are going to run on servers that never properly belong in a cloud service like ours, right? They're gonna you're gonna host them in um Azure or Google or AWS or wherever it is you want that to be. And so a a tool like Terraform um and it's not it's not like we're selling it here. I mean it's you can it's just an open source thing. It it's it's a thing it it's a tool that ought to be in your toolbox but a tool like it is a nice companion when you're building stuff on a cloud service like a confluent cloud because you're you're never gonna with a SaaS service like us, by definition it's data infrastructure. There's other stuff you have to deploy you know there's there's all of your stuff your application that's using the data infrastructure that's got to live somewhere. And so a tool like Terraform is nice.
SPEAKER_00Yeah I I uh I agree 100% and uh one thing that's interesting about this whole process of using Terraform to um to like you said like uh free the developer for the responsibility of warring with these concerns is that when you think about cloud right you are dealing with moving parts that uh you're gonna spend a lot of time handling not only the infrastructure creation but also problems like uh connectivity may arise that some problems that if you were working with Apache Kafka on-prem they would never uh surf out but uh when we bring this infrastructure to the cloud you may have problems such as okay I am writing a producer that's trying to write some records to a topic that runs in a cloud but for some reason I'm having a connection refused I am pretty sure that the VM is up I can ping the VM but my producer is not communicating so what might be happening here it might be happening that you might you might have fire rolls and secure groups and things like that between your layers so you can use any you can use interform to define those fire rolls and uh protections over layers and division by subnets that for again for the developer perspective they simply want to run the code and make sure their client application are interacting with the cluster and that's it. So they don't have to worry about things that would take them hours of troubleshooting and hours of delay and that brings agility for a whole new level.
SPEAKER_01Cool cool well if you want to know more it's uh just terraform.io t-e-r-r-a-f o r m dot io super cool tool that we get use out of and you know Ricardo you've brought some expertise to the team and started building demos based on this and it's been real effective and like I said you know we're not selling it it doesn't it doesn't have anything to do with confluence business or anything like that but um it's helpful and it's it's uh a tool growing in priority and I just wanted people to be able to learn a little bit more about it. You got anything else uh closing thoughts on Terraform you might add?
SPEAKER_00Uh well other than thank you Resh Corp for bringing this technology to the world I think we're done here.
SPEAKER_01All right Ricardo thanks for being a part of Streaming Audio. All right appreciate it thanks man and there you have it I hope that was helpful to you if you've got questions you can ask me at TLBurglund on Twitter. That's T-L-B-E-R-G-L-U-N-D or you can leave a comment on any of our YouTube videos. Your question might be featured on the next episode of Streaming Audio. And feel free to subscribe to our YouTube channel and this podcast wherever fine podcasts are sold. And if you subscribe through iTunes be sure to leave us a review there. That helps other people discover the podcast and just generally helps us get the word out. We appreciate your support. See you next time
