Reuse Don't Repeat, Creating an IaC Library

Session Information

• Session Transcript

  We’ve all had the experience. You’re writing some code or scripting a process and you realize you’ve done this before, probably more than once. At that point, hopefully your desire is to go and create something to stop you having to do it again, writing some snippets or a module that you could share with your team. Well, in the infrastructure’s code world, it’s not really that much different. There’s only so many ways you can deploy a virtual machine.

  And so ideally if you’re gonna be doing a process more than once, if you’re gonna be deploying a resource more than once, you’ll reuse rather than repeating the code. And that’s what we’re gonna talk about today. We’re gonna talk about how you can use infrastructure as code tools to build a library of modules and reusable resources that hopefully will help you not repeat yourself, but also hopefully help others in your team and your company to get the benefits that you’ll all acknowledge. My name’s Sam Cogan. I’m a Solutions Architect with a company called Willis Towers Watson, and I’m on Microsoft Azure MVP.

  I spend a lot of time doing infrastructure as code and I’m quite lazy, so if I can write code once and reuse it again later, absolutely going to do that. And so I’ve spent a lot of time working with building modules and reusable libraries to not only help myself, but to help others get to grips with building infrastructure as code and making their lives simpler. So let’s dive a bit deeper into creating reusable modules using infrastructures code and why you might want to do that. But first and perhaps most obvious reason, you want to save yourself time. You want to stop repeating yourself.

  You want to stop writing the same code over and over again and so you create a reusable copy of your code and you’ve only got to write that once and hopefully amend some parameters or so when you use it. And that’s quite often the driving reason why somebody might want to start creating reusable infrastructure as code modules, particularly if you’re working by yourself. But another benefit of this is the fact that you’re gonna promote consistency. If you were writing the same code over and over again, you’re probably not actually writing the same code. It’s gonna be slightly different.

  And so you may find that you’re deploying resources slightly differently. Maybe you’ve got some configuration settings that are different when you, you know, the first time you did it to the second. Maybe they don’t make that much of an impact, but they’re different and over time as you write it again and again, those things are gonna change. If you use the same code, the literally the same code over and over again, you’re gonna get the same output and that’s gonna make your resources a lot more consistent, hopefully prevent you making errors, or doing things like leaving some security settings that you really want to be enabled off because you forgot to write about them on this particular time. So hopefully you’ll get a lot more consistent and maybe even more secure resources when you deploy them using a reusable modules.

  A lot of the time, you’re gonna find though that you’re not working on your own and there’s other team members who are also working on the same infrastructure you’re working with and so by creating reusable modules, you can help these people as well. Firstly, you can help make things a lot less complex. Let’s say you’ve got a way you want to deploy a virtual network and you want everybody to use the same approach and there’s lots of different settings that need to be set up to make it the right approach. You need to set up certain security groups. You need to set up certain route tables, all those sorts of things.

  Getting everybody who’s doing that to follow that approach is gonna be really hard and everyone’s gonna do things slightly differently, probably miss some things, and if they’re new to the use of infrastructure of code, it might even make the job very daunting and difficult for them to get started with. So if we create some modules that can do that for them and make those modules so that they hide a lot of the complexity, it makes it a lot easier for people to get started with using the infrastructure as code and following your process and it makes sure that they’re actually following what you want them to do every time. So you can use these modules to take a very complex set of infrastructure you want to deploy and hide it behind a much simpler interface that everyone can use. And alongside that you can then use it to apply standards and these just might be the way you like things done, but they might also actually be standards you’ve got from a corporate level or from a regulatory approach, or to meet some sort of compliance that you have to have resources deployed in a certain way. Again, if you don’t have any reusable modules, then you’re relying on people actually applying all those settings every time when they write their infrastructure as code.

  Whereas if you create a module that’s got all of those standards baked in, first it makes everybody else’s life easier because they don’t have to worry about that, but secondly, it ensures that you’ve actually got resources deployed and they’re deployed in exactly the way you want with the appropriate settings, meet the standards you need. Finally, if you’re working with lots of different people, it’s quite likely you’ve got your own area of expertise, maybe you’re the networking person or the web app person, but whatever it is, you can bake your expertise into a module and then let everybody else use that. So, you know, if they’re not familiar with networking, but they just need a virtual network to deploy their application into and you can give them a module that’s got all your expertise baked in and they can just run that and they get a really well-configured and secure and compliant network that they can then use for the rest of the application and so it’s much easier for you to share your expertise rather than having to go try and go and train that person up to understand what you’re doing in that code or the very least wrote in documentation that they have to follow when they want to deploy a virtual network. You can just give them a module and say, “Run that and you’ll get all the infrastructure you need exactly as it needs to be configured.” So we know why we want to create these reusable modules, but how do we actually go about doing it? And this is where it becomes much more dependent on the tooling you’ve chosen to run your infrastructure as code.

  Generally, you were bought into one language or another, and we’ve got listed here as some of the most common languages. There are obviously other ones and they will have their own way of doing it as well, but I’ve tried to call out some of the most common ones. So we start from the top here. If you’re doing deployments into Microsoft Azure, and you’re using the native infrastructure’s code tooling there, then you’ve got either Azure Resource Manager or ARM templates and these use a feature called nested templates is their sort of modularization approach where you can write your infrastructure in one file and call it from another. Now that works, but to be honest, at this point, I wouldn’t recommend going down the ARM template and nested template approach ‘cause they can get very complicated.

  Instead, I’d look at the second option here, which is the newer language for Azure, which is Bicep. And this has a built-in module approach that allows you to create specific modules and use them in your code. If you’re doing AWS and you’re using the native cloud formation language for that, then this has a concept of modules and you can use those to create reusable chunks of code and then deliver them to your users. If you’re using Pulumi, then these use the concept of component resources and it also packages, which is a newer feature, which allows you to create reusable component resources that can be used for any language. And if you’re using Terraform, then this has the concept of modules as well.

  So whichever of those languages, if you want to dive deeper into how to do it in those languages, then go have a look at that specific option within their documentation. So let’s go and take a quick look at what a module might look like. So, here in Visual Studio Code, we’ve got an example of a module. This one’s been created using the Bicep, just ‘cause it’s fairly easy to read and a fairly small sample, but obviously you can do this in any language. And what we’ve got is a module that’s used to create a storage account.

  Now, I guess the first question you might ask is why put this in a module, because the storage account is a fairly simple resource on its own anyway. What we’re using this here for is to apply some standards and some, I guess, specific ways we want storage accounts created. So the first thing we’ll do when we create a module is train interface that we want the user who’s gonna use it to pass in data and the data they’re gonna get out the other side. So here at the top, we’ve got the parameter settings for our module, which are the interface, and so we’re asking the users to pass in a prefix, which is gonna be used as part of our naming. We’re gonna ask them to pass in a specific skew of what type of storage account they want and a location they want.

  And you’ll notice on the storage skew setting there, we’ve got a list of aloud types. So this is our first sort of standard approach. We’re saying these are the allowed types that you were allowed to create in particularly, and this is an Azure module, you’re only allowed to create standard storage accounts and not premium storage accounts. So we’re saying we don’t want the users using those more expensive accounts when they use our module, they’re only allowed to use these standard accounts. Then we get into the actual code for the resource and it’s fairly straightforward, but you’ll note, we’ve got that variable that which is creating a name.

  And so again, we’re applying another standard and this one is a naming convention so we want to name all our resources in a certain way. And so we’re taking that prefix they passed in and then appending a unique string generated based on the resource group and that will give us a sort of set naming convention for this resource. Then in the resource itself, we’re using those values to actually create it and then you’ll note on line 24 there, we are setting HTTP traffic to on. So a user who’s created using this module cannot turn off HTTPS traffic, which is a good thing from a security perspective. But again, we’re setting our standard.

  And then we’ve got the output at the end, which is the other side of the interface where we pass this to the user of the module. When it’s running as completed, they will get the name of the actual storage endpoint, the URL for the storage account out the other end, which they can then pass into other resources they’re going to deploy or get to the command line and then use it manually. So we know why we want to create modules, we know how to great modules, but how do you create good modules? So firstly, you want your module to do one thing and do it as simple as possible. It can be quite tempting to go and build really big and complex modules that do lots of different things that are really fancy, but I’ve found the best approach is to make sure your module is focused in what it wants to do. So maybe it’s deploying a storage account like we saw with the standards on it, or maybe it is doing something more complicated.

  Maybe you’re deploying a whole Kubernetes cluster, but you’re sticking to the Kubernetes cluster and you’re not then reaching out to build the firewall that also is used by that Kubernetes cluster and so on. So keep them as simple as possible. You want to provide options to your users. You want to provide them parameters and ways to customize the module, but don’t give them too many options. If you try and replicate every single property that you can configure in inside of your module on the outside, yes, it’s gonna make it very flexible, but it’s gonna make it really confusing for a user, particularly somebody who’s new to using the module or even new to infrastructure as code as a concept.

  You want to try and drive the user down at the particular approach you want to give, but give them enough options to be able to customize it the way they want and not have to go and, you know, not use your module about the code themselves because they can’t run it the way they want to. So find the balance there. Your module needs to add value. There is really no point in taking a native resource and wrapping a module around it while you don’t do anything else. If you’re not using it to either make something simpler, to apply a standard or to deploy multiple resources or so on, you’re just wrapping the native provider in your module and adding no value, there’s no point to your module.

  Modules are software packages and so you need to make sure you’re versioning them appropriately for your end users. If it’s just you using it, then maybe that’s not quite so important, but if you’ll have other team members, you need to make sure that you’re, you know, you’re giving it a version numbering approach that you update the version when you release a new version and that maybe you’re following something like or similar so that people know when there’s a breaking change and so on. The next point’s a really important one is you need to make your modules easy to obtain. If you can’t get hold of them, it’s a real pain and they’re not gonna use them. You need to make the friction to using the modules very small and so you need to use the tools that are available and we’ll talk about that on the next slide to distribute those modules to your end users so they can get them really easily.

  And alongside that, keep them updated. No one’s gonna use your modules if they’re all four versions out of date, they don’t use the latest features and people can’t deploy the latest resources with it. So make sure you keep them updated, automate some of the process if you need to, but make sure that they’re up to date. When you’re building a module, start simple and then evolve it and get more complex over time. Trying to do everything at once in a big bang can be quite difficult, especially if you’re new to writing them or you’re working with a new resource that you’re not sure how to deploy.

  So start with the basics, the stuff you really need in the module, and then you can add to it over time like any good software project. Create documentation and make sure it’s good and available. People are gonna want to know how to use your modules. If it’s not self-explanatory from the code or, you know, it’s packaged in something like NewGit and you can’t really see the code, the people aren’t gonna use it less it’s good documentation on how to use it. And it may be you want to look at some sort of automation to automate document creation, things like doc with X and so on can be used to create automated API docs, but make sure there are docs and make sure they’re easy for people to find.

  And finally, if you want people to use your modules, then you need to make sure that they know that they exist, what benefits they’re gonna give them and work with people to help them get onboarded and use your modules. Again, not so required if it’s just for you, but I can pretty much guarantee if you work in a company, if you start writing modules and they’re useful, people are gonna want to use them, so make sure people know about them, engage with them and get feedback and maybe they will even help you build the next version and maybe even start contributing to your library, which is another great point. If you can make your library of modules open source, but inner source within your company, and you can get collaboration between lots of different people, you’ll find that the library will grow much quicker and you’ll get people a lot more engaged with building and maintaining these things. So I mentioned on the last slide, distributing your modules and making them easy to get hold of is really important so I just want to mention quickly the different strategies you’ve got for doing that. Again, this is more technology-dependent, depending on what type of language you’re using.

  So with both ARM templates and now Bicep on the Azure side, you can use this concept called template specs, which allow you to basically create a library of reusable code that you can then distribute to the end users by the Azure portal. So that’s one way of doing it, or if you need to keep them sort of more closed source and within your company, you can put them into storage accounts and have people have modules there, or even things like GitHub and so on. You can also use that as a storage place to get your modules, as long as it’s accessible over HTTP. For cloud formation, this comes with a cloud formation registry that you can use to store your models and distribute them to users. With Pulumi, because you’re writing actual code that you’re writing C-sharp or Node or Python, you can use the native delivery mechanisms are like NewGit, MPM, PyPy, all those sorts of things.

  And then with Terraform, you’ve got a couple of options. You can actually just use Git and reference modules directly in Git, or if you’ve got Terraform Enterprise, you can use a Terraform registry as well, which is a similar approach. So all of these provide you with a way of distributing your modules to end users so that the friction of getting on board and getting started is much lower. So lastly, I just want to go with some real world use cases that I’ve worked on, or I’ve used the provide really good examples of modules that are useful. So starting off something like a virtual machine.

  So the virtual machine is a good example of reducing complexity and applying standards. When you deploy a virtual machine in whatever cloud, it’s very rare that it’s just a virtual machine. It usually has some network pieces with it, maybe some storage, it’s gonna have a lot of configuration settings around security and monitoring, and those sorts of things. Maybe it’s gonna have some extensions to install other software and so on. You’re probably gonna have a standard way that you want VMs to be deployed, but also the code to deploy a VM can be pretty big.

  And so creating a module that deploys a VM in a standard way that everyone can reuse is gonna save people a lot of time and it’s gonna make sure that they follow the standards and you’re not gonna have a virtual machines configured with the RDP exposed to the internet or so on. So that’s a really good sort of example, and quite a good starting point for building a module. Another one that comes up a lot is Kubernetes clusters. And this is all really about reducing complexity. If you’re gonna deploy Kubernetes into one of the cloud providers, you’re gonna probably gonna use their native cloud solution, so AKS, EKS, Google, Kubernetes engine, those sorts of things, but often you’re not just deploying Kubernetes, you’re deploying all the prerequisites that you need, so some networking, some storage, the Kubernetes cluster, but also you’re gonna deploy some things on top of the Kubernetes cluster.

  Maybe you want to run an English control and maybe you need certain manager running. You can deploy lots of things after you Kubernetes cluster as well. And so again, that can be a really complicated set of resources. And so bundling that into a module for anyone who wants to create a Kubernetes cluster, particularly if people are spinning up Kubernetes clusters when they want to do testing and then tearing them down again, you can hide an awful lot of complexity within that module and give somebody a really simple way to deploy that. Particularly for people who aren’t familiar with how to deploy and manage Kubernetes, they just want to run an application, this can be a really useful use case.

  Another great example is maybe you’re on the networking team and you’re not working with the other teams to deploy their applications or anything, but you want to make sure everybody’s deploying their network, following the standards. Maybe you’ve got a sort of a hub and spoke network and you’ve deployed the hub and you need to make sure everybody joins their spokes appropriately, build a network module that you can then distribute to the various teams and that will show them how to connect to the hub without you having to explain it and document it and so on, you just give them this network module and everything is configured correctly, follow standards and is joined appropriately. Another one here is configuration module, and this might seem a bit odd, but it’s really useful tooling to have. So this is a module doesn’t actually create any resources, but what it has is configuration settings that you’re gonna use for other resources. It’s really common within companies to have a lot of different configuration options that are unique to your company, so things like maybe external IP addresses, naming conventions, those sorts of things where people need to remember to make sure they use those within their resources.

  So why not build a module that has all of those pre-configured so people can use that module. they can update it so if you can, if you add more external IPs, they can just take the latest version of the module and then they’ve got the latest IPs and then they use those in the rest of their resources, maybe even feeding them into other modules, but it gives you a central place to store your configuration. So that’s a brief introduction on why you might want to have a look at building a library of modules and how you can go about get started. I hope you found that useful and I hope that encourage you to reuse rather than repeat.