Pulumi Blog: Refactoring

How We Used Pulumi to Safely Migrate Oso's Global Infrastructure

Shaun Verch — Mon, 05 May 2025 00:00:00 +0000

Infrastructure as Code (IaC) tools such as Pulumi can provide enormous amounts of leverage, but they must be used correctly to also provide safety. One of our main jobs as infrastructure engineers is to not break things, so leverage without safety is useless. If something is safe, we can change things easily without even thinking about it. If it isn’t, we’ll be up until 2 a.m. fixing what we broke.

At Oso, we recently had to do a large infrastructure migration and learned three key principles that make a huge difference in using Pulumi for safe migrations. First, understand how it’s even possible to safely change your infrastructure independently of Pulumi. Second, write reusable and predictable Pulumi code. Finally, refactor your Pulumi code to your new version using zero-diff refactors.

Osoʼs Migration

Oso provides Authorization as a Service, so every application decision must first query Oso. This means that if Oso goes down, our customers will also go down. We can’t afford a moment of downtime.

However, we’re also growing quickly, and one of the side effects of growth is that architectural decisions that worked for us in the past quickly reach their expiration date. One example of this is when we hit a scaling limit and needed to quickly migrate from Amazon ECS Fargate to ECS on our own self-managed EC2 instances.

We can’t overstate the Pandora’s box of complexity this opened up for us. From the lifecycle of our data to our deployment orchestration, this touched nearly every part of our stack. On top of this, because performance and data locality are critical for us, we are deployed globally in 12 different regions. We need to be extremely careful to do this correctly.

How the Migration Actually Worked

The most important thing to realize when using an infrastructure tool like Pulumi is that ultimately you’re dealing with real resources. It doesn’t matter how cleverly you refactor your Pulumi code; if something gets deleted that isn’t supposed to be deleted, or if at any point a critical component is missing, there will be an outage.

At Oso, we were already using a Route 53 Traffic Policy to do latency-based per-region routing, so Route 53 was a convenient tool for us in this migration. The plan was to create an entirely new Oso infrastructure and include it in our traffic policy, but send 0% of our traffic to it. Then, we could run tests ourselves directly against the new infrastructure and gradually ramp up traffic until we were fully migrated.

That is the first big lesson we took away from this. If we didn’t have a point in our infrastructure where we could easily reroute customer traffic from one place to the other, this transition would have been impossible without downtime.

Writing Reusable Pulumi Code

The holy grail of infrastructure development is to achieve what we already have in application development: code reuse. Besides the benefit of reducing the burden of maintaining more code, it also reduces risk because we can concentrate our testing in one place and benefit from using code that has already been proven.

In Osoʼs case, because our migration involved creating a copy of our entire infrastructure with the new changes, code reuse was vital. Even the smallest reusable component meant fewer new bugs and less time fixing bugs we had already fixed in the old infrastructure.

In Pulumi, you can use the import and function system of the underlying programming language, so that part of managing libraries is not an issue. The real challenge lies in how to actually write the business logic of libraries in Pulumi because writing Pulumi code is not like writing normal code.

Take a look at this example of creating an ECS service:


let service = new aws.ecs.Serice("api",{ });

This line of code might seem trivial, but a lot is happening here, and it’s important to understand Pulumi’s programming model on a deeper level to know how to work with it properly.

The first thing to note is the first argument: api. This is the logical resource name in Pulumi. The subsequent arguments are more about how to actually create the resource, and we wonʼt talk about that here. This logical resource is how Pulumi identifies this resource in the internal state.

The internal state is the core of the Pulumi engine. It maps the “logical resource name" we provide to the actual resource that Pulumi either created or plans to create. The code example above, at some point deep inside the Pulumi engine, turns into something that roughly looks like this:

{
 "aws:ecs:Service": {
 "api": {...},
 ...
 },
 ...
}

This is the key to understanding why writing Pulumi code is not like writing normal code. There is one “state namespace” per Pulumi execution, which effectively means that there is one global object that we are adding resources to. So the new aws.ecs.Service function call wasn’t just creating a service object that we can pass around; it was registering the service with the global list of resources we plan to deploy.

After we understand that calls to the Pulumi library have side effects on this global state, we can start to write better Pulumi libraries. For example, say we have this function call:

deployWebStack("app",true,false,"enabled");

Do you know which resources this will create? Is it safe to call deployWebStack again with the same arguments? What would happen if we deleted this line of code? These are all the familiar perils of global state, and something we must proactively contend with to write a good Pulumi library.

Here are some rules that we recommend:

DO use Pulumi Component Resources in libraries that create resources.
- This encourages a more declarative object-based approach to resources, which fits better with Pulumi’s model and also causes these objects to show up in the Pulumi output.
Do NOT conditionally create resources inside library functions.
- If the function is called, all the resources should be created, and if it is not, the resources should not be created. Conditionals should be at the top level, because that’s where you are deciding what is deployed in a particular region or component.
Do NOT check the current infrastructure state inside library functions.
- This should be passed in as arguments; otherwise, the library function will have non-deterministic behavior.
Do NOT attempt to implement idempotent resource creation functions.
- See the previous rule. The only way to implement a function that creates a resource only if it hasn’t been created yet is to check if it has been created, leading to non-determinism.
DO write functional code as inputs to resources.
- You can still write normal code in some cases, as long as it doesn’t call into Pulumi. For example, the code that generates the list of environment variables for an app can be a normal function.
DO have a strategy for resource naming.
- Because any function might register new resources, and because you may even have naming conflicts on the cloud provider side, having a coherent strategy for naming resources is helpful. We don’t have anything prescriptive here, because it’s really a matter of preferences and specific situations.
DO favor assigning resources to named objects and variables over getter functions.
- Rather than implementing a function or some kind of “getter” to refer to another resource, save that resource object in a variable at some scope where it can be referenced by the other resources that need it. This can even be in the global scope to match the reality that all the resources end up in one resource namespace in the end.

There’s a lot of advice here, but it really all comes down to keeping in mind what Pulumi is actually doing with your code and remembering that calls into the Pulumi library are actually registering resources in the global state.

Infrastructure Snapshot Testing for Zero Diff Refactors

Now that we know what we want to change in our infrastructure, and we know how to write our Pulumi code in a reusable way, we’re faced with a familiar software engineering challenge: How can we safely refactor our existing code to get to where we want to be? To answer this question, we can step back from the Pulumi specifics for a moment and think about refactoring in general.

One method that is amazing for refactoring code is Snapshot Testing. With Snapshot Testing, you run the legacy code that you want to change, save the output, and create a unit test around that. Then, when you make a code change, you can detect if your code changes result in any unintended changes in behavior.

We can imitate this with Pulumi by taking advantage of the fact that we can see a preview of the changes Pulumi will make before we deploy them. While we don’t have direct access to Pulumi’s internal data model in the code itself to make this “snapshot,” the preview can give us this comparison. If we make changes to our infrastructure, as long as pulumi preview shows no changes, we are 100% sure that the refactor we made is safe and won’t cause any issues. Because of this, we at Oso internally call this method a “zero diff refactor.”

This is a relatively simple concept, but it is necessary to make this migration possible. In our original infrastructure, most of our resources were deployed in one monolithic Pulumi project, and resources were created at the top level rather than in libraries. The 12-region deployment also made running Pulumi on this project very slow, and we wanted a clean separation between the old and new infrastructure. Using snapshot testing to refactor allowed us to build and reuse our in-house infrastructure libraries with 100% confidence and let us focus on the parts that held the real risk.

Conclusion

This migration did not go smoothly. We constantly found problems and unexpected assumptions that our code was making about the underlying infrastructure.

But no one outside Oso noticed. We had a clear migration plan, factored out common Pulumi code, and used zero-diff refactors. Putting this all together meant we could benefit from the leverage that Pulumi has the potential to provide. This allowed us to make major changes with confidence that they wouldn’t break our system.

In the early days of Oso, people were often surprised by how much we were able to accomplish with such a small core team of engineers. Part of the secret to this is using the tools we have available to gain maximum leverage, but this isn’t the entire story. Knowing how to use them safely allows us to actually benefit from that leverage in practice.

If you’re interested in exploring Pulumi further, here are several ways to get involved:

Discover how Pulumi Crosswalk for AWS simplifies “day one” tasks in our AWS guide
Register for one of our upcoming workshops
But most importantly, try Pulumi today!

Refactoring Infrastructure as Code

Sophia Parafina — Tue, 01 Jun 2021 00:00:00 +0000

The central principle of cloud engineering is adopting software engineering practices. Refactoring is a technique for making changes to code that improve maintainability, enhance performance, scalability, and security without changing its external behavior. In devops, refactoring often occurs with modern applications; however, we can apply those same techniques to cloud infrastructure with infrastructure as code.

Refactoring results in many advantages. First and foremost, the code is more readable and easier to understand for other team members –this aids in maintainability and well-organized code, providing a solid foundation for future releases. Overall, if done well, refactoring reduces complexity which makes future changes more efficient.

The topic of refactoring is a broad topic covering many different techniques. Many references, such as Martin Fowler’s Refactoring, cover refactoring in depth. In this article, we examine standard practices used with infrastructure as code.

Infrastructure is different

A major difference between refactoring application and infrastructure code is that resources, such as databases persist. Typically, if you change the name, type, or parent path of a resource, it is deleted and a new resource is created. Other infrastructure as code solutions tie a file or module to a resource’s identity. An advantage of Pulumi is that moving a resource across file boundaries does not recreate the resource because a resource’s identity is based on its object ID, irrespective of its file location.

In cases where an object’s ID changes – due to a rename, reparenting to a new component, or otherwise – Pulumi provides aliases. An Alias accepts a list of identifiers that can include old Alias objects or old resource URNs. When a resource is annotated with an alias, Pulumi can connect the old to the new object state, and the resource can be updated in place.

A similar situation occurs with component resources, i.e., a logical grouping of related resources such as VPC with subnets pre-configured. When you change a parent resource of a component resource, the identity of the component resource also changes. Pulumi provides a parent option to tie a child resource to the parent resource. The combination of alias and parent arguments to resources lets you refactor infrastructure while preserving existing resources.

Let’s examine how to apply refactoring to infrastructure.

Test-Driven Development

Test-driven development (TDD) is a common technique used for code refactoring. TDD breaks down refactoring into three steps:

Red: consider the required behavior and what needs to be coded
Green: get the code to pass a test for that behavior
Refactor: review the code for improvements and implement as needed.

In a previous blog post, Ringo de Smeti used TDD to refactor Cumundi’s code for provisioning repositories by customer. Ringo created a ComponentResource subclass which moved individual resources into a customer resource class.

// Pulumi custom resource representing a customer project
export class Project extends pulumi.ComponentResource {
 constructor(name: string, args: ProjectArgs, opts: pulumi.CustomResourceOptions = {}) {
 super('customer:Project', name, {}, opts);

 var customerId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase();
 var serviceId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1').toLowerCase();

 const gitlabProject = new gitlab.Project(name,
 {
 name: customerId,
 description: pulumi.interpolate`${args.customer} code`,
 namespaceId: args.gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 }
 )

Calling the Project class creates a new customer.

const firstCustomer = new customer.Project("FirstCustomer",
 {
 customer: 'First Customer',
 needsGoogleInfra: false,
 gitlabNamespace: gitlabNamespace
 }
)

const secondCustomerProject = new customer.Project('SecondCustomer',
 {
 customer: 'Second Customer',
 needsGoogleInfra: true,
 gitlabNamespace: gitlabNamespace
 }
)

By using TDD, Ringo reduced code duplication, resulting in a clean and efficient codebase.

Refactoring by Abstraction: Pull-Up/Push-Down

The Pull-Up/Push-Down method is used frequently to reduce duplicate code. It does this by abstracting methods into classes to implement inheritance and extraction. The Pull-Up method uses a superclass to consolidate similar methods, whereas Push-Down moves methods from a superclass into subclasses.

The Pull-Up method in this example deploys the Kubernetes Guest Book. The simple version is a port of the original YAML and the component version, shown below, creates a service class that creates both the service and deployment.

export class ServiceDeployment extends pulumi.ComponentResource {
 public readonly deployment: k8s.apps.v1.Deployment;
 public readonly service: k8s.core.v1.Service;
 public readonly ipAddress?: pulumi.Output<string>;

 constructor(name: string, args: ServiceDeploymentArgs, opts?: pulumi.ComponentResourceOptions) {
 super("k8sjs:service:ServiceDeployment", name, {}, opts);

 const labels = { app: name };
 const container: k8stypes.core.v1.Container = {
 name,
 image: args.image,
 resources: args.resources || { requests: { cpu: "100m", memory: "100Mi" } },
 env: [{ name: "GET_HOSTS_FROM", value: "dns" }],
 ports: args.ports && args.ports.map(p => ({ containerPort: p })),
 };
 this.deployment = new k8s.apps.v1.Deployment(name, {
 spec: {
 selector: { matchLabels: labels },
 replicas: args.replicas || 1,
 template: {
 metadata: { labels: labels },
 spec: { containers: [ container ] },
 },
 },
 }, { parent: this });

 this.service = new k8s.core.v1.Service(name, {
 metadata: {
 name: name,
 labels: this.deployment.metadata.labels,
 },
 spec: {
 ports: args.ports && args.ports.map(p => ({ port: p, targetPort: p })),
 selector: this.deployment.spec.template.metadata.labels,
 // Minikube does not implement services of type `LoadBalancer`; require the user to specify if we're
 // running on minikube, and if so, create only services of type ClusterIP.
 type: args.allocateIpAddress ? (args.isMinikube ? "ClusterIP" : "LoadBalancer") : undefined,
 },
 }, { parent: this });

 if (args.allocateIpAddress) {
 this.ipAddress = args.isMinikube ?
 this.service.spec.clusterIP :
 this.service.status.loadBalancer.ingress[0].ip;
 }
 }
}

With infrastructure as code, you can use your IDE to refactor.

Composing

Composing is another refactoring method for reducing duplicate code. The most common way to use composing is through extraction and inline methods.

Extraction consolidates code by creating a new method and replacing duplicate methods with a call to the new method. For example, when we create services, we often export information about the service, such as IP address or hostname, as shown in the example below.

export const publicIp = server.publicIp;
export const publicHostName = server.publicDns;

We can create a function that exports the information about a service and call it as needed.

function public_server_info(server) {
 export const publicIp = server.publicIp;
export const publicHostName = server.publicDns;
}

public_server_info(server);

Inline refactoring reduces unnecessary methods by finding all calls to the method and replacing them with the content of the method. In this policy as code example, we can refactor two methods into a single method.

export function requireEbsVolumesOnEc2Instances(name: string): ResourceValidationPolicy {
 // TODO: Check if EBS volumes are marked for deletion.
 return {
 name: name,
 description: "EBS volumes should be attached to all EC2 instances",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.ec2.Instance, (instance, args, reportViolation) => {
 if (instance.ebsBlockDevices !== undefined) {
 reportViolation("EC2 Instance should have EBS volumes attached.");
 }
 if (instance.ebsBlockDevices.length === 0) {
 reportViolation("EC2 Instance should have EBS volumes attached.");
 }

 }),
 };
}

The function can be simplified to the following:

export function requireEbsVolumesOnEc2Instances(name: string): ResourceValidationPolicy {
 // TODO: Check if EBS volumes are marked for deletion.
 return {
 name: name,
 description: "EBS volumes should be attached to all EC2 instances",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.ec2.Instance, (instance, args, reportViolation) => {
 if (instance.ebsBlockDevices !== undefined && instance.ebsBlockDevices.length === 0) {
 reportViolation("EC2 Instance should have EBS volumes attached.");
 }
 }),
 };
}

Simplifying Method Calls

Simplifying method calls makes them easier to understand by clarifying the logic for interaction between classes. One way to simplify code is to add, remove, or introduce new parameters with explicit methods and method calls.

Roles in AWS are typically stored as JSON documents. In the example below, we create the role by passing in a string.

lambda_role = iam.Role('lambdaRole',
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "lambda.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }
 ]
 }"""
)

sfn_role = iam.Role('sfnRole',
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "Service": "states.%s.amazonaws.com"
 },
 "Action": "sts:AssumeRole"
 }
 ]
 }""" % config.region
)

You can write a function called create_role and pass in the name and policy as JSON to create many roles.

const lambda_role_document = JSON.parse('{"Version": "2012-10-17","Statement": [{"Action": "sts:AssumeRole","Principal": {"Service": "lambda.amazonaws.com"},"Effect": "Allow","Sid": ""}]}')

function create_role(role :string, policy :JSON) {
 const newrole = new aws.iam.Role(role,{
 assumeRolePolicy: JSON.stringify(policy),
 })
 return role;
}

const lambdaRole = create_role("my_lambdarole",lambda_role_document);

Moving Features

Moving features creates new classes and moves functionality between old and new classes. A utility class may not have some methods you need, but you can’t add those methods to that class. The solution is to create a new class with the methods you need.

In this example, we extend a class and add new methods that configure an EC2 instance post-provisioning.

export class Provisioner<T, U> extends pulumi.dynamic.Resource {
 dep: pulumi.Output<T>;
 result: pulumi.Output<U>;
 constructor(name: string, props: ProvisionerProperties<T, U>, opts?: pulumi.CustomResourceOptions) {
 const provider: pulumi.dynamic.ResourceProvider = {
 diff: async (id: pulumi.ID, olds: State<T, U>, news: State<T, U>) => {
 let replace = false;
 let replacementProperties = [];
 if (JSON.stringify(olds.dep) !== JSON.stringify(news.dep)) {
 replace = true;
 replacementProperties.push("dep");
 }
 // Only trigger replacement due to the `changeToken` property, IFF
 // the changeToken still has a value in the new inputs, and it doesn't
 // match with the old value. If, say, the user decides to no longer specify
 // the changeToken in the new inputs, then we don't trigger a replacement.
 if (news.changeToken && olds.changeToken !== news.changeToken) {
 replace = true;
 replacementProperties.push("changeToken");
 }
 return {
 changes: replace,
 replaces: replace ? replacementProperties : undefined,
 deleteBeforeReplace: true,
 };
 },
 create: async (inputs: State<T, U>) => {
 const result = await props.onCreate(inputs.dep);
 if (result !== undefined) {
 inputs.result = result;
 }
 return { id: uuid.v4(), outs: inputs };
 },
 };
 super(provider, name, { dep: props.dep, changeToken: props.changeToken, result: null }, opts);
 }
}

Preparatory Refactoring

Preparatory refactoring occurs when adding a new feature to a release. It is typically part of a software update instead of a separate refactoring process. Preparatory refactoring reduces future technical debt.

Conclusion

Writing clean and efficient code is part of cloud engineering, and we can achieve this by using well-established refactoring methods used by application developers. Pulumi has many easy-to-understand examples in Node.js, Python, Go, and .Net. Because the examples are written explicitly to make them understandable, some could use refactoring. What code samples can you find in the Pulumi Github repository that could use some refactoring?

Refactoring Pulumi Code with `aliases`

Ringo De Smet — Thu, 26 Mar 2020 00:00:00 +0000

Guest Article: Ringo De Smet, Founder of Cumundi, standardizes on Pulumi for writing infrastructure as reusable code libraries for his customers. Pulumi enables him to rapidly iterate through the build-test-release cycle of these building blocks.

Cumundi helps companies adopt cloud infrastructure in a more integral way. It has found that 75% of companies’ needs are covered by ‘vanilla’ infrastructure patterns. However, due to a shortage of people and time, there has been limited investment to take full advantage of cloud-native configurations - which can lead to inefficiency, poor performance, and higher costs.

How Pulumi Code Keeps Up with Changing Requirements

At Cumundi, we build reusable libraries for our customers to set up their infrastructure while following best practices. These best practices include a range of “non-functional requirements” - infrastructure patterns - which often receive less focus than application features. Where feasible, Cumundi encapsulates these non-functional requirements into infrastructure code libraries. This enables Cumundi to serve clients using all major clouds while keeping application-focused infrastructure libraries clean and modular.

It may come as no surprise that we also use Pulumi to manage Cumundi’s own infrastructure. As a young company (founded January 1st!), it is hard to foresee how our infrastructure needs will evolve. With Pulumi, we write code in a modern programming language, and, good coding practice is not to optimize prematurely because the future is unpredictable. Another coding best practice is red-green-refactor or formally known as Test-Driven Development. You take the current code, write a test for the new requirement, which initially fails (red). Next, you implement the code in the most straightforward way to make the test succeed (green) and you complete the cycle by refactoring the code to keep the design in proper shape.

This blog post demonstrates a TDD cycle for Pulumi code with a reduced version of the code used to configure the internal infrastructure we provide for each customer. This example focuses on one specific Pulumi resource property: aliases.

All the code is available on Github if you want to follow along with a full project setup. Every step described here is committed as a separate branch with the starting point on master, the default branch.

The Starting Point

Every Pulumi project starts empty, so ours is no exception. This is how our main Pulumi code file looks for now.

import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config('gitlab')

Applying the above code does nothing, which you can see in the Resources tab on the hosted Pulumi platform:

Adding the Git Repository

We create a separate repository for each of our customers:

const config = new pulumi.Config('gitlab')

const gitlabNamespace = config.getNumber('namespace')

const firstCustomer = new gitlab.Project("FirstCustomer",
 {
 name: "first-customer",
 description: "First Customer code",
 namespaceId: gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 }
)

We wrote the code in the simplest way to get the job done. After pulumi up, the Gitlab project is created:

A Customer Needs Cloud Infrastructure

For our second customer, I just duplicated the creation of the Gitlab repository and created a Google Cloud project and service account.

const secondCustomer = new gitlab.Project("SecondCustomer",
 {
 ... // properties here
 }
)

const secondCustomerProject = new gcp.organizations.Project("SecondCustomer",
 {
 projectId: 'second-customer',
 name: 'Second Customer Infrastructure'
 }
)

const serviceAccountSecondCustomer = new gcp.serviceAccount.Account("ServiceAccountSecondCustomer",
 {
 accountId: 'secondcustomer',
 displayName: 'Service Account for Second Customer project',
 project: secondCustomerProject.projectId
 }
);

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 }
);

const secondCustomerGitlabCIVariable = new gitlab.ProjectVariable("SecondCustomerGCPAccess",
 {
 project: secondCustomer.id,
 variableType: 'file',
 key: "GOOGLE_APPLICATION_CREDENTIALS",
 value: serviceAccountSecondCustomerKey.privateKey.apply(value => Buffer.from(value, 'base64').toString('utf-8')),
 protected: true,
 environmentScope: "*"
 }
)

These resources are all added with the Stack as their parent in the resource view, as seen below. This is getting clumsy.

Visualization goes a long way, but a flat list of resources does not clearly show what belongs together.

Find Relationships between Resources

In the current state of the code, we created a Google Cloud service account and a key. Since a key can’t be created without a service account, we also created a Gitlab CI project variable for the Gitlab project of the second customer. Two cases of a parent-child relationship.

Although we have this relationship between our resources, the Pulumi state graph doesn’t display it. How can we change this without affecting the real resources on Gitlab and Google Cloud?

We can pass CustomResourceOptions as the last argument to every Pulumi resource that we want to create and use parent and aliases for refactoring.

To link the key to the service account, we set the parent property to the service account resource. If you run pulumi preview, Pulumi wants to recreate the key. It wants to do this because it searches for the key as a child resource of the service account. In your last applied Pulumi state, that is not the case.

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 },
 {
 parent: serviceAccountSecondCustomer
 }
);

How can we tell Pulumi that the existing key resource fulfills the expectation in our code? The aliases property comes to the rescue!

If you want to rename resources in a Pulumi state or change the parent-child relationships, then the aliases property is your friend. Let’s indicate in our code that the existing key resource, linked to the Stack, is the key of interest.

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 },
 {
 parent: serviceAccountSecondCustomer,
 aliases: [
 { parent: pulumi.rootStackResource }
 ]
 }
);

Before going forward, we also changed the relationship between the Gitlab repository and the Gitlab CI variable. When you run pulumi up, nothing changes on Gitlab and Google Cloud. But after applying the changes, refreshing the resource view on the Pulumi platform displays the following state graph.

Clean Up Duplication

In the previous step, I copied the code for the Gitlab repository from our first customer to our second customer. In this step, we resolve the duplication issue, and ensure that customer projects can conditionally create the Google infrastructure needed.

I introduced a ComponentResource subclass, which encapsulates moving the code for the individual resources into the customer resource class. As we set up Google Cloud resources for other customers, we create a Google Cloud project and related resources only when required. We can use a regular Typescript if conditional for this.

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as gitlab from "@pulumi/gitlab";
import * as util from 'util';

export interface ProjectArgs {
 // The name of the customer, e.g. "First Customer"
 customer: pulumi.Input<string>;
 // Indication whether we need to create Google Cloud infrastructure for this customer
 needsGoogleInfra: boolean;
 // Namespace in Gitlab to create the repositories in.
 gitlabNamespace?: pulumi.Input<number> | undefined
}

// Pulumi custom resource representing a customer project
export class Project extends pulumi.ComponentResource {
 constructor(name: string, args: ProjectArgs, opts: pulumi.CustomResourceOptions = {}) {
 super('customer:Project', name, {}, opts);

 var customerId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase();
 var serviceId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1').toLowerCase();

 const gitlabProject = new gitlab.Project(name,
 {
 name: customerId,
 description: pulumi.interpolate`${args.customer} code`,
 namespaceId: args.gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 }
 )

 if (args.needsGoogleInfra) {
 ... // Google Cloud and Gitlab CI Variable code here.
 }
 }
}

Now that I have our custom resource, I can use it for our existing customers. Our refactored main Pulumi file looks like this:

const firstCustomer = new customer.Project("FirstCustomer",
 {
 customer: 'First Customer',
 needsGoogleInfra: false,
 gitlabNamespace: gitlabNamespace
 }
)

const secondCustomerProject = new customer.Project('SecondCustomer',
 {
 customer: 'Second Customer',
 needsGoogleInfra: true,
 gitlabNamespace: gitlabNamespace
 }
)

When we apply this to our infrastructure, two custom resources are created and saved to your Pulumi state.

However, we have not updated the parent-child relationships.

Fix the Remaining Relationships

We have a representation of a customer project in our Pulumi state graph, but we want to see the resources of each customer as child resources of this abstraction, right?

The same trick with parent and aliases is used to re-parent our resources. Our resources are created within the constructor of our Project class. In Typescript, the language used in this example, we can use the keyword this to point to the Project instance.

export class Project extends pulumi.ComponentResource {
 constructor(name: string, args: ProjectArgs, opts: pulumi.CustomResourceOptions = {}) {
 super('customer:Project', name, {}, opts);

 var customerId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase();
 var serviceId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1').toLowerCase();

 const gitlabProject = new gitlab.Project(name,
 {
 name: customerId,
 description: pulumi.interpolate`${args.customer} code`,
 namespaceId: args.gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 },
 {
 parent: this,
 aliases: [
 { parent: pulumi.rootStackResource }
 ]
 }
 )

While we are changing parent-child relationships, the service account resource is also updated to have the Google Cloud project as its parent. The result should look like the state graph below.

If another developer reads this code, they might not immediately have a clear picture of how everything is wired together. However, looking at the Pulumi resource visualization, the following properties can immediately be deduced:

our infrastructure has 2 customer projects
one customer only has a Gitlab repository, while the other also has Google Cloud infrastructure
if we create a Google Cloud project, we create a service account and a key together with it
if we create a Google Cloud project, we also set a Gitlab CI variable. It’s a pity though I can’t link the Gitlab CI variable to the key.

Next Steps

You probably noticed by now that the aliases property accepts a list. You can provide entries like:

{ name: <some old name> }
{ parent: <some previous parent> }
{ name: <some old name>, parent: <some previous parent> }

Note that putting the first two in the aliases list is not the same as putting the third one as the single entry. I leave it up to the reader to test it and understand the difference.

The take away from this example is: values for name in the aliases property can not be Inputs as they need to be resolvable at preview time, similar to the name of the resources themselves.

Conclusion

The Pulumi programming model well designed with regards to the evolution of an infrastructure. With ComponentResource and the parent and aliases properties, refactoring your infrastructure code is a breeze with visualizing abstractions on the Pulumi platform.