Pulumi Blog: Ringo De Smet

Pulumi's Declarative and Imperative Approach to IaC

Ringo De Smet — Fri, 23 Sep 2022 00:00:00 +0000

On a regular basis, articles and tweets pass by discussing whether some specific tool is imperative or declarative.

It’s no surprise that Pulumi is often the tool being debated. What if I tell you that Pulumi is imperative, declarative and imperative?

When we look at our Frequently Asked Questions, we read the following on the declarative versus imperative topic:

Pulumi is a declarative tool that uses imperative languages to define your end state. The language is used for authoring your program. It’s not used for talking to the cloud provider API.

It is good to know to refresh what declarative and imperative mean:

Both terms refer to how the user provides direction to the automation platform. With an imperative tool, you define the steps to execute in order to reach the desired solution. With a declarative tool, you define the desired state of the final solution, and the automation platform determines how to achieve that state. (Source)

The title of the article mentions imperative and declarative both. Pulumi leverages the best of both worlds into our product.

Pulumi tries to offer a solution where our customers are only limited by their imagination rather than the tool at hand. Let me use the Pulumi architecture to highlight why I mention imperative twice.

Pulumi Architecture

Here is the diagram from our How Pulumi Works page:

You code your infrastructure in your preferred programming language. When you are done coding, you run pulumi up and the Pulumi CLI starts the language host for your selected programming language, as well as the required providers. The interaction between these 3 parts of the architecture results in the actual creation or modification of your infrastructure.

An Imperative Part: The Language Host

Under the hood, the Pulumi CLI does a lot of things, but one of the first actions is starting the language runtime, which is configured in the Pulumi.yaml project file:

name: blog_code
description: Example for declarative blog post
runtime: python
main: blog_code/

Once the language runtime is up, it turns to the program file. Here is a small Python example:

from pulumi_aws import s3

bucket = s3.Bucket('bucket')

for i in range(10):
 s3.BucketObject(
 f'object-{i}',
 s3.BucketObjectArgs(
 bucket=bucket.id,
 key=str(i),
 )
 )

A Pulumi program models the to-be state of your infrastructure. If you read this program, you can find that we define 11 resources as our to-be infrastructure:

1 AWS S3 bucket
10 Objects in the bucket created in the previous step

The stack itself is also modelled as a resource, and it is the parent to all other resources.

$ pulumi up

Updating (<masked>/dev)

View Live: https://app.pulumi.com/<masked>/blog_code/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack blog_code-dev created
 + ├─ aws:s3:Bucket bucket created
 + ├─ aws:s3:BucketObject object-0 created
 + ├─ aws:s3:BucketObject object-1 created
 + ├─ aws:s3:BucketObject object-3 created
 + ├─ aws:s3:BucketObject object-2 created
 + ├─ aws:s3:BucketObject object-6 created
 + ├─ aws:s3:BucketObject object-4 created
 + ├─ aws:s3:BucketObject object-5 created
 + ├─ aws:s3:BucketObject object-7 created
 + ├─ aws:s3:BucketObject object-9 created
 + └─ aws:s3:BucketObject object-8 created

Resources:
 + 12 created

Duration: 13s

While this is definitely an imperative program, there is one important thing to understand: Instantiating an s3.Bucket, s3.BucketObject, or any other Pulumi resource should not be interpreted as an imperative creation of the resource in the language host. Behind the scenes, any resource instantiation in the language host triggers a Register Resource request to the Pulumi deployment engine. All these Register Resource requests together form the resource model you as an infrastructure developer want to get in the end.

Running your program always sends the full resource model to the Pulumi deployment engine regardless of what state your current infrastructure is in.

Our previous example was shown in Python, but recently we also delivered support for YAML. The creation of our S3 bucket could be converted to this snippet:

resources:
 bucket:
 type: aws:s3:Bucket

The separation of language support from the engine allows us to offer both imperative & declarative solutions. But remember that none of the supported languages do any of the provisioning directly in the language host.

If the resource provisioning is not taking place in the language host, where is the magic happening then?

The Declarative Part: The CLI and Engine

In the previous step, you found out that the language host sends requests to the engine to fulfill your to-be infrastructure.

It is now that the Pulumi deployment engine gets to work. The engine combines the intended model of the infrastructure received from the language host, the current state recorded in the state backend, and the actual resource state to compute which actions need to be executed to bring the actual state in line with the intended model.

Our little example contains dependencies: every s3.BucketObject uses the bucket.id as a way to define in which bucket these objects should be stored. The property id from the s3.Bucket is an Output. Outputs are Pulumi’s way of tracking which property of one resource is required by another, hereby creating a dependency between the resources. The engine uses all these outputs passed from one resource to another as the vertices in a directed acyclic graph (DAG). The engine determines the order of actions based on this graph.

On a first run of pulumi up of our example program, the engine will first create the bucket and wait for the provisioning to be complete, after which the actual bucket ID is passed to the creation of all the bucket objects. Since none of the bucket objects depend on other resources, these bucket objects can all be provisioned concurrently.

On a second run, assuming no modifications to our example program, the Pulumi deployment engine will compare the to-be model with the actual state and conclude that nothing needs to be done.

Let’s crank up the number of bucket objects to 11.

from pulumi_aws import s3

bucket = s3.Bucket('bucket')

for i in range(11): # <- number of objects increased by 1
 s3.BucketObject(
 f'object-{i}',
 s3.BucketObjectArgs(
 bucket=bucket.id,
 key=str(i),
 )
 )

The third Pulumi run will find the bucket and the first 10 objects in the state. The only action taken now is the creation of the 11th bucket object. This brings the actual state back in sync with your intended model.

$ pulumi up

Updating (<masked>/dev)

View Live: https://app.pulumi.com/<masked>/blog_code/dev/updates/2

 Type Name Status
 pulumi:pulumi:Stack blog_code-dev
 + └─ aws:s3:BucketObject object-10 created

Resources:
 + 1 created
 12 unchanged

Duration: 5s

Although you create the intended model of your infrastructure with an imperative language, the engine definitely processes this in a declarative way.

Another Imperative Part: The Providers

These examples hopefully made clear that the Pulumi deployment engine calculates a set of actions to declaratively bring the actual state in sync with your intended model. But the engine is not the component which knows how to talk to all the different APIs of cloud and tool vendors. That’s the role of the provider.

The engine and the provider processes are connected with a gRPC connection. This is similar to the connection between the language host and the engine, with the only difference of which requests are sent. The API that providers expose are of a CRUD nature:

Create: create a new resource
Read: read information of an existing resource
Update: update an existing resource with modified information
Delete: delete an existing resource when no longer needed

The providers receive requests from the engine dependent on the set of actions the engine calculated. A provider doesn’t know anything about the state, the correlation between resource, and so forth.

The nature of the provider API is clearly imperative.

Summary

If the question ever pops up again whether Pulumi is declarative or imperative, the answer is clearly we are both. It is only based on which component of our architecture you are talking about:

Language host: imperative (JS/TS, Go, Python, C#/F#/.NET, Java) and declarative (YAML)
Pulumi engine: declarative
Providers: imperative

To us, it mainly matters if we can solve your infrastructure automation problem. If you still miss something in our offering, we want to hear from you!

Refactoring Pulumi Code with `aliases`

Ringo De Smet — Thu, 26 Mar 2020 00:00:00 +0000

Guest Article: Ringo De Smet, Founder of Cumundi, standardizes on Pulumi for writing infrastructure as reusable code libraries for his customers. Pulumi enables him to rapidly iterate through the build-test-release cycle of these building blocks.

Cumundi helps companies adopt cloud infrastructure in a more integral way. It has found that 75% of companies’ needs are covered by ‘vanilla’ infrastructure patterns. However, due to a shortage of people and time, there has been limited investment to take full advantage of cloud-native configurations - which can lead to inefficiency, poor performance, and higher costs.

How Pulumi Code Keeps Up with Changing Requirements

At Cumundi, we build reusable libraries for our customers to set up their infrastructure while following best practices. These best practices include a range of “non-functional requirements” - infrastructure patterns - which often receive less focus than application features. Where feasible, Cumundi encapsulates these non-functional requirements into infrastructure code libraries. This enables Cumundi to serve clients using all major clouds while keeping application-focused infrastructure libraries clean and modular.

It may come as no surprise that we also use Pulumi to manage Cumundi’s own infrastructure. As a young company (founded January 1st!), it is hard to foresee how our infrastructure needs will evolve. With Pulumi, we write code in a modern programming language, and, good coding practice is not to optimize prematurely because the future is unpredictable. Another coding best practice is red-green-refactor or formally known as Test-Driven Development. You take the current code, write a test for the new requirement, which initially fails (red). Next, you implement the code in the most straightforward way to make the test succeed (green) and you complete the cycle by refactoring the code to keep the design in proper shape.

This blog post demonstrates a TDD cycle for Pulumi code with a reduced version of the code used to configure the internal infrastructure we provide for each customer. This example focuses on one specific Pulumi resource property: aliases.

All the code is available on Github if you want to follow along with a full project setup. Every step described here is committed as a separate branch with the starting point on master, the default branch.

The Starting Point

Every Pulumi project starts empty, so ours is no exception. This is how our main Pulumi code file looks for now.

import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config('gitlab')

Applying the above code does nothing, which you can see in the Resources tab on the hosted Pulumi platform:

Adding the Git Repository

We create a separate repository for each of our customers:

const config = new pulumi.Config('gitlab')

const gitlabNamespace = config.getNumber('namespace')

const firstCustomer = new gitlab.Project("FirstCustomer",
 {
 name: "first-customer",
 description: "First Customer code",
 namespaceId: gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 }
)

We wrote the code in the simplest way to get the job done. After pulumi up, the Gitlab project is created:

A Customer Needs Cloud Infrastructure

For our second customer, I just duplicated the creation of the Gitlab repository and created a Google Cloud project and service account.

const secondCustomer = new gitlab.Project("SecondCustomer",
 {
 ... // properties here
 }
)

const secondCustomerProject = new gcp.organizations.Project("SecondCustomer",
 {
 projectId: 'second-customer',
 name: 'Second Customer Infrastructure'
 }
)

const serviceAccountSecondCustomer = new gcp.serviceAccount.Account("ServiceAccountSecondCustomer",
 {
 accountId: 'secondcustomer',
 displayName: 'Service Account for Second Customer project',
 project: secondCustomerProject.projectId
 }
);

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 }
);

const secondCustomerGitlabCIVariable = new gitlab.ProjectVariable("SecondCustomerGCPAccess",
 {
 project: secondCustomer.id,
 variableType: 'file',
 key: "GOOGLE_APPLICATION_CREDENTIALS",
 value: serviceAccountSecondCustomerKey.privateKey.apply(value => Buffer.from(value, 'base64').toString('utf-8')),
 protected: true,
 environmentScope: "*"
 }
)

These resources are all added with the Stack as their parent in the resource view, as seen below. This is getting clumsy.

Visualization goes a long way, but a flat list of resources does not clearly show what belongs together.

Find Relationships between Resources

In the current state of the code, we created a Google Cloud service account and a key. Since a key can’t be created without a service account, we also created a Gitlab CI project variable for the Gitlab project of the second customer. Two cases of a parent-child relationship.

Although we have this relationship between our resources, the Pulumi state graph doesn’t display it. How can we change this without affecting the real resources on Gitlab and Google Cloud?

We can pass CustomResourceOptions as the last argument to every Pulumi resource that we want to create and use parent and aliases for refactoring.

To link the key to the service account, we set the parent property to the service account resource. If you run pulumi preview, Pulumi wants to recreate the key. It wants to do this because it searches for the key as a child resource of the service account. In your last applied Pulumi state, that is not the case.

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 },
 {
 parent: serviceAccountSecondCustomer
 }
);

How can we tell Pulumi that the existing key resource fulfills the expectation in our code? The aliases property comes to the rescue!

If you want to rename resources in a Pulumi state or change the parent-child relationships, then the aliases property is your friend. Let’s indicate in our code that the existing key resource, linked to the Stack, is the key of interest.

const serviceAccountSecondCustomerKey = new gcp.serviceAccount.Key("ServiceAccountSecondCustomerKey",
 {
 serviceAccountId: serviceAccountSecondCustomer.email
 },
 {
 parent: serviceAccountSecondCustomer,
 aliases: [
 { parent: pulumi.rootStackResource }
 ]
 }
);

Before going forward, we also changed the relationship between the Gitlab repository and the Gitlab CI variable. When you run pulumi up, nothing changes on Gitlab and Google Cloud. But after applying the changes, refreshing the resource view on the Pulumi platform displays the following state graph.

Clean Up Duplication

In the previous step, I copied the code for the Gitlab repository from our first customer to our second customer. In this step, we resolve the duplication issue, and ensure that customer projects can conditionally create the Google infrastructure needed.

I introduced a ComponentResource subclass, which encapsulates moving the code for the individual resources into the customer resource class. As we set up Google Cloud resources for other customers, we create a Google Cloud project and related resources only when required. We can use a regular Typescript if conditional for this.

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as gitlab from "@pulumi/gitlab";
import * as util from 'util';

export interface ProjectArgs {
 // The name of the customer, e.g. "First Customer"
 customer: pulumi.Input<string>;
 // Indication whether we need to create Google Cloud infrastructure for this customer
 needsGoogleInfra: boolean;
 // Namespace in Gitlab to create the repositories in.
 gitlabNamespace?: pulumi.Input<number> | undefined
}

// Pulumi custom resource representing a customer project
export class Project extends pulumi.ComponentResource {
 constructor(name: string, args: ProjectArgs, opts: pulumi.CustomResourceOptions = {}) {
 super('customer:Project', name, {}, opts);

 var customerId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase();
 var serviceId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1').toLowerCase();

 const gitlabProject = new gitlab.Project(name,
 {
 name: customerId,
 description: pulumi.interpolate`${args.customer} code`,
 namespaceId: args.gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 }
 )

 if (args.needsGoogleInfra) {
 ... // Google Cloud and Gitlab CI Variable code here.
 }
 }
}

Now that I have our custom resource, I can use it for our existing customers. Our refactored main Pulumi file looks like this:

const firstCustomer = new customer.Project("FirstCustomer",
 {
 customer: 'First Customer',
 needsGoogleInfra: false,
 gitlabNamespace: gitlabNamespace
 }
)

const secondCustomerProject = new customer.Project('SecondCustomer',
 {
 customer: 'Second Customer',
 needsGoogleInfra: true,
 gitlabNamespace: gitlabNamespace
 }
)

When we apply this to our infrastructure, two custom resources are created and saved to your Pulumi state.

However, we have not updated the parent-child relationships.

Fix the Remaining Relationships

We have a representation of a customer project in our Pulumi state graph, but we want to see the resources of each customer as child resources of this abstraction, right?

The same trick with parent and aliases is used to re-parent our resources. Our resources are created within the constructor of our Project class. In Typescript, the language used in this example, we can use the keyword this to point to the Project instance.

export class Project extends pulumi.ComponentResource {
 constructor(name: string, args: ProjectArgs, opts: pulumi.CustomResourceOptions = {}) {
 super('customer:Project', name, {}, opts);

 var customerId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase();
 var serviceId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1').toLowerCase();

 const gitlabProject = new gitlab.Project(name,
 {
 name: customerId,
 description: pulumi.interpolate`${args.customer} code`,
 namespaceId: args.gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 },
 {
 parent: this,
 aliases: [
 { parent: pulumi.rootStackResource }
 ]
 }
 )

While we are changing parent-child relationships, the service account resource is also updated to have the Google Cloud project as its parent. The result should look like the state graph below.

If another developer reads this code, they might not immediately have a clear picture of how everything is wired together. However, looking at the Pulumi resource visualization, the following properties can immediately be deduced:

our infrastructure has 2 customer projects
one customer only has a Gitlab repository, while the other also has Google Cloud infrastructure
if we create a Google Cloud project, we create a service account and a key together with it
if we create a Google Cloud project, we also set a Gitlab CI variable. It’s a pity though I can’t link the Gitlab CI variable to the key.

Next Steps

You probably noticed by now that the aliases property accepts a list. You can provide entries like:

{ name: <some old name> }
{ parent: <some previous parent> }
{ name: <some old name>, parent: <some previous parent> }

Note that putting the first two in the aliases list is not the same as putting the third one as the single entry. I leave it up to the reader to test it and understand the difference.

The take away from this example is: values for name in the aliases property can not be Inputs as they need to be resolvable at preview time, similar to the name of the resources themselves.

Conclusion

The Pulumi programming model well designed with regards to the evolution of an infrastructure. With ComponentResource and the parent and aliases properties, refactoring your infrastructure code is a breeze with visualizing abstractions on the Pulumi platform.