Pulumi Blog: Next level iac

Next-level IaC: Bridging the Declarative Gap

Troy Howard — Fri, 07 Jun 2024 00:00:00 +0000

Pulumi stands out in the world of infrastructure-as-code (IaC) for its flexibility and ease of use. This is due to its unique approach of using general purpose programming languages to describe system configuration. It accomplishes this through a small but brilliant implementation detail that bridges the worlds of declarative and standard programming using asynchronous semantics.

Since this post was published, Pulumi has added first-class support for HashiCorp Configuration Language (HCL). You can now write Pulumi programs in HCL directly, alongside general-purpose languages like TypeScript, Python, Go, and C#. To see how it works, see Pulumi HCL.

In Pulumi, you describe your infrastructure in code – real code, not a DSL or YAML – using your preferred general purpose programming language. You don’t have to become a specialist in a niche proprietary declarative language, like Terraform’s HCL. Rather, you can reuse your existing programming skills, writing in standard imperative, object-oriented, and even functional language styles, while still gaining all of the benefits of the declarative style that other tools emphasize.

Pulumi provides a unique mix of a declarative model embedded and implemented inside of a standard programming language, allowing all the flexibility of custom imperative code, while still enabling Pulumi’s deployment engine to infer opportunities for parallel asynchronous execution and to converge a partially-realized system.

The secret sauce that lets Pulumi achieve this is the apply function, and the way it handles inputs and outputs via asynchronous programming constructs, similar to futures or promises. We’ll go deeper on that in this article, but first let’s talk a little bit about the flow of operations in Pulumi.

Pulumi’s architecture and flow of operations

Pulumi is a system that is made up of a number of components. At a high level, those are:

Pulumi Programs: The code you write to describe the infrastructure you need.
Deployment Engine: A process that executes requests to create/modify infrastructure.
Resource Providers: Plugins that interface with cloud services like AWS, Azure, and others.

These components interact together to manage your infrastructure across multiple cloud providers using a consistent programming and execution model.

Figure: Pulumi’s Program, Deployment Engine, and Resource Provider architecture

Let’s get into an example of how information flows back and forth between these components. Suppose you want to write a Pulumi program to create an AWS S3 Bucket to host a static website. When that bucket is created, it will be given a unique identifier that you need to use in other parts of your infrastructure. Every time you create the bucket the identifier will change, and it’s generated by the AWS API, so we can’t know it ahead of time. We need to handle this as a variable, not a fixed configuration value.

To do that in a Pulumi program, you can create a bucket and then refer to its id property later, like this:

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';

// Create an S3 bucket
const bucket = new aws.s3.Bucket("bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

import pulumi
import pulumi_aws as aws

# Create an S3 bucket
bucket = aws.s3.Bucket('bucket')

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

When this program runs, the call to create the bucket object will issue a request to the deployment engine. The deployment engine will load the resource provider for AWS, and ask it to create the S3 bucket. The AWS resource provider then issues the request to the AWS API, which will create the bucket and then return the id of the newly created resource. The provider will return that to the deployment engine, which will then set the realized value on your Pulumi program’s bucket.id property and then move on to creating anything that depends on that property.

This all seems pretty straight-forward, except that Pulumi programs don’t wait and imperatively do each operation, one at a time. Instead the deployment engine will run them in parallel, only waiting in cases where the output of one operation is needed to run the next operation. This can create complex structures with some operations needing to wait for inputs from other operations, and some that don’t, all finishing at different times, depending on how long the cloud service takes to process that request.

Inputs and Outputs as asynchronous futures

Infrastructure automation tasks can be long-running and can result in intermittent errors and may need to be retried before moving on to the next step. A robust IaC solution will need to run concurrently, be able to restart, retry, and resume itself, and understand that a set of tasks could happen in any order, not necessarily in the order you defined them in code.

Back in our Pulumi program code, that means that some of the time the bucket object from our earlier example has its id property set, and sometimes it doesn’t. We wouldn’t want to run any operation that needed that variable before it was set, but we also wouldn’t want to wait to start operations that don’t need that value.

To address this, Pulumi models the input and output values as futures. This allows you to describe a complex set of needs using a familiar value-like syntax. Instead of using a string result directly, which would only exist some of the time, you instead work with an Output<string>, that Pulumi understands as a reference to a value that currently doesn’t exist, but will exist when the operation completes. Similarly, Input<string> describes a dependency on an input string that will at some point be provided as the result of another operation.

This notion of future completion is essential to understanding why apply blocks are necessary within Pulumi programs. The apply function is called by the deployment engine only when the operation has completed and the fully realized value is available. By using Input<T> and Output<T> to build the connections between different parts of your infrastructure, the code is leveraging asynchronous value semantics not just to wrap and provide access to a future value, but also to describe a relationship between two components. This allows Pulumi programs to be very natural to author and as readable as traditional code, while also enabling Pulumi’s deployment engine to execute the entirety of your graph of operations in parallel, only waiting when necessary. This maximizes the efficiency of interacting with potentially slow and unreliable external cloud services.

Realizing values using the `apply` function

An important distinction though, is that Input<T>/Output<T> values are just declarative placeholders for the values that will eventually be present. At a certain point you will need to access the values directly, for example in the context of a transformation. The output of one operation might produce a string, which you need to transform into another data structure before passing to the next operation. Pulumi really shines in this context!

Using the apply function we can pass an Input<T> or Output<T> to a transformation function which will accept the realized value as its input (and only run when that value is available). Within that function you can handle the string as a normal string, then return it back as an Output<string>. This allows you to use the full power of your programming language to perform a custom operation inline with the rest of your infrastructure automation.

Here’s a simple example of using apply to transform an output to something new using your programming language’s libraries. In this case, Base64 encoding the id of our bucket object.

Example: Using `apply` to Base64 encode an Output

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';

// Create an S3 bucket
const bucket = new aws.s3.Bucket("bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

// Export the name of the bucket, in Base64 encoding
export const bucketNameBase64 = bucket.id.apply(id=> {
 // Transform the bucket name into a Base64 encoded string
 Buffer.from(id).toString('base64')
})

import base64
import pulumi
import pulumi_aws as aws

# Create an S3 bucket
bucket = aws.s3.Bucket('bucket')

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

# A Base64 encoder function to use in the apply function's lambda
def base64_encode(data):
 data_bytes = data.encode("ascii")
 base64_bytes = base64.b64encode(data_bytes)
 base64_string = base64_bytes.decode("ascii")

# Export the name of the bucket, in Base64 encoding
pulumi.export('bucket_name', bucket.id.apply(lambda data: base64_encode))

Our example shows how to do Base64 encoding, but this technique can be used for any kind of transformation function. These can be as elaborate as your use case requires, and leverage the full power of your programming language and its entire ecosystem. Doing custom transformations like this in other popular tools is often not possible. This leaves you stuck using external coordination (e.g. gluing things together with Bash scripts) which breaks their declarative-only model (because you cannot describe everything you need to do in it) and interrupts these tools’ ability to fully understand your infrastructure’s dependency graph.

While Pulumi’s asynchronous semantics may seem somewhat complex for basic use cases, they unlock abilities in more complex use cases, which is when you really need your IaC tooling to show up for you. Because real life is never as simple as demo code.

Next steps

If you haven’t already, install Pulumi today, and follow our self-directed Getting Started guides to learn more about making the most of Pulumi’s next-level infrastructure management features at your organization.

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

The Pulumi Cloud is a fully managed service that helps you adopt Pulumi’s open source SDK with ease. It provides built-in state and secrets management, integrates with source control and CI/CD, and offers a web console and API that make it easier to visualize and manage infrastructure. It is free for individual use, with features available for teams.

Create an Account

Next-level IaC: Breakpoint Debugging for Pulumi Programs

Troy Howard — Fri, 31 May 2024 00:00:00 +0000

For many of us, life is suspended between presses of the F5 key. Our IDE is the only place where everything is organized and makes sense. And while we know that the likelihood of a code hole-in-one is very rare (that magical moment where you run your build and everything just works the first time), we always hope that this time, this will be the run where you see the output that lets you know you’ve gotten it right. And in between? Debugging.

Debugging is a complicated topic. But what is not complicated is understanding the cost of debugging. Debugging is the biggest unknown-unknown in software. It makes you miss your sprint goals, it holds back scaling and company growth, it slows your time to market. Debugging is what makes estimating so difficult, because until you know what the problem is, you don’t know how long it will take to fix, or indeed if it is even possible to fix at all. Which is why savvy developers invest a lot of time into refining their debugging process and seek out the best-in-class tools to streamline this process.

In the infrastructure world, debugging is a pretty big problem. Some of the most popular tools have nothing more than simple unstructured text output logs. While tools like Terraform call themselves “Infrastructure as Code” the “code” is a proprietary DSL, not a general purpose programming language, and so there is little-to-no IDE support. There is no F5. Even worse, there is no F9 for setting a breakpoint. There’s no way to inspect state mid-run, or more importantly to explore in an unstructured way, because this is where the “aha!” moments usually happen.

Pulumi gives you a rich IaC debugging experience by enabling breakpoint debugging in your favorite IDE. Let’s walk through an example of how to enable a full-featured debugging workflow in VS Code for TypeScript.

How Breakpoint Debugging Works

Using a debugger can feel like that moment in The Matrix where Neo first sees the green streaming code raining down. A debugger lifts the veil from your otherwise opaque executable code. Tools like VS Code provide a user experience where you set breakpoints, i.e. certain lines of code that are designated as points where you want to pause execution and inspect state. Hitting the F9 key in VS Code sets a breakpoint on the current line of code. Later, you run your code in debug mode. This is a special execution state where the underlying runtime (e.g. Node’s V8) exposes a TCP/IP port hosting a debugging protocol, and when run, it will pause execution until something attaches to it (i.e. connects to the port to start the debugging session).

At this point a debugging client (such as VS Code) will control the flow of the program, moving forward step-by-step, one line of code at a time. But how does it know which line of code it’s executing? When you run in debug mode, the binary code is instrumented with metadata about the line of code it represents. The debugger will tell the runtime, via the debugging protocol, “run until you reach line 109 in file data.ts, then pause and wait for a signal from me before continuing”.

While paused, the protocol also allows you to inspect the values of any variables that are in scope at that moment. A more sophisticated debugger will provide a REPL environment where you can execute ad-hoc code in the scope of the line where you are paused. This can be very powerful to inspect the values of various objects and to see in real-time what the effects of executing variations of code would be. After a debugging session you can take what you’ve learned and edit the code accordingly.

Example: Attaching a debugger to a TypeScript Pulumi program in VS Code

For this example, we will set up a new Pulumi project from the command-line and then set up VS Code to run that program in a debugger.

Step 1: Create a Pulumi project from a template

For this example, we will create a new Pulumi project using the TypeScript AWS static website template. Make sure you’ve set up Pulumi, logged into Pulumi Cloud, and configured your AWS credentials. Follow the prompts in the pulumi CLI to configure your project.

$ mkdir good-morning && cd good-morning
$ pulumi new static-website-aws-typescript

Then launch this project to make sure it works as expected:

$ pulumi up
...

Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/troy-pulumi-corp/good-morning/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack good-morning-dev created (200s)
 + ├─ aws:s3:Bucket bucket created (1s)
 + ├─ aws:s3:BucketPublicAccessBlock public-access-block created (0.57s)
 + ├─ aws:s3:BucketOwnershipControls ownership-controls created (1s)
 + ├─ aws:cloudfront:Distribution cdn created (196s)
 + └─ synced-folder:index:S3BucketFolder bucket-folder created (0.41s)
 + ├─ aws:s3:BucketObject error.html created (0.46s)
 + └─ aws:s3:BucketObject index.html created (0.46s)

Outputs:
 cdnHostname : "##############.cloudfront.net"
 cdnURL : "https://##############.cloudfront.net"
 originHostname: "bucket-#######.s3-website-us-west-2.amazonaws.com"
 originURL : "http://bucket-#######.s3-website-us-west-2.amazonaws.com"

Resources:
 + 8 created

Great! You should be able to navigate to the website using the URL from the cdnURL output value.

$ open $(pulumi stack output cdnURL)

If all looks good, let’s move on to the next step by opening your project directory in VS Code:

$ code .

Step 2: Lets break stuff!

Now that we have a perfectly running Pulumi program, in order to have something to debug, we will need to break it!

For this example, we are going to change our indexDocument from index.html to good-morning.html.

$ cp ./wwww/index.html ./www/good-morning.html

Next, let’s edit the contents to say “Good morning, world!” instead of “Hello, world!”:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="utf-8">
 <title>Good morning, world!</title>
</head>
<body>
 <h1>Good morning, world! 👋</h1>
 <p>Deployed with 💜 by <a href="https://pulumi.com/">Pulumi</a>.</p>
</body>
</html>

Once that’s edited and saved, go ahead let’s redeploy the website:

$ pulumi up

Everything runs successfully… but if we navigate to the website, we don’t see our updated content?! Oh no! What went wrong? Why isn’t there an error message?! Time to put on our debugging hats and figure out what is going on. First, let’s set up VS Code to enable breakpoint debugging right in the IDE.

Step 3: Create a debug wrapper script

VS Code can run an arbitrary command when you hit F5. Typically this would be a one-liner that runs your language runtime like node foo.js or python foo.py. In our case, we need to run the pulumi CLI and need to do more than one step. The easiest way to do this is to create a small wrapper script and call that from VS Code.

In the root of the directory, create a file called pulumi-debug.sh:

#!/bin/bash

pulumi login
pulumi stack select dev
pulumi up -f

Let’s walk through these lines to explain what’s going on here:

pulumi login to set up credentials for the pulumi backend
pulumi stack select dev to run this against our dev stack
pulumi up -f to run our program update. The -f flag skips the preview phase and also skips the need to interactively confirm that we want to run the update.

Open the VS Code terminal panel and first set it to be executable, then try running the script:

$ chmod +x pulumi-debug.sh
$ ./pulumi-debug.sh

Ok now let’s set up VS Code to run that in debug mode when we hit F5!

Step 4: Configure VS Code `launch.json` and `tasks.json`

To run our Pulumi program in debug mode we will need to create a couple of VS Code configuration files, launch.json and tasks.json, in a hidden directory called .vscode within your project.

Create the .vscode directory and then create a new file called tasks.json file in that directory, with the following contents:

{
 "version": "2.0.0",
 "tasks": [
 {
 "label": "pulumi-debug",
 "type": "shell",
 "command": ". ./pulumi-debug.sh",
 "isBackground": true,
 "options": {
 "cwd": "${workspaceFolder}"
 }
 }
 ]
}

This creates a VS Code task named pulumi-debug which will run the wrapper script we just created from the root of our project.

Next, create a file named launch.json in the same directory. This will configure the behavior of the F5 key!

{
 "version": "0.2.0",
 "configurations": [
 {
 "name": "Launch Pulumi Program (debug)",
 "type": "node",
 "port": 9292,
 "request": "attach",
 "preLaunchTask": "pulumi-debug",
 "continueOnAttach": true,
 "restart": {
 "delay": 5000,
 "maxAttempts": 5
 },
 "skipFiles": [
 "${workspaceFolder}/node_modules/**/*.js",
 "${workspaceFolder}/lib/**/*.js",
 "<node_internals>/**/*.js"
 ]
 }
 ]
}

Let’s break this down a bit:

type: node tells VS Code that we are using the Node runtime
port: 9292 tells VS Code to attach the debugger to port 9292
request: attach tells VS Code that we want to attach to the debugger at launch
preLaunchTask: pulumi-debug tells VS Code to run the task we configured in tasks.json before we connect the debugger. This will run our pulumi-debug.sh script and then attach to the Node.js runtime
continueOnAttach: true tells VS Code to start running our code as soon as we attach to the waiting Node.js runtime (until we reach our first breakpoint that is, where it will pause and return control to the debugger client, aka VS Code)
restart: {...} configures VS Code to wait 5 seconds and then retry connecting, up to 5 times. This is here in case it takes a while for the Node.js runtime to start running. It’s unlikely, but possible!
skipFiles: {...} sets up some filters of files not to try to debug. This includes everything that is part of node’s internals, 3rdparty node modules, and other libraries. That way we don’t end up deep in someone else’s code unexpectedly and can just focus on our Pulumi program.
name sets a human-readable name which shows up in the drop down box in VS Code when you choose which [launch configuration][vs-code-launch-configs] to run.

Finally, we need to make one more edit to the pulumi-debug.sh file we created. Here we will pass the necessary options to the Node.js runtime to set it to run in debug mode.

Add the following line to the beginning of the file:

#!/bin/bash

export NODE_OPTIONS='--inspect-brk=127.0.0.1:9292'

pulumi login
pulumi stack select dev
pulumi up -f

The NODE_OPTIONS environment variable passes options to the Node.js V8 runtime. In this case, we are passing --inspect-brk which tells the Node.js runtime to run in debug mode, which will cause it to pause before running the code, until a debugging client connects to localhost on port 9292.

Ok, now we should be ready to run our Pulumi program from VS Code using the F5 key! Let’s give it a try. You should see Pulumi running inside of the “Terminal” tab. You will also see output indicating that it is waiting for a debugger to attach:

Debugger listening on ws://127.0.0.1:9292/########-####-####-####-############
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
Waiting for the debugger to disconnect...

Step 5: Setting a debugging breakpoint

Now we can try setting a breakpoint in our code. Getting back to the problem at hand – our content doesn’t seem to be updating, despite no errors. Let’s start at the beginning, by investigating the inputs to our program, aka our configuration… at runtime!

Navigate to line 12 within the code and hit the F9 key. You’ll see a small red dot appear next to the line indicating that a breakpoint has been set. This line defines the bucket our program creates and seems like a good place to stop if we want to check the input configuration before we run the AWS S3 provider call.

Figure: A breakpoint set on line 12.

You can then run the program again using F5. When the execution of the code reaches the line it will pause there. At this point you can navigate around the code inspecting the state of the local variables, like you would in any other debugging session.

Let’s inspect the configuration variables. On the left side of the screen you’ll find a “Variables” panel listing all the local variables that are in scope.

Figure: Local variables displayed in the Variables side panel.

There’s also the “Debug” console which provides a REPL environment to navigate those variables and anything else that is in scope. You can run arbitrary functions here and see their output. In this case we can simply type the variable name at the prompt there, hit enter, and see the value. What’s great about the debug console is that Intellisense (e.g. using TAB to auto-complete) works here, so you can quickly navigate possible options.

Figure: Debug console tab completion and inspecting variable values.

In our case, we’re interested in the value of indexDocument variable. We can see that it is getting set to index.html… not good-morning.html. The Pulumi program didn’t error because index.html still exists. So we are still publishing our old index.html. That makes sense. A quick fix here is to modify index.ts and set the default to good-morning.html:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as synced_folder from "@pulumi/synced-folder";

// Import the program's configuration settings.
const config = new pulumi.Config();
const path = config.get("path") || "./www";
const indexDocument = config.get("indexDocument") || "good-morning.html";
const errorDocument = config.get("errorDocument") || "error.html";

...

Ok, we edited the document, which fixes the program for future runs, but for the current run that we are debugging, indexDocument is still set to index.html. Let’s use the debug console to set the value to good-morning.html before we continue the run.

> indexDocument="good-morning.html`

The output in the debug window will show the new value, and if you hover over the inputs being passed to the bucket creation call, it will now show as good-morning.html:

Figure: Screenshots of debug console and editor window after changing the indexDocument variable

When you’re ready for the program to continue on, hit F5 once again and the program will resume operation, either until the program exits or until it reaches the next breakpoint. Optionally you can step through line-by-line using the F10 key. VS Code provides a variety of debug actions once the session is established.

Most importantly, check the published website and make sure we fixed the issue! You can find the cdnURL output in the Terminal tab and click on it from there to open the link.

Considerations and Limitations

Breakpoint debugging is a powerful tool for inspecting the state of your Pulumi program. However, there are some important limitations to consider:

Pulumi programs are only part of the story: This technique will let you inspect the program you authored, but it doesn’t give access to what’s happening with the Pulumi backend, the providers, or anything else outside of the program itself. You might find yourself needing to dig deeper to resolve a problem with the stack as a whole. Luckily there are other techniques we can use for that, which we will delve into in an upcoming post.
Inputs and Outputs are asynchronous futures: If you’ve used Pulumi before you will have encountered Input<T> and Output<T> values, which are a core function of the tool. However, at the time that the debugger hits a breakpoint on those values, they are likely not populated with their final value. That means you won’t be able to inspect them in the debugger very well. Instead, you will need to add an apply(...) or all(..) block to the code, and place the breakpoint on the materialized value within that block. This follows all the same rules as accessing those variables in code in a Pulumi program.
```
bucket.id.apply(id => {
 // NOTE: The `debugger` keyword automatically sets a breakpoint here
 // so you don't need to set it via the IDE.
 debugger
});
```
Limited integration with VS Code: Unfortunately our technique of running Pulumi in wrapper script as a backgrounded pre-launch task in VS Code leaves some things to be desired. It is different enough from the use case that VS Code had in mind for these features (e.g. interacting with a long-running tsc-watch or similar task) that making things “just work” is a little difficult. For example, we can’t easily report on errors in the “Problems” tab, and VS Code complains that a problem matcher is missing the first time you run it. VS Code can’t tell that the program finished and so you’ll have to manually stop the debugging session after Pulumi has completed (or wait for the retry/timeout to expire).

Despite these limitations, Pulumi with VS Code as your IDE provides a rich experience compared to trawlling through unstructured logs, or adding in console.log(...) calls to echo out ad-hoc debugging information (aka. printf debugging). Also, you can enable this for any language that Pulumi supports, and most popular IDEs support breakpoint debugging (e.g. Visual Studio, Webstorm, neovim, etc). Python, Go, and C# all have popular debuggers and IDEs that can be used in the same manner. Whatever your preferred IDE or language is, you can use their full range of breakpoint debugging features for your Pulumi programs.

Pulumi also offers detailed logging which can be used to look into the other aspects that breakpoint debugging doesn’t cover. In an upcoming article we’ll show how to use these tools as well. Until then, have fun spelunking through your code with your favorite IDE!

Next Steps

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

Next-level IaC: How Pulumi Supports Your API Economy Strategy

Engin Diri — Wed, 15 May 2024 00:00:00 +0000

When I am talking with community members, who are not using Pulumi yet, I often get asked what would be a good way to include their Infrastructure as Code (IaC) into existing software like a REST API. And my answer is always the same: Use our Pulumi Automation API.

With the Pulumi Automation API you can include Pulumi IaC into your existing software, and this for any of the Pulumi supported programming languages like TypeScript, Python, Go, or C#. This gives you a greater flexibility and control, which you will not have with other IaC tools like CloudFormation or Terraform.

This opens up many doors of opportunity to expose your services as an API, whether internally to your organization or externally to your potential customers. This opportunity for innovation fuels what is known as the API Economy.

What is API Economy?

As a matter of fact: — APIs enable innovation.

Over 40% of large organizations have 250+ APIs. And 71% plan to use even more APIs this year.

API Economy is a term that is used to describe the way that organizations are using APIs to create new business models and revenue streams. APIs are the building blocks of modern software development, and they allow developers to build on top of existing services and create new services that are greater than the sum of their parts.

Major international corporations leveraging the API economy through robust API strategies include Google, Amazon, Stripe, Twilio, Slack, and many other renowned brands. Google’s initial Maps service, for instance, has evolved into a comprehensive platform offering Maps, Routes, and Places APIs and SDKs. These tools enable companies to incorporate both dynamic and static maps into their applications and websites, offer navigational guidance, access 360° Street View images, utilize geolocation services, and more. This transformation exemplifies the concept of value multiplication in the API economy, as described by Gartner Vice President Kristin Moyer:

“The API economy is an enabler for turning a business or organisation into a platform. Platforms multiply value creation because they enable business ecosystems inside and outside of the enterprise to consummate matches among users and facilitate the creation and exchange of goods, services and social currency so that all participants are able to capture value.”

What is Pulumi Automation API in a nutshell?

Pulumi Automation API is a set of SDKs that allow you to interact with your Pulumi stacks programmatically. This means that you can use the Automation API to create, update, and delete stacks, as well as to query their current state. The Automation API is built on top of the Pulumi engine, which means that it is fully compatible with all of the existing Pulumi providers and resources.

And it is this capability that allows you to build your own Service API on top of Pulumi. We can now build a REST or gRPC API endpoint that, depending on the request parameters, will create, update, or delete a Pulumi stack. This API can then be exposed to the rest of the organization, allowing developers to interact with their infrastructure in a programmatic way.

How to build your own Service API with Pulumi Automation API?

Let’s take a look at how you can build your own Service API with Pulumi Automation API.

For this purpose, we will build two basic REST services written in TypeScript and in Python.

import express from "express"
import * as dotevnv from "dotenv"
import cors from "cors"
import helmet from "helmet"
import {fullyQualifiedStackName, RemoteWorkspace} from "@pulumi/pulumi/automation";

dotevnv.config()

if (!process.env.PORT) {
 console.log(`No port value specified...`)
}

const PORT = parseInt(process.env.PORT as string, 10)

const app = express()

app.use(express.json())
app.use(express.urlencoded({extended: true}))
app.use(cors())
app.use(helmet())

// Define the payload interface
interface ServicePayload {
 name: string;
 org: string;
 project: string;
 stack: string;
 region: string;
}

// Define a helper function to set up the Pulumi program
async function setUpPulumiProgram(payload: ServicePayload) {
 const org = payload.org;
 const project = payload.project;
 const stackName = fullyQualifiedStackName(org, project, payload.stack);
 const awsRegion = payload.region;

 // Using the Pulumi Automation API to create or select a stack in the remote workspace
 // and set the stack's configuration and secrets. This will use Pulumi Deployments to
 // deploy the stack to the specified AWS region.
 return await RemoteWorkspace.createOrSelectStack({
 stackName,
 url: "https://github.com/pulumi/examples.git",
 branch: "refs/heads/master",
 projectPath: project,
 }, {
 envVars: {
 AWS_REGION: awsRegion,
 AWS_ACCESS_KEY_ID: process.env.AWS_ACCESS_KEY_ID ?? "",
 AWS_SECRET_ACCESS_KEY: {secret: process.env.AWS_SECRET_ACCESS_KEY ?? ""},
 AWS_SESSION_TOKEN: {secret: process.env.AWS_SESSION_TOKEN ?? ""},
 },
 });
}

// Define the API endpoint for creating a new stack in the remote workspace.
app.post('/', async (req, res) => {
 const payload: ServicePayload = req.body;

 if ((payload.name) || !payload.org || !payload.project || !payload.stack || !payload.region) {
 res.status(400).send('Name, org, project, stack and region are required');
 return;
 }
 // Create a new stack in the remote workspace and set the stack's configuration and secrets.
 const stack = await setUpPulumiProgram(payload);

 // Deploy the stack to the specified AWS region. We'll print the stack's outputs to the console.
 const upRes = await stack.up({onOutput: console.log});
 res.status(200).send(`url: ${upRes.outputs.websiteUrl.value}`);
});

// Define the API endpoint for destroying a stack in the remote workspace.
app.delete('/', async (req, res) => {
 const {name, org, project, stack, region} = req.query;

 if (!name || !org || !project || !stack || !region) {
 res.status(400).send('Name, org, project, stack, and region are required');
 return;
 }

 const payload: ServicePayload = {
 name: name as string,
 org: org as string,
 project: project as string,
 stack: stack as string,
 region: region as string
 }

 // Define the stack and set the stack's configuration and secrets.
 const prog = await setUpPulumiProgram(payload);
 // Destroy the stack in the remote workspace. We'll print the stack's outputs to the console.
 await prog.destroy({onOutput: console.log});
 res.status(200).send('Stack destroyed');
});


app.listen(PORT, () => {
 console.log(`Server is listening on port ${PORT}`)
})

import os
import pulumi.automation as auto
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


# Define the payload that the API will receive. This payload will contain the name of the stack
# to create, the organization and project to use, the stack name, and the AWS region to deploy the stack to.
class ServicePayload(BaseModel):
 name: str
 org: str
 project: str
 stack: str
 region: str


# Define a helper function that will create a new stack in the remote workspace and set the stack's configuration
def setup_pulumi_program(payload: ServicePayload):
 org = payload.org
 project = payload.project
 stack_name = auto.fully_qualified_stack_name(org, project, payload.stack)
 aws_region = payload.region

 # Using the Pulumi Automation API to create or select a stack in the remote workspace
 # and set the stack's configuration and secrets. This will use Pulumi Deployments to
 # deploy the stack to the specified AWS region.
 return auto.create_or_select_remote_stack_git_source(
 stack_name=stack_name,
 url="https://github.com/pulumi/examples.git",
 branch="refs/heads/master",
 project_path=project,
 opts=auto.RemoteWorkspaceOptions(
 env_vars={
 "AWS_REGION": aws_region,
 "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
 "AWS_SECRET_ACCESS_KEY": auto.Secret(os.environ["AWS_SECRET_ACCESS_KEY"]),
 "AWS_SESSION_TOKEN": auto.Secret(os.environ["AWS_SESSION_TOKEN"]),
 },
 ),
 )


# Define the API endpoints for creating and deleting a stack in the remote workspace.
@app.post("/")
async def create(payload: ServicePayload):
 # Create a new stack in the remote workspace and set the stack's configuration and secrets.
 stack = setup_pulumi_program(payload)
 # Deploy the stack to the specified AWS region. We'll print the stack's outputs to the console.
 up_res = stack.up(on_output=print)
 return {"url": up_res.outputs['websiteUrl'].value}


# Define the API endpoint for deleting a stack in the remote workspace.
@app.delete("/")
async def delete(name: str, org: str, project: str, stack: str, region: str):
 payload = ServicePayload(name=name, org=org, project=project, stack=stack, region=region)
 # Define the stack and set the stack's configuration and secrets.
 stack = setup_pulumi_program(payload)
 # Destroy the stack in the remote workspace. We'll print the stack's outputs to the console.
 stack.destroy(on_output=print)
 return {"message": "Stack destroyed"}

Let us run the above Python example in a FastAPI server to see how the Automation API works in practice.

Let’s start by creating a new Python file called main.py and pasting the above code into it. Next, we need to install the required dependencies:

python3 -m venv ven
source venv/bin/activate

Create a new file called requirements.txt and paste the following dependencies into it:

pulumi
fastapi
pydantic

We need to install the dependencies using the following command:

pip install -r requirements.txt --force-reinstall

Finally, we can run the FastAPI server using the following command:

fastapi dev main.py

This will start the FastAPI server on port 8000.

INFO Using path main.py
INFO Resolved absolute path /private/tmp/automation-api-example/python/main.py
INFO Searching for package file structure from directories with __init__.py files
INFO Importing from /private/tmp/automation-api-example/python

 ╭─ Python module file ─╮
 │ │
 │ 🐍 main.py │
 │ │
 ╰──────────────────────╯

INFO Importing module main
INFO Found importable FastAPI app

 ╭─ Importable FastAPI app ─╮
 │ │
 │ from main import app │
 │ │
 ╰──────────────────────────╯

INFO Using import string main:app

 ╭────────── FastAPI CLI - Development mode ───────────╮
 │ │
 │ Serving at: http://127.0.0.1:8000 │
 │ │
 │ API docs: http://127.0.0.1:8000/docs │
 │ │
 │ Running in development mode, for production use: │
 │ │
 │ fastapi run │
 │ │
 ╰─────────────────────────────────────────────────────╯

INFO: Will watch for changes in these directories: ['/private/tmp/automation-api-example/python']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [55849] using WatchFiles
INFO: Started server process [55851]
INFO: Waiting for application startup.
INFO: Application startup complete.

To test the REST services, you can use a tool like curl or Postman. Here is an example of how you can create a new stack using curl:

curl -X POST -H "Content-Type: application/json" -d '{"name": "myexample", "org": "myorg", "project": "myproject", "stack": "mystack", "region": "us-west-2"}' http://localhost:3000

In the logs of the FastAPI server, you should see following output from the Automation API:

INFO: Create stack
Preparing deployment...

View Live: https://app.pulumi.com/dirien/aws-ts-s3-folder/dev/deployments/3

Setup:
Preparing environment
Pulling container image "pulumi/pulumi:3.115.2"
3.115.2: Pulling from pulumi/pulumi
Digest: sha256:a7b9022d7c1a40d81adeca30a21eb1cd49add474e3de923ade663db046e7ee1a
Status: Downloaded newer image for pulumi/pulumi:3.115.2
Starting container
Container created (6b92928bdcdfe6e54286dcdcb622480004f18c98b23b5b60775f71959a423708)
Container started (6b92928bdcdfe6e54286dcdcb622480004f18c98b23b5b60775f71959a423708)

Download deployment executor:
$ curl --retry 5 -H "Authorization: token $PULUMI_ACCESS_TOKEN" "$PULUMI_SERVICE_URL/deployments/executor" -o /pulumi-deploy-executor.gz && gzip -d /pulumi-deploy-executor.gz && chmod +x /pulumi-deploy-executor
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 15.9M 0 15.9M 0 0 42.2M 0 --:--:-- --:--:-- --:--:-- 42.4M

Get source:
$ /pulumi-deploy-executor getsource git --repoURL="https://github.com/pulumi/examples.git" --workDir="/deployment" --branch="refs/heads/master" --repoDir="aws-ts-s3-folder"
Successfully cloned: https://github.com/pulumi/examples.git

Download dependencies:
$ /pulumi-deploy-executor dependencies download --workDir="/deployment/aws-ts-s3-folder"
npm WARN deprecated @opentelemetry/api-metrics@0.32.0: Please use @opentelemetry/api >= 1.3.0

added 294 packages, and audited 295 packages in 14s

43 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
npm notice
npm notice New major version of npm available! 9.6.7 -> 10.7.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.7.0>
npm notice Run `npm install -g npm@10.7.0` to update!
npm notice

pulumi update:
$ /pulumi-deploy-executor pulumi update --stackIdentity="dirien/aws-ts-s3-folder/dev" --workDir="/deployment/aws-ts-s3-folder"
Updating (dev)

View Live: https://app.pulumi.com/dirien/aws-ts-s3-folder/dev/updates/3


 + pulumi:pulumi:Stack aws-ts-s3-folder-dev creating (0s)
 + aws:s3:Bucket s3-website-bucket creating (0s)
 + aws:s3:Bucket s3-website-bucket created (4s)
 + aws:s3:BucketObject favicon.png creating (0s)
 + aws:s3:BucketObject index.html creating (0s)
 + aws:s3:BucketPublicAccessBlock public-access-block creating (0s)
 + aws:s3:BucketObject index.html created (0.98s)
 + aws:s3:BucketObject favicon.png created (1s)
 + aws:s3:BucketPublicAccessBlock public-access-block created (1s)
 + aws:s3:BucketPolicy bucketPolicy creating (0s)
 + aws:s3:BucketPolicy bucketPolicy created (0.83s)
 + pulumi:pulumi:Stack aws-ts-s3-folder-dev created (18s)
Outputs:
 bucketName: "s3-website-bucket-d55e9bd"
 websiteUrl: "s3-website-bucket-d55e9bd.s3-website.eu-central-1.amazonaws.com"

Resources:
 + 6 created

Duration: 20s

Result: succeeded
INFO: 127.0.0.1:49220 - "POST / HTTP/1.1" 200 OK

You should see the URL of the created infrastructure in the response from the FastAPI server. If you head over to the Pulumi console, you should see the created stack and the resources that were created and also that Pulumi Deployments were used to deploy the stack to the specified AWS region.

And here is an example of how you can delete a stack using curl:

curl -X DELETE "http://localhost:8000/?name=myexample&org=myorg&project=myproject&stack=mystack&region=us-west-2"

In the logs of the FastAPI server, you should see the following output from the Automation API:

INFO: Deleting stack
Preparing deployment...

View Live: https://app.pulumi.com/dirien/aws-ts-s3-folder/dev/deployments/4

Setup:
Preparing environment
Pulling container image "pulumi/pulumi:3.115.2"
3.115.2: Pulling from pulumi/pulumi
Digest: sha256:a7b9022d7c1a40d81adeca30a21eb1cd49add474e3de923ade663db046e7ee1a
Status: Downloaded newer image for pulumi/pulumi:3.115.2
Starting container
Container created (26af871d4e26a6dab5656a1c8dc8a7e87bf247f42bde77e2269c269f15054e73)
Container started (26af871d4e26a6dab5656a1c8dc8a7e87bf247f42bde77e2269c269f15054e73)

Download deployment executor:
$ curl --retry 5 -H "Authorization: token $PULUMI_ACCESS_TOKEN" "$PULUMI_SERVICE_URL/deployments/executor" -o /pulumi-deploy-executor.gz && gzip -d /pulumi-deploy-executor.gz && chmod +x /pulumi-deploy-executor
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 15.9M 0 15.9M 0 0 43.5M 0 --:--:-- --:--:-- --:--:-- 43.5M

Get source:
$ /pulumi-deploy-executor getsource git --repoURL="https://github.com/pulumi/examples.git" --workDir="/deployment" --branch="refs/heads/master" --repoDir="aws-ts-s3-folder"
Successfully cloned: https://github.com/pulumi/examples.git

Download dependencies:
$ /pulumi-deploy-executor dependencies download --workDir="/deployment/aws-ts-s3-folder"
npm WARN deprecated @opentelemetry/api-metrics@0.32.0: Please use @opentelemetry/api >= 1.3.0

added 294 packages, and audited 295 packages in 14s

43 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
npm notice
npm notice New major version of npm available! 9.6.7 -> 10.7.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.7.0>
npm notice Run `npm install -g npm@10.7.0` to update!
npm notice

pulumi destroy:
$ /pulumi-deploy-executor pulumi destroy --stackIdentity="dirien/aws-ts-s3-folder/dev" --workDir="/deployment/aws-ts-s3-folder"
Destroying (dev)

View Live: https://app.pulumi.com/dirien/aws-ts-s3-folder/dev/updates/4


 - aws:s3:BucketPolicy bucketPolicy deleting (0s)
 - aws:s3:BucketPolicy bucketPolicy deleted (1s)
 - aws:s3:BucketPublicAccessBlock public-access-block deleting (0s)
 - aws:s3:BucketObject favicon.png deleting (0s)
 - aws:s3:BucketObject index.html deleting (0s)
 - aws:s3:BucketObject index.html deleted (0.61s)
 - aws:s3:BucketObject favicon.png deleted (0.73s)
 - aws:s3:BucketPublicAccessBlock public-access-block deleted (0.77s)
 - aws:s3:Bucket s3-website-bucket deleting (0s)
 - aws:s3:Bucket s3-website-bucket deleted (0.79s)
 - pulumi:pulumi:Stack aws-ts-s3-folder-dev deleting (0s)
 - pulumi:pulumi:Stack aws-ts-s3-folder-dev deleted (0.10s)
Outputs:
 - bucketName: "s3-website-bucket-d55e9bd"
 - websiteUrl: "s3-website-bucket-d55e9bd.s3-website.eu-central-1.amazonaws.com"

Resources:
 - 6 deleted

Duration: 7s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
If you want to remove the stack completely, run `pulumi stack rm dev`.
Result: succeeded
INFO: 127.0.0.1:51509 - "DELETE /?name=my-test&org=dirien&project=aws-ts-s3-folder&stack=dev&region=eu-central-1 HTTP/1.1" 200 OK

Next Steps

Now that we have our basic REST services, we can deploy them on a server (using a service like AWS Lambda, Google Cloud Functions, or Azure Functions) and expose them to the rest of the organization via an API Gateway. In parallel you can publish release notes and documentation to your organization, so that they can start using your new Service API.

One way could be to create an OpenAPI specification for your Service API and use a tool like Swagger to generate a client library for your organization. This way, developers can interact with your Service API in a programmatic way, without having to write any code.

Here is an example of an OpenAPI specification for our Service API:

openapi: 3.0.0
info:
 title: My Organization Service API
 description: API for creating and destroying a specific piece of infrastructure
 version: 1.0.0
servers:
- url: 'http://localhost:8000/'
 description: Development server
paths:
 /:
 post:
 summary: Create Infrastructure
 operationId: createInfrastructure
 requestBody:
 required: true
 content:
 application/json:
 schema:
 $ref: '#/components/schemas/ServicePayload'
 responses:
 '200':
 description: Infrastructure successfully created
 content:
 application/json:
 schema:
 type: object
 properties:
 url:
 type: string
 description: URL of the created infrastructure
 '400':
 description: Bad request if the input payload is incorrect
 '500':
 description: Internal server error
 delete:
 summary: Destroy Infrastructure
 operationId: destroyInfrastructure
 parameters:
 - in: query
 name: name
 schema:
 type: string
 required: true
 description: Name of the service
 - in: query
 name: org
 schema:
 type: string
 required: true
 description: Organization in Pulumi
 - in: query
 name: project
 schema:
 type: string
 required: true
 description: Project name in Pulumi
 - in: query
 name: stack
 schema:
 type: string
 required: true
 description: Stack name in Pulumi
 - in: query
 name: region
 schema:
 type: string
 required: true
 description: AWS region where the service will be deployed
 responses:
 '200':
 description: Infrastructure successfully destroyed
 content:
 application/json:
 schema:
 type: object
 properties:
 message:
 type: string
 description: Confirmation message of the destruction
 '400':
 description: Bad request if the required parameters are missing
 '500':
 description: Internal server error
components:
 schemas:
 ServicePayload:
 type: object
 required:
 - name
 - org
 - project
 - stack
 - region
 properties:
 name:
 type: string
 description: Name of the service
 org:
 type: string
 description: Organization in Pulumi
 project:
 type: string
 description: Project name in Pulumi
 stack:
 type: string
 description: Stack name in Pulumi
 region:
 type: string
 description: AWS region where the service will be deployed

Navigate to the Swagger Editor and paste the above OpenAPI specification to see how it looks.

Conclusion

In this blog post, we’ve explored how Pulumi’s Automation API can be used to build and expose a Service API within your organization. By embedding Pulumi as a library in any application, you can drive your Infrastructure as Code (IaC) deployments directly through code. This sets Pulumi apart from other tools like Terraform, which relies on static configuration files and a command-line interface.

Additionally, we touched on the concept of the API economy, where APIs are leveraged to connect services, accelerate product delivery, and drive business value through the possibilities of creating new touchpoints. The Pulumi Automation API is the perfect piece of technology, which makes it now possible to define, deploy, and manage infrastructure programmatically as part of your API strategy.

I hope this blog post has inspired you with ideas on how to take your IaC to the next level, making it more accessible and practical for your organization.

Next Steps

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

Next-level IaC: Powering Up Pulumi with Package Ecosystems

Troy Howard — Mon, 13 May 2024 00:00:00 +0000

Every experienced tech professional I know has a programming language they love. But is it the syntax and symbols that make it so loveable? Not really. It’s the community and package ecosystem surrounding the language that makes a real impact on your heart… and on your productivity!

If we look at some of the biggest success stories in tech — Python, Node.js, Ruby, Perl, and Go — the common thread between all of them is an extensive ecosystem of packages, libraries, modules (or whatever you decide to call them… ahem, Gems?!). A great language will allow you to build anything you can imagine, but a great ecosystem will have already written it for you, and made it available in a convenient to install-and-use package.

One of the amazing things about Pulumi is that it is built around general-purpose programming languages, and that means your Pulumi programs have access to the entire ecosystem of packages that come with each language. This is a stark difference between Pulumi and other infrastructure automation tools that use proprietary domain-specific languages with not much in the way of community around them. Some tools might allow you to write custom code, but they certainly don’t make it convenient, and still… you have to write it yourself, which just adds so much overhead to a project where your core concern isn’t writing that custom function, but rather shipping your own product on the infrastructure you are trying to automate.

The flexibility that Pulumi gives you, and the access to these powerful libraries of code, means your productivity can skyrocket when you get to write in your favorite language, using your preferred tools and the package ecosystem you already know, love, and rely on every day.

Let’s have a look at a concrete example where this package ecosystem can get you out of a tough spot.

Use Case: Determining S3 MIME types on-the-fly

A popular example of cloud infrastructure automation is creating an S3 bucket and pushing some files to it. This is a common task that many of us have had to do, and most infrastructure automation tools have a way to accomplish this.

Here’s one way to do that in Pulumi:

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';

// Create an S3 bucket where we will upload the file.
const bucket = new aws.s3.Bucket("my-bucket", {
 // You can apply additional bucket settings here.
});

const filePath = "path/to/your/local/file.txt"; // Replace with the path to the file you want to upload

// Upload the file to the S3 bucket.
const bucketObject = new aws.s3.BucketObject("my-bucket-object", {
 bucket: bucket, // Reference to the bucket created above.
 key: "file.txt", // This will be the file's name in the bucket.
 source: pulumi.FileAsset(filePath), // Contents of the file.
 contentType: "text/plain", // MIME-type of the file. Adjust if you upload a different type of file.
});

// Export the name of the bucket
export const bucketName = bucket.id;

// Export the URL of the file in the S3 bucket
export const objectUrl = pulumi.interpolate`s3://${bucket.id}/${bucketObject.key}`;

import pulumi
import pulumi_aws as aws

# Create an S3 bucket
bucket = aws.s3.Bucket('my-bucket')

# The path to the file to upload to S3
file_path = "path/to/your/local/file.txt" # Replace with the path to the file you want to upload

# Upload the file to the S3 bucket
bucket_object = aws.s3.BucketObject('my-bucket-object',
 bucket=bucket.id, # Reference to the bucket created above.
 key='file.txt', # This will be the file's name in the bucket.
 source=pulumi.FileAsset(file_path),
 content_type='text/plain') # MIME-type of the file. Adjust if you upload a different type of file.

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

# Export the URL of the file in the S3 bucket
pulumi.export('object_url', pulumi.Output.concat('s3://', bucket.id, '/', bucket_object.key))

However, something that stands out as a weak link here is the line:

contentType: "text/plain", // MIME-type of the file. Adjust if you upload a different type of file.

content_type='text/plain') # MIME-type of the file. Adjust if you upload a different type of file.

Yeah, hard-coding the MIME type for every file I need to upload is a not going to scale. In fact, I’m probably going to get a random directory full of files I need to iterate over, each with different file types, and it will change every time it runs, and we won’t know what type of file it is ahead of time.

Packages to the rescue!! Turns out that determining the MIME type of a file only involves importing a community-created module like mime-types in TypeScript, or using the mimetypes module from Python’s standard library, and a single function call to guess the MIME type:

import * as mime from 'mime-types'

// Detect the MIME type of the file
const mimeType = mime.lookup('path/to/your/local/file.txt');

import mimetypes

# Detect the MIME type of the file
mime_type, _ = mimetypes.guess_type('path/to/your/local/file.txt')

So, to work that into our Pulumi program from above, we only need to make a few small edits:

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';
import * as mime from 'mime-types';

// Create an S3 bucket where we will upload the file.
const bucket = new aws.s3.Bucket("my-bucket", {
 // You can apply additional bucket settings here.
});

const filePath = "path/to/your/local/file.txt"; // Replace with the path to the file you want to upload

// Upload the file to the S3 bucket.
const bucketObject = new aws.s3.BucketObject("my-bucket-object", {
 bucket: bucket, // Reference to the bucket created above.
 key: "file.txt", // This will be the file's name in the bucket.
 source: pulumi.FileAsset(filePath), // Contents of the file.
 contentType: mime.lookup(filePath) || "application/octet-stream", // MIME-type of the file
});

// Export the name of the bucket
export const bucketName = bucket.id;

// Export the URL of the file in the S3 bucket
export const objectUrl = pulumi.interpolate`s3://${bucket.id}/${bucketObject.key}`;

import pulumi
import pulumi_aws as aws
import mimetypes

# Create an S3 bucket
bucket = aws.s3.Bucket('my-bucket')

# The path to the file to upload to S3
file_path = "path/to/your/local/file.txt" # Replace with the path to the file you want to upload

# Detect the MIME type of the file
mime_type, _ = mimetypes.guess_type(file_path)

# Upload the file to the S3 bucket
bucket_object = aws.s3.BucketObject('my-bucket-object',
 bucket=bucket.id, # Reference to the bucket created above.
 key='file.txt', # This will be the file's name in the bucket.
 source=pulumi.FileAsset(file_path),
 content_type=mime_type or 'application/octet-stream') # MIME-type of the file

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

# Export the URL of the file in the S3 bucket
pulumi.export('object_url', pulumi.Output.concat('s3://', bucket.id, '/', bucket_object.key))

Well that was pretty straightforward, wasn’t it? Trying to do this in another tool might take all day to figure out the right way to implement it, but in Pulumi it worked exactly like you would expect it to, in no time at all.

It’s a real game-changer when you can just quickly grab something complicated like MIME typing from a package. But when you can’t, you’re going to be stuck figuring out how to write that functionality yourself, from scratch, using things like Bash and primitive data types, and then find yourself copy-pasting that stuff all over the place. All the while fully aware of how much time you’re wasting by having to do so. That’s a real pain! It’s the kind of pain most people managing infrastructure have been through before.

The possibilities are endless

What other packages can you imagine using in your infrastructure solutions? Maybe you could use the text-to-image library to generate an image badge for your website? Maybe use the crypto module to encrypt a file before uploading it to S3? Or possibly using octokit to check on the state of something in git before taking the next step in your automation?

Whatever you can do with code in your favorite language, you can do in Pulumi as part of your infrastructure automation. I’m not saying you should use cowsay to generate custom MOTD banners for your ssh logins, I’m just saying you could.

Next Steps

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

Next-level IaC: Drop those wrapper scripts and let your language do that for you

Christian Nunciato — Tue, 30 Apr 2024 00:00:00 +0000

Our users are always telling us (particularly the ones who come to Pulumi from other IaC tools) that being able to use general-purpose languages to manage their infrastructure was a game changer for them.

I know it was for me. As a JavaScript developer, when I discovered Pulumi and saw that I could do pretty much everything I was doing with Terraform but with TypeScript, I was immediately hooked; that’s all it took. Just being able to write my resource declarations in a language I knew well (and that my IDE understood) was huge.

But it’s easy to forget, even when you’ve been using Pulumi for a while, that you aren’t just getting the language alone — you’re getting the full capability of the language’s runtime environment also. Not just JavaScript, but all of Node.js. Not just C#, but everything the .NET framework, runtime, and ecosystem have to offer.

This is quite a big deal, actually. Whereas with other tools, you’re often bound by the limits of the tool’s DSL, with Pulumi, you have no such constraints; you can do just about anything your chosen language can do. That means much more than just declaring long lists of cloud resources — it means unlocking all kinds of capabilities that can help bring your infrastructure-management practice to a whole new level.

In this post, the first of a new series we’re calling Next-level IaC, we’ll show you a few small but useful ways you can take advantage of this reality — first, by spotting some opportunities to simplify your deployments.

Run pre-deployment tasks with Pulumi

Fundamentally, a Pulumi program is just a regular program. That means you can take actions programmatically at runtime — before the Pulumi engine gets going.

Suppose you were tasked with writing the code to deploy and manage a static website. Conceptually, you might break this process up into a few separate steps:

Build the website
Deploy the supporting infrastructure (e.g, the cloud storage, maybe a content-delivery network)
Copy the content of the website into the infrastructure

This you might lead you to reach for Bash to write up a shell script like the one below, which uses Hugo (a popular static-site generator) to build the website, Pulumi to provision the infrastructure, and the AWS CLI to push the built website up into the AWS cloud:

#!/bin/bash

# Build the website.
hugo --destination ./public

# Provision the infrastructure.
pulumi up --yes

# Copy the website content into the infrastructure.
aws s3 sync ./public s3://$(pulumi stack output bucketName)

And this would definitely work. But it’s also unnecessary. You can avoid both the Bash wrapper and the dependency on the AWS CLI by using your chosen language’s built-in support for spawning shell processes:

import * as aws from "@pulumi/aws";
import * as childProcess from "child_process";

// Build the website.
const result = childProcess.execSync("hugo --destination ./public", { stdio: "pipe", cwd: "./www" });

// Provision a storage bucket for the website.
const bucket = new aws.s3.Bucket("bucket", {
 website: {
 indexDocument: "index.html",
 },
});

// Copy the website home page into the bucket.
const homepage = new aws.s3.BucketObject(
 "index.html",
 {
 bucket: bucket.id,
 content: fs.readFileSync("./www/public/index.html", "utf-8"),
 contentType: "text/html",
 acl: "public-read",
 },
);

import pulumi_aws as aws
import subprocess

# Build the website.
result = subprocess.run(
 ["hugo", "--destination", "./public"],
 stdout=subprocess.PIPE,
 cwd="./www",
 check=True,
 shell=True,
)

# Provision a storage bucket for the website.
bucket = aws.s3.Bucket(
 "bucket", website=aws.s3.BucketWebsiteArgs(index_document="index.html")
)

# Copy the website home page into the bucket
homepage = aws.s3.BucketObject(
 "index.html",
 bucket=bucket.id,
 content=open("./www/public/index.html", "r").read(),
 content_type="text/html",
 acl="public-read",
)

Running processes like these synchronously, as above, means they’ll complete before Pulumi starts registering resources — important here because the BucketObject resource needs that public folder to exist for the deployment to succeed. Fortunately it does, so all is well:

$ pulumi up
Updating (dev)
Type Name Plan Info
+ pulumi:pulumi:Stack example-dev create 15 messages
+ ├─ aws:s3:Bucket bucket create
+ └─ aws:s3:BucketObject index.html create
Diagnostics:
pulumi:pulumi:Stack (example-dev):
Start building sites ...
| EN
-------------------+-----
Pages | 4
Paginator pages | 0
Non-page files | 0
Static files | 0
Processed images | 0
Aliases | 0
Sitemaps | 1
Cleaned | 0
Total in 11 ms
Resources:
+ 3 created

Also notice how the example captures the output of the process in result and writes it to the terminal. Standard logging like this tends not to be possible with static languages (at least not without some hackery) because there’s no real way to do it programmatically. Similarly, those that do support spawning processes tend to require them to return strictly formatted JSON strings — which, as the example shows, not all processes do. (Forcing you to have to wrap them with yet more shell scripts.)

Simple stuff, but flexible and powerful. With one line, you’ve kept your deployment process simple, dropping two tools you don’t need thanks to the built-in capabilities of Pulumi and your language of choice.

Fetch data to generate resources dynamically

To build on the static-website scenario, suppose you wanted to retrieve some data from an external source and use it to provision some cloud resources dynamically. For example, you might want to let your marketing team create short URLs on their own for social media posts. For this, you might once again turn to Bash and the AWS CLI to pull those URLs from your corporate CMS (e.g., with curl), either before or after running pulumi, and create the corresponding redirects imperatively using a follow-up shell command of some sort.

Again, a totally reasonable approach. But again, why bother, when you have the full capability of Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET at fingertips? Just add a few lines to your program to fetch the data and be on your way:

// ...

// Fetch some redirects from a hypothetical CMS.
const redirects = fetch(`${process.env.CMS_ENDPOINT}/redirects.json`)
 .then(response => response.json())
 .then(items =>
 items.forEach((redirect: any, i: number) => {
 // Create an S3 website redirect for each one.
 new aws.s3.BucketObject(
 `redirect-${i}`,
 {
 bucket: bucket.id,
 key: redirect.from,
 websiteRedirect: redirect.to,
 acl: "public-read",
 },
 { dependsOn: [ownershipControls, publicAccess] },
 );
 }),
 );

#...
import json
import os
import requests

# ...

# Fetch some redirects from a hypothetical CMS.
response = requests.get(f"{os.environ['CMS_ENDPOINT']}/redirects.json")
redirects = json.loads(response.text)

# Create an S3 website redirect for each one.
for i, redirect in enumerate(redirects):
 aws.s3.BucketObject(
 f"redirect-{i}",
 bucket=bucket.id,
 key=redirect["from"],
 website_redirect=redirect["to"],
 acl="public-read",
 )

Notice the example parses the response using the language’s built-in JSON support. If your CMS happened to return data in some other format, or required some additional credentials for authorization, no problem — your language and its package ecosystem are going to be there to help you.

Also notice the example reads the source URL directly from the environment. This is typically more cumbersome (if even possible) with other tools, requiring you to name your environment variables in tool-specific ways and then tack on additional blocks of configuration before you’re able to use them. As you can see, Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET makes this much more convenient.

Finally, notice the redirects are translated into cloud resources using a familiar JavaScript TypeScript Python Go C# Java Pulumi YAML F# VB looping construct. We’ll cover this topic in much more depth in an upcoming post. (Stay tuned!)

Hook the event loop to run tasks after deployment

Finally, you can do nifty things like run tasks programmatically after a Pulumi deployment completes — once again, without ever having to leave the Pulumi program.

If your hypothetical website were sitting behind a CloudFront CDN, for example, you might want some way to clear the CDN’s cache after a deployment. This generally requires some imperative action, like using the AWS CLI to run aws cloudfront create-invalidation sometime later. How might you do something like this with Pulumi, given Pulumi programs exit automatically when their operations complete?

One way — again, since a Pulumi program is just a regular Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET program — would be to write a JavaScript TypeScript Python Go C# Java Pulumi YAML F# VB function to create an invalidation request, and then ask the Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET runtime to run that function before exiting.

The mechanics of this vary slightly by language, but it works, and it’s a very handy way to keep all of your deployment logic in one place — again avoiding the need to relegate anything to a Bash script:

// ...
import * as cloudfront from "@aws-sdk/client-cloudfront";

const config = new pulumi.Config("aws");
const region = config.require("region");

// Create a CloudFront distribution for the website.
const cdn = new aws.cloudfront.Distribution("cdn", {
 enabled: true,
 defaultRootObject: "index.html",
 origins: [{
 originId: bucket.arn,
 domainName: bucket.websiteEndpoint,
 // ...
 }],
 // ...
});

function createInvalidation(id: string) {
 // Only invalidate after a deployment.
 if (pulumi.runtime.isDryRun()) {
 console.log("This is a Pulumi preview, so skipping cache invalidation.");
 return;
 }

 process.on("beforeExit", () => {
 const client = new cloudfront.CloudFrontClient({ region });
 const command = new cloudfront.CreateInvalidationCommand({
 DistributionId: id,
 InvalidationBatch: {
 CallerReference: `invalidation-${Date.now()}`,
 Paths: {
 Quantity: 1,
 Items: ["/*"],
 },
 },
 });

 client
 .send(command)
 .then(result => {
 console.log(`Invalidation status for ${id}: ${result.Invalidation?.Status}.`);
 process.exit(0);
 })
 .catch(error => {
 console.error(error);
 process.exit(1);
 });
 });
}

// Register a function to be invoked before the program exits.
cdn.id.apply(id => createInvalidation(id));

The relevant code is the call to process.on("beforeExit"), which registers a function to be invoked just before the program exits using the resolved ID of the distribution. (For more on how the apply() method works, see Inputs and Outputs.) The function returns early for Pulumi previews — no sense clearing the cache if the site hasn’t changed — and uses the AWS SDK for JavaScript to submit the invalidation request to CloudFront, naming it uniquely with a timestamp and logging the result to the console:

# ...
import boto3
import asyncio

def create_invalidation(id):
 # Don't bother invalidating unless it's an actual deployment.
 if pulumi.runtime.is_dry_run():
 print("This is a Pulumi preview, so skipping cache invalidation.")
 return

 client = boto3.client("cloudfront")
 result = client.create_invalidation(
 DistributionId=id,
 InvalidationBatch={
 "CallerReference": f"invalidation-{time.time()}",
 "Paths": {
 "Quantity": 1,
 "Items": ["/*"],
 },
 },
 )

 print(
 f"Cache invalidation for distribution {id}: {result['Invalidation']['Status']}."
 )


# Register a function to be invoked before the program exits.
cdn.id.apply(lambda id: atexit.register(lambda: create_invalidation(id)))

The relevant code is the call to atexit.register(), which registers a function to be invoked just before the program exits using the resolved ID of the distribution. (For more on how the apply() method works, see Inputs and Outputs.) The function returns early for Pulumi previews — no sense clearing the cache if the site hasn’t changed — and uses Boto3 to submit the invalidation request to CloudFront, naming it uniquely with a timestamp and logging the result to the console:

$ pulumi preview
Previewing update (dev)
Type Name Info
pulumi:pulumi:Stack example-dev 16 messages
Diagnostics:
pulumi:pulumi:Stack (example-dev):
...
This is a Pulumi preview, so skipping cache invalidation.

$ pulumi up
Updating (dev)
Type Name Status Info
pulumi:pulumi:Stack example-dev 16 messages
Diagnostics:
pulumi:pulumi:Stack (example-dev):
...
Cache invalidation of distribution EAJOVC1QNGT13: InProgress.

Techniques like these are a great way to eliminate unnecessary scripts, glue code, and other dependencies while keeping your deployment logic clear, well contained, and amenable to change.

Wrapping up

Hopefully this first post has given you a few new ways to think about how to do more with Pulumi and your language of choice. Over the next few weeks, as this series unfolds, we’ll share a lot more, so keep an eye on this space!

You’ll find the full source for the examples above on GitHub:

Happy coding!

Next Steps

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

Pulumi Blog: Next level iac

Next-level IaC: Bridging the Declarative Gap

Pulumi’s architecture and flow of operations

Inputs and Outputs as asynchronous futures

Realizing values using the apply function

Example: Using apply to Base64 encode an Output

Next steps

Pulumi Cloud

Next-level IaC: Breakpoint Debugging for Pulumi Programs

How Breakpoint Debugging Works

Example: Attaching a debugger to a TypeScript Pulumi program in VS Code

Step 1: Create a Pulumi project from a template

Step 2: Lets break stuff!

Step 3: Create a debug wrapper script

Step 4: Configure VS Code launch.json and tasks.json

Step 5: Setting a debugging breakpoint

Considerations and Limitations

Next Steps

Pulumi Cloud

Next-level IaC: How Pulumi Supports Your API Economy Strategy

What is API Economy?

What is Pulumi Automation API in a nutshell?

How to build your own Service API with Pulumi Automation API?

Next Steps

Conclusion

Next Steps

Pulumi Cloud

Next-level IaC: Powering Up Pulumi with Package Ecosystems

Use Case: Determining S3 MIME types on-the-fly

The possibilities are endless

Next Steps

Pulumi Cloud

Next-level IaC: Drop those wrapper scripts and let your language do that for you

Run pre-deployment tasks with Pulumi

Fetch data to generate resources dynamically

Hook the event loop to run tasks after deployment

Wrapping up

Next Steps

Pulumi Cloud

Realizing values using the `apply` function

Example: Using `apply` to Base64 encode an Output

Step 4: Configure VS Code `launch.json` and `tasks.json`