Pulumi Blog: Nodejs

Next-level IaC: Breakpoint Debugging for Pulumi Programs

Troy Howard — Fri, 31 May 2024 00:00:00 +0000

For many of us, life is suspended between presses of the F5 key. Our IDE is the only place where everything is organized and makes sense. And while we know that the likelihood of a code hole-in-one is very rare (that magical moment where you run your build and everything just works the first time), we always hope that this time, this will be the run where you see the output that lets you know you’ve gotten it right. And in between? Debugging.

Debugging is a complicated topic. But what is not complicated is understanding the cost of debugging. Debugging is the biggest unknown-unknown in software. It makes you miss your sprint goals, it holds back scaling and company growth, it slows your time to market. Debugging is what makes estimating so difficult, because until you know what the problem is, you don’t know how long it will take to fix, or indeed if it is even possible to fix at all. Which is why savvy developers invest a lot of time into refining their debugging process and seek out the best-in-class tools to streamline this process.

In the infrastructure world, debugging is a pretty big problem. Some of the most popular tools have nothing more than simple unstructured text output logs. While tools like Terraform call themselves “Infrastructure as Code” the “code” is a proprietary DSL, not a general purpose programming language, and so there is little-to-no IDE support. There is no F5. Even worse, there is no F9 for setting a breakpoint. There’s no way to inspect state mid-run, or more importantly to explore in an unstructured way, because this is where the “aha!” moments usually happen.

Pulumi gives you a rich IaC debugging experience by enabling breakpoint debugging in your favorite IDE. Let’s walk through an example of how to enable a full-featured debugging workflow in VS Code for TypeScript.

How Breakpoint Debugging Works

Using a debugger can feel like that moment in The Matrix where Neo first sees the green streaming code raining down. A debugger lifts the veil from your otherwise opaque executable code. Tools like VS Code provide a user experience where you set breakpoints, i.e. certain lines of code that are designated as points where you want to pause execution and inspect state. Hitting the F9 key in VS Code sets a breakpoint on the current line of code. Later, you run your code in debug mode. This is a special execution state where the underlying runtime (e.g. Node’s V8) exposes a TCP/IP port hosting a debugging protocol, and when run, it will pause execution until something attaches to it (i.e. connects to the port to start the debugging session).

At this point a debugging client (such as VS Code) will control the flow of the program, moving forward step-by-step, one line of code at a time. But how does it know which line of code it’s executing? When you run in debug mode, the binary code is instrumented with metadata about the line of code it represents. The debugger will tell the runtime, via the debugging protocol, “run until you reach line 109 in file data.ts, then pause and wait for a signal from me before continuing”.

While paused, the protocol also allows you to inspect the values of any variables that are in scope at that moment. A more sophisticated debugger will provide a REPL environment where you can execute ad-hoc code in the scope of the line where you are paused. This can be very powerful to inspect the values of various objects and to see in real-time what the effects of executing variations of code would be. After a debugging session you can take what you’ve learned and edit the code accordingly.

Example: Attaching a debugger to a TypeScript Pulumi program in VS Code

For this example, we will set up a new Pulumi project from the command-line and then set up VS Code to run that program in a debugger.

Step 1: Create a Pulumi project from a template

For this example, we will create a new Pulumi project using the TypeScript AWS static website template. Make sure you’ve set up Pulumi, logged into Pulumi Cloud, and configured your AWS credentials. Follow the prompts in the pulumi CLI to configure your project.

$ mkdir good-morning && cd good-morning
$ pulumi new static-website-aws-typescript

Then launch this project to make sure it works as expected:

$ pulumi up
...

Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/troy-pulumi-corp/good-morning/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack good-morning-dev created (200s)
 + ├─ aws:s3:Bucket bucket created (1s)
 + ├─ aws:s3:BucketPublicAccessBlock public-access-block created (0.57s)
 + ├─ aws:s3:BucketOwnershipControls ownership-controls created (1s)
 + ├─ aws:cloudfront:Distribution cdn created (196s)
 + └─ synced-folder:index:S3BucketFolder bucket-folder created (0.41s)
 + ├─ aws:s3:BucketObject error.html created (0.46s)
 + └─ aws:s3:BucketObject index.html created (0.46s)

Outputs:
 cdnHostname : "##############.cloudfront.net"
 cdnURL : "https://##############.cloudfront.net"
 originHostname: "bucket-#######.s3-website-us-west-2.amazonaws.com"
 originURL : "http://bucket-#######.s3-website-us-west-2.amazonaws.com"

Resources:
 + 8 created

Great! You should be able to navigate to the website using the URL from the cdnURL output value.

$ open $(pulumi stack output cdnURL)

If all looks good, let’s move on to the next step by opening your project directory in VS Code:

$ code .

Step 2: Lets break stuff!

Now that we have a perfectly running Pulumi program, in order to have something to debug, we will need to break it!

For this example, we are going to change our indexDocument from index.html to good-morning.html.

$ cp ./wwww/index.html ./www/good-morning.html

Next, let’s edit the contents to say “Good morning, world!” instead of “Hello, world!”:

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="utf-8">
 <title>Good morning, world!</title>
</head>
<body>
 <h1>Good morning, world! 👋</h1>
 <p>Deployed with 💜 by <a href="https://pulumi.com/">Pulumi</a>.</p>
</body>
</html>

Once that’s edited and saved, go ahead let’s redeploy the website:

$ pulumi up

Everything runs successfully… but if we navigate to the website, we don’t see our updated content?! Oh no! What went wrong? Why isn’t there an error message?! Time to put on our debugging hats and figure out what is going on. First, let’s set up VS Code to enable breakpoint debugging right in the IDE.

Step 3: Create a debug wrapper script

VS Code can run an arbitrary command when you hit F5. Typically this would be a one-liner that runs your language runtime like node foo.js or python foo.py. In our case, we need to run the pulumi CLI and need to do more than one step. The easiest way to do this is to create a small wrapper script and call that from VS Code.

In the root of the directory, create a file called pulumi-debug.sh:

#!/bin/bash

pulumi login
pulumi stack select dev
pulumi up -f

Let’s walk through these lines to explain what’s going on here:

pulumi login to set up credentials for the pulumi backend
pulumi stack select dev to run this against our dev stack
pulumi up -f to run our program update. The -f flag skips the preview phase and also skips the need to interactively confirm that we want to run the update.

Open the VS Code terminal panel and first set it to be executable, then try running the script:

$ chmod +x pulumi-debug.sh
$ ./pulumi-debug.sh

Ok now let’s set up VS Code to run that in debug mode when we hit F5!

Step 4: Configure VS Code `launch.json` and `tasks.json`

To run our Pulumi program in debug mode we will need to create a couple of VS Code configuration files, launch.json and tasks.json, in a hidden directory called .vscode within your project.

Create the .vscode directory and then create a new file called tasks.json file in that directory, with the following contents:

{
 "version": "2.0.0",
 "tasks": [
 {
 "label": "pulumi-debug",
 "type": "shell",
 "command": ". ./pulumi-debug.sh",
 "isBackground": true,
 "options": {
 "cwd": "${workspaceFolder}"
 }
 }
 ]
}

This creates a VS Code task named pulumi-debug which will run the wrapper script we just created from the root of our project.

Next, create a file named launch.json in the same directory. This will configure the behavior of the F5 key!

{
 "version": "0.2.0",
 "configurations": [
 {
 "name": "Launch Pulumi Program (debug)",
 "type": "node",
 "port": 9292,
 "request": "attach",
 "preLaunchTask": "pulumi-debug",
 "continueOnAttach": true,
 "restart": {
 "delay": 5000,
 "maxAttempts": 5
 },
 "skipFiles": [
 "${workspaceFolder}/node_modules/**/*.js",
 "${workspaceFolder}/lib/**/*.js",
 "<node_internals>/**/*.js"
 ]
 }
 ]
}

Let’s break this down a bit:

type: node tells VS Code that we are using the Node runtime
port: 9292 tells VS Code to attach the debugger to port 9292
request: attach tells VS Code that we want to attach to the debugger at launch
preLaunchTask: pulumi-debug tells VS Code to run the task we configured in tasks.json before we connect the debugger. This will run our pulumi-debug.sh script and then attach to the Node.js runtime
continueOnAttach: true tells VS Code to start running our code as soon as we attach to the waiting Node.js runtime (until we reach our first breakpoint that is, where it will pause and return control to the debugger client, aka VS Code)
restart: {...} configures VS Code to wait 5 seconds and then retry connecting, up to 5 times. This is here in case it takes a while for the Node.js runtime to start running. It’s unlikely, but possible!
skipFiles: {...} sets up some filters of files not to try to debug. This includes everything that is part of node’s internals, 3rdparty node modules, and other libraries. That way we don’t end up deep in someone else’s code unexpectedly and can just focus on our Pulumi program.
name sets a human-readable name which shows up in the drop down box in VS Code when you choose which [launch configuration][vs-code-launch-configs] to run.

Finally, we need to make one more edit to the pulumi-debug.sh file we created. Here we will pass the necessary options to the Node.js runtime to set it to run in debug mode.

Add the following line to the beginning of the file:

#!/bin/bash

export NODE_OPTIONS='--inspect-brk=127.0.0.1:9292'

pulumi login
pulumi stack select dev
pulumi up -f

The NODE_OPTIONS environment variable passes options to the Node.js V8 runtime. In this case, we are passing --inspect-brk which tells the Node.js runtime to run in debug mode, which will cause it to pause before running the code, until a debugging client connects to localhost on port 9292.

Ok, now we should be ready to run our Pulumi program from VS Code using the F5 key! Let’s give it a try. You should see Pulumi running inside of the “Terminal” tab. You will also see output indicating that it is waiting for a debugger to attach:

Debugger listening on ws://127.0.0.1:9292/########-####-####-####-############
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
Waiting for the debugger to disconnect...

Step 5: Setting a debugging breakpoint

Now we can try setting a breakpoint in our code. Getting back to the problem at hand – our content doesn’t seem to be updating, despite no errors. Let’s start at the beginning, by investigating the inputs to our program, aka our configuration… at runtime!

Navigate to line 12 within the code and hit the F9 key. You’ll see a small red dot appear next to the line indicating that a breakpoint has been set. This line defines the bucket our program creates and seems like a good place to stop if we want to check the input configuration before we run the AWS S3 provider call.

Figure: A breakpoint set on line 12.

You can then run the program again using F5. When the execution of the code reaches the line it will pause there. At this point you can navigate around the code inspecting the state of the local variables, like you would in any other debugging session.

Let’s inspect the configuration variables. On the left side of the screen you’ll find a “Variables” panel listing all the local variables that are in scope.

Figure: Local variables displayed in the Variables side panel.

There’s also the “Debug” console which provides a REPL environment to navigate those variables and anything else that is in scope. You can run arbitrary functions here and see their output. In this case we can simply type the variable name at the prompt there, hit enter, and see the value. What’s great about the debug console is that Intellisense (e.g. using TAB to auto-complete) works here, so you can quickly navigate possible options.

Figure: Debug console tab completion and inspecting variable values.

In our case, we’re interested in the value of indexDocument variable. We can see that it is getting set to index.html… not good-morning.html. The Pulumi program didn’t error because index.html still exists. So we are still publishing our old index.html. That makes sense. A quick fix here is to modify index.ts and set the default to good-morning.html:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as synced_folder from "@pulumi/synced-folder";

// Import the program's configuration settings.
const config = new pulumi.Config();
const path = config.get("path") || "./www";
const indexDocument = config.get("indexDocument") || "good-morning.html";
const errorDocument = config.get("errorDocument") || "error.html";

...

Ok, we edited the document, which fixes the program for future runs, but for the current run that we are debugging, indexDocument is still set to index.html. Let’s use the debug console to set the value to good-morning.html before we continue the run.

> indexDocument="good-morning.html`

The output in the debug window will show the new value, and if you hover over the inputs being passed to the bucket creation call, it will now show as good-morning.html:

Figure: Screenshots of debug console and editor window after changing the indexDocument variable

When you’re ready for the program to continue on, hit F5 once again and the program will resume operation, either until the program exits or until it reaches the next breakpoint. Optionally you can step through line-by-line using the F10 key. VS Code provides a variety of debug actions once the session is established.

Most importantly, check the published website and make sure we fixed the issue! You can find the cdnURL output in the Terminal tab and click on it from there to open the link.

Considerations and Limitations

Breakpoint debugging is a powerful tool for inspecting the state of your Pulumi program. However, there are some important limitations to consider:

Pulumi programs are only part of the story: This technique will let you inspect the program you authored, but it doesn’t give access to what’s happening with the Pulumi backend, the providers, or anything else outside of the program itself. You might find yourself needing to dig deeper to resolve a problem with the stack as a whole. Luckily there are other techniques we can use for that, which we will delve into in an upcoming post.
Inputs and Outputs are asynchronous futures: If you’ve used Pulumi before you will have encountered Input<T> and Output<T> values, which are a core function of the tool. However, at the time that the debugger hits a breakpoint on those values, they are likely not populated with their final value. That means you won’t be able to inspect them in the debugger very well. Instead, you will need to add an apply(...) or all(..) block to the code, and place the breakpoint on the materialized value within that block. This follows all the same rules as accessing those variables in code in a Pulumi program.
```
bucket.id.apply(id => {
 // NOTE: The `debugger` keyword automatically sets a breakpoint here
 // so you don't need to set it via the IDE.
 debugger
});
```
Limited integration with VS Code: Unfortunately our technique of running Pulumi in wrapper script as a backgrounded pre-launch task in VS Code leaves some things to be desired. It is different enough from the use case that VS Code had in mind for these features (e.g. interacting with a long-running tsc-watch or similar task) that making things “just work” is a little difficult. For example, we can’t easily report on errors in the “Problems” tab, and VS Code complains that a problem matcher is missing the first time you run it. VS Code can’t tell that the program finished and so you’ll have to manually stop the debugging session after Pulumi has completed (or wait for the retry/timeout to expire).

Despite these limitations, Pulumi with VS Code as your IDE provides a rich experience compared to trawlling through unstructured logs, or adding in console.log(...) calls to echo out ad-hoc debugging information (aka. printf debugging). Also, you can enable this for any language that Pulumi supports, and most popular IDEs support breakpoint debugging (e.g. Visual Studio, Webstorm, neovim, etc). Python, Go, and C# all have popular debuggers and IDEs that can be used in the same manner. Whatever your preferred IDE or language is, you can use their full range of breakpoint debugging features for your Pulumi programs.

Pulumi also offers detailed logging which can be used to look into the other aspects that breakpoint debugging doesn’t cover. In an upcoming article we’ll show how to use these tools as well. Until then, have fun spelunking through your code with your favorite IDE!

Next Steps

If you haven’t already, install Pulumi today, and follow our self-directed Getting Started guides to learn more about making the most of Pulumi’s next-level infrastructure management features at your organization.

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

The Pulumi Cloud is a fully managed service that helps you adopt Pulumi’s open source SDK with ease. It provides built-in state and secrets management, integrates with source control and CI/CD, and offers a web console and API that make it easier to visualize and manage infrastructure. It is free for individual use, with features available for teams.

Create an Account

Next-level IaC: Powering Up Pulumi with Package Ecosystems

Troy Howard — Mon, 13 May 2024 00:00:00 +0000

Every experienced tech professional I know has a programming language they love. But is it the syntax and symbols that make it so loveable? Not really. It’s the community and package ecosystem surrounding the language that makes a real impact on your heart… and on your productivity!

If we look at some of the biggest success stories in tech — Python, Node.js, Ruby, Perl, and Go — the common thread between all of them is an extensive ecosystem of packages, libraries, modules (or whatever you decide to call them… ahem, Gems?!). A great language will allow you to build anything you can imagine, but a great ecosystem will have already written it for you, and made it available in a convenient to install-and-use package.

One of the amazing things about Pulumi is that it is built around general-purpose programming languages, and that means your Pulumi programs have access to the entire ecosystem of packages that come with each language. This is a stark difference between Pulumi and other infrastructure automation tools that use proprietary domain-specific languages with not much in the way of community around them. Some tools might allow you to write custom code, but they certainly don’t make it convenient, and still… you have to write it yourself, which just adds so much overhead to a project where your core concern isn’t writing that custom function, but rather shipping your own product on the infrastructure you are trying to automate.

The flexibility that Pulumi gives you, and the access to these powerful libraries of code, means your productivity can skyrocket when you get to write in your favorite language, using your preferred tools and the package ecosystem you already know, love, and rely on every day.

Let’s have a look at a concrete example where this package ecosystem can get you out of a tough spot.

Use Case: Determining S3 MIME types on-the-fly

A popular example of cloud infrastructure automation is creating an S3 bucket and pushing some files to it. This is a common task that many of us have had to do, and most infrastructure automation tools have a way to accomplish this.

Here’s one way to do that in Pulumi:

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';

// Create an S3 bucket where we will upload the file.
const bucket = new aws.s3.Bucket("my-bucket", {
 // You can apply additional bucket settings here.
});

const filePath = "path/to/your/local/file.txt"; // Replace with the path to the file you want to upload

// Upload the file to the S3 bucket.
const bucketObject = new aws.s3.BucketObject("my-bucket-object", {
 bucket: bucket, // Reference to the bucket created above.
 key: "file.txt", // This will be the file's name in the bucket.
 source: pulumi.FileAsset(filePath), // Contents of the file.
 contentType: "text/plain", // MIME-type of the file. Adjust if you upload a different type of file.
});

// Export the name of the bucket
export const bucketName = bucket.id;

// Export the URL of the file in the S3 bucket
export const objectUrl = pulumi.interpolate`s3://${bucket.id}/${bucketObject.key}`;

import pulumi
import pulumi_aws as aws

# Create an S3 bucket
bucket = aws.s3.Bucket('my-bucket')

# The path to the file to upload to S3
file_path = "path/to/your/local/file.txt" # Replace with the path to the file you want to upload

# Upload the file to the S3 bucket
bucket_object = aws.s3.BucketObject('my-bucket-object',
 bucket=bucket.id, # Reference to the bucket created above.
 key='file.txt', # This will be the file's name in the bucket.
 source=pulumi.FileAsset(file_path),
 content_type='text/plain') # MIME-type of the file. Adjust if you upload a different type of file.

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

# Export the URL of the file in the S3 bucket
pulumi.export('object_url', pulumi.Output.concat('s3://', bucket.id, '/', bucket_object.key))

However, something that stands out as a weak link here is the line:

contentType: "text/plain", // MIME-type of the file. Adjust if you upload a different type of file.

content_type='text/plain') # MIME-type of the file. Adjust if you upload a different type of file.

Yeah, hard-coding the MIME type for every file I need to upload is a not going to scale. In fact, I’m probably going to get a random directory full of files I need to iterate over, each with different file types, and it will change every time it runs, and we won’t know what type of file it is ahead of time.

Packages to the rescue!! Turns out that determining the MIME type of a file only involves importing a community-created module like mime-types in TypeScript, or using the mimetypes module from Python’s standard library, and a single function call to guess the MIME type:

import * as mime from 'mime-types'

// Detect the MIME type of the file
const mimeType = mime.lookup('path/to/your/local/file.txt');

import mimetypes

# Detect the MIME type of the file
mime_type, _ = mimetypes.guess_type('path/to/your/local/file.txt')

So, to work that into our Pulumi program from above, we only need to make a few small edits:

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';
import * as mime from 'mime-types';

// Create an S3 bucket where we will upload the file.
const bucket = new aws.s3.Bucket("my-bucket", {
 // You can apply additional bucket settings here.
});

const filePath = "path/to/your/local/file.txt"; // Replace with the path to the file you want to upload

// Upload the file to the S3 bucket.
const bucketObject = new aws.s3.BucketObject("my-bucket-object", {
 bucket: bucket, // Reference to the bucket created above.
 key: "file.txt", // This will be the file's name in the bucket.
 source: pulumi.FileAsset(filePath), // Contents of the file.
 contentType: mime.lookup(filePath) || "application/octet-stream", // MIME-type of the file
});

// Export the name of the bucket
export const bucketName = bucket.id;

// Export the URL of the file in the S3 bucket
export const objectUrl = pulumi.interpolate`s3://${bucket.id}/${bucketObject.key}`;

import pulumi
import pulumi_aws as aws
import mimetypes

# Create an S3 bucket
bucket = aws.s3.Bucket('my-bucket')

# The path to the file to upload to S3
file_path = "path/to/your/local/file.txt" # Replace with the path to the file you want to upload

# Detect the MIME type of the file
mime_type, _ = mimetypes.guess_type(file_path)

# Upload the file to the S3 bucket
bucket_object = aws.s3.BucketObject('my-bucket-object',
 bucket=bucket.id, # Reference to the bucket created above.
 key='file.txt', # This will be the file's name in the bucket.
 source=pulumi.FileAsset(file_path),
 content_type=mime_type or 'application/octet-stream') # MIME-type of the file

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

# Export the URL of the file in the S3 bucket
pulumi.export('object_url', pulumi.Output.concat('s3://', bucket.id, '/', bucket_object.key))

Well that was pretty straightforward, wasn’t it? Trying to do this in another tool might take all day to figure out the right way to implement it, but in Pulumi it worked exactly like you would expect it to, in no time at all.

It’s a real game-changer when you can just quickly grab something complicated like MIME typing from a package. But when you can’t, you’re going to be stuck figuring out how to write that functionality yourself, from scratch, using things like Bash and primitive data types, and then find yourself copy-pasting that stuff all over the place. All the while fully aware of how much time you’re wasting by having to do so. That’s a real pain! It’s the kind of pain most people managing infrastructure have been through before.

The possibilities are endless

What other packages can you imagine using in your infrastructure solutions? Maybe you could use the text-to-image library to generate an image badge for your website? Maybe use the crypto module to encrypt a file before uploading it to S3? Or possibly using octokit to check on the state of something in git before taking the next step in your automation?

Whatever you can do with code in your favorite language, you can do in Pulumi as part of your infrastructure automation. I’m not saying you should use cowsay to generate custom MOTD banners for your ssh logins, I’m just saying you could.

Next Steps

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

Next-level IaC: Drop those wrapper scripts and let your language do that for you

Christian Nunciato — Tue, 30 Apr 2024 00:00:00 +0000

Our users are always telling us (particularly the ones who come to Pulumi from other IaC tools) that being able to use general-purpose languages to manage their infrastructure was a game changer for them.

I know it was for me. As a JavaScript developer, when I discovered Pulumi and saw that I could do pretty much everything I was doing with Terraform but with TypeScript, I was immediately hooked; that’s all it took. Just being able to write my resource declarations in a language I knew well (and that my IDE understood) was huge.

But it’s easy to forget, even when you’ve been using Pulumi for a while, that you aren’t just getting the language alone — you’re getting the full capability of the language’s runtime environment also. Not just JavaScript, but all of Node.js. Not just C#, but everything the .NET framework, runtime, and ecosystem have to offer.

This is quite a big deal, actually. Whereas with other tools, you’re often bound by the limits of the tool’s DSL, with Pulumi, you have no such constraints; you can do just about anything your chosen language can do. That means much more than just declaring long lists of cloud resources — it means unlocking all kinds of capabilities that can help bring your infrastructure-management practice to a whole new level.

In this post, the first of a new series we’re calling Next-level IaC, we’ll show you a few small but useful ways you can take advantage of this reality — first, by spotting some opportunities to simplify your deployments.

Run pre-deployment tasks with Pulumi

Fundamentally, a Pulumi program is just a regular program. That means you can take actions programmatically at runtime — before the Pulumi engine gets going.

Suppose you were tasked with writing the code to deploy and manage a static website. Conceptually, you might break this process up into a few separate steps:

Build the website
Deploy the supporting infrastructure (e.g, the cloud storage, maybe a content-delivery network)
Copy the content of the website into the infrastructure

This you might lead you to reach for Bash to write up a shell script like the one below, which uses Hugo (a popular static-site generator) to build the website, Pulumi to provision the infrastructure, and the AWS CLI to push the built website up into the AWS cloud:

#!/bin/bash

# Build the website.
hugo --destination ./public

# Provision the infrastructure.
pulumi up --yes

# Copy the website content into the infrastructure.
aws s3 sync ./public s3://$(pulumi stack output bucketName)

And this would definitely work. But it’s also unnecessary. You can avoid both the Bash wrapper and the dependency on the AWS CLI by using your chosen language’s built-in support for spawning shell processes:

import * as aws from "@pulumi/aws";
import * as childProcess from "child_process";

// Build the website.
const result = childProcess.execSync("hugo --destination ./public", { stdio: "pipe", cwd: "./www" });

// Provision a storage bucket for the website.
const bucket = new aws.s3.Bucket("bucket", {
 website: {
 indexDocument: "index.html",
 },
});

// Copy the website home page into the bucket.
const homepage = new aws.s3.BucketObject(
 "index.html",
 {
 bucket: bucket.id,
 content: fs.readFileSync("./www/public/index.html", "utf-8"),
 contentType: "text/html",
 acl: "public-read",
 },
);

import pulumi_aws as aws
import subprocess

# Build the website.
result = subprocess.run(
 ["hugo", "--destination", "./public"],
 stdout=subprocess.PIPE,
 cwd="./www",
 check=True,
 shell=True,
)

# Provision a storage bucket for the website.
bucket = aws.s3.Bucket(
 "bucket", website=aws.s3.BucketWebsiteArgs(index_document="index.html")
)

# Copy the website home page into the bucket
homepage = aws.s3.BucketObject(
 "index.html",
 bucket=bucket.id,
 content=open("./www/public/index.html", "r").read(),
 content_type="text/html",
 acl="public-read",
)

Running processes like these synchronously, as above, means they’ll complete before Pulumi starts registering resources — important here because the BucketObject resource needs that public folder to exist for the deployment to succeed. Fortunately it does, so all is well:

$ pulumi up
Updating (dev)
Type Name Plan Info
+ pulumi:pulumi:Stack example-dev create 15 messages
+ ├─ aws:s3:Bucket bucket create
+ └─ aws:s3:BucketObject index.html create
Diagnostics:
pulumi:pulumi:Stack (example-dev):
Start building sites ...
| EN
-------------------+-----
Pages | 4
Paginator pages | 0
Non-page files | 0
Static files | 0
Processed images | 0
Aliases | 0
Sitemaps | 1
Cleaned | 0
Total in 11 ms
Resources:
+ 3 created

Also notice how the example captures the output of the process in result and writes it to the terminal. Standard logging like this tends not to be possible with static languages (at least not without some hackery) because there’s no real way to do it programmatically. Similarly, those that do support spawning processes tend to require them to return strictly formatted JSON strings — which, as the example shows, not all processes do. (Forcing you to have to wrap them with yet more shell scripts.)

Simple stuff, but flexible and powerful. With one line, you’ve kept your deployment process simple, dropping two tools you don’t need thanks to the built-in capabilities of Pulumi and your language of choice.

Fetch data to generate resources dynamically

To build on the static-website scenario, suppose you wanted to retrieve some data from an external source and use it to provision some cloud resources dynamically. For example, you might want to let your marketing team create short URLs on their own for social media posts. For this, you might once again turn to Bash and the AWS CLI to pull those URLs from your corporate CMS (e.g., with curl), either before or after running pulumi, and create the corresponding redirects imperatively using a follow-up shell command of some sort.

Again, a totally reasonable approach. But again, why bother, when you have the full capability of Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET at fingertips? Just add a few lines to your program to fetch the data and be on your way:

// ...

// Fetch some redirects from a hypothetical CMS.
const redirects = fetch(`${process.env.CMS_ENDPOINT}/redirects.json`)
 .then(response => response.json())
 .then(items =>
 items.forEach((redirect: any, i: number) => {
 // Create an S3 website redirect for each one.
 new aws.s3.BucketObject(
 `redirect-${i}`,
 {
 bucket: bucket.id,
 key: redirect.from,
 websiteRedirect: redirect.to,
 acl: "public-read",
 },
 { dependsOn: [ownershipControls, publicAccess] },
 );
 }),
 );

#...
import json
import os
import requests

# ...

# Fetch some redirects from a hypothetical CMS.
response = requests.get(f"{os.environ['CMS_ENDPOINT']}/redirects.json")
redirects = json.loads(response.text)

# Create an S3 website redirect for each one.
for i, redirect in enumerate(redirects):
 aws.s3.BucketObject(
 f"redirect-{i}",
 bucket=bucket.id,
 key=redirect["from"],
 website_redirect=redirect["to"],
 acl="public-read",
 )

Notice the example parses the response using the language’s built-in JSON support. If your CMS happened to return data in some other format, or required some additional credentials for authorization, no problem — your language and its package ecosystem are going to be there to help you.

Also notice the example reads the source URL directly from the environment. This is typically more cumbersome (if even possible) with other tools, requiring you to name your environment variables in tool-specific ways and then tack on additional blocks of configuration before you’re able to use them. As you can see, Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET makes this much more convenient.

Finally, notice the redirects are translated into cloud resources using a familiar JavaScript TypeScript Python Go C# Java Pulumi YAML F# VB looping construct. We’ll cover this topic in much more depth in an upcoming post. (Stay tuned!)

Hook the event loop to run tasks after deployment

Finally, you can do nifty things like run tasks programmatically after a Pulumi deployment completes — once again, without ever having to leave the Pulumi program.

If your hypothetical website were sitting behind a CloudFront CDN, for example, you might want some way to clear the CDN’s cache after a deployment. This generally requires some imperative action, like using the AWS CLI to run aws cloudfront create-invalidation sometime later. How might you do something like this with Pulumi, given Pulumi programs exit automatically when their operations complete?

One way — again, since a Pulumi program is just a regular Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET program — would be to write a JavaScript TypeScript Python Go C# Java Pulumi YAML F# VB function to create an invalidation request, and then ask the Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET runtime to run that function before exiting.

The mechanics of this vary slightly by language, but it works, and it’s a very handy way to keep all of your deployment logic in one place — again avoiding the need to relegate anything to a Bash script:

// ...
import * as cloudfront from "@aws-sdk/client-cloudfront";

const config = new pulumi.Config("aws");
const region = config.require("region");

// Create a CloudFront distribution for the website.
const cdn = new aws.cloudfront.Distribution("cdn", {
 enabled: true,
 defaultRootObject: "index.html",
 origins: [{
 originId: bucket.arn,
 domainName: bucket.websiteEndpoint,
 // ...
 }],
 // ...
});

function createInvalidation(id: string) {
 // Only invalidate after a deployment.
 if (pulumi.runtime.isDryRun()) {
 console.log("This is a Pulumi preview, so skipping cache invalidation.");
 return;
 }

 process.on("beforeExit", () => {
 const client = new cloudfront.CloudFrontClient({ region });
 const command = new cloudfront.CreateInvalidationCommand({
 DistributionId: id,
 InvalidationBatch: {
 CallerReference: `invalidation-${Date.now()}`,
 Paths: {
 Quantity: 1,
 Items: ["/*"],
 },
 },
 });

 client
 .send(command)
 .then(result => {
 console.log(`Invalidation status for ${id}: ${result.Invalidation?.Status}.`);
 process.exit(0);
 })
 .catch(error => {
 console.error(error);
 process.exit(1);
 });
 });
}

// Register a function to be invoked before the program exits.
cdn.id.apply(id => createInvalidation(id));

The relevant code is the call to process.on("beforeExit"), which registers a function to be invoked just before the program exits using the resolved ID of the distribution. (For more on how the apply() method works, see Inputs and Outputs.) The function returns early for Pulumi previews — no sense clearing the cache if the site hasn’t changed — and uses the AWS SDK for JavaScript to submit the invalidation request to CloudFront, naming it uniquely with a timestamp and logging the result to the console:

# ...
import boto3
import asyncio

def create_invalidation(id):
 # Don't bother invalidating unless it's an actual deployment.
 if pulumi.runtime.is_dry_run():
 print("This is a Pulumi preview, so skipping cache invalidation.")
 return

 client = boto3.client("cloudfront")
 result = client.create_invalidation(
 DistributionId=id,
 InvalidationBatch={
 "CallerReference": f"invalidation-{time.time()}",
 "Paths": {
 "Quantity": 1,
 "Items": ["/*"],
 },
 },
 )

 print(
 f"Cache invalidation for distribution {id}: {result['Invalidation']['Status']}."
 )


# Register a function to be invoked before the program exits.
cdn.id.apply(lambda id: atexit.register(lambda: create_invalidation(id)))

The relevant code is the call to atexit.register(), which registers a function to be invoked just before the program exits using the resolved ID of the distribution. (For more on how the apply() method works, see Inputs and Outputs.) The function returns early for Pulumi previews — no sense clearing the cache if the site hasn’t changed — and uses Boto3 to submit the invalidation request to CloudFront, naming it uniquely with a timestamp and logging the result to the console:

$ pulumi preview
Previewing update (dev)
Type Name Info
pulumi:pulumi:Stack example-dev 16 messages
Diagnostics:
pulumi:pulumi:Stack (example-dev):
...
This is a Pulumi preview, so skipping cache invalidation.

$ pulumi up
Updating (dev)
Type Name Status Info
pulumi:pulumi:Stack example-dev 16 messages
Diagnostics:
pulumi:pulumi:Stack (example-dev):
...
Cache invalidation of distribution EAJOVC1QNGT13: InProgress.

Techniques like these are a great way to eliminate unnecessary scripts, glue code, and other dependencies while keeping your deployment logic clear, well contained, and amenable to change.

Wrapping up

Hopefully this first post has given you a few new ways to think about how to do more with Pulumi and your language of choice. Over the next few weeks, as this series unfolds, we’ll share a lot more, so keep an eye on this space!

You’ll find the full source for the examples above on GitHub:

Happy coding!

Next Steps

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

Run Pulumi with Any TypeScript Version

Julien Poissonnier — Tue, 16 Apr 2024 10:28:28 +0200

Pulumi allows you to use the best features of programming languages to build your infrastructure while ensuring compatibility with your current development environments. To this end Pulumi provides a seamless experience when authoring TypeScript programs, managing the compilation of your program for you. With the latest release of our Node.js SDK, we’ve made this even more flexible by making it easier than ever to choose the version of TypeScript you want to use.

Each new release of TypeScript comes with new features and improved type checking, as well as support for the latest JavaScript features. Using the latest version of TypeScript provides you with the maximum flexibility for creating types and gives you more confidence that your program will behave as expected.

Until now, Pulumi’s seamless experience was limited to using Pulumi’s builtin TypeScript version 3.8. If you wanted to use a more recent version, you had to introduce a build step to your project and let Pulumi run in plain Javascript mode. With the release of the Pulumi Node.js SDK 3.113.0 you can now use any version of TypeScript by adding it as a dependency to your project’s package.json file.

Pulumi supports all TypeScript versions from 3.8 and up, including TypeScript 5.4.

If you have an existing project, you can upgrade your project to the latest versions of TypeScript and Pulumi by running:

npm add typescript@^5.4.5 @pulumi/pulumi@^3.113.0

We have also updated our Pulumi templates to include the latest version of TypeScript. To create a new project, run:

pulumi new

Adding TypeScript as a dependency to your project is optional, as TypeScript and ts-node are optional peer dependencies of the Node.js SDK. This makes it easy to use a single version of TypeScript or ts-node within monorepos. To ensure backward compatibility, Pulumi ships with bundled versions of TypeScript 3.8 and ts-node 7, and falls back to these versions if they are not installed in your project’s node modules.

Example

Let’s show an example of how to use one of TypeScripts’s new features with Pulumi. TypeScript 4.1 introduced Template Literal Types, which allow you to build types using template literals. Imagine you wanted to enforce that names of S3 buckets are all lower case and always start with a given prefix. Of course you can check this at runtime, but with Template Literal Types this can be enforced during typechecking.

type prefix = `acme_corp`
type BucketName = Lowercase<`${prefix}_${string}`>;

We can create a custom Bucket subclass that expects a name matching our new BucketName type.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

type prefix = `acme_corp`
type BucketName = Lowercase<`${prefix}_${string}`>;

class AcmeCorpBucket extends aws.s3.Bucket {
 constructor(name: BucketName, args?: aws.s3.BucketArgs, opts?: pulumi.CustomResourceOptions) {
 super(name, args, opts);
 }
}

const myBucketWrong = new AcmeCorpBucket("MyBucket");
// ❌ Fails typechecking with: Argument of type '"MyBucket"' is not assignable to parameter of type '`acme_corp_${Lowercase<string>}`'.ts(2345)

const myBucketWrongAgain = new AcmeCorpBucket("acme_corp_MyBucket");
// ❌ Fails typechecking with: Argument of type '"acme_corp_MyBucket"' is not assignable to parameter of type '`acme_corp_${Lowercase<string>}`'.ts(2345)

const myBucketCorrect = new AcmeCorpBucket("acme_corp_my_bucket");
// ✅ Passes typechecking

This example shows just one of Typescript’s many new features that have shipped since version 3.8, and now all are available for you to use in your Pulumi programs without jumping through hoops.

Conclusion

We recently shipped improved support for Node.js monorepos, and with the latest version of the Pulumi Node.js SDK, we further improved how Pulumi integrates into your development environment by letting you easily pick the version of TypeScript to use.

Node.js Native Binary Compilation Using vercel/pkg

Daniel Bradley — Tue, 13 Sep 2022 00:00:00 +0000

In Pulumi’s engineering department, we often build and distribute tools as native binaries to avoid the need for additional dependencies on user machines. Most of these tools are written in Go, which has good support for building self-contained binaries that target modern operating systems.

While other Pulumi-supported languages like Node.js, Python, and .NET require additional runtime dependencies, it’s possible to bundle dependencies with the program. In this article, we’ll show you how to do that for a Node.js program.

This is a problem that can be solved using vercel/pkg command line tool. Here’s the summary from their readme.

This command line interface enables you to package your Node.js project into an executable that can be run even on devices without Node.js installed.

Let’s take a look at how to use pkg and some issues we encountered on the way.

Setting up `pkg`

1 - Install

pkg is distributed as an npm package which can be installed into your “devDependencies” using:

npm install -D pkg

yarn add -D pkg

or run without installing with npx:

npx pkg [args]

2 - Set `bin` in `package.json`

pkg will use the bin field from your package.json to find the entry point so you just have to specify a path to the directory containing your package.json.

{
 "name": "my-program"
 "bin": "bin/index.js",
 ...
}

If no target is specified, then a set of defaults will be chosen for you. If the output path is not specified, pkg will infer the name from the package.json “name” field and write to the current working directory.

3 - Execute

The main inputs that pkg needs is:

The entry point to your program for packaging
The target machine to build for
The output path to write the finished binary

Here’s an example of building the project in the current directory using node v18 for macOS ARM architectures:

# pkg [options] <input>
# -t, --targets comma-separated list of targets
# -o, --output output file name or template for several files
pkg -t node18-macos-arm64 -o bin/my-program .

It’s possible to specify multiple targets in a comma-separated list to build them all at the same time, but it does come with the limitation where the output file names follow a fixed pattern. Instead, we chose to just run the pkg command multiple times with different arguments from our makefile in parallel as this fits with our existing workflows well.

Issues encountered

“Inspector Not Available”

As soon as we started executing the provider we started seeing some interesting warnings printed to the console stating “Inspector is not available” (here’s our tracking issue and some pkg discussion). This is because we use the Node.js Inspector API as part of our automatic closure serialization, however this is not available by default when packaging with pkg.

pkg provides an option to fix this by building your own base image with custom Node.js flags set to enable debugging. However, on investigation, these issues were caused by the Pulumi Typescript SDK creating and caching an inspector instance at the point of being imported even though we never actually call this code in our plugin. Therefore, we opted to make this eager singleton creation to be lazy – only created on first use, as we’re not using in our providers at this time.

Unrunnable MacOS ARM binaries

When trying to use the binaries produced by our CI, we found that the binaries weren’t runnable on MacOS ARM architectures - and were forcibly killed by the operating system.

This led me down quite a bit of a rabbit hole investigating signing of binaries, but it was actually resolved by simply installing the ‘ldid’ tool on our linux CI environments. The ldid source is available via git.saurik.com/ldid and binaries are available from various sources. Our solution was to use the “Install Ldid” GitHub Action to install the ldid binary and add it to the PATH in our CI workflow. My learning here is that sometimes reading the warnings in logs more carefully can save you lots of time!

Static Binaries

One adjustment to our configuration came from a community contribution by @afreakk where the provider was being used in a nixos environment. Nixos adds the requirement for all binaries to be static rather than dynamic - so there’s no requirement for the operating system to dynamically map link functions from system libraries at runtime. Statically compiled programs sometimes result in a larger size, but avoid any possible issues with different versions of the libraries it depends on.

To resolve this issue, it’s as simple as changing the ‘linux’ targets to ‘linuxstatic’. E.g. node18-linux-amd64 becomes node18-linuxstatic-amd64.

Build infrastructure with Node.js

Pulumi lets you define, deploy, and manage cloud infrastructure using TypeScript and JavaScript, with the same npm packages and tooling you already rely on. Start building your first stack in Pulumi Cloud.

Get started

Multi-platform builds with a makefile

We use makefiles to build our providers, so here’s a brief outline of how we build for multiple platforms using GNU Make.

# Set the correct pkg TARGET for each binary we build
# when building for linux-amd64, set the pkg target to node18-linuxstatic-x64
bin/linux-amd64/my-program: TARGET := node18-linuxstatic-x64
# output binary file ^ ^ variable ^ pkg target
bin/linux-arm64/my-program: TARGET := node18-linuxstatic-arm64
bin/darwin-amd64/my-program: TARGET := node18-macos-x64
bin/darwin-arm64/my-program: TARGET := node18-macos-arm64
bin/windows-amd64/my-program.exe: TARGET := node18-win-x64

# Wildcard rule to build any of binary outputs
# "To build any bin file, ensure node_modules are up to date..."
bin/%: node_modules
 # "... then run pkg for actual output name & target"
 yarn run pkg . --target ${TARGET} --output $@
 # "TARGET" is the variable defined above, depending on the output
 # "$@" is the current makefile target - e.g. 'bin/linux-amd64/my-program'

# Running `make bins` will build all the listed outputs
bins: bin/linux-amd64/my-program
bins: bin/linux-arm64/my-program
bins: bin/darwin-amd64/my-program
bins: bin/darwin-arm64/my-program
bins: bin/windows-amd64/my-program.exe

# Let Make know that `bins` doesn't really exist (is phony) - it's just a helpful shortcut
.PHONY: bins

In summary the above code does the following:

Define each output file we need
- Set the correct pkg TARGET for each output
Define a rule for building any binary output:
- Ensure module_modules are up-to-date (this is another make target not shown here).
- Run pkg with the $TARGET set in (1) and the name of the current output ($@).
Define bins “phony target” which builds all listed bins.

You might also like:

Recap

I hope this gives you a good overview of the vercel/pkg tool and how you can use it to create standalone programs using Node.js.

Historical Note

We used to do a similar process using the nexe project, but there’s been no releases since 2017 and therefore no support for newer versions of Node.js and we therefore consider this package as unmaintained at this point in time.

Pulumi Blog: Nodejs

Next-level IaC: Breakpoint Debugging for Pulumi Programs

How Breakpoint Debugging Works

Example: Attaching a debugger to a TypeScript Pulumi program in VS Code

Step 1: Create a Pulumi project from a template

Step 2: Lets break stuff!

Step 3: Create a debug wrapper script

Step 4: Configure VS Code launch.json and tasks.json

Step 5: Setting a debugging breakpoint

Considerations and Limitations

Next Steps

Pulumi Cloud

Next-level IaC: Powering Up Pulumi with Package Ecosystems

Use Case: Determining S3 MIME types on-the-fly

The possibilities are endless

Next Steps

Pulumi Cloud

Next-level IaC: Drop those wrapper scripts and let your language do that for you

Run pre-deployment tasks with Pulumi

Fetch data to generate resources dynamically

Hook the event loop to run tasks after deployment

Wrapping up

Next Steps

Pulumi Cloud

Run Pulumi with Any TypeScript Version

Example

Conclusion

Node.js Native Binary Compilation Using vercel/pkg

Setting up pkg

1 - Install

2 - Set bin in package.json

3 - Execute

Issues encountered

“Inspector Not Available”

Unrunnable MacOS ARM binaries

Static Binaries

Multi-platform builds with a makefile

Recap

Historical Note

Step 4: Configure VS Code `launch.json` and `tasks.json`

Setting up `pkg`

2 - Set `bin` in `package.json`