Pulumi Release Notes: Colorized Stack Traces, Output Methods for JSON, OIDC integration, and much more!

Posted on

Mikhail Shilkov Mikhail Shilkov Meagan Cojocar Meagan Cojocar

We have been busy shipping improvements in the last 2 months. Let’s walk through the release highlights across Pulumi engineering areas from January and February. If you want to learn more between release blogs, follow the CLI improvements in the pulumi/pulumi repo changelog and Pulumi Service features in the new features blogs.

Quick shortcuts to jump to your favorite area:

Pulumi CLI

Colorized stack traces for Node.js errors

When an error happens in a Pulumi program, stack traces can be instrumental to understand the reasons behind the failure. Stack traces can often be large and contain frames unrelated to user code, which makes debugging harder.

We have now changed the CLI to display Node.js stack traces with colors, where the important bits are highlighted and less important parts faded out. The new display technique helps draw attention to relevant details and is similar to running vanilla node programs.

Colorized stack traces

πŸ‘‰ Learn more in the Emit colorized stack traces pull request.

New –shell option for pulumi stack output

We added a new --shell option to the pulumi stack output command that writes the outputs for a stack as a shell script:

% pulumi stack output --shell
bucketName=mybucket-1234
websiteURL="http://mybucket-1234.example.com"

On Windows, it will produce a Powershell script:

% pulumi stack output --shell
$bucketName = 'mybucket-1234'
$websiteURL = 'http://mybucket-1234.example.com'

As a result, Pulumi users are now able to write shell scripts like the one below:

eval "$(pulumi stack output --shell)"
echo "Created bucket $bucketName"
# => Created bucket mybucket-1234

echo "Visit at $websiteURL"
# => Visit at http://mybucket-1234.example.com

πŸ‘‰ Learn more in the Add –shell option pull request.

pulumi new in directories with source control

By default, pulumi new won’t run unless the current directory is empty to avoid overriding any existing files. This behavior meant that the command failed in directories with initialized source control.

We shipped an improvement where Pulumi CLI will ignore .git, .hg, and .bzr files and directories when considering whether a directory is empty.

πŸ‘‰ Learn more in the Ignore VCS directories pull request.

Support plugins hosted in GitLab

Pulumi comes with a wealth of packages but users can also create and use custom providers and components. By default, Pulumi expects the binaries to be hosted on GitHub. On top of that, the CLI now supports GitLab as a first-class source of package downloads.

If your package binary is hosted on GitLab, you can now set its pluginDownloadURL property (see Publish your package) to point to your GitLab project, e.g. to gitlab://gitlab.com/43429536.

πŸ‘‰ Learn more in the Support custom plugins from other sources issue.

Use access tokens to authenticate to Google Cloud backend

When storing the state files in Google Cloud backend, you can now authenticate to the Google Cloud Storage using Google access tokens. If the environment variable GOOGLE_OAUTH_ACCESS_TOKEN is set during a pulumi login gs://..., Pulumi will utilize the provided token to login to the bucket, assuming this has not been pre-empted by another authentication type.

πŸ‘‰ Learn more in the Pulumi login to support google oauth access tokens for GCS buckets pull request.

Delete temporary files from plugin downloads

While a plugin is being installed, the CLI downloads the necessary archive and stores the archive in /tmp/pulumi-plugin-tar*. The CLI is now more mindful of cleaning the temporary file after plugin installation completes to save disk space.

πŸ‘‰ Learn more in the Delete temporary files from plugin downloads pull request.

“Yes, using Update Plans” prompt removed

Update Plans is a preview feature to constrain the update only to the operations that were planned during preview. To get extra feedback from our users, we experimented with showing an extra option “yes, using Update Plans” at the update prompt. The experiment is now complete, so we removed the option.

Lean more in the Remove the “yes, using Update Plans” prompt issue.

Pulumi Language SDKs

Output methods to serialize and deserialize JSON

Building JSON from values and parsing strings into JSON are two of the most common operations in Pulumi programs. All programming languages have built-in libraries to do so. However, quite often JSON operations need to be combined with Pulumi Inputs and Outputs, which required writing a cumbersome combination of Apply and serialization method calls.

To streamline this experience, Pulumi SDKs shipped helper serialization and deserialization functions that undertand the type Output<T> natively. Here is a simple usage example in Python, where table.arn is an output coming from another resource:

policy = iam.Policy(
    "iam-policy",
    policy= Output.json_dumps({
        "Version": "2012-10-17",
        "Statement": [{
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:GetItem"
            ],
            "Effect": "Allow",
            "Resource": [table.arn]
        }]
    }))

πŸ‘‰ Learn more in the Output methods issue that also tracks a number of futher additions to the toolset of helper methods.

Retrieve stack reference outputs as plain values

You can read outputs from one stack in another stack’s program using Stack References. In the past, the resulting values were always wrapped inside an Output<T> container in order to preserve secret values (when needed).

With recent SDKs, we shipped an alternative method StackReference.getOutputDetails. This method returns a plain object with two fields: value and secretValue. At most one of these fields is set, depending on whether the stack reference output is a secret or not.

If you know that the output is not a secret, you can write a program to use the plain value immediately, for example, to iterate though a list in a loop and create resources in each iteration.

πŸ‘‰ Learn more for each language:

Native arm64 binaries in Python SDK

Pulumi Python SDK has been upgraded to use the grpcio library version 1.51.3, which comes with a wheel with native arm64 binaries (universal2). The change eases the installation of the Pulumi Python SDK on Apple M1/M2 machines.

πŸ‘‰ Learn more in the Python grpc 1.51.3 contains universal2 binaries pull request.

Go version raised to 1.18

The minimum version of Go required by Pulumi was raised to 1.18. Raising this directive allows us to use language features like generics in the codbase. The change is applied to both pkg and sdk.

πŸ‘‰ Learn more in the go.mod: Raise to ‘go 1.18 pull request.

Delegate alias computation to the engine

Sometimes, two classes in Pulumi SDKs represent the same cloud resource. To enable migration from one to another, Pulumi sets aliases for those classes and declares them to be compatible.

In the past, each SDK was responsible for calculating all possible alias combinations for a given resource and its inherited parent. We recently added support in the engine for this computation along with an optimization to reduce complexity from a multiplicative factor to a linear one.

As a follow-up step, we updated our SDKs to only provide the base aliases, and allow the engine to do this computation. This enables smaller SDKs and faster runtime performance, especially for longer chains of parented resources. Other than that, the user experience hasn’t changed.

πŸ‘‰ Learn more in the Delegate alias computation from SDKs to Engine issue.

Pulumi Automation API

Adding tags to stacks through Automation API

You can now use Automation API to get, set, list, or remove tags on stacks programmatically. Here is a quick example in TypeScript:

const workspace = await LocalWorkspace.create({
    projectSettings: projectSettings,
});
await workspace.createStack(stackName);
await workspace.setTag(stackName, "foo", "bar");
const tags = await workspace.listTags(stackName);
// tags will contain tags for `pulumi:project`, `pulumi:runtime`, and `foo` keys.

πŸ‘‰ Learn more in the Add support for adding tags to stacks through Automation API issue.

Support cloning from Azure DevOps in Automation API

We implemented a workaround to support Azure DevOps repositories when driving Pulumi from Automation API in Go. This is now also utilized in the Pulumi Kubernetes Operator.

πŸ‘‰ Learn more in the Support clones from Azure DevOps pull request.

Cloud Providers and Packages

Every week we keep shipping provider updates. Two highlights on our native providers:

  • Azure Native shipped 20 new resources, including support for networkcloud and voiceservices (changelog)
  • AWS Native shipped 30 new resources in apigatewayv2, appflow, applicationautoscaling, cloudtrail, connect, directoryservice, ec2, fms, gamecast, gamelift, kendraranking, networkmanager, omics, organizations, sagemaker, secretsmanager, simspaceweaver, and systemsmanagersap modules (changelog)

Five new community packages were added to the Registry: Vultr, Zscaler Private Access (ZPA), Zscaler Internet Access (ZIA), Statuscake, and Nuage.

Pulumi Service & Pulumi.com

After launching Pulumi Deployments a few months ago and receiving a ton of community feedback (thank you!) we launched an integration with OpenID Connect (OIDC) to enable security first deployments and industry best practices.

OpenID Connect (OIDC) is an identity layer built on top of the OAuth 2.0 framework. It allows third-party applications to verify the identity of the end-user and to obtain basic user profile information. OIDC uses JSON web tokens (JWTs), which you can obtain using flows conforming to the OAuth 2.0 specifications.

Overview

The main benefits of using the OIDC integration for Pulumi Deployments are as follows:

  • Temporary credentials: You can move away from long term credentials within your organization to leverage temporary credentials that are generated every time a deployment runs. The Pulumi Service issues a new OIDC token that is limited to that deployment. This token is used to obtain cloud credentials that automatically expire, meaning you do not need to store sensitive credentials that are reused with each deployment.

  • Granular access control: The OIDC token generated by the Pulumi Service contains detailed information about the deployment with which it is associated. In AWS for example, you can configure our OIDC integration to only use temporary credentials if that token is associated with a particular Pulumi stack or Pulumi organization. This enables you to follow the principle of least privilege access and scope access down to only what is needed. For AWS, Azure, and GCP, Pulumi Deployments can automatically exchange its OIDC token for temporary credentials and make those credentials available to providers. For advanced scenarios or other cloud providers, the OIDC token can be manually exchanged for credentials by the user.

πŸ‘‰ Learn more on how to get started with Pulumi Deployments OIDC by reading the blog post.

If you haven’t yet tried out Pulumi Deployments

You should now see a β€œDeploy” tab under Stack Settings in the Pulumi Service console and have access to the Deployments REST API. View the documentation for help getting started and join our Slack (#pulumi-deployments channel) for questions and feedback.

Share this post

Introducing Drift Detection, TTL Stacks, and Scheduled Deployments. Learn More.