Pulumi Blog: Fraser Waters

Introducing pulumi do: Direct Resource Operations for Any Cloud

Fraser Waters — Fri, 22 May 2026 00:00:00 +0000

Infrastructure as code is the right model for production systems. State tracking, drift detection, and repeatable deployments all matter when you’re managing real workloads.

But sometimes, you also need a quick, one-off interaction with the cloud: create a bucket or a database, look up a VPC, delete a stray resource.

Today we’re introducing pulumi do, a new command for direct resource operations. With pulumi do, you can create, read, update, delete, and query any cloud resource from the terminal with a single command, across thousands of Pulumi-supported providers — no project, code, or state required.

The problem: Sometimes IaC is more than you need

When you’re managing production workloads, IaC is the proven solution. Code lets you declare complex systems, state tracking catches drift before it becomes a problem, dependency graphs sequence changes safely, and policy keeps everything in bounds. That full lifecycle, especially with the backing of a platform like Pulumi Cloud, is exactly what you want to build systems that scale.

But when you (or your coding agent) need an ad-hoc Postgres database, the simplest path with IaC still takes several steps: make a directory, create a project, configure your credentials, write the code, preview, deploy. It works, but it’s not always necessary for what should be a simple operation. pulumi do collapses all of those steps into one, using the same Pulumi providers, resource model, and ecosystem that powers the core Pulumi platform.

Resource creation is also only part of the problem. As Joe laid out in The Agentic Infrastructure Era, the real challenge for AI agents isn’t with code or CLI commands, it’s with everything else: getting a cloud account, resolving credentials, wiring configuration across multiple services. Agent accounts, also released this week, simplify this by letting an agent provision its own ephemeral Pulumi Cloud account, and Pulumi ESC takes care of consolidating credentials across providers. Together, with pulumi do, agents can now go from zero to deployed infrastructure without requiring a human in the loop — and when that one-off resource needs to grow into a more permanent system, there’s a clear graduation path back to full Pulumi IaC.

What it looks like

As an example, say you wanted to provision an S3 bucket. With the AWS CLI, you’d need to assemble an aws s3api create-bucket invocation with the right set of command-line flags, region constraints, a globally unique name, and so on. With pulumi do, it’s just this:

$ pulumi do aws:s3:Bucket create

That might not look all that different on the surface — but because you’re using the Pulumi engine and resource model, you can provide a minimal set of input properties, take advantage of provider-defined defaults, and use Pulumi’s auto-naming feature to give the bucket a unique name automatically:

$ pulumi do aws:s3:Bucket create

This will create aws:s3/bucket:Bucket with the following inputs:
{
 "bucket": "bucket-279ea56",
 "tagsAll": {}
}

Please confirm that this is what you'd like to do by typing `yes`:

Answer yes (or just pass --yes), and you’re done. To delete the bucket:

$ pulumi do aws:s3:Bucket delete bucket-279ea56 --yes

Need to look up an existing resource? Use a provider function:

$ pulumi do aws:ec2:getVpc --default

{
 "arn": "arn:aws:ec2:us-west-2:663782525873:vpc/vpc-d7b311af",
 "cidrBlock": "172.31.0.0/16",
 "enableDnsHostnames": true,
 "enableDnsSupport": true,
 "enableNetworkAddressUsageMetrics": false,
 "id": "vpc-d7b311af",
 ...
}

Same CLI, same output contract, same provider ecosystem.

The command shape

The do command accepts a Pulumi resource type, or type token, to determine the action to take. Type tokens have the form <package:module:resource>. For example, aws:s3:Bucket refers to the Amazon S3 Bucket resource that belongs to the s3 module of the aws package.

You can also provide a portion of the token to help you find what you’re looking for without ever having to leave the terminal:

$ pulumi do aws:s3

Functions and resources for the s3 module.

Run 'pulumi do <module/resource/function> --help' for more details on usage.

Functions:
 aws:s3:getAccessPoint
 aws:s3:getAccountPublicAccessBlock
 aws:s3:getBucket
 aws:s3:getBucketObject
 ...

Resources:
 aws:s3:AccessPoint
 aws:s3:AccountPublicAccessBlock
 aws:s3:AnalyticsConfiguration
 aws:s3:Bucket
 ...

$ pulumi do aws:s3:Bucket read bucket-d20976f

{
 "arn": "arn:aws:s3:::bucket-d20976f",
 "bucket": "bucket-d20976f",
 "bucketDomainName": "bucket-d20976f.s3.amazonaws.com",
 "bucketNamespace": "global",
 ...
}

The package, module, and resource/function segments all come directly from the Pulumi provider schema, so --help works at every level of the tree. Pass a package name, optional module, and optional function or resource type, and do returns the appropriate level of detail.

You can also provide the input properties of a resource in a YAML or JSON file with the --input option. To create a container service in Google Cloud Run for example:

# service.yaml
location: us-central1
deletionProtection: false
template:
 containers:
 - image: us-docker.pkg.dev/cloudrun/container/hello

$ pulumi do gcp:cloudrunv2:Service create \
 --input yaml \
 --input-file service.yaml

This will create gcp:cloudrunv2/service:Service with the following inputs:
{
 "deletionProtection": false,
 "location": "us-central1",
 "name": "service-b8af752",
 "template": {
 "containers": [
 {
 "image": "us-docker.pkg.dev/cloudrun/container/hello"
 }
 ]
 }
}

The result:

{
 "createTime": "2026-05-22T23:00:22.415839Z",
 ...
 "urls": [
 "https://service-b8af752-921927215178.us-central1.run.app",
 "https://service-b8af752-ctnulmzwoa-uc.a.run.app"
 ]
}

Resource operations

Most resources support the full set of CRUD operations — create, read, update, delete, and list — directly from the CLI. Each operation maps to a provider CRUD method using the same provider logic a full Pulumi program would use, and resources are addressable by their cloud provider IDs:

# Create a resource
$ pulumi do aws:s3:Bucket create --yes | jq -r ".name"
bucket-4f5cb22

# Fetch it
$ pulumi do aws:s3:Bucket read bucket-4f5cb22 | jq -r ".hostedZoneId"
Z3BJ6K6RIION7M

# Update/patch it
$ pulumi do aws:s3:Bucket patch bucket-4f5cb22 --input yaml --input-file tags.yaml

$ pulumi do aws:s3:Bucket read bucket-4f5cb22 | jq ".tags"
{
 "key": "value"
}

# Delete it
$ pulumi do aws:s3:Bucket delete bucket-4f5cb22

Provider configuration

Today, pulumi do resolves provider configuration — for example, applying your AWS credentials — using environment variables or credential files as supported by each individual Pulumi provider. See the Pulumi Registry for provider-specific configuration details.

Designed for humans and agents

We’ve designed pulumi do to serve humans and coding agents equally well, guided by three fundamental ideas:

Consistent command structure across every provider. The do <package:module:type> <operation> pattern is the same for AWS, Azure, Google Cloud, Kubernetes, Cloudflare, Datadog, and every provider, including packages containing higher-level component resources. Once an agent learns that pattern, it applies across the board.
Predictable output contract. JSON on stdout, progress on stderr, consistent exit codes. An agent can parse the result programmatically without scraping human-formatted tables.
A single CLI command that works across every cloud. Many cloud and SaaS providers don’t have a full CLI at all. pulumi do generates commands from the provider schema, so if a Pulumi provider exists for it, the CLI just works. Neither humans nor agents need to install, learn, or even know about cloud provider-specific tooling.

What’s next

Resource operations and provider functions are the foundation. The pulumi do roadmap extends the same direct-operation model with credential management, state tracking, and a path to full IaC.

Unified credentials with Pulumi ESC

One of the hardest parts of multi-cloud operations is credential management. Every provider has its own authentication scheme, environment variables, and session lifecycle. An agent working across AWS, Cloudflare, and Datadog today manages three separate credential mechanisms.

We’re building Pulumi ESC integration into pulumi do so you can manage credentials in one place and resolve them everywhere. ESC handles credential resolution (including OIDC-based dynamic credential generation and short-lived tokens) across all of your providers. Name the credential set, reference it, and ESC does the rest, with rotation, RBAC, and audit built in.

Cross-resource references

Real infrastructure has dependencies — subnets need VPCs, security group rules need their security groups, and so on. When you’re building resources one at a time, those references need to flow between commands somehow.

A future version of pulumi do will let resource inputs reference outputs from previously created resources, allowing the CLI to resolve them automatically and preserve the dependency graph. Later, when the time comes to graduate to a full IaC program, the generated code contains proper resource references rather than hard-coded strings.

Stateful mode and the graduation path

Today, pulumi do is stateless. Each command runs independently. A planned stateful mode will persist resource state across operations, enabling drift detection, lifecycle management, and a graduation path to full infrastructure as code.

Here’s what we’re planning:

Zero setup. Your first pulumi do implicitly creates a project and stack. No manual initialization.
Accumulate resources. Each operation stores resource state. After a few commands, you have a lightweight representation of your infrastructure.
Eject to a full project. When the time comes, generate a Pulumi project in your chosen language with all resources imported and dependency graphs intact.
Connect to Pulumi Cloud. Layer on governance, compliance, team collaboration, and deployment automation through Pulumi Cloud. Resources created via pulumi do can be governed by Pulumi Insights from day one, even before you opt into full IaC.

This path works because pulumi do uses the same providers, resource types, and property schemas as every other pulumi operation. Provisioned cloud resources stay where they are as management capabilities are added as needed.

Get started

pulumi do ships as a research preview in Pulumi CLI v3.242.0 and later. Install or update the CLI, install a provider plugin, and start running commands. The documentation has the full reference.

We can’t wait to hear your feedback. Give it a try today, tell us what works (and what doesn’t), and help shape the CLI that agents and humans both reach for first.

Introducing the pulumi policy analyze Command for Existing Stacks

Fraser Waters — Fri, 03 Apr 2026 00:00:00 +0000

You can now run policy packs against your existing stack state without running your Pulumi program or making provider calls. The new pulumi policy analyze command evaluates your current infrastructure against local policy packs directly, turning policy validation into a fast, repeatable check.

Why this command matters

Policy authoring and policy updates usually involve an iteration loop:

Make a policy change.
Run a policy check.
Inspect violations or remediations.
Repeat until the policy behavior matches intent.

Before this command, that loop often depended on pulumi preview or pulumi up, which can be heavier than you need when your goal is validating policy logic against known state.

With pulumi policy analyze, you can evaluate your current stack state directly and quickly.

Basic usage

At minimum, provide a policy pack path and optionally a stack:

pulumi policy analyze \
 --policy-pack ./policy-pack \
 --stack dev

You can also pass a config file for each policy pack:

pulumi policy analyze \
 --policy-pack ./policy-pack \
 --policy-pack-config ./policy-config.dev.json \
 --stack dev

If any mandatory policy violations are found, the command exits non-zero.

If remediation policies fire, those changes are reported in output, but stack state is not modified.

Testing new policy packs as a developer

For policy pack development, this command is useful as a tight local feedback loop:

Pick a representative stack (dev, staging, or a fixture stack).
Run pulumi policy analyze against that stack after each policy change.
Use the output to verify mandatory, advisory, and remediation behavior.
Repeat before publishing the policy pack or attaching it to broader policy groups.

Two output modes are especially useful:

--diff for a concise, human-readable view while iterating locally.
--json for structured output that can be consumed in scripts and CI.

Using it in AI and agent workflows

This command is also a good primitive for AI-assisted policy workflows.

Because pulumi policy analyze can emit JSON and a clear process exit code, agents can use it for deterministic policy evaluation steps:

Propose or edit policy rules.
Run pulumi policy analyze --json against target stacks.
Parse violations and remediation signals.
Suggest policy fixes, config adjustments, or targeted infrastructure changes.
Re-run analysis until mandatory violations are resolved.

For example, an agent tasked with fixing a policy violation can run pulumi policy analyze --json to get a structured list of violations, identify which resources are non-compliant, generate targeted infrastructure changes, then re-run analysis to confirm the violations are resolved, all without triggering a full preview on each iteration. The same loop works for policy authoring: an agent can propose a new policy rule, test it against several representative stacks, and surface unintended violations before the rule is published.

This works well for automation because the command doesn’t execute your Pulumi program or make provider calls, so there are no side effects or runtime variance between runs. The JSON output and non-zero exit code on failure give agents a clear pass/fail contract to build on.

Try it out

pulumi policy analyze is available in Pulumi v3.229.0.

If you are authoring or tuning policy packs, start by running this command against a known stack in your environment. It is a quick way to validate policy behavior before rollout.

For implementation details, see the merged PR: pulumi/pulumi#22250.

Get started with policy as code

Introducing the Stash Resource in Pulumi IaC

Fraser Waters — Mon, 12 Jan 2026 00:00:00 +0000

We’re excited to announce the Stash resource, a new built-in Pulumi resource that lets you save arbitrary values directly to your stack’s state. Whether you need to capture a computed result, record who first deployed your infrastructure, or persist configuration that should remain stable across updates, Stash provides a simpler and more ergonomic solution.

Why Stash?

Infrastructure code often produces values that need to persist beyond a single deployment. Maybe you’re generating a random identifier that should stay consistent, tracking which team member initially set up a stack, or recording a timestamp from the first deployment. Previously, you’d need workarounds like external storage, custom resources, or careful state manipulation.

The Stash resource helps with that. It takes an input value, stores it in your stack’s state, and makes it available as an output property. The output property preserves the original value even when the input changes in subsequent deployments, making Stash perfect for “first-run” scenarios where you want to capture and preserve a value from the initial deployment.

Using Stash

Creating a Stash is straightforward. Here’s how you’d capture the username of whoever first deploys the stack (using Node.js):

import * as pulumi from "@pulumi/pulumi";
import * as os from "os"; // Node.js built-in module

const firstDeployer = new pulumi.Stash("firstDeployer", {
 input: os.userInfo().username,
});

export const originalDeployer = firstDeployer.output;

The first time this runs, both input and output will show the current user. On subsequent deployments by different users, input will update to show the new user, but output will continue returning the original deployer’s name.

Stash supports any value type—strings, numbers, objects, arrays, and nested structures. It also respects secret annotations, so if you stash a secret value, it stays encrypted in your state.

When to replace

Since Stash preserves the original value by design, updating the stored value requires a replacement. You have several options:

Use --target-replace during pulumi up:

pulumi up --target-replace urn:pulumi:dev::my-project::pulumi:index:Stash::firstDeployer

Run pulumi state taint to mark the resource for replacement:

pulumi state taint urn:pulumi:dev::my-project::pulumi:index:Stash::firstDeployer

Use the replacementTrigger resource option to automate replacements based on value changes:

import * as pulumi from "@pulumi/pulumi";
import * as os from "os";

const remoteConfig = fetch("https://example.com/my-service").then(response => response.json())

const myStash = new pulumi.Stash("myStash",
 {
 input: os.userInfo().username,
 }, {
 replacementTrigger: remoteConfig.someValue,
 }
);

export const stashedValue = myStash.output;

With replacementTrigger, when remoteConfig.someValue changes, the Stash resource will be replaced and the new input value will be captured.

Learn more

The Stash resource is available now in Pulumi v3.208.0 and later across all supported languages. Check out the Stash documentation for detailed examples in TypeScript, Python, Go, .NET, and YAML.

We’d love to hear how you’re using Stash and any feedback you have on it! Share your use cases in our Community Slack or open an issue if you see any on pulumi/pulumi.

Happy hacking!

Introducing Resource Transforms: Enhancing Flexibility for Packaged Component Resources

Fraser Waters — Fri, 19 Jul 2024 00:00:00 +0000

Pulumi has supported a Transformations system for a number of years now. This has proved to be a powerful and flexible escape hatch for modifying resource properties and options across your entire program. For example, you could use Transformations to automatically apply tags to all taggable resources in your program, including the children of component resources.

However, there is one major limitation with the existing Transformations system: it isn’t able to transform the children of packaged component resources, such as those in awsx and eks. This limitation is due to the fact that packaged component resources are created in a separate provider process and Transformations only work with resources created in your program’s process.

To address this limitation we’re introducing a new system called Transforms, which works with all resources, including packaged component resources and their children. The new Transforms system is intended to fully replace the old Transformations system (we plan to deprecate the old system in the future).

Here’s an example that demonstrates adding tags to all Subnet child resources created by the Vpc packaged component:

import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx";

const myTags = { test: "TEST" };

pulumi.runtime.registerResourceTransform(args => {
 if (args.type === "aws:ec2/subnet:Subnet") {
 const tags = pulumi.output(args.props.tags || {});
 args.props.tags = tags.apply(t => ({ ...t, ...myTags }));
 return { props: args.props, opts: args.opts };
 }
 return undefined;
});

const vpc = new awsx.ec2.Vpc("test");

import pulumi
from pulumi_awsx import ec2

my_tags = { "test": "TEST" }

def tag_transform(args):
 if args.type_ == "aws:ec2/subnet:Subnet":
 tags = pulumi.Output.from_input(args.props.get("tags", {}))
 args.props["tags"] = tags.apply(lambda v: {**v, **my_tags})
 return pulumi.ResourceTransformResult(args.props, args.opts)

pulumi.runtime.register_resource_transform(tag_transform)

my_vpc = ec2.Vpc("my-vpc")

package main

import (
 "context"

 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 myTags := map[string]string{"test": "TEST"}

 err := ctx.RegisterResourceTransform(func(_ context.Context, args *pulumi.ResourceTransformArgs) *pulumi.ResourceTransformResult {
 if args.Type == "aws:ec2/subnet:Subnet" {
 tags, ok := args.Props["tags"].(pulumi.StringMapInput)
 if !ok {
 tags = pulumi.StringMap{}
 }
 args.Props["tags"] = tags.ToStringMapOutput().ApplyT(func(m map[string]string) map[string]string {
 result := make(map[string]string)
 for k, v := range m {
 result[k] = v
 }
 for k, v := range myTags {
 result[k] = v
 }
 return result
 })
 return &pulumi.ResourceTransformResult{
 Props: args.Props,
 Opts: args.Opts,
 }
 }
 return nil
 })
 if err != nil {
 return err
 }

 _, err = ec2.NewVpc(ctx, "test", nil)
 if err != nil {
 return err
 }

 return nil
 })
}

using System;
using System.Collections.Immutable;
using Pulumi;
using Pulumi.Awsx.Ec2;

await Deployment.RunAsync<MyStack>();

class MyStack : Stack
{
 public MyStack() : base(new StackOptions
 {
 ResourceTransforms =
 {
 async (args, _) => {
 var myTags = ImmutableDictionary.Create<string, object>()
 .Add("test", "Test");

 if (args.Type == "aws:ec2/subnet:Subnet")
 {
 InputMap<object> tags =
 args.Args.TryGetValue("tags", out var tagsValue) && tagsValue is not null ?
 tagsValue is Output<ImmutableDictionary<string, object>> tagsOutput ? tagsOutput :
 tagsValue is ImmutableDictionary<string, object> tagsDictionary ? tagsDictionary :
 throw new InvalidOperationException($"Unexpected tags type: {tagsValue.GetType()}") :
 ImmutableDictionary<string, object>.Empty;

 tags = tags.Apply(t => t.SetItems(myTags));
 return new(args.Args.SetItem("tags", tags), args.Options);
 }
 return null;
 },
 }
 })
 {
 var vpc = new Vpc("test");
 }
}

Using the new Transforms system is similar to the old system, but there are some differences between the two systems. A guide on Migrating from Transformations to Transforms is available to help transition to the new system.

We’re excited to make it possible to transform all resources in your program, including those from packaged components. If you have any questions or feedback, please don’t hesitate to reach out to us on Slack.

Converting Full Terraform States to Pulumi

Fraser Waters — Wed, 25 Oct 2023 00:00:00 +0000

Building on our work of converting Terraform projects, we now have support for adopting resources from Terraform state.

In the prior blog post about converting from Terraform we called out that while it was great that the converter could migrate a Terraform project, but what if you had existing infrastructure resources.

Historically we’ve shared some AWS specific code for TypeScript and Go to import from .tfstate files (previously available in the tf2pulumi repository, now replaced by the Terraform converter plugin). These have been useful, but they’re limited to the AWS provider and either Go or TypeScript programs.

From v3.87.0 we have extended our converter system to also include state in addition to code. So now it’s possible to import resources from other state systems, the first of which is Terraform.

Converting a Real World Workspace

Terraform stores its state in workspaces. In the most simple case this is just a terraform.tfstate file in the directory with the code. For remote state it can be fetched with terraform state pull.

Given a terraform.tfstate file we can set up a Pulumi stack and tell it to import the resource from Terraform.

$ pulumi stack init
$ pulumi import --from terraform ./terraform.tfstate

Pulumi will try to convert the resources in the Terraform state into their matching Pulumi types and import strings (normally just the resource ID).

Some resources don’t use their ID as the import string. For some well known AWS resources the converter will handle this; for others, they might still cause an error at import time. However, if the import does fail (for any reason) the import file listing all the resources will be written out to the current directory, allowing you the opportunity to manually fix up issues with type tokens or import strings, and then try again using pulumi import --file.

We’re working on ensuring more types correctly translate, so feel free to open issues with any import errors you get while using this new converter to help us improve.

Get Started

Support for the new pulumi import --from terraform command is now available in v3.87.0 of the Pulumi CLI. Download the latest Pulumi CLI and give the new converter a try today. If you run into any issues, please let us know or reach out in the Pulumi Community Slack with any questions!

Preview of .NET resource providers

Fraser Waters — Wed, 18 Jan 2023 00:00:00 +0000

Today we are pleased to announce the Preview of .NET support for custom resource providers. This means you can build custom providers using your favorite .NET language, including C#, F#, and VB.NET.

This is a first look preview release of the provider interface for .NET. It has minimal documentation but is feature complete. We hope to gather early feedback to refine the API and quickly bring this to a production quality release.

You will need to use version 3.52.0 to access the new APIs. You can install this in your project with:

dotnet add package Pulumi --version 3.52.0

You can see an example of using the new provider interface in the dotnet integration tests directory. This has replaced the old test provider that was written in Go.

Please view the provider interface and raise issues and discussions on GitHub or our Community Slack in the #dotnet channel — we can’t wait to hear what you think!

New CLI prompt to use Update Plans

Fraser Waters — Wed, 23 Nov 2022 06:00:00 -0700

Earlier this year we announced the experimental introduction of Update Plans as we heard from many of you that you need a strong guarantee about exactly which changes an update will make to your infrastructure, especially in critical and production environments. We have been making steady progress on this feature and are excited to further integrate it into your workflows. In the latest release of the Pulumi CLI (v3.48.0), there’s a new prompt to use experimental Update Plans when running an update.

Why use Update Plans

Update Plans help catch any unexpected changes that might happen between when you preview a change and when you apply that change. Update Plans work by saving the results of a pulumi preview to a plan file, which enables you to restrict subsequent pulumi up operations to only the actions saved in the plan file. This helps you ensure that what you saw in the pulumi preview is what will actually happen when you run pulumi up.

With Update Plans, this same workflow is now possible with infrastructure managed by Pulumi. Assuming you already have CI set up for your pull/merge requests, you can run pulumi preview --save-plan PLAN-FILENAME and output the resulting plan file as a CI artifact. Then, update the CI workflow that runs when changes are merged to run pulumi up --plan PLAN-FILENAME.

Until now to leverage Update Plans users had to pass --save-plan <file> and --plan <file> options while using non-interactive CLI commands preview and up respectively with the PULUMI_EXPERIMENTAL=true environment variable. This restriction limited the feature discoverability, while enabling us to scale gradually. The --save-plan <file> and --plan <file> options continue to be available and still require a PULUMI_EXPERIMENTAL=true environment variable.

New option in CLI interactive prompt

We believe that Update Plans have matured to be evaluated by our larger community. To make it more visible, we are introducing a new option to the update confirmation prompt. If you run a pulumi up command interactively, you will now see the following choice:

Do you want to perform this update? [Use arrows to move, type to filter]
> [experimental] yes, using Update Plans (https://pulumi.com/updateplans)
 yes
 no
 details

The no option is still the default choice, and the yes option is still immediately above it, so the change will hopefully align with user habits. However, there is a top option added, which suggests running an update with a plan built during the preview.

Each interactive preview now calculates an Update Plan. If you choose the yes, using Update Plans option, the Update Plan will be automatically passed to the update command and applied during the deployment. Any inconsistencies between the preview and the update will be reported back to the user.

Try Update Plans

Update Plans are in public preview. We recommend using it only in non-critical, non-production scenarios during the preview period.

We’re eager for you to try the public preview of Update Plans and let us know what you think. To try out the new interactive option, make sure you’ve updated to Pulumi 3.48.0 or higher.

We’d love to hear your thoughts on the Update Plans feature! Feel free to ask questions in our Slack or open an issue on GitHub.

Simplify Configuration Management with Project-level Config

Fraser Waters — Wed, 02 Nov 2022 06:00:00 -0700

This blog post discusses project-level config before Pulumi ESC was released. ESC makes this and hierarchical config easy.

One of our most up-voted feature requests (with 78 thumbs ups) is to support hierarchical config. We’re happy to announce that we’ve now released the first part of plans to support this feature.

Pulumi will now allow you to set configuration values in your Pulumi.yaml file, using the given value as a default for all stacks in the project. While we expect even this first level of support will be incredibly useful to many people we also want to assure you that we have many more plans in place to make this feature even better.

How to Use Project-level Configuration

Pulumi has always had the concept of configuration per stack, and the changes for project level configuration haven’t fundamentally changed that. You can still set config for stacks via the pulumi config CLI command, or by editing the Pulumi.<stack>.yaml file. Project level config just gives you the ability to set default values for all stacks in your Pulumi.yaml.

A common request we get for this is making sure that all stacks use the same AWS region, or don’t have default providers enabled. You can now do this by setting these in your Pulumi.yaml.

name: project
runtime: go
config:
 pulumi:disable-default-providers: [“*”]
 aws:region: us-east-1

As you can see this allows setting default configuration values per-stack for configuring the engine or providers, but you can also set default values for project configuration. For example maybe you have a default count of the number of virtual machines to create:

name: project
runtime: go
config:
 vm-count: 2

The only caveat to mention about this system is that if you have object values they must be nested under a key value. So if you want to set the key obj to the object value { foo: "bar" } you would need to write it as such:

name: project
runtime: go
config:
 obj:
 value:
 foo: bar

This allows the engine to distinguish between values and schemas, which we will show below.

Defining Configuration Schemas in Pulumi

A related feature to defining values for config at the project level is being able to define a schema for the config. This allows you to define what configuration settings you expect in your Pulumi.yaml so that the engine can use that information to validate configuration for every stack.

This is done using a very similar declaration to JSON schema, or our Pulumi JSON schema (the schemas from resource providers that drive our code generators). Below is a small example of some of the things you can declare:

config:
 vm-count:
 type: integer
 description: the number of virtual machines to create
 default: 2
 subnets:
 type: array
 description: an array of subnets to create
 items:
 type: string
 vaultKey:
 type: string
 description: which vault key to use for this stack
 secret: true

These are declared in the same config block as the other project level config shown through this post. You can combine the two by declaring your project settings and defining defaults for providers together:

config:
 aws:region: us-east-2
 createVpc:
 type: boolean
 default: false

Pulumi YAML Configuration: What’s Changing?

When we released our support for simple Pulumi programs in YAML we added some support for configuration with it. This was done via the configuration key in the Pulumi.yaml file. We’ll be deprecating support for that key now, as YAML will instead use the new standard project configuration.

What’s Next for Pulumi Configuration?

This is just the start of our support for better config. We posted a long comment to GitHub earlier this year to explain some of our future plans, and we do intend to continue improving this part of the Pulumi system as described in that comment.

FAQ: Project-Level Configuration in Pulumi

What is project-level config in Pulumi?

Project-level config allows you to set default configuration values for all stacks in a Pulumi.yaml file.

Can I override project-level config for specific stacks?

Yes! You can override default values in Pulumi.<stack>.yaml or via the CLI.

How does Pulumi handle object values in config?

Objects must be nested under a value key to distinguish them from schemas.

Is the configuration key still supported in YAML?

No, it is now deprecated in favor of project-level configuration.

Where can I give feedback or ask questions?

Join the Pulumi Community Slack or join the GitHub Discussions.

Easier IaC adoption with improved `pulumi import` experience

Fraser Waters — Tue, 15 Mar 2022 00:00:00 +0000

Last year, we introduced a new Pulumi feature that allows you to import existing infrastructure into your Pulumi program. Not only did it bring the resource into the Pulumi state file, but it could generate the source code for your Pulumi program too. Today, we’re excited to announce that we’ve listened to feedback and delivered a plethora of updates and fixes to streamline the import experience; to make it more useful, more convenient, and more powerful.

At Pulumi, we understand that many cloud engineers and platform teams around the world don’t have the luxury of greenfield projects, more often than not we’re stuck with the impossible task of “refactoring” or “migrating” existing projects to more modern stacks to help increase team productivity, velocity, and stability. These projects aren’t trivial and we want to make it easier for teams and organizations to bring their infrastructure into a cloud engineering world. Oh, and worry not you lucky greenfielders … even if you just wanna ClickOps your way through some resources and import them into your program; that’s gonna work just fine too; we won’t tell if you don’t 😉.

To help you understand the changes we’ve made to help you on your journey, let’s take a look at some side-by-side examples of how pulumi import used to work vs. what we’re releasing today in Pulumi 3.26.

Example: AWS S3 Bucket

Previously when importing an S3 bucket, we would set the default values for properties like acl and forceDestroy. The new behavior will not include properties with default values in the generated code, keeping your code inline with how you’d write it yourself. Artisanal codegen for the artisanal cloud engineer.

Code Comparison

This code would have been generated by the older implementation:

const my_bucket = new aws.s3.Bucket("my-bucket", {
 acl: "private",
 bucket: "my-bucket-3f85c54",
 forceDestroy: false,
}, {
 protect: true,
});

This code will be generated by the newer implementation:

const my_bucket = new aws.s3.Bucket("my-bucket", {
 arn: "arn:aws:s3:::my-bucket-3f85c54",
 bucket: "my-bucket-3f85c54",
 hostedZoneId: "Z3BJ6K6RIION7M",
 requestPayer: "BucketOwner",
}, {
 protect: true,
});

The above accurately reflects what the providers Read function has told us of the inputs set for this bucket, although due to the way the AWS provider works, we could elide all these properties. This is because the provider will use outputs saved in the state file and the new inputs to calculate the update diff. A property that exists in the old output and is missing from the new inputs is simply left at the old output value. As none of the properties for Bucket are marked required the following works as well and is equivalent:

const my_bucket = new aws.s3.Bucket("my-bucket", { protect: true });

Example: AWS EC2 Instances

Previously, when trying to import an EC2 instance with pulumi import, you’d be presented with a list of required properties that weren’t satisfied. Like above, the new behavior will read from the provider and inherit the missing properties, providing everything required to correctly import. This provides a cleaner and more intuitive experience for the user.

Old Behavior

$ pulumi import aws:ec2/instance:Instance test i-085d780737c600c7e
aws:ec2:Instance (test):
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "instance_type": one of `instance_type,launch_template` must be specified. Examine values at 'Instance.InstanceType'.
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "launch_template": one of `ami,instance_type,launch_template` must be specified. Examine values at 'Instance.LaunchTemplate'.
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "ami": one of `ami,launch_template` must be specified. Examine values at 'Instance.Ami'.

New Behavior

The import will be successful and we’ll be presented with a rich resource in your language of choice.

const test = new aws.ec2.Instance("test", {
 ami: "ami-082b5a644766e0e6f",
 associatePublicIpAddress: true,
 availabilityZone: "us-west-2c",
 capacityReservationSpecification: {
 capacityReservationPreference: "open",
 },
 cpuCoreCount: 1,
 cpuThreadsPerCore: 1,
 creditSpecification: {
 cpuCredits: "standard",
 },
 iamInstanceProfile: "",
 instanceInitiatedShutdownBehavior: "stop",
 instanceType: "t2.micro",
 metadataOptions: {
 httpEndpoint: "enabled",
 httpPutResponseHopLimit: 1,
 httpTokens: "optional",
 },
 privateIp: "172.31.0.188",
 rootBlockDevice: {
 iops: 100,
 volumeSize: 8,
 volumeType: "gp2",
 },
 securityGroups: ["default"],
 subnetId: "subnet-43f43a1e",
 tenancy: "default",
 vpcSecurityGroupIds: ["sg-4d436f12"],
}, {
 protect: true,
});

test = aws.ec2.Instance("test",
 ami="ami-082b5a644766e0e6f",
 associate_public_ip_address=True,
 availability_zone="us-west-2c",
 capacity_reservation_specification=aws.ec2.InstanceCapacityReservationSpecificationArgs(
 capacity_reservation_preference="open",
 ),
 cpu_core_count=1,
 cpu_threads_per_core=1,
 credit_specification=aws.ec2.InstanceCreditSpecificationArgs(
 cpu_credits="standard",
 ),
 iam_instance_profile="",
 instance_initiated_shutdown_behavior="stop",
 instance_type="t2.micro",
 metadata_options=aws.ec2.InstanceMetadataOptionsArgs(
 http_endpoint="enabled",
 http_put_response_hop_limit=1,
 http_tokens="optional",
 ),
 private_ip="172.31.0.188",
 root_block_device=aws.ec2.InstanceRootBlockDeviceArgs(
 iops=100,
 volume_size=8,
 volume_type="gp2",
 ),
 security_groups=["default"],
 subnet_id="subnet-43f43a1e",
 tenancy="default",
 vpc_security_group_ids=["sg-4d436f12"],
 opts=pulumi.ResourceOptions(protect=True))

 _, err := ec2.NewInstance(ctx, "test", &ec2.InstanceArgs{
 Ami: pulumi.String("ami-082b5a644766e0e6f"),
 AssociatePublicIpAddress: pulumi.Bool(true),
 AvailabilityZone: pulumi.String("us-west-2c"),
 CapacityReservationSpecification: &ec2.InstanceCapacityReservationSpecificationArgs{
 CapacityReservationPreference: pulumi.String("open"),
 },
 CpuCoreCount: pulumi.Int(1),
 CpuThreadsPerCore: pulumi.Int(1),
 CreditSpecification: &ec2.InstanceCreditSpecificationArgs{
 CpuCredits: pulumi.String("standard"),
 },
 IamInstanceProfile: pulumi.Any(""),
 InstanceInitiatedShutdownBehavior: pulumi.String("stop"),
 InstanceType: pulumi.String("t2.micro"),
 MetadataOptions: &ec2.InstanceMetadataOptionsArgs{
 HttpEndpoint: pulumi.String("enabled"),
 HttpPutResponseHopLimit: pulumi.Int(1),
 HttpTokens: pulumi.String("optional"),
 },
 PrivateIp: pulumi.String("172.31.0.188"),
 RootBlockDevice: &ec2.InstanceRootBlockDeviceArgs{
 Iops: pulumi.Int(100),
 VolumeSize: pulumi.Int(8),
 VolumeType: pulumi.String("gp2"),
 },
 SecurityGroups: pulumi.StringArray{
 pulumi.String("default"),
 },
 SubnetId: pulumi.String("subnet-43f43a1e"),
 Tenancy: pulumi.String("default"),
 VpcSecurityGroupIds: pulumi.StringArray{
 pulumi.String("sg-4d436f12"),
 },
}, pulumi.Protect(true))

var test = new Aws.Ec2.Instance("test", new Aws.Ec2.InstanceArgs
 {
 Ami = "ami-082b5a644766e0e6f",
 AssociatePublicIpAddress = true,
 AvailabilityZone = "us-west-2c",
 CapacityReservationSpecification = new Aws.Ec2.Inputs.InstanceCapacityReservationSpecificationArgs
 {
 CapacityReservationPreference = "open",
 },
 CpuCoreCount = 1,
 CpuThreadsPerCore = 1,
 CreditSpecification = new Aws.Ec2.Inputs.InstanceCreditSpecificationArgs
 {
 CpuCredits = "standard",
 },
 IamInstanceProfile = "",
 InstanceInitiatedShutdownBehavior = "stop",
 InstanceType = "t2.micro",
 MetadataOptions = new Aws.Ec2.Inputs.InstanceMetadataOptionsArgs
 {
 HttpEndpoint = "enabled",
 HttpPutResponseHopLimit = 1,
 HttpTokens = "optional",
 },
 PrivateIp = "172.31.0.188",
 RootBlockDevice = new Aws.Ec2.Inputs.InstanceRootBlockDeviceArgs
 {
 Iops = 100,
 VolumeSize = 8,
 VolumeType = "gp2",
 },
 SecurityGroups =
 {
 "default",
 },
 SubnetId = "subnet-43f43a1e",
 Tenancy = "default",
 VpcSecurityGroupIds =
 {
 "sg-4d436f12",
 },
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 ```

Again this reflects what Read has told us of the inputs set for this instance, and again due to the way the AWS provider works many of these inputs could be elided and the provider would pick up the values from the saved output set. The following is again equivalent to the above given the saved state:

const test = new aws.ec2.Instance("test", {
 ami: "ami-082b5a644766e0e6f",
 instanceType: "t2.micro"
}, {
 protect: true,
});

Everything is Better, Yes?

To quote our old friend, Harlan, “Everything is better, yes?”

While we’ve only shown you a couple of examples of this new behaviour, rest assured that the benefits of it stretch far and wide across the majority of Pulumi resources.

If you’ve ever had a problem importing a resource before, we encourage you to try it again and let us know if you run into any problems.

Curious how this all works? Let’s dive in.

Technical details

We’ve never really explained how the import system works before, which has lead to a lot of confusion from users when they encountered errors with it. Given the changes we’re making here we felt it a good time to give a more detailed write up of how the engine works to try and help users understand why their import results are the way they are.

This section requires some understanding of the Pulumi architecture, but that understanding is not necessary to be able to use pulumi import.

The importance of `Provider.Read`

Our import system depends on the ability for a resource provider to be able to read the existing state of a resource and report back to Pulumi the current value of it’s inputs and outputs.

This isn’t always possible to return accurately. For example if a resource’s inputs don’t match up 1-to-1 with it’s output state and the underlying provider can only read the current output state there’s no way for it to always construct the correct input state. There could also be bugs in the provider’s Read method that result in inaccurate reads. We’ve designed the import system to be tolerant to these cases, but it does mean we don’t expect perfect import results for every resource.

The current import system

To import resources with the currently the engine issues the following steps:

Use the providers Read function to get the inputs and outputs for the resource.
Remove all properties from the inputs except for those flagged as required.
Pass that reduced input property set to the providers Check method.
If there are any check failures return an error and fail the import.
Take the result of Check which will have filled in default properties and diff it with the original input set returned from Read.
If there are any diffs copy in the values from the original Read to the resulting input set.
Save the updated input set and the original output set into the stacks state.
Pass the resulting input set to the code generator to generate a new Resource call to match the imported state.

There are a number of problems with the above. The biggest is trying to strip the input set down to only properties that are marked as required. The required flag is a fairly blunt tool that can’t capture that some fields are only required sometimes based on the value of other fields. As such this stripping to just required properties causes many of the following calls to Check to raise check failures.

The next major issue is that we hard fail if check returns failures. There is always the risk of this happening due to limitations of Read, and it doesn’t help the user just throwing an error.

Finally in the cases where the above two issues weren’t hit this data flow had the odd result of setting optional fields to their default values. This isn’t needed and these fields would be better left as blank.

The new import system aims to improve the described issues.

The new import system

The engine will now issue the following steps to import resources:

Use the providers Read function to get the inputs and outputs for the resource.
Pass the input set to the providers Check method.
Warn if there are any check failures.
Save the input and output set from Read into the stacks state.
Pass the input set to the code generator to generate a new Resource call to match the imported state.

Note that we no longer end up with filled in default values in our final result, and we carry on in the face of check failures and so can still generate some code. If providers can give complete and accurate results from Read the above flow will result in correctly generated code. However it doesn’t result in quite the correct stack state, this is due to how the data flow of our import system currently works. Currently we generate code based on the value of inputs in the state file, however normally the value stored at inputs in the state file is the result from Check, not what is directly in the users program. As such when you run the first pulumi up after an import your state file will change slightly as the engine calls Check and saves the inputs with defaults filled in. This should normally be transparent to the user, but there may be some cases where providers think this is a trivial diff.

There are two complications with fixing the above. The first is that during import if Read doesn’t return valid results then Check can fail and so we won’t have a property set returned from it that we can save to the state file. This case will always have to fall back to just storing the properties returned by Read and triggering a check failure during up that will need to be manually resolved. The second complication is that the code generation for import works off the state file rather than a separate data flow specifically for imported resources. We plan on improving this but in the spirit of agility we felt that a slightly better import feature today would be more useful to our users than a perfect import feature later.

Terraform `Read`

Many of Pulumi’s resource providers make use of Terraform providers. We call these bridged providers and all of them use our “terraform-bridge” library to translate between the Terraform API and Pulumi API.

Terraform currently has a slightly odd behavior that missing values for some properties are zero initialized by its Refresh method (what we use to implement Pulumi’s Read method in the terraform bridge). This has the result that when Pulumi asks to read a given Terraform resource some top level properties (such as S3 buckets acl property) come back as missing, while other top level properties and all nested properties (such as S3 buckets versioning.mfaDelete) come back with some zero value.

For example reading an AWS S3 bucket object with the current Terraform bridge results in the following property set:

{
"__defaults": [],
"accelerationStatus": "",
"arn": "arn:aws:s3:::my-bucket-3f85c54",
"bucket": "my-bucket-3f85c54",
"corsRules": [],
"grants": [],
"hostedZoneId": "Z3BJ6K6RIION7M",
"lifecycleRules": [],
"loggings": [],
"objectLockConfiguration": null,
"replicationConfiguration": null,
"requestPayer": "BucketOwner",
"serverSideEncryptionConfiguration": null,
"tags": {
"__defaults": []
},
"versioning": {
"__defaults": [],
"enabled": false,
"mfaDelete": false
},
"website": null
}

We’ve made a change to the Terraform bridge to try and elide these zero properties that should just be missing. Reading the same S3 bucket with these changes gives the following much smaller property set:

{
"__defaults": [],
"arn": "arn:aws:s3:::my-bucket-3f85c54",
"bucket": "my-bucket-3f85c54",
"hostedZoneId": "Z3BJ6K6RIION7M",
"requestPayer": "BucketOwner",
"website": null
}

We’ll continue to work on the Terraform bridge to give better results here. For example the website property on the bucket should probably be missing rather than null.

Feedback

We’re aware this could have a large impact to some of our users so we wanted to make sure we explained these changes well. If you have questions or concerns about this change let us know in our community Slack or in GitHub Discussions.

Retain on Delete

Fraser Waters — Wed, 23 Feb 2022 00:00:00 +0000

Pulumi is frequently used to manage the entire lifecycle of a resource, from creation, to updates, to replacement, to deletion. However, there are some cases where it is important to ensure that a resource’s life can extend beyond the lifetime of the Pulumi program that created it. To support these use cases, Pulumi now supports a new resource option RetainOnDelete which allows a resource to be retained in a cloud provider even after it is deleted from the Pulumi stack it is part of.

We’ve seen a number of user questions over the years about keeping resources after Pulumi deployments, or sharing resources between deployments. All of these worked down to needing a new resource option to tell the Pulumi engine to not actually delete a resource when replacing or removing it from the stack.

Retain on delete

We now support a new resource option retain on delete that allows you to specify this case to the engine.

When a resource marked with RetainOnDelete is deleted or replaced by a Pulumi update, the resource is removed from the Pulumi state but Pulumi does not delete the actual resource in the cloud. That is, it won’t call through to the resource provider’s Delete function. This is the similar to AWS CloudFormation’s Retain DeletionPolicy.

New workflows enabled

This feature enables a number of new workflows with Pulumi.

Supporting data storage policies

Using this new feature it is now possible to uncouple resource lifecycles from deployment cycles. This can be especially useful to teams with policy around data and storage. For example your company may require that all S3 storage is backed up and retained for some amount of time. Using RetainOnDelete you can ensure that the S3 buckets that Pulumi creates aren’t deleted by Pulumi, leaving the data alive for your storage policy team to backup and then delete when no longer required.

Supporting shared resources

Another workflow this option enables is the ability to manage externally owned resources via Pulumi deployments. You may have an external long lived resource that you want to temporarily manage via a Pulumi program, but then relinquish back to the external owner rather than deleting when you’re finished with it.

Using a combination of import and RetainOnDelete you can bring a resource under management and then relinquish it back to the external owner.

Working around provider issues

Although it isn’t really a workflow, RetainOnDelete can also be used to work around some provider issues. Some providers may fail to refresh and detect that a resource is now deleted, leading Pulumi to issue a delete call to a resource that no longer exists. While you could always manually edit your state file to resolve cases like this, RetainOnDelete give you the option to fix this via program changes instead. Simply mark the faulty resource with RetainOnDelete, run pulumi up to apply the mark and then delete the resource from your program and run pulumi up again. The engine will cleanly delete the resource from your state file without communicating with the faulty provider.

Using retain on delete

Setting retain on delete is done via a resource option. The following example builds two AWS S3 bucket objects, with one used for logging which will not be deleted by Pulumi.

import * as pulumi from "@pulumi/pulumi";
import { s3 } from "@pulumi/aws";

const logBucket = new aws.s3.Bucket("logBucket", {
 acl: "log-delivery-write"
}, {
 retainOnDelete: true
});
const bucket = new aws.s3.Bucket("bucket", {
 acl: "private",
 loggings: [{
 targetBucket: logBucket.id,
 targetPrefix: "log/",
 }],
});

Read more about retain on delete in our docs. Let us know how you get on with this new feature at our community Slack!

Announcing the Public Preview of Update Plans

Fraser Waters — Wed, 09 Feb 2022 00:00:00 +0000

Update Plans now no longer require the PULUMI_EXPERIMENTAL environment variable. For the most up-to-date information about using Update Plans, please see the Update Plans documentation.

Pulumi’s previews are an important part of any workflow where you want to see the changes that will be made to your infrastructure before actually making the changes (with pulumi up). However, today there is no guarantee that the pulumi up operation will do only what was previewed; if the program, or your infrastructure, changes between the preview and the update, the update might make additional changes to bring your infrastructure back in line with what’s defined in your program. We’ve heard from many of you that you need a strong guarantee about exactly which changes an update will make to your infrastructure, especially in critical and production environments.

Today, I’m excited to announce the public preview of Update Plans, a new Pulumi feature which guarantees that operations shown in pulumi preview will run on pulumi up. Update Plans also help catch any unexpected changes that might happen between when you preview a change and when you apply that change. Update Plans work by saving the results of a pulumi preview to a plan file, which enables you to restrict subsequent pulumi up operations to only the actions saved in the plan file. This helps you ensure that what you saw in the pulumi preview is what will actually happen when you run pulumi up.

Here’s an example of Update Plans in action. In this example we’re creating a new AWS EC2 instance and associated security group. First, we plan the change with pulumi preview --save-plan plan.json, and then we examine the plan file to see which resources the plan expects to change. Finally, we run an update that’s constrained to the plan with pulumi up --plan plan.json.

To get started, you only need to add --save-plan <file> to the preview command and --plan <file> to the up command. In the video above, we demonstrated a pretty trivial example where our program did what our plan expected, so everything ran just as expected. But what if we changed our program to have a different userData value and ran it against the same plan? If we clear the infrastructure we just created and use the same plan with the changed program, then Update Plans will return an error and block the change.

When updates fail to validate against the plan, Pulumi will print what constraint failed. In the example above, the resource urn:pulumi:dev::aws-ts-webserver::aws:ec2/instance:Instance::web-server-www changed the value of the property userData.

New Scenario: Pull Request Validation Workflows

Software development teams commonly use version control-based workflows to review and monitor the code that is added to a codebase. For example, many teams protect their primary branch (e.g. main, master, trunk) from direct code pushes and instead require a pull or merge request where the changes can be reviewed and continuous integration (CI) and tests can be run.

Try Update Plans

Update Plans are in public preview. We recommend using it only in non-critical, non-production scenarios during the preview period.

We’re eager for you to try the public preview of Update Plans and let us know what you think. To try it out, make sure you’ve updated to Pulumi 3.24.1 then set the following environment variable in your shell:


PULUMI_EXPERIMENTAL=true

Then, save a plan file:


pulumi preview --save-plan plan.json

Finally, pass the plan file to the pulumi up command:


pulumi up --plan-file plan.json

When PULUMI_EXPERIMENTAL is set pulumi up will also generate a plan during it’s preview stage which will then apply to the update if an explicit plan hasn’t been given via --plan. This ensures your up previews match what happens in the resulting operations.

We’d love to hear your thoughts on the Update Plans feature! Feel free to start discussions or ask questions using GitHub Discussions in the pulumi/pulumi repository.

Exploring how to solve circular dependencies

Daniel Bradley — Wed, 12 Jan 2022 00:00:00 +0000

This blog post originated from a hackathon and exploratory exercise, and should be considered in that context. None of the solutions exist in the Pulumi SDKs. For more information on this, please view our blog post on Circular Dependencies.

As part of our hackathon near the end of last year, we decided to explore solutions to a common problem when people are using Pulumi for their systems. A question that’s been asked in a few different forms is how to resolve circular dependencies between resources in a Pulumi program.

A simple example of this idea is a modern web application with a static front-end and an API, where the front-end needs to know the URL of the API to be able to call it and the API needs to know the source domain of the front-end to allow it access via CORS. As these two resources rely on one another to be created, they are circular dependencies.

For a deployable example, let’s build a serverless game of ping-pong—two functions that invoke one another until one misses!

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name, // <-- Can't access `pong` yet!
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

Programming languages don’t let us create one object that takes a value from another object that doesn’t yet exist. Therefore, we can’t express a circular dependency in this way.

The simplest workaround for this error is to get a reference to our own stack, which contains the outputs from the previous deployment. Now, we set the reference to pong within ping from the output of the last deployment.

const currentStack = new pulumi.StackReference(
 `acme-org/${pulumi.getProject()}/${pulumi.getStack()}`
);

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: currentStack
 .getOutput("pongName")
 .apply((pongName) => pongName ?? ""),
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

export const pingName = ping.name;
export const pongName = pong.name;

The downside of this approach is that we have to deploy twice to reach the desired state because we need the results of the previous deployment, though this approach might be fine for simple one-off circumstances.

How then could we make the process easier and clearer for users who need to resolve this kind of circular dependency regularly? We explored four options as part of the hackathon, so let’s go through each one, getting progressively closer to a possible solution.

Option 1: State

Instead of referencing outputs directly from our stack’s previous deploy and manually deploying twice, we could introduce a special type called “State”.

The State object does 3 tasks:

Read the value from the previous deployment, or provide a default if it’s the first deployment.
Update the value in the stack state using a “set” method.
Automatically triggers another deployment if its value has changed.

const pongName = new pulumi.State("pong-name", {
 initialValue: "",
});

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pongName.value,
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

// Engine re-runs "up" if this changes the value
pongName.set(pong.name);

The first deployment will match the preview, but when triggering the re-deployment, we’re making a change that wasn’t in the first preview. This action could also result in a never-ending loop where every deployment makes another change to the state and triggers another deployment. So let’s explore a different option that might not have this issue.

Option 2: Refs

This option looks almost identical to the “state” concept above, except we’re calling them references (refs):

const pongName = new pulumi.Ref("pong-name", {
 initialValue: "",
});

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pongName.value, // Special type the Pulumi engine will track
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

// Determine affected resources and re-update
pongName.set(pong.name);

The difference between refs and state is that, instead of triggering a whole re-deploy when the value changes, refs would be integrated into the dependency graph similarly to resources. When the value of the ref changes, we could internally work out which resources depend on that ref and change the target state for only those resources. This avoids the possibility of a never-ending update loop.

Both the “state” and “refs” approaches have the downside of not making the final outputs of the updated resource available to the rest of the program. The resource object would still have the values from after it first deployed. This means that other parts of the program might still deploy differently on subsequent deployments. Let’s explore another option that might address that problem.

Option 3: Create patch resources

We could explicitly model the third step by using a new resource type suffixed with “patch”:

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

const pingPatch = new CallbackFunctionPatch("ping-patch", {
 id: ping.id,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name,
 },
 },
});

The provider presents pseudo-resources for update or patch operations. The patch resource is using the same underlying resource as the original resource. This option is a very flexible approach that could be implemented by providers right now. These patch resources could also be used for resources that exist outside the Pulumi stack. The drawback is that the patch operation has to be implemented in each provider individually, and it significantly increases the overall size of each provider and its associated SDK. We’re still evaluating whether this option would be a good one considering that size increase, which increases download times as an example impact, and the engineering resources to implement the change. We have one more option to consider, as well.

Option 4: Allow a resource to update its goal state

When using the SDK, we could write something like the following code to create then update a little later on:

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

const updatedPing = ping.update({
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name,
 },
 },
});

When declaring a resource in a Pulumi program, you’re essentially saying, “This is the state I want my resource to be in.” The update method returns a whole new copy of the resource with new properties containing the outputs of the update.

What’s next?

The final option is quite close to a fully working solution, but there still remains a few tasks to complete before we could release it as a feature:

Finalise what the internal messages between the Pulumi’s internal components should look like.
Keep a record of circular dependencies in the stack state without causing knock-on issues elsewhere in Pulumi’s internal workings.
Decide what should happen if an update triggers a replacement—if this is potentially valid behaviour or not. We should be able to generate the set of properties which are able to be updated without causing replacement and provide these properties to the user as the arguments for the update method to avoid these issues before execution of the program.

Some other ideas that would be interesting to explore in the future are as follows:

How could we utilise patching of resources where a provider can support it?
Could we delete a resource that was created earlier in the program to allow the creation of temporary resources?

This hackathon project has opened up a number of new pathways for us to explore. If you want to participate, we always welcome contributions in our open-source repos at https://github.com/pulumi. We hope you have enjoyed this dive into how our team explored this concept as part of the hackathon. We’ll be exploring more of the hackathon projects soon on the blog!

Pulumi Blog: Fraser Waters

Introducing pulumi do: Direct Resource Operations for Any Cloud

The problem: Sometimes IaC is more than you need

What it looks like

The command shape

Resource operations

Provider configuration

Designed for humans and agents

What’s next

Unified credentials with Pulumi ESC

Cross-resource references

Stateful mode and the graduation path

Get started

Introducing the pulumi policy analyze Command for Existing Stacks

Why this command matters

Basic usage

Testing new policy packs as a developer

Using it in AI and agent workflows

Try it out

Introducing the Stash Resource in Pulumi IaC

Why Stash?

Using Stash

When to replace

Learn more

Introducing Resource Transforms: Enhancing Flexibility for Packaged Component Resources

Converting Full Terraform States to Pulumi

Converting a Real World Workspace

Get Started

Preview of .NET resource providers

New CLI prompt to use Update Plans

Why use Update Plans

New option in CLI interactive prompt

Try Update Plans

Simplify Configuration Management with Project-level Config

How to Use Project-level Configuration

Defining Configuration Schemas in Pulumi

Pulumi YAML Configuration: What’s Changing?

What’s Next for Pulumi Configuration?

FAQ: Project-Level Configuration in Pulumi

What is project-level config in Pulumi?

Can I override project-level config for specific stacks?

How does Pulumi handle object values in config?

Is the configuration key still supported in YAML?

Where can I give feedback or ask questions?

Easier IaC adoption with improved `pulumi import` experience

Example: AWS S3 Bucket

Code Comparison

Example: AWS EC2 Instances

Old Behavior

New Behavior

Everything is Better, Yes?

Technical details

The importance of Provider.Read

The current import system

The new import system

Terraform Read

Feedback

Retain on Delete

Retain on delete

New workflows enabled

Supporting data storage policies

Supporting shared resources

Working around provider issues

Using retain on delete

Announcing the Public Preview of Update Plans

New Scenario: Pull Request Validation Workflows

Try Update Plans

Exploring how to solve circular dependencies

Option 1: State

Option 2: Refs

Option 3: Create patch resources

Option 4: Allow a resource to update its goal state

What’s next?

The importance of `Provider.Read`

Terraform `Read`