Pulumi Blog: Thomas Gummerer

Introducing OTel Tracing in the Pulumi CLI

Thomas Gummerer — Wed, 01 Apr 2026 00:00:00 +0000

Tracing is an important part of our CLI observability story. So far we’ve relied on (the now deprecated) OpenTracing for this. We have now added OTel tracing to the CLI, which is more future-proof, and should in most cases give you a better view over what the CLI is doing.

Background

We introduced tracing using OpenTracing all the way back in 2017, before OpenTelemetry was a thing. This served us well over the years, but as OpenTracing was deprecated, and OTel emerged as the new and maintained thing, it got harder and harder to justify further investment in a tracing infrastructure that was deprecated. Last year we started focusing more on performance, and it became more and more clear that we’d either have to enhance our current OpenTracing setup, or do the work to switch to OTel.

In the end it was a relatively easy decision to move to the more modern and fully supported OTel, especially as more and more tooling around it starts emerging.

Enter OTel

With the decision to move to OTel made, the only thing left to decide was how to implement it. There were a couple of constraints we faced here. First, we wanted to make sure that traces are easily shareable. This means ideally a text file in whatever format, that can be shared easily. Second, pulumi’s plugin system works by spawning a new process per plugin. We want to get traces from each of these plugins to make sure we have as much coverage as possible. And third, ideally we also want to get traces from plugins that only implement OpenTracing, but not OTel yet, since someone can upgrade the CLI, but not the plugins for example.

Given these constraints, we decided to implement an OTel collector in the CLI, that could then forward the traces to whatever output format we want. This means that plugins only ever need to send traces back to the CLI over gRPC, and the CLI will do any further processing. This means only one process will write to the file, if requested.

For plugins we always request both OpenTracing and OTel traces. If both are requested and OTel is supported by the plugin, the plugin is expected to only send the OTel version of the traces. For OpenTracing traces, we collect them in the collector in the CLI, and then translate them internally to OTel traces. This way we can still get the traces from older plugins, without them needing to change anything.

Try it out

Currently the OTel exporter supports both exporting the traces directly via gRPC to a collector, or to a file, where the traces are JSON encoded. This file can then be shared and imported into a trace viewer at a later time. To do this, you can use the --otel-traces <file://<filename>|grpc://<exporter-address>> flag, using pulumi version v3.226 or newer. For further documentation see our performance tracing docs.

To view the traces, you can use one of the various exporters that exist. Popular options include Jaeger, OTel Desktop Viewer, or OTel TUI if you prefer not leaving your terminal. Once you’ve ingested the logs there either by uploading the trace file, or sending them directly by giving pulumi the exporter address, look for the pulumi-cli: pulumi root span.

All further spans will be parented to that root span, and you should thus be able to see a nice flow diagram in the viewer of your choice.

As always, we would love any feedback either in the Community Slack, or through a GitHub issue.

Now GA: Up to 20x Faster Pulumi Operations for Everyone

Thomas Gummerer — Thu, 05 Mar 2026 00:00:00 +0000

In January, we introduced a major performance enhancement for Pulumi Cloud through a fundamental change to how Pulumi manages state that speeds up operations by up to 20x. After a staged rollout across many organizations, it is now enabled by default for every Pulumi Cloud operation. No opt-in required—just use Pulumi CLI v3.225.0+ with Pulumi Cloud. The improvement applies to pulumi up, pulumi destroy, and pulumi refresh; pulumi preview does not modify state, so it is unchanged.

What this means for you

First and foremost, nothing about how you work with pulumi needs to change. Your updates now benefit from better parallelism and should thus complete faster. Before this change, pulumi always saved a full snapshot to the cloud, so the current state could always be recovered if something goes wrong. With journaling, we now only send the state of each operation, which allows us to send these updates in parallel, as long as resources are not related to each other. For the full deep dive, see the blog post linked above.

Production results

Since January, we’ve had many early adopters of journaling. This helped us shake out one final bug on the server side, and journaling has been stable since then. With that we feel confident in rolling this out to all our users.

We’ve also gathered some real-world data on how journaling is performing. The data from the preview period shows some significant improvements for update times. For stacks with fewer than 100 resources, the median improvement is 25.3%, while the p90 improvement is 75.2%, and we’ve seen a p99 improvement of up to 92.6% Meanwhile, for larger stacks, the median improvement is 60.2%. We need more data for stacks with more than 100 resources, we will update this blog once that comes in.

This data already shows the expected significant improvement in update times, especially for larger stacks, though the improvements strongly depend on the shape and type of resources that are being set up. Stacks with many resources, that are quick to update benefit more than smaller stacks with slower to set up resources. For more numbers see also the Benchmarks section in the previous blog post

What you need to do

While this was an opt-in process using the PULUMI_ENABLE_JOURNALING environment variable, this opt-in is no longer required. Just upgrade your Pulumi CLI to v3.225.0+ and use the Pulumi Cloud backend, and journaling will automatically speed up your updates.

If you encounter any issues, reach out on the Pulumi Community Slack or through Pulumi Support. You can also set the PULUMI_DISABLE_JOURNALING=true env variable to opt out of journaling.

Speeding up Pulumi Operations by up to 20x

Thomas Gummerer — Mon, 12 Jan 2026 18:57:55 +0200

Today we’re introducing an improvement that can speed up operations by up to 20x. At every operation, and at every step within an operation, pulumi saves a snapshot of your cloud infrastructure. This gives pulumi a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here’s how we fixed it.

Benchmarks

Before getting into the more technical details, here are a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks we picked a couple of Pulumi projects: one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another that looks a little more like a real world example. Note that we conducted all of these benchmarks in Europe, connecting to Pulumi Cloud, which runs in AWS’s us-west-2 region, so exact numbers may vary based on your location and internet connection. This should however give a good indication of the performance improvements.

We’re benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We’re using the aws-ts-static-website example here with a small subset of the fraction from our docs site. This means we’re setting up more than 3000 bucket objects, with 3222 resources in total.

The benchmarks were measured using the time built-in command and using the best time in a best-of-three benchmark. The network traffic was measured using tcpdump, limiting the measured traffic to only the IP addresses for Pulumi Cloud. Finally, tshark was used to process the packet captures and count the bytes sent.

All the benchmarks are run with journaling off (the default experience) and with journaling on (the new experience). To begin with, let’s look at the results when creating our stack from scratch:

	Time	Bytes sent
Without journaling	58m26s	16.5MB
With journaling	02m50s	2.3MB

Now let’s have a look at what this looks like if we only change half the resources, but the remaining ones remain unchanged:

	Time	Bytes sent
Without journaling	34m49s	13.8MB
With journaling	01m45s	2.3MB

The second example is setting up an instance of the Pulumi app and API. Here we’ll have an example that’s a bit more dominated by the cost of setting up the actual infrastructure in the cloud, but we still have a very noticeable improvement in the time it takes to set up the stack.

	Time	Bytes sent
Without journaling	17m52s	18.5MB
With journaling	9m12s	5.9MB

To use this feature, you need a pulumi version newer than v3.211.0, and set the PULUMI_ENABLE_JOURNALING environment variable to true.

If you are interested in the more technical details read on!

Introduction to snapshotting

pulumi keeps track of all resources in a stack in a snapshot. This snapshot is stored in the stack’s configured backend, which is either the Pulumi Cloud or a DIY backend. Future operations on the stack then use this snapshot to figure out which resources need to be created, updated or deleted.

pulumi creates a new snapshot at the beginning and at the end of each resource operation to minimize the possibility of untracked changes even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs).

At the beginning of the operation, pulumi adds a new “pending operation” to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but pulumi couldn’t record the end of it), the next operation will try to resolve this. If we have an ID for the resource already, for example on partial updates/deletes, pulumi will try to read the resource state from the cloud and resolve it internally. If there is no ID yet, pulumi will ask the user to check the actual state of the resource. Depending on the user’s response, pulumi will either remove the operation from the snapshot or import the resource. This is because it is possible that the resource has been set up correctly or that the resource creation failed. If pulumi aborted midway through the operation, it’s impossible to know which state the resource is in.

Once an operation finishes, the pending operation is removed and the resource’s final state is recorded in the snapshot.

There’s also some additional metadata that is stored in the snapshot that is only updated infrequently.

Here’s how the snapshot looks in code. This snapshot is serialized and sent to the backend. Resources holds the list of known resource states and is updated after each operation finishes, and PendingOperations is the list of pending operations described above.

type Snapshot struct {
 Manifest Manifest // a deployment manifest of versions, checksums, and so on.
 SecretsManager secrets.Manager // the secrets manager to use when serializing this snapshot.
 Resources []*resource.State // all resources and their associated states.
 PendingOperations []resource.Operation // all currently pending resource operations.
 Metadata SnapshotMetadata // metadata associated with the snapshot.
}

Before we dive in deeper, we also need to understand a little bit about how the pulumi engine works internally. Whenever a pulumi operation is run (e.g. pulumi up, pulumi destroy, pulumi refresh etc.), the engine internally generates and executes a series of steps, to create, update, delete etc. resources. To maintain correct relationships between resources, the steps need to be executed in a partial order such that no step is executed until all of the steps it depends on have executed successfully. Steps may otherwise execute concurrently.

As each step is responsible for updating a single resource, we can generate a snapshot of the state before each step starts, and after it completes. Before each step starts, we create a pending operation, and add it to the PendingOperations list. After that step completes, we remove the pending operation from that list, and update the Resources list, either adding a resource, removing it, or updating it, depending on the kind of operation we just executed.

After this introduction, we can dive into what’s slow, how we fixed it, and some benchmarks.

Why is it slow?

To make sure the state is always as up-to-date as possible, even if there are any network hiccups/power outages etc., a step won’t start until the snapshot that includes the pending operation is confirmed to be stored in the backend. Similarly an operation won’t be considered finished until the snapshot with an updated resources list is confirmed to be stored in the backend.

To send the current state to the backend, we simply serialize it as a JSON file, and send it to the backend. However, as mentioned above, steps can be executed in parallel. If we uploaded the snapshot at the beginning and end of every step with no serialization, there would be a risk that we overwrite a new snapshot with an older one, leading to incorrect data.

Our workaround for that is to serialize the snapshot uploads, uploading one snapshot at a time. This gives us the data integrity properties we want, however it can slow step execution down, especially on internet connections with lower bandwidth, and/or high latency.

This impacts performance especially for large stacks, as we upload the whole snapshot every time, which can take some time if the snapshot is getting big. For the Pulumi Cloud backend we improved on this a little at the end of 2022. We implemented a diff based protocol, which is especially helpful for large snapshots, as we only need to send the diff between the old and the new snapshot, and Pulumi Cloud can then reconstruct the full snapshot based on that. This reduces the amount of data that needs to be transferred, thus improving performance.

However, the snapshotting is still a major bottleneck for large pulumi operations. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. Furthermore, the time spent performing textual diffs between snapshots scales in proportion to the size of the data being processed, which adds additional execution time to each operation.

Fast, but lacking data integrity?

As long as pulumi can complete its operation, there’s no need for the intermediate checkpoints. We could allow pulumi operations to skip uploading the intermittent checkpoints to the backend. This, of course, avoids the single serialization point we have sending the snapshots to the backend, and thus makes the operation much more performant.

However, it also has the serious disadvantage of compromising some of the data integrity guarantees pulumi gives you. If anything goes wrong during the update, pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist.

Neither of these solutions is very satisfying, as the tradeoff is either performance or data integrity. We would like to have our cake and eat it too, and that’s exactly what we’re doing with journaling.

Enter journaling

To achieve this, we went back to the drawing board, and asked ourselves, “What would a solution look like that’s both performant and preserves data integrity throughout the update?”.

Making that happen is possible because of three facts:

We always start with the same snapshot on the backend and the CLI.
Every step the engine executes affects only one resource.
We have a service that can reconstruct a snapshot from what is given to it.

(The third point here already hints at it, but this feature is only available and made possible by Pulumi Cloud, not on the DIY backend).

What if instead of sending the whole snapshot, or a diff of the snapshot, we could send the individual changes to the base snapshot to the service, which could then apply it, and reconstruct a full snapshot from it? This is exactly what we are doing here, in the form of what we call journal entries. Each journal entry has the following form:

const (
 JournalEntryKindBegin JournalEntryKind = 0
 JournalEntryKindSuccess JournalEntryKind = 1
 JournalEntryKindFailure JournalEntryKind = 2
 JournalEntryKindRefreshSuccess JournalEntryKind = 3
 JournalEntryKindOutputs JournalEntryKind = 4
 JournalEntryKindWrite JournalEntryKind = 5
 JournalEntryKindSecretsManager JournalEntryKind = 6
 JournalEntryKindRebuiltBaseState JournalEntryKind = 7
)

type JournalEntry struct {
 // Version of the journal entry format.
 Version int `json:"version"`
 // Kind of journal entry.
 Kind JournalEntryKind `json:"kind"`
 // Sequence ID of the operation.
 SequenceID int64 `json:"sequenceID"`
 // ID of the operation this journal entry is associated with.
 OperationID int64 `json:"operationID"`
 // ID for the delete Operation that this journal entry is associated with.
 RemoveOld *int64 `json:"removeOld"`
 // ID for the delete Operation that this journal entry is associated with.
 RemoveNew *int64 `json:"removeNew"`
 // PendingReplacementOld is the index of the resource that's to be marked as pending replacement
 PendingReplacementOld *int64 `json:"pendingReplacementOld,omitempty"`
 // PendingReplacementNew is the operation ID of the new resource to be marked as pending replacement
 PendingReplacementNew *int64 `json:"pendingReplacementNew,omitempty"`
 // DeleteOld is the index of the resource that's to be marked as deleted.
 DeleteOld *int64 `json:"deleteOld,omitempty"`
 // DeleteNew is the operation ID of the new resource to be marked as deleted.
 DeleteNew *int64 `json:"deleteNew,omitempty"`
 // The resource state associated with this journal entry.
 State *ResourceV3 `json:"state,omitempty"`
 // The operation associated with this journal entry, if any.
 Operation *OperationV2 `json:"operation,omitempty"`
 // If true, this journal entry is part of a refresh operation.
 RebuildDependencies bool `json:"isRefresh,omitempty"`
 // The secrets manager associated with this journal entry, if any.
 SecretsProvider *SecretsProvidersV1 `json:"secretsProvider,omitempty"`

 // NewSnapshot is the new snapshot that this journal entry is associated with.
 NewSnapshot *DeploymentV3 `json:"newSnapshot,omitempty"`
}

These journal entries encode all the information needed to reconstruct the snapshot from them. Each journal entry can be sent in parallel from the engine, and the snapshot will still be fully valid. All journal entries have a Sequence ID attached to them, and they need to be replayed in that order on the service side to make sure we get a valid snapshot. It is however okay to replay without journal entries that have not yet been received by the service, and whose sequence ID is thus missing. This is safe because the engine only sends entries in parallel for resources whose parents/dependencies have been fully created and confirmed by the service.

This way we make sure that the resources list is always in the correct partial order that is required by the engine to function correctly, and for the snapshot to be considered valid.

The algorithm looks as follows:

# Apply snapshot writes. This replaces the full snapshot we have on the service.
# We do this if default providers change, because we don't emit steps for that, as
# we do for the rest of the operations.
snapshot = find_write_journal_entry_or_use_base(base, journal)
# Track changes
deletes, snapshot_deletes, mark_deleted, mark_pending = set(), set(), set(), set()
operation_id_to_resource_index = {}
# Process journal entries. This is the main algorithm, that adds new resources
# to the snapshot, removes existing ones, deals with refreshes, and operations
# that update outputs.
incomplete_ops = {}
has_refresh = false
index = 0
for entry in journal:
match entry.type:
case BEGIN:
incomplete_ops[entry.op_id] = entry
case SUCCESS:
del incomplete_ops[entry.op_id]
if entry.state and entry.op_id:
resources.append(entry.state)
operation_id_to_resource_index.add(entry.op_id, index)
index++
if entry.remove_old:
snapshot_deletes.add(entry.remove_old)
if entry.remove_new:
deletes[remove_new] = true
if entry.pending_replacement:
mark_pending(entry.pending_replacement)
if entry.delete:
mark_deleted(entry.delete)
has_refresh |= entry.is_refresh
case REFRESH_SUCCESS:
del incomplete_ops[entry.op_id]
has_refresh = true
if entry.remove_old:
if entry.state:
snapshot_replacements[entry.remove_old] = entry.state
else:
snapshot_deletes.add(entry.remove_old)
if entry.remove_new:
if entry.state:
deletes[entry.remove_new] = true
else:
resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state)
case FAILURE:
del incomplete_ops[entry.op_id]
case OUTPUTS:
if entry.state and entry.remove_old:
snapshot_replacements[entry.remove_old] = entry.state
if entry.state and entry.remove_new:
resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state)
deletes = deletes.map(|i| => operation_id_to_resource_index[i])
# Now that we have marked all the operations, and created a new list of resources, we can
# go through them, and merge the list of new resources and old resources from the snapshot
# that remain together.
for i, res in resources:
if i in deletes:
remove_from_resources(resources, i)
# Merge snapshot resources. These resources have not been touched by the update, and will
# thus be appended to the end of the resource list. We also need to mark existing resources as
# `Delete` and `PendingReplacement` here.
for i, res in enumerate(snapshot.resources):
if i not in snapshot_deletes:
if i in snapshot_replacements:
resources.append(snapshot_replacements[i])
else:
if i in mark_deleted:
res.delete = true
if i in mark_pending:
res.pending_replacement = true
resources.append(res)
# Collect pending operations. These are stored separately from the resources list
# in the snapshot.
pending_ops = [op.operation for op in incomplete_ops.values() if op.operation]
pending_ops.extend([op for op in snapshot.pending_ops if op.type == CREATE])
# Rebuild dependencies if necessary. Refreshes can delete parents or dependencies
# of resources, without affecting the resource itself directly. We need to now remove
# these relationships to make sure the snapshot remains valid.
if has_refresh:
rebuild_dependencies(resources)

The full documentation of the algorithm can be found in our developer docs.

Rollout

pulumi state is a very central part of pulumi, so we wanted to be extra careful with the rollout to make sure we don’t break anything. We did this in a few stages:

We implemented the replay interface inside the pulumi CLI, and ran it in parallel with the current snapshotting implementation in our tests. The snapshots were then compared automatically, and tests made to fail when the result didn’t match.
Since tests can’t cover all possible edge cases, the next step was to run the journaler in parallel with the current snapshotting implementation internally. This was still without sending the results to the service. However we would compare the snapshot, and send an error event to the service if the snapshot didn’t match. In our data warehouse we could then inspect any mismatches, and fix them. Since this does involve the service in a minor way, we would only do this if the user is using the Cloud backend.
Next up was adding a feature flag for the service, so journaling could be turned on selectively for some orgs. At the same time we implemented an opt-in environment variable in the CLI (PULUMI_ENABLE_JOURNALING), so the feature could be selectively turned on by users, if both the feature flag is enabled and the user sets the environment variable. This way we could slowly start enabling this in our repos, e.g. first in the integration tests for pulumi/pulumi, then in the tests for pulumi/examples and pulumi/templates, etc.
Allow users to start opting in. If you want to opt-in with your org, please reach out to us, either on the Community Slack, or through our Support channels, and we’ll opt your org into the feature flag. Then you can begin seeing the performance improvements by setting the PULUMI_ENABLE_JOURNALING env variable to true.
Turn on the feature flag for everyone, but still require the PULUMI_ENABLE_JOURNALING env variable to be set to true. (We are here right now).
Flip the feature on by default, but still allow users to opt out using a PULUMI_DISABLE_JOURNALING env variable.

What’s next

While these performance improvements hopefully make your day to day use of pulumi quicker and more enjoyable, we’re not quite done here. We’re looking at some other performance improvements, that will hopefully speed up your workflows even more.

Introducing `pulumi state move`: Move Resources Between Stacks or Projects

Thomas Gummerer — Thu, 25 Jul 2024 00:00:00 +0000

Today we’re announcing the pulumi state move command, which can be used to move resources that are managed by Pulumi between different stacks and/or projects. With the pulumi state move command, you can refactor your Pulumi Infrastructure as Code without any disruption to your deployed cloud infrastructure, enabling you to evolve and scale with confidence.

When refactoring a Pulumi project from a monolithic structure to micro stacks, you might need to move resources between different projects or stacks, without recreating them. While this has been possible with some significant manual effort to hand-modify Pulumi state files (our users came up with automated solutions for this!), doing so can be error prone and time consuming. Having a simpler, integrated solution for this problem was a much requested feature. The new pulumi state move command, combined with the aliases command for refactoring resources within stacks, offers the richest collection of tools for refactoring IaC at scale available in the industry today.

The pulumi state move command works as follows:

$ pulumi state move --help
Move resources from one stack to another
This command can be used to move resources from one stack to another. This can be useful when
splitting a stack into multiple stacks or when merging multiple stacks into one.
Usage:
pulumi state move [flags] <urn>...
Flags:
--dest string The name of the stack to move resources to
-h, --help help for move
--include-parents Include all the parents of the moved resources as well
--source string The name of the stack to move resources from
-y, --yes Automatically approve and perform the move

Both dest and source can be either stacks in the current project, or stacks in a different project, using the fully qualified stack names. Note that this works only for stacks within the same backend, it is currently not possible to move a resource between different backends (though you can move stacks between backends using other existing tools).

The resources being moved have to be specified by their full URN, and multiple URNs can be passed at once. For each resource being moved, all the children of that resource will also be moved, and the relationships between all resources being moved is preserved. Resources with other types of dependencies will however not be moved to the target stack by default. The easiest way to get the full URN of the resources is to use pulumi stack --show-urns. Note that URNs can contain characters that get interpreted by the shell, so it is always a good idea to wrap them in single quotes (') when passing them as arguments.

The pulumi state move command won’t modify the code of your program directly, it will only modify the state of the source and destination stacks. So following running the command, you should ensure that you modify both programs to match the changes you have made. This can typically be accomplished by copy/pasting source code for the resources and/or components between the two codebases. Inputs and outputs of resources that were moved may need to be adjusted as part of this process. This can be done either by using stack references, or recreating the inputs in the program.

Example

To demonstrate how the command works in practice, let’s go through an example. Let’s assume we have the following program:

const randomPet = new random.RandomPet("a-random-pet");

const b = new aws.s3.Bucket("b");

const index = new aws.s3.BucketObject("index.html", {
 bucket: b.bucket,
 content: "Thanks for using Pulumi!",
}, {
 parent: b,
});

const randomSite = new aws.s3.BucketObject("random.html", {
 bucket: b.bucket,
 content: randomPet.id,
}, {
 parent: b,
});

We create a bucket, and two bucket objects, both of them having the bucket as their parent. One of the objects has a random pet name as its content. At some point we decide we’d rather group all the AWS resources in a separate program, because this one grew too large. We can do that using the pulumi state move command, and then adjusting the code.

First we can find out which stacks already exist and we can move the resources to:

$ pulumi stack ls --all
NAME LAST UPDATE RESOURCE COUNT URL
v-thomas-pulumi-corp/pulumi-demo-move-aws/dev 23 minutes ago 0 <url>
v-thomas-pulumi-corp/pulumi-demo-move/dev 6 minutes ago 7 <url>

Next we can find the URN of the resources using pulumi stack --show-urns:

$ pulumi stack --show-urns
Current stack is dev:
 Owner: v-thomas-pulumi-corp
 Last updated: 13 seconds ago (2024-07-23 13:52:17.58884965 +0200 CEST)
 Pulumi version used: 3.125.1-dev.0
Current stack resources (7):
 TYPE NAME
 pulumi:pulumi:Stack pulumi-demo-move-dev
 │ URN: urn:pulumi:dev::pulumi-demo-move::pulumi:pulumi:Stack::pulumi-demo-move-dev
 ├─ random:index/randomPet:RandomPet a-random-pet
 │ URN: urn:pulumi:dev::pulumi-demo-move::random:index/randomPet:RandomPet::a-random-pet
 ├─ aws:s3/bucket:Bucket b
 │ │ URN: urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket::b
 │ ├─ aws:s3/bucketObject:BucketObject random.html
 │ │ URN: urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket$aws:s3/bucketObject:BucketObject::random.html
 │ └─ aws:s3/bucketObject:BucketObject index.html
 │ URN: urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket$aws:s3/bucketObject:BucketObject::index.html
 ├─ pulumi:providers:random default_4_16_3
 │ URN: urn:pulumi:dev::pulumi-demo-move::pulumi:providers:random::default_4_16_3
 └─ pulumi:providers:aws default_6_45_0
 URN: urn:pulumi:dev::pulumi-demo-move::pulumi:providers:aws::default_6_45_0

Next we can actually move the resources. Since we’re moving the resources from the currently selected stack, we can omit the --source argument:

$ pulumi state move --dest v-thomas-pulumi-corp/pulumi-demo-move-aws/dev 'urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket::b'
Planning to move the following resources from v-thomas-pulumi-corp/pulumi-demo-move/dev to v-thomas-pulumi-corp/pulumi-demo-move-aws/dev:

 - urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket::b
 - urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket$aws:s3/bucketObject:BucketObject::random.html
 - urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket$aws:s3/bucketObject:BucketObject::index.html

The following resources being moved to v-thomas-pulumi-corp/pulumi-demo-move-aws/dev have dependencies on resources in v-thomas-pulumi-corp/pulumi-demo-move/dev:

 - urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket$aws:s3/bucketObject:BucketObject::random.html has a dependency on urn:pulumi:dev::pulumi-demo-move::random:index/randomPet:RandomPet::a-random-pet
 - urn:pulumi:dev::pulumi-demo-move::aws:s3/bucket:Bucket$aws:s3/bucketObject:BucketObject::random.html (content) has a property dependency on urn:pulumi:dev::pulumi-demo-move::random:index/randomPet:RandomPet::a-random-pet

If you go ahead with moving these dependencies, it will be necessary to create the appropriate inputs and outputs in the program for the stack the resources are moved to.

Do you want to perform this move? yes
Successfully moved resources from v-thomas-pulumi-corp/pulumi-demo-move/dev to v-thomas-pulumi-corp/pulumi-demo-move-aws/dev

There’s a few things worth explaining above:

We’re moving the resources to a different project, so we need to use the fully qualified stack name. If the resources are being moved between different stacks in the same project, the argument can be only the stack name.
Before actually moving the resources, the command gives a list of resources that is being moved. Since the bucket objects are children of the bucket, they will be included in the move. However the random pet is not a child resource, so it will not be moved.
The command also warns us about dependencies that will not be transferred to the destination stack. Since we didn’t want to include the pet, but the bucket has a dependency on it, pulumi will warn us about that.
Before going ahead with the move, we get prompted to confirm whether we want to go ahead with it. In this example we do want to go ahead with the move, so we confirm the move, and the resources will be moved for us.

Now that the resources are moved, we need to adjust the program to take advantage of that. The source program will now look as follows:

const randomPet = new random.RandomPet("a-random-pet");
export const randomPetName = randomPet.id;

Only the random pet remains here. We also export its name, so we can continue using it in the destination program. Meanwhile the destination program is mostly just a copy of the original AWS resources, but there is an important change:

const stackRef = new pulumi.StackReference(`v-thomas-pulumi-com/pulumi-demo-move/dev`)

const b = new aws.s3.Bucket("b");

const index = new aws.s3.BucketObject("index.html", {
 bucket: b.bucket,
 content: "Thanks for using Pulumi!",
}, {
 parent: b,
});

const randomSite = new aws.s3.BucketObject("random.html", {
 bucket: b.bucket,
 content: stackRef.getOutput("randomPetName"),
}, {
 parent: b,
});

Note how we now need to specify the content of random.html, since we no longer have the random pet in the same program. In this case we use a stack reference to reference the output from the source program. It is of course up to the user how to re-create the output. It could also come from config, or be hardcoded depending on the use-case.

Refactor your infrastructure safely

Use pulumi state move to split a growing project into focused stacks without recreating your cloud resources. Organize your infrastructure as code and manage every stack in Pulumi Cloud.

Learn more

As always, we would love to hear your feedback in the community slack. If you encounter any issues with the command, please open an issue.

Continue on Error in the Pulumi CLI

Thomas Gummerer — Thu, 09 May 2024 00:00:00 +0000

When managing many resources with Pulumi, a pulumi up or pulumi destroy can lead to a complex graph of resource operations being performed as concurrently as possible relative to the dependencies in your program. If one of those operations fails, you may have noticed that Pulumi takes the most conservative approach, letting already in-flight operations complete, but not starting any new operations. Most often, that’s what you want – there might be no point in bringing up more infrastructure if a resource fails.

However, in some cases it can be useful to keep going to try to bring resources that are independent from the failed one into the requested state, be that resources being created or destroyed. For example, when doing a pulumi destroy, you may want to have Pulumi destroy as many resources as it can, without stopping when the first error occurs.

You can now do exactly that with the new --continue-on-error flag for pulumi up and pulumi destroy.

Using --continue-on-error means that resources that are not in the same dependency tree as the failed resource will still continue to be updated or destroyed, as they would normally. To make sure dependencies are still correctly respected, resources that depend on a successful update or destroy of the failed resource will not continue to be updated. This means that this flag is always safe to use, as Pulumi will continue to manage the failed resources, and they can be updated or destroyed in subsequent runs of Pulumi.

When the execution finishes, Pulumi will report the resource failures as you would currently expect from a failure, and exit with a non-zero exit code. This indicates that even though we continued to update resources independent from the failed one, there was an error during the deployment.

What do I need to do?

pulumi destroy --continue-on-error was introduced in Pulumi v3.112.0. After upgrading to this version or later of the Pulumi CLI you can go ahead and use this feature.

pulumi up --continue-on-error was introduced in Pulumi v3.114.0. However, it also requires updated language SDKs to work best. Since we now have resources that fail to create, but the Pulumi program continues to execute, the SDK has to deal with the outputs of these resources. Currently during pulumi up there are no “unknown” values expected by the SDK. Since we may now have a failed resource, the outputs for that resource might still be unknown, and the SDK will have to deal with that. This means that to fully support this feature some SDK changes were necessary. So in addition to upgrading the Pulumi CLI to v3.114.0, you will also have to upgrade the Pulumi SDK to v3.114.0 or later for Go, Python and Node.js, and 3.63.0 or later for .NET. For Pulumi YAML programs, you’ll need to use v3.115.0 or later of the Pulumi CLI.

When --continue-on-error is used with older SDK versions, the Pulumi engine will return an error to your program, which then will need to be handled, or the rest of the program might not be executed, and thus some resources may not be updated.

This feature was a highly requested feature in the Pulumi open source project, and we’re constantly working to address these highly requested features across the open source projects, so jump in and give features you’d like to see us add a 👍.

Give --continue-on-error a try and let us know what you think!

Introducing Dev Releases for the Pulumi CLI and SDKs

Thomas Gummerer — Tue, 02 Apr 2024 00:00:00 +0000

At Pulumi, the stability of our CLI and SDK releases is critically important. A lot of infrastructure is built and deployed using Pulumi, and any bugs in how that infrastructure is deployed can cause outages. While we put a lot of emphasis on unit and integration testing, we are also constantly investing in new ways to provide additional safeguards to ensure we always deliver stable releases. To that end, we are introducing new dev releases of our CLI and SDKs, which will enable users who want to work with the very latest bits to try them out before they are deployed widely to the entire Pulumi user base. This blog post will walk you through how we are using them internally, and how you can too!

What are dev releases?

On every push to the main branch, in other words every time a pull request is merged, we publish a new dev release for both the CLI and the Pulumi core SDKs. For the CLI, we publish the binaries for all supported platforms to S3, while for the SDKs we create pre-release versions in whatever form the various package managers support.

This means these dev releases always include all the latest features and bug fixes, that will be in the next stable release. And while it was always possible to get the latest CLI or SDKs with some moderate effort depending on the platform and language, having these be proper releases makes for a much more standardized experience on how to install them.

Dev releases of the CLI can be installed on Linux, macOS, and Windows. For example, on Linux or macOS:

$ curl -fsSL https://get.pulumi.com | sh -s -- --version dev

Dev SDKs are currently available for Node.js, Python, .NET, and Go. For example, installing the dev release of the Node.js SDK with npm:

$ npm add @pulumi/pulumi@dev

How are we using them internally at Pulumi?

At Pulumi we are not only developers of our product, but we also love using it internally. All deploys for Pulumi Cloud are orchestrated by Pulumi. In addition we use Pulumi for managing other things such as this blog, and even access management to services we are using is partly done with Pulumi. All of these are a great way for us to test our software in real world use cases, before releasing it.

In addition to that, we also use the dev releases for testing both our templates and examples. This is a huge set of Pulumi programs, that can help figure out problems early.

How you can leverage dev releases

In addition to making it easier for us to test new code that gets merged, and making sure we can release the most stable software possible, we also wanted to make it easier for users to use these dev releases. There’s a couple reasons why these could help your organization:

A bug fix has been merged, but not yet released? You can get it a few minutes after it is merged via the dev release channel
You want to make sure your production stacks work well with the latest version of Pulumi? Use dev releases with your dev stacks for some early testing. We’d be delighted with any bugs that are reported before they make it into a release.

How to get started

For instructions on how to use the CLI dev releases see the Install Docs, and for pre-release versions of dev SDKs see the docs for Node.js, Python, .NET, and Go.

Happy building!