Pulumi Blog: Performance

Now GA: Up to 20x Faster Pulumi Operations for Everyone

Thomas Gummerer — Thu, 05 Mar 2026 00:00:00 +0000

In January, we introduced a major performance enhancement for Pulumi Cloud through a fundamental change to how Pulumi manages state that speeds up operations by up to 20x. After a staged rollout across many organizations, it is now enabled by default for every Pulumi Cloud operation. No opt-in required—just use Pulumi CLI v3.225.0+ with Pulumi Cloud. The improvement applies to pulumi up, pulumi destroy, and pulumi refresh; pulumi preview does not modify state, so it is unchanged.

What this means for you

First and foremost, nothing about how you work with pulumi needs to change. Your updates now benefit from better parallelism and should thus complete faster. Before this change, pulumi always saved a full snapshot to the cloud, so the current state could always be recovered if something goes wrong. With journaling, we now only send the state of each operation, which allows us to send these updates in parallel, as long as resources are not related to each other. For the full deep dive, see the blog post linked above.

Production results

Since January, we’ve had many early adopters of journaling. This helped us shake out one final bug on the server side, and journaling has been stable since then. With that we feel confident in rolling this out to all our users.

We’ve also gathered some real-world data on how journaling is performing. The data from the preview period shows some significant improvements for update times. For stacks with fewer than 100 resources, the median improvement is 25.3%, while the p90 improvement is 75.2%, and we’ve seen a p99 improvement of up to 92.6% Meanwhile, for larger stacks, the median improvement is 60.2%. We need more data for stacks with more than 100 resources, we will update this blog once that comes in.

This data already shows the expected significant improvement in update times, especially for larger stacks, though the improvements strongly depend on the shape and type of resources that are being set up. Stacks with many resources, that are quick to update benefit more than smaller stacks with slower to set up resources. For more numbers see also the Benchmarks section in the previous blog post

What you need to do

While this was an opt-in process using the PULUMI_ENABLE_JOURNALING environment variable, this opt-in is no longer required. Just upgrade your Pulumi CLI to v3.225.0+ and use the Pulumi Cloud backend, and journaling will automatically speed up your updates.

If you encounter any issues, reach out on the Pulumi Community Slack or through Pulumi Support. You can also set the PULUMI_DISABLE_JOURNALING=true env variable to opt out of journaling.

Speeding up Pulumi Operations by up to 20x

Thomas Gummerer — Mon, 12 Jan 2026 18:57:55 +0200

Today we’re introducing an improvement that can speed up operations by up to 20x. At every operation, and at every step within an operation, pulumi saves a snapshot of your cloud infrastructure. This gives pulumi a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here’s how we fixed it.

Benchmarks

Before getting into the more technical details, here are a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks we picked a couple of Pulumi projects: one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another that looks a little more like a real world example. Note that we conducted all of these benchmarks in Europe, connecting to Pulumi Cloud, which runs in AWS’s us-west-2 region, so exact numbers may vary based on your location and internet connection. This should however give a good indication of the performance improvements.

We’re benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We’re using the aws-ts-static-website example here with a small subset of the fraction from our docs site. This means we’re setting up more than 3000 bucket objects, with 3222 resources in total.

The benchmarks were measured using the time built-in command and using the best time in a best-of-three benchmark. The network traffic was measured using tcpdump, limiting the measured traffic to only the IP addresses for Pulumi Cloud. Finally, tshark was used to process the packet captures and count the bytes sent.

All the benchmarks are run with journaling off (the default experience) and with journaling on (the new experience). To begin with, let’s look at the results when creating our stack from scratch:

	Time	Bytes sent
Without journaling	58m26s	16.5MB
With journaling	02m50s	2.3MB

Now let’s have a look at what this looks like if we only change half the resources, but the remaining ones remain unchanged:

	Time	Bytes sent
Without journaling	34m49s	13.8MB
With journaling	01m45s	2.3MB

The second example is setting up an instance of the Pulumi app and API. Here we’ll have an example that’s a bit more dominated by the cost of setting up the actual infrastructure in the cloud, but we still have a very noticeable improvement in the time it takes to set up the stack.

	Time	Bytes sent
Without journaling	17m52s	18.5MB
With journaling	9m12s	5.9MB

To use this feature, you need a pulumi version newer than v3.211.0, and set the PULUMI_ENABLE_JOURNALING environment variable to true.

If you are interested in the more technical details read on!

Introduction to snapshotting

pulumi keeps track of all resources in a stack in a snapshot. This snapshot is stored in the stack’s configured backend, which is either the Pulumi Cloud or a DIY backend. Future operations on the stack then use this snapshot to figure out which resources need to be created, updated or deleted.

pulumi creates a new snapshot at the beginning and at the end of each resource operation to minimize the possibility of untracked changes even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs).

At the beginning of the operation, pulumi adds a new “pending operation” to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but pulumi couldn’t record the end of it), the next operation will try to resolve this. If we have an ID for the resource already, for example on partial updates/deletes, pulumi will try to read the resource state from the cloud and resolve it internally. If there is no ID yet, pulumi will ask the user to check the actual state of the resource. Depending on the user’s response, pulumi will either remove the operation from the snapshot or import the resource. This is because it is possible that the resource has been set up correctly or that the resource creation failed. If pulumi aborted midway through the operation, it’s impossible to know which state the resource is in.

Once an operation finishes, the pending operation is removed and the resource’s final state is recorded in the snapshot.

There’s also some additional metadata that is stored in the snapshot that is only updated infrequently.

Here’s how the snapshot looks in code. This snapshot is serialized and sent to the backend. Resources holds the list of known resource states and is updated after each operation finishes, and PendingOperations is the list of pending operations described above.

type Snapshot struct {
 Manifest Manifest // a deployment manifest of versions, checksums, and so on.
 SecretsManager secrets.Manager // the secrets manager to use when serializing this snapshot.
 Resources []*resource.State // all resources and their associated states.
 PendingOperations []resource.Operation // all currently pending resource operations.
 Metadata SnapshotMetadata // metadata associated with the snapshot.
}

Before we dive in deeper, we also need to understand a little bit about how the pulumi engine works internally. Whenever a pulumi operation is run (e.g. pulumi up, pulumi destroy, pulumi refresh etc.), the engine internally generates and executes a series of steps, to create, update, delete etc. resources. To maintain correct relationships between resources, the steps need to be executed in a partial order such that no step is executed until all of the steps it depends on have executed successfully. Steps may otherwise execute concurrently.

As each step is responsible for updating a single resource, we can generate a snapshot of the state before each step starts, and after it completes. Before each step starts, we create a pending operation, and add it to the PendingOperations list. After that step completes, we remove the pending operation from that list, and update the Resources list, either adding a resource, removing it, or updating it, depending on the kind of operation we just executed.

After this introduction, we can dive into what’s slow, how we fixed it, and some benchmarks.

Why is it slow?

To make sure the state is always as up-to-date as possible, even if there are any network hiccups/power outages etc., a step won’t start until the snapshot that includes the pending operation is confirmed to be stored in the backend. Similarly an operation won’t be considered finished until the snapshot with an updated resources list is confirmed to be stored in the backend.

To send the current state to the backend, we simply serialize it as a JSON file, and send it to the backend. However, as mentioned above, steps can be executed in parallel. If we uploaded the snapshot at the beginning and end of every step with no serialization, there would be a risk that we overwrite a new snapshot with an older one, leading to incorrect data.

Our workaround for that is to serialize the snapshot uploads, uploading one snapshot at a time. This gives us the data integrity properties we want, however it can slow step execution down, especially on internet connections with lower bandwidth, and/or high latency.

This impacts performance especially for large stacks, as we upload the whole snapshot every time, which can take some time if the snapshot is getting big. For the Pulumi Cloud backend we improved on this a little at the end of 2022. We implemented a diff based protocol, which is especially helpful for large snapshots, as we only need to send the diff between the old and the new snapshot, and Pulumi Cloud can then reconstruct the full snapshot based on that. This reduces the amount of data that needs to be transferred, thus improving performance.

However, the snapshotting is still a major bottleneck for large pulumi operations. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. Furthermore, the time spent performing textual diffs between snapshots scales in proportion to the size of the data being processed, which adds additional execution time to each operation.

Fast, but lacking data integrity?

As long as pulumi can complete its operation, there’s no need for the intermediate checkpoints. We could allow pulumi operations to skip uploading the intermittent checkpoints to the backend. This, of course, avoids the single serialization point we have sending the snapshots to the backend, and thus makes the operation much more performant.

However, it also has the serious disadvantage of compromising some of the data integrity guarantees pulumi gives you. If anything goes wrong during the update, pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist.

Neither of these solutions is very satisfying, as the tradeoff is either performance or data integrity. We would like to have our cake and eat it too, and that’s exactly what we’re doing with journaling.

Enter journaling

To achieve this, we went back to the drawing board, and asked ourselves, “What would a solution look like that’s both performant and preserves data integrity throughout the update?”.

Making that happen is possible because of three facts:

We always start with the same snapshot on the backend and the CLI.
Every step the engine executes affects only one resource.
We have a service that can reconstruct a snapshot from what is given to it.

(The third point here already hints at it, but this feature is only available and made possible by Pulumi Cloud, not on the DIY backend).

What if instead of sending the whole snapshot, or a diff of the snapshot, we could send the individual changes to the base snapshot to the service, which could then apply it, and reconstruct a full snapshot from it? This is exactly what we are doing here, in the form of what we call journal entries. Each journal entry has the following form:

const (
 JournalEntryKindBegin JournalEntryKind = 0
 JournalEntryKindSuccess JournalEntryKind = 1
 JournalEntryKindFailure JournalEntryKind = 2
 JournalEntryKindRefreshSuccess JournalEntryKind = 3
 JournalEntryKindOutputs JournalEntryKind = 4
 JournalEntryKindWrite JournalEntryKind = 5
 JournalEntryKindSecretsManager JournalEntryKind = 6
 JournalEntryKindRebuiltBaseState JournalEntryKind = 7
)

type JournalEntry struct {
 // Version of the journal entry format.
 Version int `json:"version"`
 // Kind of journal entry.
 Kind JournalEntryKind `json:"kind"`
 // Sequence ID of the operation.
 SequenceID int64 `json:"sequenceID"`
 // ID of the operation this journal entry is associated with.
 OperationID int64 `json:"operationID"`
 // ID for the delete Operation that this journal entry is associated with.
 RemoveOld *int64 `json:"removeOld"`
 // ID for the delete Operation that this journal entry is associated with.
 RemoveNew *int64 `json:"removeNew"`
 // PendingReplacementOld is the index of the resource that's to be marked as pending replacement
 PendingReplacementOld *int64 `json:"pendingReplacementOld,omitempty"`
 // PendingReplacementNew is the operation ID of the new resource to be marked as pending replacement
 PendingReplacementNew *int64 `json:"pendingReplacementNew,omitempty"`
 // DeleteOld is the index of the resource that's to be marked as deleted.
 DeleteOld *int64 `json:"deleteOld,omitempty"`
 // DeleteNew is the operation ID of the new resource to be marked as deleted.
 DeleteNew *int64 `json:"deleteNew,omitempty"`
 // The resource state associated with this journal entry.
 State *ResourceV3 `json:"state,omitempty"`
 // The operation associated with this journal entry, if any.
 Operation *OperationV2 `json:"operation,omitempty"`
 // If true, this journal entry is part of a refresh operation.
 RebuildDependencies bool `json:"isRefresh,omitempty"`
 // The secrets manager associated with this journal entry, if any.
 SecretsProvider *SecretsProvidersV1 `json:"secretsProvider,omitempty"`

 // NewSnapshot is the new snapshot that this journal entry is associated with.
 NewSnapshot *DeploymentV3 `json:"newSnapshot,omitempty"`
}

These journal entries encode all the information needed to reconstruct the snapshot from them. Each journal entry can be sent in parallel from the engine, and the snapshot will still be fully valid. All journal entries have a Sequence ID attached to them, and they need to be replayed in that order on the service side to make sure we get a valid snapshot. It is however okay to replay without journal entries that have not yet been received by the service, and whose sequence ID is thus missing. This is safe because the engine only sends entries in parallel for resources whose parents/dependencies have been fully created and confirmed by the service.

This way we make sure that the resources list is always in the correct partial order that is required by the engine to function correctly, and for the snapshot to be considered valid.

The algorithm looks as follows:

# Apply snapshot writes. This replaces the full snapshot we have on the service.
# We do this if default providers change, because we don't emit steps for that, as
# we do for the rest of the operations.
snapshot = find_write_journal_entry_or_use_base(base, journal)
# Track changes
deletes, snapshot_deletes, mark_deleted, mark_pending = set(), set(), set(), set()
operation_id_to_resource_index = {}
# Process journal entries. This is the main algorithm, that adds new resources
# to the snapshot, removes existing ones, deals with refreshes, and operations
# that update outputs.
incomplete_ops = {}
has_refresh = false
index = 0
for entry in journal:
match entry.type:
case BEGIN:
incomplete_ops[entry.op_id] = entry
case SUCCESS:
del incomplete_ops[entry.op_id]
if entry.state and entry.op_id:
resources.append(entry.state)
operation_id_to_resource_index.add(entry.op_id, index)
index++
if entry.remove_old:
snapshot_deletes.add(entry.remove_old)
if entry.remove_new:
deletes[remove_new] = true
if entry.pending_replacement:
mark_pending(entry.pending_replacement)
if entry.delete:
mark_deleted(entry.delete)
has_refresh |= entry.is_refresh
case REFRESH_SUCCESS:
del incomplete_ops[entry.op_id]
has_refresh = true
if entry.remove_old:
if entry.state:
snapshot_replacements[entry.remove_old] = entry.state
else:
snapshot_deletes.add(entry.remove_old)
if entry.remove_new:
if entry.state:
deletes[entry.remove_new] = true
else:
resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state)
case FAILURE:
del incomplete_ops[entry.op_id]
case OUTPUTS:
if entry.state and entry.remove_old:
snapshot_replacements[entry.remove_old] = entry.state
if entry.state and entry.remove_new:
resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state)
deletes = deletes.map(|i| => operation_id_to_resource_index[i])
# Now that we have marked all the operations, and created a new list of resources, we can
# go through them, and merge the list of new resources and old resources from the snapshot
# that remain together.
for i, res in resources:
if i in deletes:
remove_from_resources(resources, i)
# Merge snapshot resources. These resources have not been touched by the update, and will
# thus be appended to the end of the resource list. We also need to mark existing resources as
# `Delete` and `PendingReplacement` here.
for i, res in enumerate(snapshot.resources):
if i not in snapshot_deletes:
if i in snapshot_replacements:
resources.append(snapshot_replacements[i])
else:
if i in mark_deleted:
res.delete = true
if i in mark_pending:
res.pending_replacement = true
resources.append(res)
# Collect pending operations. These are stored separately from the resources list
# in the snapshot.
pending_ops = [op.operation for op in incomplete_ops.values() if op.operation]
pending_ops.extend([op for op in snapshot.pending_ops if op.type == CREATE])
# Rebuild dependencies if necessary. Refreshes can delete parents or dependencies
# of resources, without affecting the resource itself directly. We need to now remove
# these relationships to make sure the snapshot remains valid.
if has_refresh:
rebuild_dependencies(resources)

The full documentation of the algorithm can be found in our developer docs.

Rollout

pulumi state is a very central part of pulumi, so we wanted to be extra careful with the rollout to make sure we don’t break anything. We did this in a few stages:

We implemented the replay interface inside the pulumi CLI, and ran it in parallel with the current snapshotting implementation in our tests. The snapshots were then compared automatically, and tests made to fail when the result didn’t match.
Since tests can’t cover all possible edge cases, the next step was to run the journaler in parallel with the current snapshotting implementation internally. This was still without sending the results to the service. However we would compare the snapshot, and send an error event to the service if the snapshot didn’t match. In our data warehouse we could then inspect any mismatches, and fix them. Since this does involve the service in a minor way, we would only do this if the user is using the Cloud backend.
Next up was adding a feature flag for the service, so journaling could be turned on selectively for some orgs. At the same time we implemented an opt-in environment variable in the CLI (PULUMI_ENABLE_JOURNALING), so the feature could be selectively turned on by users, if both the feature flag is enabled and the user sets the environment variable. This way we could slowly start enabling this in our repos, e.g. first in the integration tests for pulumi/pulumi, then in the tests for pulumi/examples and pulumi/templates, etc.
Allow users to start opting in. If you want to opt-in with your org, please reach out to us, either on the Community Slack, or through our Support channels, and we’ll opt your org into the feature flag. Then you can begin seeing the performance improvements by setting the PULUMI_ENABLE_JOURNALING env variable to true.
Turn on the feature flag for everyone, but still require the PULUMI_ENABLE_JOURNALING env variable to be set to true. (We are here right now).
Flip the feature on by default, but still allow users to opt out using a PULUMI_DISABLE_JOURNALING env variable.

What’s next

While these performance improvements hopefully make your day to day use of pulumi quicker and more enjoyable, we’re not quite done here. We’re looking at some other performance improvements, that will hopefully speed up your workflows even more.

Faster Secrets in Pulumi IaC

Daniel Bradley — Thu, 13 Mar 2025 01:00:00 -0700

Pulumi now handles secrets more efficiently through optimized encryption and decryption processes, reducing deployment times while maintaining security standards. Users of Pulumi Cloud for state management will notice the most improvement due to new batch API capabilities.

Secrets Management in Pulumi

Pulumi Infrastructure as Code (IaC)’s built-in secrets management encrypts individual secrets within the stack for fine-grained protection. Pulumi Cloud transmits and stores stack state securely, ensuring that sensitive information—such as database passwords, API keys, and access tokens—remains protected even if someone gains access to your stack file. For organizations with more complex secrets management needs, Pulumi also offers Pulumi ESC (Environments, Secrets, and Configuration), which provides centralized secrets management with hierarchical environments and dynamic credential generation.

Beyond simple encryption, Pulumi tracks the transitive use of secrets to prevent accidental exposure of sensitive values. This tracking works across your entire infrastructure definition, ensuring secrets remain protected:

Within the stack state
In CLI output
In the Pulumi Cloud console

When using Pulumi Cloud as your secrets provider, all encryption and decryption operations happen server-side without transmitting the encryption key to the Pulumi program. Combined with Pulumi Cloud’s robust user authorization and SSO integration, this creates a comprehensive security framework that ensures only authorized team members can access sensitive information.

Performance Challenges with Secrets

While security is paramount, performance is equally crucial for developer productivity. Every second spent waiting for encryption or decryption operations is time not spent building and improving your infrastructure.

Previously, when working with stacks containing numerous secrets, users could experience noticeable delays during operations that required encryption or decryption. This was primarily due to:

Individual network requests for each secret operation
Network latency compounding with each separate request

For teams with complex infrastructure containing dozens or hundreds of secrets, these delays could add up to significant wait times.

Our Performance Improvements

We’ve reduced network overhead by batching encryption work into a single request instead of making separate encryption requests for each secret. This batching approach:

Reduces the total number of HTTP requests
Minimizes the impact of network latency
Allows for more efficient server-side processing

This is combined with identifying when no secrets have changed between intermediate deployment steps to bypass the work of re-encrypting all secrets.

Measurable Performance Gains

The performance improvements are particularly noticeable for stacks with many secrets or in environments with higher network latency:

Secrets Count	Network Latency	Typical Time Savings
10	100ms	~1 second
25	150ms	~4 seconds
50+	150ms	10+ seconds
100+	150ms	20+ seconds

The performance improvements are noticeable for stacks with many secrets or in environments with higher network latency. For teams running CI/CD pipelines with frequent deployments, these savings compound throughout the day, potentially saving significant cumulative wait time across your engineering organization.

Getting Started

To benefit from these performance improvements:

Update to Pulumi IaC version 3.155.0 or later
No configuration changes are needed—the optimizations work automatically
Ensure you’re using Pulumi Cloud as your secrets provider for additional speed gains

These improvements maintain complete backward compatibility with existing stacks and secret values. Your existing encrypted secrets will continue to work without any migration steps required.

Conclusion

At Pulumi, we’re committed to providing both robust security and exceptional performance. These secrets management optimizations represent our ongoing effort to improve the developer experience without compromising on security.

By reducing wait times during encryption and decryption operations, we’re helping teams be more productive while maintaining the same high level of protection for sensitive information. This update exemplifies our philosophy that security and performance should complement rather than compete with each other.

Beyond Stack Secrets: Pulumi ESC

While these performance improvements enhance Pulumi’s built-in secrets management, some organizations require more advanced capabilities. If you need secrets to be centrally managed, composable, and reusable across teams and applications, Pulumi ESC (Environments, Secrets, and Configuration) may be a better fit. Pulumi ESC keeps sensitive information out of source control, supports hierarchical environments, and enables dynamic credential generation. It’s designed for organizations that need to share secrets between multiple teams, applications, and infrastructure deployments while maintaining strict access controls and auditability.

Get Started Today

Update to Pulumi IaC 3.155.0 today to experience these performance improvements and let us know what you think!

Benchmarking Python Performance

Justin Van Patten — Thu, 28 Sep 2023 00:00:00 +0000

This is the second post in a series about performance optimizations we’ve made to the Pulumi CLI and SDKs. In this post, we’ll go deep on a performance improvement we made for Pulumi Python programs. You can read more about Amazing Performance in the first post in the series.

Late last year, we took a hard look at the performance of Python programs when we realized they weren’t performing up to our expectations. We uncovered a major bug limiting Python performance, and we ran a number of rigorous experiments to evaluate just how performant Pulumi Python programs are after the bug had been repaired. The results indicate Pulumi Python programs are significantly faster than they were, and now Pulumi Python has reached performance parity with Pulumi Node.js!

The Bug

When you execute a Pulumi program, Pulumi internally builds a dependency graph between the resources in your program. In every Pulumi program, some resources have all their input arguments available at the time of their construction. In contrast, other resources may depend on Outputs from other resources.

For example, consider a sample program where we create two AWS S3 buckets, where one bucket is used to store logs for the other bucket:

import pulumi
import pulumi_aws as aws

log_bucket = aws.s3.Bucket("logBucket", acl="log-delivery-write")

bucket = aws.s3.Bucket("bucket",
 acl="private",
 loggings=[aws.s3.BucketLoggingArgs(
 target_bucket=log_bucket.id,
 target_prefix="log/",
 )])

Because bucket takes an Output from log_bucket as an input, we can’t create the bucket until after the log_bucket is created. We have to create the log_bucket first to compute its ID, which we can pass to bucket. This idea extends inductively for arbitrary programs – before any resource can be run, we must resolve the Outputs of all of its arguments. To do this, Pulumi builds a dependency graph between all resources in your program. Then, it walks the graph topologically to schedule provisioning operations.

Provisioning operations that are not dependent on each other can be executed in parallel, and Pulumi defaults to unbounded parallelism, but users can ratchet this down if they so desire. Consider this embarrassingly parallel Python program:

import pulumi
import pulumi_aws as aws

# SQS
for i in range(100):
 name = f'pulumi-{str(i).rjust(3, "0")}'
 aws.sqs.Queue(name)

# SNS
for i in range(100):
 name = f'pulumi-{str(i).rjust(3, "0")}'
 aws.sns.Topic(name)

In this program, we can create 200 resources in parallel because none of them take inputs from other resources. This program should be entirely network-bound because Pulumi can issue all 200 API calls in parallel and wait for AWS to provision the instances. We discovered, however, that it did not! Strangely, API calls were issued in an initial batch of 20; as one completed, another would start.

The Fix

The culprit was the Python default future executor, ThreadPoolExecutor. We observed that benchmark was run on a four-core computer, and in Python 3.5 to Python 3.7, the number of max workers is five times the number of cores, or 20 (in Python 3.8, this number was changed to min(32, os.cpu_count() + 4)). We realized we shouldn’t be using the default ThreadExecutor, and instead we should provide a ThreadExecutor with an adjusted number of max_workers based on the configured parallelism value. That way, when users run pulumi up --parallel, which issues an upper bound on parallel resource construction, the ThreadExecutor will respect that bound. We merged a fix that would plumb the value of --parallel through to a custom ThreadExecutor and measured the impact this change had on the performance of our benchmark.

Experimental Setup

We designed and implemented two independent experiments to evaluate this change. The first experiment measures how well the patched Python runtime stacks up against the control group, Pulumi Python without the patch. The second experiment compares Pulumi Python to Pulumi TypeScript using the same benchmark ported to TypeScript. We used the awesome benchmarking tool hyperfine to record wall clock time as our indicator of performance.

The experiments ran overnight on a 2021 MacBook Pro with 32GB RAM, the M1 chip, and 10 cores. Experimental code is available on GitHub, and release tags pin the version of the code used for each experiment. We also made an effort to run the experiments on a quiet machine connected to power. For all experiment groups, --parallel was unset, translating to unbounded parallelism.

Before between samples, we ran pulumi destroy –yes to ensure a fresh environment. Hyperfine measures shell startup time and subtracts the value before final measurements are recorded to more precisely represent the true cost of execution. All groups collected 20 samples each. We also discard stderr and stdout to reduce noise associated with logging to a tty, but we do record the status code of each command so can show they executed successfully.

Python: Pre- and Post-patch

This experiment compares the performance of Pulumi Python before and after the patch was applied. The control group used Pulumi v3.43.1, while the experimental group used Pulumi v3.44.3. The primary difference between these two groups is that a fix was introduced for a Python runtime concurrency bug as part of v3.44.0. Both groups use the same benchmark program, which created 100 AWS SNS and 100 AWS SQS resources in parallel, as described earlier. Only the version of the Pulumi CLI is different between groups.

Control vs. Fix

Group	Mean	Standard Deviation
Control	222.232 s	0.908 s
Experimental	70.189 s	1.497 s

Summary: The Experimental Group ran 3.17 ± 0.07 times faster than the Control Group, accounting for a +300% speedup in performance. Running Welch T-Test indicated statical significance (p = 2.93e-59, α=0.05).

Python vs. TypeScript

After seeing very promising results from the first experiment, we wanted to determine just how promising these results were. We decided to compare Pulumi Python to Pulumi TypeScript to see if this fix had narrowed the gap in performance between the two runtimes. We ported the Python program to TypeScript:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// SQS
[...Array(100).map((_, i) => {
 const name = `pulumi-${i}`;
 new aws.sqs.Queue(name);
})];

// SQS
[...Array(100).map((_, i) => {
 const name = `pulumi-${i}`;
 new aws.sqs.Queue(name);
})];

For this experiment, we fixed the version of the CLI to v3.44.3, which included the patch to the Python runtime. Here are the result.

TypeScript vs. Python

Group	Mean	Standard Deviation
Python	70.975 s	0.909 s
TypeScript	73.741 s	1.574 s

Summary: The Python Group performed the best and ran 1.04 ± 0.03 times faster than the TypeScript Group. This accounts for a 4% difference in performance. A second T-Test indicated statical significance (p = 1.4e-07, α=0.05). Not only did Python close the gap with TypeScript, but it is also now marginally faster than its Node.js competitor.

Conclusion

It’s rare to have a small PR result in such a massive performance increase, but when it happens, we want to shout it from the rooftops. This change, which shipped last year in v3.44.3, does not require Python users to opt-in; their programs are now faster. This patch has closed the gap with the Node.js runtime. Users can now expect highly parallel Pulumi programs to run in a similar amount of time between either language.

Artifacts

You can check out the artifacts of the experiments on GitHub, including the source code.

Here are some useful links:

The GitHub repository
Artifacts from the first experiment, “Control vs. Fix” or “Pre- and Post-patch”.
More statistics about the first experiment.
Artifacts from the second experiment
More statistics about the second experiment.
Pulumi Internals

Achieving Amazing Performance in the Pulumi CLI

Robbie McKinstry — Thu, 26 Jan 2023 00:00:00 +0000

This is the first post in a series about performance optimizations we’ve made to the Pulumi CLI. Over the last six months at Pulumi, the Platform Team has been working on a project we call “Amazing Performance.” Amazing Performance is a new initiative to improve the throughput and latency of the Pulumi CLI not only for power users but for everyone. By the end of June 2022, we assembled a list of issues containing both high-value improvements requiring a sizable investment and low-hanging fruit for quick wins. The full list, including the items we have yet to tackle, is contained in a tracking issue on GitHub. This blog series will cover the highlights.

This post has two sections. First, we’ll describe the tools we built previously to track performance. Secondly, we’ll recap a few quick wins. Future posts will detail our major wins as part of the Amazing Performance initiative.

Measuring and Tracking Performance

The Amazing Performance initiative didn’t come from a vacuum. We’ve been working to understand and improve the performance of the Pulumi CLI for years. Much of the earlier effort has focused on gaining performance insights – building an understanding of where our performance pains were the sharpest. The two most impactful tools at our disposal for understanding performance pains are our analytics dashboard and our OpenTracing support.

Analytics Dashboard

We’ve built an analytics dashboard and monitoring system to let us know when performance dips. This system has three components: a suite of benchmarks, a data warehouse for analytics, and an alerting system.

Benchmark Suite: We’ve written a suite of benchmarks measuring different performance characteristics of the CLI. Each benchmark answers a simple question: “How long does the CLI take to run when the plan has no changes?” or “How long does the CLI take to run an empty plan?” In addition to these simple questions, we also have a few benchmarks to stress test creating many resources. We’ve implemented these benchmarks in Go, JavaScript, TypeScript, C#, and Python. Each night, we exercise the benchmarks, capturing samples. Once execution is complete, our nightly job uploads the results to a serverless function which stores them in our data warehouse.

Warehouse Analytics: We use Metabase to query and analyze data in our data warehouse. Pulumi engineers can log into Metabase and view our performance dashboard, which charts the nightly data as a time series, where each data point is the average of that night’s samples.

Plotting the chart as a line graph allows us to identify any performance dips visually. Sometimes the data from our nightly runs are noisy, so it can take a few nights before we can attribute a change in the runtime to a change in the code. In the absence of noise, the line graph allows us to pinpoint the day the regression was introduced so we can leaf through the pull requests merged that day. More often, there’s enough noise in the data that we go through a few days of PRs to find the regression.

Shortly after shipping the performance dashboard internally, we used the initial numbers we observed to establish a service-level objective (SLO).

Each language has an SLO for each of the benchmarks implemented in that language. Our goal for the SLO is to ensure that we set a “do not breach” expectation across the Platform Core team. We want everyone on the team to know when performance slips beyond an acceptable level.

Alerting: What good is an SLO if it’s not observable? Whether the breach comes from a major regression or creeps up over time, any time a benchmark violates an SLO, a Slackbot alerts the Platform Core team that performance has slowed beyond an acceptable level. One objective we had for the Amazing Performance initiative was to ensure there were no alerts by the end of Q3 2022, which we achieved.

Lastly, we established the accuracy of these benchmarks as part of Amazing Performance. We compared the benchmark results we collected in the nightly job with numbers observed on our local laptops and found they were similar. Consequently, we believe our benchmarks are a good predictor of an actual developer’s experience.

Tracing Support

If you ever encounter a particularly sluggish Pulumi program and want to know what is taking so long, you can use Pulumi’s tracing support to look into the details. When run with the --tracing flag, Pulumi will capture an OpenTracing trace of the program’s execution, giving you an idea of where time is spent. We regularly trace programs large and small to see how we can shave off time.

Pulumi can send traces to a Zipkin server, or aggregate them to a local file and let you view them offline. If you provide an HTTP URI to the --tracing flag, then Pulumi assumes that URI points to a web server that accepts Zipkin-format traces, and it will send all tracing data there. If you provide a file URI instead, Pulumi spins up a local Zipkin server to which child processes send their traces. Once execution is complete, Pulumi aggregates the traces into a single local file and spins down the Zipkin server. You can run PULUMI_DEBUG_COMMANDS=1 pulumi view-trace ./up.trace to view the trace in an embedded web UI.

Not all spans are exceptionally well-named, and there are blind spots in our traces. However, traces are typically a reliable way to subdivide the program execution, so hotspots are easily identifiable. In the future, we plan to migrate to OpenTelemetry and improve our trace coverage. (Feel free to get in touch on the Community Slack if you want to help with this effort!)

Quick Wins for Performance Gains

Some of the work we completed for Amazing Performance targeted simple, isolated changes to make the system a little snappier. The rest of this blog details two quick examples.

Skipping a Slow Import

We cut 300ms in boot time a class of programs by dynamically importing TypeScript.

When Pulumi evaluates NodeJS programs, it compiles TypeScript into JavaScript using TS-Node. Consequently, Pulumi bundles the TypeScript compiler into its NodeJS runtime so TS-Node can perform the compilation. However, we noticed JavaScript programs were importing the TypeScript compiler even when unused.

Users typically hit this bug in one of two ways. Some users choose to write Pulumi programs in pure JavaScript. Pulumi was needlessly importing the TypeScript compiler, even when there was nothing to compile.

The second way a user might encounter this bug is in a more advanced deployment pipeline. One common advanced pattern in CI environments or with Automation API is to typecheck Pulumi programs separately from execution. CI will enforce typechecking during pull requests to guarantee that all code hitting the repository’s main branch is valid. Once it’s typechecked and merged, the Pulumi program can precompile the TypeScript into JavaScript before execution. Finally, when it comes time to execute the Pulumi program, the deployment pipeline will run the compiled JavaScript, which is known to be well-typed. The motivation for precompiling comes from using your own preprocessor. Some customers want to ditch TS-Node for compilation and prefer to use SWC, ESBuild, or other options for increased performance, or for consistency with the rest of their codebase.

In both scenarios, importing TypeScript when it’s not used results in a 300ms slowdown. In a small PR, we detect cases that don’t require TypeScript and dynamically import it only when needed.

Timely Lease Renewal

Another quick win involved the renewal of a new lease, which resulted in another 120ms shaved off of boot time. When the Pulumi CLI kicks off a preview, projects that use the Service backend are authenticated with the Service so the CLI can fetch the current state. During authentication, the CLI exchanges user credentials for a short-lived access token. This access token is valid for a fixed amount of time, called a lease, after which they expire. The CLI can renew a lease by making another request to the Service to extend the duration for which the lease is valid.

We noticed that the first time the CLI renewed the lease, it blocked further execution until the renewal was complete, introducing an unnecessary slowdown of 120ms. The fix was to remove this renewal from the critical path by running it in the backend. By starting the HTTP call on a background thread and only blocking if the token is needed but not renewed, we’ve eliminated the 120ms slowdown.

Since this bug affects anyone using the Pulumi Service backend, most users will benefit from this improvement. It’s small, but it’s noticeable to most users. Program start-up will feel slightly more snappy.

Stay Tuned!

This post covered tools we built before Amazing Performance and a few smaller changes we made as part of the initiative. Future posts will describe some of the larger advances we made to drive Pulumi performance ever forward! Keep an eye out for our next post in the series!