Pulumi Blog: Robbie McKinstry

Benchmarking Python Performance

Justin Van Patten — Thu, 28 Sep 2023 00:00:00 +0000

This is the second post in a series about performance optimizations we’ve made to the Pulumi CLI and SDKs. In this post, we’ll go deep on a performance improvement we made for Pulumi Python programs. You can read more about Amazing Performance in the first post in the series.

Late last year, we took a hard look at the performance of Python programs when we realized they weren’t performing up to our expectations. We uncovered a major bug limiting Python performance, and we ran a number of rigorous experiments to evaluate just how performant Pulumi Python programs are after the bug had been repaired. The results indicate Pulumi Python programs are significantly faster than they were, and now Pulumi Python has reached performance parity with Pulumi Node.js!

The Bug

When you execute a Pulumi program, Pulumi internally builds a dependency graph between the resources in your program. In every Pulumi program, some resources have all their input arguments available at the time of their construction. In contrast, other resources may depend on Outputs from other resources.

For example, consider a sample program where we create two AWS S3 buckets, where one bucket is used to store logs for the other bucket:

import pulumi
import pulumi_aws as aws

log_bucket = aws.s3.Bucket("logBucket", acl="log-delivery-write")

bucket = aws.s3.Bucket("bucket",
 acl="private",
 loggings=[aws.s3.BucketLoggingArgs(
 target_bucket=log_bucket.id,
 target_prefix="log/",
 )])

Because bucket takes an Output from log_bucket as an input, we can’t create the bucket until after the log_bucket is created. We have to create the log_bucket first to compute its ID, which we can pass to bucket. This idea extends inductively for arbitrary programs – before any resource can be run, we must resolve the Outputs of all of its arguments. To do this, Pulumi builds a dependency graph between all resources in your program. Then, it walks the graph topologically to schedule provisioning operations.

Provisioning operations that are not dependent on each other can be executed in parallel, and Pulumi defaults to unbounded parallelism, but users can ratchet this down if they so desire. Consider this embarrassingly parallel Python program:

import pulumi
import pulumi_aws as aws

# SQS
for i in range(100):
 name = f'pulumi-{str(i).rjust(3, "0")}'
 aws.sqs.Queue(name)

# SNS
for i in range(100):
 name = f'pulumi-{str(i).rjust(3, "0")}'
 aws.sns.Topic(name)

In this program, we can create 200 resources in parallel because none of them take inputs from other resources. This program should be entirely network-bound because Pulumi can issue all 200 API calls in parallel and wait for AWS to provision the instances. We discovered, however, that it did not! Strangely, API calls were issued in an initial batch of 20; as one completed, another would start.

The Fix

The culprit was the Python default future executor, ThreadPoolExecutor. We observed that benchmark was run on a four-core computer, and in Python 3.5 to Python 3.7, the number of max workers is five times the number of cores, or 20 (in Python 3.8, this number was changed to min(32, os.cpu_count() + 4)). We realized we shouldn’t be using the default ThreadExecutor, and instead we should provide a ThreadExecutor with an adjusted number of max_workers based on the configured parallelism value. That way, when users run pulumi up --parallel, which issues an upper bound on parallel resource construction, the ThreadExecutor will respect that bound. We merged a fix that would plumb the value of --parallel through to a custom ThreadExecutor and measured the impact this change had on the performance of our benchmark.

Experimental Setup

We designed and implemented two independent experiments to evaluate this change. The first experiment measures how well the patched Python runtime stacks up against the control group, Pulumi Python without the patch. The second experiment compares Pulumi Python to Pulumi TypeScript using the same benchmark ported to TypeScript. We used the awesome benchmarking tool hyperfine to record wall clock time as our indicator of performance.

The experiments ran overnight on a 2021 MacBook Pro with 32GB RAM, the M1 chip, and 10 cores. Experimental code is available on GitHub, and release tags pin the version of the code used for each experiment. We also made an effort to run the experiments on a quiet machine connected to power. For all experiment groups, --parallel was unset, translating to unbounded parallelism.

Before between samples, we ran pulumi destroy –yes to ensure a fresh environment. Hyperfine measures shell startup time and subtracts the value before final measurements are recorded to more precisely represent the true cost of execution. All groups collected 20 samples each. We also discard stderr and stdout to reduce noise associated with logging to a tty, but we do record the status code of each command so can show they executed successfully.

Python: Pre- and Post-patch

This experiment compares the performance of Pulumi Python before and after the patch was applied. The control group used Pulumi v3.43.1, while the experimental group used Pulumi v3.44.3. The primary difference between these two groups is that a fix was introduced for a Python runtime concurrency bug as part of v3.44.0. Both groups use the same benchmark program, which created 100 AWS SNS and 100 AWS SQS resources in parallel, as described earlier. Only the version of the Pulumi CLI is different between groups.

Control vs. Fix

Group	Mean	Standard Deviation
Control	222.232 s	0.908 s
Experimental	70.189 s	1.497 s

Summary: The Experimental Group ran 3.17 ± 0.07 times faster than the Control Group, accounting for a +300% speedup in performance. Running Welch T-Test indicated statical significance (p = 2.93e-59, α=0.05).

Python vs. TypeScript

After seeing very promising results from the first experiment, we wanted to determine just how promising these results were. We decided to compare Pulumi Python to Pulumi TypeScript to see if this fix had narrowed the gap in performance between the two runtimes. We ported the Python program to TypeScript:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// SQS
[...Array(100).map((_, i) => {
 const name = `pulumi-${i}`;
 new aws.sqs.Queue(name);
})];

// SQS
[...Array(100).map((_, i) => {
 const name = `pulumi-${i}`;
 new aws.sqs.Queue(name);
})];

For this experiment, we fixed the version of the CLI to v3.44.3, which included the patch to the Python runtime. Here are the result.

TypeScript vs. Python

Group	Mean	Standard Deviation
Python	70.975 s	0.909 s
TypeScript	73.741 s	1.574 s

Summary: The Python Group performed the best and ran 1.04 ± 0.03 times faster than the TypeScript Group. This accounts for a 4% difference in performance. A second T-Test indicated statical significance (p = 1.4e-07, α=0.05). Not only did Python close the gap with TypeScript, but it is also now marginally faster than its Node.js competitor.

Conclusion

It’s rare to have a small PR result in such a massive performance increase, but when it happens, we want to shout it from the rooftops. This change, which shipped last year in v3.44.3, does not require Python users to opt-in; their programs are now faster. This patch has closed the gap with the Node.js runtime. Users can now expect highly parallel Pulumi programs to run in a similar amount of time between either language.

Artifacts

You can check out the artifacts of the experiments on GitHub, including the source code.

Here are some useful links:

The GitHub repository
Artifacts from the first experiment, “Control vs. Fix” or “Pre- and Post-patch”.
More statistics about the first experiment.
Artifacts from the second experiment
More statistics about the second experiment.
Pulumi Internals

Manage Pulumi Teams in Bulk with the New CLI Teams Flag

Robbie McKinstry — Fri, 31 Mar 2023 00:00:00 +0000

We’ve been hearing feedback from our customers that they need ways to manage permissions for their stacks at scale. Today we are announcing a --teams flag for pulumi stack init, which allows customers to assign Teams to stacks from the CLI. This flag offers a third programmatic method for assigning permissions, supplementing Pulumi Service REST API or the Pulumi Service Provider. Developers can now initialize their stacks with the right permissions directly from the CLI.

Here’s some background for any Pulumi newcomers. Pulumi projects consist of stacks, which are separate deployments of the same infrastructure. Stacks may also be configured with separate inputs. Organization admins can leverage RBAC to grant access permissions at the organization or Pulumi Teams levels. Pulumi Teams and RBAC are available for Enterprise and Business Critical customers.

Stacks are often created via the CLI with the pulumi stack init command. This command initializes a new stack. If you’re using the Pulumi Service as your backend, you can view your newly created stack in the Pulumi Service console. If your organization uses teams, you will want to give your teammates access to the stack.

With the release of Pulumi v3.59.0, developers can assign Pulumi Team access during stack creation. To do this, pass in the --teams flag followed by the team’s name. For instance:

pulumi stack init --teams Red

This saves time by eliminating context switching, allowing users to stay on task. Previously, the Service console was the most direct way to assign team access to a stack, meaning a trip to the browser before your teammates could access the stack. The --teams flag improves upon the process of managing stack permissions by removing this step.

Use the flag multiple times to assign access to multiple Pulumi Teams, as in:

pulumi stack init --teams Red --teams Blue

Currently, the feature always grants read and write access, the most commonly assigned permissions.

We implemented this feature as a direct result of feedback from daily users. Context switching can be expensive when you’re creating enough stacks. At Pulumi, we enable our users to squeak out as much productivity from our tooling as possible. We’re always happy to make ergonomic improvements. We look forward to continuing to raise the bar for cloud productivity.

Achieving Amazing Performance in the Pulumi CLI

Robbie McKinstry — Thu, 26 Jan 2023 00:00:00 +0000

This is the first post in a series about performance optimizations we’ve made to the Pulumi CLI. Over the last six months at Pulumi, the Platform Team has been working on a project we call “Amazing Performance.” Amazing Performance is a new initiative to improve the throughput and latency of the Pulumi CLI not only for power users but for everyone. By the end of June 2022, we assembled a list of issues containing both high-value improvements requiring a sizable investment and low-hanging fruit for quick wins. The full list, including the items we have yet to tackle, is contained in a tracking issue on GitHub. This blog series will cover the highlights.

This post has two sections. First, we’ll describe the tools we built previously to track performance. Secondly, we’ll recap a few quick wins. Future posts will detail our major wins as part of the Amazing Performance initiative.

Measuring and Tracking Performance

The Amazing Performance initiative didn’t come from a vacuum. We’ve been working to understand and improve the performance of the Pulumi CLI for years. Much of the earlier effort has focused on gaining performance insights – building an understanding of where our performance pains were the sharpest. The two most impactful tools at our disposal for understanding performance pains are our analytics dashboard and our OpenTracing support.

Analytics Dashboard

We’ve built an analytics dashboard and monitoring system to let us know when performance dips. This system has three components: a suite of benchmarks, a data warehouse for analytics, and an alerting system.

Benchmark Suite: We’ve written a suite of benchmarks measuring different performance characteristics of the CLI. Each benchmark answers a simple question: “How long does the CLI take to run when the plan has no changes?” or “How long does the CLI take to run an empty plan?” In addition to these simple questions, we also have a few benchmarks to stress test creating many resources. We’ve implemented these benchmarks in Go, JavaScript, TypeScript, C#, and Python. Each night, we exercise the benchmarks, capturing samples. Once execution is complete, our nightly job uploads the results to a serverless function which stores them in our data warehouse.

Warehouse Analytics: We use Metabase to query and analyze data in our data warehouse. Pulumi engineers can log into Metabase and view our performance dashboard, which charts the nightly data as a time series, where each data point is the average of that night’s samples.

Plotting the chart as a line graph allows us to identify any performance dips visually. Sometimes the data from our nightly runs are noisy, so it can take a few nights before we can attribute a change in the runtime to a change in the code. In the absence of noise, the line graph allows us to pinpoint the day the regression was introduced so we can leaf through the pull requests merged that day. More often, there’s enough noise in the data that we go through a few days of PRs to find the regression.

Shortly after shipping the performance dashboard internally, we used the initial numbers we observed to establish a service-level objective (SLO).

Each language has an SLO for each of the benchmarks implemented in that language. Our goal for the SLO is to ensure that we set a “do not breach” expectation across the Platform Core team. We want everyone on the team to know when performance slips beyond an acceptable level.

Alerting: What good is an SLO if it’s not observable? Whether the breach comes from a major regression or creeps up over time, any time a benchmark violates an SLO, a Slackbot alerts the Platform Core team that performance has slowed beyond an acceptable level. One objective we had for the Amazing Performance initiative was to ensure there were no alerts by the end of Q3 2022, which we achieved.

Lastly, we established the accuracy of these benchmarks as part of Amazing Performance. We compared the benchmark results we collected in the nightly job with numbers observed on our local laptops and found they were similar. Consequently, we believe our benchmarks are a good predictor of an actual developer’s experience.

Tracing Support

If you ever encounter a particularly sluggish Pulumi program and want to know what is taking so long, you can use Pulumi’s tracing support to look into the details. When run with the --tracing flag, Pulumi will capture an OpenTracing trace of the program’s execution, giving you an idea of where time is spent. We regularly trace programs large and small to see how we can shave off time.

Pulumi can send traces to a Zipkin server, or aggregate them to a local file and let you view them offline. If you provide an HTTP URI to the --tracing flag, then Pulumi assumes that URI points to a web server that accepts Zipkin-format traces, and it will send all tracing data there. If you provide a file URI instead, Pulumi spins up a local Zipkin server to which child processes send their traces. Once execution is complete, Pulumi aggregates the traces into a single local file and spins down the Zipkin server. You can run PULUMI_DEBUG_COMMANDS=1 pulumi view-trace ./up.trace to view the trace in an embedded web UI.

Not all spans are exceptionally well-named, and there are blind spots in our traces. However, traces are typically a reliable way to subdivide the program execution, so hotspots are easily identifiable. In the future, we plan to migrate to OpenTelemetry and improve our trace coverage. (Feel free to get in touch on the Community Slack if you want to help with this effort!)

Quick Wins for Performance Gains

Some of the work we completed for Amazing Performance targeted simple, isolated changes to make the system a little snappier. The rest of this blog details two quick examples.

Skipping a Slow Import

We cut 300ms in boot time a class of programs by dynamically importing TypeScript.

When Pulumi evaluates NodeJS programs, it compiles TypeScript into JavaScript using TS-Node. Consequently, Pulumi bundles the TypeScript compiler into its NodeJS runtime so TS-Node can perform the compilation. However, we noticed JavaScript programs were importing the TypeScript compiler even when unused.

Users typically hit this bug in one of two ways. Some users choose to write Pulumi programs in pure JavaScript. Pulumi was needlessly importing the TypeScript compiler, even when there was nothing to compile.

The second way a user might encounter this bug is in a more advanced deployment pipeline. One common advanced pattern in CI environments or with Automation API is to typecheck Pulumi programs separately from execution. CI will enforce typechecking during pull requests to guarantee that all code hitting the repository’s main branch is valid. Once it’s typechecked and merged, the Pulumi program can precompile the TypeScript into JavaScript before execution. Finally, when it comes time to execute the Pulumi program, the deployment pipeline will run the compiled JavaScript, which is known to be well-typed. The motivation for precompiling comes from using your own preprocessor. Some customers want to ditch TS-Node for compilation and prefer to use SWC, ESBuild, or other options for increased performance, or for consistency with the rest of their codebase.

In both scenarios, importing TypeScript when it’s not used results in a 300ms slowdown. In a small PR, we detect cases that don’t require TypeScript and dynamically import it only when needed.

Timely Lease Renewal

Another quick win involved the renewal of a new lease, which resulted in another 120ms shaved off of boot time. When the Pulumi CLI kicks off a preview, projects that use the Service backend are authenticated with the Service so the CLI can fetch the current state. During authentication, the CLI exchanges user credentials for a short-lived access token. This access token is valid for a fixed amount of time, called a lease, after which they expire. The CLI can renew a lease by making another request to the Service to extend the duration for which the lease is valid.

We noticed that the first time the CLI renewed the lease, it blocked further execution until the renewal was complete, introducing an unnecessary slowdown of 120ms. The fix was to remove this renewal from the critical path by running it in the backend. By starting the HTTP call on a background thread and only blocking if the token is needed but not renewed, we’ve eliminated the 120ms slowdown.

Since this bug affects anyone using the Pulumi Service backend, most users will benefit from this improvement. It’s small, but it’s noticeable to most users. Program start-up will feel slightly more snappy.

Stay Tuned!

This post covered tools we built before Amazing Performance and a few smaller changes we made as part of the initiative. Future posts will describe some of the larger advances we made to drive Pulumi performance ever forward! Keep an eye out for our next post in the series!