Pulumi Blog: Justin Van Patten

Pulumi Java is Now Generally Available

Mark Huber — Mon, 10 Feb 2025 16:15:00 -0800

One of Pulumi’s core Infrastructure as Code (IaC) features is the ability to model infrastructure using well-traveled, familiar general-purpose programming languages. Today, we’re thrilled to announce that Java, one of the world’s most popular programming languages, is now generally available in Pulumi. This release joins our existing first-class support for TypeScript, Python, Go, YAML, and C#, enabling Java developers to manage cloud infrastructure using the language they know and trust.

We’ve grown our Java support based on feedback from the expanding set of companies that have adopted Pulumi Java in production workloads. Let’s look at some examples of using Java to power your Pulumi programs.

Familiar Composability

Using Java with Pulumi lets you model your infrastructure using familiar patterns. Paired with rich abstractions, you can efficiently build Pulumi programs. It only seems appropriate that we take a look at a Java example using Oracle Cloud:

We start our program by grabbing our configuration values that describe which distro and AD to use.

var config = ctx.config();
var compartmentId = config.require("compartmentId");
var availabilityDomain = config.require("availabilityDomain");
var ubuntuId = config.require("ubuntuId");

Next, we create a set of abstractions to describe the networking and OS.

var vcn = new VirtualNetwork("virtualNetworkResource", VirtualNetworkArgs.builder()
 .compartmentId(compartmentId)
 .cidrBlock("10.0.0.0/16")
 .build());

var subnet = new Subnet("testSubnet", SubnetArgs.builder()
 .cidrBlock("10.0.0.0/24")
 .compartmentId(compartmentId)
 .vcnId(vcn.id())
 .availabilityDomain(availabilityDomain)
 .build());

var vnicDetails = InstanceCreateVnicDetailsArgs.builder()
 .subnetId(subnet.id())
 .assignPublicIp("true")
 .build();

var sourceDetails = InstanceSourceDetailsArgs.builder()
 .sourceType("image")
 .sourceId(ubuntuId)
 .build();

Finally, we compose our instance.

var instance = new Instance("instanceResource", InstanceArgs.builder()
 .availabilityDomain(availabilityDomain)
 .compartmentId(compartmentId)
 .shape("VM.Standard.E2.1.Micro")
 .preserveBootVolume(false)
 .createVnicDetails(vnicDetails)
 .sourceDetails(sourceDetails)
 .build());

With just a few lines of code (especially by Java standards), we composed a set of strongly typed, easy-to-reason about resources using the builder pattern (a Java favorite) and without introducing any new language paradigms.

New: Java Automation API

With the general availability of our Java language support, we’re also excited to announce that the Automation API is now supported in Java. The Automation API is a fully typed SDK that allows you to interact with Pulumi programs outside the Pulumi CLI. You can directly access and orchestrate your Pulumi projects and stacks with the SDK, allowing you to supercharge your Infrastructure as Code.

In our Automation API examples repo, we have an example demonstrating how you could construct a Pulumi program with the Automation API to perform a database migration. Let’s take a look at a few of the interesting aspects:

Like regular Pulumi programs, we’re able to obtain the current operating context and use the details to construct new resources, inline with the rest of the Automation API program.

Consumer<Context> program = ctx -> {
 ...

 var subnetGroup = new SubnetGroup("db-subnet", ...);

 // Make a public security group for our cluster for the migration
 var securityGroup = new SecurityGroup("public-security-group", ...);

 // Example values for our db
 var dbName = "hellosql";
 var dbUser = "hellosql";
 var dbPassword = "hellosql";

 // Provision our db
 var cluster = new Cluster("db", ClusterArgs.builder()
 .engine(EngineType.AuroraMysql)
 .engineVersion("8.0.mysql_aurora.3.08.0")
 .databaseName(dbName)
 .masterUsername(dbUser)
 .masterPassword(dbPassword)
 .skipFinalSnapshot(true)
 .dbSubnetGroupName(subnetGroup.name())
 .vpcSecurityGroupIds(securityGroup.id().applyValue(List::of))
 .build());

 var clusterInstance = new ClusterInstance("db-instance", ClusterInstanceArgs.builder()
 .clusterIdentifier(cluster.clusterIdentifier())
 .instanceClass("db.t3.medium")
 .engine(EngineType.AuroraMysql.getValue())
 .engineVersion("8.0.mysql_aurora.3.08.0")
 .publiclyAccessible(true)
 .dbSubnetGroupName(subnetGroup.name())
 .build());

 ctx.export("host", cluster.endpoint());
 ctx.export("db_name", dbName);
 ctx.export("db_user", dbUser);
 ctx.export("db_pass", dbPassword);
};

With the Automation API, however, we’re also able to directly interact with project and stack objects, giving us the opportunity to orchestrate workflows. This could include scaffolding projects, working across stacks, or in our case, completing a one-time workflow to migrate a database.

var projectName = "database_migration_project";
var stackName = "dev";

try (var stack = LocalWorkspace.createOrSelectStack(projectName, stackName, program)) {
 ...

 stack.workspace().installPlugin("aws", "v6.68.0");

 stack.setConfig("aws:region", new ConfigValue("us-west-2"));

 System.out.println("updating stack...");
 var result = stack.up(UpOptions.builder()
 .onStandardOutput(System.out::println)
 .build());

 var changes = result.summary().resourceChanges();
 if (!changes.isEmpty()) {
 System.out.println("update summary:");
 changes.forEach((key, value) -> {
 System.out.printf(" %s: %d%n", key, value);
 });
 }

 System.out.printf("db host url: %s%n", result.outputs().get("host").value());
 configureDatabase(
 ...
 );
}

The full example, along with others, is available in our Automation API examples repo. We’re excited to bring the Automation API to the Java SDK for the first time and we’re looking forward to your feedback.

Get Started with Pulumi Java

To learn more about and to get started with Pulumi Java, you can check out the following resources:

Java SDK Repo: The Java SDK is fully open-source, and we’re excited to get your feedback or review any pull requests.
Examples Repo: The Pulumi examples repo has many examples of Pulumi Java programs to help you get up and running with Pulumi Java.
Pulumi Java Converter: The Pulumi converter tool efficiently migrates existing IaC applications, such as Terraform, ARM, and Bicep, to Pulumi Java.

Introducing Resource Transforms: Enhancing Flexibility for Packaged Component Resources

Fraser Waters — Fri, 19 Jul 2024 00:00:00 +0000

Pulumi has supported a Transformations system for a number of years now. This has proved to be a powerful and flexible escape hatch for modifying resource properties and options across your entire program. For example, you could use Transformations to automatically apply tags to all taggable resources in your program, including the children of component resources.

However, there is one major limitation with the existing Transformations system: it isn’t able to transform the children of packaged component resources, such as those in awsx and eks. This limitation is due to the fact that packaged component resources are created in a separate provider process and Transformations only work with resources created in your program’s process.

To address this limitation we’re introducing a new system called Transforms, which works with all resources, including packaged component resources and their children. The new Transforms system is intended to fully replace the old Transformations system (we plan to deprecate the old system in the future).

Here’s an example that demonstrates adding tags to all Subnet child resources created by the Vpc packaged component:

import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx";

const myTags = { test: "TEST" };

pulumi.runtime.registerResourceTransform(args => {
 if (args.type === "aws:ec2/subnet:Subnet") {
 const tags = pulumi.output(args.props.tags || {});
 args.props.tags = tags.apply(t => ({ ...t, ...myTags }));
 return { props: args.props, opts: args.opts };
 }
 return undefined;
});

const vpc = new awsx.ec2.Vpc("test");

import pulumi
from pulumi_awsx import ec2

my_tags = { "test": "TEST" }

def tag_transform(args):
 if args.type_ == "aws:ec2/subnet:Subnet":
 tags = pulumi.Output.from_input(args.props.get("tags", {}))
 args.props["tags"] = tags.apply(lambda v: {**v, **my_tags})
 return pulumi.ResourceTransformResult(args.props, args.opts)

pulumi.runtime.register_resource_transform(tag_transform)

my_vpc = ec2.Vpc("my-vpc")

package main

import (
 "context"

 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 myTags := map[string]string{"test": "TEST"}

 err := ctx.RegisterResourceTransform(func(_ context.Context, args *pulumi.ResourceTransformArgs) *pulumi.ResourceTransformResult {
 if args.Type == "aws:ec2/subnet:Subnet" {
 tags, ok := args.Props["tags"].(pulumi.StringMapInput)
 if !ok {
 tags = pulumi.StringMap{}
 }
 args.Props["tags"] = tags.ToStringMapOutput().ApplyT(func(m map[string]string) map[string]string {
 result := make(map[string]string)
 for k, v := range m {
 result[k] = v
 }
 for k, v := range myTags {
 result[k] = v
 }
 return result
 })
 return &pulumi.ResourceTransformResult{
 Props: args.Props,
 Opts: args.Opts,
 }
 }
 return nil
 })
 if err != nil {
 return err
 }

 _, err = ec2.NewVpc(ctx, "test", nil)
 if err != nil {
 return err
 }

 return nil
 })
}

using System;
using System.Collections.Immutable;
using Pulumi;
using Pulumi.Awsx.Ec2;

await Deployment.RunAsync<MyStack>();

class MyStack : Stack
{
 public MyStack() : base(new StackOptions
 {
 ResourceTransforms =
 {
 async (args, _) => {
 var myTags = ImmutableDictionary.Create<string, object>()
 .Add("test", "Test");

 if (args.Type == "aws:ec2/subnet:Subnet")
 {
 InputMap<object> tags =
 args.Args.TryGetValue("tags", out var tagsValue) && tagsValue is not null ?
 tagsValue is Output<ImmutableDictionary<string, object>> tagsOutput ? tagsOutput :
 tagsValue is ImmutableDictionary<string, object> tagsDictionary ? tagsDictionary :
 throw new InvalidOperationException($"Unexpected tags type: {tagsValue.GetType()}") :
 ImmutableDictionary<string, object>.Empty;

 tags = tags.Apply(t => t.SetItems(myTags));
 return new(args.Args.SetItem("tags", tags), args.Options);
 }
 return null;
 },
 }
 })
 {
 var vpc = new Vpc("test");
 }
}

Using the new Transforms system is similar to the old system, but there are some differences between the two systems. A guide on Migrating from Transformations to Transforms is available to help transition to the new system.

We’re excited to make it possible to transform all resources in your program, including those from packaged components. If you have any questions or feedback, please don’t hesitate to reach out to us on Slack.

Pulumi Release Notes: Pulumi ESC, Deployments GA, Pulumi AI Improvements, AWS S3 Express One Zone Support, and more!

Arun Loganathan — Thu, 01 Feb 2024 11:06:04 -0800

We’ve had a busy last few months at Pulumi. From shipping a brand new product offering, Pulumi ESC, to adding several major features in Pulumi Cloud and updates to various Pulumi providers - there is lots to catch up on. In addition to reading these release notes, check out the pulumi/pulumi repo changelog to stay up to date with CLI enhancements after each CLI release. For Pulumi Cloud updates, follow the new features blogs to stay updated on the latest feature releases. Let’s walk through the major updates across Pulumi from the last few months!

AI
Pulumi Cloud
Core
- Import Improvements
Providers and Packages
Wrap up

AI

Deploy with Pulumi within Pulumi AI

Pulumi AI’s new feature enables deploying cloud infrastructure using AI prompts. Users can rapidly generate and deploy templates through natural language inputs in ‘pulumi new’ CLI command and new project wizard, significantly streamlining the cloud setup process. Additionally, you now have the option to deploy the Pulumi AI generated Pulumi programs from within Pulumi Neo and in the New Project Wizard in the Pulumi Cloud console.

Pulumi AI coverage and performance

Pulumi AI can now write Pulumi programs for all 150 cloud providers in the Pulumi registry - up from 20 cloud providers. Pulumi AI has also been updated to OpenAI’s GPT-4 Turbo for enhanced performance along with fine tuning our prompts to improve our AI’s ability to write code.

Pulumi AI Answers

Pulumi AI Answers, launched late last year, has become a go-to repository, hosting an extensive collection of over 50,000 anonymized, infrastructure-as-code commonly asked queries, all carefully selected and presented by Pulumi. As this repository continues to expand, we’re seeing an exponential increase in user traffic, underscoring the immense value it offers. It’s inspiring to see the Pulumi community actively leveraging it as they build and scale their cloud infrastructure

Pulumi Cloud

Pulumi ESC Preview

Pulumi Environment, Secrets and Configuration (ESC) is our answer to the growing needs of our customers to manage secret sprawl and streamline config management. Pulumi ESC allows teams to store and aggregate secrets and configuration from various sources into a composable collection called an environment. You can dynamically generate OIDC credentials from all three major cloud providers (AWS, Azure and GCP), and integrate with other secrets managers like AWS Secrets Manager, Hashicorp Vault, Azure Vault and GCP Secret manager to pull secrets during runtime. Its hierarchical structure simplifies the composition and reuse of configurations, ensuring secure, auditable management and robust access control. With Pulumi ESC, the trend is clear: organizations are choosing its comprehensive approach for managing secrets and configurations, and they are here to stay. Get started with Pulumi ESC

Developer Portal Gallery

We launched a new template gallery in the New Project Wizard, supercharging Platform teams to provision and distribute organization templates for their development teams to get up and running fast. Developers can generate Pulumi programs from these templates, commit and push code to GitHub and trigger deployment within the Pulumi Cloud console. The gallery supports three template types: Organization Templates for custom internal projects, Pulumi Templates authored by Pulumi to cover common architectural patterns and AI-Generated Templates created through Pulumi AI. For more information, check out the full blog post.

Deployments GA

Pulumi Deployments was made generally available with new improvements such as support for GitHub Enterprise. Since launch, Pulumi Deployments has made infrastructure management at scale seamless for our customers, offering out of the box features such as Review Stacks and multiple deployment triggers.

Customer Managed Agents

Customer Managed Agents for Pulumi Deployments enable customers with hard security and compliance requirements to take advantage of the powerful capabilities of Pulumi Deployments. By self-hosting the agents, you can host them anywhere within your infrastructure on any hardware and environment in order to meet compliance requirements to keep cloud provider credentials within your private network.

MFA in Pulumi Cloud

We now support multi-factor authentication to prevent unauthorized access, ensuring robust security within your organization. Enable MFA by navigating to your account settings.

Pricing Calculator

Pulumi cost calculator for Team Edition enables you to quickly estimate the cost of using Pulumi Cloud. Simply input the number of resources in all your cloud accounts and the percentage of time they will be running to quickly get the total cost per month with a break down of estimated credits, free credits and cost per credit. You can also use the calculator to estimate the cost of using Pulumi Deployments.

Historical Views

The Pulumi Cloud stack update page now contains the historical resources in the stack and the historical stack outputs. Simply click on details on each Pulumi update to see the resource state and stack output at the time of the update. This history extends all the way back to the first update of the stack. See the historical views blog post for more details.

Core

Import Improvements

We have made several improvements to the pulumi import command to make it easier to bulk import resources into complex programs and components. You can now:

Import one resource and then use that resource as the parent for another imported resource. Previously, parents could only refer to resources that already existed; i.e., you would have had to do two imports
Specify that a resource is a component to be used as the parent of other imported resources via a new component property in the import file
Specify parents by name without having to include the parent URN in the nameTable
Specify the logicalName of resources
Run pulumi preview --import-file <file> on a program to generate a placeholder import file for every resource that would be created as part of the program. The generated file will contain all the names, types, and URNs already filled in, with blank id fields that need to be filled in. This is useful when you already have the structure of the resources for your program and only need to fill in the IDs of existing resources

Additionally, the import system no longer renames resources to try and make unique names and no longer generates errors just because two resources of different types have the same name. The rules for uniqueness now match what’s valid when writing a Pulumi program. The import system will generate an error if you try to import two resources that would end up with the same URN.

Providers and Packages

AWSX now uses AWS 6.0

We have launched AWSx 2.0, bringing all the benefits AWS 6.0 to Pulumi Crosswalk for AWS (AWSx). The new versions offer several fixes and improvements, including support for the latest Terraform Plugin Framework, 56 new resources and 23 new functions for various AWS services.

Pulumi Google Cloud Classic 7.0

We have updated the Pulumi Google Cloud Classic Provider to reflect the latest developments from Google Cloud, including updates to existing resources and adding support for new resources, functions and input properties. One of the key update is the fix for renaming of the Service Account resource for consistency and ease of use. We also made changes to align with upstream changes to labels. Upgrading to this new version is straightforward. Check out our blog post and migration guide.

AWS S3 Express One Zone Support

We added support for Amazon S3 Express One Zone, a new storage class offering up to 10x faster performance and reduced request costs by 50%. S3 Express One Zone is ideal for data-intensive applications such as AI/ML, HPC, etc. This update is part of our commitment to provide timely and effective support for the latest AWS offerings.

New Provider Resources

We have added 641 functions and 662 resources across our most popular providers, along with many more throughout our ecosystem.

New Community Providers

We welcome new providers to our registry!

Wrap up

That’s a wrap on our latest Pulumi release notes! With each new feature and improvement, from AI advancements to enhanced cloud deployment tools, we’re excited to see how you leverage these updates. Your feedback is crucial in shaping our path forward, feel free to open issues in our Pulumi Cloud requests repository or in the pulumi/pulumi repository for CLI feedback. Stay connected for more updates, and here’s to building a more efficient cloud future together!

Benchmarking Python Performance

Justin Van Patten — Thu, 28 Sep 2023 00:00:00 +0000

This is the second post in a series about performance optimizations we’ve made to the Pulumi CLI and SDKs. In this post, we’ll go deep on a performance improvement we made for Pulumi Python programs. You can read more about Amazing Performance in the first post in the series.

Late last year, we took a hard look at the performance of Python programs when we realized they weren’t performing up to our expectations. We uncovered a major bug limiting Python performance, and we ran a number of rigorous experiments to evaluate just how performant Pulumi Python programs are after the bug had been repaired. The results indicate Pulumi Python programs are significantly faster than they were, and now Pulumi Python has reached performance parity with Pulumi Node.js!

The Bug

When you execute a Pulumi program, Pulumi internally builds a dependency graph between the resources in your program. In every Pulumi program, some resources have all their input arguments available at the time of their construction. In contrast, other resources may depend on Outputs from other resources.

For example, consider a sample program where we create two AWS S3 buckets, where one bucket is used to store logs for the other bucket:

import pulumi
import pulumi_aws as aws

log_bucket = aws.s3.Bucket("logBucket", acl="log-delivery-write")

bucket = aws.s3.Bucket("bucket",
 acl="private",
 loggings=[aws.s3.BucketLoggingArgs(
 target_bucket=log_bucket.id,
 target_prefix="log/",
 )])

Because bucket takes an Output from log_bucket as an input, we can’t create the bucket until after the log_bucket is created. We have to create the log_bucket first to compute its ID, which we can pass to bucket. This idea extends inductively for arbitrary programs – before any resource can be run, we must resolve the Outputs of all of its arguments. To do this, Pulumi builds a dependency graph between all resources in your program. Then, it walks the graph topologically to schedule provisioning operations.

Provisioning operations that are not dependent on each other can be executed in parallel, and Pulumi defaults to unbounded parallelism, but users can ratchet this down if they so desire. Consider this embarrassingly parallel Python program:

import pulumi
import pulumi_aws as aws

# SQS
for i in range(100):
 name = f'pulumi-{str(i).rjust(3, "0")}'
 aws.sqs.Queue(name)

# SNS
for i in range(100):
 name = f'pulumi-{str(i).rjust(3, "0")}'
 aws.sns.Topic(name)

In this program, we can create 200 resources in parallel because none of them take inputs from other resources. This program should be entirely network-bound because Pulumi can issue all 200 API calls in parallel and wait for AWS to provision the instances. We discovered, however, that it did not! Strangely, API calls were issued in an initial batch of 20; as one completed, another would start.

The Fix

The culprit was the Python default future executor, ThreadPoolExecutor. We observed that benchmark was run on a four-core computer, and in Python 3.5 to Python 3.7, the number of max workers is five times the number of cores, or 20 (in Python 3.8, this number was changed to min(32, os.cpu_count() + 4)). We realized we shouldn’t be using the default ThreadExecutor, and instead we should provide a ThreadExecutor with an adjusted number of max_workers based on the configured parallelism value. That way, when users run pulumi up --parallel, which issues an upper bound on parallel resource construction, the ThreadExecutor will respect that bound. We merged a fix that would plumb the value of --parallel through to a custom ThreadExecutor and measured the impact this change had on the performance of our benchmark.

Experimental Setup

We designed and implemented two independent experiments to evaluate this change. The first experiment measures how well the patched Python runtime stacks up against the control group, Pulumi Python without the patch. The second experiment compares Pulumi Python to Pulumi TypeScript using the same benchmark ported to TypeScript. We used the awesome benchmarking tool hyperfine to record wall clock time as our indicator of performance.

The experiments ran overnight on a 2021 MacBook Pro with 32GB RAM, the M1 chip, and 10 cores. Experimental code is available on GitHub, and release tags pin the version of the code used for each experiment. We also made an effort to run the experiments on a quiet machine connected to power. For all experiment groups, --parallel was unset, translating to unbounded parallelism.

Before between samples, we ran pulumi destroy –yes to ensure a fresh environment. Hyperfine measures shell startup time and subtracts the value before final measurements are recorded to more precisely represent the true cost of execution. All groups collected 20 samples each. We also discard stderr and stdout to reduce noise associated with logging to a tty, but we do record the status code of each command so can show they executed successfully.

Python: Pre- and Post-patch

This experiment compares the performance of Pulumi Python before and after the patch was applied. The control group used Pulumi v3.43.1, while the experimental group used Pulumi v3.44.3. The primary difference between these two groups is that a fix was introduced for a Python runtime concurrency bug as part of v3.44.0. Both groups use the same benchmark program, which created 100 AWS SNS and 100 AWS SQS resources in parallel, as described earlier. Only the version of the Pulumi CLI is different between groups.

Control vs. Fix

Group	Mean	Standard Deviation
Control	222.232 s	0.908 s
Experimental	70.189 s	1.497 s

Summary: The Experimental Group ran 3.17 ± 0.07 times faster than the Control Group, accounting for a +300% speedup in performance. Running Welch T-Test indicated statical significance (p = 2.93e-59, α=0.05).

Python vs. TypeScript

After seeing very promising results from the first experiment, we wanted to determine just how promising these results were. We decided to compare Pulumi Python to Pulumi TypeScript to see if this fix had narrowed the gap in performance between the two runtimes. We ported the Python program to TypeScript:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// SQS
[...Array(100).map((_, i) => {
 const name = `pulumi-${i}`;
 new aws.sqs.Queue(name);
})];

// SQS
[...Array(100).map((_, i) => {
 const name = `pulumi-${i}`;
 new aws.sqs.Queue(name);
})];

For this experiment, we fixed the version of the CLI to v3.44.3, which included the patch to the Python runtime. Here are the result.

TypeScript vs. Python

Group	Mean	Standard Deviation
Python	70.975 s	0.909 s
TypeScript	73.741 s	1.574 s

Summary: The Python Group performed the best and ran 1.04 ± 0.03 times faster than the TypeScript Group. This accounts for a 4% difference in performance. A second T-Test indicated statical significance (p = 1.4e-07, α=0.05). Not only did Python close the gap with TypeScript, but it is also now marginally faster than its Node.js competitor.

Conclusion

It’s rare to have a small PR result in such a massive performance increase, but when it happens, we want to shout it from the rooftops. This change, which shipped last year in v3.44.3, does not require Python users to opt-in; their programs are now faster. This patch has closed the gap with the Node.js runtime. Users can now expect highly parallel Pulumi programs to run in a similar amount of time between either language.

Artifacts

You can check out the artifacts of the experiments on GitHub, including the source code.

Here are some useful links:

The GitHub repository
Artifacts from the first experiment, “Control vs. Fix” or “Pre- and Post-patch”.
More statistics about the first experiment.
Artifacts from the second experiment
More statistics about the second experiment.
Pulumi Internals

Using Go Generics with Pulumi

Zaid Ajaj — Wed, 20 Sep 2023 00:00:00 +0000

Pulumi loves Go, it’s what powers Pulumi. We’ve kept a close eye on the design and development of support for generics in the Go programming language over the years, a feature that allows developers to write type-safe, concise, and reusable code. We’ve been exploring what it’d look like to improve Pulumi’s Go SDKs with generics and recently published a public RFC detailing our plans. We’ve been making progress on the implementation and are excited to announce preview support for Go generics in our core and AWS Go SDKs. If you’re using Go with Pulumi, we’d love for you to give it a try and share your feedback!

// Given
var a pulumi.IntOutput
var b pulumi.StringOutput

// Before (could panic at runtime if you got something wrong)
o := pulumi.All(a, b).ApplyT(func(vs []interface{}) string { // could panic
 a := vs[0].(int) // could panic
 b := vs[1].(string) // could panic
 return strconv.Itoa(a) + b
}).(pulumi.StringOutput) // could panic

// After (compile-time type-safety)
o := pulumix.Apply2(a, b, func(a int, b string) string {
 return strconv.Itoa(a) + b
})

Why Generics?

Before looking at what’s available in the preview, let’s first review the motivations for adding support for generics, from the RFC.

Unsafe Inputs, Outputs, and Apply

A common complaint about the Pulumi Go SDK is the lack of type safety. The Input and Output types, and the apply operation rely heavily on interface{}. This makes them effectively untyped at compile-time, instead relying on type matching at runtime.

For instance, the apply operation is implemented in Go as the following method:

func (*OutputState) ApplyT(applier interface{}) Output

The contract for ApplyT states that applier must be a function with one of the following signatures:

func(U) T
func(U) (T, error)

Where U must be assignable from the type contained in the underlying Output. If it isn’t, the operation will panic.

The caller will usually want to cast the result into a specific output type. If they choose the wrong output type, this operation will panic.

So given a typical usage like the following, there are two spots that can panic: the ApplyT invocation and the IntOutput cast.

intOutput := stringOutput.ApplyT(func(v string) int {
 return len(v)
}).(pulumi.IntOutput)

Excessive code generation

The Go SDK relies heavily on code generation in an attempt to alleviate the pain of untyped operations. It generates type-specific input and output wrapper types for nearly every type it encounters.

In the core Pulumi SDK, this includes all primitive types, their pointers, containers, and containers of containers. For instance, the core SDK has 22 String* types besides type String itself, covering nearly all variations of the following pattern:

String(Ptr|(Array|Map)?(Array|Map))?(Input|Output)?

This pattern of code generation carries over to code generated for provider SDKs, though not to this degree of nesting. Every type gets an input and output type, a pointer variant if the type is referenced as a pointer, and container variants if the type is referenced from a container.

In general, this generates extremely large packages that are ungainly to navigate—both in source form and in their API reference.

Introducing Go Generics in the Pulumi Go SDK

We can address the above by leveraging Go generics to incorporate type-safety into the core concepts (inputs, outputs, and apply) and reduce the amount of generated code.

Starting with v3.80 of the Pulumi Go SDK, we have added a new pulumix subpackage (containing “extensions”), which includes new generic Input[T] and Output[T] types as well as type-safe generics-based APIs to work with them such as Apply.

With the new support for generics, the previous untyped example can now be written in a type-safe manner as follows:

intOutput := pulumix.Apply(stringOutput, func(v string) int {
 return len(v)
})

This is checked at compile-time: if you mistakenly specified an incorrect type, it will be caught at compile-time rather than panicking at runtime.

We recommend using Go 1.21 or later, which has improved type inference for generics, allowing you to write the call to Apply above without having to explicitly specify generic type parameters.

Adopting Generics in Your Pulumi Programs

Here’s a larger example that uses the non-generic pulumi.All function and ApplyT method to produce a JSON string representing a policy for a BucketPolicy:

package main

import (
 "encoding/json"

 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/cloudfront"
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 ErrorDocument: pulumi.String("404.html"),
 },
 })
 if err != nil {
 return err
 }

 originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
 Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
 })
 if err != nil {
 return err
 }

 _, err = s3.NewBucketPolicy(ctx, "cloudfront-bucket-policy", &s3.BucketPolicyArgs{
 Bucket: bucket.ID(),
 Policy: pulumi.All(bucket.Arn, originAccessIdentity.IamArn).ApplyT(
 func(args []any) (string, error) {
 bucketArn := args[0].(string)
 iamArn := args[1].(string)

 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }).(pulumi.StringOutput),
 }, pulumi.Parent(bucket))
 if err != nil {
 return err
 }

 return nil
 })
}

pulumi.All is used to combine bucket.Arn and originAccessIdentity.IamArn into a single Output and then uses ApplyT construct a policy in JSON using the values.

If we wanted to take advantage of the type-safety provided by the new pulumix subpackage, we could rewrite the above example to use pulumix.Apply. First, import "github.com/pulumi/pulumi/sdk/v3/go/pulumix". Then, replace the use of pulumi.All and ApplyT with the generic pulumix.Apply:

 Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }),

Here, we’re specifically using pulumix.Apply2Err, a variant of pulumix.Apply that accepts two generic arguments and a callback function that supports returning an error. There are other variants of pulumix.Apply that accept up to 8 generic arguments, and variants that return an error and accept a context.Context.

The result of our call to pulumix.Apply2Err in this example is pulumix.Output[string]. We can pass this directly to the Policy argument because it happens to be weakly typed as pulumi.Input, which pulumix.Output[T] implements.

If the Policy argument had been typed as pulumi.StringInput, for example, we would have had to cast the pulumix.Output[string] to pulumi.StringOutput using the pulumix.Cast function:

policy := pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 // ...
 })

// ...

Policy: pulumix.Cast[pulumi.StringOutput](policy),

Here’s the updated example in full:

package main

import (
 "encoding/json"

 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/cloudfront"
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 ErrorDocument: pulumi.String("404.html"),
 },
 })
 if err != nil {
 return err
 }

 originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
 Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
 })
 if err != nil {
 return err
 }

 _, err = s3.NewBucketPolicy(ctx, "bucket-policy", &s3.BucketPolicyArgs{
 Bucket: bucket.ID(),
 Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }),
 }, pulumi.Parent(bucket))
 if err != nil {
 return err
 }

 return nil
 })
}

Notice how we are importing pulumix alongside pulumi. Being a subpackage means you can start using the generics-based types and functions alongside the current APIs as necessary without having to switch your code fully from using the old types to the new generic ones. Instead, you can migrate your code gradually to these new APIs where it makes sense. When needed, there are APIs to convert between generics-based APIs and the old ones using functions such as pulumix.Cast and pulumix.ConvertTyped.

Generics-Based Cloud Provider SDKs

We’ve seen how the pulumix subpackage on its own can be used with existing Pulumi provider SDKs. We can go further and adopt generics in the provider SDKs themselves. To make the transition non-breaking and gradual for your Pulumi programs, we intend to publish generic-based variants of the provider Go SDKs under a subpackage called x (for “extensions”). This allows you to migrate your code resource by resource.

Migrating an AWS Program to leverage Go generics

The following example is an AWS Pulumi program which we’ll migrate into the generics-based API. Here is what pulumi new aws-go gives you today:

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an AWS resource (S3 Bucket)
 bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 },
 })
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("bucketName", bucket.ID())
 return nil
 })
}

To start using the generics-based version of the S3 module from AWS, you need to change the import from "https://github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3" to "https://github.com/pulumi/pulumi-aws/sdk/v7/go/aws/x/s3" (notice the x after aws/), then change input values to use pulumix.Ptr instead of pulumi.String.

Rather than passing values as pulumi.String("foo") and pulumi.Int(42), with generics, you can simply use pulumix.Val("foo") and pulumix.Val(42). Similarly, you can use the pulumix.Ptr function for pointer values.

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/x/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an AWS resource (S3 Bucket)
 bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumix.Ptr("index.html"),
 },
 })
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("bucketName", bucket.ID())
 return nil
 })
}

You’ll also need to update your go.mod to use the preview branch of the Pulumi AWS Go SDK that includes the x subpackage with generic APIs. Run the following:

go get github.com/pulumi/pulumi-aws/sdk/v7@generics
go mod tidy

The changes look minimal, except the fact that the inputs of s3.BucketArgs and the outputs of Bucket are now fully generics-based. Here is the Bucket type you get from NewBucket:

type Bucket struct {
 pulumi.CustomResourceState
 AccelerationStatus pulumix.Output[string] `pulumi:"accelerationStatus"`
 Acl pulumix.Output[*string] `pulumi:"acl"`
 Arn pulumix.Output[string] `pulumi:"arn"`
 Bucket pulumix.Output[string] `pulumi:"bucket"`
 // ....
}

And here are its input arguments:

type BucketArgs struct {
 AccelerationStatus pulumix.Input[*string]
 Acl pulumix.Input[*string]

 Arn pulumix.Input[*string]
 Bucket pulumix.Input[*string]
 // ...
}

Authoring Generics-based Go SDKs

For Pulumi package authors, it is possible to start generating Go SDKs for your providers that include the generics-based variant under the x subpackage using v3.84.0 of Pulumi. By default, this is not enabled so you will have to explicitly opt-in to it using a specific option in the go language section of your Pulumi schema. The new option is called "generics" and it has three possible values:

"none" is the default value when the option is not set. This doesn’t generate the generics-based variant
"side-by-side" is the option we are using for our select providers which generates the variant under the x subdirectory.
"generics-only" if you want your provider to only emit generics-based APIs. Since this is a preview feature, things might still be rough around the edges, so use this option with caution as fixes in the SDK-generation code might result in breaking changes in your APIs.

Here is how this looks in JSON:

{
 "language": {
 "go": {
 "generics": "side-by-side"
 }
 }
}

We eventually intend to make "side-by-side" be the default, such that all providers get a generics-based variant under the x subpackage, so that all users can use the full generics API.

Try it out!

We are excited for you to try out these new APIs and would love to hear your feedback! You can learn more details on the design and add comments to the RFC, or chat with us in the #golang channel on the Pulumi Community Slack.

If you’d like to learn more about building cloud applications with Go and Pulumi, we have a workshop coming up in October. Sign up here.

We’ll also be at GopherCon, if you’d like to meet with us.

Converting Full Terraform Programs to Pulumi

Justin Van Patten — Mon, 12 Jun 2023 00:00:00 +0000

Over the last 2 years, we’ve seen an increasing trend of cloud development teams migrating to Pulumi from Terraform. These teams often have experience with and meaningful investment in Terraform, but have also typically run into limits of expressivity, productivity, scalability, or reliability with their existing tools. One of the first questions we hear when they decide to move to Pulumi is “how will I migrate my existing Terraform projects over?”.

Today, we’re excited to announce new support for converting whole Terraform projects to Pulumi via the pulumi convert command in the Pulumi CLI. The new Terraform converter includes support for Terraform modules, core features of Terraform 1.4, and the majority of Terraform built-in functions, converting to Pulumi TypeScript, Python, Go, or C#. The new converter can significantly reduce the amount of time it takes to migrate Terraform to Pulumi. Let’s dig in to learn more about the new converter and how to use it.

Historically, we have offered a separate tf2pulumi tool to convert small snippets of Terraform to Pulumi (now replaced by the Terraform converter plugin). The new converter is no longer a separate tool. As of v3.71.0, you can run the new converter directly from the Pulumi CLI with the pulumi convert --from terraform command. And you can convert more than small snippets – the new converter supports converting full Terraform programs.

The new support in pulumi convert builds upon Pulumi’s CrossCode foundations for providing universal infrastructure as code support across a wide variety of programming languages and conversion tooling between them. It also introduces a new concept of converter plugin in the Pulumi engine, which allows conversion tools from other Infrastructure as Code platforms to be integrated into the same pulumi convert experience in the future, both as part of the core project, as well as by other ecosystem partners and contributors.

Several common use cases are supported via the new pulumi convert --from terraform support in the Pulumi CLI:

Converting your organization’s existing Terraform projects to Pulumi
Converting your organization’s existing Terraform modules to Pulumi, to be consumed as part of existing Pulumi projects
Converting 3rd party open source Terraform modules or projects which address a use case you want to incorporate into your existing Pulumi projects

Supported Terraform Features

The following major features are supported:

Variables, outputs, resources, and data sources
Terraform modules are converted to Pulumi components
Almost all HCL2 expression syntax

In cases where the converter does not yet support a feature, the pulumi convert command succeeds but generates a TODO in the form of a call to a notImplemented not_implemented notImplemented NotImplemented function that will need to be filled in manually. For most projects, the converter should be able to convert 90-95% of the code without any TODOs, with only a small percentage of items to address manually, significantly reducing migration time compared to doing an entire migration by hand. We are actively improving the converter by adding support for missing features and improving the overall quality of the converted code to reduce the amount of manual fix-ups required.

Converting a Real World Program

Let’s walk through converting a Terraform codebase to Pulumi. Avant Terraform Vault Setup is an open source project that provides a high-availability installation of Vault using a variety of managed AWS services. It defines a fairly complex installation with dozens of AWS resources in over 1,000 lines of Terraform HCL, including the main program and a Vault module. Let’s convert it to Pulumi.

First, clone the repo and cd into the directory containing the Terraform project:

$ git clone https://github.com/avantoss/vault-infra.git
$ cd vault-infra/terraform/main

Next, run the converter:

$ pulumi convert --from terraform --language typescript --out pulumi

$ pulumi convert --from terraform --language python --out pulumi

$ pulumi convert --from terraform --language go --out pulumi

$ pulumi convert --from terraform --language csharp --out pulumi

The converted code is generated in the specified pulumi output directory. A complete Pulumi project is generated, including two primary code files specific to this this program’s conversion:

index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf contains the converted code for the main program
vault.ts vault.py vault.go vault.cs contains the Vault Pulumi component, converted from the Terraform module

Addressing TODOs

The file for the Vault component makes up the bulk of the implementation and contains some TODOs emitted by the converter that must be addressed manually.

For example, the converter doesn’t yet support the replace, element, and split built-in functions.

const plainDomain = notImplemented("replace(element(split(\":\",var.vault_dns_address),1),\"////\",\"\")");

plain_domain = not_implemented("replace(element(split(\":\",var.vault_dns_address),1),\"////\",\"\")")

plainDomain := notImplemented("replace(element(split(\":\",var.vault_dns_address),1),\"////\",\"\")");

var plainDomain = NotImplemented("replace(element(split(\":\",var.vault_dns_address),1),\"////\",\"\")");

We can fill in an implementation. Note that we get to use the full expressiveness of the native and familiar string manipulation libraries in our target programming language, instead of the relatively constrained options of the Terraform DSL.

const plainDomain = pulumi.output(args.vaultDnsAddress)
 .apply(a => a.split(":")[1].replace("//", ""));

plain_domain = pulumi.Output.from_input(args["vaultDnsAddress"]) \
 .apply(lambda a: a.split(":")[1].replace("//", ""))

plainDomain := args.VaultDnsAddress.ToStringOutput().ApplyT(func(a string) string {
 return strings.ReplaceAll(strings.Split(a, ":")[1], "//", "")
}).(pulumi.StringOutput)

var plainDomain = args.VaultDnsAddress
 .Apply(a => a.Split(':')[1].Replace("//", ""));

Wrapping Up

After addressing the remaining TODOs and some other tweaks so that the code compiles, we can now run the converted program with pulumi up to provision the Vault installation with Pulumi.

The converter has saved us a ton of time, converting over 1,000 lines of Terraform to a modern Pulumi language, with only a small number of manual fix-ups required. From here, we can leverage our IDE and compiler to further refactor and improve the code, one of the many benefits of Pulumi!

Convert your Terraform to Pulumi

Run pulumi convert to migrate your existing Terraform projects and modules to TypeScript, Python, Go, or C#, then refine the result with your language’s own tooling.

Learn more

Importing State

It’s great that the new converter can migrate Terraform projects to Pulumi for new deployments, but what if you want to import existing resource states from a .tfstate file to avoid unnecessarily recreating your infrastructure?

If you’re using TypeScript or Go, there is some additional code that can be added to your converted Pulumi program to import resource states from a .tfstate file. See the Importing Resources reference documentation for more details.

We’re working to make this even more seamless with built-in support for importing state from .tfstate files in a future update coming soon.

Get Started

Support for the new pulumi convert --from terraform command is now available in v3.71.0 of the Pulumi CLI. Download the latest Pulumi CLI and give the new converter a try today. If you run into any issues, please let us know or reach out in the Pulumi Community Slack with any questions!

Aligning Projects between Service and DIY Backend

Justin Van Patten — Mon, 03 Apr 2023 00:00:00 +0000

At Pulumi, our goal is to offer the best Infrastructure as Code experience for all cloud developers. From the very beginning, we’ve believed that the best IaC experience is made possible by combining a great open source SDK and CLI with a great backend management service. This is why we built Pulumi Cloud, a rich management platform for your Infrastructure as Code, which includes a forever free option for individuals, a generous free tier for teams, and critical tools for enterprises to manage IaC at scale.

Over the last few years, we’ve continued to expand the features of the Pulumi Service - with Deployments, Audit Logs, SAML SSO and SCIM, Teams, Stack Transfers, Favorites, Organization and Team Access Tokens and much more.

While the majority of Pulumi users do choose to use the Pulumi Service, we also know that there are good reasons why some organizations would prefer to use Pulumi IaC alone without the Pulumi Service. And so we support and continue to invest in enabling a variety of additional backends that allow the Pulumi CLI to be used with state stored in the local filesystem or in cloud storage like S3, Azure Blob Storage, or Google Cloud Storage.

Historically the Pulumi Service backend and the DIY backend have differed in their handling of “projects”. The Pulumi Service stores state for a Pulumi stack in a separate namespace per project. The DIY backends have historically stored all stacks in a single namespace across all projects. This inconsistency has been a common source of confusion for users getting started with Pulumi when using the file storage backends.

Today, we are aligning how projects are managed across all backends, adding Project-Scoped Stacks support to DIY backends.

Background

This section goes over some of the background necessary to understand project-scoping and why it’s important. Feel free to skip ahead to the next section if you’re already familiar with this.

Projects and stacks

Pulumi projects are a unit of organization. A project holds the code and configuration for your infrastructure, and can be used to deploy multiple stacks. A stack is a single, isolated instance of your infrastructure.

For example, you might have an “app” project for your application, deploying to the “test” and “prod” stacks.

What’s a state backend?

Each stack tracks metadata about your infrastructure so it can manage your cloud resources. This metadata is called state and it is stored in a backend of your choosing: Service or DIY.

The Service backend is hosted at app.pulumi.com or on your own server. It provides the best combination of usability, safety, and security for most users.

The DIY backend stores state in a JSON file, deferring to you to manage persistence of this information. You can store it in Amazon S3, Azure Blob Store, Google Cloud Storage, a system such as Minio or Ceph with an API compatible with Amazon S3, or on your local filesystem. Using this backend trades reliability and convenience of the Service backend for additional control over where the information is stored.

If you’re unsure of which backend to use, see Deciding On a State Backend.

Project-scoped stacks

Depending on your needs, you can have several Pulumi projects. For example, you might have a “net” project to set up an Amazon VPC, a “k8s” project to set up a Kubernetes cluster deployed to it, and an “app” project to build and deploy your application to the cluster. Each of these projects will have their own stacks, e.g. “test” and “prod”.

The Pulumi Service backend scopes stacks to the project they belong to, so we can re-use the names “test” and “prod” across different projects.

This was not the case for the DIY backend, where stack names were global across all projects. To work around this, users resorted to qualifying stack names in the form <project>-<stack> when using the DIY backend.

We’re pleased to announce that such workarounds are no longer necessary!

Project-scoped stacks in DIY backends

With the latest release of Pulumi, we’ve added support for project-scoped stacks to the DIY backend. Stacks created in new or empty DIY backends will automatically use the new project-scoped layout.

With this change, you can now use stack names like “test” and “prod” across your projects without risk of conflict—the same way you do with the Service backend.

Upgrading existing backends

Only stacks created in new or empty DIY backends will scope stacks to projects. Existing backends will continue to operate as they do today.

If you have an existing DIY backend that you’d like to upgrade to use project-scoped stacks, you can do so with the new pulumi state upgrade command, which upgrades all stacks in the backend to use the new project-scoped layout.

$ pulumi state upgrade
This will upgrade the current backend to the latest supported version.
Older versions of Pulumi will not be able to read the new format.
Are you sure you want to proceed?
Please confirm that this is what you'd like to do by typing `yes`:

Once stacks in a DIY backend are upgraded, older versions of the Pulumi CLI will be unable to access those stacks. This change cannot be reverted.

Cleaning up old stack names

If you previously qualified stack names with the project name to avoid conflicts, you can use the pulumi stack rename command to clean up these names after the upgrade.

$ pulumi stack rename --stack app-test test

Manage stacks across your projects

Organize your infrastructure into projects and reuse stack names like test and prod across them. Store state in Pulumi Cloud or a self-managed backend, whichever fits your team.

Get started

Referencing stacks in the CLI

Many commands in the Pulumi CLI expect a stack name to be specified. These commands accept a fully-qualified stack name in the form:

<org>/<project>/<stack>

Where, <org> is the organization name, <project> the project name, and <stack> the stack name.

They also accept one of the following shorthand forms if we’re inside the project directory:

<org>/<stack>
<stack>

For instance, given a stack “prod” inside the “app” project, the following are all valid ways to refer to it:

my-org/app/prod
my-org/prod // if inside the app directory
prod // if inside the app directory

With the latest release of Pulumi, the above is true for both the Service backend and the DIY backend.

Referencing DIY project-scoped stacks

All projects in a DIY backend are placed under a virtual organization named “organization”. This value is constant and cannot be changed.

Repeating the above examples with a DIY backend, we can use any of the following forms to refer to the same stack:

organization/app/prod
organization/prod // if inside the app directory
prod // if inside the app directory

Availability

Support for project-scoped stacks in DIY backends is now available in Pulumi v3.61.0. Give it a try and share your thoughts with us on the Pulumi Community Slack!

Introducing Resource Methods for Pulumi Packages

Justin Van Patten — Mon, 11 Oct 2021 08:00:00 -0700

It’s now possible to provide resource methods from Pulumi Packages. Resource methods are similar to functions, but instead of being exposed as top-level functions in a module, methods are exposed as methods on a resource class. This allows for a more object-oriented approach to exposing functionality—operations performed by a resource (that potentially use the resource’s state) can now be exposed as methods on the resource. Resource methods can be implemented once, in your language of choice, and made available to users in all Pulumi languages.

When authoring component resources, it’s often useful to provide additional functionality through methods on the component. For example, the Cluster component in the eks package has a getKubeconfig method that can be used to generate a kubeconfig for authentication with the cluster that does not use the default AWS credential provider chain, but is instead scoped based on the passed-in arguments. Until now, that method has only been available from JavaScript/TypeScript (the language the Cluster component was written in). With the new support for resource methods for Pulumi Packages, we can make this method available to all the Pulumi languages, which is exactly what we’ve done in pulumi-eks v0.34.0.

const cluster = new eks.Cluster("mycluster");
const kubeconfig = cluster.getKubeconfig(...);

cluster = eks.Cluster("mycluster")
kubeconfig = cluster.get_kubeconfig(...)

var cluster = new Cluster("mycluster");
var kubeconfig = cluster.GetKubeconfig(...);

cluster, err := eks.NewCluster(ctx, "mycluster", nil)
if err != nil {
 return err
}
kubeconfig, err := cluster.GetKubeconfig(ctx, &eks.ClusterGetKubeconfigArgs{...})
if err != nil {
 return err
}

This post will show how you can provide resource methods from your component packages.

Authoring Methods

It’s always been possible to provide module-level functions from Pulumi Packages. For example, the aws package provides many module-level functions, such as the getAmi function, which can be used to get the ID of an existing Amazon Machine Image (AMI). Functions are declared in the package’s schema, and their functionality is implemented in the provider through the Invoke remote procedure call (RPC).

Methods are authored in a similar manner to functions. Methods are declared in the schema and implemented in the provider’s Call RPC (similar to Invoke).

Let’s walk through an example. We’ll author a Message component that accepts a message as input. The component then provides a getMessage method that accepts a recipient name and returns the message customized for the recipient. To get started authoring a component package, refer to the package documentation.

Schema

We’ll start with declaring the method and component in the Pulumi schema. First, define the function representing the method:

 "functions": {
 "example:index:Message/getMessage": {
 "inputs": {
 "properties": {
 "__self__": {
 "$ref": "#/resources/example:index:Message"
 },
 "recipient": {
 "type": "string"
 }
 },
 "required": ["__self__", "recipient"]
 },
 "outputs": {
 "properties": {
 "result": {
 "type": "string"
 }
 },
 "required": ["result"]
 }
 }
 },

Our method has two required arguments: __self__ and recipient. __self__ is a special input required by all methods that represents the component resource and is typed as such. The method has one result named result.

Next, define our Message component:

 "resources": {
 "example:index:Message": {
 "isComponent": true,
 "inputProperties": {
 "message": {
 "type": "string"
 },
 },
 "requiredInputs": ["message"],
 "properties": {
 "message": {
 "type": "string"
 },
 },
 "required": ["message"],
 "methods": {
 "getMessage": "example:index:Message/getMessage"
 }
 }
 },

Our component has a single required input/output property: message. The method is specified in the methods property, which references the method’s function definition: "getMessage": "example:index:Message/getMessage".

Here’s our schema all together:

{
 "version": "0.0.1",
 "name": "example",
 "functions": {
 "example:index:Message/getMessage": {
 "inputs": {
 "properties": {
 "__self__": {
 "$ref": "#/resources/example:index:Message"
 },
 "recipient": {
 "type": "string"
 }
 },
 "required": ["__self__", "recipient"]
 },
 "outputs": {
 "properties": {
 "result": {
 "type": "string"
 }
 },
 "required": ["result"]
 }
 }
 },
 "resources": {
 "example:index:Message": {
 "isComponent": true,
 "inputProperties": {
 "message": {
 "type": "string"
 },
 },
 "requiredInputs": ["message"],
 "properties": {
 "message": {
 "type": "string"
 },
 },
 "required": ["message"],
 "methods": {
 "getMessage": "example:index:Message/getMessage"
 }
 }
 },
 "language": {
 "csharp": {
 "packageReferences": {
 "Pulumi": "3.12"
 },
 "liftSingleValueMethodReturns": true
 },
 "go": {
 "liftSingleValueMethodReturns": true
 },
 "nodejs": {
 "devDependencies": {
 "@types/node": "latest"
 },
 "liftSingleValueMethodReturns": true
 },
 "python": {
 "liftSingleValueMethodReturns": true
 }
 }
}

To make it easier for users of our method, we also set "liftSingleValueMethodReturns": true for each language, which makes single-value methods return the single value directly, rather than wrapping the results in a Result type.

Component Implementation

We can implement the component and make it available to any Pulumi language. We’re going to show some implementation examples in TypeScript/JavaScript, Python, and Go.

Here’s the implementation of the Message component:

import * as pulumi from "@pulumi/pulumi";

interface MessageArgs {
 message: pulumi.Input<string>;
}

class Message extends pulumi.ComponentResource {
 public readonly message!: pulumi.Output<string>;

 constructor(name: string, args: MessageArgs, opts?: pulumi.ComponentResourceOptions) {
 const props = { message: args?.message }
 super("example:index:Message", name, props, opts);

 if (opts?.urn) {
 // Skip further initialization when being constructed from a resource reference.
 return;
 }

 this.registerOutputs(props);
 }

 getMessage(recipient: pulumi.Input<string>): pulumi.Output<string> {
 return pulumi.iterpolate `${recipient}, ${this.message}!`;
 }
}

from typing import Optional

import pulumi

class Message(pulumi.ComponentResource):
 @property
 @pulumi.getter
 def message(self) -> pulumi.Output[str]:
 return pulumi.get(self, "message")

 def __init__(self,
 resource_name: str,
 opts: Optional[pulumi.ResourceOptions] = None,
 message: Optional[pulumi.Input[str]] = None) -> None:

 args = {"message": message}
 super().__init__("example:index:Message", resource_name, args, opts)

 if opts and opts.urn:
 # Skip further initialization when being constructed from a resource reference.
 return

 self.register_outputs(args)

 def get_message(self, recipient: pulumi.Input[str]) -> pulumi.Output[str]:
 return pulumi.Output.concat(recipient, ", ", self.message, "!")

Authoring multi-language components in .NET is not yet supported.

import (
 "errors"

 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

type Message struct {
 pulumi.ResourceState
 Message pulumi.StringOutput `pulumi:"message"`
}

type MessageArgs struct {
 Message pulumi.StringInput `pulumi:"message"`
}

func NewMessage(ctx *pulumi.Context, name string, args *MessageArgs, opts ...pulumi.ResourceOption) (*Message, error) {
 if args == nil {
 return nil, errors.New("args is required")
 }

 message := &Message{}
 err := ctx.RegisterComponentResource("example:index:Message", name, message, opts...)
 if err != nil {
 return nil, err
 }

 message.Message = args.Message.ToStringOutput()
 if err := ctx.RegisterResourceOutputs(message, pulumi.Map{
 "message": args.Message,
 }); err != nil {
 return nil, err
 }

 return message, nil
}

type GetMessageArgs struct {
 Recipient pulumi.StringInput `pulumi:"recipient"`
}

func (c *Message) GetMessage(args *GetMessageArgs) StringOutput {
 return pulumi.Sprintf("%s, %s!", args.Recipient, c.Message)
}

Provider RPCs

Next, wire up the provider RPCs with the component implementation:

import * as pulumi from "@pulumi/pulumi";
import * as provider from "@pulumi/pulumi/provider";

class Provider implements provider.Provider {
 public readonly version = "0.0.1";

 constructor() {
 // Register any resources that can come back as resource references that need to be rehydrated.
 pulumi.runtime.registerResourceModule("example", "index", {
 version: this.version,
 construct: (name, type, urn) => {
 switch (type) {
 case "example:index:Message":
 return new Component(name, <any>undefined, { urn });
 default:
 throw new Error(`unknown resource type ${type}`);
 }
 },
 });
 }

 async construct(name: string, type: string, inputs: pulumi.Inputs,
 options: pulumi.ComponentResourceOptions): Promise<provider.ConstructResult> {
 if (type != "example:index:Message") {
 throw new Error(`unknown resource type ${type}`);
 }

 const message = new Message(name, <MessageArgs>inputs, options);
 return {
 urn: message.urn,
 state: inputs,
 };
 }

 async call(token: string, inputs: pulumi.Inputs): Promise<provider.InvokeResult> {
 switch (token) {
 case "example:index:Message/getMessage":
 const self: Component = inputs.__self__;
 return {
 outputs: {
 result: self.getMessage(inputs.recipient),
 },
 };

 default:
 throw new Error(`unknown method ${token}`);
 }
 }
}

export function main(args: string[]) {
 return provider.main(new Provider(), args);
}

main(process.argv.slice(2));

from typing import Optional
import sys

import pulumi
import pulumi.provider as provider


class Provider(provider.Provider):
 VERSION = "0.0.1"

 class Module(pulumi.runtime.ResourceModule):
 def version(self):
 return Provider.VERSION

 def construct(self, name: str, typ: str, urn: str) -> pulumi.Resource:
 if typ == "example:index:Message":
 return Component(name, pulumi.ResourceOptions(urn=urn))
 else:
 raise Exception(f"unknown resource type {typ}")

 def __init__(self):
 super().__init__(Provider.VERSION)
 pulumi.runtime.register_resource_module("example", "index", Provider.Module())

 def construct(self, name: str, resource_type: str, inputs: pulumi.Inputs,
 options: Optional[pulumi.ResourceOptions] = None) -> provider.ConstructResult:

 if resource_type != "example:index:Message":
 raise Exception(f"unknown resource type {resource_type}")

 message = Message(name, opts=options, message=inputs["message"])

 return provider.ConstructResult(
 urn=component.urn,
 state=inputs)

 def call(self, token: str, args: pulumi.Inputs) -> provider.CallResult:
 if token != "example:index:Message/getMessage":
 raise Exception(f'unknown method {token}')

 message: Message = args["__self__"]
 outputs = {
 "result": message.get_message(args["recipient"])
 }
 return provider.CallResult(outputs=outputs)


if __name__ == "__main__":
 provider.main(Provider(), sys.argv[1:])

Authoring multi-language components in .NET is not yet supported.

import (
 "errors"
 "fmt"

 "github.com/blang/semver"

 "github.com/pulumi/pulumi/pkg/v3/resource/provider"
 "github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 pulumiprovider "github.com/pulumi/pulumi/sdk/v3/go/pulumi/provider"
)

const providerName = "example"
const version = "0.0.1"

type module struct {
 version semver.Version
}

func (m *module) Version() semver.Version {
 return m.version
}

func (m *module) Construct(ctx *pulumi.Context, name, typ, urn string) (r pulumi.Resource, err error) {
 if typ != "example:index:Message" {
 return nil, fmt.Errorf("unknown resource type: %s", typ)
 }
 r = &Component{}
 err = ctx.RegisterResource(typ, name, nil, r, pulumi.URN_(urn))
 return
}

type GetMessageResult struct {
 Result pulumi.StringOutput `pulumi:"result"`
}

func main() {
 pulumi.RegisterResourceModule("example", "index", &module{semver.MustParse(version)})

 if err := provider.MainWithOptions(provider.Options{
 Name: providerName,
 Version: version,
 Construct: func(ctx *pulumi.Context, typ, name string, inputs pulumiprovider.ConstructInputs,
 options pulumi.ResourceOption) (*pulumiprovider.ConstructResult, error) {

 if typ != "example:index:Message" {
 return nil, fmt.Errorf("unknown resource type %s", typ)
 }

 args := &MessageArgs{}
 if err := inputs.CopyTo(args); err != nil {
 return nil, fmt.Errorf("setting args: %w", err)
 }

 message, err := NewMessage(ctx, name, args, options)
 if err != nil {
 return nil, fmt.Errorf("creating component: %w", err)
 }

 return pulumiprovider.NewConstructResult(message)
 },
 Call: func(ctx *pulumi.Context, tok string, args pulumiprovider.CallArgs) (*pulumiprovider.CallResult, error) {
 if tok != "example:index:Message/getMessage" {
 return nil, fmt.Errorf("unknown method %s", tok)
 }

 methodArgs := &GetMessageArgs{}
 res, err := args.CopyTo(methodArgs)
 if err != nil {
 return nil, fmt.Errorf("setting args: %w", err)
 }
 message := res.(*Message)

 result, message.GetMessage(methodArgs)
 return pulumiprovider.NewCallResult(&GetMessageResult{
 Result: result
 })
 },
 }); err != nil {
 cmdutil.ExitError(err.Error())
 }
}

The Construct RPC is called when creating an instance of the component. The Call RPC is called when a method is called.

Using the Component

Now we can build and try out using the component and its method from any Pulumi language:

const component = new example.Message("mycomponent", {
 message: "hello world",
});
export const message = component.getMessage({ recipient: "Alice" }); // Exports "Alice, hello world!"

component = example.Message("mycomponent", message="hello world")
message = component.get_message(recipient="Alice")
pulumi.export("message", message) # Exports "Alice, hello world!"

var component = new Message("mycomponent", new MessageArgs
{
 Message = "hello world",
});
// Exports "Alice, hello world!"
this.Message = component.GetMessage(new MessageGetMessageArgs
{
 Recipient = "Alice",
});

component, err := example.NewMessage(ctx, "mycomponent", &example.MessageArgs{
 Message: "hello world",
})
if err != nil {
 return err
}
message, err := component.GetMessage(ctx, &example.MessageGetMessageArgs{Recipient: "Alice"})
if err != nil {
 return err
}
pulumi.Export("message", message) // Exports "Alice, hello world!"

Wrapping Up

With support for resource methods for Pulumi Packages, you can now create component resources with methods and make them available to use from any Pulumi language. We look forward to seeing the components you create!

👉 Author your first Pulumi Package

Announcing Python Tooling Improvements

Justin Van Patten — Thu, 27 Aug 2020 00:00:00 +0000

Today we’re excited to announce some fairly significant improvements to the experience of writing Pulumi programs in Python. We’ve added type annotations to APIs and now allow passing nested data as strongly typed classes instead of raw dicts. This provides a much better editing experience in IDEs, improved type checking, and overall consistency.

Type Annotations

PEP 484 type annotations have been added to all Python APIs, allowing IDEs like PyCharm and VS Code and tools like mypy to catch errors earlier.

Consider the following example for retrieving an AMI, with an incorrect most_recent argument:

from pulumi_aws import get_ami

ami = get_ami(most_recent="true",
 owners=["137112412989"],
 filters=[{"name": "name", "values": ["amzn-ami-hvm-*"]}])

PyCharm shows the error:

As does VS Code with mypy:

And running mypy manually also reports the issue:

$ mypy __main__.py
__main__.py:3: error: Argument "most_recent" to "get_ami" has incompatible type "str"; expected "Optional[bool]"
Found 1 error in 1 file (checked 1 source file)

Data Classes

Consider an AWS bucket hosting a static website. Previously, the website and cors_rules inputs needed to be specified as raw dicts:

from pulumi_aws import s3

my_bucket = s3.Bucket("my-bucket",
 acl="public-read",
 website={
 "indexDocument": "index.html",
 "errorDocument": "error.html",
 },
 cors_rules=[{
 "allowedHeaders": ["*"],
 "allowedMethods": ["PUT", "POST"],
 "allowedOrigins": ["https://s3-website-test.mydomain.com"],
 "exposeHeaders": ["ETag"],
 "maxAgeSeconds": 3000,
 }],
)

You would likely have several questions if writing the above example from scratch. What keys can be set for website and cors_rules? Should the keys be camelCase or snake_case? Which keys are required? What are the types of values? To answer these questions, you’d need to refer to Bucket’s documentation.

We now allow optionally passing nested values using data classes:

from pulumi_aws import s3

my_bucket = s3.Bucket("my-bucket",
 acl="public-read",
 website=s3.BucketWebsiteArgs(
 index_document="index.html",
 error_document="error.html",
 ),
 cors_rules=[s3.BucketCorsRuleArgs(
 allowed_headers=["*"],
 allowed_methods=["PUT", "POST"],
 allowed_origins=["https://s3-website-test.mydomain.com"],
 expose_headers=["ETag"],
 max_age_seconds=3000,
 )],
)

IDEs can display argument names and types, reducing the need to refer to documentation. If a required argument is missing, or a value isn’t the right type, IDEs will immediately show the problem, allowing errors to be caught earlier.

The use of data classes makes setting nested values consistent with top-level values, with arguments that are always snake_case, for a more Pythonic feel.

Here’s what this looks like in VS Code using the new Pylance language server for Python:

Support for passing data classes is a fully compatible change. Nested data are passed as either raw dicts or data classes, so existing programs using dicts will continue to work as-is without modification.

Accessing Outputs

Nested outputs can now be accessed in a more strongly-typed way as well.

Consider accessing one of the website settings from the bucket. Previously, nested outputs were returned as raw dicts, and you’d get the value by looking it up in the dict by its key:

index_document = my_bucket.website.apply(lambda w: w["indexDocument"])

Now, nested outputs are returned as data classes, and values can be accessed via properties.

index_document = my_bucket.website.apply(lambda w: w.index_document)

Existing programs that lookup the values in the dict continue to work as-is because the new output data classes are subclasses of dict and continue to return the same data with the same keys as before.

Other improvements

Another improvement we’ve made is to hover tooltips in IDEs. Hovering over members in PyCharm and VS Code with Pylance now shows the member’s associated documentation:

We’ve additionally made some changes to reduce the amount of noise in IDE completion lists when “dotting” into individual modules.

Conclusion

All considered, these changes provide a major improvement to the Python experience.

The improvements are available in the latest releases of the AWS, Azure, AzureAD, Google Cloud, and Random providers. They will be rolling out to the rest of the providers this week, along with updates to examples and documentation.

Take the new providers for a spin and let us know what you think in the Community Slack!

Enforcing Different Kinds of Policies for Cloud Resources

Justin Van Patten — Thu, 19 Dec 2019 00:00:00 +0000

This post describes an early version of Pulumi CrossGuard (Policy as Code). The API and implementation details may have changed. For the most up-to-date information, please see the CrossGuard documentation.

We recently announced a new policy as code solution, CrossGuard that validates policies at deployment time. Policies are expressed as code and are used to prevent the creation of out-of-compliance resources. This allows an organization to prevent entire classes of security and reliability defects to ensure infrastructure is following best practices. Because policies are written using full-blown programming languages, it’s possible to do interesting things such as combining IAM Access Analyzer and Pulumi CrossGuard. In this post, we’ll take a closer look at the different types of policies that can be written.

Authoring Policies

A policy contains specific logic you want to enforce. For example, you may want to prevent the creation of public, world-readable storage objects on AWS S3, Azure BlobStore or other providers, or prevent the creation of a virtual machine without the proper firewalls in-place. Policies are run during pulumi preview and pulumi update, ensuring that cloud resource definitions comply.

Policies, written as validation functions, are evaluated against all resources in your Pulumi stack. If the validation function calls reportViolation, the associated resource is considered in violation of the policy.

There are two types of policies:

ResourceValidationPolicy - Validates a particular resource in a stack.
StackValidationPolicy - Validates the stack as a whole.

Let’s take a closer look at these and why you would use one over the other.

Resource Validation Policies

Pulumi CrossGuard calls ResourceValidationPolicy type policies for each resource in a stack. These policies are run before a resource is registered and thus block an out-of-compliance resource from ever being created or modified.

A resource validation function is passed args with more information about the resource (such as the resource’s type, input properties, name, and URN) and a reportViolation for reporting a policy violation. In most cases, you can use the helper function validateResourceOfType to filter the resource type you want to validate and provide stronger typing of the resource’s input properties.

The following is an example of a resource validation policy:

const ec2DetailedMonitoring: ResourceValidationPolicy = {
 name: "ec2-detailed-monitoring",
 description: "Detailed monitoring should be enabled for all EC2 instances.",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.ec2.Instance, (instance, args, reportViolation) => {
 if (instance.monitoring !== true) {
 reportViolation("EC2 Instance should have monitoring enabled.");
 }
 }),
};

If you have multiple resources that require a similar policy, you can group them under one policy by setting validateResource to an array of callback functions. The example below shows a case where we want to run similar checks against multiple types of resources.

const elbLoggingEnabled: ResourceValidationPolicy = {
 name: "elb-logging-enabled",
 description: "Checks whether the Application Load Balancers and the Classic Load Balancers have logging enabled.",
 enforcementLevel: "mandatory",
 validateResource: [
 validateResourceOfType(aws.elasticloadbalancing.LoadBalancer, (loadBalancer, args, reportViolation) => {
 if (loadBalancer.accessLogs === undefined || !loadBalancer.accessLogs.enabled) {
 reportViolation("Elastic Load Balancer must have access logs enabled.");
 }
 }),
 validateResourceOfType(aws.elasticloadbalancingv2.LoadBalancer, (loadBalancer, args, reportViolation) => {
 if (loadBalancer.accessLogs === undefined || !loadBalancer.accessLogs.enabled) {
 reportViolation("Elastic Load Balancer must have access logs enabled.");
 }
 }),
 validateResourceOfType(aws.applicationloadbalancing.LoadBalancer, (loadBalancer, args, reportViolation) => {
 if (loadBalancer.accessLogs === undefined || !loadBalancer.accessLogs.enabled) {
 reportViolation("Application Load Balancer must have access logs enabled.");
 }
 }),
 ],
};

The upside to resource validation policies is that they prevent resources from being created/modified during updates. However, since resource validation policies are run before resources are created, you cannot verify things like the assigned name or properties of the “live” state the resources have. This means there are some types of policies that cannot be expressed in a resource validation policy. The good news is that there are other types of policies you can write, such as stack validation policies.

Stack Validation Policies

Policies of StackValidationPolicy are run against the stack as a whole. These policies are run after all stack resources are registered and thus do not block an out-of-compliance resource from being created or modified, but do fail the preview or update. To avoid creating out-of-compliance resources, we recommend running a preview command before an update.

However, there are cases where a policy requires a resource to exist in order to perform its validation, such as inspecting the resource’s output properties that are only available to already provisioned resources (e.g. checking if a resource’s id is referenced by another resource). In such cases, the policy can skip reporting violations during previews and only report violations during updates when the information it needs is available. Thus, the preview will succeed and the resources will be created or modified during the update, but the overall update operation will fail if any resources are in violation.

Stack validation policies also allow you to write policies that require inspecting multiple resources at once. For example, that a virtual machine and its persistent hard drive are in the same stack, and have congruent settings.

A stack validation function is passed args with information about all of the resources in the stack (such as each resource’s type, output properties, name, and URN) and a reportViolation callback for reporting a policy violation.

An example stack validation policy is shown below:

import { Resource } from "@pulumi/pulumi";
import * as q from "@pulumi/pulumi/queryable";
import * as aws from "@pulumi/aws";
import { PolicyPack, ReportViolation, StackValidation, StackValidationArgs, StackValidationPolicy } from "@pulumi/policy";

const s3BucketLoggingEnabled: StackValidationPolicy = {
 name: "s3-bucket-logging-enabled",
 description: "Checks whether logging is enabled for your S3 buckets.",
 enforcementLevel: "mandatory",
 validateStack: validateStackResourcesOfType(aws.s3.Bucket, (buckets, args, reportViolation) => {
 // First, save any bucket IDs that are being used as logging targets.
 const logBucketIDs: Set<string> = new Set();
 for (const bucket of buckets) {
 if (bucket.loggings) {
 for (const logging of bucket.loggings) {
 if (logging.targetBucket) {
 logBucketIDs.add(logging.targetBucket);
 }
 }
 }
 }

 // Then, check the buckets for violations.
 for (const bucket of buckets) {
 // Skip any buckets not provisioned (i.e., during previews).
 if (!bucket.id) {
 continue;
 }
 // If the bucket doesn't have any loggings and the bucket itself isn't being used as
 // a log target, it's in violation of the policy.
 if (!bucket.loggings || bucket.loggings.length === 0) {
 if (!logBucketIDs.has(bucket.id)) {
 // Report and associate the violation with the non-compliant bucket resource.
 reportViolation("Bucket logging must be defined.", bucket.urn);
 }
 }
 }
 }),
};

The example above checks whether logging is enabled on S3 buckets. If a bucket does not have a loggings property defined, it violates the policy unless the bucket is a logging targetBucket of another bucket. The policy check skips the check if the bucket’s id isn’t available, meaning this policy only runs during updates when the resource ids are available.

Resource Validation vs. Stack Validation

	Resource Validation	Stack Validation
What does it check?	Individual resources	All resources in the stack
When is the check performed?	Before resources are created/modified	After all stack resources have been created/modified
What information is available?	Resource input properties	Resource output properties (Note: inputs are propagated to outputs during preview)

With two different ways to write policies, you have more flexibility about how you want to describe and enforce your rules:

Use resource validation policies when you want to check individual types of resources and block them from being created or modified.
Use stack validation policies when you want to check the stack as a whole (all resources in the stack at once) after they’ve been created or modified.

Trying It Out

In this post, we’ve seen the different types of CrossGuard policies that can be written.

To give it a try, download and install Pulumi and get started with policy as code. For examples of policies, check out Pulumi CrossGuard policies for AWS (AWSGuard) or our policy examples. And as always, if you’d like to share what policies you have come up with on your own you can discuss them on the Pulumi Community Slack.