Pulumi Blog: Komal Ali

Announcing the Pulumi ESC GitHub Action

Komal Ali — Wed, 05 Mar 2025 00:00:00 +0000

We’re excited to share our latest addition to the Pulumi Ecosystem: the Pulumi ESC GitHub Action. This Action lets you inject secrets and configuration securely into your GitHub Actions workflows as they are needed, rather than storing them as static, long-lived secrets.

Why we built it

Secret management continues to be a challenge for organizations of all sizes. Storing long-lived credentials in multiple places (e.g., GitHub, third-party CI systems, and internal servers) often leads to “secret sprawl” and potential security gaps. With Pulumi ESC, you can migrate to dynamic credentials, and ensure secrets don’t persist beyond their necessary lifetime. If your process requires longer-lived static credentials, with ESC you can manage credentials in a central place and rotate them automatically.

By packaging ESC functionality into a GitHub Action, we’re enabling teams to inject secrets and run ESC commands directly within their GitHub workflows. This allows for a streamlined approach to environment management and secrets rotation—without leaving GitHub.

Key benefits

Secure Secrets Injection - Inject secrets into your GitHub Actions workflows as they are needed, rather than storing them as static, long-lived secrets. This dynamic approach reduces the risk of secrets leaking in logs or pulled from unauthorized sources.
Automatic Secret Rotation - Pulumi ESC handles rotation for you behind the scenes, ensuring that any secrets or credentials you inject into GitHub Actions are valid at the time of use, but rotated on a regular schedule to avoid the risks that come with long-lived static secrets.
Streamlined CI/CD Pipeline - In addition to injecting environment variables, you can run any Pulumi ESC command directly in your GitHub Actions workflows. Automate the creation, update, or teardown of ESC Environments as part of your CI/CD process.
Seamless Integration with Pulumi Cloud - Combine the Pulumi ESC GitHub Action with pulumi/auth-actions to authenticate securely with Pulumi Cloud. No need for long-lived tokens — simply leverage short-lived tokens provided at runtime.

How it works

At a high level, the Pulumi ESC GitHub Action can do three things:

Install the Pulumi CLI - By default, the action ensures the Pulumi CLI is installed in your GitHub Actions environment. If you provide a specific version, that version is installed.
Inject all environment variables from an ESC environment - If you specify an environment (for example, tinyco/someProject/myEnv), the action will inject all environment variables and projected files associated with that environment into the GitHub Actions context.
Inject specific environment variables from an ESC environment - If you only need certain keys, you can specify them in the action’s inputs to pick and choose what gets injected, giving you fine-grained control.

Example workflow

Below is a minimal workflow that showcases injecting an entire environment’s variables into your GitHub Actions job:

on:
 - pull_request

permissions:
 id-token: write
 contents: read

jobs:
 test-env-injection:
 runs-on: ubuntu-latest
 steps:
 - name: Check out repository
 uses: actions/checkout@v4
 - name: Authenticate with Pulumi Cloud
 uses: pulumi/auth-actions@v1
 with:
 organization: pulumi
 requested-token-type: urn:pulumi:token-type:access_token:organization
 - name: Install and inject ESC environment variables
 uses: pulumi/esc-action@v3
 with:
 environment: 'tinyco/someProject/myEnv@stable'
 - name: Verify environment variables
 run: |
 echo "FOO=$FOO"
 echo "SOME_IMPORTANT_KEY=$SOME_IMPORTANT_KEY"
 echo "TEST_ENV=$TEST_ENV"

The GitHub Action workflow above does the following:

Authenticate with Pulumi Cloud using pulumi/auth-actions. Instead of storing a permanent token, we use a short-lived token to authenticate with Pulumi Cloud.
Inject the environment variables from tinyco/someProject/myEnv into the current GitHub Actions environment, making them accessible to subsequent steps in the workflow.

What’s Next?

Explore More ESC Features: Dive deeper into ESC’s environment management, secret rotation, and dynamic credentials.
Contribute to the GitHub Action: The Pulumi ESC GitHub Action is open source. Feel free to open issues, submit PRs, or suggest enhancements.
Join the Pulumi Community: Connect with other Pulumi users and share best practices in our Community Slack.

We’re thrilled to bring you the Pulumi ESC GitHub Action, allowing you to automate your infrastructure and streamline secrets management from within GitHub. This integration makes it simpler than ever to adopt best-in-class secrets management and environment configuration practices, all while speeding up your CI/CD pipelines.

Have questions or feedback? Feel free to reach out to us in our Community Slack or submit an issue on GitHub. We can’t wait to see what you build!

Announcing dependency caching for Pulumi Deployments

Komal Ali — Tue, 12 Nov 2024 11:30:00 -0700

We’re excited to announce dependency caching for Pulumi Deployments! This new feature is designed to significantly speed up deployment times by reducing dependency installation time by up to 80%, ensuring faster and more efficient workflows for every Pulumi user.

When using Pulumi Deployments, installing dependencies can be a time-consuming part of a deployment. With dependency caching, Pulumi can reuse previously installed dependencies, allowing your deployments to skip redundant installation steps and focus on the task at hand. This optimization is especially helpful for frequent deployments, reducing both time and compute resources.

How dependency caching works

Pulumi Deployments now can automatically cache dependencies for popular package managers, such as npm, pip, and go modules. When a deployment is triggered, Pulumi will:

Check for an existing cache of your dependencies.
If a cache already exists, Pulumi will reuse those dependencies, bypassing the need to install them from scratch.
If a cache does not exist, or if dependencies have changed, Pulumi will install fresh dependencies and create an updated cache to be used for subsequent deployments.

The end result? Faster, more efficient deployments to push out infrastructure changes quickly.

How to enable dependency caching

Dependency caching can be enabled through deployment settings, however you choose to configure them.

In the Pulumi Console

Open the Pulumi Console and navigate to a stack’s deployment settings (i.e. Stack page > Settings > Deploy).
Under the “Dependency caching” section, toggle the switch to enable dependency caching for your stack.
Click “Save deployment configuration” to apply the changes.

Defined as code with the Pulumi Cloud Service provider

If you prefer to define your deployment settings as code, you can use the Pulumi Cloud Service provider to enable dependency caching. Here’s an example of how to do this in TypeScript:

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();

const settings = new service.DeploymentSettings("deployment_settings", {
 organization: "service-provider-test-org",
 project: "test-deployment-settings-project",
 stack: "dev",
 operationContext: {
 environmentVariables: {
 TEST_VAR: "foo",
 SECRET_VAR: config.requireSecret("my_secret"),
 }
 },
 sourceContext: {
 git: {
 repoUrl: "https://github.com/pulumi/deploy-demos.git",
 branch: "refs/heads/main",
 repoDir: "pulumi-programs/simple-resource"
 }
 },
 cacheOptions: {
 enable: true, // <-- enable dependency caching
 }
});

Using the REST API

You can also enable dependency caching using the Pulumi REST API. See the dependency caching documentation for more information.

For more information on how dependency caching works with Pulumi Cloud Deployments and the list of supported package managers, check out the dependency caching documentation.

What’s next?

We’re continually working to enhance Pulumi Deployments and optimize the developer experience. Dependency caching is just one of the many ways we’re committed to improving this experience, and we’re excited to hear your feedback. Give it a try, and let us know how it improves your deployments!

Ready to get started? Sign in to Pulumi Cloud and experience faster deployments today!

Pushing Pulumi ESC Secrets into External Platforms

Komal Ali — Tue, 17 Sep 2024 00:00:00 -0300

This content now lives in the Pulumi docs. See Sync secrets to external platforms for the maintained version.

Managing secrets across multi-cloud infrastructures has long been a challenge for developers and operations teams. This article explores Pulumi IaC-based strategy to centrally define secrets and configuration in Pulumi ESC and automatically sync these values across the external platforms where they will be utilized, effectively reducing secret sprawl and manual overhead.

The Secret Management Conundrum

Traditional secret management often leads to a phenomenon known as secret sprawl. This occurs when secrets are duplicated across multiple systems, including various repositories, cloud providers, and CI/CD pipelines. The result is a tangled web of inconsistencies that makes tracking and managing secrets a Herculean task.

Updating secrets manually across all these systems is not only tedious but also error-prone. Each time a secret changes, it needs to be updated in every repository, cloud platform, and CI/CD tool where it’s used. This process is time-consuming and frustrating, often leading to mistakes that can compromise security.

Moreover, the lack of a centralized system makes it difficult to track where secrets are stored and who has access to them. This complicates security audits and compliance efforts. When a secret is compromised, the process of revoking and rotating it across multiple systems can be slow and disruptive, potentially leading to application downtime.

ESC Sync with Infrastructure as Code

Using Pulumi building blocks like ESC and IaC, we can create a pattern where secrets and configuration can be centrally defined in ESC, and then automatically synced across the external platforms where they will be utilized.

Consider the following ESC environment, which imports an environment “my-project/my-imported-env” and defines a set of secrets and configuration values to be synced:

imports:
 - my-project/my-imported-env@stable
values:
 sync:
 awsSecretsManager:
 value:
 myConfigKey: ${my-imported-env-foo}
 myNestedKey:
 haha: ${my-imported-env-bar}
 mySecret: ${my-imported-env-password}
 name: name-in-secrets-manager

The sync block defines the secrets and configuration values to be synced. In this example, we are syncing values to AWS Secrets Manager. The value field contains the actual values to be synced, while the name field specifies the name of the secret in AWS Secrets Manager.

To automate the synchronization process, we define a Pulumi program that creates a Pulumi ESC environment, Pulumi IaC stack, and Pulumi deployment settings for the target stack. We then define a set of pre-run commands that extract the values from the environment and set them as configuration values in the target stack. The syncCronSchedule variable specifies how often the synchronization process should run.

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();
const orgName = pulumi.getOrganization();

// projectName is the name of the project where the target stack is located
const projectName = config.require("projectName");

// stackName is the name of the target stack
const stackName = config.get("stackName") || "dev";

// repository is the GitHub repository where the target stack is located (for deployment settings)
const repository = config.require("repository");

// how often you want to sync. Default is hourly
const syncCronSchedule = config.get("syncCronSchedule") || "0 * * * *"

// envPath is the path to the environment file that contains the secrets or configuration to be synced
const envPath = config.get("envPath") || "syncEnv.yaml";

const env = new service.Environment("env", {
 organization: orgName,
 project: projectName,
 name: stackName,
 yaml: new pulumi.asset.FileAsset(envPath)
});

const stack = new service.Stack("esc-sync-aws-secretsmanager", {
 organizationName: orgName,
 projectName,
 stackName,
})

const fullyQualifiedStackName = pulumi.interpolate`${orgName}/${projectName}/${stackName}`;
const fullyQualifiedEnvName = pulumi.interpolate`${orgName}/${projectName}/${env.name}`;

const settings = new service.DeploymentSettings("deployment_settings", {
 organization: orgName,
 project: stack.projectName,
 stack: stack.stackName,
 github: {
 repository,
 },
 sourceContext: {
 git: {
 branch: "main",
 repoDir: "sync/target",
 }
 },
 operationContext: {
 preRunCommands: [
 'pulumi login',
 pulumi.interpolate`pulumi config env add ${projectName}/${env.name} -s ${fullyQualifiedStackName} --yes`,
 pulumi.interpolate`pulumi env open ${fullyQualifiedEnvName} sync.awsSecretsManager.value > sync.json`,
 pulumi.interpolate`pulumi config set -s ${fullyQualifiedStackName} secretName $(pulumi env open ${fullyQualifiedEnvName} sync.awsSecretsManager.name)`,
 ]
 }
});

const schedule = new service.DeploymentSchedule("update_schedule", {
 organization: orgName,
 project: settings.project,
 stack: settings.stack,
 scheduleCron: syncCronSchedule,
 pulumiOperation: "update",
})

Finally, we define a Pulumi program that creates the AWS Secrets Manager secret and secret version based on the values extracted from the environment file. The sync.json file contains the values to be synced, which are read and set as the secret string in the secret version. This Pulumi program is executed on a schedule to ensure that the secrets and configuration values are up-to-date between Pulumi ESC and AWS Secrets Manager.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from "fs";

const config = new pulumi.Config();
const name = config.require("secretName");

// Read a json file from the local filesystem using node.js fs module
const json = fs.readFileSync("sync.json", "utf8");

const secret = new aws.secretsmanager.Secret(name, {
 recoveryWindowInDays: 0,
})

const secretVersion = new aws.secretsmanager.SecretVersion(`${name}-version`, {
 secretId: secret.id,
 secretString: json,
})

// Export the name of the secret
export const secretName = secret.name;

This pattern allows you to define secrets and configuration values in a single location and automatically sync them across external platforms. The same pattern can be extended to create secrets in other platforms like Azure Key Vault, GCP Secrets Manager, GitHub Secrets, HashiCorp Vault and countless others.

Webhooks: An Event-Driven Alternative Approach

In addition to the scheduled synchronization approach, Pulumi ESC also supports webhooks. Webhooks allow you to respond to events in real-time, such as when a secret is updated in ESC, and take immediate action to sync the new value to an external platform.

In the example below, instead of a scheduled deployment, we define a webhook that triggers when the Pulumi ESC environment is updated. This ensures that the secrets and configuration values are always in sync between Pulumi ESC and the external platform.

const webhook = new service.Webhook("webhook", {
 organizationName: orgName,
 projectName: settings.project,
 stackName: settings.stack,
 displayName: "Sync to AWS Secrets Manager",
 environmentName: pulumi.interpolate`${env.project}/${env.name}`,
 filters: [WebhookFilters.ImportedEnvironmentChanged],
 format: WebhookFormat.PulumiDeployments,
 payloadUrl: pulumi.interpolate`${stack.projectName}/${stack.stackName}`,
 active: true,
})

When the webhook is triggered (i.e. the “my-project/my-imported-env” environment is updated), a deployment is automatically triggered, and the secrets and configuration values are synced to AWS Secrets Manager.

Embracing a New Era of Secret Management

Using Pulumi IaC to synchronize secrets and configuration across platform boundaries represents a giant step forward in secret management. By centralizing and automating the handling of sensitive data, it addresses the key challenges of traditional approaches. Secret sprawl becomes a thing of the past, and the risks associated with manual updates are significantly reduced.

Whether you’re working with AWS Secrets Manager, Azure Key Vault, GCP Secrets Manager, GitHub Secrets or other platforms, Pulumi ESC provides a streamlined solution that ensures your secrets are always secure and up-to-date. It offers the convenience and security of a single source of truth for all your secrets, allowing you to focus on building and improving your applications rather than wrestling with secret management.

We encourage you to explore Pulumi’s documentation and dive into the Pulumi ESC Sync Examples repository to see how you can implement this powerful tool in your own projects. Join our vibrant community on Slack to discuss your experiences, ask questions, and share insights. Together, we can make secret management simpler, more secure, and more efficient.

Pulumi Announces Drift Detection and Remediation: Continuously Monitor Your Infrastructure Changes

Meagan Cojocar — Wed, 24 Apr 2024 07:00:04 -0800

At Pulumi, we recognize the challenges platform teams face in maintaining the stability and compliance of their cloud infrastructures. One of the primary challenges is configuration drift, where the actual state of the infrastructure deviates from its intended state. This deviation can occur for various reasons, including manual adjustments made directly in the cloud provider’s console, unintended consequences of scripts, or unauthorized changes. Such drift can lead to significant problems including security vulnerabilities that open up potential breaches, compliance violations that can result in penalties, operational disruptions that affect user experience and business operations, and increased costs from unnecessary resource spend.

Today, we are excited to launch a new Drift Detection and Remediation feature in Pulumi Cloud to address these challenges. This new functionality is designed to automate the detection and correction of drift in your cloud environments, seamlessly integrating into your existing workflows, whether you use Pulumi Deployments or other CI/CD systems, and is supported for any resource you can manage with Pulumi, whether they are from Amazon Web Services, Microsoft Azure, Google Cloud Platform, or any of the 180+ supported providers. Drift Detection and Remediation is available today on the Enterprise and Business Critical editions of Pulumi Cloud.

Why Pulumi Cloud Drift Detection and Remediation?

Drift Detection is the process of identifying changes in the actual state of your cloud environment that deviate from the expected configuration stored in Pulumi Cloud. This can occur when an operator makes an out-of-band change directly in the cloud provider console, bypassing the usual code deployment processes.

Key benefits:

Enhanced Security: Stay ahead of potential security breaches caused by unauthorized changes to your infrastructure.
Compliance Assurance: Ensure that your infrastructure remains compliant with organizational policies and standards after out of band changes.
Time and Cost Efficiency: Reduce the time spent on tracking and rectifying configuration drift or managing a custom drift solution: leading to cost savings.
Risk Mitigation: Minimize the risk of errors and potential outages by promptly identifying and addressing accidental drifts in your environments.

How Drift Detection Works

Pulumi Cloud’s Drift Detection and Remediation operates continuously, on a schedule of your choosing, comparing the state of your resources with the expected configurations defined in your Pulumi setups. Any discrepancies triggered by modifications, deletions, or additions of resources are promptly reported in the new Drift tab in the Pulumi Cloud console. Alerts can be configured to be sent via webhooks, Slack, or Microsoft Teams when drift occurs, with detailed information about the drift’s nature and scope provided directly within the alerts.

For those using Pulumi Deployments, the system not only regularly detects but can also remedy the drift to align with your Pulumi program. This auto-remediation applies the last known infrastructure as code state, overwriting the drift. Auto-remediation can be set to occur by default or turned off to allow for manual intervention in the remediation of drift, depending on the criticality and nature of the stacks involved.

Under the hood, drift is detected by regularly running a new Pulumi operation we have added, pulumi refresh --preview-only. This operation collects the set of changes that exist between the cloud provider and the current desired state in Pulumi. These scheduled operations build on top of the new Scheduled Deployments functionality as is Time-to-Live Stacks, both of which were also announced today. If there is a delta between the cloud and the current desired state, it is presented in the Drift tab along with a “Drift detected” notification on the stack. If auto-remediation is configured, a pulumi update --refresh is run automatically following any detection of drift. All of this is automated by Pulumi Deployments for users who have configured Pulumi Deployments on their stack. For users who are not using Pulumi Deployments, they can configure their existing CI/CD system to run these same operations regularly, and the results will still be displayed as part of the Drift tab in Pulumi Cloud and you can receive notifications when drift is detected.

Getting Started with Drift Detection

To get started with Drift Detection in Pulumi, select which deployment route you will be using:

Pulumi Deployments: Pulumi programs are run on Pulumi-hosted compute, allowing you to set a schedule and let us handle the rest, including detecting drift, auto-remediation if you turn it on, history of each drift run and what was detected and notifications for drift events.
Existing CI/CD system: Run Drift Detection by regularly running pulumi refresh --preview-only to see what drift is detected in the Pulumi Cloud console and get notified on drift events.

Setting it Up in the Pulumi Cloud Console

In order to set up Drift Detection and Remediation in the Pulumi Cloud console, follow these steps:

Ensure Deployments Settings are configured on the stack see the docs
Navigate to the Stack > Settings > Schedules page
Select “Drift”
(Optional) Turn on auto-remediation if applicable
Set the schedule using a cron expression
Save the Schedule

And just like that, you have Drift Detection! To test what the output will look like you can use the Actions button in the top right on the Stack page to run an ad hoc Drift Detection operation or an Auto-remediate operation. Note: if you have never run a pulumi refresh on that stack before, you will want to run one before setting up Drift Detection and ensure a clean refresh state. More details on getting a clean refresh state in our documentation.

You will now see on the Drift tab, a summary of what resources have been updated or deleted, the properties that have changed, when the drift run happened and a link to the Deployment with further details.

When drift is detected, you have a couple of options:

Run remediation from Pulumi Cloud: If you want to overwrite the changes made in your cloud provider with the most recently specified desired state of your Pulumi program. This can be done via the “Remediate drift” option in the Actions drop down on your stack page.
Run refresh from Pulumi Cloud: If you want to accept the changes made in your cloud provider back into your Pulumi desired state. This can be done via the “Refresh” option in the Actions drop down on your stack page. You may also want to update your Pulumi program to align with the changes made in the cloud provider.

Setting it Up via the API

For those who prefer to automate and script their infrastructure tasks, Drift Detection and Remediation can be configured programmatically using simple HTTP requests. Here are the new endpoints we have added:

Create a Drift schedule
Get a Drift schedule
Update or delete a Drift schedule
Pause or resume a Drift schedule
List all schedules (includes raw Pulumi operations and Time-to-Live schedules)

Below is an example of setting up Drift Detection and Remediation on a stack, see the Pulumi Deployments REST API documentation for more details on how to set Drift Detection and Remediation up programmatically.

Create a Drift Detection and Remediation schedule:

curl -H "Accept: application/vnd.pulumi+json" \
 -H "Content-Type: application/json" \
 -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
 --request POST \
 --data '{"scheduleCron":"0 0 * * *","autoRemediate":true}' \
 https://api.pulumi.com/api/stacks/{organization}/{project}/{stack}/deployments/drift/schedules

Setting it Up via the Pulumi Service Provider

The Pulumi Service Provider allows you to set up automated Drift Detection and Remediation in source control.

import * as pulumi from "@pulumi/pulumi";
import * as pulumiservice from "@pulumi/pulumiservice";

const organizationName = "my-org";
const projectName = "my-project";
const stackName = "production";

// Creating a DriftSchedule for automatically running drift detection
const driftDetectionSchedule = new pulumiservice.DriftSchedule("driftDetectionSchedule", {
 organization: organizationName,
 project: projectName,
 stack: stackName,
 scheduleCron: "0 0 * * *", // Run drift detection daily at midnight
 autoRemediate: true, // Automatically remediate any drift detected
});

export const driftScheduleId = driftDetectionSchedule.scheduleId;

import pulumi
import pulumi_pulumiservice as pulumiservice

organization_name = "my-org"
project_name = "my-project"
stack_name = "production"

# Create a drift detection schedule
drift_detection_schedule = pulumiservice.DriftSchedule("driftDetectionSchedule",
 organization=organization_name,
 project=project_name,
 stack=stack_name,
 schedule_cron="0 0 * * *", # Run drift detection daily at midnight
 auto_remediate=True) # Automatically remediate any drift detected

pulumi.export('driftScheduleId', drift_detection_schedule.schedule_id)

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 driftDetectionSchedule, err := pulumiservice.NewDriftSchedule(ctx, "driftDetectionSchedule", &pulumiservice.DriftScheduleArgs{
 Organization: pulumi.String("my-org"),
 Project: pulumi.String("my-project"),
 Stack: pulumi.String("production"),
 ScheduleCron: pulumi.String("0 0 * * *"), // Run drift detection daily at midnight
 AutoRemediate: pulumi.Bool(true), // Automatically remediate any drift detected
 })
 if err != nil {
 return err
 }

 ctx.Export("driftScheduleId", driftDetectionSchedule.ScheduleId)
 return nil
 })
}

using Pulumi;
using PulumiService = Pulumi.PulumiService;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(() => {
 var driftDetectionSchedule = new PulumiService.DriftSchedule("driftDetectionSchedule", new PulumiService.DriftScheduleArgs
 {
 Organization = "my-org",
 Project = "my-project",
 Stack = "production",
 ScheduleCron = "0 0 * * *", // Run drift detection daily at midnight
 AutoRemediate = true, // Automatically remediate any drift detected
 });

 return new Dictionary<string, object?>
 {
 { "driftScheduleId", driftDetectionSchedule.ScheduleId }
 };
 });
}

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.pulumiservice.Webhook;
import com.pulumi.pulumiservice.WebhookArgs;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 private static void stack(Context ctx) {
 var driftDetectionSchedule = new Webhook("driftDetectionSchedule", WebhookArgs.builder()
 .organization("my-org")
 .project("my-project")
 .stack("production")
 .scheduleCron("0 0 * * *") // Run drift detection daily at midnight
 .autoRemediate(true) // Automatically remediate any drift detected
 .build());

 ctx.export("driftScheduleId", driftDetectionSchedule.scheduleId());
 }
}

name: drift-detection-setup
runtime: yaml
description: Setup of automated drift detection with Pulumi

resources:
 driftDetectionSchedule:
 type: pulumiservice:index:DriftSchedule
 properties:
 organization: my-org
 project: my-project
 stack: production
 scheduleCron: "0 0 * * *" # Run drift detection daily at midnight
 autoRemediate: true # Automatically remediate any drift detected

outputs:
 driftScheduleId: ${driftDetectionSchedule.scheduleId}

See the Pulumi Service Provider documentation for more details on how to manage Drift Detection and Remediation in source control.

Wrapping it up

We are thrilled to be releasing a top customer ask today and are looking forward to hearing about your experience using Drift Detection and Remediation.

Until next time, happy building!

Pulumi Cloud Gets Full Historical Views of Resources and Stack Outputs

Meagan Cojocar — Tue, 12 Dec 2023 10:34:32 -0800

As part of our ongoing commitment to providing developers and infrastructure teams with powerful tools, we’re thrilled to introduce the latest enhancement to Pulumi Cloud’s stack update pages. We have shipped an improvement to Pulumi Cloud stack update pages where all update pages now show the resources in the stack at that time and the stack outputs going back since your first update on the stack.

Expanded Resource Information

Now, each update will provide detailed information about the resources. This includes a list of resources created, modified, or deleted in each update, offering a clearer understanding of the changes made to your infrastructure.

Stack Outputs

The updated pages also now include information on the stack’s outputs for each update instead of just the current point in time. The value of this is being able to go back and see what output values were after any update.

The Value of Enhanced Data Visibility

Having a more detailed and transparent view of your stack updates provides several key benefits:

Resource Lifecycle Management: Understanding when and why a resource was added, modified, or deleted helps in better managing your infrastructure’s lifecycle and planning future changes.
Enhanced Troubleshooting and Analysis: The ability to go back to a specific point in time and analyze the state of your infrastructure at that moment can be invaluable in troubleshooting issues or understanding the impact of certain changes.

In summary, the improved Pulumi Cloud stack update pages offer a more detailed, transparent, and insightful view of your infrastructure changes. This upgrade came directly from customer asks, and underscores our commitment to empowering developers with the tools and information they need to effectively manage and understand their cloud resources and how they change over time.

Stay tuned for more updates as we continue to enhance our platform to better meet your needs. As always, submit any feedback on the feature in the Pulumi Cloud Requests repository.

Announcing Microsoft Teams Webhook Integration for Pulumi Cloud

Meagan Cojocar — Mon, 11 Sep 2023 00:00:00 +0000

Getting your Pulumi notifications into Microsoft Teams is now easier than ever. Organizations using Pulumi Cloud have previously been setting up Microsoft Teams notifications for their stacks using Pulumi Cloud Webhooks. Webhooks can attach to a Pulumi organization or a Pulumi stack. Starting today, customers can set up a Microsoft Teams integration, for organization or stack notifications, with fewer steps and without needing to host the infrastructure themselves.

Pulumi Cloud Webhooks, including the Microsoft Teams integration, are available to all Pulumi Cloud organizations.

See it in action!

Pulumi Microsoft Teams notifications enable central visibility for your team. In addition to getting the notification of a successful or failed update, you will have links to take you directly to the stack and directly to the update itself. You can see which Pulumi operation was being run, which user initiated it and the number of resources changed and unaffected. All of this information in one place lets you spend less time context switching and more time collaborating.

Setting up the Microsoft Teams Integration

Before today, Pulumi customers used Pulumi Cloud Webhooks to set up generic JSON webhooks. When events occur, we send a HTTP POST request to any registered listeners. Webhooks can then be used to send notifications to an app (like Microsoft Teams), start running automated tests, or even to update another stack! We have now built support for Microsoft Teams formatted webhooks, which allow you to quickly enable notifications about your Pulumi stacks and organizations into your Microsoft Teams workspace by simply providing a incoming webhook URL.

By following these steps, which can also be found in our webhooks documentation, in a few clicks you will have a Microsoft Teams incoming webhook URL which you can use to set up a webhook in Pulumi Cloud, as shown in the GIF below.

There are Pulumi customers with hundreds to thousands of stacks, making setting up webhooks for each via the UI a time consuming task. In order to enable Pulumi notifications at scale, we have added Pulumi Cloud REST API endpoints for creating webhooks as well as Pulumi Service Provider support.

Pulumi Cloud REST API

Create a webhook via the Pulumi Cloud REST API formatted for Microsoft Teams, as shown in the example below.


curl \
 -H "Accept: application/vnd.pulumi+8" \
 -H "Content-Type: application/json" \
 -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
 --request POST \
 --data '{
 "organizationName": "{organization}",
 "projectName": "{project}",
 "stackName": "{stack}",
 "displayName": "my-display-name",
 "payloadUrl": "https://xxxxx.webhook.office.com/xxxxxxxxx",
 "format": "ms_teams",
 "active": true
 }' \
 https://api.pulumi.com/api/orgs/{organization}/{project}/{stack}/hooks

Pulumi Service Provider

The Pulumi Service provider allows you to create Pulumi Cloud resources via Pulumi. You can provision and manage webhooks, including Microsoft Teams webhooks, using this provider.

import * as pulumi from "@pulumi/pulumi";
import { Webhook, WebhookFormat, WebhookFilters } from "@pulumi/pulumiservice";

const orgName = "my-org";

const orgWebhook = new Webhook("org-webhook", {
 active: true,
 displayName: "my-teams-channel",
 organizationName: orgName,
 payloadUrl: "https://xxxxx.webhook.office.com/xxxxxxxxx",
 format: WebhookFormat.MicrosoftTeams,
 filters: [
 WebhookFilters.StackCreated,
 WebhookFilters.StackDeleted,
 WebhookFilters.DeploymentSucceeded,
 WebhookFilters.DeploymentFailed,
 ],
});

const stackWebhook = new Webhook("stack-webhook", {
 active: true,
 displayName: "my-teams-channel",
 organizationName: orgName,
 projectName: pulumi.getProject(),
 stackName: pulumi.getStack(),
 payloadUrl: "https://xxxxx.webhook.office.com/xxxxxxxxx",
 format: WebhookFormat.MicrosoftTeams,
});

export const orgWebhookName = orgWebhook.name;
export const stackWebhookName = stackWebhook.name;


name: webhook-examples
runtime: yaml
description: An example of provisioning a Pulumi Cloud Webhook via Pulumi Service Provider

resources:
 orgWebhook:
 type: pulumiservice:index:Webhook
 properties:
 active: true
 displayName: my-teams-channel
 organizationName: my-org
 payloadUrl: "https://xxxxx.webhook.office.com/xxxxxxxxx"
 format: ms_teams
 filters:
 - stack_created
 - stack_deleted
 - deployment_succeeded
 - deployment_failed
 stackWebhook:
 type: pulumiservice:index:Webhook
 properties:
 active: true
 displayName: my-teams-channel
 organizationName: my-org
 projectName: webhook-examples
 stackName: dev
 payloadUrl: "https://xxxxx.webhook.office.com/xxxxxxxxx"
 format: ms_teams

outputs:
 # export the name of the webhooks
 orgWebhookName: ${orgWebhook.name}
 stackWebhookName: ${stackWebhook.name}

import pulumi
import pulumi_pulumiservice as pulumiservice

org_webhook = pulumiservice.Webhook("orgWebhook",
 active=True,
 display_name="my-teams-channel",
 organization_name="my-org",
 payload_url="https://xxxxx.webhook.office.com/xxxxxxxxx",
 format=pulumiservice.WebhookFormat.MICROSOFT_TEAMS,
 filters=[
 pulumiservice.WebhookFilters.STACK_CREATED,
 pulumiservice.WebhookFilters.STACK_DELETED,
 pulumiservice.WebhookFilters.DEPLOYMENT_SUCCEEDED,
 pulumiservice.WebhookFilters.DEPLOYMENT_FAILED,
 ])

stack_webhook = pulumiservice.Webhook("stackWebhook",
 active=True,
 display_name="my-teams-channel",
 organization_name="my-org",
 project_name="webhook-examples",
 stack_name="dev",
 payload_url="https://xxxxx.webhook.office.com/xxxxxxxxx",
 format=pulumiservice.WebhookFormat.MICROSOFT_TEAMS)

pulumi.export("orgWebhookName", org_webhook.name)
pulumi.export("stackWebhookName", stack_webhook.name)

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 orgWebhook, err := pulumiservice.NewWebhook(ctx, "orgWebhook", &pulumiservice.WebhookArgs{
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("my-teams-channel"),
 OrganizationName: pulumi.String("my-org"),
 PayloadUrl: pulumi.String("https://xxxxx.webhook.office.com/xxxxxxxxx"),
 Format: pulumiservice.WebhookFormatMicrosoftTeams,
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersStackCreated,
 pulumiservice.WebhookFiltersStackDeleted,
 pulumiservice.WebhookFiltersDeploymentSucceeded,
 pulumiservice.WebhookFiltersDeploymentFailed,
 },
 })
 if err != nil {
 return err
 }
 stackWebhook, err := pulumiservice.NewWebhook(ctx, "stackWebhook", &pulumiservice.WebhookArgs{
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("my-teams-channel"),
 OrganizationName: pulumi.String("my-org"),
 ProjectName: pulumi.String("webhook-examples"),
 StackName: pulumi.String("dev"),
 PayloadUrl: pulumi.String("https://xxxxx.webhook.office.com/xxxxxxxxx"),
 Format: pulumiservice.WebhookFormatMicrosoftTeams,
 })
 if err != nil {
 return err
 }
 ctx.Export("orgWebhookName", orgWebhook.Name)
 ctx.Export("stackWebhookName", stackWebhook.Name)
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using PulumiService = Pulumi.PulumiService;

return await Deployment.RunAsync(() =>
{
 var orgWebhook = new PulumiService.Webhook("orgWebhook", new()
 {
 Active = true,
 DisplayName = "my-teams-channel",
 OrganizationName = "my-org",
 PayloadUrl = "https://xxxxx.webhook.office.com/xxxxxxxxx",
 Format = PulumiService.WebhookFormat.MicrosoftTeams,
 Filters = new[]
 {
 PulumiService.WebhookFilters.StackCreated,
 PulumiService.WebhookFilters.StackDeleted,
 PulumiService.WebhookFilters.DeploymentSucceeded,
 PulumiService.WebhookFilters.DeploymentFailed,
 },
 });

 var stackWebhook = new PulumiService.Webhook("stackWebhook", new()
 {
 Active = true,
 DisplayName = "my-teams-channel",
 OrganizationName = "my-org",
 ProjectName = "webhook-examples",
 StackName = "dev",
 PayloadUrl = "https://xxxxx.webhook.office.com/xxxxxxxxx",
 Format = PulumiService.WebhookFormat.MicrosoftTeams,
 });

 return new Dictionary<string, object?>
 {
 ["orgWebhookName"] = orgWebhook.Name,
 ["stackWebhookName"] = stackWebhook.Name,
 };
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.pulumiservice.Webhook;
import com.pulumi.pulumiservice.WebhookArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var orgWebhook = new Webhook("orgWebhook", WebhookArgs.builder()
 .active(true)
 .displayName("my-teams-channel")
 .organizationName("my-org")
 .payloadUrl("https://xxxxx.webhook.office.com/xxxxxxxxx")
 .format("ms_teams")
 .filters(
 "stack_created",
 "stack_deleted",
 "deployment_succeeded",
 "deployment_failed")
 .build());

 var stackWebhook = new Webhook("stackWebhook", WebhookArgs.builder()
 .active(true)
 .displayName("my-teams-channel")
 .organizationName("my-org")
 .projectName("webhook-examples")
 .stackName("dev")
 .payloadUrl("https://xxxxx.webhook.office.com/xxxxxxxxx")
 .format("ms_teams")
 .build());

 ctx.export("orgWebhookName", orgWebhook.name());
 ctx.export("stackWebhookName", stackWebhook.name());
 }
}

Wrapping up

Our webhooks improvements enable ChatOps workflows and more visibility into your infrastructure. As always, please let us know if you have feedback on the feature by opening an issue in the Pulumi Cloud requests repository or if you have other features you would like to see in Pulumi Cloud.

Happy building! 👷

Dependent Stack Updates with Pulumi Deployments

Evan Boyle — Tue, 20 Jun 2023 00:00:00 +0000

As infrastructure projects grow in size and complexity, you need to decompose infrastructure into smaller stacks to limit the blast radius of errors, extract and reference common layers like networking, and limit access to sensitive components. This comes with a coordination cost as you now need to figure out how to detect and propagate changes to downstream stacks in your dependency graph. Today we’re announcing two features that can help you manage this complexity by automatically updating dependent stacks:

Deployment Webhook Destinations for triggering downstream stack updates automatically
The Pulumi Auto Deploy Package for easily expressing dependencies and configuring downstream updates via Deployment Webhooks.

The following example shows automatic deployment of stacks with the following dependency graph
a
├── b
│ ├── d
│ ├── e
│ └── f
└── c
Whenever a node in the graph is updated, all downstream nodes will be automatically updated
via a webhook triggering Pulumi Deployments.

Automatically updating dependent stacks and providing a way to write Pulumi programs that can express nested stack updates are two of the most requested features across the Pulumi project, and we’re excited to enable both of these via Pulumi Cloud and Pulumi Deployments.

Deployment Webhook Destinations

Pulumi Webhooks allow you to pick one or more event types on a stack (i.e. update succeeded, or refresh failed) and automatically deliver an event to a destination such a slack channel or a webserver you’ve deployed with custom code. Now you can choose Pulumi Deployments as a webhook destination. When an update completes on the parent stack, a deployment will automatically be triggered on the destination stack. This makes it easy to keep dependent stacks up to date. For instance, you can use Deployment Webhook Destinations to configure every successful update on your networking stack to trigger downstream updates on your database and compute stacks.

import * as pulumi from "@pulumi/pulumi";
import * as pulumiservice from "@pulumi/pulumiservice";

const databaseWebhook = new pulumiservice.Webhook("databaseWebhook", {
 organizationName: "org",
 projectName: "network",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "database/prod",
 active: true,
 displayName: "deploy-database",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});
const computeWebhook = new pulumiservice.Webhook("computeWebhook", {
 organizationName: "org",
 projectName: "database",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "compute/prod",
 active: true,
 displayName: "deploy-compute",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using PulumiService = Pulumi.PulumiService;

return await Deployment.RunAsync(() =>
{
 var databaseWebhook = new PulumiService.Webhook("databaseWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "network",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "database/prod",
 Active = true,
 DisplayName = "deploy-database",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

 var computeWebhook = new PulumiService.Webhook("computeWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "database",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "compute/prod",
 Active = true,
 DisplayName = "deploy-compute",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

});

import pulumi
import pulumi_pulumiservice as pulumiservice

database_webhook = pulumiservice.Webhook("databaseWebhook",
 organization_name="org",
 project_name="network",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="database/prod",
 active=True,
 display_name="deploy-database",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])
compute_webhook = pulumiservice.Webhook("computeWebhook",
 organization_name="org",
 project_name="database",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="compute/prod",
 active=True,
 display_name="deploy-compute",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := pulumiservice.NewWebhook(ctx, "databaseWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("network"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("database/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-database"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 _, err = pulumiservice.NewWebhook(ctx, "computeWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("database"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("compute/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-compute"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
resources:
 databaseWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: network
 stackName: prod
 format: pulumi_deployments
 payloadUrl: database/prod
 active: true
 displayName: deploy-database
 filters:
 - update_succeeded
 computeWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: database
 stackName: prod
 format: pulumi_deployments
 payloadUrl: compute/prod
 active: true
 displayName: deploy-compute
 filters:
 - update_succeeded

Expressing Stack Dependencies with Pulumi Auto Deploy

We’ve also shipped a new Pulumi package called Pulumi Auto Deploy. It lets you simply express dependencies between stacks, and takes care of creating and updating the necessary Deployment Webhooks under the hood.

import * as autodeploy from "@pulumi/auto-deploy";
import * as pulumi from "@pulumi/pulumi";

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

const organization = pulumi.getOrganization();
const project = "dependency-example"

export const f = new autodeploy.AutoDeployer("auto-deployer-f", {
 organization,
 project,
 stack: "f",
 downstreamRefs: [],
});

export const e = new autodeploy.AutoDeployer("auto-deployer-e", {
 organization,
 project,
 stack: "e",
 downstreamRefs: [],
});

export const d = new autodeploy.AutoDeployer("auto-deployer-d", {
 organization,
 project,
 stack: "d",
 downstreamRefs: [],
});

export const c = new autodeploy.AutoDeployer("auto-deployer-c", {
 organization,
 project,
 stack: "c",
 downstreamRefs: [],
});

export const b = new autodeploy.AutoDeployer("auto-deployer-b", {
 organization,
 project,
 stack: "b",
 downstreamRefs: [d.ref, e.ref, f.ref],

});

export const a = new autodeploy.AutoDeployer("auto-deployer-a", {
 organization,
 project,
 stack: "a",
 downstreamRefs: [b.ref, c.ref],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AutoDeploy = Pulumi.AutoDeploy;

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

return await Deployment.RunAsync(() =>
{
 var projectVar = "dependency-example";

 var organization = "pulumi";

 var f = new AutoDeploy.AutoDeployer("f", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "f",
 DownstreamRefs = new[] {},
 });

 var e = new AutoDeploy.AutoDeployer("e", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "e",
 DownstreamRefs = new[] {},
 });

 var d = new AutoDeploy.AutoDeployer("d", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "d",
 DownstreamRefs = new[] {},
 });

 var c = new AutoDeploy.AutoDeployer("c", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "c",
 DownstreamRefs = new[] {},
 });

 var b = new AutoDeploy.AutoDeployer("b", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "b",
 DownstreamRefs = new[]
 {
 d.Ref,
 e.Ref,
 f.Ref,
 },
 });

 var a = new AutoDeploy.AutoDeployer("a", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "a",
 DownstreamRefs = new[]
 {
 b.Ref,
 c.Ref,
 },
 });

});

import pulumi
import pulumi_auto_deploy as auto_deploy

'''

The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
Whenever a node in the graph is updated,
all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
'''

project_var = "dependency-example"
organization = pulumi.get_organization()
f = auto_deploy.AutoDeployer("f",
 organization=organization,
 project=project_var,
 stack="f",
 downstream_refs=[])
e = auto_deploy.AutoDeployer("e",
 organization=organization,
 project=project_var,
 stack="e",
 downstream_refs=[])
d = auto_deploy.AutoDeployer("d",
 organization=organization,
 project=project_var,
 stack="d",
 downstream_refs=[])
c = auto_deploy.AutoDeployer("c",
 organization=organization,
 project=project_var,
 stack="c",
 downstream_refs=[])
b = auto_deploy.AutoDeployer("b",
 organization=organization,
 project=project_var,
 stack="b",
 downstream_refs=[
 d.ref,
 e.ref,
 f.ref,
 ])
a = auto_deploy.AutoDeployer("a",
 organization=organization,
 project=project_var,
 stack="a",
 downstream_refs=[
 b.ref,
 c.ref,
 ])

package main

import (
 "github.com/pulumi/pulumi-auto-deploy/sdk/go/autodeploy"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 /**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

 pulumi.Run(func(ctx *pulumi.Context) error {
 projectVar := "dependency-example"
 organization := "pulumi"
 f, err := autodeploy.NewAutoDeployer(ctx, "f", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("f"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 e, err := autodeploy.NewAutoDeployer(ctx, "e", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("e"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 d, err := autodeploy.NewAutoDeployer(ctx, "d", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("d"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 c, err := autodeploy.NewAutoDeployer(ctx, "c", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("c"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 b, err := autodeploy.NewAutoDeployer(ctx, "b", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("b"),
 DownstreamRefs: pulumi.StringArray{
 d.Ref,
 e.Ref,
 f.Ref,
 },
 })
 if err != nil {
 return err
 }
 _, err = autodeploy.NewAutoDeployer(ctx, "a", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("a"),
 DownstreamRefs: pulumi.StringArray{
 b.Ref,
 c.Ref,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
variables:
 project: dependency-example
 # TODO: update once https://github.com/pulumi/pulumi-yaml/issues/461 is fixed
 organization: pulumi
resources:
 f:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: f
 downstreamRefs: []
 e:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: e
 downstreamRefs: []
 d:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: d
 downstreamRefs: []
 c:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: c
 downstreamRefs: []
 b:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: b
 downstreamRefs:
 - ${d.ref}
 - ${e.ref}
 - ${f.ref}
 a:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: a
 downstreamRefs:
 - ${b.ref}
 - ${c.ref}

Just declare an AutoDeployer resource for each stack, and any downstream stacks will automatically be updated whenever a stack in its dependency chain is updated.

Automate your deployment pipeline

Use Pulumi Deployments to trigger downstream stack updates automatically when an upstream stack changes. Keep your dependent infrastructure in sync without manual runbooks.

Get started

Scale Infrastructure with Software

At Pulumi, we build tools to help you scale your infrastructure footprint with software instead of just humans. Deployment Webhook Destinations and pulumi-auto-deploy give you the tools to keep all of your dependent infrastructure up to date with automation instead of manual runbooks and human-driven operations.

Check out these resources to get started today:

Check out the Pulumi Deployments documentation and learn about configuring Deployment Settings for your stacks.
Get started with the Pulumi Auto Deploy Package.
Learn more about Pulumi Cloud Webhooks.
Join us in the #pulumi-deployments community slack channel.

🚀🚀🚀 Happy building! 🚀🚀🚀

Announcing Slack and Deployment Notifications for Pulumi Cloud

Meagan Cojocar — Tue, 30 May 2023 00:00:00 +0000

Today, we are excited to introduce a set of improvements to Pulumi Cloud Webhooks designed to deliver your deployment notifications to where you already spend your time, enabling faster response times to critical issues. Getting your deployment notifications into Slack is now easier than ever on Pulumi Cloud with our new Slack integration. We are also announcing two new improvements to our webhooks feature: Pulumi Deployments events and fine-grained event filtering.

Teams using Pulumi Cloud have been setting up Slack notifications for their stacks using Pulumi Cloud Webhooks. Webhooks can attach to a Pulumi organization or a Pulumi stack. Starting today, customers can set up a Slack integration, for organization or stack notifications, with fewer steps and without needing to host the infrastructure themselves. We have also expanded the types of event notifications and the filtering options on both Pulumi Cloud Webhooks and for the Slack integration.

Pulumi Cloud Webhooks, including the Slack integration, are available to all Pulumi Cloud organizations.

Slack integration

Pulumi Slack notifications enable central visibility for your team. In addition to getting the notification of a successful or failed update, you will have links to take you directly to the stack and directly to the update itself. You can see what Pulumi operation was being run, which user initiated it and the number of resources changed and unaffected. All of this information in one place lets you spend less time context switching and more time collaborating.

Setting up the Slack Integration

Before today Pulumi customers used Pulumi Cloud Webhooks to set up generic JSON webhooks. When events occur, we send a HTTP POST request to any registered listeners. Webhooks can then be used to send notifications to an app (like Slack), start running automated tests, or even to update another stack! We have now built support for Slack-formatted webhooks, which allow you to quickly enable notifications about your Pulumi stacks and organizations into your Slack workspace by simply providing a Slack incoming webhook URL.

You can either create your own Slack app (or use an existing one you may already have installed in your workspace) or you can use the link below to quickly get started with a predefined Slack app manifest.

Create a Slack app from manifest

By following these steps, which can also be found in our webhooks documentation, in a few clicks you will have a Slack incoming webhook URL which you can use to set up a webhook in Pulumi Cloud, as shown in the GIF below.

Pulumi Cloud REST API

Create a webhook via the Pulumi Cloud REST API formatted for Slack, as shown in the example below.


curl \
 -H "Accept: application/vnd.pulumi+8" \
 -H "Content-Type: application/json" \
 -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
 --request POST \
 --data '{
 "organizationName":"{organization}",
 "projectName":"{project}",
 "stackName":"{stack}",
 "displayName":"#some-slack-channel",
 "payloadUrl":"https://hooks.slack.com/services/...",
 "format": "slack",
 "active":true
 }' \
 https://api.pulumi.com/api/orgs/{organization}/{project}/{stack}/hooks

Pulumi Service Provider

The Pulumi Service provider allows you to create Pulumi Cloud resources via Pulumi. You can provision and manage webhooks, including Slack-formatted webhooks, using this provider.

import * as pulumi from "@pulumi/pulumi";
import { Webhook, WebhookFormat, WebhookFilters } from "@pulumi/pulumiservice";

const orgName = "my-org";

const orgWebhook = new Webhook("org-webhook", {
 active: true,
 displayName: "#my-org-activity-channel",
 organizationName: orgName,
 payloadUrl: "https://hooks.slack.com/services/...",
 format: WebhookFormat.Slack,
 filters: [
 WebhookFilters.StackCreated,
 WebhookFilters.StackDeleted,
 WebhookFilters.DeploymentSucceeded,
 WebhookFilters.DeploymentFailed,
 ],
});

const stackWebhook = new Webhook("stack-webhook", {
 active: true,
 displayName: "#my-stack-activity-channel",
 organizationName: orgName,
 projectName: pulumi.getProject(),
 stackName: pulumi.getStack(),
 payloadUrl: "https://hooks.slack.com/services/...",
 format: WebhookFormat.Slack,
});

export const orgWebhookName = orgWebhook.name;
export const stackWebhookName = stackWebhook.name;


name: webhook-examples
runtime: yaml
description: An example of provisioning a Pulumi Cloud Webhook via Pulumi Service Provider

resources:
 orgWebhook:
 type: pulumiservice:index:Webhook
 properties:
 active: true
 displayName: "#my-org-activity-channel"
 organizationName: my-org
 payloadUrl: "https://hooks.slack.com/services/..."
 format: slack
 filters:
 - stack_created
 - stack_deleted
 - deployment_succeeded
 - deployment_failed
 stackWebhook:
 type: pulumiservice:index:Webhook
 properties:
 active: true
 displayName: "#my-stack-activity-channel"
 organizationName: my-org
 projectName: webhook-examples
 stackName: dev
 payloadUrl: "https://hooks.slack.com/services/..."
 format: slack

outputs:
 # export the name of the webhooks
 orgWebhookName: ${orgWebhook.name}
 stackWebhookName: ${stackWebhook.name}

import pulumi
import pulumi_pulumiservice as pulumiservice

org_webhook = pulumiservice.Webhook("orgWebhook",
 active=True,
 display_name="#my-org-activity-channel",
 organization_name="my-org",
 payload_url="https://hooks.slack.com/services/...",
 format=pulumiservice.WebhookFormat.SLACK,
 filters=[
 pulumiservice.WebhookFilters.STACK_CREATED,
 pulumiservice.WebhookFilters.STACK_DELETED,
 pulumiservice.WebhookFilters.DEPLOYMENT_SUCCEEDED,
 pulumiservice.WebhookFilters.DEPLOYMENT_FAILED,
 ])

stack_webhook = pulumiservice.Webhook("stackWebhook",
 active=True,
 display_name="#my-stack-activity-channel",
 organization_name="my-org",
 project_name="webhook-examples",
 stack_name="dev",
 payload_url="https://hooks.slack.com/services/...",
 format=pulumiservice.WebhookFormat.SLACK)

pulumi.export("orgWebhookName", org_webhook.name)
pulumi.export("stackWebhookName", stack_webhook.name)

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 orgWebhook, err := pulumiservice.NewWebhook(ctx, "orgWebhook", &pulumiservice.WebhookArgs{
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("#my-org-activity-channel"),
 OrganizationName: pulumi.String("my-org"),
 PayloadUrl: pulumi.String("https://hooks.slack.com/services/..."),
 Format: pulumiservice.WebhookFormatSlack,
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersStackCreated,
 pulumiservice.WebhookFiltersStackDeleted,
 pulumiservice.WebhookFiltersDeploymentSucceeded,
 pulumiservice.WebhookFiltersDeploymentFailed,
 },
 })
 if err != nil {
 return err
 }
 stackWebhook, err := pulumiservice.NewWebhook(ctx, "stackWebhook", &pulumiservice.WebhookArgs{
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("#my-stack-activity-channel"),
 OrganizationName: pulumi.String("my-org"),
 ProjectName: pulumi.String("webhook-examples"),
 StackName: pulumi.String("dev"),
 PayloadUrl: pulumi.String("https://hooks.slack.com/services/..."),
 Format: pulumiservice.WebhookFormatSlack,
 })
 if err != nil {
 return err
 }
 ctx.Export("orgWebhookName", orgWebhook.Name)
 ctx.Export("stackWebhookName", stackWebhook.Name)
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using PulumiService = Pulumi.PulumiService;

return await Deployment.RunAsync(() =>
{
 var orgWebhook = new PulumiService.Webhook("orgWebhook", new()
 {
 Active = true,
 DisplayName = "#my-org-activity-channel",
 OrganizationName = "my-org",
 PayloadUrl = "https://hooks.slack.com/services/...",
 Format = PulumiService.WebhookFormat.Slack,
 Filters = new[]
 {
 PulumiService.WebhookFilters.StackCreated,
 PulumiService.WebhookFilters.StackDeleted,
 PulumiService.WebhookFilters.DeploymentSucceeded,
 PulumiService.WebhookFilters.DeploymentFailed,
 },
 });

 var stackWebhook = new PulumiService.Webhook("stackWebhook", new()
 {
 Active = true,
 DisplayName = "#my-stack-activity-channel",
 OrganizationName = "my-org",
 ProjectName = "webhook-examples",
 StackName = "dev",
 PayloadUrl = "https://hooks.slack.com/services/...",
 Format = PulumiService.WebhookFormat.Slack,
 });

 return new Dictionary<string, object?>
 {
 ["orgWebhookName"] = orgWebhook.Name,
 ["stackWebhookName"] = stackWebhook.Name,
 };
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.pulumiservice.Webhook;
import com.pulumi.pulumiservice.WebhookArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var orgWebhook = new Webhook("orgWebhook", WebhookArgs.builder()
 .active(true)
 .displayName("#my-org-activity-channel")
 .organizationName("my-org")
 .payloadUrl("https://hooks.slack.com/services/...")
 .format("slack")
 .filters(
 "stack_created",
 "stack_deleted",
 "deployment_succeeded",
 "deployment_failed")
 .build());

 var stackWebhook = new Webhook("stackWebhook", WebhookArgs.builder()
 .active(true)
 .displayName("#my-stack-activity-channel")
 .organizationName("my-org")
 .projectName("webhook-examples")
 .stackName("dev")
 .payloadUrl("https://hooks.slack.com/services/...")
 .format("slack")
 .build());

 ctx.export("orgWebhookName", orgWebhook.name());
 ctx.export("stackWebhookName", stackWebhook.name());
 }
}

Pulumi Deployment Notification Events

Pulumi Deployments is a fully managed platform for managing infrastructure at scale. With Pulumi Deployments, you can run a Pulumi infrastructure as code action (a preview, update, destroy or refresh) inside Pulumi Cloud. Pulumi provides scalability, observability and security for deployments. Both Slack-formatted and generic JSON webhooks in Pulumi Cloud now send notifications events on Pulumi Deployments statuses: when a deployment is queued, started, succeeds and fails.

Filtering Notification Events

Event filtering allows you to select which events should be delivered to each webhook. You may want to receive all events, or filter to specific events (only failures, only deployment events, etc.). The following table describes the various event filters available and the context in which they are relevant.

Filter	Event Kind	Webhook Type	Triggered
`stack_created`	`stack`	Organization webhooks only	When a stack is created.
`stack_deleted`	`stack`	Organization webhooks only	When a stack is deleted.
`preview_succeeded`	`stack_preview`	Both	When a stack `preview` succeeds.
`preview_failed`	`stack_preview`	Both	When a stack `preview` fails.
`update_succeeded`	`stack_update`	Both	When a stack `update` succeeds.
`update_failed`	`stack_update`	Both	When a stack `update` fails.
`destroy_succeeded`	`stack_update`	Both	When a stack `destroy` succeeds.
`destroy_failed`	`stack_update`	Both	When a stack `destroy` fails.
`refresh_succeeded`	`stack_update`	Both	When a stack `refresh` succeeds.
`refresh failed`	`stack_update`	Both	When a stack `refresh` fails.
`deployment_queued`	`deployment`	Both	When a deployment is queued.
`deployment_started`	`deployment`	Both	When a deployment starts running.
`deployment_succeeded`	`deployment`	Both	When a deployment succeeds.
`deployment_failed`	`deployment`	Both	When a deployment fails.

Wrapping up

We hope you and your team can streamline deployment related notifications to where you spend your time. Our webhooks improvements enable ChatOps workflows and more visibility into your infrastructure. As always, please let us know if you have feedback on the feature by opening an issue in the Pulumi Cloud requests repository or if you have other features you would like to see in Pulumi Cloud.

Happy building! 👷

Pulumi Deployments: API-Driven Infrastructure at Scale

Evan Boyle — Wed, 29 Mar 2023 00:00:00 +0000

Delivering software has been conventionally driven by CI/CD workflows. A single commit is merged into a codebase, and a small, and static set of workflow runs are triggered by the CI system to update the appropriate environments (Dev -> Staging -> Production). This may have been fine when the only product a company had to offer was a single stateless web service, but increasingly companies are called upon to deliver cloud infrastructure as a product.

There are obvious forms of this, such as IaaS companies building database-as-a-service offerings and SaaS companies deploying self-hosted installations into customer environments. But there are less apparent forms like internal platform teams that are chartered with enabling self-service of Kubernetes clusters configured with best practices to thousands of application engineers within an enterprise. In any of these cases, engineers are delivering infrastructure as the product but only have conventional tools like gitops and CI/CD to reach for.

The assumptions that CI/CD systems make are limiting for infrastructure. We are not pushing a single application commit across three environments. We are templating infrastructure hundreds and thousands of times. Once that infrastructure is templated, we’re responsible for ensuring it is secure, up-to-date, and can be customized by self-service users. CI/CD is a poor fit due to the design choices that are locked in around source control and source events. CI/CD is typically configured via templated YAML that is checked into source control, with no way to create or customize behavior dynamically from a REST API. And beyond configuration, there aren’t REST APIs for triggering updates and workflow runs. The best you can expect is the ability to add more YAML to configure a cron job. What do you do if you need to create a self-service platform, or configure CI/CD across thousands of projects across your organization?

Today we’re excited to announce the release of a new version of Pulumi Deployments, a fully managed platform for shipping infrastructure at scale. It provides:

Compute - run infrastructure deployments and workloads (i.e. pulumi up) on fully managed, secure, and elastic compute.
Configuration - A simple, centralized format for specifying everything you need to deploy your infrastructure at the stack level. Source code, cloud credentials, OpenID Connect (OIDC), environment variables, build prerequisites, and dependencies are bundled in an atomic unit.
Composition - Trigger deployments via our REST API, click to deploy in the Pulumi Service UI, Git Push to Deploy, Remote Automation API, and other programmatic building blocks that you can compose to build custom cloud automation.

Pulumi Deployments is built on top of a powerful set of public REST APIs. Unlike any other platform, Deployments lets you configure CI/CD settings programmatically. Service-catalog scenarios and guard rails that enforce CI/CD best practices across thousands of projects are far easier to achieve when you have REST APIs that you can build software on top of. No more generating and templating YAML to configure infrastructure deployments at scale.

Deployment Settings and Triggers

Deployments can be configured at a stack level via Deployment Settings in three ways:

The Pulumi Service UI enables setting up OIDC and git push to deploy in just a few clicks
The Pulumi Service Provider lets you define deployment configuration as Pulumi code in you language of choice, reviewed and managed in source control.
The Deployment Settings REST API provides programmatic access to configure Deployments and CI/CD dynamically, without needing to generate or check in YAML.

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();

// deployment settings configured via a pulumi program
const settings = new service.DeploymentSettings("deployment_settings", {
 organization: "service-provider-test-org",
 project: "test-deployment-settings-proj",
 stack: "dev",
 operationContext: {
 environmentVariables: {
 TEST_VAR: "foo",
 SECRET_VAR: config.requireSecret("my_secret"),
 }
 },
 sourceContext: {
 git: {
 repoUrl: "https://github.com/pulumi/deploy-demos.git",
 branch: "refs/heads/main",
 repoDir: "pulumi-programs/simple-resource"
 }
 }
});

Each stack configured with Pulumi Deployments gets access to a number of triggers that can be used to start a Pulumi operation (i.e. pulumi up):

Deployments REST API - an endpoint available at api.pulumi.com that can trigger a variety of pulumi (up/destroy/refresh/preview) operations. Calls to this trigger can override individual settings to customize deployment behavior (i.e. - deploy a different branch, or override a particular environment variable).
git push to Deploy - Deep integration with the Pulumi GitHub App that enables automatic previews on every pull request, and an automated update whenever code is pushed to the target branch.
Click to Deploy - trigger a deployment from the Pulumi Service Console with the push of a button. Very useful for day-to-day operational activities such as debugging a stuck stack, or rectifying drift via a refresh without having to pull a stack and its accompanying source code onto your machine.
Remote Automation API - Your favorite strongly typed Pulumi orchestration tool, Automation API, lets you offload update workloads into the cloud. Write deployment scripts that update thousands of environments simultaneously without worrying about compute.

You can mix, match, and compose Deployment Settings and Triggers to create a golden path to production that is perfect for your enterprise’s needs.

Example: Building RESTful Infrastructure APIs

Increasingly, infrastructure updates happen in response to user action and not source control events like a git commit. For instance, a SaaS database company might need to spin up dedicated compute and storage when a customer puts in a credit card and self-serves onto a specific SKU. This signup workflow might happen hundreds or even thousands of times an hour. Traditional deployment systems are optimized for infrastructure that spans, at most, a handful of environments (dev/staging/production).

Leading SaaS and infrastructure companies want the best of both worlds:

Custom, domain-specific REST APIs to create infrastructure on demand.
Desired state configuration that keeps track of what is deployed, and where.

You can build on top of the Deployments REST API to build your own infrastructure APIs. Here we have a simple Go web server that offers a RESTful interface over top of a static website. Users can create, update, and delete static websites, specifying the content for the site in the POST payload. Here’s our web server:

 router := httprouter.New()
 router.POST("/sites", server.create)
 router.GET("/sites/:id", server.get)
 router.POST("/sites/:id", server.update)
 router.DELETE("/sites/:id", server.delete)

 http.ListenAndServe(*addr, router)

Within the create handler, we first dynamically configure Deployment Settings for the stack, including the branch of the Pulumi program to be deployed, the AWS OIDC provider used for cloud credentials, and the configuration of git push Deployment Triggers`:

 err := s.client.patchDeploymentSettings(r.Context(), s.org, s.project, stack, DeploymentSettings{
 SourceContext: &sourceContext{
 Git: gitContext{
 Branch: s.branch,
 RepoDir: s.dir,
 },
 },
 OperationContext: &operationContext{
 Environment: map[string]string{
 "AWS_REGION": s.region,
 },
 OIDC: &oidcContext{
 AWS: &awsOIDCContext{
 RoleARN: s.roleARN,
 SessionName: s.sessionName,
 },
 },
 },
 GitHub: &gitHubContext{
 Repository: s.repository,
 Paths: paths,
 DeployCommits: true,
 PreviewPullRequests: false,
 },
 })

These settings configure the stack to trigger an update when new commits are pushed to the branch. In addition, our create handler also kicks off the initial update to create the infrastructure:

err = s.client.createDeployment(ctx, s.org, s.project, stack, createDeploymentRequest{
 DeploymentSettings: DeploymentSettings{
 OperationContext: &operationContext{
 Environment: map[string]string{
 "SITE_CONTENT": content,
 },
 },
 },
 InheritSettings: true,
 Operation: "update",
 })

Notice how this REST API call specifies InheritSettings: true so that Deployment Settings are read from the Pulumi Service and merged with the incoming request payload to create deployment configuration for this run. This stack also has to Click to Deploy via the Pulumi Service UI, so operational tasks like refreshing the stack can be done on demand without pulling down source code onto your local machine.

The Deployments Platform does all of the heavy lifting of managing deployment compute, providing asynchronous workflow orchestration, queueing, status and logging API access, and more. At the end, you get a Pulumi Stack and state file containing a manifest of all cloud resources and their properties managed by the Pulumi Service.

Deployment Settings and Triggers can be combined to ship infrastructure in novel ways with just a few lines of code. See the full RESTful infrastructure API example to try it out yourself.

Pave your Golden Path Today

Pulumi Deployments is the platform for shipping infrastructure at scale. It gives you the programmatic building blocks you need to deliver infrastructure on demand in a large organization. Get started with Deployments today. Happy automating!

Announcing Cross-Language Enum Support

Komal Ali — Tue, 15 Dec 2020 00:00:00 +0000

Here at Pulumi, we believe in leveraging the best features of programming languages to create a delightful development experience for our users. Today, we continue our contributions in this area by announcing cross-language support for enum types in our provider SDKs, available in all Pulumi languages - Python, TypeScript, .NET and Go.

Enum types explained

An enum (short for enumerated) type is a data type consisting of a set of named values. In the Pulumi resource model, a property is an enum type when its value can only be set to a finite number of predetermined values.

For instance, consider a simple S3 bucket.

import * as pulumi from "@pulumi/pulumi";
import { s3 } from "@pulumi/aws";

const myBucket = new s3.Bucket("myBucket", { acl: "private" });

import pulumi
from pulumi_aws import s3

my_bucket = s3.Bucket("myBucket", acl="private")

using Pulumi;
using S3 = Pulumi.Aws.S3;

class MyStack : Stack
{
 public MyStack()
 {
 var myBucket = new S3.Bucket("myBucket", new S3.BucketArgs
 {
 Acl = "private",
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v3/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 myBucket, err := s3.NewBucket(ctx, "myBucket", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

In the above code, the S3 Bucket resource has a property called acl, where we pass in the string private to indicate that this is a private bucket.

If we look at the resource docs, we can see that the acl property can only be set to one of a few different values: private, public-read, public-read-write, aws-exec-read, authenticated-read, and log-delivery-write. The acl property is the perfect candidate for an enum type and is emitted as one, so you can use the following code instead.

import * as pulumi from "@pulumi/pulumi";
import { s3 } from "@pulumi/aws";

const myBucket = new s3.Bucket("myBucket", { acl: s3.CannedAcl.Private });

import pulumi
from pulumi_aws import s3

my_bucket = s3.Bucket("myBucket", acl=s3.CannedAcl.PRIVATE)

using Pulumi;
using S3 = Pulumi.Aws.S3;

class MyStack : Stack
{
 public MyStack()
 {
 var myBucket = new S3.Bucket("myBucket", new S3.BucketArgs
 {
 Acl = S3.CannedAcl.Private,
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v3/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 myBucket, err := s3.NewBucket(ctx, "myBucket", &s3.BucketArgs{
 Acl: s3.CannedAclPrivate,
 })
 if err != nil {
 return err
 }
 return nil
 })
}

IDE Superpowers

Enum properties provide discoverable and normalized constants that can be used in place of raw strings, allowing you to move faster with the added boosts of IDE tooling like type hints and autocomplete.

Using the provided constants, you can avoid referring back to the documentation to remember the valid values and save precious moments in the development cycle that would be lost to debugging errors caused by typos.

Optimized for flexibility

While some properties make sense as “strict” enums (i.e., the input value must be one of the enumerated values), there are times when it would be beneficial to have constants for commonly used values without restricting the input to only those values. We call this second category “relaxed” enums, and we model them a little differently in our SDKs.

“Strict” enums

A property is a “strict” enum when the input value must be one of the enumerated values. In this case, the property type is specified as the enum type.

We will use “strict” enums when we are sure that the enum will include all legal values, such as when a provider is auto-generated from a cloud provider specification (like our Azure-Native or Kubernetes providers).

export class StorageAccount extends pulumi.CustomResource {
 constructor(name: string, args: StorageAccountArgs, opts?: pulumi.CustomResourceOptions) {...}
}

export interface StorageAccountArgs {
 readonly accessTier?: pulumi.Input<enums.storage.latest.AccessTier>;
 ...
}

class StorageAccount(pulumi.CustomResource):
 def __init__(__self__,
 resource_name: str,
 access_tier: Optional[pulumi.Input['AccessTier']] = None,
 ...):
 ...

namespace Pulumi.AzureNative.Storage
{
 public partial class StorageAccount : Pulumi.CustomResource
 {
 public StorageAccount(string name, StorageAccountArgs args, ...)
 {
 }
 }

 public sealed class StorageAccountArgs : Pulumi.ResourceArgs
 {
 [Input("accessTier")]
 public Input<Pulumi.AzureNative.Storage.AccessTier>? AccessTier { get; set; }

 ...
 }
}

func NewStorageAccount(ctx *pulumi.Context,
name string, args *StorageAccountArgs, ...) (*StorageAccount, error) {
 ...
}

type StorageAccountArgs struct {
 AccessTier AccessTier
 ...
}

When using TypeScript, “strict” enums will accept both the constant (AccessTier.Cool) or the literal string ("Cool"). If there is an accidental spelling error in the literal, you are immediately alerted to the issue rather than having to wait until runtime.

“Relaxed” enums

When a property is a “relaxed” enum, the property type is specified as the Union of the enum type and the underlying primitive type. This means that you have the convenience of using the enum constants (i.e., s3.CannedAcl.Private), but you may also pass in the raw string (i.e., "private").

In the AWS provider (and other Terraform-based providers), we have opted for only using “relaxed” enums. The reasoning is twofold. Allowing the primitive type maintains backward compatibility and also allows users to use values that may not yet be represented in the Pulumi schema (e.g., a new Managed Policy ARN, EC2 Instance type, etc.).

export class Bucket extends pulumi.CustomResource {
 constructor(name: string, args?: BucketArgs, opts?: pulumi.CustomResourceOptions) {...}
}

export interface BucketArgs {
 readonly acl?: pulumi.Input<string | enums.s3.CannedAcl>;
 ...
}

class Bucket(pulumi.CustomResource):
 def __init__(__self__,
 resource_name: str,
 acl: Optional[pulumi.Input[Union[str, 'CannedAcl']]] = None,
 ...):
 ...

namespace Pulumi.Aws.S3
{
 public partial class Bucket : Pulumi.CustomResource
 {
 public Bucket(string name, BucketArgs? args = null, ...)
 {
 }
 }

 public sealed class BucketArgs : Pulumi.ResourceArgs
 {
 [Input("acl")]
 public InputUnion<string, Pulumi.Aws.S3.CannedAcl>? Acl { get; set; }

 ...
 }
}

func NewBucket(ctx *pulumi.Context,
name string, args *BucketArgs, ...) (*Bucket, error) {
 ...
}

// In Go, relaxed enums retain the type of the underlying primitive.
// However, the CannedAcl constants may still be passed in, as they are
// of the same underlying type.
type BucketArgs struct {
 Acl pulumi.StringPtrInput
 ...
}

Try them out!

You can find enum types integrated into v3.19.0 of the AWS provider and v0.7.1 of the Azure-Native provider, and we will be adding enums to other providers in the coming weeks and months.

Azure-Native: In the native Azure provider, all properties labeled as enums in the OpenAPI spec are represented as such. In all, there are over 1300 enums provided in the SDKs. The native Azure provider uses both “strict” and “relaxed” enums since the OpenAPI specification explicitly defines its properties as such.

AWS: The AWS provider enums are manually identified and maintained as part of the provider schema. We’ve already added many that you might find useful, such as Lambda Runtimes, EC2 Instance Types, and IAM Managed Policies, and will continue to add more in the coming months.

Take the new providers for a spin and let us know what you think in the Community Slack! If there are properties that you would like to see represented as enums, let us know. Or even better, submit a PR like this one to add them to the schema!