Pulumi Blog: Deployments

Dependent Stack Updates with Pulumi Deployments

Evan Boyle — Tue, 20 Jun 2023 00:00:00 +0000

As infrastructure projects grow in size and complexity, you need to decompose infrastructure into smaller stacks to limit the blast radius of errors, extract and reference common layers like networking, and limit access to sensitive components. This comes with a coordination cost as you now need to figure out how to detect and propagate changes to downstream stacks in your dependency graph. Today we’re announcing two features that can help you manage this complexity by automatically updating dependent stacks:

Deployment Webhook Destinations for triggering downstream stack updates automatically
The Pulumi Auto Deploy Package for easily expressing dependencies and configuring downstream updates via Deployment Webhooks.

The following example shows automatic deployment of stacks with the following dependency graph
a
├── b
│ ├── d
│ ├── e
│ └── f
└── c
Whenever a node in the graph is updated, all downstream nodes will be automatically updated
via a webhook triggering Pulumi Deployments.

Automatically updating dependent stacks and providing a way to write Pulumi programs that can express nested stack updates are two of the most requested features across the Pulumi project, and we’re excited to enable both of these via Pulumi Cloud and Pulumi Deployments.

Deployment Webhook Destinations

Pulumi Webhooks allow you to pick one or more event types on a stack (i.e. update succeeded, or refresh failed) and automatically deliver an event to a destination such a slack channel or a webserver you’ve deployed with custom code. Now you can choose Pulumi Deployments as a webhook destination. When an update completes on the parent stack, a deployment will automatically be triggered on the destination stack. This makes it easy to keep dependent stacks up to date. For instance, you can use Deployment Webhook Destinations to configure every successful update on your networking stack to trigger downstream updates on your database and compute stacks.

import * as pulumi from "@pulumi/pulumi";
import * as pulumiservice from "@pulumi/pulumiservice";

const databaseWebhook = new pulumiservice.Webhook("databaseWebhook", {
 organizationName: "org",
 projectName: "network",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "database/prod",
 active: true,
 displayName: "deploy-database",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});
const computeWebhook = new pulumiservice.Webhook("computeWebhook", {
 organizationName: "org",
 projectName: "database",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "compute/prod",
 active: true,
 displayName: "deploy-compute",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using PulumiService = Pulumi.PulumiService;

return await Deployment.RunAsync(() =>
{
 var databaseWebhook = new PulumiService.Webhook("databaseWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "network",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "database/prod",
 Active = true,
 DisplayName = "deploy-database",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

 var computeWebhook = new PulumiService.Webhook("computeWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "database",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "compute/prod",
 Active = true,
 DisplayName = "deploy-compute",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

});

import pulumi
import pulumi_pulumiservice as pulumiservice

database_webhook = pulumiservice.Webhook("databaseWebhook",
 organization_name="org",
 project_name="network",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="database/prod",
 active=True,
 display_name="deploy-database",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])
compute_webhook = pulumiservice.Webhook("computeWebhook",
 organization_name="org",
 project_name="database",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="compute/prod",
 active=True,
 display_name="deploy-compute",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := pulumiservice.NewWebhook(ctx, "databaseWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("network"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("database/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-database"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 _, err = pulumiservice.NewWebhook(ctx, "computeWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("database"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("compute/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-compute"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
resources:
 databaseWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: network
 stackName: prod
 format: pulumi_deployments
 payloadUrl: database/prod
 active: true
 displayName: deploy-database
 filters:
 - update_succeeded
 computeWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: database
 stackName: prod
 format: pulumi_deployments
 payloadUrl: compute/prod
 active: true
 displayName: deploy-compute
 filters:
 - update_succeeded

Expressing Stack Dependencies with Pulumi Auto Deploy

We’ve also shipped a new Pulumi package called Pulumi Auto Deploy. It lets you simply express dependencies between stacks, and takes care of creating and updating the necessary Deployment Webhooks under the hood.

import * as autodeploy from "@pulumi/auto-deploy";
import * as pulumi from "@pulumi/pulumi";

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

const organization = pulumi.getOrganization();
const project = "dependency-example"

export const f = new autodeploy.AutoDeployer("auto-deployer-f", {
 organization,
 project,
 stack: "f",
 downstreamRefs: [],
});

export const e = new autodeploy.AutoDeployer("auto-deployer-e", {
 organization,
 project,
 stack: "e",
 downstreamRefs: [],
});

export const d = new autodeploy.AutoDeployer("auto-deployer-d", {
 organization,
 project,
 stack: "d",
 downstreamRefs: [],
});

export const c = new autodeploy.AutoDeployer("auto-deployer-c", {
 organization,
 project,
 stack: "c",
 downstreamRefs: [],
});

export const b = new autodeploy.AutoDeployer("auto-deployer-b", {
 organization,
 project,
 stack: "b",
 downstreamRefs: [d.ref, e.ref, f.ref],

});

export const a = new autodeploy.AutoDeployer("auto-deployer-a", {
 organization,
 project,
 stack: "a",
 downstreamRefs: [b.ref, c.ref],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AutoDeploy = Pulumi.AutoDeploy;

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

return await Deployment.RunAsync(() =>
{
 var projectVar = "dependency-example";

 var organization = "pulumi";

 var f = new AutoDeploy.AutoDeployer("f", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "f",
 DownstreamRefs = new[] {},
 });

 var e = new AutoDeploy.AutoDeployer("e", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "e",
 DownstreamRefs = new[] {},
 });

 var d = new AutoDeploy.AutoDeployer("d", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "d",
 DownstreamRefs = new[] {},
 });

 var c = new AutoDeploy.AutoDeployer("c", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "c",
 DownstreamRefs = new[] {},
 });

 var b = new AutoDeploy.AutoDeployer("b", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "b",
 DownstreamRefs = new[]
 {
 d.Ref,
 e.Ref,
 f.Ref,
 },
 });

 var a = new AutoDeploy.AutoDeployer("a", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "a",
 DownstreamRefs = new[]
 {
 b.Ref,
 c.Ref,
 },
 });

});

import pulumi
import pulumi_auto_deploy as auto_deploy

'''

The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
Whenever a node in the graph is updated,
all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
'''

project_var = "dependency-example"
organization = pulumi.get_organization()
f = auto_deploy.AutoDeployer("f",
 organization=organization,
 project=project_var,
 stack="f",
 downstream_refs=[])
e = auto_deploy.AutoDeployer("e",
 organization=organization,
 project=project_var,
 stack="e",
 downstream_refs=[])
d = auto_deploy.AutoDeployer("d",
 organization=organization,
 project=project_var,
 stack="d",
 downstream_refs=[])
c = auto_deploy.AutoDeployer("c",
 organization=organization,
 project=project_var,
 stack="c",
 downstream_refs=[])
b = auto_deploy.AutoDeployer("b",
 organization=organization,
 project=project_var,
 stack="b",
 downstream_refs=[
 d.ref,
 e.ref,
 f.ref,
 ])
a = auto_deploy.AutoDeployer("a",
 organization=organization,
 project=project_var,
 stack="a",
 downstream_refs=[
 b.ref,
 c.ref,
 ])

package main

import (
 "github.com/pulumi/pulumi-auto-deploy/sdk/go/autodeploy"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 /**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

 pulumi.Run(func(ctx *pulumi.Context) error {
 projectVar := "dependency-example"
 organization := "pulumi"
 f, err := autodeploy.NewAutoDeployer(ctx, "f", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("f"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 e, err := autodeploy.NewAutoDeployer(ctx, "e", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("e"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 d, err := autodeploy.NewAutoDeployer(ctx, "d", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("d"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 c, err := autodeploy.NewAutoDeployer(ctx, "c", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("c"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 b, err := autodeploy.NewAutoDeployer(ctx, "b", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("b"),
 DownstreamRefs: pulumi.StringArray{
 d.Ref,
 e.Ref,
 f.Ref,
 },
 })
 if err != nil {
 return err
 }
 _, err = autodeploy.NewAutoDeployer(ctx, "a", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("a"),
 DownstreamRefs: pulumi.StringArray{
 b.Ref,
 c.Ref,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
variables:
 project: dependency-example
 # TODO: update once https://github.com/pulumi/pulumi-yaml/issues/461 is fixed
 organization: pulumi
resources:
 f:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: f
 downstreamRefs: []
 e:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: e
 downstreamRefs: []
 d:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: d
 downstreamRefs: []
 c:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: c
 downstreamRefs: []
 b:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: b
 downstreamRefs:
 - ${d.ref}
 - ${e.ref}
 - ${f.ref}
 a:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: a
 downstreamRefs:
 - ${b.ref}
 - ${c.ref}

Just declare an AutoDeployer resource for each stack, and any downstream stacks will automatically be updated whenever a stack in its dependency chain is updated.

Automate your deployment pipeline

Use Pulumi Deployments to trigger downstream stack updates automatically when an upstream stack changes. Keep your dependent infrastructure in sync without manual runbooks.

Get started

Scale Infrastructure with Software

At Pulumi, we build tools to help you scale your infrastructure footprint with software instead of just humans. Deployment Webhook Destinations and pulumi-auto-deploy give you the tools to keep all of your dependent infrastructure up to date with automation instead of manual runbooks and human-driven operations.

Check out these resources to get started today:

Check out the Pulumi Deployments documentation and learn about configuring Deployment Settings for your stacks.
Get started with the Pulumi Auto Deploy Package.
Learn more about Pulumi Cloud Webhooks.
Join us in the #pulumi-deployments community slack channel.

🚀🚀🚀 Happy building! 🚀🚀🚀

Review Stacks: Collaborate in the Cloud

Evan Boyle — Wed, 14 Jun 2023 00:00:00 +0000

Today we’re excited to announce Review Stacks – dedicated cloud environments that get created automatically every time a pull request is opened, all powered by Pulumi Deployments. Open a pull request, and Pulumi Deployments will stand up a stack with your changes and the Pulumi GitHub App will add a PR comment with the outputs from your deployment. Merge the PR and Pulumi Deployments will destroy the stack and free up the associated resources. It has never been simpler to pick up an unfamiliar codebase, make changes to both application and infrastructure code, and share a live environment for review with your teammates.

Ephemeral environments for front-end development and static websites have become commonplace in recent years, but infrastructure and complex systems have been unsupported up until this point. Review Stacks enable you to iterate on both application code changes and infrastructure code changes at the same time. Making changes to HTML, CSS, Next.js, Go APIs, Pulumi configuration, subnet configuration, databases, and load balancers are all possible with Review Stacks. No more fumbling with out-of-date READMEs and developer documentation. Just open a pull request and you can start testing changes against everything from simple static websites to API servers, microservices, data pipelines, Kubernetes clusters, and any other piece of infrastructure across Pulumi’s 100+ cloud providers.

Review Stacks manage the full lifecycle of your cloud development environment including creating it when the PR is opened, updating it every time a new commit is pushed, and destroying and cleaning up all cloud resources when the pull request is merged or closed. They even integrate with GitHub checks so that you can gate your pull requests on successful Review Stack deployments.

Pulumi customers already using Review Stacks find that it saves their developers time. Having a standardized process for development environments and change management makes it easier to onboard new developers, and load-balance teammates onto unfamiliar projects and codebases. It also makes it substantially easier for developers to work on multiple feature branches and bug fixes at the same time, as they don’t have to worry about manually maintaining and juggling development environments.

“Pulumi allows every engineer to build and update infrastructure instead of only our infrastructure team, which enables us to ship faster than if we were using a domain-specific language,” said Aaron Gibralter, head of engineering, Mighty. “With Pulumi Deployments, we could easily set up a standardized CI/CD infrastructure workflow for every engineer that immediately makes them productive. Review Stacks automates the work of managing our own ephemeral infrastructure and makes our lives easier.”

Review Stacks also provide an avenue for optimizing cloud costs. Rather than keeping development infrastructure running 24/7, Review Stacks only exist as long as an engineer works on a feature. Since Review Stacks are flexible and built on Pulumi IaC, it is also possible to configure it to deploy to multi-tenant environments that share development infrastructure to further streamline costs.

“At GitHub, we are constantly seeking to improve the developer experience and our partner ecosystem plays a big role in that.” said Matthew Manning, Senior Business Development Manager for GitHub Actions, “The launch of Pulumi Review Stacks with GitHub makes it easier for teams to review and test their products before shipping which ultimately improves the developer experience.”

Configuring Review Stacks

Configuring Review Stacks is a simple three-step process:

Create a new stack, by convention named pr, and corresponding Pulumi.pr.yaml configuration file - this config will be copied into every review stack that gets created, and can even be modified within a PR.
Configure Deployment Settings for the stack - this specifies how to acquire source code, cloud credentials and more when deploying via Pulumi Deployments.
Set the pullRequestTemplate Deployment Setting to true - this indicates that all pull requests against this stack’s branch should reference this stack as a Review Stack template.

You can use an existing stack as a Review Stack template, as long as it has Deployment Settings configured. This will result in Review Stacks being deployed into the same cloud account. If you want to separate the cloud resources in your production stack from the resources created via Review Stacks then you can create a separate stack and template that references a different cloud account (AWS, Azure, GCP, etc).

Review Stacks and Deployment Settings can be configured via the Pulumi Cloud console, the Pulumi Cloud REST API, or within a Pulumi Program using the Pulumi Cloud Resource Provider.

Pulumi Cloud UI

It is just one click to turn on Review Stacks via the Pulumi Cloud console.

REST API

You can programmatically configure Review Stacks and Deployment Settings at scale across thousands of projects using the Deployments REST API.

curl -i -XPOST -H "Content-Type: application/json" -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
--location "https://api.pulumi.com/api/stacks/org/project/stack/deployments/settings" \
-d '{
"gitHub":{
"pullRequestTemplate": true
}
}'

Pulumi Cloud Resource Provider

You can use Pulumi to manage and code review Deployment Settings and Review Stacks with the Pulumi Cloud Provider.

import * as pulumiservice from "@pulumi/pulumiservice";

const deploymentSettings = new pulumiservice.DeploymentSettings("deploymentSettings", {
 organization: pulumi.getOrganization(),
 project: "your project",
 stack: "your stack",
 github: {
 deployCommits: true,
 previewPullRequests: true,
 pullRequestTemplate: true,
 repository: "pulumi/deployment-automation",
 },
 sourceContext: {
 git: {
 branch: "refs/heads/main",
 repoDir: "pulumi-pet-shop",
 },
 },
});

Collaborate in the Cloud

At Pulumi, our engineering team writes a ton of code. We have hundreds of repositories with programs managing thousands of stacks and tens of thousands of cloud resources. These tools and services power production workloads, internal tools, business intelligence and process, and more. And of course we use Pulumi to ship all of it. When you’re an engineer working in one of these repositories, making a code change is the easy part. The hard part is figuring out how to get a tight dev loop going—in particular, faithfully verifying your changes before merging and deploying them. Are there instructions in the README for running locally? A Docker Compose file? Or maybe standalone dev environments? Can I figure out what config and upstream stack references are required to stand up a fresh dev stack? And what about service dependencies? Pulumi makes it incredibly easy to build in the cloud and write prolific amounts of code, however, with that increase in development velocity comes a need to standardize the entire process by which we develop, test, review, and ship our code.

We’ve heard similar stories from hundreds of customers. From startup teams of five to enterprises of thousands, they’re all looking for the same thing - so we went ahead and built it. With Review Stacks and Pulumi Deployments, you can ship with greater velocity, confidence, and collaboration.

Check out these resources and get started today:

Review Stacks documentation
Learn more about Pulumi Deployments
Join us in the #pulumi-deployments community slack channel

🚀🚀🚀 Happy building! 🚀🚀🚀

Pulumi Deployments: API-Driven Infrastructure at Scale

Evan Boyle — Wed, 29 Mar 2023 00:00:00 +0000

Delivering software has been conventionally driven by CI/CD workflows. A single commit is merged into a codebase, and a small, and static set of workflow runs are triggered by the CI system to update the appropriate environments (Dev -> Staging -> Production). This may have been fine when the only product a company had to offer was a single stateless web service, but increasingly companies are called upon to deliver cloud infrastructure as a product.

There are obvious forms of this, such as IaaS companies building database-as-a-service offerings and SaaS companies deploying self-hosted installations into customer environments. But there are less apparent forms like internal platform teams that are chartered with enabling self-service of Kubernetes clusters configured with best practices to thousands of application engineers within an enterprise. In any of these cases, engineers are delivering infrastructure as the product but only have conventional tools like gitops and CI/CD to reach for.

The assumptions that CI/CD systems make are limiting for infrastructure. We are not pushing a single application commit across three environments. We are templating infrastructure hundreds and thousands of times. Once that infrastructure is templated, we’re responsible for ensuring it is secure, up-to-date, and can be customized by self-service users. CI/CD is a poor fit due to the design choices that are locked in around source control and source events. CI/CD is typically configured via templated YAML that is checked into source control, with no way to create or customize behavior dynamically from a REST API. And beyond configuration, there aren’t REST APIs for triggering updates and workflow runs. The best you can expect is the ability to add more YAML to configure a cron job. What do you do if you need to create a self-service platform, or configure CI/CD across thousands of projects across your organization?

Today we’re excited to announce the release of a new version of Pulumi Deployments, a fully managed platform for shipping infrastructure at scale. It provides:

Compute - run infrastructure deployments and workloads (i.e. pulumi up) on fully managed, secure, and elastic compute.
Configuration - A simple, centralized format for specifying everything you need to deploy your infrastructure at the stack level. Source code, cloud credentials, OpenID Connect (OIDC), environment variables, build prerequisites, and dependencies are bundled in an atomic unit.
Composition - Trigger deployments via our REST API, click to deploy in the Pulumi Service UI, Git Push to Deploy, Remote Automation API, and other programmatic building blocks that you can compose to build custom cloud automation.

Pulumi Deployments is built on top of a powerful set of public REST APIs. Unlike any other platform, Deployments lets you configure CI/CD settings programmatically. Service-catalog scenarios and guard rails that enforce CI/CD best practices across thousands of projects are far easier to achieve when you have REST APIs that you can build software on top of. No more generating and templating YAML to configure infrastructure deployments at scale.

Deployment Settings and Triggers

Deployments can be configured at a stack level via Deployment Settings in three ways:

The Pulumi Service UI enables setting up OIDC and git push to deploy in just a few clicks
The Pulumi Service Provider lets you define deployment configuration as Pulumi code in you language of choice, reviewed and managed in source control.
The Deployment Settings REST API provides programmatic access to configure Deployments and CI/CD dynamically, without needing to generate or check in YAML.

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();

// deployment settings configured via a pulumi program
const settings = new service.DeploymentSettings("deployment_settings", {
 organization: "service-provider-test-org",
 project: "test-deployment-settings-proj",
 stack: "dev",
 operationContext: {
 environmentVariables: {
 TEST_VAR: "foo",
 SECRET_VAR: config.requireSecret("my_secret"),
 }
 },
 sourceContext: {
 git: {
 repoUrl: "https://github.com/pulumi/deploy-demos.git",
 branch: "refs/heads/main",
 repoDir: "pulumi-programs/simple-resource"
 }
 }
});

Each stack configured with Pulumi Deployments gets access to a number of triggers that can be used to start a Pulumi operation (i.e. pulumi up):

Deployments REST API - an endpoint available at api.pulumi.com that can trigger a variety of pulumi (up/destroy/refresh/preview) operations. Calls to this trigger can override individual settings to customize deployment behavior (i.e. - deploy a different branch, or override a particular environment variable).
git push to Deploy - Deep integration with the Pulumi GitHub App that enables automatic previews on every pull request, and an automated update whenever code is pushed to the target branch.
Click to Deploy - trigger a deployment from the Pulumi Service Console with the push of a button. Very useful for day-to-day operational activities such as debugging a stuck stack, or rectifying drift via a refresh without having to pull a stack and its accompanying source code onto your machine.
Remote Automation API - Your favorite strongly typed Pulumi orchestration tool, Automation API, lets you offload update workloads into the cloud. Write deployment scripts that update thousands of environments simultaneously without worrying about compute.

You can mix, match, and compose Deployment Settings and Triggers to create a golden path to production that is perfect for your enterprise’s needs.

Example: Building RESTful Infrastructure APIs

Increasingly, infrastructure updates happen in response to user action and not source control events like a git commit. For instance, a SaaS database company might need to spin up dedicated compute and storage when a customer puts in a credit card and self-serves onto a specific SKU. This signup workflow might happen hundreds or even thousands of times an hour. Traditional deployment systems are optimized for infrastructure that spans, at most, a handful of environments (dev/staging/production).

Leading SaaS and infrastructure companies want the best of both worlds:

Custom, domain-specific REST APIs to create infrastructure on demand.
Desired state configuration that keeps track of what is deployed, and where.

You can build on top of the Deployments REST API to build your own infrastructure APIs. Here we have a simple Go web server that offers a RESTful interface over top of a static website. Users can create, update, and delete static websites, specifying the content for the site in the POST payload. Here’s our web server:

 router := httprouter.New()
 router.POST("/sites", server.create)
 router.GET("/sites/:id", server.get)
 router.POST("/sites/:id", server.update)
 router.DELETE("/sites/:id", server.delete)

 http.ListenAndServe(*addr, router)

Within the create handler, we first dynamically configure Deployment Settings for the stack, including the branch of the Pulumi program to be deployed, the AWS OIDC provider used for cloud credentials, and the configuration of git push Deployment Triggers`:

 err := s.client.patchDeploymentSettings(r.Context(), s.org, s.project, stack, DeploymentSettings{
 SourceContext: &sourceContext{
 Git: gitContext{
 Branch: s.branch,
 RepoDir: s.dir,
 },
 },
 OperationContext: &operationContext{
 Environment: map[string]string{
 "AWS_REGION": s.region,
 },
 OIDC: &oidcContext{
 AWS: &awsOIDCContext{
 RoleARN: s.roleARN,
 SessionName: s.sessionName,
 },
 },
 },
 GitHub: &gitHubContext{
 Repository: s.repository,
 Paths: paths,
 DeployCommits: true,
 PreviewPullRequests: false,
 },
 })

These settings configure the stack to trigger an update when new commits are pushed to the branch. In addition, our create handler also kicks off the initial update to create the infrastructure:

err = s.client.createDeployment(ctx, s.org, s.project, stack, createDeploymentRequest{
 DeploymentSettings: DeploymentSettings{
 OperationContext: &operationContext{
 Environment: map[string]string{
 "SITE_CONTENT": content,
 },
 },
 },
 InheritSettings: true,
 Operation: "update",
 })

Notice how this REST API call specifies InheritSettings: true so that Deployment Settings are read from the Pulumi Service and merged with the incoming request payload to create deployment configuration for this run. This stack also has to Click to Deploy via the Pulumi Service UI, so operational tasks like refreshing the stack can be done on demand without pulling down source code onto your local machine.

The Deployments Platform does all of the heavy lifting of managing deployment compute, providing asynchronous workflow orchestration, queueing, status and logging API access, and more. At the end, you get a Pulumi Stack and state file containing a manifest of all cloud resources and their properties managed by the Pulumi Service.

Deployment Settings and Triggers can be combined to ship infrastructure in novel ways with just a few lines of code. See the full RESTful infrastructure API example to try it out yourself.

Pave your Golden Path Today

Pulumi Deployments is the platform for shipping infrastructure at scale. It gives you the programmatic building blocks you need to deliver infrastructure on demand in a large organization. Get started with Deployments today. Happy automating!

Pulumi Deployments: the Fastest Way to Go from Code to Cloud

Evan Boyle — Wed, 02 Nov 2022 06:10:00 -0700

Portions of this blog post are out of date. See the drift detection and time-to-live stacks blog posts for an updated overview and examples.

Adoption of modern cloud technologies and services is driving enormous value for organizations, but many companies are finding that deploying and managing cloud infrastructure is a bottleneck on how fast they can scale. When your tools can’t keep up, scaling your cloud footprint means proportionally scaling your headcount. Things quickly fall apart as demand for cloud infrastructure often outpaces the bandwidth of infrastructure teams. This leads to slow product releases, longer lead times to get new products to market, and burned-out operational teams.

At Pulumi, we want every engineer and organization to be able to take full advantage of the cloud. The cloud should be an accelerant to your business and not a bottleneck. Today we are excited to be launching Pulumi Deployments, a new collection of features to power infrastructure and platform automation and unlock the scale of the cloud.

With Pulumi Deployments, you can run a Pulumi infrastructure as code action (a preview, update, destroy or refresh) inside Pulumi’s managed service. Pulumi provides scalability, observability, and security for deployments. It consists of five components:

The Deployments REST API - A fully managed REST API to execute Pulumi programs with the Pulumi Service. This includes APIs to observe your deployment and all associated logs.
Deployment Visualization - A dashboard for deployment observability including deployment status, logs, and queueing via the Pulumi Service console.
Click to Deploy - Deploy infrastructure with the click of a button from the Pulumi Service console.
git push to Deploy - Configure automatic infrastructure deployments in response to git push events by installing and configuring the Pulumi GitHub App.
Remote Automation API - Our strongly typed infrastructure orchestration technology can now run deployments remotely in the cloud.

Both the Programmatic Deployment API and Remote Automation API are flexible building blocks that simplify the creation and management of common platform automation scenarios like multi-region deployments, drift detection, and ephemeral infrastructure. Click to Deploy and git push to Deploy gives your organization an out-of-the-box golden path to production. Whether you are building your internal cloud platform or using git push to Deploy, every deployment you schedule reaps the benefits of observability in the Pulumi Service. We’ve built an intuitive user experience that allows anyone in your organization to check deployment status, debug logs, and view deployment history.

Automation API & Deployments - Do More with Less

Two years ago, we released Automation API, a programmable SDK for the Pulumi engine that unlocks the full power of infrastructure as code automation. Organizations use it to do things like provisioning infrastructure at runtime for SaaS platforms and building higher-level systems that automate the full lifecycle of infrastructure management. Since then, it has performed millions of automated infrastructure updates. Analyzing our usage data, we found that organizations that center their cloud workflow around Automation API can manage more than ten times the infrastructure per engineer as compared to traditional IaC usage patterns.

This is our motivation for building Pulumi Deployments. We took the product lessons we learned from working hand in hand with leading SaaS and infrastructure companies building on Automation API and turned them into a fully managed service. Pulumi Deployments is the platform to power your platform. The tools that enable the giants to create and manage ten times the cloud footprint per engineer are now available to you as a fully managed service with increased scalability, reliability, and observability baked in.

A Golden Path to Production

Out of the box Pulumi Deployments offers your cloud engineering team a turn-key infrastructure delivery workflow without writing a single line of code. Every piece of infrastructure within your organization can rely on automatic cloud delivery.

`git push` to Deploy

Using the Pulumi Github App (GitLab, Bitbucket, and others coming soon), git push to Deploy enables a rich git-based workflow for safely reviewing infrastructure changes and rolling them out across environments. Opening a Pull Request on GitHub will automatically trigger a pulumi preview and post the results back to the PR. You can also enforce successful previews as a check that gates merging.

Once your PR is reviewed and merged, a Deployment runs automatically:

Setting this up is as easy as installing the Pulumi GitHub App and selecting the repository and branch for your stack:

You can ensure that every environment and every commit is automatically validated and kept up to date with just a few clicks. GitHub integration is available today, with support for GitLab, Atlassian Bitbucket, and others coming soon.

Click to Deploy

Sometimes infrastructure requires ad-hoc operations. You may want to run a refresh to rectify some cloud state changes:

Whatever the situation calls for, you can easily update, refresh, and modify infrastructure with the click of a button in the Pulumi Service console.

Power-up Your Platform with The Deployments API

Both git push to Deploy and Click to Deploy are built on top of the Pulumi Deployments REST API, the same API that is available to you as a building block. The Deployments API is fully managed and enables you to orchestrate cloud rollouts without worrying about capacity. Let’s look at a few ways you can embed it into your custom cloud platform.

Drift Detection

Manual edits within the AWS console can wreak havoc on your infrastructure if those changes aren’t brought back into source control. The last thing you want is to scale out a service just to have the next CI run undo that change.

You can build drift detection into your platform and get automated alerts on unexpected infrastructure changes.

aws.cloudwatch.onSchedule("drift-lambda", "cron(0/5 * * * ? *)", async() => {
 const outstandingDeployments = [];
 for(let s of stacks) {
 const url = `https://api.pulumi.com/api/stacks/${s.organization}/${s.project}/${s.stack}/deployments`;
 const response = await fetch(url, {
 method: "POST",
 headers,
 body: JSON.stringify(refreshPayload),
 });

 const deployment = await response.json();
 outstandingDeployments.push(deployment.id);
 }
})

This scheduled Lambda runs refresh operations through the Deployments REST API. When one of those deployments detects changes, we get notified. With platform automation that automatically and proactively detects changes, your cloud team can reduce the number of surprise incidents, freeing them up to focus on higher-leverage activities. The full source code is available for the Drift Detection example.

TTL Stacks and Temporary Infrastructure

Infrastructure waste is a common problem. It’s too easy to leave development infrastructure running accidentally and end up with a huge bill. Many companies have entire teams devoted to solving this problem.

With Pulumi Webhooks and the Deployments API we can build a system for tagging and reclaiming temporary infrastructure. Anyone in your organization can create temporary infrastructure with a simple workflow:

$ pulumi stack tag set ttl 360 # this stack expires six hours after it deploys
$ pulumi up # with the `ttl` tag preset, the infrastructure will be automatically destroyed at expiration

After the expiration period has elapsed, a deployment will be queued and the development infrastructure is automatically destroyed. The Deployments API can be embedded into your platform to reduce cloud waste and automate cost controls. The full source code is available for the TTL Stack Processor.

Managing Scale with Remote Automation API

With Automation API, Pulumi can be fully embedded inside your software projects to power a wide range of custom cloud infrastructure automation. Previously, Automation API could only run locally. With Remote Automation API, these same scripts can run locally while offloading deployment workloads to the cloud via the Deployments API. The infrastructure as code action (a preview, update, destroy or refresh) runs inside of our fully managed service.

const stack = await RemoteWorkspace.createOrSelectStack({
 stackName,
 url: "https://github.com/pulumi/examples.git",
 branch: "refs/heads/master",
 projectPath: project,
}, {
 envVars: {
 AWS_REGION: "us-west-2",
 AWS_ACCESS_KEY_ID: process.env.AWS_ACCESS_KEY_ID ?? "",
 AWS_SECRET_ACCESS_KEY: { secret: process.env.AWS_SECRET_ACCESS_KEY ?? "" },
 AWS_SESSION_TOKEN: { secret: process.env.AWS_SESSION_TOKEN ?? "" },
 },
});

const upRes = await stack.up({ onOutput: console.log });
console.log(`url: ${upRes.outputs.websiteUrl.value}`);

repo := auto.GitRepo{
 URL: "https://github.com/pulumi/examples.git",
 Branch: "refs/heads/master",
 ProjectPath: project,
}

env := map[string]auto.EnvVarValue{
 "AWS_REGION": {Value: "us-west-2"},
 "AWS_ACCESS_KEY_ID": {Value: os.Getenv("AWS_ACCESS_KEY_ID")},
 "AWS_SECRET_ACCESS_KEY": {Value: os.Getenv("AWS_SECRET_ACCESS_KEY"), Secret: true},
 "AWS_SESSION_TOKEN": {Value: os.Getenv("AWS_SESSION_TOKEN"), Secret: true},
}

s, err := auto.UpsertRemoteStackGitSource(ctx, stackName, repo, auto.RemoteEnvVars(env))
if err != nil {
 fmt.Printf("Failed to create or select stack: %v\n", err)
 os.Exit(1)
}

res, err := s.Up(ctx, optremoteup.ProgressStreams(os.Stdout))

stack = auto.create_or_select_remote_stack_git_source(
 stack_name=stack_name,
 url="https://github.com/pulumi/examples.git",
 branch="refs/heads/master",
 project_path=project,
 opts=auto.RemoteWorkspaceOptions(
 env_vars={
 "AWS_REGION": "us-west-2",
 "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
 "AWS_SECRET_ACCESS_KEY": auto.Secret(os.environ["AWS_SECRET_ACCESS_KEY"]),
 "AWS_SESSION_TOKEN": auto.Secret(os.environ["AWS_SESSION_TOKEN"]),
 },
 ),
)

up_res = stack.up(on_output=print)

var stack = await RemoteWorkspace.CreateOrSelectStackAsync(new RemoteGitProgramArgs(stackName, "https://github.com/pulumi/examples.git")
{
 Branch = "refs/heads/master",
 ProjectPath = project,
 EnvironmentVariables =
 {
 { "AWS_REGION", new EnvironmentVariableValue("us-west-2") },
 { "AWS_ACCESS_KEY_ID", RequireFromEnvironment("AWS_ACCESS_KEY_ID") },
 { "AWS_SECRET_ACCESS_KEY", RequireFromEnvironment("AWS_SECRET_ACCESS_KEY", isSecret: true) },
 { "AWS_SESSION_TOKEN", RequireFromEnvironment("AWS_SESSION_TOKEN", isSecret: true) },
 },
});

var result = await stack.UpAsync(new RemoteUpOptions { OnStandardOutput = Console.WriteLine });

Multi-region infrastructure build outs become a breeze, as workloads can be easily parallelized. Now you can run thousands of updates concurrently without having to manage the compute that runs them.

Centralized Observability

Whether you use the built-in cloud delivery workflows like Push to Deploy, or build your own custom platform with the REST API and Remote Automation API, you get full observability into every deployment. Anyone in your organization can view updates status, update logs, and the full history of deployments.

You can build platform automation without worrying about how you’ll surface the results across your product, engineering, and infrastructure teams.

Automation Accelerates the Cloud

We’ve taken the lessons from our industry-leading customers in cloud SaaS and infrastructure and packaged them into Pulumi Deployments. This suite of tools enables you to manage and scale ten times the cloud footprint as opposed to traditional IaC usage patterns. Whether you want a turn-key delivery workflow or highly-customizable platform building blocks, there is something for your organization.

Pulumi Deployments removes the cloud as a bottleneck and unlocks it as an accelerant for your business. It is available in preview today. Deployments are free during the preview, with usage-based pricing to come. Check out these resources to get started today:

Request access to the Pulumi Deployments preview
Pulumi Deployments examples repo
The Pulumi Deployments demo video
Pulumi Deployments documentation
Pulumi Deployments REST API documentation
Automation API examples

🚀🚀🚀 Happy building! 🚀🚀🚀

Pulumi Blog: Deployments

Dependent Stack Updates with Pulumi Deployments

Deployment Webhook Destinations

Expressing Stack Dependencies with Pulumi Auto Deploy

Scale Infrastructure with Software

Review Stacks: Collaborate in the Cloud

Configuring Review Stacks

Pulumi Cloud UI

REST API

Pulumi Cloud Resource Provider

Collaborate in the Cloud

Pulumi Deployments: API-Driven Infrastructure at Scale

Deployment Settings and Triggers

Example: Building RESTful Infrastructure APIs

Pave your Golden Path Today

Pulumi Deployments: the Fastest Way to Go from Code to Cloud

Automation API & Deployments - Do More with Less

A Golden Path to Production

git push to Deploy

Click to Deploy

Power-up Your Platform with The Deployments API

Drift Detection

TTL Stacks and Temporary Infrastructure

Managing Scale with Remote Automation API

Centralized Observability

Automation Accelerates the Cloud

`git push` to Deploy