Pulumi Blog: Evan Boyle

Dependent Stack Updates with Pulumi Deployments

Evan Boyle — Tue, 20 Jun 2023 00:00:00 +0000

As infrastructure projects grow in size and complexity, you need to decompose infrastructure into smaller stacks to limit the blast radius of errors, extract and reference common layers like networking, and limit access to sensitive components. This comes with a coordination cost as you now need to figure out how to detect and propagate changes to downstream stacks in your dependency graph. Today we’re announcing two features that can help you manage this complexity by automatically updating dependent stacks:

Deployment Webhook Destinations for triggering downstream stack updates automatically
The Pulumi Auto Deploy Package for easily expressing dependencies and configuring downstream updates via Deployment Webhooks.

The following example shows automatic deployment of stacks with the following dependency graph
a
├── b
│ ├── d
│ ├── e
│ └── f
└── c
Whenever a node in the graph is updated, all downstream nodes will be automatically updated
via a webhook triggering Pulumi Deployments.

Automatically updating dependent stacks and providing a way to write Pulumi programs that can express nested stack updates are two of the most requested features across the Pulumi project, and we’re excited to enable both of these via Pulumi Cloud and Pulumi Deployments.

Deployment Webhook Destinations

Pulumi Webhooks allow you to pick one or more event types on a stack (i.e. update succeeded, or refresh failed) and automatically deliver an event to a destination such a slack channel or a webserver you’ve deployed with custom code. Now you can choose Pulumi Deployments as a webhook destination. When an update completes on the parent stack, a deployment will automatically be triggered on the destination stack. This makes it easy to keep dependent stacks up to date. For instance, you can use Deployment Webhook Destinations to configure every successful update on your networking stack to trigger downstream updates on your database and compute stacks.

import * as pulumi from "@pulumi/pulumi";
import * as pulumiservice from "@pulumi/pulumiservice";

const databaseWebhook = new pulumiservice.Webhook("databaseWebhook", {
 organizationName: "org",
 projectName: "network",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "database/prod",
 active: true,
 displayName: "deploy-database",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});
const computeWebhook = new pulumiservice.Webhook("computeWebhook", {
 organizationName: "org",
 projectName: "database",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "compute/prod",
 active: true,
 displayName: "deploy-compute",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using PulumiService = Pulumi.PulumiService;

return await Deployment.RunAsync(() =>
{
 var databaseWebhook = new PulumiService.Webhook("databaseWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "network",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "database/prod",
 Active = true,
 DisplayName = "deploy-database",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

 var computeWebhook = new PulumiService.Webhook("computeWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "database",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "compute/prod",
 Active = true,
 DisplayName = "deploy-compute",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

});

import pulumi
import pulumi_pulumiservice as pulumiservice

database_webhook = pulumiservice.Webhook("databaseWebhook",
 organization_name="org",
 project_name="network",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="database/prod",
 active=True,
 display_name="deploy-database",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])
compute_webhook = pulumiservice.Webhook("computeWebhook",
 organization_name="org",
 project_name="database",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="compute/prod",
 active=True,
 display_name="deploy-compute",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := pulumiservice.NewWebhook(ctx, "databaseWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("network"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("database/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-database"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 _, err = pulumiservice.NewWebhook(ctx, "computeWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("database"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("compute/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-compute"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
resources:
 databaseWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: network
 stackName: prod
 format: pulumi_deployments
 payloadUrl: database/prod
 active: true
 displayName: deploy-database
 filters:
 - update_succeeded
 computeWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: database
 stackName: prod
 format: pulumi_deployments
 payloadUrl: compute/prod
 active: true
 displayName: deploy-compute
 filters:
 - update_succeeded

Expressing Stack Dependencies with Pulumi Auto Deploy

We’ve also shipped a new Pulumi package called Pulumi Auto Deploy. It lets you simply express dependencies between stacks, and takes care of creating and updating the necessary Deployment Webhooks under the hood.

import * as autodeploy from "@pulumi/auto-deploy";
import * as pulumi from "@pulumi/pulumi";

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

const organization = pulumi.getOrganization();
const project = "dependency-example"

export const f = new autodeploy.AutoDeployer("auto-deployer-f", {
 organization,
 project,
 stack: "f",
 downstreamRefs: [],
});

export const e = new autodeploy.AutoDeployer("auto-deployer-e", {
 organization,
 project,
 stack: "e",
 downstreamRefs: [],
});

export const d = new autodeploy.AutoDeployer("auto-deployer-d", {
 organization,
 project,
 stack: "d",
 downstreamRefs: [],
});

export const c = new autodeploy.AutoDeployer("auto-deployer-c", {
 organization,
 project,
 stack: "c",
 downstreamRefs: [],
});

export const b = new autodeploy.AutoDeployer("auto-deployer-b", {
 organization,
 project,
 stack: "b",
 downstreamRefs: [d.ref, e.ref, f.ref],

});

export const a = new autodeploy.AutoDeployer("auto-deployer-a", {
 organization,
 project,
 stack: "a",
 downstreamRefs: [b.ref, c.ref],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AutoDeploy = Pulumi.AutoDeploy;

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

return await Deployment.RunAsync(() =>
{
 var projectVar = "dependency-example";

 var organization = "pulumi";

 var f = new AutoDeploy.AutoDeployer("f", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "f",
 DownstreamRefs = new[] {},
 });

 var e = new AutoDeploy.AutoDeployer("e", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "e",
 DownstreamRefs = new[] {},
 });

 var d = new AutoDeploy.AutoDeployer("d", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "d",
 DownstreamRefs = new[] {},
 });

 var c = new AutoDeploy.AutoDeployer("c", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "c",
 DownstreamRefs = new[] {},
 });

 var b = new AutoDeploy.AutoDeployer("b", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "b",
 DownstreamRefs = new[]
 {
 d.Ref,
 e.Ref,
 f.Ref,
 },
 });

 var a = new AutoDeploy.AutoDeployer("a", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "a",
 DownstreamRefs = new[]
 {
 b.Ref,
 c.Ref,
 },
 });

});

import pulumi
import pulumi_auto_deploy as auto_deploy

'''

The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
Whenever a node in the graph is updated,
all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
'''

project_var = "dependency-example"
organization = pulumi.get_organization()
f = auto_deploy.AutoDeployer("f",
 organization=organization,
 project=project_var,
 stack="f",
 downstream_refs=[])
e = auto_deploy.AutoDeployer("e",
 organization=organization,
 project=project_var,
 stack="e",
 downstream_refs=[])
d = auto_deploy.AutoDeployer("d",
 organization=organization,
 project=project_var,
 stack="d",
 downstream_refs=[])
c = auto_deploy.AutoDeployer("c",
 organization=organization,
 project=project_var,
 stack="c",
 downstream_refs=[])
b = auto_deploy.AutoDeployer("b",
 organization=organization,
 project=project_var,
 stack="b",
 downstream_refs=[
 d.ref,
 e.ref,
 f.ref,
 ])
a = auto_deploy.AutoDeployer("a",
 organization=organization,
 project=project_var,
 stack="a",
 downstream_refs=[
 b.ref,
 c.ref,
 ])

package main

import (
 "github.com/pulumi/pulumi-auto-deploy/sdk/go/autodeploy"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 /**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

 pulumi.Run(func(ctx *pulumi.Context) error {
 projectVar := "dependency-example"
 organization := "pulumi"
 f, err := autodeploy.NewAutoDeployer(ctx, "f", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("f"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 e, err := autodeploy.NewAutoDeployer(ctx, "e", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("e"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 d, err := autodeploy.NewAutoDeployer(ctx, "d", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("d"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 c, err := autodeploy.NewAutoDeployer(ctx, "c", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("c"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 b, err := autodeploy.NewAutoDeployer(ctx, "b", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("b"),
 DownstreamRefs: pulumi.StringArray{
 d.Ref,
 e.Ref,
 f.Ref,
 },
 })
 if err != nil {
 return err
 }
 _, err = autodeploy.NewAutoDeployer(ctx, "a", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("a"),
 DownstreamRefs: pulumi.StringArray{
 b.Ref,
 c.Ref,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
variables:
 project: dependency-example
 # TODO: update once https://github.com/pulumi/pulumi-yaml/issues/461 is fixed
 organization: pulumi
resources:
 f:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: f
 downstreamRefs: []
 e:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: e
 downstreamRefs: []
 d:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: d
 downstreamRefs: []
 c:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: c
 downstreamRefs: []
 b:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: b
 downstreamRefs:
 - ${d.ref}
 - ${e.ref}
 - ${f.ref}
 a:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: a
 downstreamRefs:
 - ${b.ref}
 - ${c.ref}

Just declare an AutoDeployer resource for each stack, and any downstream stacks will automatically be updated whenever a stack in its dependency chain is updated.

Automate your deployment pipeline

Use Pulumi Deployments to trigger downstream stack updates automatically when an upstream stack changes. Keep your dependent infrastructure in sync without manual runbooks.

Get started

Scale Infrastructure with Software

At Pulumi, we build tools to help you scale your infrastructure footprint with software instead of just humans. Deployment Webhook Destinations and pulumi-auto-deploy give you the tools to keep all of your dependent infrastructure up to date with automation instead of manual runbooks and human-driven operations.

Check out these resources to get started today:

Check out the Pulumi Deployments documentation and learn about configuring Deployment Settings for your stacks.
Get started with the Pulumi Auto Deploy Package.
Learn more about Pulumi Cloud Webhooks.
Join us in the #pulumi-deployments community slack channel.

🚀🚀🚀 Happy building! 🚀🚀🚀

Review Stacks: Collaborate in the Cloud

Evan Boyle — Wed, 14 Jun 2023 00:00:00 +0000

Today we’re excited to announce Review Stacks – dedicated cloud environments that get created automatically every time a pull request is opened, all powered by Pulumi Deployments. Open a pull request, and Pulumi Deployments will stand up a stack with your changes and the Pulumi GitHub App will add a PR comment with the outputs from your deployment. Merge the PR and Pulumi Deployments will destroy the stack and free up the associated resources. It has never been simpler to pick up an unfamiliar codebase, make changes to both application and infrastructure code, and share a live environment for review with your teammates.

Ephemeral environments for front-end development and static websites have become commonplace in recent years, but infrastructure and complex systems have been unsupported up until this point. Review Stacks enable you to iterate on both application code changes and infrastructure code changes at the same time. Making changes to HTML, CSS, Next.js, Go APIs, Pulumi configuration, subnet configuration, databases, and load balancers are all possible with Review Stacks. No more fumbling with out-of-date READMEs and developer documentation. Just open a pull request and you can start testing changes against everything from simple static websites to API servers, microservices, data pipelines, Kubernetes clusters, and any other piece of infrastructure across Pulumi’s 100+ cloud providers.

Review Stacks manage the full lifecycle of your cloud development environment including creating it when the PR is opened, updating it every time a new commit is pushed, and destroying and cleaning up all cloud resources when the pull request is merged or closed. They even integrate with GitHub checks so that you can gate your pull requests on successful Review Stack deployments.

Pulumi customers already using Review Stacks find that it saves their developers time. Having a standardized process for development environments and change management makes it easier to onboard new developers, and load-balance teammates onto unfamiliar projects and codebases. It also makes it substantially easier for developers to work on multiple feature branches and bug fixes at the same time, as they don’t have to worry about manually maintaining and juggling development environments.

“Pulumi allows every engineer to build and update infrastructure instead of only our infrastructure team, which enables us to ship faster than if we were using a domain-specific language,” said Aaron Gibralter, head of engineering, Mighty. “With Pulumi Deployments, we could easily set up a standardized CI/CD infrastructure workflow for every engineer that immediately makes them productive. Review Stacks automates the work of managing our own ephemeral infrastructure and makes our lives easier.”

Review Stacks also provide an avenue for optimizing cloud costs. Rather than keeping development infrastructure running 24/7, Review Stacks only exist as long as an engineer works on a feature. Since Review Stacks are flexible and built on Pulumi IaC, it is also possible to configure it to deploy to multi-tenant environments that share development infrastructure to further streamline costs.

“At GitHub, we are constantly seeking to improve the developer experience and our partner ecosystem plays a big role in that.” said Matthew Manning, Senior Business Development Manager for GitHub Actions, “The launch of Pulumi Review Stacks with GitHub makes it easier for teams to review and test their products before shipping which ultimately improves the developer experience.”

Configuring Review Stacks

Configuring Review Stacks is a simple three-step process:

Create a new stack, by convention named pr, and corresponding Pulumi.pr.yaml configuration file - this config will be copied into every review stack that gets created, and can even be modified within a PR.
Configure Deployment Settings for the stack - this specifies how to acquire source code, cloud credentials and more when deploying via Pulumi Deployments.
Set the pullRequestTemplate Deployment Setting to true - this indicates that all pull requests against this stack’s branch should reference this stack as a Review Stack template.

You can use an existing stack as a Review Stack template, as long as it has Deployment Settings configured. This will result in Review Stacks being deployed into the same cloud account. If you want to separate the cloud resources in your production stack from the resources created via Review Stacks then you can create a separate stack and template that references a different cloud account (AWS, Azure, GCP, etc).

Review Stacks and Deployment Settings can be configured via the Pulumi Cloud console, the Pulumi Cloud REST API, or within a Pulumi Program using the Pulumi Cloud Resource Provider.

Pulumi Cloud UI

It is just one click to turn on Review Stacks via the Pulumi Cloud console.

REST API

You can programmatically configure Review Stacks and Deployment Settings at scale across thousands of projects using the Deployments REST API.

curl -i -XPOST -H "Content-Type: application/json" -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
--location "https://api.pulumi.com/api/stacks/org/project/stack/deployments/settings" \
-d '{
"gitHub":{
"pullRequestTemplate": true
}
}'

Pulumi Cloud Resource Provider

You can use Pulumi to manage and code review Deployment Settings and Review Stacks with the Pulumi Cloud Provider.

import * as pulumiservice from "@pulumi/pulumiservice";

const deploymentSettings = new pulumiservice.DeploymentSettings("deploymentSettings", {
 organization: pulumi.getOrganization(),
 project: "your project",
 stack: "your stack",
 github: {
 deployCommits: true,
 previewPullRequests: true,
 pullRequestTemplate: true,
 repository: "pulumi/deployment-automation",
 },
 sourceContext: {
 git: {
 branch: "refs/heads/main",
 repoDir: "pulumi-pet-shop",
 },
 },
});

Collaborate in the Cloud

At Pulumi, our engineering team writes a ton of code. We have hundreds of repositories with programs managing thousands of stacks and tens of thousands of cloud resources. These tools and services power production workloads, internal tools, business intelligence and process, and more. And of course we use Pulumi to ship all of it. When you’re an engineer working in one of these repositories, making a code change is the easy part. The hard part is figuring out how to get a tight dev loop going—in particular, faithfully verifying your changes before merging and deploying them. Are there instructions in the README for running locally? A Docker Compose file? Or maybe standalone dev environments? Can I figure out what config and upstream stack references are required to stand up a fresh dev stack? And what about service dependencies? Pulumi makes it incredibly easy to build in the cloud and write prolific amounts of code, however, with that increase in development velocity comes a need to standardize the entire process by which we develop, test, review, and ship our code.

We’ve heard similar stories from hundreds of customers. From startup teams of five to enterprises of thousands, they’re all looking for the same thing - so we went ahead and built it. With Review Stacks and Pulumi Deployments, you can ship with greater velocity, confidence, and collaboration.

Check out these resources and get started today:

Review Stacks documentation
Learn more about Pulumi Deployments
Join us in the #pulumi-deployments community slack channel

🚀🚀🚀 Happy building! 🚀🚀🚀

Pulumi Deployments: API-Driven Infrastructure at Scale

Evan Boyle — Wed, 29 Mar 2023 00:00:00 +0000

Delivering software has been conventionally driven by CI/CD workflows. A single commit is merged into a codebase, and a small, and static set of workflow runs are triggered by the CI system to update the appropriate environments (Dev -> Staging -> Production). This may have been fine when the only product a company had to offer was a single stateless web service, but increasingly companies are called upon to deliver cloud infrastructure as a product.

There are obvious forms of this, such as IaaS companies building database-as-a-service offerings and SaaS companies deploying self-hosted installations into customer environments. But there are less apparent forms like internal platform teams that are chartered with enabling self-service of Kubernetes clusters configured with best practices to thousands of application engineers within an enterprise. In any of these cases, engineers are delivering infrastructure as the product but only have conventional tools like gitops and CI/CD to reach for.

The assumptions that CI/CD systems make are limiting for infrastructure. We are not pushing a single application commit across three environments. We are templating infrastructure hundreds and thousands of times. Once that infrastructure is templated, we’re responsible for ensuring it is secure, up-to-date, and can be customized by self-service users. CI/CD is a poor fit due to the design choices that are locked in around source control and source events. CI/CD is typically configured via templated YAML that is checked into source control, with no way to create or customize behavior dynamically from a REST API. And beyond configuration, there aren’t REST APIs for triggering updates and workflow runs. The best you can expect is the ability to add more YAML to configure a cron job. What do you do if you need to create a self-service platform, or configure CI/CD across thousands of projects across your organization?

Today we’re excited to announce the release of a new version of Pulumi Deployments, a fully managed platform for shipping infrastructure at scale. It provides:

Compute - run infrastructure deployments and workloads (i.e. pulumi up) on fully managed, secure, and elastic compute.
Configuration - A simple, centralized format for specifying everything you need to deploy your infrastructure at the stack level. Source code, cloud credentials, OpenID Connect (OIDC), environment variables, build prerequisites, and dependencies are bundled in an atomic unit.
Composition - Trigger deployments via our REST API, click to deploy in the Pulumi Service UI, Git Push to Deploy, Remote Automation API, and other programmatic building blocks that you can compose to build custom cloud automation.

Pulumi Deployments is built on top of a powerful set of public REST APIs. Unlike any other platform, Deployments lets you configure CI/CD settings programmatically. Service-catalog scenarios and guard rails that enforce CI/CD best practices across thousands of projects are far easier to achieve when you have REST APIs that you can build software on top of. No more generating and templating YAML to configure infrastructure deployments at scale.

Deployment Settings and Triggers

Deployments can be configured at a stack level via Deployment Settings in three ways:

The Pulumi Service UI enables setting up OIDC and git push to deploy in just a few clicks
The Pulumi Service Provider lets you define deployment configuration as Pulumi code in you language of choice, reviewed and managed in source control.
The Deployment Settings REST API provides programmatic access to configure Deployments and CI/CD dynamically, without needing to generate or check in YAML.

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();

// deployment settings configured via a pulumi program
const settings = new service.DeploymentSettings("deployment_settings", {
 organization: "service-provider-test-org",
 project: "test-deployment-settings-proj",
 stack: "dev",
 operationContext: {
 environmentVariables: {
 TEST_VAR: "foo",
 SECRET_VAR: config.requireSecret("my_secret"),
 }
 },
 sourceContext: {
 git: {
 repoUrl: "https://github.com/pulumi/deploy-demos.git",
 branch: "refs/heads/main",
 repoDir: "pulumi-programs/simple-resource"
 }
 }
});

Each stack configured with Pulumi Deployments gets access to a number of triggers that can be used to start a Pulumi operation (i.e. pulumi up):

Deployments REST API - an endpoint available at api.pulumi.com that can trigger a variety of pulumi (up/destroy/refresh/preview) operations. Calls to this trigger can override individual settings to customize deployment behavior (i.e. - deploy a different branch, or override a particular environment variable).
git push to Deploy - Deep integration with the Pulumi GitHub App that enables automatic previews on every pull request, and an automated update whenever code is pushed to the target branch.
Click to Deploy - trigger a deployment from the Pulumi Service Console with the push of a button. Very useful for day-to-day operational activities such as debugging a stuck stack, or rectifying drift via a refresh without having to pull a stack and its accompanying source code onto your machine.
Remote Automation API - Your favorite strongly typed Pulumi orchestration tool, Automation API, lets you offload update workloads into the cloud. Write deployment scripts that update thousands of environments simultaneously without worrying about compute.

You can mix, match, and compose Deployment Settings and Triggers to create a golden path to production that is perfect for your enterprise’s needs.

Example: Building RESTful Infrastructure APIs

Increasingly, infrastructure updates happen in response to user action and not source control events like a git commit. For instance, a SaaS database company might need to spin up dedicated compute and storage when a customer puts in a credit card and self-serves onto a specific SKU. This signup workflow might happen hundreds or even thousands of times an hour. Traditional deployment systems are optimized for infrastructure that spans, at most, a handful of environments (dev/staging/production).

Leading SaaS and infrastructure companies want the best of both worlds:

Custom, domain-specific REST APIs to create infrastructure on demand.
Desired state configuration that keeps track of what is deployed, and where.

You can build on top of the Deployments REST API to build your own infrastructure APIs. Here we have a simple Go web server that offers a RESTful interface over top of a static website. Users can create, update, and delete static websites, specifying the content for the site in the POST payload. Here’s our web server:

 router := httprouter.New()
 router.POST("/sites", server.create)
 router.GET("/sites/:id", server.get)
 router.POST("/sites/:id", server.update)
 router.DELETE("/sites/:id", server.delete)

 http.ListenAndServe(*addr, router)

Within the create handler, we first dynamically configure Deployment Settings for the stack, including the branch of the Pulumi program to be deployed, the AWS OIDC provider used for cloud credentials, and the configuration of git push Deployment Triggers`:

 err := s.client.patchDeploymentSettings(r.Context(), s.org, s.project, stack, DeploymentSettings{
 SourceContext: &sourceContext{
 Git: gitContext{
 Branch: s.branch,
 RepoDir: s.dir,
 },
 },
 OperationContext: &operationContext{
 Environment: map[string]string{
 "AWS_REGION": s.region,
 },
 OIDC: &oidcContext{
 AWS: &awsOIDCContext{
 RoleARN: s.roleARN,
 SessionName: s.sessionName,
 },
 },
 },
 GitHub: &gitHubContext{
 Repository: s.repository,
 Paths: paths,
 DeployCommits: true,
 PreviewPullRequests: false,
 },
 })

These settings configure the stack to trigger an update when new commits are pushed to the branch. In addition, our create handler also kicks off the initial update to create the infrastructure:

err = s.client.createDeployment(ctx, s.org, s.project, stack, createDeploymentRequest{
 DeploymentSettings: DeploymentSettings{
 OperationContext: &operationContext{
 Environment: map[string]string{
 "SITE_CONTENT": content,
 },
 },
 },
 InheritSettings: true,
 Operation: "update",
 })

Notice how this REST API call specifies InheritSettings: true so that Deployment Settings are read from the Pulumi Service and merged with the incoming request payload to create deployment configuration for this run. This stack also has to Click to Deploy via the Pulumi Service UI, so operational tasks like refreshing the stack can be done on demand without pulling down source code onto your local machine.

The Deployments Platform does all of the heavy lifting of managing deployment compute, providing asynchronous workflow orchestration, queueing, status and logging API access, and more. At the end, you get a Pulumi Stack and state file containing a manifest of all cloud resources and their properties managed by the Pulumi Service.

Deployment Settings and Triggers can be combined to ship infrastructure in novel ways with just a few lines of code. See the full RESTful infrastructure API example to try it out yourself.

Pave your Golden Path Today

Pulumi Deployments is the platform for shipping infrastructure at scale. It gives you the programmatic building blocks you need to deliver infrastructure on demand in a large organization. Get started with Deployments today. Happy automating!

Read Every Single Error

Evan Boyle — Tue, 21 Mar 2023 00:00:00 +0000

At Pulumi we read every single error message that our API produces. This is the primary mechanism that led to a 17x YoY reduction in our error rate. You’re probably wondering how reading error messages make them go away.

Doesn’t common wisdom tell us that we need a fancy observability toolchain, or to follow the Google SRE model? I can confidently say that you don’t. I’ll go a step further and state that throughout my career, every system I’ve worked on that relied on aggregate views of errors was a complete dumpster fire. In every team where we instead chose to read all the errors, reliability naturally improved over time.

I offer a concrete process that will drive your error rates down over time with math to back it up.

Read Every Error Message That Your System Produces

You should read every error message that your system produces. Simple but effective. Our team pumps every 5XX into a Slack channel and reviewing each of these is a top priority for the current on-call engineer. There’s a little more to it, but that’s the gist! Commit to this process and your error rates are guaranteed to drop. And I can prove it!

Reliability from First Principles

Why does reading error messages imply improving error rates? It isn’t magic. You still have to dedicate time to fix the bugs you’re shipping and make proactive investments. But you can model this process with a simple inequality:

(API Call Volume) * (Error Rate) * (Time to Triage an Error) < On-Call Attention

This comes with a few important constraints:

On-call attention is a fixed commodity
API call volume scales with your business
This process requires reading every error message

The end result is simple. Your error rate must improve over time to keep the scales balanced. If it doesn’t, the process becomes untenable.

Let’s assume you have one engineer on call, and they spend at most an hour a day triaging errors (they have other responsibilities too!) on each of the ~250 business days per year. That’s a cap of 250 hours per year that can be spent triaging errors. This is effectively fixed. Sure you can hire and split systems out into separate on-call rotations to increase capacity, but our goal is to scale exponentially with respect to humans, not linearly!

Let’s say that triaging an error is a 5-minute process that might involve any of the following:

Checking the issue tracker for a pre-existing ticket to add additional context, a +1, etc.
Reaching out to a teammate who just shipped some buggy code so they can get a fix out
Filing a well-documented bug
Starting a Slack thread with the team to raise a known issue that seems to be cropping up more regularly
Just opening a PR if it’s a simple fix

Remember we only have 250 hours or 15,000 minutes per year to triage with a single on-call engineer. At five minutes a pop we can triage ~3000 errors per year until things start dropping on the floor.

Now imagine you work on a new product and you just launched your MVP. It is early days, but there is some traction and you’re seeing API traffic at a rate of 1,000,000 requests per year. Your annual budget for triaging as we previously determined is 3000 errors, which yields a maximum permissible error rate of 0.3%, or a 99.7% success rate.

Over the next six months, the team iterates, listens to customers, and delivers a ton of value. As a result, traffic levels grow to a rate of 10,000,000 requests per year. But still, our team hasn’t grown much and we still have just one engineer on call at any given point in time meaning our error triage capacity remains fixed at 3000 errors per year. In order to keep up with triaging the error stream at the increased levels of traffic, the team must improve the error rate from 0.3% to 0.03%. And hopefully, this team continues to be successful, increasing API traffic superlinearly in years to come.

If you want to be able to read every error message, then the error rate has to come down as API traffic increases.

Why Should You Care?

Pulumi aspires to be the most reliable infrastructure that our customers interact with, and the benefits towards that end are reason enough for us. But this process is by no means free, and there is always an opportunity cost.

However, we noticed a powerful second-order effect emerge over time. The team began obsessing over the user experience. Following this process builds a visceral understanding of how your system behaves. You know when a new feature has a bug before the first support tickets get opened. You know which customer workloads will require scaling investments in the next 90 days. You know what features see heavy usage and which ones customers ignore. You’re forced to confront every wart in your application. Slowly, your team builds a better understanding of your customers and this trickles down into every aspect of product development. You begin to hold yourselves to higher standards.

A friend and engineer at a large-cap software company read a draft of this post and told me:

“I can’t imagine this process being set on any of the systems I work on at [redacted]. They are so prone to 500s and everybody shrugs their shoulders as if it is just commonplace.”

Apathy in place of customer obsession is not an option if you are a startup that wants to disrupt anything.

The SRE’s Folly

Error budgets and the SRE model are haute couture. Some preach that we should never look at errors at this level of granularity and instead use expensive tools that aggregate, categorize, and collect statistics on errors flowing through your system. But all of this automation can actually make things worse when you reach for it prematurely. Aggregating errors is a great way to gloss over important details early on. Collecting fancy metrics does not matter if your users are not happy. Cutting your teeth with the tools and processes that make sense for your level of scale is the only way to build a high-performance culture. Skipping straight to step 100 does not always help.

Admittedly, this process does not work for Google-level scale. But it works a lot longer than you might imagine. Pulumi manages a meaningful percentage of the resources deployed across all clouds. I asked large-cap software engineer about traffic levels and believe it or not, it is in the same order of magnitude as what we see at Pulumi.

We’re still reading every error.

Pulumi Deployments: the Fastest Way to Go from Code to Cloud

Evan Boyle — Wed, 02 Nov 2022 06:10:00 -0700

Portions of this blog post are out of date. See the drift detection and time-to-live stacks blog posts for an updated overview and examples.

Adoption of modern cloud technologies and services is driving enormous value for organizations, but many companies are finding that deploying and managing cloud infrastructure is a bottleneck on how fast they can scale. When your tools can’t keep up, scaling your cloud footprint means proportionally scaling your headcount. Things quickly fall apart as demand for cloud infrastructure often outpaces the bandwidth of infrastructure teams. This leads to slow product releases, longer lead times to get new products to market, and burned-out operational teams.

At Pulumi, we want every engineer and organization to be able to take full advantage of the cloud. The cloud should be an accelerant to your business and not a bottleneck. Today we are excited to be launching Pulumi Deployments, a new collection of features to power infrastructure and platform automation and unlock the scale of the cloud.

With Pulumi Deployments, you can run a Pulumi infrastructure as code action (a preview, update, destroy or refresh) inside Pulumi’s managed service. Pulumi provides scalability, observability, and security for deployments. It consists of five components:

The Deployments REST API - A fully managed REST API to execute Pulumi programs with the Pulumi Service. This includes APIs to observe your deployment and all associated logs.
Deployment Visualization - A dashboard for deployment observability including deployment status, logs, and queueing via the Pulumi Service console.
Click to Deploy - Deploy infrastructure with the click of a button from the Pulumi Service console.
git push to Deploy - Configure automatic infrastructure deployments in response to git push events by installing and configuring the Pulumi GitHub App.
Remote Automation API - Our strongly typed infrastructure orchestration technology can now run deployments remotely in the cloud.

Both the Programmatic Deployment API and Remote Automation API are flexible building blocks that simplify the creation and management of common platform automation scenarios like multi-region deployments, drift detection, and ephemeral infrastructure. Click to Deploy and git push to Deploy gives your organization an out-of-the-box golden path to production. Whether you are building your internal cloud platform or using git push to Deploy, every deployment you schedule reaps the benefits of observability in the Pulumi Service. We’ve built an intuitive user experience that allows anyone in your organization to check deployment status, debug logs, and view deployment history.

Automation API & Deployments - Do More with Less

Two years ago, we released Automation API, a programmable SDK for the Pulumi engine that unlocks the full power of infrastructure as code automation. Organizations use it to do things like provisioning infrastructure at runtime for SaaS platforms and building higher-level systems that automate the full lifecycle of infrastructure management. Since then, it has performed millions of automated infrastructure updates. Analyzing our usage data, we found that organizations that center their cloud workflow around Automation API can manage more than ten times the infrastructure per engineer as compared to traditional IaC usage patterns.

This is our motivation for building Pulumi Deployments. We took the product lessons we learned from working hand in hand with leading SaaS and infrastructure companies building on Automation API and turned them into a fully managed service. Pulumi Deployments is the platform to power your platform. The tools that enable the giants to create and manage ten times the cloud footprint per engineer are now available to you as a fully managed service with increased scalability, reliability, and observability baked in.

A Golden Path to Production

Out of the box Pulumi Deployments offers your cloud engineering team a turn-key infrastructure delivery workflow without writing a single line of code. Every piece of infrastructure within your organization can rely on automatic cloud delivery.

`git push` to Deploy

Using the Pulumi Github App (GitLab, Bitbucket, and others coming soon), git push to Deploy enables a rich git-based workflow for safely reviewing infrastructure changes and rolling them out across environments. Opening a Pull Request on GitHub will automatically trigger a pulumi preview and post the results back to the PR. You can also enforce successful previews as a check that gates merging.

Once your PR is reviewed and merged, a Deployment runs automatically:

Setting this up is as easy as installing the Pulumi GitHub App and selecting the repository and branch for your stack:

You can ensure that every environment and every commit is automatically validated and kept up to date with just a few clicks. GitHub integration is available today, with support for GitLab, Atlassian Bitbucket, and others coming soon.

Click to Deploy

Sometimes infrastructure requires ad-hoc operations. You may want to run a refresh to rectify some cloud state changes:

Whatever the situation calls for, you can easily update, refresh, and modify infrastructure with the click of a button in the Pulumi Service console.

Power-up Your Platform with The Deployments API

Both git push to Deploy and Click to Deploy are built on top of the Pulumi Deployments REST API, the same API that is available to you as a building block. The Deployments API is fully managed and enables you to orchestrate cloud rollouts without worrying about capacity. Let’s look at a few ways you can embed it into your custom cloud platform.

Drift Detection

Manual edits within the AWS console can wreak havoc on your infrastructure if those changes aren’t brought back into source control. The last thing you want is to scale out a service just to have the next CI run undo that change.

You can build drift detection into your platform and get automated alerts on unexpected infrastructure changes.

aws.cloudwatch.onSchedule("drift-lambda", "cron(0/5 * * * ? *)", async() => {
 const outstandingDeployments = [];
 for(let s of stacks) {
 const url = `https://api.pulumi.com/api/stacks/${s.organization}/${s.project}/${s.stack}/deployments`;
 const response = await fetch(url, {
 method: "POST",
 headers,
 body: JSON.stringify(refreshPayload),
 });

 const deployment = await response.json();
 outstandingDeployments.push(deployment.id);
 }
})

This scheduled Lambda runs refresh operations through the Deployments REST API. When one of those deployments detects changes, we get notified. With platform automation that automatically and proactively detects changes, your cloud team can reduce the number of surprise incidents, freeing them up to focus on higher-leverage activities. The full source code is available for the Drift Detection example.

TTL Stacks and Temporary Infrastructure

Infrastructure waste is a common problem. It’s too easy to leave development infrastructure running accidentally and end up with a huge bill. Many companies have entire teams devoted to solving this problem.

With Pulumi Webhooks and the Deployments API we can build a system for tagging and reclaiming temporary infrastructure. Anyone in your organization can create temporary infrastructure with a simple workflow:

$ pulumi stack tag set ttl 360 # this stack expires six hours after it deploys
$ pulumi up # with the `ttl` tag preset, the infrastructure will be automatically destroyed at expiration

After the expiration period has elapsed, a deployment will be queued and the development infrastructure is automatically destroyed. The Deployments API can be embedded into your platform to reduce cloud waste and automate cost controls. The full source code is available for the TTL Stack Processor.

Managing Scale with Remote Automation API

With Automation API, Pulumi can be fully embedded inside your software projects to power a wide range of custom cloud infrastructure automation. Previously, Automation API could only run locally. With Remote Automation API, these same scripts can run locally while offloading deployment workloads to the cloud via the Deployments API. The infrastructure as code action (a preview, update, destroy or refresh) runs inside of our fully managed service.

const stack = await RemoteWorkspace.createOrSelectStack({
 stackName,
 url: "https://github.com/pulumi/examples.git",
 branch: "refs/heads/master",
 projectPath: project,
}, {
 envVars: {
 AWS_REGION: "us-west-2",
 AWS_ACCESS_KEY_ID: process.env.AWS_ACCESS_KEY_ID ?? "",
 AWS_SECRET_ACCESS_KEY: { secret: process.env.AWS_SECRET_ACCESS_KEY ?? "" },
 AWS_SESSION_TOKEN: { secret: process.env.AWS_SESSION_TOKEN ?? "" },
 },
});

const upRes = await stack.up({ onOutput: console.log });
console.log(`url: ${upRes.outputs.websiteUrl.value}`);

repo := auto.GitRepo{
 URL: "https://github.com/pulumi/examples.git",
 Branch: "refs/heads/master",
 ProjectPath: project,
}

env := map[string]auto.EnvVarValue{
 "AWS_REGION": {Value: "us-west-2"},
 "AWS_ACCESS_KEY_ID": {Value: os.Getenv("AWS_ACCESS_KEY_ID")},
 "AWS_SECRET_ACCESS_KEY": {Value: os.Getenv("AWS_SECRET_ACCESS_KEY"), Secret: true},
 "AWS_SESSION_TOKEN": {Value: os.Getenv("AWS_SESSION_TOKEN"), Secret: true},
}

s, err := auto.UpsertRemoteStackGitSource(ctx, stackName, repo, auto.RemoteEnvVars(env))
if err != nil {
 fmt.Printf("Failed to create or select stack: %v\n", err)
 os.Exit(1)
}

res, err := s.Up(ctx, optremoteup.ProgressStreams(os.Stdout))

stack = auto.create_or_select_remote_stack_git_source(
 stack_name=stack_name,
 url="https://github.com/pulumi/examples.git",
 branch="refs/heads/master",
 project_path=project,
 opts=auto.RemoteWorkspaceOptions(
 env_vars={
 "AWS_REGION": "us-west-2",
 "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
 "AWS_SECRET_ACCESS_KEY": auto.Secret(os.environ["AWS_SECRET_ACCESS_KEY"]),
 "AWS_SESSION_TOKEN": auto.Secret(os.environ["AWS_SESSION_TOKEN"]),
 },
 ),
)

up_res = stack.up(on_output=print)

var stack = await RemoteWorkspace.CreateOrSelectStackAsync(new RemoteGitProgramArgs(stackName, "https://github.com/pulumi/examples.git")
{
 Branch = "refs/heads/master",
 ProjectPath = project,
 EnvironmentVariables =
 {
 { "AWS_REGION", new EnvironmentVariableValue("us-west-2") },
 { "AWS_ACCESS_KEY_ID", RequireFromEnvironment("AWS_ACCESS_KEY_ID") },
 { "AWS_SECRET_ACCESS_KEY", RequireFromEnvironment("AWS_SECRET_ACCESS_KEY", isSecret: true) },
 { "AWS_SESSION_TOKEN", RequireFromEnvironment("AWS_SESSION_TOKEN", isSecret: true) },
 },
});

var result = await stack.UpAsync(new RemoteUpOptions { OnStandardOutput = Console.WriteLine });

Multi-region infrastructure build outs become a breeze, as workloads can be easily parallelized. Now you can run thousands of updates concurrently without having to manage the compute that runs them.

Centralized Observability

Whether you use the built-in cloud delivery workflows like Push to Deploy, or build your own custom platform with the REST API and Remote Automation API, you get full observability into every deployment. Anyone in your organization can view updates status, update logs, and the full history of deployments.

You can build platform automation without worrying about how you’ll surface the results across your product, engineering, and infrastructure teams.

Automation Accelerates the Cloud

We’ve taken the lessons from our industry-leading customers in cloud SaaS and infrastructure and packaged them into Pulumi Deployments. This suite of tools enables you to manage and scale ten times the cloud footprint as opposed to traditional IaC usage patterns. Whether you want a turn-key delivery workflow or highly-customizable platform building blocks, there is something for your organization.

Pulumi Deployments removes the cloud as a bottleneck and unlocks it as an accelerant for your business. It is available in preview today. Deployments are free during the preview, with usage-based pricing to come. Check out these resources to get started today:

Request access to the Pulumi Deployments preview
Pulumi Deployments examples repo
The Pulumi Deployments demo video
Pulumi Deployments documentation
Pulumi Deployments REST API documentation
Automation API examples

🚀🚀🚀 Happy building! 🚀🚀🚀

Stack READMEs in the Pulumi Service

Evan Boyle — Thu, 19 May 2022 09:35:01 -0700

Starting today, users can create Stack READMEs in the Pulumi Service that dynamically update based on Stack Outputs.

Each Pulumi Stack you deploy manages a key set of cloud infrastructure for your organization. The Pulumi Console includes a variety of features for exposing key information about your stack for other users within your organization - configuration, outputs, resources under management, links to cloud providers, and a graph of all resources. However, it’s often useful to allow the author of a Pulumi Stack to describe in their own words the key elements of a stack, so future viewers can quickly understand the components and cloud resources that are managed.

A README is a text file that introduces and explains a project. A Pulumi Service Stack README can contain documentation and common links for the Stack. The key difference between a source control README, such as a GitHub README, and a Pulumi Service Stack README, is that it is dynamically populated with details from your stack outputs. It does this by interpolating output variables on the stack (${outputs.instances[0].ARN}) so that each stack can construct links to dashboards, shell commands, and other pieces of documentation. All of this content stays up to date as you stand up new stacks, rename resources, and refactor your infrastructure.

There are a lot of operational activities that happen around Pulumi Stacks, but critical information is spread across many tools. We have heard from our users that it can be time consuming switching from different tools and web apps to manage their deployments- from their cloud Console, to the Pulumi Service User Interface (UI), to the CLI, and so on. By collecting resource outputs in the Pulumi Service UI, we are reducing this friction and keeping relevant Stack information in one place so that is kept up to date automatically.

README templates can reference Stack outputs and resource properties, for example ${outputs.vpc.ARN}. To walk you through a tangible example, a user can deploy an RDS instance and then use a variable in their README template to link to their CloudWatch dashboard to keep an eye on operational metrics. This CloudWatch dashboard link will dynamically update when deployments happen. You can use the same README template across dev, testing and production and have the correct dashboard links for each Stack. Learn more in the Stack READMEs documentation page.

The new experience lives in the Stack page, which can be navigated to through Projects and clicking on the specific Stack you want to view the README for. Let’s take a look at this new feature!

The Pulumi.README.md file we added to the Stack ‘Production’ above is a template README for our Pulumi Service. In it we have links to our AWS authentication tool, our CloudWatch metrics, to our production deployment documentation, and so on. The common places that we navigate to and from when managing this Stack. Here is some of the Markdown for this README:

# Pulumi Service README

[Sign in to AWS to view stack resources!](https://top-secret-url.com)

## On Call Operations

### Monitor

**Cloudwatch Metrics**
Monitor holistic metrics tracking overall service health
[Link](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#dashboards:name=${outputs.dashboardName})

**RDS Performance Metrics**
Monitor RDS performance (wait times, top queries)
[Link](https://us-west-2.console.aws.amazon.com/rds/home?region=us-west-2#performance-insights-v20206:/resourceId/${outputs.databaseClusterId}/resourceName/${outputs.rdsClusterWriterInstance})

**Cloudwatch Logs**
Search across service logs
[Link](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logStream:group=${outputs.cloudwatchLogGroup})

How to add a Stack README to your Stack

In order to add a README to your Pulumi Stack, you will need to do the following:

Step 1

Export a Stack output named readme that contains your templated Stack README markdown, commonly by reading a file, i.e. Pulumi.README.md.

import { readFileSync } from "fs";
export const strVar = "foo";
export const arrVar = ["fizz", "buzz"];
// add readme to stack outputs. must be named "readme".
export const readme = readFileSync("./Pulumi.README.md").toString();

import pulumi
pulumi.export('strVar', 'foo')
pulumi.export('arrVar', ['fizz', 'buzz'])
# open template readme and read contents into stack output
with open('./Pulumi.README.md') as f:
 pulumi.export('readme', f.read())

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 strVar := "foo"
 arrVar := []string{"fizz", "buzz"}
 readmeBytes, err := ioutil.ReadFile("./Pulumi.README.md")
 if err != nil {
 return fmt.Errorf("failed to read readme: %w", err)
 }
 ctx.Export("strVar", pulumi.String(strVar))
 ctx.Export("arrVar", pulumi.ToStringArray(arrVar))
 ctx.Export("readme", pulumi.String(string(readmeBytes)))
 return nil
 })
}

using Pulumi;
class MyStack : Stack
{
 public MyStack()
 {
 this.StrVar = "foo";
 this.ArrVar = new string[] { "fizz", "buzz" };
 this.Readme = System.IO.File.ReadAllText("./Pulumi.README.md");
 }
 [Output]
 public Output<string> StrVar { get; set; }
 [Output]
 public Output<string[]> ArrVar { get; set; }
 [Output]
 public Output<string> Readme { get; set; }
}

package stackreadme;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
public class App {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 var strVar = "foo";
 var arrVar = new String[]{ "fizz", "buzz" };
 try {
 var readme = Files.readString(Paths.get("./Pulumi.README.md"));
 ctx.export("strVar", Output.of(strVar));
 ctx.export("arrVar", Output.of(arrVar));
 ctx.export("readme", Output.of(readme));
 } catch (IOException e) {
 throw new RuntimeException(e);
 }
 });
 }
}

name: stack-readme-yaml
runtime: yaml
description: A minimal Pulumi YAML program demonstrating stack readme feature
variables:
 readme:
 Fn::ReadFile: ./Pulumi.README.md
outputs:
 strVar: foo
 arrVar:
 - fizz
 - buzz
 readme: ${readme}

Step 2

Create a README template for the Stack. In this example, we will create a Pulumi.README.md file that looks as follows:

# Stack README

Full markdown support! Substitute stack outputs dynamically so that links can depend on your infrastructure! Link to dashboards, logs, metrics, and more.

1. Reference a string stack output: ${outputs.strVar}
2. Reference an array stack output: ${outputs.arrVar[1]}

Step 3

Run pulumi up on that Stack

Step 4

Open the Pulumi Service UI, navigate to Projects and then the Stack you have updated. Once on the Stack page you will see the README tab with your README file.

Ta da! 🎉

We now have a README on the Stack.

Refer to the Stack README documentation page for more details on how to use this feature. As always, please feel free to submit feature requests and bug reports to the Pulumi Service GitHub Repo. We love hearing feedback from users about ways we can improve your productivity when using Pulumi. We look forward to seeing how you make Stack READMEs fit your needs!

The Pulumi Automation API - The Next Quantum Leap in IaC

Evan Boyle — Mon, 19 Oct 2020 00:00:00 +0000

Today’s Infrastructure as Code platforms let organizations build rich, reliable, and complex cloud-based applications and architectures. But as teams move to modern cloud technologies, they continue to search for ways to drive increased software-driven automation. Although modern Infrastructure as Code tools bring key software engineering benefits to cloud engineering, they remain focused on human-driven workflows. For example, a person running pulumi up at their terminal or wiring the Pulumi CLI into their CI/CD system.

To scale up how we deploy and manage cloud infrastructure and ultimately unlock the cloud’s value and agility, we believe it will be critical to build software systems around our Infrastructure as Code platforms - systems that scale with software, not just humans.

Pulumi’s approach of using general purpose languages enables Infrastructure as Software – bringing all of the great lessons learned over decades of building software to your infrastructure (languages, IDEs, testing, debugging, componentization, packaging, versioning, and more).

Today we’re excited to announce the Pulumi Automation API, a robust programmatic layer on top of Pulumi’s declarative Infrastructure as Software. The Automation API exposes Pulumi programs and stacks as strongly-typed and composable building blocks.

With Automation API, Pulumi can be fully embedded inside your software projects - to power a wide range of custom cloud infrastructure automation projects. No CLI, no human-in-the-loop, just code.

Enabling Cloud Engineering

Pulumi customers are already applying the Automation API to an incredible breadth of different scenarios. We continue to be amazed by the creativity that it unlocks as a new fundamental building block for cloud engineering.

So far, our users have applied the Pulumi Automation API to these scenarios:

Platform APIs - Embedding Pulumi opens up a world of possibilities for building internal and public cloud platforms that serve your infrastructure abstractions. It enables new approaches to SaaS, including IaaS and PaaS products and platforms. You can create declarative infrastructure defined by your best practices and expose it behind a REST, gRPC, or Custom Resource API that developers and operators alike can easily consume.
Complex Workflow Orchestration - Distributed systems and microservices incur a significant coordination burden, and dependencies across API boundaries are difficult to track and maintain. Automation API is the layer that orchestrates multi-stack deployments, codifying dependencies, and enabling safe incremental deployment. Networking, database systems, application code, schema migrations, and more can be coordinated in a single strongly typed, familiar programming environment.
Cloud Development Frameworks - Automation API provides building blocks in your favorite language. With them, you can author higher-level frameworks and components that blur the lines between infrastructure and application. We expect to see more frameworks that focus on application code that ambiently creates its own infrastructure.
Supercharged Ops Tooling - Create hardened and tested tooling to automate anything entering your ticketing system. From allocating timebound developer VMs to upgrading TLS certificates and scaling servers, Automation API enables you to build custom CLIs and tools that leverage the full power of Infrastructure as Software underneath custom user interfaces.

The Automation API is a new subpackage in each of Pulumi’s language-specific SDKs that provides APIs to create and manage Stacks and perform lifecycle operations like update, refresh, preview, and destroy. It is a strongly typed and safe way to use Pulumi in embedded contexts such as web servers without having to shell out to a CLI.

You can define a Pulumi program as a function within your codebase rather than in a separate project and use methods to get and set configuration parameters programmatically. The Automation API uses a gRPC interface to execute programs that control and communicate with the core Pulumi engine. It still requires a Pulumi CLI installation, as this is how we bundle and distribute the core engine. Today it’s available in preview for TypeScript/JavaScript, Go and Python, with support for C# under active development.

Platform APIs

For SaaS scenarios like deploying infrastructure in response to user action, relying on shell scripts for online services brings operational challenges with observability and reliability. Automation API is just another package and can be instrumented, profiled, and monitored with your existing APM tools like Honeycomb, Prometheus, and Datadog.

Automation API runs inside your favorite frameworks and interacts with your other packages exactly as you’d expect. We can define custom declarative infrastructure and expose it to end-users behind a familiar REST interface:

// an HTTP handler to create instances of our declarative infra
// each request creates an S3-backed static website
// with content from the POST body
const createHandler: express.RequestHandler = async (req, res) => {
 const stackName = req.body.id;
 const content = req.body.content as string;
 try {
 // create a new stack
 const stack = await LocalWorkspace.createStack({
 stackName,
 projectName,
 // generate our pulumi program on the fly from the POST body
 program: createPulumiProgram(content),
 });
 await stack.setConfig("aws:region", { value: "us-west-2" });
 // deploy the stack, tailing the logs to console
 const upRes = await stack.up({ onOutput: console.info });
 res.json({ id: stackName, url: upRes.outputs.websiteUrl.value });
 } catch (e) {
 // built in support for structured error handling
 if (e instanceof StackAlreadyExistsError) {
 res.status(409).send(`stack "${stackName}" already exists`);
 } else {
 res.status(500).send(e);
 }
 }
};

// an HTTP handler to create instances of our declarative infra
// each request creates an S3-backed static website
// with dynamic content from the POST body
func createHandler(w http.ResponseWriter, req *http.Request) {
 w.Header().Set("Content-Type", "application/json")
 var createReq CreateSiteReq
 _ := json.NewDecoder(req.Body).Decode(&createReq)

 stackName := createReq.ID
 // dynamically define our Pulumi Program
 program := createPulumiProgram(createReq.Content)

 s, err := auto.NewStackInlineSource(ctx, stackName, project, program)
 if err != nil {
 // built in support for structured error handling
 // if stack already exists, 409
 if auto.IsCreateStack409Error(err) {
 w.WriteHeader(409)
 fmt.Fprintf(w, fmt.Sprintf("stack %q already exists", stackName))
 return
 }

 w.WriteHeader(500)
 fmt.Fprintf(w, err.Error())
 return
 }
 s.SetConfig(ctx, "aws:region", auto.ConfigValue{Value: "us-west-2"})

 // deploy the stack
 upRes, err := s.Up(ctx, optup.ProgressStreams(os.Stdout))

 response := &SiteResponse{
 ID: stackName,
 URL: upRes.Outputs["websiteUrl"].Value.(string),
 }
 json.NewEncoder(w).Encode(&response)
}

# an HTTP handler to create instances of our declarative infra
# each request creates an S3-backed static website with dynamic content from the POST body
@app.route("/sites", methods=["POST"])
def create_handler():
 """creates new sites"""
 stack_name = request.json.get('id')
 content = request.json.get('content')
 try:
 def pulumi_program():
 return create_pulumi_program(content)
 # create a new stack, generating our pulumi program on the fly from the POST body
 stack = auto.create_stack(stack_name=stack_name,
 project_name=project_name,
 program=pulumi_program)
 stack.set_config("aws:region", auto.ConfigValue("us-west-2"))
 # deploy the stack, tailing the logs to stdout
 up_res = stack.up(on_output=print)
 return jsonify(id=stack_name, url=up_res.outputs['website_url'].value)
 except auto.StackAlreadyExistsError:
 return make_response(f"stack '{stack_name}' already exists", 409)
 except Exception as exn:
 return make_response(str(exn), 500)

The handlers work with your favorite HTTP frameworks like mux, express and flask:

// install necessary plugins once upon boot
ensurePlugins();

// configure express
const app = express();
app.use(express.json());

// setup our RESTful routes for our Site resource
app.post("/sites", createHandler);
app.get("/sites", listHandler);
app.get("/sites/:id", getHandler);
app.put("/sites/:id", updateHandler);
app.delete("/sites/:id", deleteHandler);

// start our http server
app.listen(1337, () => console.info("server running on :1337"));

func main() {
 ensurePlugins()

 router := mux.NewRouter()

 // setup our RESTful routes for our Site resource
 router.HandleFunc("/sites", createHandler).Methods("POST")
 router.HandleFunc("/sites", listHandler).Methods("GET")
 router.HandleFunc("/sites/{id}", getHandler).Methods("GET")
 router.HandleFunc("/sites/{id}", updateHandler).Methods("PUT")
 router.HandleFunc("/sites/{id}", deleteHandler).Methods("DELETE")

 // define and start our http server
 s := &http.Server{
 Addr: ":1337",
 Handler: router,
 }
 fmt.Println("starting server on :1337")
 log.Fatal(s.ListenAndServe())
}

import Flask

# install necessary plugins once upon boot
ensure_plugins()

# initialize our Flask app
app = Flask(__name__)

# set up our routes
@app.route("/sites", methods=["POST"])
def create_handler():
 ...

@app.route("/sites", methods=["GET"])
def list_handler():
 ...

@app.route("/sites/<string:id>", methods=["GET"])
def get_handler(id: str):
 ...

@app.route("/sites/<string:id>", methods=["UPDATE"])
def update_handler(id: str):
 ...

@app.route("/sites/<string:id>", methods=["DELETE"])
def delete_handler(id: str):
 ...

Check out the full Pulumi over HTTP example in TypeScript, Go and Python.

Here at Pulumi, we used Automation API as the foundation of our Kubernetes Operator that exposes Pulumi stacks as Custom Resources. See how your own platform’s operators can take advantage of the Automation API.

We’re seeing customers reach for Automation API to build internal cloud platforms to empower their application teams, as well as independent software vendors using it to build and sell public cloud offerings like SaaS database products.

Complex Workflow Orchestration

Choosing between small, decoupled units of infrastructure and a maintainable orchestration process is no longer necessary. The Automation API executes just like any other code in your application, meaning that a single process can deploy infrastructure and manage related activities such as database migrations and blue/green deployments.

Like any other Pulumi program, we define our cloud resources. In this case, an AWS RDS database:

const clusterInstance = new aws.rds.ClusterInstance("dbInstance", {
 clusterIdentifier: cluster.clusterIdentifier,
 instanceClass: "db.t3.small",
 engine: "aurora-mysql",
 engineVersion: "5.7.mysql_aurora.2.03.2",
 publiclyAccessible: true,
 dbSubnetGroupName: subnetGroup.name,
});

return {
 host: cluster.endpoint,
 dbName,
 dbUser,
 dbPass
};

_, err = rds.NewClusterInstance(ctx, "dbInstance", &rds.ClusterInstanceArgs{
 ClusterIdentifier: cluster.ClusterIdentifier,
 InstanceClass: pulumi.String("db.t3.small"),
 Engine: pulumi.String("aurora-mysql"),
 EngineVersion: pulumi.String("5.7.mysql_aurora.2.03.2"),
 PubliclyAccessible: pulumi.Bool(true),
 DbSubnetGroupName: subnetGroup.Name,
})
if err != nil {
 return err
}

ctx.Export("host", cluster.Endpoint)
ctx.Export("dbName", dbName)
ctx.Export("dbUser", dbUser)
ctx.Export("dbPass", dbPass)
}

cluster_instance = aws.rds.ClusterInstance(
 "db_instance",
 cluster_identifier=cluster.cluster_identifier,
 instance_class=aws.rds.InstanceType.T3_SMALL,
 engine=aws.rds.EngineType.AURORA_MYSQL,
 engine_version="5.7.mysql_aurora.2.03.2",
 publicly_accessible=True,
 db_subnet_group_name=subnet_group.name
)

pulumi.export("host", cluster.endpoint)
pulumi.export("db_name", db_name)
pulumi.export("db_user", db_user)
pulumi.export("db_pass", db_pass)

But with Automation API, we can perform database migrations and preload rows in the same program:

// create (or select if one already exists) a stack that uses our inline program
const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "dev",
 projectName: "databaseMigration",
 // our pulumi program as an "inline" function
 program: pulumiProgram
});

// run the update to deploy our database
const upRes = await stack.up({ onOutput: console.info });
console.log(`db host url: ${upRes.outputs.host.value}`);
console.info("configuring db...");

// establish mysql client
const connection = mysql.createConnection({
 host: upRes.outputs.host.value,
 user: upRes.outputs.dbUser.value,
 password: upRes.outputs.dbPass.value,
 database: upRes.outputs.dbName.value
});

connection.connect();
console.log("creating table...")

// create our table
connection.query(`
CREATE TABLE IF NOT EXISTS hello_pulumi(
 id int(9) NOT NULL,
 color varchar(14) NOT NULL,
 PRIMARY KEY(id)
);
`, function (error, results, fields) {
 if (error) throw error;
 console.log("table created!")
 console.log('Result: ', JSON.stringify(results));
});

// create our stack with an "inline" Pulumi program (deployFunc)
stack := auto.UpsertStackInlineSource(ctx, stackName, projectName, deployFunc)
// run the update to deploy our database
res, err := stack.Up(ctx, stdoutStreamer)

fmt.Println("Update succeeded!")

// get the connection info
host := res.Outputs["host"].Value
dbName := res.Outputs["dbName"].Value
dbUser := res.Outputs["dbUser"].Value
dbPass := res.Outputs["dbPass"].Value

// establish db connection
db := sql.Open("mysql", fmt.Sprintf("%s:%s@tcp(%s:3306)/%s", dbUser, dbPass, host, dbName))
defer db.Close()

// run our database "migration"
fmt.Println("creating table...")
db.Query(`
CREATE TABLE IF NOT EXISTS hello_pulumi(
 id int(9) NOT NULL,
 color varchar(14) NOT NULL,
 PRIMARY KEY(id)
);
`)

# create (or select if one already exists) a stack that uses our inline program
stack = auto.create_or_select_stack(stack_name=stack_name,
 project_name=project_name,
 program=pulumi_program)

up_res = stack.up(on_output=print)
print(f"db host url: {up_res.outputs['host'].value}")

print("configuring db...")
with connect(
 host=up_res.outputs['host'].value,
 user=up_res.outputs['db_user'].value,
 password=up_res.outputs['db_pass'].value,
 database=up_res.outputs['db_name'].value) as connection:
 print("db configured!")

 # make sure the table exists
 print("creating table...")
 create_table_query = """CREATE TABLE IF NOT EXISTS hello_pulumi(
 id int(9) NOT NULL PRIMARY KEY,
 color varchar(14) NOT NULL);
 """
 with connection.cursor() as cursor:
 cursor.execute(create_table_query)
 connection.commit()

See the full database deployment and migration Automation API program in TypeScript, Go and Python.

Cloud Framework Development

The Automation API removes any distinction between infrastructure and application code. Its unique form factor lends all of the predictability and reliability of immutable infrastructure and the desired state configuration you expect from Pulumi while naturally enabling automation and composition in a way that CLIs and DSLs were never designed to handle.

We prototyped a Heroku-like developer experience called halloumi that demonstrates this. All you worry about is writing HTTP handlers. Halloumi worries about the rest, including orchestrating cross-service dependencies, networking, and deploying docker images to ECS Fargate or serialized JavaScript to AWS Lambda.

// this application will deploy multiple services publicly available in the cloud
const application = new App();

const svc1 = new WebService("hello", (expressApp: express.Express) => {
 // a simple static handler
 expressApp.get('/', (req, res) => res.send("omg i'm alive\n"));
});
application.addService(svc1);


const svc2 = new WebService("world", (expressApp: express.Express) => {
 expressApp.get('/', async (req, res) => {
 // track dependencies across services and make them available at runtime
 const svc1Url = svc1.discover();
 // this handler calls svc1 and passes the result through
 const result = await (await fetch(svc1Url)).text()

 res.send(`this is the world. With a window from hello:\n${result} \n`);
 });
});
application.addService(svc2);

application.run().catch(err => { console.error(err) });

app := app.NewApp("petStore")

// a cloud web service that returns a number [0-100)
randomNumberService := web.NewWebService("randomNumber", func() http.Handler {
 // a normal HTTP handler, halloumi takes care of running this in the cloud
 r := mux.NewRouter()
 handler := func(w http.ResponseWriter, r *http.Request) {
 rand.Seed(time.Now().UnixNano())
 fmt.Fprintf(w, "%d", rand.Intn(100))
 }
 r.HandleFunc("/", handler)
 return r
})
app.Register(randomNumberService)

// a cloud web service that returns N of a random animal.
// this service takes a dependency on the URLs of the previously defined services
nAnimalsService := web.NewWebService("nAnimals", func() http.Handler {
 r := mux.NewRouter()
 // we'll read these variables in via requests to other services
 var num string
 var animal string
 handler := func(w http.ResponseWriter, r *http.Request) {
 // Notice how we have the URL of randomNumberService
 // available to consume here!
 numResp := http.Get(randomNumberService.URL())
 num = readResponse(numResp)

 // Notice how we have the URL of randomAnimalService
 // available to consume here!
 animalResp, err := http.Get(randomAnimalService.URL())
 animal = readResponse(animalResp)

 fmt.Fprintf(w, "Wow, you got %s %ss!", num, animal)
 }
 r.HandleFunc("/", handler)
 return r
})

Check out the full halloumi source in TypeScript and Go.

Like any other framework, you can even set a breakpoint to debug orchestration logic or the pulumi program itself, including Apply callbacks.

Supercharged Ops Tooling

Cloud infrastructure is beyond the days of driving deployments and infrastructure by hand, even though many of our tools are CLIs for manual workflows. Cloud engineering demands automated tools to keep up with growth, not just in terms of pace but also in complexity.

Have a look at our temporary development VM provisioner for Azure vmgr. A simple Cobra CLI command creates a new stack with a VM ready to go:

func NewAddCmd() *cobra.Command {
 return &cobra.Command{
 Use: "add",
 Short: "add deploys an additional vm stack",
 Run: func(cmd *cobra.Command, args []string) {
 stackName := fmt.Sprintf("vmgr%d", rangeIn(10000000, 99999999))
 // create a new stack for our VM with an Inline program.
 // the program is nil for now, but we'll set it later after retrieving some dependent outputs.
 s, err := auto.NewStackInlineSource(ctx, stackName, projectName, nil /* Program */)
 fmt.Println("ensuring network is configured...")
 // lookout network and resource group info
 subnetID, rgName, err := EnsureNetwork(ctx, projectName)
 // set our program for the deployment with the resulting network info
 stack.SetProgram(GetDeployVMFunc(subnetID, rgName))
 fmt.Println("deploying vm...")
 // deploy our VM stack
 stdoutStreamer := optup.ProgressStreams(os.Stdout)

 res, err := s.Up(ctx, stdoutStreamer)
 if err != nil {
 fmt.Printf("Failed to deploy vm stack: %v\n", err)
 os.Exit(1)
 }
 fmt.Printf("deployed server running at public IP %s\n", res.Outputs["ip"].Value)
 },
 }
}

Our vmgr cron environment cleanup job takes advantage of Pulumi’s declarative model. It queries the Pulumi Service for stacks older than five days and removes them along with any child resources:

// query all stacks in the project
stacks := w.ListStacks(ctx)
expiration := time.Now().Add(-1 * FIVE_DAYS)
var stacksToDestroy []string
for _, stackSummary := range stacks {
 lastUpdateTime := time.Parse(time.RFC3339, stackSummary.LastUpdate)
 // mark all stacks past expriation date for deletion
 if lastUpdateTime.Before(expiration) {
 stacksToDestroy = append(stacksToDestroy, stackSummary.Name)
 fmt.Printf("Found expired stack %s last deployed at %s\n", stackSummary.Name, stackSummary.LastUpdate)
 }
}

fmt.Printf("Found %d stacks to clean up\n", len(stacksToDestroy))

success := 0
for _, sName := range stacksToDestroy {
 stack := auto.SelectStack(ctx, sName, w)

 fmt.Printf("destroying stack %s\n", sName)

 // tail logs to stdout
 stdoutStreamer := optdestroy.ProgressStreams(os.Stdout)
 // destroy the expired stack
 stack.Destroy(ctx, stdoutStreamer)
 success++
 fmt.Printf("removing stack %s and all associated config and history\n", sName)
 w.RemoveStack(ctx, sName)
 fmt.Printf("stack %s successfully reaped \n", sName)
}

fmt.Printf("destroyed %d stack(s)\n", success)

Our VM provisioner uses Automation API, backed by Pulumi’s desired state model. This means that there’s no chance of resources leaking or getting lost. Check out the full vmgr example here. For a cloud-native example, take a look at our eksctl inspired aksctl for Azure AKS.

Give it a Try Today!

The Automation API is your tool to tame Cloud Engineering complexity and give your team the leverage to automate your cloud infrastructure. It is completely open source and available today for TypeScript/JavaScript, Go and Python. Want to learn more? Come hang out with us in the #automation-api community slack channel. Download the latest Pulumi release and check out these resources to get started:

Go documentation
JavaScript/TypeScript documentation
Python documentation
The Automation API examples repo
The list of known issues. Please file more as you find them!
The Pulumi Kubernetes Operator
aksctl
Halloumi

Keep an eye out! C# support is under active development and coming soon.

We’d like to give a special thank you to the community members who have consulted with us, chimed in on the original GitHub issue, and have been there for the journey in community slack.

🏭 🏭 🏭 Happy automating! 🏭 🏭 🏭

Welcoming Go to the Pulumi Family

Evan Boyle — Tue, 19 May 2020 00:00:00 +0000

Over the last 10 years, Go has quickly become the “language of the cloud” for building application servers and services that run in and on today’s cloud platforms. With Pulumi 2.0, Go can also be used to manage and provision modern infrastructure as well. Across any cloud (AWS, Azure, GCP, Kubernetes and more than 50 others!) and across a variety of workloads (containers, serverless, kubernetes, core infrastructure and more), you can now use the rich software engineering strengths of the Go language and ecosystem to manage your cloud infrastructure. The Pulumi open source project itself has been built on Go from day 1, and so we’re really excited to bring full Go support for cloud infrastructure as code to the same language ecosystem that Pulumi itself has been part of.

What is Pulumi?

Pulumi is a modern infrastructure as code tool for the cloud. It is built around four key ideas.

Programming Languages. Pulumi offers best-in-class productivity and software engineering capabilities by allowing you to use existing rich programming languages like JavaScript, Python, .NET and of course Go. Pulumi combines this with still getting the reliability of a desired-state infrastructure as code. The best of both worlds!
Multi-Cloud. Give operators the tools to build consistent workflows across any cloud(s). Whether public, private, or hybrid — you can use the full breadth and depth of any cloud services.
Modern Application Architectures. Pulumi fully supports legacy workloads, but it is designed for the world of modern containers, serverless, and cloud-native architectures. Build the architecture that works for you by mixing and matching infrastructure components.
Cloud Engineering Teams. Pulumi is for the entire team; everyone ranging from developers, infrastructure and operations engineers, and security engineers can use Pulumi. Together, they can tackle cloud solutions together without technology or team silos.

Giving you superpowers

Pulumi gives you the software engineering superpowers of the Go programming language for working with cloud infrastructure. But what can you do with them? In this post, we’ll walk through a few vignettes of how Pulumi can be used to work with serverless, containers and Kubernetes workloads across GCP, Azure, AWS and self-managed clouds.

Serverless

Pulumi has first-class support for serverless workloads, allowing your application and infrastructure code to live side by side (and both be written in Go if you want!). Easily deploy a lambda running a Go handler on AWS:

args := &lambda.FunctionArgs{
 Handler: pulumi.String("handler"),
 Role: role.Arn,
 Runtime: pulumi.String("go1.x"),
 Code: pulumi.NewFileArchive("./handler/handler.zip"),
 }

// Create the lambda using the args.
function, err := lambda.NewFunction(
 ctx,
 "basicLambda",
 args,
 pulumi.DependsOn([]pulumi.Resource{logPolicy}),
 )
if err != nil {
 return err
 }

// Export the lambda ARN.
ctx.Export("lambda", function.Arn)

And with Pulumi’s multi-cloud approach you can do the equivalent on Google Cloud:

// Set arguments for creating the function resource.
args := &cloudfunctions.FunctionArgs{
 SourceArchiveBucket: bucket.Name,
 Runtime: pulumi.String("go111"),
 SourceArchiveObject: bucketObject.Name,
 EntryPoint: pulumi.String("Handler"),
 TriggerHttp: pulumi.Bool(true),
 AvailableMemoryMb: pulumi.Int(128),
}

// Create the function using the args.
function, err := cloudfunctions.NewFunction(ctx, "basicFunction", args)
if err != nil {
 return err
}

// Export the trigger URL.
ctx.Export("function", function.HttpsTriggerUrl)

Containers

Pulumi makes it easy to work with containers. To start with, you can build and publish your docker images as part of a Pulumi deployment. This example uses the Azure Container Registry:

// Create a registry.
registryArgs := containerservice.RegistryArgs{
 ResourceGroupName: resourceGroup.Name,
 AdminEnabled: pulumi.Bool(true),
 Sku: pulumi.String("Premium"),
}
registry, err := containerservice.NewRegistry(ctx, "registry", &registryArgs)
if err != nil {
 return err
}

// Create the docker image.
imageArgs := docker.ImageArgs{
 ImageName: pulumi.Sprintf("%s/mynodeapp:v1.0.0", registry.LoginServer),
 Build: &docker.DockerBuildArgs{
 Context: pulumi.String("./app"),
 },
 Registry: &docker.ImageRegistryArgs{
 Server: registry.LoginServer,
 Username: registry.AdminUsername,
 Password: registry.AdminPassword,
 },
}
image, err := docker.NewImage(ctx, "node-app", &imageArgs)
if err != nil {
 return err
}

You can even deploy a container into your cloud environment based on that image from within the same Pulumi program ensuring that application and infrastructure deployments are always in sync:

portArgs := containerservice.GroupContainerPortArgs{
 Port: pulumi.Int(80),
 Protocol: pulumi.String("TCP"),
}
containerArgs := containerservice.GroupContainerArgs{
 Cpu: pulumi.Float64(0.5),
 Image: image.ImageName,
 Memory: pulumi.Float64(1.5),
 Name: pulumi.String("hello-world"),
 Ports: containerservice.GroupContainerPortArray{portArgs},
}
groupArgs := containerservice.GroupArgs{
 ResourceGroupName: resourceGroup.Name,
 ImageRegistryCredentials: containerservice.GroupImageRegistryCredentialArray{credentialArgs},
 OsType: pulumi.String("Linux"),
 Containers: containerservice.GroupContainerArray{containerArgs},
 IpAddressType: pulumi.String("public"),
 DnsNameLabel: pulumi.String("acigo"),
}
group, err := containerservice.NewGroup(ctx, "aci", &groupArgs)
if err != nil {
 return err
}

ctx.Export("endpoint", group.Fqdn)

Kubernetes

Pulumi provides access not just to your cloud (AWS, Azure, GCP and more), but also to the whole Kubernetes API. The Pulumi Kubernetes provider is based on the OpenAPI specification, so is always up to date with the full functionality of the Kubernetes API. Using Pulumi to provision your Kubernetes resources gives you the declarative benefits of YAML without any of the drawbacks. Using a familiar, general-purpose language to orchestrate resources brings the benefits of strong typing, safety, and opens up options for encapsulation. Here’s a snippet from the guestbook example:

// Redis leader Service
_, err = corev1.NewService(ctx, "redis-leader", &corev1.ServiceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("redis-master"),
 Labels: redisLeaderLabels,
 },
 Spec: &corev1.ServiceSpecArgs{
 Ports: corev1.ServicePortArray{
 corev1.ServicePortArgs{
 Port: pulumi.Int(6379),
 TargetPort: pulumi.Int(6379),
 },
 },
 Selector: redisLeaderLabels,
 },
})
if err != nil {
 return err
}

With the expressiveness of go, we have the power to encapsulate services and deployments behind higher-order components:

type ServiceDeployment struct {
 pulumi.ResourceState

 FrontendIP pulumi.StringPtrOutput
 Deployment *appsv1.Deployment
 Service *corev1.Service
}

type ServiceDeploymentArgs struct {
 AllocateIPAddress pulumi.Bool
 Image pulumi.StringInput
 IsMinikube pulumi.Bool
 Ports pulumi.IntArrayInput
 Replicas pulumi.IntPtrInput
}

func NewServiceDeployment(...){...}

This allows us to simplify boilerplate and design strongly-typed APIs for our Kubernetes infrastructure (in this case, combining a Service and a Deployment into a single, simple, API with nice defaults for our use case):

_, err := NewServiceDeployment(ctx, "redis-leader", &ServiceDeploymentArgs{
 Image: pulumi.String("k8s.gcr.io/redis:e2e"),
 Ports: pulumi.IntArray{pulumi.Int(6379)},
})

Kubernetes + Infra

The power doesn’t stop with managing Kubernetes resources. You can provision cloud resources right alongside. Here we use Go to provision an EKS cluster:

// Create EKS Cluster
eksCluster, err := eks.NewCluster(ctx, "eks-cluster", &eks.ClusterArgs{
 RoleArn: pulumi.StringInput(eksRole.Arn),
 VpcConfig: &eks.ClusterVpcConfigArgs{
 PublicAccessCidrs: pulumi.StringArray{
 pulumi.String("0.0.0.0/0"),
 },
 SecurityGroupIds: pulumi.StringArray{
 clusterSg.ID().ToStringOutput(),
 },
 SubnetIds: toPulumiStringArray(subnet.Ids),
 },
})

Try Go + Pulumi Today

The magic doesn’t stop here. There’s a wealth of examples, and we’re excited to see where the community takes it. If you’d like to try out Pulumi today, you can get started with your favorite cloud here:

We’d love to hear what you think. We look forward to hanging out on Pulumi Slack or hearing from your thoughts in a GitHub issue.

Modern Cloud Infrastructure in Go - The Road to 2.0

Evan Boyle — Thu, 27 Feb 2020 00:00:00 +0000

Here at Pulumi, everyone on our engineering team is a Gopher. Go has quickly become the “language of the cloud,” and so when we chose to build our open-source pulumi/pulumi engine and SaaS backend, we chose Go. As such, we are very excited to welcome Go to the family of supported infrastructure as code languages as part of Pulumi 2.0.

What is Pulumi?

Go has become the lingua franca of cloud-native infrastructure development. From projects like Docker, to Kubernetes, the community has embraced Go. The result is an ecosystem rich in tooling and packages designed for systems and cloud engineering.

Pulumi lets you use familiar languages to express your application’s infrastructure needs, using a powerful technique called “infrastructure as code.” Using infrastructure as code, you declare desired infrastructure, and an engine provisions it for you so that it’s automated, easy to replicate, and robust enough for demanding production requirements. Pulumi improves this process by leveraging general-purpose languages and making modern cloud infrastructure patterns, such as containers and serverless programs, first-class and easy.

Automatically create, update, or delete cloud resources using Pulumi’s infrastructure as code engine, removing manual point-and-clicking in the cloud provider consoles, UIs, and ad-hoc scripts.
Use your favorite IDEs and tools, including Visual Studio Code, taking advantage of features like auto-completion, refactoring, and interactive documentation.
Catch mistakes early on with standard compiler errors, and an infrastructure-specific policy engine for enforcing security, compliance, and best practices.
Reuse any existing go packages, or distribute your own, whether that’s for infrastructure best practices, productivity, or just general programming patterns.
Build scalable cloud applications using classic infrastructure cloud-native technologies. including serverless functions, and highly scalable databases such as AWS Aurora, into your core development experience, bringing them closer to your application code.

Pulumi’s free open source SDK, which includes a CLI and assortment of libraries, enables these capabilities.

Creating an AWS S3 Bucket is a familiar and simple example that demonstrates these advantages:

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an AWS resource (S3 Bucket)
 bucket, err := s3.NewBucket(ctx, "my-bucket", nil)
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("bucketName", bucket.ID())
 return nil
 })
}

Although this code looks remarkably similar to using your cloud’s Go SDK to imperatively change infrastructure, there is a major and important difference: this is declarative infrastructure as code. This code declares to Pulumi a goal state for our infrastructure which Pulumi can then carry out. That means we can preview changes with full diffs before they are made, that we always have a history of changes we’ve made to our infrastructure, and that it works both for the initial provisioning of a new environment as well as revisions to an existing one.

An Expressive and Idiomatic Type System

Making Go a first-class citizen is a goal for the Pulumi 2.0 release. We especially wanted Go to have more idiomatic and stronger typing than interface{}.

Pulumi’s programming model is inherently asynchronous. Consider creating a new AWS SecurityGroup, and using it to provision an EC2 instance. The EC2 instance must wait for the Group to finish provisioning, and the Pulumi engine must track this dependency. While two resources, modern cloud deployments commonly manage hundreds of resources. Pulumi models these resources using promise-like wrappers referred to as Inputs and Outputs allowing us to build a dependency graph, and parallelize cloud resource provisioning where possible.

Inputs, Outputs, and functions that operate over them (All, Apply, Sprintf) must be compatible with raw type primitives (string, integer, list, map, boolean), and user-defined structs. In other words, we must support polymorphism in a language that doesn’t have generics. We hide the gritty details behind a rich, strongly-typed API that we were able to iterate on easily thanks to the utilization of code generation.

Try Pulumi + Go Today

Although Pulumi for Go is in “preview” status until 2.0 is released, it supports all of the essential Pulumi programming model features (and the rest is on its way). Our goal is to gather feedback and over the next few weeks, and work hard to improve the Go experience across the board, including more examples and better documentation. To get started check out:

GoDoc
Getting started with Go and AWS, Azure, or Google Cloud.
Author and deploy serverless functions in Go on AWS Lambda and GCP Cloud Functions.
Build a serverless container-based application on AWS ECS Fargate.
Deploy a web server on a virtual machine with AWS or Azure.
Deploy a Kubernetes cluster using Azure Kubernetes Service.

To keep up with our progress, follow Pulumi on Github, and feel free to open a new issue with ideas, feedback, and bugs. Join the community in Slack to discuss your scenarios, and to get any needed assistance from the team and other end users.

Coming Soon

Some exciting things to keep an eye out for:

AWS Serverless Analytics

Evan Boyle — Thu, 30 Jan 2020 00:00:00 +0000

Whether it’s an IoT installation, a website, or a mobile app, modern software systems generate a trove of usage and performance data. While it can be daunting to collect and manage, surfacing data empowers the business to make informed product investments. In this article, we’ll explore the following:

An overview of the traditional Redshift analytics stack on AWS, the use cases it excels at, and where it falls apart.
An alternative architecture utilizing serverless and streaming.
How to implement this architecture as code in a reusable library with Pulumi.
How to automate the development loop when writing Pulumi libraries.

If you’d like to follow along, you can clone and run the reference implementation. If you’re new to Pulumi, you can follow this guide to get started.

ACME E-Commerce

ACME is a burgeoning online retailer. They ship goods around the world to millions of happy customers. ACME has recently started an incubation project to test out running ads on its website. They are unsure of how customers will react or whether their merchants will be interested in participating, so they staff a small team of 5 tasked with launching an MVP.

ACME wants to track the performance of its ads, understand how its customers are interacting with them, and look for patterns that can inform business decisions and further investment. With such a small team, they look for the path of least resistance as analytics is just a small slice of the overall work to be done to launch on their aggressive timeline.

The Traditional Analytics Stack

A familiar SQL interface and the ease of loading data through built-in features like S3 COPY have made Redshift the goto choice for bootstrapping an analytics stack on new projects. It’s low friction to get from zero to something, and Redshift scales to petabytes.

Like all tools, Redshift has targeted use cases that it excels at. Broadly speaking, it’s niche is heavy BI and analytics workloads, reading more than you write. Redshift can serve millisecond-level queries on massive datasets through a horizontally scalable, distributed processing architecture. At the core of this is a reliance on sorting and data distribution mechanisms. Schemas are configured by administrators to distribute the data across the cluster in a variety of methods that optimize for different use cases, such as highly denormalized tables, or tables making heavy use of joins. Distributed data is stored in a specified sort order on disk, allowing the query planner to make optimizations.

New data is sorted locally when a COPY command is run, but this doesn’t mean that the global sort order is maintained. A VACUUM and ANALYZE command must be run to re-sort the data, rebuild indexes, and update metadata used by the query planner. It can be a resource-intensive process relative to how much data you’ve loaded, and how it’s distributed amongst your existing sort key range. This limits the effective use of Redshift.

ACME Ads MVP

The team defines its initial product. They’ll begin with testing ads in the search results of ACME’s North American website. When customers search for “Soap”, advertisers can bid to have their product show up first. They’ve picked a cost per click (CPC) billing model, where advertisers pay a small sum whenever a user clicks on their ad. To support this, they’ve designed an initial architecture:

They’ve defined the following key events to track:

Impression: Emitted when the ad is rendered on a customer device.
Click: Emitted when the customer clicks on the product shown in the ad.

The shopping web app has a mechanism to deliver these events into an S3 bucket. On an as-needed basis, engineers on the team can manually run a COPY command to refresh the data in the cluster. They do this on a monthly cadence to generate invoices to send to advertisers, and sometimes more frequently to generate ad-hoc reports on ad performance.

Scaling a Real Business

The team is delighted to find that the ads are performing well. Advertisers see a return on their investment, and data shows that the impact on the consumer experience is minimal. The ACME team has scaled the number of advertisers from five in the beta program, to over a thousand. The team struggled with the ad-hoc nature of its MVP analytics stack and has invested heavily in automation:

Loading of event data (pre-impression, clicks, etc.) into Redshift has been automated into an hourly cron that runs an incremental COPY.
Billing is automated to run on an hourly basis, to limit the amount of time an advertiser can be allowed to serve ads after exceeding their allocated budget.

The automated load, in addition to the data generated by the recently hired Product Manager, Business Intelligence Analyst, and Data Scientist, strained the Redshift deployment. It has quickly become the most expensive part of their system, not just in terms of dollars spent, but operator time as well. An overloaded cluster leads to a variety of operational issues that the team must learn to detect and mitigate:

Data delivery is regularly delayed: COPY commands that used to take minutes are taking over an hour when the cluster is under load. Dependent cron jobs over the data need to be updated to handle delays.
The cluster must be resized regularly to keep up with growth: The team often uses Elastic Resize operations, which require a few minutes of downtime and only allow the cluster to increase in size by factors of two. Upgrading node types to better match the workload requires downtime as a new cluster is stood up, and data is copied over.
Ad-hoc workloads interfere with production: Business users write sophisticated queries that accidentally bring the cluster to a standstill. The team puts a complicated WLM policy in place to try and isolate ad-hoc from production, but this is more of an art than a science.

Designing the V2 Analytics Stack

The team understands how to keep the analytics stack limping along at this point, but the cost in both hardware and developer time is too high. They decide to pursue a redesign with the following goals in mind:

Little to no operator intervention required. Developer time is their most valuable asset, and going forward, they want to focus engineering on delivering new features to advertisers, and new experiences to customers.
Scale the analytics stack to 1B events per day and beyond as the program grows in terms of advertisers, and geography.
Enable workload isolation between production services and ad-hoc business intelligence users.

JSON events are queued in Kinesis, consumed by a Firehose that validates their schema against the definition in Glue, then converted into parquet, and finally stored in S3 in hourly folders based on arrival time. Structured data in S3 and a Glue metastore are flexible primitives that end-users and production systems can build on. Both production systems and ad-hoc users can bring their own compute or take advantage of serverless solutions like Athena (the AWS serverless version of Presto) to query over the data with isolation. The operator overhead is low because it relies almost entirely on serverless offerings. This architecture brings some additional benefits:

Scale the read and write workloads independently. Redshift was overprovisioned to handle the mismatch of a continuous write workload at peak hours. A decrease in operating cost was the result of adopting this architecture.
Pay as you go. Firehose, S3, Lambda, and Athena are all billed based on usage. Kinesis is the exception charging per shard hour, which for most deployments would be provisioned for peak load.
Configurable latency. Firehose can dump batches of data in intervals of 1-15 minutes. If low latency is a concern, you can achieve it (see the end for caveats on compaction). We don’t have to worry about an increased write workload impacting reads.
Malformed records won’t block the data ingestion pipeline. Redshift COPY has a maxerror that allows skipping over bad records, but there isn’t any automation to isolate and alert on those records. Firehose writes unsuccessful events to a separate directory and allows processing to continue.

While our new serverless streaming architecture brings a slew of benefits that enable ease of operation and the next level of scale, there are some drawbacks. For one, there is substantially more infrastructure to configure than the single Redshift cluster. How do we implement this in a repeatable fashion?

Architecture as Code with Pulumi

We want repeatable infrastructure that we can reliably develop, test, and deploy. Creating a single Redshift instance through the console was manageable, but our serverless architecture utilizes multiple instances of five AWS services. Serverless solutions frequently require many more granular resources, and thus the complexity of the infrastructure as code increases, and we need more robust tools to manage this complexity - just like we manage software complexity.

Pulumi takes care of the complexity by bringing software development best practices like abstraction, encapsulation, and modularity to infrastructure. You can use design patterns in your favorite programming language to provision modern cloud infrastructure. Component Resources are our tools for encapsulating low-level cloud infrastructure into Architecture as Code that is shareable across teams.

With Pulumi the code we write looks very similar to our architecture diagram:

import { ServerlessDataWarehouse, StreamingInputTableArgs }
 from "./datawarehouse";

// declare our shared schema for clicks and impressions
const columns = [
 {
 name: "id",
 type: "string",
 },
 {
 name: "session_id",
 type: "string",
 },
 {
 name: "event_time",
 type: "string",
 },
];

const impressionsTableName = "impressions";
const clicksTableName = "clicks";

const tableArgs: StreamingInputTableArgs = {
 columns,
 inputStreamShardCount: 1,
 region: "us-west-2",
};

// create two tables with kinesis input streams,
// writing data into hourly partitions in S3.
const dataWarehouse = new ServerlessDataWarehouse("analytics_dw", { isDev })
 .withStreamingInputTable(impressionsTableName, genericTableArgs)
 .withStreamingInputTable(clicksTableName, genericTableArgs);

// Make input streams available to external programs as stack outputs
export const impressionInputStream = impressionsInputStream.name;
export const clickInputStream = clicksInputStream.name;

In a dozen lines of code, we’ve provisioned two of our desired tables, “clicks” and “impressions”, using our serverless streaming input architecture. It takes care of everything, including implicitly creating an arrival time partition scheme with the key “inserted_at”. We export our Kinesis input streams as stack outputs that can be referenced in other projects, such as the instrumentation within our ad server or consumer-facing web app.

While on the surface this Pulumi component is described imperatively, it produces a declarative output in the form of a state file that can be managed locally, in an object store like S3, or by the Pulumi Service backend. Running a ‘pulumi up’ shows that we’ve created 45 AWS resources, and lists our stack outputs to the console.

After putting some sample events into Kinesis, we can use Athena to query our impressions table:

What does ServerlessDataWarehouse look like under the hood? Just like any other typescript class:

export class ServerlessDataWarehouse extends pulumi.ComponentResource {

 public dataWarehouseBucket: aws.s3.Bucket;
 public queryResultsBucket: aws.s3.Bucket;
 public database: aws.glue.CatalogDatabase;
 private tables: { [key: string]: aws.glue.CatalogTable } = {};
 private inputStreams: { [key: string]: aws.kinesis.Stream } = {};

 constructor(name: string,
 args: DataWarehouseArgs = {},
 opts: pulumi.ComponentResourceOptions = {}
 ) {
 super("serverless:data_warehouse", name, opts);

 const bucketArgs: BucketArgs | undefined = args.isDev ?
 { forceDestroy: true } : undefined;

 const dataWarehouseBucket = new aws.s3.Bucket("datawarehouse-bucket",
 bucketArgs, { parent: this });
 const queryResultsBucket = new aws.s3.Bucket("query-results-bucket",
 bucketArgs, { parent: this });

 const database = args.database || new aws.glue.CatalogDatabase(name, {
 name,
 }, { parent: this });


 this.dataWarehouseBucket = dataWarehouseBucket;
 this.queryResultsBucket = queryResultsBucket;
 this.database = database;
 }
}

The ServerlessDataWarehouse class creates a shared S3 data bucket, Athena query results bucket, and Glue database. We’ve exposed a fluent method, .withStreamingInputTable(), that creates the Glue table, Kinesis input stream, Firehose destination stream, and the Lambda cron that registers the hourly partitions. Many of these pieces themselves are also encapsulated into higher-order components.

public withStreamingInputTable(name: string, args: StreamingInputTableArgs):
ServerlessDataWarehouse {
 const {
 partitionKeyName, columns,
 inputStreamShardCount, fileFlushIntervalSeconds
 } = args;
 const partitionKey = partitionKeyName || "inserted_at";

 const partitionKeys: input.glue.CatalogTablePartitionKey[] = [{
 name: partitionKey,
 type: "string",
 }];

 const tableArgs: TableArgs = {
 columns,
 partitionKeys,
 };

 this.withTable(name, tableArgs);

 const streamArgs: InputStreamArgs = {
 destinationBucket: this.dataWarehouseBucket,
 shardCount: inputStreamShardCount,
 databaseName: this.database.name,
 tableName: name,
 fileFlushIntervalSeconds,
 };

 const { inputStream } = new InputStream(`inputstream-${name}`,
 streamArgs, { parent: this });
 this.inputStreams[name] = inputStream;

 const registrarArgs: PartitionRegistrarArgs = {
 database: this.database,
 partitionKey,
 region: args.region,
 dataWarehouseBucket: this.dataWarehouseBucket,
 athenaResultsBucket: this.queryResultsBucket,
 table: name,
 scheduleExpression: args.partitionScheduleExpression,
 };
 const partitionRegistrar = new HourlyPartitionRegistrar(
 `${name}-partitionregistrar`, registrarArgs, { parent: this });

 return this;
}

Our withStreamingInputTable call makes use of another fluent API, .withTable(), that encapsulates the logic around various storage and SerDe details required for storing different formats in glue (Parquet vs. JSON) in a way that Athena can understand:

public withTable(name: string, args: TableArgs): ServerlessDataWarehouse {
 if (this.tables[name]) {
 throw new Error(`Duplicate table! Name: ${name}`);
 }

 const dataFormat = this.validateFormatAndGetDefault(args.dataFormat);

 const table = this.createTable(name, args.columns,
 dataFormat, args.partitionKeys);
 this.tables[name] = table;

 return this;
}

private createTable(name: string,
 columns: input.glue.CatalogTableStorageDescriptorColumn[],
 dataFormat: DataFormat,
 partitionKeys?: input.glue.CatalogTablePartitionKey[]):
 aws.glue.CatalogTable {
 const location = getS3Location(this.dataWarehouseBucket, name);

 const parquetStorageDescriptor = {
 location,
 inputFormat:
 "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
 outputFormat:
 "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
 serDeInfo: {
 parameters: { "serialization.format": "1" },
 name: "ParquetHiveSerDe",
 serializationLibrary:
 "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
 },
 columns,
 };

 const jsonStorageDescriptor = {
 location,
 inputFormat:
 "org.apache.hadoop.mapred.TextInputFormat",
 outputFormat:
 "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
 serDeInfo: {
 name: "OpenXJSONSerDe",
 serializationLibrary: "org.openx.data.jsonserde.JsonSerDe",
 },
 columns,
 };

 const storageDescriptor = dataFormat === "JSON" ?
 jsonStorageDescriptor : parquetStorageDescriptor;
 return new aws.glue.CatalogTable(name, {
 name: name,
 databaseName: this.database.name,
 tableType: "EXTERNAL_TABLE",
 storageDescriptor,
 partitionKeys,
 }, { parent: this });
}

Encapsulating core functionality like this enables other analytics use cases, and can act as an escape hatch for more complex ones. Users can create a data warehouse with component owned and managed streaming tables, use .withTable() to enable the same database to own the table, and then write custom external logic to populate that table.

// create a static fact table
const factTableName = "facts";
const factColumns = [
 {
 name: "thing",
 type: "string",
 },
 {
 name: "color",
 type: "string",
 },
];

const factTableArgs: TableArgs = {
 columns: factColumns,
 dataFormat: "JSON",
};

dataWarehouse.withTable("facts", factTableArgs);

// Load a static facts file into the facts table.
const data = `{"thing": "sky", "color": "blue"}\n` +
 `{ "thing": "seattle sky", "color": "grey"}\n` +
 `{ "thing": "oranges", "color": "orange"}`;

const factJSON = new aws.s3.BucketObject("factsFile", {
 bucket: dataWarehouse.dataWarehouseBucket,
 content: data,
 key: `${factTableName}/facts.json`,
});

Here we create a static fact table that never changes. While this population mechanism is simple, you could imagine something much more sophisticated, like a sequence of EMR jobs instead.

Creating a Batch Table API

We now have a very opinionated streaming table API (withStreamingInputTable), and an escape hatch (withTable). Can we provide something for the general batch use case? It is quite a significant problem, but we can come up with a useful API if we constrain it a bit:

Execute user-specified code on a user-specified interval.
The runtime platform is AWS Lambda. Job must execute within Lambda resource constraints for time, memory, CPU, and disk (15 mins, 3 GB, and 500 MB at the time of writing).
The user must specify policies to attach to the lambda execution role for access to resources needed during the job (S3, Athena, and others).

public withBatchInputTable(name: string, args: BatchInputTableArgs):
ServerlessDataWarehouse {
 const {
 columns, partitionKeys, jobFn, scheduleExpression,
 policyARNsToAttach, dataFormat
 } = args;

 const tableArgs: TableArgs = {
 columns,
 partitionKeys,
 dataFormat,
 };

 // create a table
 this.withTable(name, tableArgs);

 const lambdaCronArgs: LambdaCronJobArgs = {
 jobFn,
 scheduleExpression,
 policyARNsToAttach,
 };

 // configure a lambda to execute on a schedule and
 // run the function to write records to the table
 const batchInputJob = new LambdaCronJob(name, lambdaCronArgs,
 { parent: this });

 return this;
}

Our .withBatchInputTable() API creates a table and schedules a lambda that writes data into the table in batches. In our example below, we create a function that queries our clicks and impressions streaming tables over a given hour and writes a JSON file with aggregations. In fifty lines of code, we can publish higher-level statistics about our streaming tables that can be consumed by business users:

// Configure batch input table 'aggregates'
const aggregateTableName = "aggregates";

const aggregateTableColumns = [
 {
 name: "event_type",
 type: "string",
 },
 {
 name: "count",
 type: "int",
 },
 {
 name: "time",
 type: "string",
 },
];

// function that lambda runs.
// writes records to table that summarize clicks and impressions
const aggregationFunction = async (event: EventRuleEvent) => {
 // configure an athena client for our DW
 const athena = require("athena-client");
 const bucketUri = `s3://${athenaResultsBucket.get()}`;
 const clientConfig = {
 bucketUri,
 };
 const awsConfig = {
 region,
 };
 const athenaClient = athena.createClient(clientConfig, awsConfig);
 const date = moment(event.time);
 // query impressions and clicks tables for hourly event counts
 const partitionKey = date.utc().format("YYYY/MM/DD/HH");
 const getAggregateQuery = (table: string) =>
 `select count(*) from ${databaseName.get()}.${table} ` +
 `where inserted_at='${partitionKey}'`;
 const clicksPromise =
 athenaClient.execute(getAggregateQuery(clicksTableName))
 .toPromise();
 const impressionsPromise =
 athenaClient.execute(getAggregateQuery(impressionsTableName))
 .toPromise();

 const clickRows = await clicksPromise;
 const impressionRows = await impressionsPromise;
 // write the event counts to our table in S3 in JSON format
 const clickCount = clickRows.records[0]["_col0"];
 const impressionsCount = impressionRows.records[0]["_col0"];
 const data = `{ "event_type": "${clicksTableName}", `+
 `"count": ${clickCount}, "time": "${partitionKey}" }\n` +
 `{ "event_type": "${impressionsTableName}", ` +
 `"count": ${impressionsCount}, "time": "${partitionKey}"}`;
 const s3Client = new S3();
 await s3Client.putObject({
 Bucket: dwBucket.get(),
 Key: `${aggregateTableName}/${partitionKey}/results.json`,
 Body: data,
 }).promise();
};

//configure resource access policies required by the function
const policyARNsToAttach: pulumi.Input<ARN>[] = [
 aws.iam.ManagedPolicies.AmazonAthenaFullAccess,
 aws.iam.ManagedPolicies.AmazonS3FullAccess,
];

const aggregateTableArgs: BatchInputTableArgs = {
 columns: aggregateTableColumns,
 jobFn: aggregationFunction,
 scheduleExpression,
 policyARNsToAttach,
 dataFormat: "JSON",
};

dataWarehouse.withBatchInputTable(aggregateTableName, aggregateTableArgs);

While Lambda’s limits restrict the utility of this method, the concept is powerful. Pulumi gives you the building blocks to define your abstractions. Instead of Lambda, the execution plane could be ECS, Kubernetes, or EMR.

Automating the Dev Environment

Developing software is an activity in seeking a specific state. For functions with simple inputs and outputs, this is a matter of well-factored code and unit test coverage. You can and should write this sort of test in your Pulumi applications. But often, the state that you’re seeking can’t be validated in unit tests. It raises many questions such as, “Am I stitching these cloud services together in a valid way? Will data flow from input service through to output destination? If I try to tighten down this IAM role, will everything still work?”

Our workflow is to make a change, run a pulumi up, and then verify that data ingested into Kinesis is read out the other end by Athena. A simple shell script using stack outputs and a query in the AWS console are enough to get started here:

#/bin/sh

aws kinesis put-record \
--stream-name $(pulumi stack output impressionInputStream) \
--region $(pulumi config get aws:region) \
--data '{"session_id": "1", "id": "2", "event_time": "2020-01-23T20:38:42.151Z"}' \
--partition-key '1'

This workflow works well enough, but it has a few problems. First of all, it requires a bunch of manual work. We have to run a pulumi up, wait for it to complete, execute this script, logon to the AWS console, execute an Athena query, and wait for the results to come back. We can automate the data ingestion with more Pulumi in our dev stack.


export class EventGenerator extends pulumi.ComponentResource {
 constructor(name: string,
 args: EventGeneratorArgs,
 opts?: pulumi.CustomResourceOptions
 ) {
 super("serverless:event_generator", name, opts);

 const { eventType, inputStreamName } = args;

 // puts a sample record into our input stream
 const eventGenCallback = (event: EventRuleEvent) => {
 const AWS = require("aws-sdk");
 const uuid = require("uuid/v4");
 const kinesis = new AWS.Kinesis();
 const records: any = [];

 const sessionId = uuid();
 const eventId = uuid();
 const record = {
 Data: JSON.stringify({
 id: eventId,
 session_id: sessionId,
 event_time: event.time,
 }),
 PartitionKey: sessionId,
 };
 records.push(record);

 kinesis.putRecords({
 Records: records,
 StreamName: inputStreamName.get(),
 }, (err: any) => {
 if (err) {
 console.error(err);
 }
 });
 };

 const lambdaCronArgs: LambdaCronJobArgs = {
 jobFn: eventGenCallback,
 scheduleExpression: "rate(1 minute)",
 policyARNsToAttach: [
 aws.iam.ManagedPolicies.AmazonKinesisFullAccess,
 ],
 };

 // schedule a lambda to run our function once a minute.
 const eventGenerator = new LambdaCronJob(`${eventType}-eventGenerator`,
 lambdaCronArgs, { parent: this });
 }
}

This helper creates a Lambda that publishes sample events to Kinesis on a minutely cron. It doesn’t completely automate our validation, but it’s certainly a step closer. Now when we run an incremental update, data automatically propagates through the system. We can use the AWS console to execute Athena queries and observe the output.

Incremental updates in our dev loop are fast, but when dealing with AWS services, entities like IAM roles and Glue schemas can be cached by services like Firehose. It’s entirely possible to make an update that appears to work for a few minutes, just to have it fail minutes later when some implementation detail clears its cache.

To bring 100% automation and certainty, we can write an isolated integration test that stands up new infrastructure, executes our tests, and then tears everything down.

jest.setTimeout(360000);

let runner: PulumiRunner;
const region = "us-west-2";

beforeAll(async () => {
 const config: { [key: string]: string } = {
 "aws:region": region,
 "aws-ts-serverless-datawarehouse:dev": "true",
 };

 const pulumiProjDir = resolve("./");
 runner = new PulumiRunner(config, pulumiProjDir);
 const setupResult = await runner.setup();
 if (!setupResult.success) {
 throw new Error(`Pulumi setup failed, aborting: ${setupResult.error}`);
 }
});

afterAll(async () => {
 const teardownResult = await runner.teardown();
 if (!teardownResult.success) {
 throw new Error(
 `Teardown failed. Test stack has leaked: ${teardownResult.error}`
 );
 }
});

test("WithStreamingInput integrtion test", async () => {
 // assert we see the expected stack outputs
 expect(runner.getStackOutputKeys().length).toBe(6);
 const db = runner.getStackOutput("databaseName");
 const clickTable = runner.getStackOutput("clickTableName");
 const impressionTable = runner.getStackOutput("impressionTableName");
 const bucket = runner.getStackOutput("athenaResultsBucket");

 // query our tables on a retry loop until we find records or timeout and fail
 const clickPromise = verifyRecordsInTable(db, clickTable, bucket);
 const impressionPromise = verifyRecordsInTable(db, impressionTable, bucket);

 const [clickTableHasRecords, impressionTableHasRecords] = await
 Promise.all([clickPromise, impressionPromise]);

 expect(clickTableHasRecords).toBe(true);
 expect(impressionTableHasRecords).toBe(true);
});

This test gives us a great deal of confidence that our changes are valid. While it certainly does a lot, it’s a little slow. We can run this suite with npm run test:int & tput bel to notify us upon completion while we work in the background, but four minutes might not meet the latency bar for core inner dev loop.

Here we’ve seen three different ways to use Pulumi to automate and improve the inner dev loop. Simple scripting with stack outputs to get started, automation of that script using Pulumi, and full blow integration testing to bring 100% confidence in tricky changes. All three are tools in our arsenal to be applied appropriately.

Wrapping Up

ACME re-evaluated their analytics architecture and shipped the next-generation serverless streaming solution that can take them to the next click-stop, and well through the next order of magnitude. Along the way, they learned how to reason about streaming data and examined the underlying assumptions about time and completeness of data in their existing system. ACME developed its streaming architecture using Pulumi and learned how to automate and create reproducible developer environments along the way. The outcome isn’t just an instance of the serverless streaming architecture, but a component library that implements best practices. That investment in time is bundled up and shipped in a format that the next generation of incubators at ACME can pick up and run with.

If you’d like to get started delivering best practices in infrastructure across your organization, get started with Pulumi free today. Clone and run the reference implementation of the Serverless Data Warehouse to build your own serverless analytics stack.

Architectural Caveats

Like every architecture, the serverless streaming model has limitations and pitfalls.

Processing Time vs. Event Time: Our data is now ordered and indexed by when it’s processed as opposed to when the events occur. We need to reason about data arriving late and make sure that our queries and dependent systems account for spikes in latency. For a detailed primer, see Streaming 101.
Data Freshness & Compaction: While it is possible to emit files by the minute to achieve low latency, it can lead to performance issues. In practice, this sort of setup can cause an explosion of tiny files that degrade the performance of query engines. See this article on optimizing file sizes for Athena for more details. A standard solution is to run an hourly compaction job that rewrites data into optimally sized chunks, leaving only the most recent hour or two of data in smaller files.
Cross Account Athena: Due to Glue’s limitations, you cannot grant cross-account access for Athena directly. There are workarounds like temporary credentials and using assume role.
Service Limits: Just as with Redshift, nothing is truly a black box. Athena has a maximum number of concurrent queries, and S3 has default limits on the rate at which you can list and get objects. As you scale, make sure you understand your service limits and reach out to AWS support to proactively increase limits where appropriate.
Query Translation: If you’re migrating an existing system from Redshift to Athena, the queries won’t quite map 1:1. For one, there’s the need to restructure queries to limit first on arrival time, and then narrow down on specific event time. And some of the traditional SQL operations like ORDER BY and GROUP BY will perform differently. See the section on query optimization for more details.
Unbounded Athena Queries: Athena charges $5 per TB scanned during a query. Most business users aren’t used to the concept of pay per query. It’s quite easy to issue a ‘select *’ query that can cost a great deal of money. Guard rails like a middleware that parses queries and ensures usage of a partition key may be necessary to control cost if exposing Athena as an interface to a broader audience.

Pulumi Blog: Evan Boyle

Dependent Stack Updates with Pulumi Deployments

Deployment Webhook Destinations

Expressing Stack Dependencies with Pulumi Auto Deploy

Scale Infrastructure with Software

Review Stacks: Collaborate in the Cloud

Configuring Review Stacks

Pulumi Cloud UI

REST API

Pulumi Cloud Resource Provider

Collaborate in the Cloud

Pulumi Deployments: API-Driven Infrastructure at Scale

Deployment Settings and Triggers

Example: Building RESTful Infrastructure APIs

Pave your Golden Path Today

Read Every Single Error

Read Every Error Message That Your System Produces

Reliability from First Principles

Why Should You Care?

The SRE’s Folly

Pulumi Deployments: the Fastest Way to Go from Code to Cloud

Automation API & Deployments - Do More with Less

A Golden Path to Production

git push to Deploy

Click to Deploy

Power-up Your Platform with The Deployments API

Drift Detection

TTL Stacks and Temporary Infrastructure

Managing Scale with Remote Automation API

Centralized Observability

Automation Accelerates the Cloud

Stack READMEs in the Pulumi Service

How to add a Stack README to your Stack

Step 1

Step 2

Step 3

Step 4

The Pulumi Automation API - The Next Quantum Leap in IaC

Enabling Cloud Engineering

Platform APIs

Complex Workflow Orchestration

Cloud Framework Development

Supercharged Ops Tooling

Give it a Try Today!

Welcoming Go to the Pulumi Family

What is Pulumi?

Giving you superpowers

Serverless

Containers

Kubernetes

Kubernetes + Infra

Try Go + Pulumi Today

Modern Cloud Infrastructure in Go - The Road to 2.0

What is Pulumi?

An Expressive and Idiomatic Type System

Try Pulumi + Go Today

Coming Soon

AWS Serverless Analytics

ACME E-Commerce

The Traditional Analytics Stack

ACME Ads MVP

Scaling a Real Business

Designing the V2 Analytics Stack

Architecture as Code with Pulumi

Creating a Batch Table API

Automating the Dev Environment

Wrapping Up

Architectural Caveats

`git push` to Deploy