Pulumi Blog: Automation API

Next-level IaC: How Pulumi Supports Your API Economy Strategy

Engin Diri — Wed, 15 May 2024 00:00:00 +0000

When I am talking with community members, who are not using Pulumi yet, I often get asked what would be a good way to include their Infrastructure as Code (IaC) into existing software like a REST API. And my answer is always the same: Use our Pulumi Automation API.

With the Pulumi Automation API you can include Pulumi IaC into your existing software, and this for any of the Pulumi supported programming languages like TypeScript, Python, Go, or C#. This gives you a greater flexibility and control, which you will not have with other IaC tools like CloudFormation or Terraform.

This opens up many doors of opportunity to expose your services as an API, whether internally to your organization or externally to your potential customers. This opportunity for innovation fuels what is known as the API Economy.

What is API Economy?

As a matter of fact: — APIs enable innovation.

Over 40% of large organizations have 250+ APIs. And 71% plan to use even more APIs this year.

API Economy is a term that is used to describe the way that organizations are using APIs to create new business models and revenue streams. APIs are the building blocks of modern software development, and they allow developers to build on top of existing services and create new services that are greater than the sum of their parts.

Major international corporations leveraging the API economy through robust API strategies include Google, Amazon, Stripe, Twilio, Slack, and many other renowned brands. Google’s initial Maps service, for instance, has evolved into a comprehensive platform offering Maps, Routes, and Places APIs and SDKs. These tools enable companies to incorporate both dynamic and static maps into their applications and websites, offer navigational guidance, access 360° Street View images, utilize geolocation services, and more. This transformation exemplifies the concept of value multiplication in the API economy, as described by Gartner Vice President Kristin Moyer:

“The API economy is an enabler for turning a business or organisation into a platform. Platforms multiply value creation because they enable business ecosystems inside and outside of the enterprise to consummate matches among users and facilitate the creation and exchange of goods, services and social currency so that all participants are able to capture value.”

What is Pulumi Automation API in a nutshell?

Pulumi Automation API is a set of SDKs that allow you to interact with your Pulumi stacks programmatically. This means that you can use the Automation API to create, update, and delete stacks, as well as to query their current state. The Automation API is built on top of the Pulumi engine, which means that it is fully compatible with all of the existing Pulumi providers and resources.

And it is this capability that allows you to build your own Service API on top of Pulumi. We can now build a REST or gRPC API endpoint that, depending on the request parameters, will create, update, or delete a Pulumi stack. This API can then be exposed to the rest of the organization, allowing developers to interact with their infrastructure in a programmatic way.

How to build your own Service API with Pulumi Automation API?

Let’s take a look at how you can build your own Service API with Pulumi Automation API.

For this purpose, we will build two basic REST services written in TypeScript and in Python.

import express from "express"
import * as dotevnv from "dotenv"
import cors from "cors"
import helmet from "helmet"
import {fullyQualifiedStackName, RemoteWorkspace} from "@pulumi/pulumi/automation";

dotevnv.config()

if (!process.env.PORT) {
 console.log(`No port value specified...`)
}

const PORT = parseInt(process.env.PORT as string, 10)

const app = express()

app.use(express.json())
app.use(express.urlencoded({extended: true}))
app.use(cors())
app.use(helmet())

// Define the payload interface
interface ServicePayload {
 name: string;
 org: string;
 project: string;
 stack: string;
 region: string;
}

// Define a helper function to set up the Pulumi program
async function setUpPulumiProgram(payload: ServicePayload) {
 const org = payload.org;
 const project = payload.project;
 const stackName = fullyQualifiedStackName(org, project, payload.stack);
 const awsRegion = payload.region;

 // Using the Pulumi Automation API to create or select a stack in the remote workspace
 // and set the stack's configuration and secrets. This will use Pulumi Deployments to
 // deploy the stack to the specified AWS region.
 return await RemoteWorkspace.createOrSelectStack({
 stackName,
 url: "https://github.com/pulumi/examples.git",
 branch: "refs/heads/master",
 projectPath: project,
 }, {
 envVars: {
 AWS_REGION: awsRegion,
 AWS_ACCESS_KEY_ID: process.env.AWS_ACCESS_KEY_ID ?? "",
 AWS_SECRET_ACCESS_KEY: {secret: process.env.AWS_SECRET_ACCESS_KEY ?? ""},
 AWS_SESSION_TOKEN: {secret: process.env.AWS_SESSION_TOKEN ?? ""},
 },
 });
}

// Define the API endpoint for creating a new stack in the remote workspace.
app.post('/', async (req, res) => {
 const payload: ServicePayload = req.body;

 if ((payload.name) || !payload.org || !payload.project || !payload.stack || !payload.region) {
 res.status(400).send('Name, org, project, stack and region are required');
 return;
 }
 // Create a new stack in the remote workspace and set the stack's configuration and secrets.
 const stack = await setUpPulumiProgram(payload);

 // Deploy the stack to the specified AWS region. We'll print the stack's outputs to the console.
 const upRes = await stack.up({onOutput: console.log});
 res.status(200).send(`url: ${upRes.outputs.websiteUrl.value}`);
});

// Define the API endpoint for destroying a stack in the remote workspace.
app.delete('/', async (req, res) => {
 const {name, org, project, stack, region} = req.query;

 if (!name || !org || !project || !stack || !region) {
 res.status(400).send('Name, org, project, stack, and region are required');
 return;
 }

 const payload: ServicePayload = {
 name: name as string,
 org: org as string,
 project: project as string,
 stack: stack as string,
 region: region as string
 }

 // Define the stack and set the stack's configuration and secrets.
 const prog = await setUpPulumiProgram(payload);
 // Destroy the stack in the remote workspace. We'll print the stack's outputs to the console.
 await prog.destroy({onOutput: console.log});
 res.status(200).send('Stack destroyed');
});


app.listen(PORT, () => {
 console.log(`Server is listening on port ${PORT}`)
})

import os
import pulumi.automation as auto
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


# Define the payload that the API will receive. This payload will contain the name of the stack
# to create, the organization and project to use, the stack name, and the AWS region to deploy the stack to.
class ServicePayload(BaseModel):
 name: str
 org: str
 project: str
 stack: str
 region: str


# Define a helper function that will create a new stack in the remote workspace and set the stack's configuration
def setup_pulumi_program(payload: ServicePayload):
 org = payload.org
 project = payload.project
 stack_name = auto.fully_qualified_stack_name(org, project, payload.stack)
 aws_region = payload.region

 # Using the Pulumi Automation API to create or select a stack in the remote workspace
 # and set the stack's configuration and secrets. This will use Pulumi Deployments to
 # deploy the stack to the specified AWS region.
 return auto.create_or_select_remote_stack_git_source(
 stack_name=stack_name,
 url="https://github.com/pulumi/examples.git",
 branch="refs/heads/master",
 project_path=project,
 opts=auto.RemoteWorkspaceOptions(
 env_vars={
 "AWS_REGION": aws_region,
 "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
 "AWS_SECRET_ACCESS_KEY": auto.Secret(os.environ["AWS_SECRET_ACCESS_KEY"]),
 "AWS_SESSION_TOKEN": auto.Secret(os.environ["AWS_SESSION_TOKEN"]),
 },
 ),
 )


# Define the API endpoints for creating and deleting a stack in the remote workspace.
@app.post("/")
async def create(payload: ServicePayload):
 # Create a new stack in the remote workspace and set the stack's configuration and secrets.
 stack = setup_pulumi_program(payload)
 # Deploy the stack to the specified AWS region. We'll print the stack's outputs to the console.
 up_res = stack.up(on_output=print)
 return {"url": up_res.outputs['websiteUrl'].value}


# Define the API endpoint for deleting a stack in the remote workspace.
@app.delete("/")
async def delete(name: str, org: str, project: str, stack: str, region: str):
 payload = ServicePayload(name=name, org=org, project=project, stack=stack, region=region)
 # Define the stack and set the stack's configuration and secrets.
 stack = setup_pulumi_program(payload)
 # Destroy the stack in the remote workspace. We'll print the stack's outputs to the console.
 stack.destroy(on_output=print)
 return {"message": "Stack destroyed"}

Let us run the above Python example in a FastAPI server to see how the Automation API works in practice.

Let’s start by creating a new Python file called main.py and pasting the above code into it. Next, we need to install the required dependencies:

python3 -m venv ven
source venv/bin/activate

Create a new file called requirements.txt and paste the following dependencies into it:

pulumi
fastapi
pydantic

We need to install the dependencies using the following command:

pip install -r requirements.txt --force-reinstall

Finally, we can run the FastAPI server using the following command:

fastapi dev main.py

This will start the FastAPI server on port 8000.

INFO Using path main.py
INFO Resolved absolute path /private/tmp/automation-api-example/python/main.py
INFO Searching for package file structure from directories with __init__.py files
INFO Importing from /private/tmp/automation-api-example/python

 ╭─ Python module file ─╮
 │ │
 │ 🐍 main.py │
 │ │
 ╰──────────────────────╯

INFO Importing module main
INFO Found importable FastAPI app

 ╭─ Importable FastAPI app ─╮
 │ │
 │ from main import app │
 │ │
 ╰──────────────────────────╯

INFO Using import string main:app

 ╭────────── FastAPI CLI - Development mode ───────────╮
 │ │
 │ Serving at: http://127.0.0.1:8000 │
 │ │
 │ API docs: http://127.0.0.1:8000/docs │
 │ │
 │ Running in development mode, for production use: │
 │ │
 │ fastapi run │
 │ │
 ╰─────────────────────────────────────────────────────╯

INFO: Will watch for changes in these directories: ['/private/tmp/automation-api-example/python']
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [55849] using WatchFiles
INFO: Started server process [55851]
INFO: Waiting for application startup.
INFO: Application startup complete.

To test the REST services, you can use a tool like curl or Postman. Here is an example of how you can create a new stack using curl:

curl -X POST -H "Content-Type: application/json" -d '{"name": "myexample", "org": "myorg", "project": "myproject", "stack": "mystack", "region": "us-west-2"}' http://localhost:3000

In the logs of the FastAPI server, you should see following output from the Automation API:

INFO: Create stack
Preparing deployment...

View Live: https://app.pulumi.com/dirien/aws-ts-s3-folder/dev/deployments/3

Setup:
Preparing environment
Pulling container image "pulumi/pulumi:3.115.2"
3.115.2: Pulling from pulumi/pulumi
Digest: sha256:a7b9022d7c1a40d81adeca30a21eb1cd49add474e3de923ade663db046e7ee1a
Status: Downloaded newer image for pulumi/pulumi:3.115.2
Starting container
Container created (6b92928bdcdfe6e54286dcdcb622480004f18c98b23b5b60775f71959a423708)
Container started (6b92928bdcdfe6e54286dcdcb622480004f18c98b23b5b60775f71959a423708)

Download deployment executor:
$ curl --retry 5 -H "Authorization: token $PULUMI_ACCESS_TOKEN" "$PULUMI_SERVICE_URL/deployments/executor" -o /pulumi-deploy-executor.gz && gzip -d /pulumi-deploy-executor.gz && chmod +x /pulumi-deploy-executor
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 15.9M 0 15.9M 0 0 42.2M 0 --:--:-- --:--:-- --:--:-- 42.4M

Get source:
$ /pulumi-deploy-executor getsource git --repoURL="https://github.com/pulumi/examples.git" --workDir="/deployment" --branch="refs/heads/master" --repoDir="aws-ts-s3-folder"
Successfully cloned: https://github.com/pulumi/examples.git

Download dependencies:
$ /pulumi-deploy-executor dependencies download --workDir="/deployment/aws-ts-s3-folder"
npm WARN deprecated @opentelemetry/api-metrics@0.32.0: Please use @opentelemetry/api >= 1.3.0

added 294 packages, and audited 295 packages in 14s

43 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
npm notice
npm notice New major version of npm available! 9.6.7 -> 10.7.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.7.0>
npm notice Run `npm install -g npm@10.7.0` to update!
npm notice

pulumi update:
$ /pulumi-deploy-executor pulumi update --stackIdentity="dirien/aws-ts-s3-folder/dev" --workDir="/deployment/aws-ts-s3-folder"
Updating (dev)

View Live: https://app.pulumi.com/dirien/aws-ts-s3-folder/dev/updates/3


 + pulumi:pulumi:Stack aws-ts-s3-folder-dev creating (0s)
 + aws:s3:Bucket s3-website-bucket creating (0s)
 + aws:s3:Bucket s3-website-bucket created (4s)
 + aws:s3:BucketObject favicon.png creating (0s)
 + aws:s3:BucketObject index.html creating (0s)
 + aws:s3:BucketPublicAccessBlock public-access-block creating (0s)
 + aws:s3:BucketObject index.html created (0.98s)
 + aws:s3:BucketObject favicon.png created (1s)
 + aws:s3:BucketPublicAccessBlock public-access-block created (1s)
 + aws:s3:BucketPolicy bucketPolicy creating (0s)
 + aws:s3:BucketPolicy bucketPolicy created (0.83s)
 + pulumi:pulumi:Stack aws-ts-s3-folder-dev created (18s)
Outputs:
 bucketName: "s3-website-bucket-d55e9bd"
 websiteUrl: "s3-website-bucket-d55e9bd.s3-website.eu-central-1.amazonaws.com"

Resources:
 + 6 created

Duration: 20s

Result: succeeded
INFO: 127.0.0.1:49220 - "POST / HTTP/1.1" 200 OK

You should see the URL of the created infrastructure in the response from the FastAPI server. If you head over to the Pulumi console, you should see the created stack and the resources that were created and also that Pulumi Deployments were used to deploy the stack to the specified AWS region.

And here is an example of how you can delete a stack using curl:

curl -X DELETE "http://localhost:8000/?name=myexample&org=myorg&project=myproject&stack=mystack&region=us-west-2"

In the logs of the FastAPI server, you should see the following output from the Automation API:

INFO: Deleting stack
Preparing deployment...

View Live: https://app.pulumi.com/dirien/aws-ts-s3-folder/dev/deployments/4

Setup:
Preparing environment
Pulling container image "pulumi/pulumi:3.115.2"
3.115.2: Pulling from pulumi/pulumi
Digest: sha256:a7b9022d7c1a40d81adeca30a21eb1cd49add474e3de923ade663db046e7ee1a
Status: Downloaded newer image for pulumi/pulumi:3.115.2
Starting container
Container created (26af871d4e26a6dab5656a1c8dc8a7e87bf247f42bde77e2269c269f15054e73)
Container started (26af871d4e26a6dab5656a1c8dc8a7e87bf247f42bde77e2269c269f15054e73)

Download deployment executor:
$ curl --retry 5 -H "Authorization: token $PULUMI_ACCESS_TOKEN" "$PULUMI_SERVICE_URL/deployments/executor" -o /pulumi-deploy-executor.gz && gzip -d /pulumi-deploy-executor.gz && chmod +x /pulumi-deploy-executor
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 15.9M 0 15.9M 0 0 43.5M 0 --:--:-- --:--:-- --:--:-- 43.5M

Get source:
$ /pulumi-deploy-executor getsource git --repoURL="https://github.com/pulumi/examples.git" --workDir="/deployment" --branch="refs/heads/master" --repoDir="aws-ts-s3-folder"
Successfully cloned: https://github.com/pulumi/examples.git

Download dependencies:
$ /pulumi-deploy-executor dependencies download --workDir="/deployment/aws-ts-s3-folder"
npm WARN deprecated @opentelemetry/api-metrics@0.32.0: Please use @opentelemetry/api >= 1.3.0

added 294 packages, and audited 295 packages in 14s

43 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
npm notice
npm notice New major version of npm available! 9.6.7 -> 10.7.0
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.7.0>
npm notice Run `npm install -g npm@10.7.0` to update!
npm notice

pulumi destroy:
$ /pulumi-deploy-executor pulumi destroy --stackIdentity="dirien/aws-ts-s3-folder/dev" --workDir="/deployment/aws-ts-s3-folder"
Destroying (dev)

View Live: https://app.pulumi.com/dirien/aws-ts-s3-folder/dev/updates/4


 - aws:s3:BucketPolicy bucketPolicy deleting (0s)
 - aws:s3:BucketPolicy bucketPolicy deleted (1s)
 - aws:s3:BucketPublicAccessBlock public-access-block deleting (0s)
 - aws:s3:BucketObject favicon.png deleting (0s)
 - aws:s3:BucketObject index.html deleting (0s)
 - aws:s3:BucketObject index.html deleted (0.61s)
 - aws:s3:BucketObject favicon.png deleted (0.73s)
 - aws:s3:BucketPublicAccessBlock public-access-block deleted (0.77s)
 - aws:s3:Bucket s3-website-bucket deleting (0s)
 - aws:s3:Bucket s3-website-bucket deleted (0.79s)
 - pulumi:pulumi:Stack aws-ts-s3-folder-dev deleting (0s)
 - pulumi:pulumi:Stack aws-ts-s3-folder-dev deleted (0.10s)
Outputs:
 - bucketName: "s3-website-bucket-d55e9bd"
 - websiteUrl: "s3-website-bucket-d55e9bd.s3-website.eu-central-1.amazonaws.com"

Resources:
 - 6 deleted

Duration: 7s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
If you want to remove the stack completely, run `pulumi stack rm dev`.
Result: succeeded
INFO: 127.0.0.1:51509 - "DELETE /?name=my-test&org=dirien&project=aws-ts-s3-folder&stack=dev&region=eu-central-1 HTTP/1.1" 200 OK

Next Steps

Now that we have our basic REST services, we can deploy them on a server (using a service like AWS Lambda, Google Cloud Functions, or Azure Functions) and expose them to the rest of the organization via an API Gateway. In parallel you can publish release notes and documentation to your organization, so that they can start using your new Service API.

One way could be to create an OpenAPI specification for your Service API and use a tool like Swagger to generate a client library for your organization. This way, developers can interact with your Service API in a programmatic way, without having to write any code.

Here is an example of an OpenAPI specification for our Service API:

openapi: 3.0.0
info:
 title: My Organization Service API
 description: API for creating and destroying a specific piece of infrastructure
 version: 1.0.0
servers:
- url: 'http://localhost:8000/'
 description: Development server
paths:
 /:
 post:
 summary: Create Infrastructure
 operationId: createInfrastructure
 requestBody:
 required: true
 content:
 application/json:
 schema:
 $ref: '#/components/schemas/ServicePayload'
 responses:
 '200':
 description: Infrastructure successfully created
 content:
 application/json:
 schema:
 type: object
 properties:
 url:
 type: string
 description: URL of the created infrastructure
 '400':
 description: Bad request if the input payload is incorrect
 '500':
 description: Internal server error
 delete:
 summary: Destroy Infrastructure
 operationId: destroyInfrastructure
 parameters:
 - in: query
 name: name
 schema:
 type: string
 required: true
 description: Name of the service
 - in: query
 name: org
 schema:
 type: string
 required: true
 description: Organization in Pulumi
 - in: query
 name: project
 schema:
 type: string
 required: true
 description: Project name in Pulumi
 - in: query
 name: stack
 schema:
 type: string
 required: true
 description: Stack name in Pulumi
 - in: query
 name: region
 schema:
 type: string
 required: true
 description: AWS region where the service will be deployed
 responses:
 '200':
 description: Infrastructure successfully destroyed
 content:
 application/json:
 schema:
 type: object
 properties:
 message:
 type: string
 description: Confirmation message of the destruction
 '400':
 description: Bad request if the required parameters are missing
 '500':
 description: Internal server error
components:
 schemas:
 ServicePayload:
 type: object
 required:
 - name
 - org
 - project
 - stack
 - region
 properties:
 name:
 type: string
 description: Name of the service
 org:
 type: string
 description: Organization in Pulumi
 project:
 type: string
 description: Project name in Pulumi
 stack:
 type: string
 description: Stack name in Pulumi
 region:
 type: string
 description: AWS region where the service will be deployed

Navigate to the Swagger Editor and paste the above OpenAPI specification to see how it looks.

Conclusion

In this blog post, we’ve explored how Pulumi’s Automation API can be used to build and expose a Service API within your organization. By embedding Pulumi as a library in any application, you can drive your Infrastructure as Code (IaC) deployments directly through code. This sets Pulumi apart from other tools like Terraform, which relies on static configuration files and a command-line interface.

Additionally, we touched on the concept of the API economy, where APIs are leveraged to connect services, accelerate product delivery, and drive business value through the possibilities of creating new touchpoints. The Pulumi Automation API is the perfect piece of technology, which makes it now possible to define, deploy, and manage infrastructure programmatically as part of your API strategy.

I hope this blog post has inspired you with ideas on how to take your IaC to the next level, making it more accessible and practical for your organization.

Next Steps

If you haven’t already, install Pulumi today, and follow our self-directed Getting Started guides to learn more about making the most of Pulumi’s next-level infrastructure management features at your organization.

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

The Pulumi Cloud is a fully managed service that helps you adopt Pulumi’s open source SDK with ease. It provides built-in state and secrets management, integrates with source control and CI/CD, and offers a web console and API that make it easier to visualize and manage infrastructure. It is free for individual use, with features available for teams.

Create an Account

How Starburst Data Creates Infrastructure Automation Magic With Code

George Huang — Wed, 23 Aug 2023 00:00:00 +0000

This blog post summarizes a presentation by Matt Stephenson at PulumiUP 2023.

Matt Stephenson is Senior Principal Software Engineer for Starburst Data and a Puluminary member. He’s deeply involved in the Infrastructure as Code (IaC) space, having contributed to Ansible, been a core contributor to Apache jclouds, and has written many Terraform plugins. He leads infrastructure architecture at Starburst and originally introduced Pulumi to the company. Starburst provides a data lake analytics platform that’s powered by Trino - an open-source distributed SQL query engine designed for running fast analytic queries across large datasets in multiple data sources. At Starburst, Matt helped revamp and improve how the company manages its multi-cloud and cloud native infrastructure.

Watch Matt Stephenson’s full presentation with demos. 8:03 demo of CI/CD. 15:33 demo of infrastructure automation.

Technical and Business Problems to Solve

Starburst operates a managed Trino service that runs on Kubernetes clusters on multiple clouds. Their Kubernetes clusters needed frequent manual scaling based on workloads. Furthermore, upgrading Trino versions running on the clusters ran the risk of potential service disruptions for customers. The company had been using tools like Terraform for IaC, Argo CD for deploying Helm charts, Maven, Docker containers, and a Slack bot for infrastructure and deployments. Their CI/CD workflow was difficult to maintain because they needed to create wrappers for their previous IaC tooling, making it less approachable in CI/CD environments. The pipeline was tenuously bound together using GitHub Actions and a Slack bot, requiring specialized GHA runners, leading to a messy architecture.

Why Starburst Chose Pulumi

Recognizing opportunities for improvement, Matt brought in Pulumi and its Automation API. This unified Starburst’s software development and infrastructure management practices, especially when deploying and managing Trino clusters on Kubernetes. The top two reasons Matt chose Pulumi were:

Support for programming languages: Pulumi supports the main languages used by Starburst’s developers, Java and TypeScript. This enables all engineers to write infrastructure code and promotes a cohesive application and infrastructure codebase.
Pulumi Automation API: Automation API is a programmatic interface for the Pulumi CLI and Pulumi IaC engine, which enables Starburst engineers to build infrastructure automation into their applications and CI/CD processes, and orchestrate complex provisioning workflows.

Benefits of Using Pulumi

Adopting Pulumi and its Automation API provided several benefits that made Starburst’s engineers more efficient and productive, ultimately leading to faster and more reliable releases for Starburst’s customers. Matt summed up the benefits of Pulumi for Starburst’s engineers: “When I think about building an airplane, I think about precision and having the right tools for the job.” Pulumi, for Starburst, epitomized this precision and efficiency.

Increased deployment speeds by 3x for new customer releases: Starburst engineers used the Pulumi Automation API to triple the deployment speed of multi-cloud Kubernetes clusters with blue-green deployments that complete within minutes. They did this by orchestrating blue-green updates across their data plane running on the clusters, ensuring zero customer downtime. They built intricate integrations between Pulumi and internal APIs that enabled them to seamlessly transition customers to new clusters. Now they can update their customers’ runtimes, including VPCs and Kubernetes clusters, twice a week.

Unified cluster and infrastructure management: Rather than juggling multiple tools for managing Kubernetes and infrastructure, Starburst engineers use a single tool to manage over 50 Kubernetes clusters and deploy Helm charts into the clusters, spanning multiple regions. They also use Pulumi to manage SaaS platforms like Cloudflare, Confluent, and DataDog.

Consistent deployment lifecycles: Using Pulumi’s integration with GitHub Actions, Starburst engineers built a streamlined CI/CD process where infrastructure and application code builds within the same pipelines with automated test environments. They accomplished this using Pulumi Automation API to abstract the underlying runtime environments, circumventing the need for additional layers and wrappers.

Try Pulumi for Infrastructure as Code Automation

Sign up for a free account to try deploying infrastructure on any cloud with Pulumi Deployments, or register for an upcoming workshop to learn more about how Pulumi can help you ship cloud infrastructure faster and more safely.

Go to more case studies →

IaC Best Practices: Using Automation API

Scott Lowe — Wed, 26 Jul 2023 00:00:00 +0000

Welcome to the sixth post in our series of blog posts focused on Infrastructure as Code (IaC) best practices. So far in this series, you’ve seen how Zephyr Archaeotech Emporium—the fictional company at the center of this series—uses Pulumi to manage their online retail store. You read how Zephyr’s initial use of Pulumi changed to use short-lived per-developer stacks. Later, as Zephyr continued to grow, you saw how Zephyr restructured their Pulumi projects and stacks, incorporated Stack References, and used Pulumi Cloud’s role-based access control (RBAC) functionality to control access to their stacks. This post focuses on how Zephyr takes advantage of the Pulumi Automation API to bring an even greater level of orchestration to the stacks that represent their online store.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. These practices have unfolded organically, as a direct response to Zephyr’s evolving needs. The aim is to demonstrate that best practices are not set in stone, but are rather “point in time” recommendations that adapt as your company grows.

For ease of navigation, here are links to all the posts in the series:

IaC Best Practices: Understanding Code Organization and Stacks
IaC Best Practices: Enabling Developer Stacks and Git Branches
IaC Best Practices: Structuring Pulumi Projects
IaC Best Practices: Applying Stack References
IaC Best Practices: Implementing RBAC and Security
IaC Best Practices: Using Automation API (you are here)
IaC Best Practices: Summarizing Key Learnings

Managing Complexity in a Multi-Project Pulumi Setup

Over the course of this series, you’ve seen how Zephyr’s use of Pulumi has grown and changed. For the sake of completeness, here’s a quick recap:

Zephyr started with a single project in a single repository with two stacks, one for production and one for testing. (Details are available in understanding code organization and stacks post.)
To unlock developer velocity, Zephyr quickly added per-developer stacks. (You can read about per-developer stacks post in the series.)
As Zephyr grew and expanded, their single Pulumi project grew into three different Pulumi projects: one for base infrastructure, one for their Kubernetes platform, and one for the online store application. (The structuring Pulumi projects post describes the reasoning for adopting multiple projects.)
Accompanying the switch to multiple projects, Zephyr implemented stack references (to pass information between stacks in different projects) and applied role-based access control (RBAC) to the stacks in Pulumi Cloud. (Refer back to the stack references and RBAC & security posts in the series for more details on each of these areas.)

To see the state of Zephyr’s Pulumi implementation before the changes described in this article, please look at the blog/multi-project branch in each of the GitHub repositories (zephyr-infra, zephyr-k8s, and zephyr-app).

Since we last checked in on Zephyr, one significant change has emerged. As traffic to the Zephyr online store continued to increase, Zephyr’s operations team was finding it difficult to scale the relational databases running on Kubernetes (specifically, the orders and catalog databases). After evaluating their options, the team decided to migrate to Amazon Aurora with MySQL compatibility. Using the considerations outlined for structuring Pulumi projects, they decided to create a fourth Pulumi project, named “zephyr-data”, to manage the data infrastructure for the online store. Initially, this data infrastructure project manages only the Aurora clusters and database instances for the orders and catalog databases.

Because Zephyr was already using a modular approach to their overall application stack, inserting a new project for managing the online store’s data layer involved only minimal changes:

A new Pulumi program had to be written. This new program used stack references to leverage the existing infrastructure created by the base infrastructure stack and allowed traffic to the databases from the security group created by the Kubernetes platform stack. You’ll find this new program in the zephyr-data repository on GitHub (look in the blog/automation-api branch).
The Pulumi program for the Kubernetes platform stack only needed to export the security group created for the cluster nodes.
The Pulumi program for deploying the online store microservices referenced the database endpoints created by the new data infrastructure stack, and no longer spun up containerized instances of MariaDB for the orders and catalog databases.

The resulting four project architecture looks like this:

To see the state of the code after the changes described above—the addition of the zephyr-data project and the changes to use Amazon Aurora—please review the blog/automation-api branch in each of the Zephyr repositories.

Using a modular approach with multiple projects has its benefits. However, one of the trade-offs of splitting your IaC code into separate Pulumi projects is some additional relative complexity. With additional complexity comes additional room for error. To stand up a production-like environment, developers now had to stand up four different stacks (base infrastructure, Kubernetes platform, data infrastructure, and online store) in a specific order. If a user gets the order wrong, then the online store won’t come up properly.

With only four projects, this is not an insurmountable challenge by any stretch. However, what if it was 10 projects? Or 15 projects? Those numbers are not out of reach! Right now Zephyr has a single project to deploy all the various services involved for the online store. What if the application team decides to split up that project so that individual services can be deployed independently of each other? (After all, being able to deploy microservices independently of one another is one of the benefits of such an architecture.)

The Zephyr team started asking some questions: How can we make this process simpler? How can we remove room for error? How can we prepare our teams for greater scale?

Simplifying Advanced Deployments with Automation API

The challenges facing the Zephyr teams are not unique to Zephyr. Many IaC vendors recommend using a modular approach. Unfortunately, most IaC products rely on less-than-ideal approaches—like Bash scripts or Makefiles—to “stitch” together the different modules.

In some cases, organizations can use existing build automation tools and CI/CD pipelines to assist. What’s really needed, though, is a way to programmatically control the IaC tooling, and that’s what Pulumi’s Automation API is. The Pulumi Automation API allows Zephyr’s teams to use a general purpose programming language to automate Pulumi operations like performing an update, refreshing the stack state, or destroying a stack. Like Pulumi itself, the Pulumi Automation API supports multiple languages and supports multiple state backends, including Pulumi Cloud. Further, there is no requirement that Pulumi Automation API programs are written in the same language as the Pulumi programs they automate. In the case of Zephyr, which thus far has chosen to use TypeScript for their Pulumi programs, this means they are free to use any supported language for their Pulumi Automation API programs.

When the Zephyr team started evaluating options for simplifying the deployment of their online store and associated infrastructure, they decided to move forward with the Pulumi Automation API for these reasons:

Tight integration with Pulumi and Pulumi Cloud
Support for Pulumi Deployments (something we’ll examine in the next blog post)
Support for a broad range of programming languages, just like Pulumi itself
No requirement to use the same language as the Pulumi programs being automated (in fact, a Pulumi Automation API program could automate multiple programs written in different supported languages!)

Let’s take a look at Zephyr’s initial foray into using Pulumi Automation API.

Performing Stack Operations Efficiently

You can view the Pulumi Automation API program that Zephyr wrote in the local-source directory of the pulumi/zephyr-automation repository on GitHub, in the blog/automation-api branch.

Zephyr’s team decided to write their Pulumi Automation API program in Golang. One of the first things you’ll note about the Pulumi Automation API program is that it looks more like a “regular” Go program than a typical Pulumi Go program.

For example, a typical Pulumi Automation API program written in Go starts out like any other Go program:

func main() {
 // All your Go code is here
}

In a typical Pulumi Go program, on the other hand, the main() function starts off with a pulumi.Run call, like this:

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // All the Pulumi Go code is here
 })
}

The easiest way to understand this difference is to consider the way each program is invoked. A Pulumi Automation API program written in Go is executed by running go run main.go, like running any other Go program. In contrast, a Pulumi Go program is invoked by the pulumi CLI.

Zephyr’s Automation API program starts out with checking for a command-line argument to indicate the desired mode of the operation—should it be creating/updating resources, or destroying resources? In a standard Pulumi Go program, this would be determined by the invocation of the pulumi CLI (for example, whether you ran pulumi up or pulumi destroy). Here, though, that falls to the user to check:

// Determine mode of operation; default is to refresh/update
destroy := false
argsWithoutProg := os.Args[1:]
if len(argsWithoutProg) > 0 {
 if argsWithoutProg[0] == "destroy" {
 destroy = true
 }
}

Next, the program reads some configuration data from a YAML configuration file; this configuration data includes things like the relative file system location of the Pulumi programs that this Automation API program is going to execute:

// Get configuration data
data, err := os.ReadFile("config.yaml")
if err != nil {
 fmt.Printf("Error reading configuration file: %v\n", err)
 os.Exit(1)
}
var env Environment
err = yaml.Unmarshal(data, &env)
if err != nil {
 fmt.Printf("Error parsing configuration information: %v\n", err)
 os.Exit(1)
}

Separating configuration from code in this manner allows you to use the language’s existing ability to create binary executables that can be easily distributed to your teams. For example, Zephyr has the option of running go build to create a binary executable for distribution to internal teams.

From there, the Automation API program gets into setting up the stacks it will use:

// Set up base stack
baseStackName := auto.FullyQualifiedStackName(env.Organization, env.BaseProject.Name, env.StackName)
baseStack, err := auto.UpsertStackLocalSource(ctx, baseStackName, env.BaseProject.Location)
if err != nil {
 fmt.Printf("Failed to create or select %s stack: %v\n", env.BaseProject.Nickname, err)
 os.Exit(1)
}
fmt.Printf("Successfully created/selected %s stack\n", env.BaseProject.Nickname)

In this particular program, Zephyr decided to start with a local workspace. (You can read more about workspaces in the Pulumi Automation API docs.) This is reflected in the use of the UpsertStackLocalSource method shown above, which will create or select a stack for a Pulumi program found in a local filesystem location.

Users can also use UpsertStackInlineSource, which embeds the Pulumi program into the Automation API program as a function. There are trade-offs to each approach:

Using local source (i.e., the Pulumi program is stored outside of the Automation API program) has the benefit of allowing users to independently revise the Pulumi program or the Automation API program; this may be the ideal approach where organizations have separate teams that will create, manage, or maintain the Automation API programs. In Zephyr’s case, where there is a platform team, this made a lot of sense for them. This does also mean, as you might guess, that the Pulumi programs must be reachable by the Automation API program (this typically means users need to clone down the appropriate repositories to their system first).
Using inline source (where the Pulumi program is embedded in the Automation API program) may be a better fit when there is a single team managing or maintaining both the Pulumi program(s) as well as the Automation API program(s), or when—for whatever reason—these programs should not be revised independently of one another. This has the benefit of removing the need for users to clone or copy separate Pulumi programs onto their system before being able to use the Automation API program.

After all four stacks—base, platform, data, and app—have been set up for use within the Pulumi Automation API program, then comes the “real work”: performing operations on those stacks. To reduce duplicate code, the Zephyr program makes use of three functions:

The refreshStack function (starting on line 199 of main.go) performs the equivalent of a pulumi refresh against a stack.
The updateStack function (starting on line 218 of main.go) performs an update against the stack. This is like running pulumi up against a stack.
Finally, the deleteStack function (starting on line 236 of main.go) destroys the resources in a stack. This equates to running pulumi destroy against a stack.

Using these functions, lines 94 through 197 perform the key task of taking action to stand up or tear down the Zephyr online store and all associated infrastructure. If the user passes the parameter “destroy” on the command line (meaning the Pulumi Automation API program is invoked using go run main.go destroy), then it will tear down the stacks. Otherwise, the Pulumi Automation API program will perform a refresh and then an update—much the same way that the pulumi CLI itself behaves.

Along the way, you can see the Pulumi Automation API program passing configuration values to the stacks being orchestrated (this is lines 177 through 182, for example):

// Set config values for app stack
appStack.SetConfig(ctx, "platformOrgName", auto.ConfigValue{Value: env.Organization, Secret: false})
appStack.SetConfig(ctx, "dataOrgName", auto.ConfigValue{Value: env.Organization, Secret: false})
appStack.SetConfig(ctx, "platformProjName", auto.ConfigValue{Value: env.PlatformProject.Name, Secret: false})
appStack.SetConfig(ctx, "dataProjName", auto.ConfigValue{Value: env.DataProject.Name, Secret: false})
appStack.SetConfig(ctx, "platformStackName", auto.ConfigValue{Value: env.StackName, Secret: false})
appStack.SetConfig(ctx, "dataStackName", auto.ConfigValue{Value: env.StackName, Secret: false})

Among other things, this means you can take your existing Pulumi programs, and—without making any changes—orchestrate them using an Automation API program! Further, in reviewing the code shown above, you can see that the Pulumi Automation API ties into Pulumi’s secrets functionality.

In the end, using Pulumi’s Automation API enabled Zephyr’s platform team to simplify the workflow for standing up the online store application and all of its associated infrastructure into a simple and straightforward process:

Developers ensure they have cloned the appropriate source repositories onto their system.
Developers run go run main.go (or go run main.go destroy) to stand up or tear down the online store and all associated infrastructure.

Although Zephyr uses multiple GitHub repositories, the Pulumi Automation API has no requirements for multiple repositories. You could use a monorepo, if that better suits your particular requirements.

Summarizing Best Practices: Automation API

This post discussed the following best practices when using Pulumi Automation API:

Use the Automation API to orchestrate operations on multiple stacks. In cases where you’ve split Pulumi projects (per the recommendations in the post on structuring Pulumi projects), you’ve introduced a potential opportunity for using Automation API to orchestrate operations. This is a great counter-balance to offset the relative increase in complexity that comes from breaking large Pulumi projects into smaller, more focused Pulumi projects.
Separate configuration from code. This is already a generally-accepted recommendation in most programming circles, and it applies to Automation API programs as well. Doing so enables you, if you wish, to distribute binary artifacts to teams instead of requiring them to execute the source code of your Automation API programs.
Use local source to decouple Automation API programs from Pulumi programs when separate teams are involved. When separate teams are creating, managing, or maintaining the Automation API programs and the Pulumi programs, using local source instead of inline source enables the teams to operate somewhat independently.
When there is a single team managing both Automation API programs and Pulumi programs, use inline source. This is also true when, perhaps due to compliance or regulatory concerns, the Automation API program should be more tightly coupled to the Pulumi programs it is automating.

The Zephyr series wraps up in the next post, where we summarize all our recommendations so far. We’ll also take a look at areas of potential future growth for the team at Zephyr. Don’t miss it!

Dependent Stack Updates with Pulumi Deployments

Evan Boyle — Tue, 20 Jun 2023 00:00:00 +0000

As infrastructure projects grow in size and complexity, you need to decompose infrastructure into smaller stacks to limit the blast radius of errors, extract and reference common layers like networking, and limit access to sensitive components. This comes with a coordination cost as you now need to figure out how to detect and propagate changes to downstream stacks in your dependency graph. Today we’re announcing two features that can help you manage this complexity by automatically updating dependent stacks:

Deployment Webhook Destinations for triggering downstream stack updates automatically
The Pulumi Auto Deploy Package for easily expressing dependencies and configuring downstream updates via Deployment Webhooks.

The following example shows automatic deployment of stacks with the following dependency graph
a
├── b
│ ├── d
│ ├── e
│ └── f
└── c
Whenever a node in the graph is updated, all downstream nodes will be automatically updated
via a webhook triggering Pulumi Deployments.

Automatically updating dependent stacks and providing a way to write Pulumi programs that can express nested stack updates are two of the most requested features across the Pulumi project, and we’re excited to enable both of these via Pulumi Cloud and Pulumi Deployments.

Deployment Webhook Destinations

Pulumi Webhooks allow you to pick one or more event types on a stack (i.e. update succeeded, or refresh failed) and automatically deliver an event to a destination such a slack channel or a webserver you’ve deployed with custom code. Now you can choose Pulumi Deployments as a webhook destination. When an update completes on the parent stack, a deployment will automatically be triggered on the destination stack. This makes it easy to keep dependent stacks up to date. For instance, you can use Deployment Webhook Destinations to configure every successful update on your networking stack to trigger downstream updates on your database and compute stacks.

import * as pulumi from "@pulumi/pulumi";
import * as pulumiservice from "@pulumi/pulumiservice";

const databaseWebhook = new pulumiservice.Webhook("databaseWebhook", {
 organizationName: "org",
 projectName: "network",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "database/prod",
 active: true,
 displayName: "deploy-database",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});
const computeWebhook = new pulumiservice.Webhook("computeWebhook", {
 organizationName: "org",
 projectName: "database",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "compute/prod",
 active: true,
 displayName: "deploy-compute",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using PulumiService = Pulumi.PulumiService;

return await Deployment.RunAsync(() =>
{
 var databaseWebhook = new PulumiService.Webhook("databaseWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "network",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "database/prod",
 Active = true,
 DisplayName = "deploy-database",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

 var computeWebhook = new PulumiService.Webhook("computeWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "database",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "compute/prod",
 Active = true,
 DisplayName = "deploy-compute",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

});

import pulumi
import pulumi_pulumiservice as pulumiservice

database_webhook = pulumiservice.Webhook("databaseWebhook",
 organization_name="org",
 project_name="network",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="database/prod",
 active=True,
 display_name="deploy-database",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])
compute_webhook = pulumiservice.Webhook("computeWebhook",
 organization_name="org",
 project_name="database",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="compute/prod",
 active=True,
 display_name="deploy-compute",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := pulumiservice.NewWebhook(ctx, "databaseWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("network"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("database/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-database"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 _, err = pulumiservice.NewWebhook(ctx, "computeWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("database"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("compute/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-compute"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
resources:
 databaseWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: network
 stackName: prod
 format: pulumi_deployments
 payloadUrl: database/prod
 active: true
 displayName: deploy-database
 filters:
 - update_succeeded
 computeWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: database
 stackName: prod
 format: pulumi_deployments
 payloadUrl: compute/prod
 active: true
 displayName: deploy-compute
 filters:
 - update_succeeded

Expressing Stack Dependencies with Pulumi Auto Deploy

We’ve also shipped a new Pulumi package called Pulumi Auto Deploy. It lets you simply express dependencies between stacks, and takes care of creating and updating the necessary Deployment Webhooks under the hood.

import * as autodeploy from "@pulumi/auto-deploy";
import * as pulumi from "@pulumi/pulumi";

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

const organization = pulumi.getOrganization();
const project = "dependency-example"

export const f = new autodeploy.AutoDeployer("auto-deployer-f", {
 organization,
 project,
 stack: "f",
 downstreamRefs: [],
});

export const e = new autodeploy.AutoDeployer("auto-deployer-e", {
 organization,
 project,
 stack: "e",
 downstreamRefs: [],
});

export const d = new autodeploy.AutoDeployer("auto-deployer-d", {
 organization,
 project,
 stack: "d",
 downstreamRefs: [],
});

export const c = new autodeploy.AutoDeployer("auto-deployer-c", {
 organization,
 project,
 stack: "c",
 downstreamRefs: [],
});

export const b = new autodeploy.AutoDeployer("auto-deployer-b", {
 organization,
 project,
 stack: "b",
 downstreamRefs: [d.ref, e.ref, f.ref],

});

export const a = new autodeploy.AutoDeployer("auto-deployer-a", {
 organization,
 project,
 stack: "a",
 downstreamRefs: [b.ref, c.ref],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AutoDeploy = Pulumi.AutoDeploy;

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

return await Deployment.RunAsync(() =>
{
 var projectVar = "dependency-example";

 var organization = "pulumi";

 var f = new AutoDeploy.AutoDeployer("f", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "f",
 DownstreamRefs = new[] {},
 });

 var e = new AutoDeploy.AutoDeployer("e", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "e",
 DownstreamRefs = new[] {},
 });

 var d = new AutoDeploy.AutoDeployer("d", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "d",
 DownstreamRefs = new[] {},
 });

 var c = new AutoDeploy.AutoDeployer("c", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "c",
 DownstreamRefs = new[] {},
 });

 var b = new AutoDeploy.AutoDeployer("b", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "b",
 DownstreamRefs = new[]
 {
 d.Ref,
 e.Ref,
 f.Ref,
 },
 });

 var a = new AutoDeploy.AutoDeployer("a", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "a",
 DownstreamRefs = new[]
 {
 b.Ref,
 c.Ref,
 },
 });

});

import pulumi
import pulumi_auto_deploy as auto_deploy

'''

The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
Whenever a node in the graph is updated,
all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
'''

project_var = "dependency-example"
organization = pulumi.get_organization()
f = auto_deploy.AutoDeployer("f",
 organization=organization,
 project=project_var,
 stack="f",
 downstream_refs=[])
e = auto_deploy.AutoDeployer("e",
 organization=organization,
 project=project_var,
 stack="e",
 downstream_refs=[])
d = auto_deploy.AutoDeployer("d",
 organization=organization,
 project=project_var,
 stack="d",
 downstream_refs=[])
c = auto_deploy.AutoDeployer("c",
 organization=organization,
 project=project_var,
 stack="c",
 downstream_refs=[])
b = auto_deploy.AutoDeployer("b",
 organization=organization,
 project=project_var,
 stack="b",
 downstream_refs=[
 d.ref,
 e.ref,
 f.ref,
 ])
a = auto_deploy.AutoDeployer("a",
 organization=organization,
 project=project_var,
 stack="a",
 downstream_refs=[
 b.ref,
 c.ref,
 ])

package main

import (
 "github.com/pulumi/pulumi-auto-deploy/sdk/go/autodeploy"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 /**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

 pulumi.Run(func(ctx *pulumi.Context) error {
 projectVar := "dependency-example"
 organization := "pulumi"
 f, err := autodeploy.NewAutoDeployer(ctx, "f", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("f"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 e, err := autodeploy.NewAutoDeployer(ctx, "e", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("e"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 d, err := autodeploy.NewAutoDeployer(ctx, "d", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("d"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 c, err := autodeploy.NewAutoDeployer(ctx, "c", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("c"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 b, err := autodeploy.NewAutoDeployer(ctx, "b", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("b"),
 DownstreamRefs: pulumi.StringArray{
 d.Ref,
 e.Ref,
 f.Ref,
 },
 })
 if err != nil {
 return err
 }
 _, err = autodeploy.NewAutoDeployer(ctx, "a", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("a"),
 DownstreamRefs: pulumi.StringArray{
 b.Ref,
 c.Ref,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
variables:
 project: dependency-example
 # TODO: update once https://github.com/pulumi/pulumi-yaml/issues/461 is fixed
 organization: pulumi
resources:
 f:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: f
 downstreamRefs: []
 e:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: e
 downstreamRefs: []
 d:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: d
 downstreamRefs: []
 c:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: c
 downstreamRefs: []
 b:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: b
 downstreamRefs:
 - ${d.ref}
 - ${e.ref}
 - ${f.ref}
 a:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: a
 downstreamRefs:
 - ${b.ref}
 - ${c.ref}

Just declare an AutoDeployer resource for each stack, and any downstream stacks will automatically be updated whenever a stack in its dependency chain is updated.

Scale Infrastructure with Software

At Pulumi, we build tools to help you scale your infrastructure footprint with software instead of just humans. Deployment Webhook Destinations and pulumi-auto-deploy give you the tools to keep all of your dependent infrastructure up to date with automation instead of manual runbooks and human-driven operations.

Check out these resources to get started today:

Check out the Pulumi Deployments documentation and learn about configuring Deployment Settings for your stacks.
Get started with the Pulumi Auto Deploy Package.
Learn more about Pulumi Cloud Webhooks.
Join us in the #pulumi-deployments community slack channel.

🚀🚀🚀 Happy building! 🚀🚀🚀

Review Stacks: Collaborate in the Cloud

Evan Boyle — Wed, 14 Jun 2023 00:00:00 +0000

Today we’re excited to announce Review Stacks – dedicated cloud environments that get created automatically every time a pull request is opened, all powered by Pulumi Deployments. Open a pull request, and Pulumi Deployments will stand up a stack with your changes and the Pulumi GitHub App will add a PR comment with the outputs from your deployment. Merge the PR and Pulumi Deployments will destroy the stack and free up the associated resources. It has never been simpler to pick up an unfamiliar codebase, make changes to both application and infrastructure code, and share a live environment for review with your teammates.

Ephemeral environments for front-end development and static websites have become commonplace in recent years, but infrastructure and complex systems have been unsupported up until this point. Review Stacks enable you to iterate on both application code changes and infrastructure code changes at the same time. Making changes to HTML, CSS, Next.js, Go APIs, Pulumi configuration, subnet configuration, databases, and load balancers are all possible with Review Stacks. No more fumbling with out-of-date READMEs and developer documentation. Just open a pull request and you can start testing changes against everything from simple static websites to API servers, microservices, data pipelines, Kubernetes clusters, and any other piece of infrastructure across Pulumi’s 100+ cloud providers.

Review Stacks manage the full lifecycle of your cloud development environment including creating it when the PR is opened, updating it every time a new commit is pushed, and destroying and cleaning up all cloud resources when the pull request is merged or closed. They even integrate with GitHub checks so that you can gate your pull requests on successful Review Stack deployments.

Pulumi customers already using Review Stacks find that it saves their developers time. Having a standardized process for development environments and change management makes it easier to onboard new developers, and load-balance teammates onto unfamiliar projects and codebases. It also makes it substantially easier for developers to work on multiple feature branches and bug fixes at the same time, as they don’t have to worry about manually maintaining and juggling development environments.

“Pulumi allows every engineer to build and update infrastructure instead of only our infrastructure team, which enables us to ship faster than if we were using a domain-specific language,” said Aaron Gibralter, head of engineering, Mighty. “With Pulumi Deployments, we could easily set up a standardized CI/CD infrastructure workflow for every engineer that immediately makes them productive. Review Stacks automates the work of managing our own ephemeral infrastructure and makes our lives easier.”

Review Stacks also provide an avenue for optimizing cloud costs. Rather than keeping development infrastructure running 24/7, Review Stacks only exist as long as an engineer works on a feature. Since Review Stacks are flexible and built on Pulumi IaC, it is also possible to configure it to deploy to multi-tenant environments that share development infrastructure to further streamline costs.

“At GitHub, we are constantly seeking to improve the developer experience and our partner ecosystem plays a big role in that.” said Matthew Manning, Senior Business Development Manager for GitHub Actions, “The launch of Pulumi Review Stacks with GitHub makes it easier for teams to review and test their products before shipping which ultimately improves the developer experience.”

Configuring Review Stacks

Configuring Review Stacks is a simple three-step process:

Create a new stack, by convention named pr, and corresponding Pulumi.pr.yaml configuration file - this config will be copied into every review stack that gets created, and can even be modified within a PR.
Configure Deployment Settings for the stack - this specifies how to acquire source code, cloud credentials and more when deploying via Pulumi Deployments.
Set the pullRequestTemplate Deployment Setting to true - this indicates that all pull requests against this stack’s branch should reference this stack as a Review Stack template.

You can use an existing stack as a Review Stack template, as long as it has Deployment Settings configured. This will result in Review Stacks being deployed into the same cloud account. If you want to separate the cloud resources in your production stack from the resources created via Review Stacks then you can create a separate stack and template that references a different cloud account (AWS, Azure, GCP, etc).

Review Stacks and Deployment Settings can be configured via the Pulumi Cloud console, the Pulumi Cloud REST API, or within a Pulumi Program using the Pulumi Cloud Resource Provider.

Pulumi Cloud UI

It is just one click to turn on Review Stacks via the Pulumi Cloud console.

REST API

You can programmatically configure Review Stacks and Deployment Settings at scale across thousands of projects using the Deployments REST API.

curl -i -XPOST -H "Content-Type: application/json" -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
--location "https://api.pulumi.com/api/stacks/org/project/stack/deployments/settings" \
-d '{
"gitHub":{
"pullRequestTemplate": true
}
}'

Pulumi Cloud Resource Provider

You can use Pulumi to manage and code review Deployment Settings and Review Stacks with the Pulumi Cloud Provider.

import * as pulumiservice from "@pulumi/pulumiservice";

const deploymentSettings = new pulumiservice.DeploymentSettings("deploymentSettings", {
 organization: pulumi.getOrganization(),
 project: "your project",
 stack: "your stack",
 github: {
 deployCommits: true,
 previewPullRequests: true,
 pullRequestTemplate: true,
 repository: "pulumi/deployment-automation",
 },
 sourceContext: {
 git: {
 branch: "refs/heads/main",
 repoDir: "pulumi-pet-shop",
 },
 },
});

Collaborate in the Cloud

At Pulumi, our engineering team writes a ton of code. We have hundreds of repositories with programs managing thousands of stacks and tens of thousands of cloud resources. These tools and services power production workloads, internal tools, business intelligence and process, and more. And of course we use Pulumi to ship all of it. When you’re an engineer working in one of these repositories, making a code change is the easy part. The hard part is figuring out how to get a tight dev loop going—in particular, faithfully verifying your changes before merging and deploying them. Are there instructions in the README for running locally? A Docker Compose file? Or maybe standalone dev environments? Can I figure out what config and upstream stack references are required to stand up a fresh dev stack? And what about service dependencies? Pulumi makes it incredibly easy to build in the cloud and write prolific amounts of code, however, with that increase in development velocity comes a need to standardize the entire process by which we develop, test, review, and ship our code.

We’ve heard similar stories from hundreds of customers. From startup teams of five to enterprises of thousands, they’re all looking for the same thing - so we went ahead and built it. With Review Stacks and Pulumi Deployments, you can ship with greater velocity, confidence, and collaboration.

Check out these resources and get started today:

Review Stacks documentation
Learn more about Pulumi Deployments
Join us in the #pulumi-deployments community slack channel

🚀🚀🚀 Happy building! 🚀🚀🚀

Pulumi Deployments: API-Driven Infrastructure at Scale

Evan Boyle — Wed, 29 Mar 2023 00:00:00 +0000

Delivering software has been conventionally driven by CI/CD workflows. A single commit is merged into a codebase, and a small, and static set of workflow runs are triggered by the CI system to update the appropriate environments (Dev -> Staging -> Production). This may have been fine when the only product a company had to offer was a single stateless web service, but increasingly companies are called upon to deliver cloud infrastructure as a product.

There are obvious forms of this, such as IaaS companies building database-as-a-service offerings and SaaS companies deploying self-hosted installations into customer environments. But there are less apparent forms like internal platform teams that are chartered with enabling self-service of Kubernetes clusters configured with best practices to thousands of application engineers within an enterprise. In any of these cases, engineers are delivering infrastructure as the product but only have conventional tools like gitops and CI/CD to reach for.

The assumptions that CI/CD systems make are limiting for infrastructure. We are not pushing a single application commit across three environments. We are templating infrastructure hundreds and thousands of times. Once that infrastructure is templated, we’re responsible for ensuring it is secure, up-to-date, and can be customized by self-service users. CI/CD is a poor fit due to the design choices that are locked in around source control and source events. CI/CD is typically configured via templated YAML that is checked into source control, with no way to create or customize behavior dynamically from a REST API. And beyond configuration, there aren’t REST APIs for triggering updates and workflow runs. The best you can expect is the ability to add more YAML to configure a cron job. What do you do if you need to create a self-service platform, or configure CI/CD across thousands of projects across your organization?

Today we’re excited to announce the release of a new version of Pulumi Deployments, a fully managed platform for shipping infrastructure at scale. It provides:

Compute - run infrastructure deployments and workloads (i.e. pulumi up) on fully managed, secure, and elastic compute.
Configuration - A simple, centralized format for specifying everything you need to deploy your infrastructure at the stack level. Source code, cloud credentials, OpenID Connect (OIDC), environment variables, build prerequisites, and dependencies are bundled in an atomic unit.
Composition - Trigger deployments via our REST API, click to deploy in the Pulumi Service UI, Git Push to Deploy, Remote Automation API, and other programmatic building blocks that you can compose to build custom cloud automation.

Pulumi Deployments is built on top of a powerful set of public REST APIs. Unlike any other platform, Deployments lets you configure CI/CD settings programmatically. Service-catalog scenarios and guard rails that enforce CI/CD best practices across thousands of projects are far easier to achieve when you have REST APIs that you can build software on top of. No more generating and templating YAML to configure infrastructure deployments at scale.

Deployment Settings and Triggers

Deployments can be configured at a stack level via Deployment Settings in three ways:

The Pulumi Service UI enables setting up OIDC and git push to deploy in just a few clicks
The Pulumi Service Provider lets you define deployment configuration as Pulumi code in you language of choice, reviewed and managed in source control.
The Deployment Settings REST API provides programmatic access to configure Deployments and CI/CD dynamically, without needing to generate or check in YAML.

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();

// deployment settings configured via a pulumi program
const settings = new service.DeploymentSettings("deployment_settings", {
 organization: "service-provider-test-org",
 project: "test-deployment-settings-proj",
 stack: "dev",
 operationContext: {
 environmentVariables: {
 TEST_VAR: "foo",
 SECRET_VAR: config.requireSecret("my_secret"),
 }
 },
 sourceContext: {
 git: {
 repoUrl: "https://github.com/pulumi/deploy-demos.git",
 branch: "refs/heads/main",
 repoDir: "pulumi-programs/simple-resource"
 }
 }
});

Each stack configured with Pulumi Deployments gets access to a number of triggers that can be used to start a Pulumi operation (i.e. pulumi up):

Deployments REST API - an endpoint available at api.pulumi.com that can trigger a variety of pulumi (up/destroy/refresh/preview) operations. Calls to this trigger can override individual settings to customize deployment behavior (i.e. - deploy a different branch, or override a particular environment variable).
git push to Deploy - Deep integration with the Pulumi GitHub App that enables automatic previews on every pull request, and an automated update whenever code is pushed to the target branch.
Click to Deploy - trigger a deployment from the Pulumi Service Console with the push of a button. Very useful for day-to-day operational activities such as debugging a stuck stack, or rectifying drift via a refresh without having to pull a stack and its accompanying source code onto your machine.
Remote Automation API - Your favorite strongly typed Pulumi orchestration tool, Automation API, lets you offload update workloads into the cloud. Write deployment scripts that update thousands of environments simultaneously without worrying about compute.

You can mix, match, and compose Deployment Settings and Triggers to create a golden path to production that is perfect for your enterprise’s needs.

Example: Building RESTful Infrastructure APIs

Increasingly, infrastructure updates happen in response to user action and not source control events like a git commit. For instance, a SaaS database company might need to spin up dedicated compute and storage when a customer puts in a credit card and self-serves onto a specific SKU. This signup workflow might happen hundreds or even thousands of times an hour. Traditional deployment systems are optimized for infrastructure that spans, at most, a handful of environments (dev/staging/production).

Leading SaaS and infrastructure companies want the best of both worlds:

Custom, domain-specific REST APIs to create infrastructure on demand.
Desired state configuration that keeps track of what is deployed, and where.

You can build on top of the Deployments REST API to build your own infrastructure APIs. Here we have a simple Go web server that offers a RESTful interface over top of a static website. Users can create, update, and delete static websites, specifying the content for the site in the POST payload. Here’s our web server:

 router := httprouter.New()
 router.POST("/sites", server.create)
 router.GET("/sites/:id", server.get)
 router.POST("/sites/:id", server.update)
 router.DELETE("/sites/:id", server.delete)

 http.ListenAndServe(*addr, router)

Within the create handler, we first dynamically configure Deployment Settings for the stack, including the branch of the Pulumi program to be deployed, the AWS OIDC provider used for cloud credentials, and the configuration of git push Deployment Triggers`:

 err := s.client.patchDeploymentSettings(r.Context(), s.org, s.project, stack, DeploymentSettings{
 SourceContext: &sourceContext{
 Git: gitContext{
 Branch: s.branch,
 RepoDir: s.dir,
 },
 },
 OperationContext: &operationContext{
 Environment: map[string]string{
 "AWS_REGION": s.region,
 },
 OIDC: &oidcContext{
 AWS: &awsOIDCContext{
 RoleARN: s.roleARN,
 SessionName: s.sessionName,
 },
 },
 },
 GitHub: &gitHubContext{
 Repository: s.repository,
 Paths: paths,
 DeployCommits: true,
 PreviewPullRequests: false,
 },
 })

These settings configure the stack to trigger an update when new commits are pushed to the branch. In addition, our create handler also kicks off the initial update to create the infrastructure:

err = s.client.createDeployment(ctx, s.org, s.project, stack, createDeploymentRequest{
 DeploymentSettings: DeploymentSettings{
 OperationContext: &operationContext{
 Environment: map[string]string{
 "SITE_CONTENT": content,
 },
 },
 },
 InheritSettings: true,
 Operation: "update",
 })

Notice how this REST API call specifies InheritSettings: true so that Deployment Settings are read from the Pulumi Service and merged with the incoming request payload to create deployment configuration for this run. This stack also has to Click to Deploy via the Pulumi Service UI, so operational tasks like refreshing the stack can be done on demand without pulling down source code onto your local machine.

The Deployments Platform does all of the heavy lifting of managing deployment compute, providing asynchronous workflow orchestration, queueing, status and logging API access, and more. At the end, you get a Pulumi Stack and state file containing a manifest of all cloud resources and their properties managed by the Pulumi Service.

Deployment Settings and Triggers can be combined to ship infrastructure in novel ways with just a few lines of code. See the full RESTful infrastructure API example to try it out yourself.

Pave your Golden Path Today

Pulumi Deployments is the platform for shipping infrastructure at scale. It gives you the programmatic building blocks you need to deliver infrastructure on demand in a large organization. Get started with Deployments today. Happy automating!

FinOps With Pulumi

Matt Small — Tue, 14 Feb 2023 00:00:00 +0000

What is FinOps?

The FinOps Foundation eloquently defines FinOps as “an evolving cloud financial management discipline and cultural practice that enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions.” Simply put, FinOps is the continuous effort to control cloud spend.

Just as organizations have adopted operations-focused best practices into software development cycles and have considered how to best insert security best practices along the way, financial best practices may also be codified by developers writing cloud programs.

Adopting a FinOps practice brings real world OpEx(Operating Expenses) considerations to the decisions that cloud engineers regularly make. When you add FinOps to your Cloud Center of Excellence, technical teams bring financial context into service level and recovery objectives, scaling and placement decisions, and business continuity planning. Developers are typically more conscious of the resources they are using and consider the impact of their application design decisions at both development, test and production scale. Therefore organizations usually realize substantial OpEx savings while better forecasting their growth.

Who is Responsible for Cloud Financial Decisions?

In most smaller organizations, developers themselves are often responsible for the decisions that impact cost. In larger organizations, this responsibility typically falls to a central IT or Cloud Platform team, often seen as the cost center for infrastructure, and the best team to optimize that spend continuously. Some organizations are evolving toward FinOps as a practice, with dedicated teams responsible for enabling the organization to make smarter financial decisions and negotiating with cloud vendors for discounts.

Ultimately, the cloud account owner is responsible for the cloud bill, and failure to pay that bill will result in the suspension of cloud services. Given that infrastructure is often code, developers or IT teams are often made responsible for the task of optimization. Optimization recommendations may range from simply deleting orphaned cloud resources to re-architecting the entire application to use a modern serverless framework that would consume less resources overall.

Establishing FinOps Practices

The first step of establishing many FinOps practices is determining the allocation of costs. It is easy to allocate costs to individual users launching cloud services, but it becomes compoundingly hard once pipeline automation, team structure, and business unit reporting come into play. As an organization scales, it grows increasingly difficult to allocate costs across different business units and teams, across environments such as development versus production, or across application component services such as the data layer. Further, how does an organization account for supporting systems and services running alongside applications in the cloud, including third-party commercial software?

When starting your FinOps Journey is essential to understand cost allocation, typically done with consistent and enforced tagging of cloud resources. Creating a Tag Policy is covered below.

FinOps: Reactive vs. Proactive Optimization

Most FinOps processes today are reactive; once you have received your bill, you will observe (and pay) the costs and resources accounted for, review your architecture and make necessary adjustments such as reserving, reallocating or removing resources. Next month, you will do it again, also factoring in your company’s cloud growth, application design changes, team shifts and other dynamics that occurred in the meantime. Many platforms, tools, and job descriptions are oriented toward this reactive optimization process.

Proactive optimization and enforcement means providing real time feedback on how cloud engineering architectural changes may impact cost and creating preventative policies which do not allow for budget-busting cloud services to be provisioned. This is an evolving practice in many modern cloud-centric organizations. As more cloud operations are defined as code, this preventative, proactive optimization model is becoming more popular.

Financial optimization can also come in the form of strategic sourcing; most cloud providers offer multiple discount mechanisms, including large enterprise discount plans for committed consumption that may extend to third party SaaS software purchased through the cloud marketplaces, as Pulumi is. This level of optimization will provide dramatic savings on your top-line spend.

FinOps with Pulumi

Using cloud engineering practices with Pulumi, you have the opportunity to implement both proactive and reactive models to control your cloud spend.

As an infrastructure as code platform, Pulumi is in the unique position to be proactive about managing your costs.

You can design guardrails on your provisioning to limit expensive resource types or quantities,
Direct resources to reserved allocations,
Ensure resources are tagged and traceable to the correct cost center,
Connect resources to your observability and dedicated FinOps platforms,
Provide preventative visibility and control.

Consider uses for CrossGuard Policy as Code, multi-language components for defining resource abstractions, and Automation API to package up orchestration into web service endpoints, as described in Pulumi Features below.

If you are proactively managing your infrastructure, then reactively, the largest contributor to financial overrun is drift; the misalignment between the desired state and the actual state. Pulumi is used to define your desired state; however, many things may cause the actual state to begin to deviate from the desired state in your code, such as manual actions from the cloud console or ad-hoc deployment pipelines.

Drift detection, which is the name given to continuously scanning your environment to compare the actual state with the desired state and alerting and/or automatically remediating the difference in state, is a primary method of managing these creeping cloud costs. Leverage an automation system, such as a CI tool, to implement continuous scanning against your Pulumi code as described below.

Remediation of any infrastructure mismanagement requires infrastructure reconfiguration action to be taken; often resources need to be replaced with better suited, more cost effective resources, and resources that were provisioned outside of acceptable policies may have to be destroyed. However, any remediation should carefully consider the underlying reasoning for the misalignment in the first place. That should include both the reason for the initial deployment specifications as well as any decisions for why that’s no longer acceptable. Be sure to consider all architectural decisions, such as disaster recovery and business continuity planning, regional latency, and any other impact to service level objectives (SLOs). Understanding the context will help you code preventative measures into your Pulumi programs going forward and the self-documenting nature of infrastructure-as-code will ensure your decisions are memorialized for future developers and operators, for example, in a Stack README as described.

Automation of remediation is a goal for many cloud engineering organizations; if the perfect state is defined in code, then any drift may indicate the need to reconcile back to the desired perfect state. Given that some cloud resources may necessarily drift due to limitations in how they are managed, a combination of alerting and remediation of known patterns is implemented. For example, you might choose to detect and deprovision any newly instantiated resource not provisioned by Pulumi, but allow manual resizing of resources if needed. Just be sure to capture any infrastructure and improvements you want to make default and/or repeatable as infrastructure as code.

Best First Action: Implement a Tag Policy

Getting started with a FinOps usually falls to the resource owners or Cloud Engineering teams themselves. The more resources you have, the more complex and noisy things are going to get and the harder it will be to both allocate costs and resources where they are needed, as well as identify and eliminate the unneeded ones.

Implementing Tag Policy is the best first action to take. Tags are the set of required cloud metadata tags for any resources provisioned by your organization. While not every resource in every cloud may be tagged, most are. Getting into the practice early will help you maintain accountability for what you are running, demystify your cloud bills as they grow larger, and help update the application architecture or make cost-effective product decisions long-term.

A Basic Tag Policy for FinOps will include:

Owner - the person responsible for this resource that you would contact if you had to (e.g., johndoe@pulumi.com)
Origin - the pipeline, process, and/or system through which this came to be (e.g., ArgoCD, Jenkins, Pulumi)
Business Unit - the team responsible typically maps to the cost center associated (e.g., Sales, Engineering)
Environment - the teams/customers this deployment services (e.g., Dev, Staging, Production)
Application Component - the part of the application that this resource serves (e.g., data pipeline, authentication, front end)*

You will now be able to report on and manage resources in those tag groups. It’s advisable that you do not create too many required tags, or it will quickly get difficult to manage and it increases the likelihood of multiple tags applying, which reduces the insightfulness of the data.

Read more about Tag Policy implementation with Pulumi below, and for extended reading, refer to this post, Automatically Enforcing AWS Resource Tagging Policies from CEO, Joe Duffy.

Pulumi Features in Context

Multi-Language Components

Multi-language components allow you to create repeatable infrastructure patterns, modular abstractions of resources and/or groups of resources that meet your desired configuration standards. Packaging up resources within and across clouds, implementing standard deployment best practices, and making those components available to developers reduces the cognitive load that they have when building new applications from those pre-assembled, ready-to-run components.

Use these guardrails to enforce Tag Policies, for example that every tag for “Production” is actually “Production” and not “Prod” or “production” (yes, case sensitivity matters). Or you may ensure teams only deploy resources such as VMs to regions where you have allocated Reserved Instances or other financial benefits in your application architecture. With MLCs end users have access to your component and not to the underlying infrastructure resources, guaranteeing repeatabality, however, in order to realize the benefits of MLCs, your developers must be empowered to leverage them, but limited from using other methods.

CrossGuard

CrossGuard Policy as Code allows an Organization to define global rules for how resources may be provisioned and prevent resources from being provisioned by Pulumi in any other configuration. Pulumi Stacks that do not meet the Organization’s financial, security, data protection, or other pre-defined standards are prevented from provisioning any resources, and the developer is instructed to correct their desired configuration or program in accordance with the Organization-wide policies.

Policy as code ensures that any Pulumi code written and run meets an organization’s stated policies. It is the ultimate preventative weapon to prevent financial, security, or other placement and provisioning decisions that may lead to exposure or overrun.

Go here for examples of FinOps policies to get you started!

Automation API

Of the many things Automation API allows you to do, one is you can create an HTTP endpoint for a Pulumi program. Organizations have used this to control easy on/off switches for applications such as provisioning/deprovisioning single tenant SaaS architectures, short lived environments, and complex testing and blue/green deployments. Because of the encapsulation of a Pulumi program and the simplification this provides, this endpoint may be directly exposed to those responsible for making the financial decisions in the moment, such as a Sales Engineer who must turn on a new POC environment for testing who then must be sure to shut that off when completed. Oftentimes, these decisions are automated in processes and pipelines.

Pulumi FinOps in Practice: Sweeping up Dev and QA

A common automation in FinOps is to destroy things that don’t meet your Tag Policy. These “Sweeper Bots” will continuously query AWS for resources with specific tags and delete them. Read more about how Pulumi has implemented this using Pulumi to set tags and Lambda to sweep through and remove resources that meet certain specifications in this post, Controlling AWS Costs with Lambda and Pulumi.

Conclusion

In the manner of “Crawl, Walk, Run”, it’s best to start by implementing policies as code in a manner that will cause the most benefit while causing the least amount of change and/or downtime. Make sure you have the appropriate billing tags applied to all supported resources, ensuring you are able to allocate costs to the appropriate business units in your company. Then add in additional policies to make sure you are using instances covered by any reserved instance or similar savings plans. Continue evolving your infrastructure and policy as code strategy to incorporate both preventative and reactive checks and balances.

Have questions about using Pulumi when implementing FinOps in your IaC pipelines? Contact us and let us know your questions or request a demo.

Pulumi Deployments: the Fastest Way to Go from Code to Cloud

Evan Boyle — Wed, 02 Nov 2022 06:10:00 -0700

Portions of this blog post are out of date. See the drift detection and time-to-live stacks blog posts for an updated overview and examples.

Adoption of modern cloud technologies and services is driving enormous value for organizations, but many companies are finding that deploying and managing cloud infrastructure is a bottleneck on how fast they can scale. When your tools can’t keep up, scaling your cloud footprint means proportionally scaling your headcount. Things quickly fall apart as demand for cloud infrastructure often outpaces the bandwidth of infrastructure teams. This leads to slow product releases, longer lead times to get new products to market, and burned-out operational teams.

At Pulumi, we want every engineer and organization to be able to take full advantage of the cloud. The cloud should be an accelerant to your business and not a bottleneck. Today we are excited to be launching Pulumi Deployments, a new collection of features to power infrastructure and platform automation and unlock the scale of the cloud.

With Pulumi Deployments, you can run a Pulumi infrastructure as code action (a preview, update, destroy or refresh) inside Pulumi’s managed service. Pulumi provides scalability, observability, and security for deployments. It consists of five components:

The Deployments REST API - A fully managed REST API to execute Pulumi programs with the Pulumi Service. This includes APIs to observe your deployment and all associated logs.
Deployment Visualization - A dashboard for deployment observability including deployment status, logs, and queueing via the Pulumi Service console.
Click to Deploy - Deploy infrastructure with the click of a button from the Pulumi Service console.
git push to Deploy - Configure automatic infrastructure deployments in response to git push events by installing and configuring the Pulumi GitHub App.
Remote Automation API - Our strongly typed infrastructure orchestration technology can now run deployments remotely in the cloud.

Both the Programmatic Deployment API and Remote Automation API are flexible building blocks that simplify the creation and management of common platform automation scenarios like multi-region deployments, drift detection, and ephemeral infrastructure. Click to Deploy and git push to Deploy gives your organization an out-of-the-box golden path to production. Whether you are building your internal cloud platform or using git push to Deploy, every deployment you schedule reaps the benefits of observability in the Pulumi Service. We’ve built an intuitive user experience that allows anyone in your organization to check deployment status, debug logs, and view deployment history.

Automation API & Deployments - Do More with Less

Two years ago, we released Automation API, a programmable SDK for the Pulumi engine that unlocks the full power of infrastructure as code automation. Organizations use it to do things like provisioning infrastructure at runtime for SaaS platforms and building higher-level systems that automate the full lifecycle of infrastructure management. Since then, it has performed millions of automated infrastructure updates. Analyzing our usage data, we found that organizations that center their cloud workflow around Automation API can manage more than ten times the infrastructure per engineer as compared to traditional IaC usage patterns.

This is our motivation for building Pulumi Deployments. We took the product lessons we learned from working hand in hand with leading SaaS and infrastructure companies building on Automation API and turned them into a fully managed service. Pulumi Deployments is the platform to power your platform. The tools that enable the giants to create and manage ten times the cloud footprint per engineer are now available to you as a fully managed service with increased scalability, reliability, and observability baked in.

A Golden Path to Production

Out of the box Pulumi Deployments offers your cloud engineering team a turn-key infrastructure delivery workflow without writing a single line of code. Every piece of infrastructure within your organization can rely on automatic cloud delivery.

`git push` to Deploy

Using the Pulumi Github App (GitLab, Bitbucket, and others coming soon), git push to Deploy enables a rich git-based workflow for safely reviewing infrastructure changes and rolling them out across environments. Opening a Pull Request on GitHub will automatically trigger a pulumi preview and post the results back to the PR. You can also enforce successful previews as a check that gates merging.

Once your PR is reviewed and merged, a Deployment runs automatically:

Setting this up is as easy as installing the Pulumi GitHub App and selecting the repository and branch for your stack:

You can ensure that every environment and every commit is automatically validated and kept up to date with just a few clicks. GitHub integration is available today, with support for GitLab, Atlassian Bitbucket, and others coming soon.

Click to Deploy

Sometimes infrastructure requires ad-hoc operations. You may want to run a refresh to rectify some cloud state changes:

Whatever the situation calls for, you can easily update, refresh, and modify infrastructure with the click of a button in the Pulumi Service console.

Power-up Your Platform with The Deployments API

Both git push to Deploy and Click to Deploy are built on top of the Pulumi Deployments REST API, the same API that is available to you as a building block. The Deployments API is fully managed and enables you to orchestrate cloud rollouts without worrying about capacity. Let’s look at a few ways you can embed it into your custom cloud platform.

Drift Detection

Manual edits within the AWS console can wreak havoc on your infrastructure if those changes aren’t brought back into source control. The last thing you want is to scale out a service just to have the next CI run undo that change.

You can build drift detection into your platform and get automated alerts on unexpected infrastructure changes.

aws.cloudwatch.onSchedule("drift-lambda", "cron(0/5 * * * ? *)", async() => {
 const outstandingDeployments = [];
 for(let s of stacks) {
 const url = `https://api.pulumi.com/api/stacks/${s.organization}/${s.project}/${s.stack}/deployments`;
 const response = await fetch(url, {
 method: "POST",
 headers,
 body: JSON.stringify(refreshPayload),
 });

 const deployment = await response.json();
 outstandingDeployments.push(deployment.id);
 }
})

This scheduled Lambda runs refresh operations through the Deployments REST API. When one of those deployments detects changes, we get notified. With platform automation that automatically and proactively detects changes, your cloud team can reduce the number of surprise incidents, freeing them up to focus on higher-leverage activities. The full source code is available for the Drift Detection example.

TTL Stacks and Temporary Infrastructure

Infrastructure waste is a common problem. It’s too easy to leave development infrastructure running accidentally and end up with a huge bill. Many companies have entire teams devoted to solving this problem.

With Pulumi Webhooks and the Deployments API we can build a system for tagging and reclaiming temporary infrastructure. Anyone in your organization can create temporary infrastructure with a simple workflow:

$ pulumi stack tag set ttl 360 # this stack expires six hours after it deploys
$ pulumi up # with the `ttl` tag preset, the infrastructure will be automatically destroyed at expiration

After the expiration period has elapsed, a deployment will be queued and the development infrastructure is automatically destroyed. The Deployments API can be embedded into your platform to reduce cloud waste and automate cost controls. The full source code is available for the TTL Stack Processor.

Managing Scale with Remote Automation API

With Automation API, Pulumi can be fully embedded inside your software projects to power a wide range of custom cloud infrastructure automation. Previously, Automation API could only run locally. With Remote Automation API, these same scripts can run locally while offloading deployment workloads to the cloud via the Deployments API. The infrastructure as code action (a preview, update, destroy or refresh) runs inside of our fully managed service.

const stack = await RemoteWorkspace.createOrSelectStack({
 stackName,
 url: "https://github.com/pulumi/examples.git",
 branch: "refs/heads/master",
 projectPath: project,
}, {
 envVars: {
 AWS_REGION: "us-west-2",
 AWS_ACCESS_KEY_ID: process.env.AWS_ACCESS_KEY_ID ?? "",
 AWS_SECRET_ACCESS_KEY: { secret: process.env.AWS_SECRET_ACCESS_KEY ?? "" },
 AWS_SESSION_TOKEN: { secret: process.env.AWS_SESSION_TOKEN ?? "" },
 },
});

const upRes = await stack.up({ onOutput: console.log });
console.log(`url: ${upRes.outputs.websiteUrl.value}`);

repo := auto.GitRepo{
 URL: "https://github.com/pulumi/examples.git",
 Branch: "refs/heads/master",
 ProjectPath: project,
}

env := map[string]auto.EnvVarValue{
 "AWS_REGION": {Value: "us-west-2"},
 "AWS_ACCESS_KEY_ID": {Value: os.Getenv("AWS_ACCESS_KEY_ID")},
 "AWS_SECRET_ACCESS_KEY": {Value: os.Getenv("AWS_SECRET_ACCESS_KEY"), Secret: true},
 "AWS_SESSION_TOKEN": {Value: os.Getenv("AWS_SESSION_TOKEN"), Secret: true},
}

s, err := auto.UpsertRemoteStackGitSource(ctx, stackName, repo, auto.RemoteEnvVars(env))
if err != nil {
 fmt.Printf("Failed to create or select stack: %v\n", err)
 os.Exit(1)
}

res, err := s.Up(ctx, optremoteup.ProgressStreams(os.Stdout))

stack = auto.create_or_select_remote_stack_git_source(
 stack_name=stack_name,
 url="https://github.com/pulumi/examples.git",
 branch="refs/heads/master",
 project_path=project,
 opts=auto.RemoteWorkspaceOptions(
 env_vars={
 "AWS_REGION": "us-west-2",
 "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
 "AWS_SECRET_ACCESS_KEY": auto.Secret(os.environ["AWS_SECRET_ACCESS_KEY"]),
 "AWS_SESSION_TOKEN": auto.Secret(os.environ["AWS_SESSION_TOKEN"]),
 },
 ),
)

up_res = stack.up(on_output=print)

var stack = await RemoteWorkspace.CreateOrSelectStackAsync(new RemoteGitProgramArgs(stackName, "https://github.com/pulumi/examples.git")
{
 Branch = "refs/heads/master",
 ProjectPath = project,
 EnvironmentVariables =
 {
 { "AWS_REGION", new EnvironmentVariableValue("us-west-2") },
 { "AWS_ACCESS_KEY_ID", RequireFromEnvironment("AWS_ACCESS_KEY_ID") },
 { "AWS_SECRET_ACCESS_KEY", RequireFromEnvironment("AWS_SECRET_ACCESS_KEY", isSecret: true) },
 { "AWS_SESSION_TOKEN", RequireFromEnvironment("AWS_SESSION_TOKEN", isSecret: true) },
 },
});

var result = await stack.UpAsync(new RemoteUpOptions { OnStandardOutput = Console.WriteLine });

Multi-region infrastructure build outs become a breeze, as workloads can be easily parallelized. Now you can run thousands of updates concurrently without having to manage the compute that runs them.

Centralized Observability

Whether you use the built-in cloud delivery workflows like Push to Deploy, or build your own custom platform with the REST API and Remote Automation API, you get full observability into every deployment. Anyone in your organization can view updates status, update logs, and the full history of deployments.

You can build platform automation without worrying about how you’ll surface the results across your product, engineering, and infrastructure teams.

Automation Accelerates the Cloud

We’ve taken the lessons from our industry-leading customers in cloud SaaS and infrastructure and packaged them into Pulumi Deployments. This suite of tools enables you to manage and scale ten times the cloud footprint as opposed to traditional IaC usage patterns. Whether you want a turn-key delivery workflow or highly-customizable platform building blocks, there is something for your organization.

Pulumi Deployments removes the cloud as a bottleneck and unlocks it as an accelerant for your business. It is available in preview today. Deployments are free during the preview, with usage-based pricing to come. Check out these resources to get started today:

Request access to the Pulumi Deployments preview
Pulumi Deployments examples repo
The Pulumi Deployments demo video
Pulumi Deployments documentation
Pulumi Deployments REST API documentation
Automation API examples

🚀🚀🚀 Happy building! 🚀🚀🚀

How Elkjøp Nordic enables self-service infrastructure for developers

George Huang — Fri, 01 Jul 2022 00:00:00 +0000

At PulumiUP 2022, Tomas Jansson, software architect at Elkjøp Nordic, gave a presentation on how to enable developers to self-service infrastructure by using Pulumi’s Automation API.

Elkjøp Nordic is the leading consumer electronics retailer in the Nordics. The company sells consumer electronics, mobile phones, computers, white goods, domestic appliances, and services linked to these products both directly to consumers and to businesses. It is an omnichannel retailer and serves customers both online and through more than 400 stores. Elkjøp has retail outlets in Norway, Sweden, Finland and Denmark, and franchise operations in Greenland and Faroe Islands.

To support the company’s modernization strategy, Tomas developed a self-service infrastructure platform that would enable any developer to easily provision cloud infrastructure that uses approved resources and security practices. The goal was to empower teams at Elkjøp to increase their development velocity by using the cloud, while maintaining guardrails that uphold security and compliance.

Watch the full presentation below or read on to learn the highlights of his presentation:

The Problem: Balancing Agility and Governance

To achieve his goals, Tomas knew he had to solve the problem of balancing freedom with control. On the one hand, he wanted to foster the creativity of his developers and encourage innovation. He wanted his team to feel they were trusted and had the room to create the best solutions they could. On the other hand, he knew he had to maintain some control. Elkjøp needed to keep its applications secure. It also needed good governance so that teams can keep track of who is making changes. Tomas also wanted to limit the impact of deploying untested code. Code that was still being developed needed to be confined to the developer’s own environment. Finding the right balance between independence and control was paramount. Tomas believed this would result in great software and in developers who would enjoy their work.

The Solution: A Self-Service Infrastructure Platform

Tomas’ solution was to build a self-service infrastructure platform. Self-service infrastructure makes it possible for developers to easily access the infrastructure they need while limiting access to resources they don’t need. In effect, self-service infrastructure is a walled garden. Within the garden, developers have all the freedom they need. However, nothing they do affects what’s outside the wall. Code that causes unintended changes only affects them and no one else.

Tomas built an application that provides a “walled garden” of self-service infrastructure. Developers can create the environments they need, including an Azure resource group and a Kubernetes namespace, all through a graphical user interface. Developers can define whether the environment is for development, test or production. They can also assign values, such as which team members should have admin privileges to the associated GitHub repositories or who can contribute. From there, the platform handles provisioning and configuring all of these resources.

Here is an example that shows how a developer can create a repository simply by selecting a few parameters from a drop-down list.

Tomas determined that the best way to implement self-service infrastructure was with Pulumi and its Automation API. By using Pulumi, he could use standard programming languages to define infrastructure while using familiar software tools like IDEs and test frameworks. Pulumi also natively supports Azure and Kubernetes with 100% coverage of each provider’s APIs and same-day access to new features and updates. Finally, the Pulumi Automation API would enable him to program infrastructure automation directly into his application.

The Pulumi Automation API is a programmatic interface for running Pulumi programs without the Pulumi CLI. Automation API encapsulates the functionality of the CLI (pulumi up, pulumi preview, pulumi destroy, pulumi stack init, etc.) but with more flexibility since infrastructure operations can be controlled programmatically.

How the self-service infrastructure platform was built

Here is the workflow for the Elkjøp self-service infrastructure platform.

The self-service platform is a web application that runs the Pulumi Automation API on the server side. The application uses Pulumi to provision and configure every environment resource such as Azure resource groups, Kubernetes namespaces, service principals, and Kubernetes service accounts. Each environment comes with a Git repository that’s used by developers to manage their deployments. The Git repositories are the locus of each environment because they store everything needed to deploy changes such as infrastructure code and environment credentials.

Using the Pulumi Automation API, the application creates an Azure resource group, a team in both Azure AD and GitHub, and a new GitHub repository. The application then grants administrative access to the GitHub repo to members of the AD group. In addition to creating a Git repository for each environment, the application uses stack references for each Azure environment to inject the associated credentials into the GitHub Secret Store. These credentials are scoped to each GitHub environment so that developers can only work in their own environment. If a team has multiple environments, each environment has its own credentials so developers can’t, for example, deploy from a test environment to a production environment. Secrets are managed through the Pulumi Service.

Every new repository comes with Pulumi programs that developers can use to deploy their application. GitHub Actions initiate the deployment. When an action triggers, Pulumi reads the credentials that were generated by the self-service application, allowing Pulumi to deploy new Kubernetes or Azure resources to the environment.

Watch the demo

To illustrate how Elkjøp’s self-service platform works, Tomas presented a simplified demo application that’s based on Elkjøp’s production platform. The code for the application is available at https://github.com/mastoj/pulumi-automation-demo.

The demo is a TypeScript application that includes an API that provisions Azure and GitHub resources. Pulumi Automation API code is used to deploy new stacks by calling handler code, which is a Pulumi program that defines resources and outputs. The handler code can create Azure resource groups or GitHub repositories, and stack references are used so that outputs generated by the Azure resources can be stored in GitHub (e.g., resource group credentials).

Ready to dive into how this works in detail? Watch the demo in the video below starting around 10:20.

Next steps

After watching the demo of a self-service infrastructure platform, you can try it out for yourself by checking out the demo code. You can also learn more about Pulumi Automation API in our documentation.

If you’re new to Pulumi, then follow our getting started guide.

Observable Infrastructure as Code

Kay Plößer — Thu, 20 Jan 2022 08:36:11 -0600

Pulumi community member Kay Plößer spent some time digging into setting up observability of a Pulumi deployment using Honeycomb. Read more to find out all the details on configuring Honeycomb and Pulumi together, with a side dish of Automation API!

Last few weeks, I procrastinated doing real product-related things with setting up monitoring for my infrastructure deployment.

I’m using @pulumidev, a cloud-provider independent IaC framework that allows me to define my infrastructure in TypeScript and use whatever provider I need.

I read something about Honeycomb’s observability approach to monitoring, and it got me thinking about my deployment code. Later, it will probably run with something like GitHub Actions, so it’s not on my machine when it executes. This means I need some insights in case things go wrong. But it would also be good to see how the deployment behaves over time. How do new resources affect deployment durations etc.?

This is what a trace could look like in Honeycomb:

Why not setting things up before I move deployment from my development machine to the cloud?

This article will show you how to make your Pulumi deployment observable with @honeycomb_io

Prerequisites

The Pulumi Automation-API

This API is a programmatical interface to the Pulumi CLI. Think Puppeteer for Pulumi instead of Chrome.

Since it controls the CLI, it can do everything the CLI can and more.

For our use case, we need information about which resource is deployed to tell Honeycomb about it, and it will create nice trace graphs for us.

To run a pulumi up command with the automation API, we need the following code:

const { LocalWorkspace } = require("@pulumi/pulumi/automation")

const main = async () => {
 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up()
}

main()

The @pulumi/pulumi package already includes the automation API, so it’s the only one we need.

We use the LocalWorkspace object to load our stack. This leaves the wholes stack intact; we can still deploy/destroy it with the CLI.

Then we call stack.up() and wait. The outputs generated from the API aren’t as pretty as the ones from the CLI, but they contain the same information.

Also, the up command of the API allows us to catch all events that are happening during the deployment. This is the data we’re after.

The Deployment Events

The up method of the stack object takes a config object. We can use the onEvent attribute to listen to all the changes.

stack.up({
 onEvent: (event) => {
 console.log(event)
 },
})

Here we log out every event. If we look at the output, we notice there are different types of events.

-The preludeEvent fires at the start with information about the stack config. -The resourcePreEvent fires before a resource is deployed, and the resOutputsEvent fires after it has been deployed. -The diagnosticEvent contains eventual errors that could happen. -The summaryEvent includes info about what has been done and if e everything was successful.

The event structure is a bit strange. The events don’t have a type attribute we could switch over, but an attribute called like the type of the event.

The following code shows how to get the right event, and where the data hides in that event:

onEvent: (event) => {
 if (event["preludeEvent"] !== undefined)
 return console.log(event.preludeEvent.config)

 if (event["resourcePreEvent"] !== undefined)
 return console.log(event.resourcePreEvent.metadata)

 if (event["resOutputsEvent"] !== undefined)
 return console.log(event.resOutputsEvent.metadata)

 if (event["diagnosticEvent"] !== undefined)
 return console.log(event.diagnosticEvent)

 if (event["summaryEvent"] !== undefined)
 return console.log(event.summaryEvent)
},

If we run things like this, we get all the information we need to send to Honeycomb. The next step is to tell Honeycomb what’s up.

Sending Data to Honeycomb

Honeycomb provides us with two libraries. The Node.js Beeline and Libhoney.

Libhoney is a low-level Honeycomb client that allows sending raw events to their service API. The Beeline is an abstraction on top of Libhoney to help with instrumentation and traces.

Usually, it would be best if you used the Beeline because it’s easy to set up and works with Node.js’s internal HTTP module and the Express framework out of the box.

In this case, we aren’t building an HTTP server, so the Beelines instrumentation doesn’t help much.

So for our deployment example, I will stick to Libhoney.

Honeycomb Events & Traces

Honeycomb uses events to figure out what’s happening in your system. Suppose the events have special attributes, like trace.trace_id and trace.span_id, Honeycomb can correlate them. This way, it can tell you things like, “The trace for your request took 300ms and consisted of 100ms authentication and 200ms database access.”

So you can pump all your information to Honeycomb via events. Sometimes it doesn’t need to be correlated; you only want to tell Honeycomb that your server uses 20% of memory. But in our case, we want to tie up all events related to one deployment into a trace, so we can figure out what we deploy, how long it takes, and if something goes wrong, what resource was responsible.

Linking up Pulumi & Honeycomb

We need to initialize Libhoney and send the correct “Honeycomb event” for every “Pulumi event.”

But let’s start with getting one event into Honeycomb: the pulumi up event.

const Libhoney = require("libhoney")
const { LocalWorkspace } = require("@pulumi/pulumi/automation")

const hny = new Libhoney({
 writeKey: "<HONEYCOMB_API_KEY>",
 dataset: "example-iac",
})

const id = (name) => `${name}-${Date.now()}`

const traceId = id`trace`
const rootSpanId = id`trace-root-span`
const startTimestamp = Date.now()

const main = async () => {
 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up({
 onEvent: (event) => {},
 })
}

main().then(() => {
 hny.sendNow({
 name: "up",
 service_name: "Pulumi",
 "trace.trace_id": traceId,
 "trace.span_id": rootSpanId,
 duration_ms: Date.now() - startTimestamp,
 })
})

We define a traceId, a rootSpanId, and a startTimestamp at the beginning, and then we send it off when everything is done.

An event with trace.trace_id will be grouped up with all other events with the same trace ID. In this case, we only have one event.

The name is usually the action you perform and want to measure, and the service_name is, well, the service that performs this action. In this case, we run pulumi up.

In the end, we also send a duration_ms attribute to track how long things take.

The result will look like this:

Not very interesting, but at least we know it run without crashing and how long it took.

Getting More Details

The next step is to get the details. What were the config parameters? How long did different parts of the deployment take?

For this, we need to look into two events, the preludeEvent and the summaryEvent.

Until now, we only send one Honeycomb event for the Pulumi up command.

Now, we will send three events for the stack.

The init event will contain the time it took to get to the actual deployment of resources and the config parameters of the stack
The run event will contain the duration all resources took to do their thing.
The finish event will contain the duration of shutting down the automation API and the summary data.

Let’s look at the updated example code:

...

const traceId = id`trace`
const rootSpanId = id`trace`
const startTimestamp = Date.now()
let stackFinishStartTimestamp

const main = async () => {
 const initStartTimestamp = Date.now()
 const runStackSpanId = id`stack`
 let stackRunStartTimestamp

 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up({
 onEvent: (event) => {
 if (event["preludeEvent"] !== undefined) {
 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(initStartTimestamp)
 hnyEvent.add({
 name: "init",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": id`stack`,
 duration_ms: Date.now() - initStartTimestamp,
 ...event.preludeEvent.config,
 })
 hnyEvent.send()
 stackRunStartTimestamp = Date.now()
 return
 }

 if (event["summaryEvent"] !== undefined) {
 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(stackRunStartTimestamp)
 hnyEvent.add({
 name: "run",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": runStackSpanId,
 duration_ms: Date.now() - stackRunStartTimestamp,
 })
 hnyEvent.send()
 stackFinishStartTimestamp = Date.now()
 return
 }
 },
 })
}

main().then(() => {
 let hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(stackFinishStartTimestamp)
 hnyEvent.add({
 name: "finish",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": id`stack`,
 duration_ms: Date.now() - stackFinishStartTimestamp,
 })
 hnyEvent.send()

 hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(startTimestamp)
 hnyEvent.add({
 name: "up",
 service_name: "Pulumi",
 "trace.trace_id": traceId,
 "trace.span_id": rootSpanId,
 duration_ms: Date.now() - startTimestamp,
 })
 hnyEvent.send()
})

First, we need to sprinkle variables on the code that capture the start times of our three events.

The init event starts before the call to the up command and takes until the preludeEvent is fired by the automation API.

The run event starts right after the init event finishes and takes until the summaryEvent fires.

The finish event starts right after run finishes and stops right before the parent event for Pulumi up is finished.

These events all get the trace.parent_id of the Pulumi up event, so they get nested in the Honeycomb view later.

Also, the init event gets the event.preludeEvent.config object, so we can see how the stack was configured inside Honeycomb.

If we run that example, we should see something like this:

Getting Resource Deployment Durations

The resources have additional details for us. As a name for the event, we will use the operation (create, update, delete) and as service_name we use the name of the resource in the Pulumi definition.

To get the parent of a resource, we have to save the URNs of every resource for later. But since we have to keep the start times anyway, we can do this in one step when the resourcePreEvent is fired.

Let’s look at the example code. This time I removed some boilerplate code and the other events to make things a bit more readable.

const main = async () => {
 // ...

 const resourceSpans = new Map()

 // ...

 await stack.up({
 onEvent: (event) => {
 // ...

 if (event["resourcePreEvent"] !== undefined) {
 const { metadata } = event.resourcePreEvent
 resourceSpans.set(metadata.urn, {
 spanId: id`span`,
 startTimestamp: Date.now(),
 })
 return
 }

 if (event["resOutputsEvent"] !== undefined) {
 const { metadata } = event.resOutputsEvent

 const serviceName = metadata.urn.split("::")[3]
 const { spanId, startTimestamp } = resourceSpans.get(metadata.urn)
 const parentUrn = metadata.new?.parent ?? metadata.old?.parent
 const parentId = resourceSpans.get(parentUrn)?.spanId ?? runStackSpanId

 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(startTimestamp)
 hnyEvent.add({
 name: metadata.op,
 service_name: serviceName,
 "trace.trace_id": traceId,
 "trace.parent_id": parentId,
 "trace.span_id": spanId,
 duration_ms: Date.now() - startTimestamp,
 })

 if (metadata.op === "update")
 hnyEvent.addField("diffs", metadata.diffs.join(", "))

 hnyEvent.send()
 return
 }

 // ...
 },
 })
}

So, what’s going on here?

First, the resources are deployed by their hierarchy, but siblings can be deployed in a different order. That’s why we need to save their spanId and startTimestamp when we get a resourcePreEvent.

I used a Map for this and keyed it with the URN of the resource because URNs are used for parent-child relationships and are part of all events we care about here.

Later, when the resOutputsEvent fires, it’s time to wrap it all up and send it to Honeycomb.

The event has the URN to find the related spanId and startTimestamp. We can also use the URN in the parent attribute of our resource to fetch the spanId of our resource’s parent. This way, our spans are correctly nested in Honeycomb later.

If the operation for the resource is an update, we also add the diffs to that event. This way, we can see in Honeycomb what inputs have changed since the last deployment.

The final trace should look a bit like this:

Getting Errors

The last step, getting deployment errors, I leave as an exercise for the reader.

But I’ll give you a few hints!

First, the interesting event is the diagnosticEvent, which comes with a severity and message attribute. If you only need errors, you have to filter out debug and info severity.

Also, the event includes an urn that can be used to correlate the error to a Honeycomb span later (resourceSpans etc.). If the errors isn’t related to an URN, you can simply use the run stack event as target.

If you add an error field to a Honeycomb event, it will automatically counted to your errors in Honeycomb.

Summary

IaC observability becomes rather simple when using Pulumi and Honeycomb.

Pulumi’s automation API allows us to control the CLI with JavaScript, and Honeycomb’s Libhoney is low-level enough to instrument every part of our deployment process.

This way we can keep track which resources are slowing our deployment down, or which was responsible for a failed deployment.

This article was originally posted at https://dev.to/fllstck/observable-infrastructure-as-code-52ha. Reposted here with permission.

Organizational Patterns: A Developer Portal

Matty Stratton — Thu, 30 Dec 2021 11:00:39 -0600

Using Pulumi is more than just writing code and components. In addition to common software development practices, there are also a number of success patterns related to how your company or team builds and deploys Pulumi programs to successfully build, deploy, and manage your infrastructure and applications. In this continuation of a series, I will explore one of these patterns - using the Pulumi Automation API to create a developer portal.

It is becoming more and more common to consider building out centralized web-based “portals” for developers (and other team members) to provision the infrastructure that their services and applications require. In many ways, the pattern of a developer portal takes portions of both the Single Infra Repo and Automation Team patterns covered in previous posts. The difference is that the developer portal is a single, centralized web application that can be used by multiple teams and organizations to provision infrastructure and applications, rather than writing the Pulumi programs themselves.

Depending upon the needs of the various teams, the developer portal could include options to deploy virtual machines into the cloud of their choice (or the organization’s standard cloud; perhaps the choice isn’t…a choice!), to provision cloud resources such as virtual networks, security groups, database instances, or even deployments to a kubernetes cluster.

Generally speaking, the developer portal is created and maintained by a central platform team. This is the team who knows the success patterns/standards that are employed by the organization, and can implement these patterns in the developer portal. The platform team is also responsible for maintaining the infrastructure that the developer portal requires.

One of the great advantages of providing a single portal for provisioning these resources is that all the “other stuff” that is often required (besides the resources from the hyperscaler) can be handled by the developer portal. For example, if the portal is used to provision a virtual machine, the code behind it can register the virtual machine with the company’s infrastructure management system.

A great example of what is possible in a portal can be seen with Komal Ali’s example, the Self Service Infrastructure Platyform. This is example code only, but it shows some of the capabilities of a developer portal.

In the example, the web portal is created using a combination of Python, Flask and Jinja templates (along with the Pulumi Automation API) to provide push-button access to deploy cloud resources according to the standards and patterns of the organization. The code snippets below illustrate some of the concepts (this code is not meant to be run, but rather to illustrate the concepts - the full example code can be found at github.com/komalali/self-service-platyform).

from flask import (Blueprint, current_app, request, flash,
 redirect, url_for, render_template)

import pulumi
import pulumi_aws as aws
import pulumi.automation as auto
import os
from pathlib import Path

bp = Blueprint("virtual_machines", __name__, url_prefix="/vms")
instance_types = ['c5.xlarge', 'p2.xlarge', 'p3.2xlarge']

In the initial snippet above, we are setting up what we need for the Flask application, but the key items to make note of are including the pulumi_aws and pulumi.automation packages, which will be used for the cloud specific resources and automation.


def create_pulumi_program(keydata: str, instance_type=str):
 ami = aws.ec2.get_ami(most_recent=True,
 owners=["amazon"],
 filters=[aws.GetAmiFilterArgs(name="name", values=["*amzn2-ami-minimal-hvm*"])])

 group = aws.ec2.SecurityGroup('web-secgrp',
 description='Enable SSH access',
 ingress=[aws.ec2.SecurityGroupIngressArgs(
 protocol='tcp',
 from_port=22,
 to_port=22,
 cidr_blocks=['0.0.0.0/0'],
 )])

Continuing on, we define the Pulumi program, and set up some of the standard settings that we consider our defaults and that are not selectable by the user.


 server = aws.ec2.Instance('dlami-server',
 instance_type=instance_type,
 vpc_security_group_ids=[group.id],
 key_name=keypair.id,
 ami=ami.id)

 pulumi.export('public_dns', server.public_dns)

This final snippet is what creates the actual instance, and it exports the public DNS of the instance so that it can be accessed by the user and presented in the portal.

Conclusion

This pattern has been implemented and deployed at many larger enterprises, where the platform team has the resources to dedicate to managing the portal as an actual product. This is a key part to remember - the developer portal is a product whose consumers are the developers and other teams in the organization, and it needs to be managed as such!

Don’t forget to check out the previous two posts in this series, Organizational Patterns - A Single Infra Repo and Organizational Patterns - An Automation Team".

Announcing Pulumi 3.0

Luke Hoban — Tue, 20 Apr 2021 00:00:06 -0700

Today we’re excited to announce the availability of Pulumi 3.0, the next major version of the Pulumi open source project, and the foundation for Pulumi’s Cloud Engineering Platform.

Pulumi offers the most complete infrastructure as code platform for building, deploying and managing modern cloud infrastructure and applications. Pulumi enables cloud engineers to use familiar languages to describe their cloud infrastructure - bringing core software engineering tools and practices to bear on managing and getting the maximum value from their cloud platforms of choice - across dozens of cloud and SaaS providers.

Pulumi 3.0 includes dozens of significant new features and hundreds of improvements that build on this foundation. This release includes more than 200 contributions from over 150 members of the Pulumi community, and builds on feedback from working with thousands of Pulumi users and customers over the last year.

Some of the most exciting new features in this release include:

Pulumi Automation API: All the power of Pulumi, now available to embed within your own software and applications in Node.js, Python, Go and .NET - opening up a brand new wave of modern cloud infrastructure use cases, including self-service infrastructure portals, custom CLIs, and richer CI/CD workflows.
Native Pulumi Providers for Azure (GA) and Google Cloud (Preview): New providers with 100% platform coverage, same-day support for new features, and the exact same resource models defined by the cloud providers - the most complete and consistent way to manage cloud infrastructure in Azure and Google Cloud.
Pulumi Packages and Multi-Language Components: Take any Pulumi components written in your language of choice, and make them available as a Pulumi Package to users in all other Pulumi languages - bringing together each of the language ecosystems Pulumi supports into a single Pulumi component ecosystem.
Improved Go and Python SDKs: Updates to the core Go and Python SDKs make APIs simpler and more consistent, unlock more tooling support, and improve compile times - making it even easier to use all the benefits of your favorite programming language when building and managing cloud infrastructure.
New Integrations and Features in the Pulumi Service: Alongside Pulumi 3.0, users of the Pulumi Service also now have access to new dashboards for organization-level cloud infrastructure insights, a CI/CD Assistant for easy version control and CI/CD integration with just a few clicks, and new identity, access control, and auditing capabilities.

👉 Get started with Pulumi 3.0 today or check out the Pulumi 3.0 migration guide to update your existing Pulumi projects to Pulumi 3.0.

Pulumi Automation API

Pulumi’s Automation API was released in preview last year and is now GA as part of Pulumi 3.0. It provides an API for embedding Pulumi within your own applications, and supports all Pulumi languages including TypeScript/JavaScript, Python, Go, and C#. Instead of working with the pulumi CLI directly, or writing scripts to execute the CLI within your CI/CD environment, you can now build your own software solutions that build on top of Pulumi’s robust, desired state infrastructure as code deployment engine. It’s simple to embed Pulumi in your software-defined workflows with just a few lines of code.

This opens up so many exciting new opportunities for how your team can manage cloud infrastructure; we’ve seen amazing creativity as Pulumi’s customers and communities have created solutions on top of Automation API. In addition, Pulumi Automation API powers several other new features like the Pulumi Kubernetes Operator, the Pulumi GitHub Action v3, and Concourse CI support for Pulumi.

👉 Read more in the Pulumi Automation API blog post.

Native Providers for Azure and Google Cloud

Pulumi offers packages for managing cloud infrastructure in more than 50 cloud and SaaS applications, with more than a dozen new providers added since Pulumi 2.0 including partnerships with Civo, Venafi, Okta, Auth0, Equinix Metal, Spot Ocean for Azure, and Splunk.

For the largest and most widely used cloud providers, we’ve created something new: Pulumi native providers, which give you the most complete and consistent interface for the modern cloud. These native providers are automatically generated from cloud provider APIs and resource models that are maintained directly by cloud provider service teams. This enables native providers to offer 100% API coverage, same-day support for new features, the exact same API designed by the provider service team, and fewer bugs and reliability issues. This makes native providers the best way to manage infrastructure in the supported cloud platforms, combining the benefits of Pulumi Infrastructure as Code with the resource models and pace of delivery of the cloud provider themselves.

Along with Pulumi 3.0, the new Azure Native provider, originally released in preview last year, is now generally available as version 1.0. The Azure Native provider is built on top of the Azure Resource Manager REST API specifications developed and maintained by Azure service teams. It supports more than 2x the number of resources, and 3x the number of resource properties as the classic Azure provider. Azure Native also supports every version of each resource API for maximum compatibility with your existing infrastructure. New versions of the Azure Native provider with support for the latest features of Azure are published every night, ensuring immediate availability of new features and improvements from the Azure service teams.

Today, we’re also excited to be releasing a preview of the new Google Cloud Native provider. This native provider offers 100% coverage of the Google Cloud platform, with 70% more resource properties supported compared to the existing provider. It also supports every version of each supported resource, including alpha, beta, and GA Google Cloud offerings.

👉 Read more in the Pulumi Native Providers blog post and the Pulumi Google Native Provider blog post. And of course, we haven’t forgotten AWS - an AWS Native provider is expected later this year.

Pulumi Packages and Multi-Language Components

Pulumi Packages are the core technology that enables cloud infrastructure resource provisioning to be defined once, and made available to users in all languages supported by Pulumi. If you’ve ever used a cloud provider in Pulumi, you’ve used a Pulumi Package. With Pulumi 3.0, we are extending Pulumi Packages to also support building and packaging Pulumi Components: higher-level abstractions over cloud infrastructure patterns described as normal Pulumi programs. This makes it easy to take a Pulumi program that defines a collection of resources and turn it into something reusable: a Pulumi Package that can be published and used by all Pulumi users, across all languages supported by Pulumi.

One example of building and packaging Pulumi Components as a Pulumi Package is the Pulumi EKS package. We originally released a component for easily building and managing production-ready EKS clusters nearly two years ago, but it was only available for Node.js users of Pulumi as an npm package. This component has been incredibly popular, as it abstracts away a lot of complexity in all the moving pieces that need to be managed to stand up an EKS cluster across both AWS resources and in-cluster Kubernetes resources. Recently, we turned this component into a Pulumi Package, making it available to users in Python, Go, and .NET as well, with just a single implementation. Pulumi EKS is just the first example of applying this technology; there’s a lot more in the works here in the coming months!

import pulumi
import pulumi_eks as eks

cluster = eks.Cluster("my-cluster")

pulumi.export("kubeconfig", cluster.kubeconfig)

package main

import (
 "github.com/pulumi/pulumi-eks/sdk/go/eks"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 cluster, err := eks.NewCluster(ctx, "my-cluster", nil)
 if err != nil {
 return err
 }
 ctx.Export("kubeconfig", cluster.Kubeconfig)
 return nil
 })
}

import * as eks from "@pulumi/eks";

const cluster = new eks.Cluster("my-cluster");

export const kubeconfig = cluster.kubeconfig;

using Pulumi;
using Pulumi.Eks;

class MyStack : Stack
{
 [Output("kubeconfig")]
 public Output<object> Kubeconfig { get; set; }

 public MyStack()
 {
 var cluster = new Cluster("my-cluster");

 Kubeconfig = cluster.Kubeconfig;
 }
}

All Pulumi packages define a common Pulumi schema, which describes the resource model the package exposes and can be used to generate language-specific SDKs and documentation. There are several different types of Pulumi Package:

Native Pulumi Provider Package: Use the full features of the Pulumi resource model to create a provider for a new cloud platform. Examples: the kubernetes, azure-native, and google-native packages.
Bridged Provider Package: Take an existing resource provider from another supported ecosystem (like a Terraform provider), and bridge it to be exposed as a Pulumi Package. Examples:the aws, tls, and cloudflare packages.
Component Package: Write a Pulumi Component in your language of choice and expose it to users in all languages supported by Pulumi. Example: the eks package.

Together, these options provide a combination of breadth and depth for the kinds of resources and components that can be exposed to Pulumi developers via Pulumi Packages.

👉 Read more in the Pulumi Packages blog post.

Improved Go and Python SDKs

Pulumi’s language SDKs bring the power of cloud infrastructure management into your language of choice in a natural and productive way, enabling you to use all of your existing software engineering knowledge and tools for your cloud infrastructure projects as well. In Pulumi 3.0, we’ve made improvements to the Go and Python SDKs to improve developer productivity and enable additional tooling.

For Go users, Pulumi 3.0 reduces binary sizes by 4x and improves build times by up to 3x, leading to a much faster infrastructure development loop. Along with this, the Go APIs for working with Pulumi Output values have been simplified to provide a single way of transforming outputs instead of three different options. This makes the Pulumi Go SDK more consistent and easier to get started with. You can learn more about these changes in the Pulumi 3.0 migration guide.

For Python users, Pulumi 3.0 brings a new, more strongly typed way to provide inputs to cloud resources, ensuring that users get completion lists, type checking, documentation, and more within their IDE of choice. Along with this, some inconsistencies in input property naming have also been removed; now, all property names follow a consistent and predictable naming convention. You can learn more about these changes in the Pulumi 3.0 migration guide.

New Integrations and Features in the Pulumi Service

In addition to the Pulumi 3.0 open source release, we’re also making available new features for users of the Pulumi Service: the simple and easy to use default backend for all Pulumi users, with additional features for teams and enterprises.

Integrating Pulumi into CI/CD systems is a key step in realizing the full benefits of modern infrastructure as code, and we’ve worked to make this easier than ever. Since 2.0, we’ve added partner integrations with Spinnaker, GitHub Actions and CodeFresh as well as a new Pulumi Kubernetes Operator. In addition, we’ve introduced a new CI/CD Assistant to the service to help configure version control and CI/CD integration for your Pulumi projects with just a few clicks.

For enterprise users, we’ve added several enhancements to SAML SSO and SCIM support. Pulumi’s SSO support now works seamlessly with Okta (as well as Google Identity, Azure Active Directory and more), users and groups can now be synced via SCIM support in your SAML provider, and new Team Admin roles make team management simpler for large organizations.

Lastly, the Pulumi Service now offers a dashboard landing page which brings insights about your infrastructure front-and-center whether you are an individual user, or part of a large organization.

👉 Read more in the Pulumi CI/CD Assistant blog post and the SAML SSO and SCIM blog post.

Keep exploring

Pulumi 3.0 is a significant new milestone for the Pulumi project and lays the foundations for the Pulumi Cloud Engineering Platform with support for the very best of the cloud via new Pulumi Native Providers, support for a unified ecosystem of reusable infrastructure components with Pulumi Packages, and support for a new era of cloud infrastructure management scenarios enabled by Pulumi Automation API. Each of these is just a start: there’s so much more to come on top of these foundations and we’re excited to work with our customers and community on the next wave of innovation in Cloud Engineering!

To learn more about each of the new areas - check out these blog posts with more details:

You can get started with Pulumi 3.0 today, or check out the Pulumi 3.0 migration guide to update your existing Pulumi projects to Pulumi 3.0.

We can’t wait to see what you build with Pulumi 3.0!

Build your perfect interface for the cloud: Automation API

Alex Mullans — Tue, 20 Apr 2021 00:00:02 -0700

Pulumi Automation API exposes the full power of infrastructure as code through a programmatic interface, instead of through CLI commands. With Automation API, you can create software that has the capability to provision and configure infrastructure: create, update, configure, and destroy infrastructure dynamically. You can create new classes of Software-as-a-Service that provision unique infrastructure for every customer. You can build the perfect cloud interface for your organization, combining all of your infrastructure knowledge and best practices into a purpose-built tool, whether it’s a SaaS offering, an internal web app, a purpose-built CLI tool, a CI/CD integration, or something else entirely.

We announced the general availability of Automation API today at PulumiUP and we’re excited to see what you create with it!

Making self-service infrastructure platforms easier

So, why might you want to create your own interface to the cloud? Let’s take a step back and review the journey that many organizations take when they embrace the modern cloud. Infrastructure engineers across the organization will typically start designing and iterating on infrastructure, using Pulumi interactively on the command line or sometimes using a graphical interface like the AWS Console and Azure Portal or an imperative CLI like aws or az. In this early phase, rapid iteration is important, as infrastructure engineers try different combinations of services, different configurations, and more until they land on a configuration that they want to make ready for production.

When they’re ready to go to production, the easy exploration and rapid iteration afforded by the cloud provider’s graphical and CLI interfaces become less important. Instead, new concerns about scalability, security, compliance, and cost control come to the forefront. At this point, it’s common to turn to infrastructure as code templates. Pulumi’s modern IaC allows infrastructure engineers to codify the infrastructure they’ve designed into a code file that can be managed like any other piece of software: checked into source control, forked and modified, and executed when needed by an engineer or application developer using the pulumi CLI in a terminal. In some organizations, this is the end of the journey: application developers start from the canonical infrastructure as code solutions created by their infrastructure team and modify them as needed to suit their unique needs. In those organizations, Pulumi’s policy-as-code offering can help infrastructure teams keep application teams “within the guardrails” as they customize their infrastructure.

However, not all organizations want to expose the full complexity of infrastructure, or an infrastructure as code tool like Pulumi, to their application developers. This is where Pulumi’s Automation API and the custom cloud interfaces that can be created with it come into play. Automation API allows infrastructure teams to provide self-service tools (e.g. developer portals, purpose-built CLIs, custom frameworks, and CI/CD workflows) for building, deploying, and managing infrastructure that offer the right levels of complexity and customization for the tool’s target audience. With Automation API, infrastructure teams scale their knowledge and expertise across the entire organization by encoding it into the tool they create. Application teams get self-service access to infrastructure without having to understand every underlying resource and option. Security and compliance teams get peace of mind that all of the organization’s infrastructure is created and managed through an approved tool that follows the right processes and procedures.

Unlocking new classes of Software-as-a-Service offerings

Before Automation API, most Software-as-a-Service (SaaS) products used multi-tenant infrastructure: infrastructure that’s shared across many of the SaaS’s customers. They did so because it wasn’t possible to create and manage unique infrastructure for each customer. But, some SaaS customers—especially in regulated industries like healthcare and finance—were reluctant to use multi-tenant SaaS offerings because of security and performance concerns, like data leakage between tenants or inadequate performance when a neighboring tenant was heavily using the system.

Pulumi’s Automation API unlocks new customers for existing SaaS products and entirely new classes of single-tenant SaaS products, like Pulumi customers CockroachDB and Snowflake. For these leading providers of cloud software, Automation API unlocks the capability to make infrastructure a fundamental part of their offering. With Automation API, they can create and manage unique infrastructure for every customer, even as they scale to 1,000s of customers. They can provide solutions that weren’t feasible with multi-tenant infrastructure or bring existing solutions to customers with higher needs for security, performance, and compliance. In short, they can leverage cloud infrastructure as a competitive advantage.

Start using Automation API: from `pulumi up` to `stack.up()`

To get started with Automation API, you can take any existing Pulumi program. Instead of running commands like pulumi up interactively in a terminal to provision your infrastructure, with Automation API you write an application that calls functions like stack.up(). Then, when you deploy that application and give your customers access to it, they can stand up infrastructure when they’re ready.

At its core, the Automation API enables you to use the entire Pulumi engine as an SDK for the applications you create. Like all of Pulumi, Automation API is available in multiple languages, so you can create applications that use it in TypeScript/JavaScript, Python, Go, and C#.

👉 Get started with the Automation API User Guide

Automation API in the wild

In the short while that Automation API has been in preview, we’ve already seen it used for a wide variety of custom platforms and other automation-centric use cases like CI/CD. Automation API powers custom infrastructure platforms—like Mercedes-Benz Research & Development’s platform—that empower teams across the organization to stand up the infrastructure they need in the cloud when they need it. It’s also used to operate Pulumi natively in automation and CI/CD environments; if you’ve used Pulumi’s GitHub Action, you’ve already used Automation API. Read on to see how Automation API has been used across languages, scenarios, and ecosystems.

Pulumipus’ Self-Service Platform, a web app in Python

Today at PulumiUP, Komal Ali—one of the primary authors of Automation API—will show “Pulumipus’ Self-Service Platyform”: a Python-based interface for creating customized static sites, databases, virtual machines, and more. The demo shows how easy it is to build a self-service web interface for infrastructure by combining Pulumi infrastructure as code and a Flask web app. The code is open source and available on GitHub.

Ploy, a CLI in Go

It’s also possible to create custom command-line interfaces with Automation API. Lee Briggs, a Pulumi community engineer, shows one such interface in this video, called Ploy. Ploy is a Heroku-like interface for deploying Docker images to a Kubernetes cluster. Ploy is fully open source on GitHub and combines Pulumi infrastructure with a Go CLI using Promptui.

Python IaC in your Jupyter notebook

Many data scientists work in Jupyter notebooks, where they can interactively execute Python code as they train models, create analyses, and more. Because these notebooks use Python, it’s possible to use Pulumi’s Python SDK and Automation API to create infrastructure interactively. Pulumian Sophia Parafina showed how to do so in this blog post from earlier this year. Examples of using Pulumi in a Jupyter notebook are available in our open source examples repository on GitHub.

Pulumi Resource Type for Concourse, a CI/CD extension in Go

Some users’ perfect interface to the cloud is a continuous integration and delivery (CI/CD) platform that builds applications and infrastructure together in a unified pipeline. Pulumi’s Automation API is a key component that has helped many of our customers plug Pulumi into their CI/CD platform of choice. One such extension is the Pulumi Resource Type for Concourse, which allows users of the Concourse CI product to add infrastructure to their pipelines. Authored by Pulumi community member Ringo De Smet, the extension is open source on GitHub.

Pulumi GitHub Action, a CI/CD extension in TypeScript

Pulumi works great with GitHub Actions, one of the most popular CI/CD systems, thanks to our GitHub Action, which Automation API also powers. Co-developed by Pulumi community member Simen A. W. Olsen and Pulumian Paul Stack, Pulumi’s GitHub Action gives you complete control over Pulumi in your CI/CD and other automation workflows. Read more about how it was created using Automation API or see the TypeScript source code on GitHub.

👉 See more examples of Automation API in action

Keep exploring

Automation API is just one part of the Pulumi Cloud Engineering Platform, announced today at PulumiUP. Watch the PulumiUP event for more.

👉 Get started with the Automation API User Guide 👉 PulumiUP

Data Science in the Cloud

Sophia Parafina — Thu, 18 Mar 2021 00:00:00 +0000

Data science has advanced because tools like Jupyter Notebook hide complexity by running high level code for the specific problem they are trying to solve. Increasing the level of abstraction lets a data scientist be more productive by reducing the effort to try multiple approaches to near zero, which encourages experimentation and better results.

Data scientists typically work locally, but they often store data for analyses and models in the cloud. There are clear advantages to using cloud resources for these tasks:

Data scientists generally don’t want to manage their storage and databases.
They need to be able to store large data sets cheaply.
They need large capacity swings available on-demand.

SDKs like AWS’ Python library, boto3, can create resources, but they still require domain expertise to manage and properly architect a solution. The Pulumi Automation API improves on raw SDKs by providing high-level abstractions for creating and managing cloud services, letting data scientists concentrate on analyses and models without being well-versed in cloud APIs.

How Pulumi Automation API Improves on Cloud SDKs

Pick any cloud provider, and you will find a cloud SDK in various languages, for example AWS SDKs, Azure SDKs, and Google Cloud SDKs. These SDKs directly access cloud resources, letting data scientists provision resources as required. The drawback to using these libraries is that the data scientist is also responsible for providing resources reliably and securely and performing modeling and analysis.

Pulumi reduces the burden of creating and maintaining repeatable and robust deployments by providing the latest language runtimes, concurrency management, support for diffing updates, and maintaining state with robust checkpoints. Your infrastructure will always be in a well-defined and recoverable state.

Automation API takes this one step further by providing an API overlay to Pulumi’s Infrastructure as Code engine. A data scientist can build infrastructure programmatically within Jupyter to automate building and creating infrastructure as functions and call them as needed. Let’s take a look at an example using a Jupyter notebook.

Infrastructure as a function

As a data scientist, you might want to share data, charts, or graphs with remote team members and make them available through a shared website. You want to keep iterating on your model but don’t want to update the site manually, or worse, let it get out of date.

This example is a simple overview of Automation API that demonstrates a basic deployment of a static website from within a Jupyter notebook. This example uses an inline program to define infrastructure within a function alongside your other code. The function called s3_static_site creates an s3 bucket, sets it up as a basic static website, and exports the URL. This is the equivalent of creating a Pulumi stack using the CLI.

We’ve added code to the example to upload files written to the www directory. Every time you want to update the website, you can write a file to the directory and call the function to upload it to the s3 bucket. Pulumi maintains state, so it will only upload new files when the function is called.

from pulumi_aws import s3

def s3_static_site():
 # Create a bucket and expose a website index document
 site_bucket = s3.Bucket("s3-website-bucket", website=s3.BucketWebsiteArgs(index_document="index.html"))
 index_content = """
 <html>
 <head><title>Hello S3</title><meta charset="UTF-8"></head>
 <body>
 <p>Hello, world!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
 </html>
 """

 # Write our index.html into the site bucket
 s3.BucketObject("index",
 bucket=site_bucket.id, # reference to the s3.Bucket object
 content=index_content,
 key="index.html", # set the key of the object
 content_type="text/html; charset=utf-8") # set the MIME type of the file

 # Set the access policy for the bucket so all objects are readable
 s3.BucketPolicy("bucket-policy", bucket=site_bucket.id, policy={
 "Version": "2012-10-17",
 "Statement": {
 "Effect": "Allow",
 "Principal": "*",
 "Action": ["s3:GetObject"],
 # Policy refers to bucket explicitly
 "Resource": [pulumi.Output.concat("arn:aws:s3:::", site_bucket.id, "/*")]
 },
 })

 # Export the website URL
 pulumi.export("website_url", site_bucket.website_endpoint)

We can write additional functions to deploy and destroy the S3 static website. We deploy the stack by calling the deploy_project function and pass it the s3_static_site function, the Pulumi AWS plugin, and the AWS region.

s3_site = deploy_project("my_first_project",
 s3_static_site,
 plugins=[("aws", "v3.23.0")],
 config={"aws:region": auto.ConfigValue(value="us-west-2")})

You can view the resources that the function created in two ways. The standard method is to open the AWS console to see your resources.

You can also view them with the Pulumi Service as a resource graph.

The Pulumi Service also displays updates to the static site and the changes made.

Now that the static site has been deployed, we want to get the site URL. The s3_static_site function exports the website url. Pulumi returns the s3_site as an object with outputs that we use to retrieve the URL. There are several ways to check if the site is working, but a simple way is to open the URL in a web browser.

import webbrowser

outputs = s3_site.outputs()

webbrowser.open(url)

Data Science Your Way

In this simple example, we demonstrated how to create a static website from an S3 bucket, write content into the bucket, and set the access policy as a Python function. We called the function, which created a stack for the website. The stack is returned as a Python object with the site URL as an output. You can manage the site and view updates using the Pulumi Service.

Because it’s code, you can create a separate module containing all the functions to create your infrastructure, import it into your notebooks, and reuse it. Pulumi lets you perform analysis and build models within a familiar environment while reducing the burden of managing infrastructure. You can find the complete example on GitHub including a Jupyter notebook for database migration.

Be sure to watch our video that explains the code in greater detail.

Automate Your Infrastructure with Automation API and C#

Joshua Studt — Mon, 08 Mar 2021 00:00:00 +0000

Joshua Studt is a Solutions Architect at Financial Independence Group and a Pulumi Community member who contributed the C# package for Automation API.

Currently available in public preview, Pulumi’s Automation API enables you to provision your infrastructure programmatically using the Pulumi engine. Today, we are excited to announce C# support for Automation API, enabling .NET developers to automate infrastructure deployments, create complex orchestration workflows, build custom ops tooling, and build cloud frameworks. Read more about the Automation API here.

Using Automation API in .NET

The Pulumi.Automation NuGet package exposes a LocalWorkspace for creating and managing Pulumi Stacks, and a WorkspaceStack that is a programmatic representation of a Stack for updating, refreshing, previewing, and destroying cloud resources. The Automation API makes it trivial to run Pulumi programs inline:

var program = PulumiFn.Create(() =>
{
 var bucket = new Pulumi.Aws.S3.Bucket("s3-website-bucket");

 return new Dictionary<string, object?>
 {
 ["bucket_name"] = bucket.BucketName,
 };
});

var stackArgs = new InlineProgramArgs("projectName", "stackName", program);
using var stack = await LocalWorkspace.CreateOrSelectStackAsync(stackArgs);

await stack.Workspace.InstallPluginAsync("aws", "v3.30.1");
var result = await stack.UpAsync();
var bucketName = result.Outputs["bucket_name"];

You can use your existing Pulumi projects and invoke them from the Automation API:

var stackArgs = new LocalProgramArgs("stackName", "C:\path\to\pulumi\project\dir");
using var stack = await LocalWorkspace.CreateOrSelectStackAsync(stackArgs);

await stack.UpAsync();

Since the Automation API can be invoked and debugged like any other code, it enables you to put together complex deployment workflows such as a blue-green deployment model:

using var workspace = await LocalWorkspace.CreateAsync(new LocalWorkspaceOptions
{
 Program = PulumiFn.Create<ApplicationStack>(), // use your existing Pulumi.Stack implementation
 ProjectSettings = new ProjectSettings("projectName", ProjectRuntimeName.Dotnet),
});

// your "blue" production code is already running

var green = await WorkspaceStack.SelectAsync("green", workspace);
await green.UpAsync();

// do your cutover to "green" logic
// and then teardown "blue"

var blue = await WorkspaceStack.SelectAsync("blue", workspace);
await blue.DestroyAsync();

See full Automation API examples in C#. Let us know if you’re using Automation API on Twitter or in our Community Slack.

Supercharging our GitHub Action with the Automation API

Paul Stack — Mon, 01 Mar 2021 00:00:00 +0000

Guest Article: Simen A. W. Olsen is a Software Architect and Manager at Bjerk, a software development agency based in Oslo, Norway. He joins Paul Stack to talk about the new GitHub Action powered by the Pulumi Automation API.

In this article, we’ll take a look at how our GitHub Action has been re-architected to be built on top of our Automation API.

As a GitHub Actions launch partner in 2018, we released our original GitHub Action, built as a Docker container action. This Action packaged all the prerequisites with the Pulumi CLI to let users run Pulumi commands. As the Pulumi ecosystem has grown, so too did the container run in GitHub Actions.

Since then, GitHub released an alternative way to create a GitHub Action by writing JavaScript. At Pulumi, and in our community, we love expressing everything we possibly can in code, so the new JavaScript support got us quite excited about the “better together” possibilities! Enter Pulumi’s new Automation API, a new feature we launched in public preview in October 2020. This new capability lets you build infrastructure as code—in pure code—without even needing to run a CLI. This sounded like a perfect match for a JavaScript Action. Creating a JavaScript Action allows us to programmatically invoke the right Pulumi commands in response to actions, without needing to awkwardly script a CLI. Instead, we were able to leverage the Automation API to fundamentally rethink our GitHub Action.

Under the Hood

Let’s take a look at the new JavaScript-based GitHub Action and see how it uses Automation API to execute Pulumi:

const main = async () => {
 const config = await makeConfig();

 if (environmentVariables.PULUMI_ACCESS_TOKEN !== '') {
 await pulumiCli.run('login');
 } else if (config.cloudUrl) {
 await pulumiCli.run('login', config.cloudUrl);
 }

 const workDir = resolve(
 environmentVariables.GITHUB_WORKSPACE,
 config.workDir,
 );

 const stack = await LocalWorkspace.selectStack({
 stackName: config.stackName,
 workDir: workDir,
 });

 const onOutput = (msg: string) => {
 .....
 };

 const actions: Record<Commands, () => Promise<string>> = {
 up: () => stack.up({ onOutput }).then((r) => r.stdout),
 refresh: () => stack.refresh({ onOutput }).then((r) => r.stdout),
 destroy: () => stack.destroy({ onOutput }).then((r) => r.stdout),
 preview: async () => {
 const { stdout, stderr } = await stack.preview();
 onOutput(stdout);
 onOutput(stderr);
 return stdout;
 },
 };

 const output = await actions[config.command]();

 const outputs = await stack.outputs();

 for (const [outKey, outExport] of Object.entries(outputs)) {
 core.setOutput(outKey, outExport.value);
 if (outExport.secret) {
 core.setSecret(outExport.value);
 }
 }
};

The Action code is clean and concise because the power lies in the Automation API itself. The Action uses the Automation API to drive Pulumi as part of your GitHub Actions workflows. This means that your workflows can take advantage of changes to Pulumi and the Automation API as soon as they’re released! And, of course, you can always write your own custom logic using the Automation API like our Action does.

Customizing Your Environment

The new Action puts you in control of the environment that it runs on. By comparison, the previous Docker-based Action bundled specific tool versions, which left your CI pipeline at the mercy of our opinionated container. Now, if you want to pin to specific versions of NodeJS / Python / Go / .NET or even Pulumi, you have that power!

You can even include a matrix strategy to test your Pulumi code against different language runtime versions and different operating systems:

name: 'Sample Pulumi NodeJS Application'
on:
 pull_request:

jobs:
 test-workflow:
 runs-on: ${{ matrix.os }}
 name: Testing Pulumi Preview on ${{ matrix.os }} with NodeJS ${{ matrix.node_version }}
 strategy:
 matrix:
 os: [ubuntu-latest, macos-latest, windows-latest]
 node_version: [10.x, 12.x, 14.x]
 fail-fast: false
 steps:
 - uses: actions/checkout@v2
 - uses: actions/setup-node@v1
 with:
 node-version: ${{ matrix.node_version }}
 - name: Install pulumi
 uses: pulumi/action-install-pulumi-cli@v1.0.1
 - run: npm install
 - uses: pulumi/actions@v2
 env:
 PULUMI_CONFIG_PASSPHRASE: not-a-secret
 with:
 command: preview
 cloud-url: file://~
 stack-name: dev

Cloud Interaction

The previous Docker-based Action would look for the GOOGLE_CREDENTIALS environment variable; if your workflow set that variable, the Action would run the following commands on your behalf:

gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud --quiet auth configure-docker $GOOGLE_DOCKER_HOSTNAME_LIST

With the new Action, you’re fully in control of authentication to cloud providers. To authenticate with AWS / Azure / Google Cloud, you can now use their official Actions:

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1
 with:
 aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
 aws-region: ${{ secrets.AWS_REGION }}
 aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

 - uses: azure/login@v1
 with:
 creds: ${{ secrets.AZURE_CREDENTIALS }}

 - uses: google-github-actions/setup-gcloud@v0
 with:
 service_account_key: ${{ secrets.GCP_KEY }}
 project_id: ${{ env.PROJECT_ID }}
 export_default_credentials: true

We recommend that you store all sensitive environment variables, like cloud credentials, in the Actions secret storage and consume them using the secrets attribute on your workflow’s action.

Faster

The new GitHub Action is much smaller and no longer needs to download a (very!) large Docker container. In the Docker-based Action, we frequently saw workflows take up to 3 minutes to download the container. The new Action is much faster; let’s take a look:

Old Docker-based Action (v1)

New JavaScript-based Action, powered by Automation API (v2)

The new Pulumi GitHub Action is ready for you to use. Here’s how to get started:

Already use the v1 Pulumi GitHub Action? Migrate to the new one with our migration guide.
New to Pulumi and GitHub Actions? Use our getting started guide to set up your first Pulumi workflow.

We’re eager to hear your feedback on this new Action. If you haven’t already signed up for our Community Slack, it’s quick and easy! You can join in on conversations you like and get help from other community members, as well as the Pulumi Team. That’s it for now!

Observability with Infrastructure as Code

Andy Davies — Fri, 26 Feb 2021 00:00:00 +0000

Guest Article: Andy Davies is a Senior Software Developer at Reaktor, a tech company based in Amsterdam, New York, and Finland, writing about using the Pulumi Automation API to add observability to infrastructure provisioning.

When using the Pulumi Automation API to create applications that provisions infrastructure, it is very handy to use observability techniques to ensure the application functions correctly and to find where performance bottlenecks are.

One of the applications I work on creates a VPC and Bastion host and then stores the credentials in a Vault instance. The problem is that the ‘create infrastructure’ part is an opaque blob, in that I can see it takes 129 seconds to create, but I can’t see what it’s doing or why it takes this amount of time.

So can I do better?

The Initial Application

In this example, I use Honeycomb’s Go Beeline to capture all the data I care about; durations, errors, and any context which is ‘interesting’:

func main() {

 beeline.Init(beeline.Config{
 WriteKey: os.Getenv("HONEYCOMB_API_KEY"),
 Dataset: "pulumi-demo",
 })
 defer beeline.Close()

 ctx, span := beeline.StartSpan(context.Background(), "basic-vpc")
 defer span.Send()

 name := auto.FullyQualifiedStackName(os.Getenv("PULUMI_USERNAME"), "basic-vpc", "dev")
 stack, err := auto.UpsertStackInlineSource(ctx, name, "basic-vpc", func(pc *pulumi.Context) error {

 azs, err := getAvailabilityZones(ctx)
 if err != nil {
 beeline.AddField(ctx, "err", err)
 return err
 }

 v, err := vpc.NewVpc(ctx, pc, "dev", &vpc.VpcArgs{
 Description: "dev",
 BaseCidr: "192.168.0.0/16",
 AvailabilityZoneNames: azs,
 S3Endpoint: true,
 DynamoEndpoint: true,
 })
 if err != nil {
 beeline.AddField(ctx, "err", err)
 return err
 }
 })

 if err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 if err := stack.SetConfig(ctx, "aws:region", auto.ConfigValue{Value: os.Getenv("PULUMI_REGION")}); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 ws := stack.Workspace()
 if err := ws.InstallPlugin(ctx, "aws", "v3.23.0"); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 if _, err := stack.Refresh(ctx); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 stream := optup.ProgressStreams(os.Stdout)
 if _, err := stack.Up(ctx, stream); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
 }

 //vault code

}

Adding Infrastructure Observability

To get a handle on what happens when stack.Up() runs, I have implemented a custom io.Writer, which is passed into the ProgressStream constructor.

The custom progress stream’s Write method is called once for each line emitted, which allows us to start new spans when a resource starts being constructed, and send them when construction completes. This is currently achieved by parsing the console output text, but I gather in the future, it will be possible to get streamed JSON blobs that can be unmarshaled into go structs.

type pulumiBeeline struct {
 ctx context.Context
 contexts map[string]func()
}

func NewPulumiBeeline(ctx context.Context) *pulumiBeeline {
 return &pulumiBeeline{
 ctx: ctx,
 contexts: map[string]func(){},
 }
}

func (cw *pulumiBeeline) Write(p []byte) (n int, err error) {

 // todo: make more robust, support modifications, deletions etc.
 line := strings.TrimSpace(string(p))
 parts := strings.Split(line, " ")
 if len(parts) < 5 {
 return len(p), nil
 }

 //+ aws-vpc dev creating
 //+ <type> <name> <action>
 resourceType := parts[2]
 resourceName := parts[3]
 resourceAction := parts[4]

 if resourceAction == "creating" {
 c, s := beeline.StartSpan(cw.ctx, resourceName)
 beeline.AddField(c, "type", resourceType)
 // add other things here

 cw.contexts[resourceName] = s.Send
 }

 if resourceAction == "created" {
 cw.contexts[resourceName]()
 }

 return len(p), nil
}

Modifying the optup.ProgressStreams is the only change needed to the original application:

stream := optup.ProgressStreams(os.Stdout, NewPulumiBeeline(ctx))
if _, err := stack.Up(ctx, stream); err != nil {
 beeline.AddField(ctx, "err", err)
 os.Exit(1)
}

When I rerun this program, I can see a lot more information in my Honeycomb traces, which shows me that Pulumi is highly parallelized and provides a better idea of where the time is spent when creating infrastructure. In this example, it’s the NAT Gateways:

In the future, I want to expand this to cover far more details. Among those details are why resources were created/modified/destroyed and collecting more information on what causes resources to fail.

Wrapping Up

In the end, this turned out to be much easier to achieve than I had hoped. Using Pulumi programmatically was a huge productivity boost, instead of running os.Exec directly.

I am looking forward to all the new kinds of tooling I can build to solve my user’s problems by continuing to utilize Honeycomb for my observability and Pulumi for my infrastructure.

Automate Your Infrastructure with Automation API and Python

Sophia Parafina — Tue, 09 Feb 2021 00:00:00 +0000

General-purpose languages enable Infrastructure as Software – bringing tested toolchains and best practices to building infrastructure, e.g., languages, IDEs, testing, debugging, componentization, packaging, and versioning. Available in public preview, Pulumi’s Automation API is a robust programmatic layer on top of Pulumi’s infrastructure engine. It exposes Pulumi programs and stacks as strongly-typed and composable building blocks. Automation API allows you to embed the Pulumi engine inside your software projects so you can build software automation around entire infrastructure provisioning processes that normally require humans to operate.

Today, we are excited to announce Python support for this powerful feature, opening up a world of possibilities for Python developers.

Enabling Cloud Engineering

The Automation API is a subpackage in Pulumi’s language-specific SDKs that provides a programmable interface for creating and managing Stacks and performing infrastructure updates, refresh, previews, and destroy. You can define a Pulumi program as a function within your codebase and use methods to get and set configuration parameters programmatically. The Automation API uses a gRPC interface to execute programs that control and communicate with the core Pulumi engine.

To use Automation API, install the Pulumi CLI, which bundles and distributes the core engine. Today it’s available for Python, TypeScript/JavaScript and Go, with support for C# under active development.

Automation Use Cases

Let’s look at two use cases where Pulumi customers are applying the Automation API:

Platform APIs

Deploying infrastructure with shell scripts for online services creates operational challenges with observability and reliability, especially for SaaS scenarios. Automation API is just another package and can be instrumented, profiled, and monitored with your existing APM tools like Honeycomb, Prometheus, and Datadog.

Automation API runs inside your favorite frameworks and interacts with your other packages precisely as you’d expect. We can define custom declarative infrastructure and expose it to end-users behind a familiar REST interface, as in this example, that creates an S3 website with a POST request.

# an HTTP handler to create instances of our declarative infra
# each request creates an S3-backed static website with dynamic content from the POST body
@app.route("/sites", methods=["POST"])
def create_handler():
 """creates new sites"""
 stack_name = request.json.get('id')
 content = request.json.get('content')
 try:
 def pulumi_program():
 return create_pulumi_program(content)
 # create a new stack, generating our pulumi program on the fly from the POST body
 stack = auto.create_stack(stack_name=stack_name,
 project_name=project_name,
 program=pulumi_program)
 stack.set_config("aws:region", auto.ConfigValue("us-west-2"))
 # deploy the stack, tailing the logs to stdout
 up_res = stack.up(on_output=print)
 return jsonify(id=stack_name, url=up_res.outputs['website_url'].value)
 except auto.StackAlreadyExistsError:
 return make_response(f"stack '{stack_name}' already exists", 409)
 except Exception as exn:
 return make_response(str(exn), 500)

The handlers work with your favorite HTTP frameworks, including flask:

import Flask

# install necessary plugins once upon boot
ensure_plugins()

# initialize our Flask app
app = Flask(__name__)

# set up our routes
@app.route("/sites", methods=["POST"])
def create_handler():
 ...

@app.route("/sites", methods=["GET"])
def list_handler():
 ...

@app.route("/sites/<string:id>", methods=["GET"])
def get_handler(id: str):
 ...

@app.route("/sites/<string:id>", methods=["UPDATE"])
def update_handler(id: str):
 ...

@app.route("/sites/<string:id>", methods=["DELETE"])
def delete_handler(id: str):
 ...

Check out the full Pulumi over HTTP example in Python.

Complex Workflow Orchestration

The Automation API executes just like any other code in your application, meaning that a single process can deploy infrastructure and manage related activities such as database migrations and blue/green deployments.

In this example, we define an AWS RDS database:

cluster_instance = aws.rds.ClusterInstance(
 "db_instance",
 cluster_identifier=cluster.cluster_identifier,
 instance_class=aws.rds.InstanceType.T3_SMALL,
 engine=aws.rds.EngineType.AURORA_MYSQL,
 engine_version="5.7.mysql_aurora.2.03.2",
 publicly_accessible=True,
 db_subnet_group_name=subnet_group.name
)

pulumi.export("host", cluster.endpoint)
pulumi.export("db_name", db_name)
pulumi.export("db_user", db_user)
pulumi.export("db_pass", db_pass)

With Automation API, we can perform database migrations and preload rows in the same program:

# create (or select if one already exists) a stack that uses our inline program
stack = auto.create_or_select_stack(stack_name=stack_name,
 project_name=project_name,
 program=pulumi_program)

up_res = stack.up(on_output=print)
print(f"db host url: {up_res.outputs['host'].value}")

print("configuring db...")
with connect(
 host=up_res.outputs['host'].value,
 user=up_res.outputs['db_user'].value,
 password=up_res.outputs['db_pass'].value,
 database=up_res.outputs['db_name'].value) as connection:
 print("db configured!")

 # make sure the table exists
 print("creating table...")
 create_table_query = """CREATE TABLE IF NOT EXISTS hello_pulumi(
 id int(9) NOT NULL PRIMARY KEY,
 color varchar(14) NOT NULL);
 """
 with connection.cursor() as cursor:
 cursor.execute(create_table_query)
 connection.commit()

See the full database deployment and migration Automation API program in Python.

Automation API Resources

To get started with Automation API check out these links and watch our video.

Automation API is still in alpha. We maintain a list of known issues. Please file more as you find them!

🏭 🏭 🏭 Happy automating! 🏭 🏭 🏭

Automation API: Supercharged Cloud Tooling

Sophia Parafina — Thu, 14 Jan 2021 00:00:00 +0000

“Why use a programming language to build and maintain infrastructure?” is a question we hear frequently. There are apparent advantages such as using a mature and well-known language across a team, enabling cloud engineers to use software development best practices, and an ecosystem of tools for building robust systems.

Infrastructure as code enables you to build tools and environments to automate routine tasks, letting cloud engineers concentrate on efficiency and resilience. In this article, we’ll take a look at how Pulumi’s Automation API lets you build custom ops tooling that improves your workflow.

What is the Automation API?

Automation API is a programming interface overlaid on the Pulumi engine for deploying infrastructure. Instead of using the Pulumi CLI to deploy infrastructure, you can directly call the engine to run code from your chosen language. With the Automation API, your Pulumi programs and stacks are strongly-typed and composable infrastructure building blocks. You can fully embed Pulumi inside your software projects, which means your code can power a wide range of cloud automation projects without the Pulumi CLI and a reduced human-in-the-loop footprint.

This alpha release of Automation API currently supports Node.js, Go and Python. .NET (C#) will be released soon.

Cleaning Up

To demonstrate how Automation API enables you to build tools, let’s tackle the common problem of unused virtual machines in the cloud. Companies spent an estimated $11 billion on idle resources in 2020, according to ParkMyCloud. Merely shutting down unused resources can significantly decrease costs for any organization.

We can use the Automation API to build a specialized CLI tool for creating and managing virtual machines. We wrote vmgr to demonstrate provisioning temporary virtual machines that expire after a user-specified amount of time. The complete example is available on Github. The vmgrCLI is a tool for provisioning temporary development virtual machines in Azure. Let’s take a look at this project.

An advantage of using a programming language is that you can use common software packages in your code. In the vmgr example, we use Cobra, a library for CLI applications used in Go projects Kubernetes, Hugo, and Github CLI. Instead of writing CLI boilerplate code with arguments, we can use Cobra with Automation API. VMs are provisioned as Stacks and their state is held by the Pulumi service. The cleanup job queries the Pulumi Service for stacks older than specified and removes them along with any child resources.

‘vmgr’ takes two commands:

vmgr add: creates a new VM web server with a public IP. These web servers share a resource group and virtual network that is referenced from a shared stack.
vmgr cron : scans all VM stacks for stacks that are older than the specified expiration. This command will continue to retry deletion even if it sees failures and will typically succeed in deleting the VM and public IP on the second try.

To delete VMs older than 5 minutes:

$ vmgr cron 5m

To delete VMs older than five days.

$vmgr cron 5d

vmgr is a tool you can give to developers that they can use immediately with the added benefit of reducing costs by removing unused instances. Automation API enables building tools that speed up the inner dev loop for application developers and reduces the operation team’s burden to clean up unused instances.

Automate All the Things

Cloud engineering will automate many tasks that are currently performed by operators. Using infrastructure as code and SDKs such as Automation API, we can build self-service tools that improve the inner dev loop for application developers and let operators perform higher-level tasks such as maintenance and infrastructure monitoring. This simple example demonstrates the effective use of infrastructure as code by importing other software libraries to build a specialized tool that is not possible with DSLs or YAML.

Automation API is currently in preview and will be part of the Pulumi 3.0 GA release, including additional language support. To get started with Automation API, try out the examples and documentation for node.js and go. If you are looking for a more complex example, check out ploy, a CLI tool that builds Docker images and deploys them on Kubernetes.

Orchestrating Cloud Workflows with Automation API

Sophia Parafina — Thu, 19 Nov 2020 00:00:00 +0000

There are many moving parts when deploying infrastructure and applications. Playbooks are step-by-step maps that standardize how infrastructure and applications are deployed across your organization. Typically playbooks describe every action to build and deploy, requiring an operator to complete each step before moving on to the next. It’s a process that can be tedious and prone to human error.

What if you could encapsulate a playbook into a single action? This is the promise of declarative infrastructure. You declare the desired state of your infrastructure and the infrastructure as code engine builds the infrastructure. However, you must still deploy the application and perform maintenance, and this is where you hit the limits of templating languages and where programming languages excel. In this hands-on article, we’ll demonstrate how to use Pulumi’s Automation API to create a program that builds infrastructure, installs an application, and can perform application maintenance.

Starting Point

Pulumi’s Automation API has a set of examples to get you started. These examples are sketches that demonstrate what is possible with Automation API. We’ll start with the database migration example in Node.js, described as:

“This example provisions an AWS Aurora SQL database and executes a database “migration” using the resulting connection info. This migration creates a table, inserts a few rows of data, and reads the data to verify the setup. This is all done in a single inline Pulumi program.”

We’ll expand on this example and add the following:

show how to retrieve a password from AWS Secrets Manager
add functions for database backup, mysqlcheck, optimize, and analyze

Let’s get started by cloning the repository and open the index.ts.

$ git clone https://github.com/pulumi/automation-api-examples.git
$ cd ./automation-api-examples/nodejs/databaseMigration-ts/

Program overview

Several pieces comprise the program; the first is the program’s main scope, where we can pass values from the functions that make up the script. The next part is the pulumiProgram function that declares the infrastructure.

const run = async () => {
 // This is our pulumi program in "inline function" form
 const pulumiProgram = async () => {

To deploy the infrastructure, we use the Automation API to instantiate the resources and create a stack. Like a Pulumi program invoked by the CLI, we need to provide a stack and project name and the inline program’s name that declares the infrastructure.

 // Create our stack
 const args: InlineProgramArgs = {
 stackName: "dev",
 projectName: "databaseMigration",
 program: pulumiProgram
 };

 // create (or select if one already exists) a stack that uses our inline program
 const stack = await LocalWorkspace.createOrSelectStack(args);

Once the infrastructure is created, we can call provisioning functions, such as createdb, maintenance functions, or other functions to manage the database.

Adding arguments for new commands

First, we’ll add the arguments for database creation, mysqlcheck, optimize, and analyze. These arguments call the functions that build the infrastructure, create and populate the database, and perform maintenance.

const process = require('process');

const args = process.argv.slice(2);
let destroy = false;
let createdb =false;
let backup = false;
let mysqlcheck = false;
let optimize = false;
let analyze = false;

if (args.length > 0 && args[0]) {
 destroy = args[0] === "destroy";
 createdb = args[0] === "createdb";
 backup = args[0] === "backup";
 mysqlcheck = args[0] === "mysqlcheck";
 optimize = args[0] === "optimize";
 analyze = args[0] === "analyze";
}

Declare infrastructure

The all the code used to create the infrastructure (a VPC, a security group, and an RDS, provision the database, and maintain the database is in an inline program, which is a function that we can call using Node.js instead of the Pulumi CLI.

Let’s break down the infrastructure piece by piece, starting with setting up configuration variables for the database. In the original code, the database name, admin name, and admin password were hardcoded into the program. We would never do that in practice and get the database, admin name, and secret name from a configuration file and a secrets provider’s password. We’ll keep those properties in the program for simplicity, but in our improved program, we’ll retrieve the password from the AWS Secrets Manager. We set the password using the aws CLI.

$ aws secretsmanager create-secret --name dbpass --secret-string hellosql

We retrieve the secret with a function that retrieves it as a Pulumi output and returns it as a string type, which is the type that Aurora MySQL requires for the masterpassword property. This is a little complicated because we use the secret name to retrieve the secret properties with getSecret, which returns the secret ARN. Then, we use the ARN to retrieve the password with getSecretVersion. Note that the function is wrapped in async/await because we will need the password before creating the database.

async function getSecret(name: string) {
 const secret = await aws.secretsmanager.getSecret({name: name});
 const sm = aws.secretsmanager.getSecretVersion( {secretId: secret.arn});
 return (await sm).secretString;
 }

The next part creates the VPC, the subnet, a security group for access, and the Aurora database cluster. Aside from calling the getSecret function, this part of the program remains unchanged. Note that the function returns several parameters that are available to other functions in the same scope.

 // setup infrastructure
 const vpc = awsx.ec2.Vpc.getDefault();
 const subnetGroup = new aws.rds.SubnetGroup("dbsubnet", {
 subnetIds: vpc.publicSubnetIds,
 });

 // make a public SG for our cluster for the migration
 const securityGroup = new awsx.ec2.SecurityGroup("publicGroup", {
 egress: [
 {
 protocol: "-1",
 fromPort: 0,
 toPort: 0,
 cidrBlocks: ["0.0.0.0/0"],
 }
 ],
 ingress: [
 {
 protocol: "-1",
 fromPort: 0,
 toPort: 0,
 cidrBlocks: ["0.0.0.0/0"],
 }
 ]
 });

 // provision our db
 const cluster = new aws.rds.Cluster("db", {
 engine: "aurora-mysql",
 engineVersion: "5.7.mysql_aurora.2.03.2",
 databaseName: dbUser,
 masterUsername: dbName,
 masterPassword: getSecret("dbpass"),
 skipFinalSnapshot: true,
 dbSubnetGroupName: subnetGroup.name,
 vpcSecurityGroupIds: [securityGroup.id],
 });

 const clusterInstance = new aws.rds.ClusterInstance("dbInstance", {
 clusterIdentifier: cluster.clusterIdentifier,
 instanceClass: "db.t3.small",
 engine: "aurora-mysql",
 engineVersion: "5.7.mysql_aurora.2.03.2",
 publiclyAccessible: true,
 dbSubnetGroupName: subnetGroup.name,
 });

 return {
 host: pulumi.interpolate`${cluster.endpoint}`,
 dbName,
 dbUser,
 dbPass: getSecret("dbpass"),
 };
 };

Creating the stack

If you’re not familiar with the Pulumi programming model, a stack is an instantiation of the infrastructure resources declared in the code. We start with calling the createOrSelectStack method from the LocalWorkSpace import. This method takes three arguments: a stack name, a project name, and the program to call, which is the pulumiProgram function, that creates the infrastructure. The stack installs the required plugins, sets the AWS region, and creates the infrastructure. We can also destroy the stack if we use the destroy argument, e.g., yarn start destroy. .

 // Create our stack
 const args: InlineProgramArgs = {
 stackName: "dev",
 projectName: "databaseMigration",
 program: pulumiProgram
 };

 // create (or select if one already exists) a stack that uses our inline program
 const stack = await LocalWorkspace.createOrSelectStack(args);

 console.info("successfully initialized stack");
 console.info("installing plugins...");
 await stack.workspace.installPlugin("aws", "v3.6.1");
 console.info("plugins installed");
 console.info("setting up config");
 await stack.setConfig("aws:region", { value: "us-west-2" });
 console.info("config set");
 console.info("refreshing stack...");
 await stack.refresh({ onOutput: console.info });
 console.info("refresh complete");

 if (destroy) {
 console.info("destroying stack...");
 await stack.destroy({ onOutput: console.info });
 console.info("stack destroy complete");
 process.exit(0);
 }

 console.info("updating stack...");
 const upRes = await stack.up({ onOutput: console.info });
 console.log(`update summary: \n${JSON.stringify(upRes.summary.resourceChanges, null, 4)}`);
 console.log(`db host url: ${upRes.outputs.host.value}`);
 console.info("configuring db...");

Adding a table and data

Once our infrastructure is up, we can add a table and insert data with SQL statements. First, we need to create a connection object to talk to the database. The upRes variable is set to the stack, so the outputs it returns are available to the program’s main scope.

 // establish mysql client
 const connection = mysql.createConnection({
 host: upRes.outputs.host.value,
 user: upRes.outputs.dbUser.value,
 password: upRes.outputs.dbPass.value,
 database: upRes.outputs.dbName.value
 });

In the original program, the table and data are created as part of the inline program. We wrap this in a function that is called by yarn, e.g., yarn start createdb.

 // create table and populate it
 if (createdb) {
 connection.connect();
 console.log("creating table...")

 // make sure the table exists
 connection.query(`
 CREATE TABLE IF NOT EXISTS hello_pulumi(
 id int(9) NOT NULL,
 color varchar(14) NOT NULL,
 PRIMARY KEY(id)
 );
 `, function (error, results, fields) {
 if (error) throw error;
 console.log("table created!")
 console.log('Result: ', JSON.stringify(results));
 console.log("seeding initial data...")
 });

 // seed the table with some data to start
 connection.query(`
 INSERT IGNORE INTO hello_pulumi (id, color)
 VALUES
 (1, 'Purple'),
 (2, 'Violet'),
 (3, 'Plum');
 `, function (error, results, fields) {
 if (error) throw error;
 console.log("rows inserted!")
 console.log('Result: ', JSON.stringify(results));
 console.log("querying to veryify data...")
 });


 // read the data back
 connection.query(`SELECT COUNT(*) FROM hello_pulumi;`, function (error, results, fields) {
 if (error) throw error;
 console.log('Result: ', JSON.stringify(results));
 console.log("database, tables, and rows successfuly configured!")
 });

 connection.end();
 }

Adding maintenance features

The createdb function establishes the pattern for adding other MySQL operations. The pattern is to connect to the database, send a SQL query, and return the results and errors if any. These functions take the table name as input. To use these commands, we can run the program with yarn, as before:

$ yarn start <command> hello_pulumi

The program parses the arguments and sets the command and table name in the main scope. The additional MySQL operations are:

checktable: CHECK TABLE,
analyze: ANALYZE TABLE
optimize: OPTIMIZE TABLE.

We created the hello_pulumi table with the createdb function, but we can leave it as a future exercise to make a generic table creation and list table functions.

 // MySQL command to check tables for integrity errors
 if (checktable) {
 connection.connect();
 console.log("checking table...")
 if (tableName) {
 connection.query(`CHECK TABLE ${tableName};`, function (error, results, fields) {
 if (error) throw error;
 console.log('Result: ', JSON.stringify(results));
 console.log("table checked for integrity errors!")});
 } else {
 console.log("Error: Table not specified.");
 console.log("Try: yarn start checktable {table name}");
 }

 connection.end();
 }

 // MySQL command to rebuild index for slow database response
 if (analyze) {
 connection.connect();
 console.log(" analyze table...")
 if (tableName) {
 connection.query(`ANALYZE TABLE ${tableName};`, function (error, results, fields) {
 if (error) throw error;
 console.log('Result: ', JSON.stringify(results));
 console.log("table index rebuilt!")});
 } else {
 console.log("Error: Table not specified.");
 console.log("Try: yarn start analyze {table name}");
 }

 connection.end();
 }

 // MySQL command to reclaim space and rebuild tables and indices
 if (optimize) {
 connection.connect();
 console.log(" optimize table...")
 if (tableName) {
 connection.query(`OPTIMIZE TABLE ${tableName};`, function (error, results, fields) {
 if (error) throw error;
 console.log('Result: ', JSON.stringify(results));
 console.log("table space reclaimed, table and index rebuilt!")});
 } else {
 console.log("Error: Table not specified.");
 console.log("Try: yarn start analyze {table name}");
 }

 connection.end();
 }

Summary

So what have we accomplished? Most importantly, we can turn a playbook into code. Encapsulating your workflow with code means that you have modern software tools such as an IDE with code completion, hints, type checking, debugging, and versioning available to you. In addition to declaring infrastructure, you can also provision it with applications and data. Because it’s code, you can extend it as we have done by adding maintenance functions, something not possible with a markup language. Finally, you don’t have to use a proprietary CLI to deploy infrastructure and apps. Automation API lets you do all these things and much more.

You can download our improved program as a gist to replace the one included in the example.

Automation API is currently in alpha and available in Typescript and Go. Check out these resources to learn more.

Build Self-Service Cloud Infrastructure with Automation API

Sophia Parafina — Tue, 10 Nov 2020 00:00:00 +0000

If you could create infrastructure without using a cloud provider’s console, a CLI, or a templating engine, what would you build? Pulumi’s Automation API lets you create declarative infrastructure defined by your best practices and expose it behind a REST, gRPC, or custom API.

So just what is Automation API? Think of it as Pulumi’s infrastructure as code engine as an SDK. Instead of writing code and using the CLI to declare infrastructure, you can directly tell the engine to build your infrastructure. This means that you’re using the same declarative IaC tooling with the predictability, robustness, safety, and desired state management, except it has a new programmatic surface area. Imagine building an application that creates infrastructure via a REST interface. Get ready, because that’s what we’re going to do.

Self-service Cloud Infrastructure

Automation API is just another package that runs inside your favorite frameworks and works with other packages. You can create methods or functions to create infrastructure that you can call from other frameworks. One Automation API use case is a self-service infrastructure. You can create declarative infrastructure and expose it behind a REST interface. Your infrastructure is also backed by a JSON state file managed by the Pulumi SaaS that allows you to create and update them dynamically.

This example demonstrates how to create infrastructure with Automation API and the Express Node.js framework. To keep this example simple, we’ll deploy the REST API locally. To deploy the code in the cloud, for example, in a virtual machine, we’ll need to create an instance profile to store AWS credentials. Deploying Automation API is beyond this article’s scope, so we’ll use localhost as the server’s endpoint. We’ll cover deploying Automation API in the cloud in subsequent articles.

Infrastructure as a function

We use an inline Pulumi program that doesn’t require a separate package with index.ts and Pulumi.yaml files, although they can be authored in an index.ts file or imported from another package. The example deploys an AWS S3 website with the context and deployment automation defined in a function. Functions can be components, in this case, a static S3 website. However, they can be more complex, such as a Kubernetes cluster with accompanying containers and AWS resources.

const projectName = "pulumi_over_http";

const createPulumiProgram = (content: string) => async () => {
 // Create a bucket and expose a website index document
 const siteBucket = new s3.Bucket("s3-website-bucket", {
 website: {
 indexDocument: "index.html",
 },
 });

This part of the function lets users write content.

 // Our HTML is defined based on what the caller sends.
 const indexContent = content;

 // write our index.html into the site bucket
 let object = new s3.BucketObject("index", {
 bucket: siteBucket,
 content: indexContent,
 contentType: "text/html; charset=utf-8",
 key: "index.html"
 });

To make the contents accessible, we create and set a policy for the S3 bucket.

 // Create an S3 Bucket Policy to allow public read of all objects in bucket
 function publicReadPolicyForBucket(bucketName): PolicyDocument {
 return {
 Version: "2012-10-17",
 Statement: [{
 Effect: "Allow",
 Principal: "*",
 Action: [
 "s3:GetObject"
 ],
 Resource: [
 `arn:aws:s3:::${bucketName}/*` // policy refers to bucket name explicitly
 ]
 }]
 };
 }

 // Set the access policy for the bucket so all objects are readable
 let bucketPolicy = new s3.BucketPolicy("bucketPolicy", {
 bucket: siteBucket.bucket, // refer to the bucket created earlier
 policy: siteBucket.bucket.apply(publicReadPolicyForBucket) // use output property `siteBucket.bucket`
 });

 return {
 websiteUrl: siteBucket.websiteEndpoint,
 };
};

Building the REST API

We use the Express RequestHandler callback to create the S3 static website buckets by creating the stack locally. Calling this handler creates a new bucket which represents an individual site. It’s important to note that one of the advantages of using code is the ability to detect and handle error scenarios, which is necessary for online infrastructure or when embedding it in other complex software.

// creates new sites
const createHandler: express.RequestHandler = async (req, res) => {
 const stackName = req.body.id;
 const content = req.body.content as string;
 try {
 // create a new stack
 const stack = await LocalWorkspace.createStack({
 stackName,
 projectName,
 // generate our pulumi program on the fly from the POST body
 program: createPulumiProgram(content),
 });
 await stack.setConfig("aws:region", { value: "us-west-2" });
 // deploy the stack, tailing the logs to console
 const upRes = await stack.up({ onOutput: console.info });
 res.json({ id: stackName, url: upRes.outputs.websiteUrl.value });
 } catch (e) {
 if (e instanceof StackAlreadyExistsError) {
 res.status(409).send(`stack "${stackName}" already exists`);
 } else {
 res.status(500).send(e);
 }
 }
};

These callbacks implement the range of REST API requests for a bucket which includes adding content with a POST request, updating content with a PUT request, getting site content and a list of sites with GET requests, and deleting a site/bucket with a DELETE request.

// lists all sites
const listHandler: express.RequestHandler = async (req, res) => {
 try {
 // set up a workspace with only enough information for the list stack operations
 const ws = await LocalWorkspace.create({ projectSettings: { name: projectName, runtime: "nodejs" } });
 const stacks = await ws.listStacks();
 res.json({ ids: stacks.map(s => s.name) });
 } catch (e) {
 res.status(500).send(e);
 }
};
// gets info about a specific site
const getHandler: express.RequestHandler = async (req, res) => {
 const stackName = req.params.id;
 try {
 // select the existing stack
 const stack = await LocalWorkspace.selectStack({
 stackName,
 projectName,
 // don't need a program just to get outputs
 program: async () => { },
 });
 const outs = await stack.outputs();
 res.json({ id: stackName, url: outs.websiteUrl.value });
 } catch (e) {
 if (e instanceof StackNotFoundError) {
 res.status(404).send(`stack "${stackName}" does not exist`);
 } else {
 res.status(500).send(e);
 }
 }
};
// updates the content for an existing site
const updateHandler: express.RequestHandler = async (req, res) => {
 const stackName = req.params.id;
 const content = req.body.content as string;
 try {
 // select the existing stack
 const stack = await LocalWorkspace.selectStack({
 stackName,
 projectName,
 // generate our pulumi program on the fly from the POST body
 program: createPulumiProgram(content),
 });
 await stack.setConfig("aws:region", { value: "us-west-2" });
 // deploy the stack, tailing the logs to console
 const upRes = await stack.up({ onOutput: console.info });
 res.json({ id: stackName, url: upRes.outputs.websiteUrl.value });
 } catch (e) {
 if (e instanceof StackNotFoundError) {
 res.status(404).send(`stack "${stackName}" does not exist`);
 } else if (e instanceof ConcurrentUpdateError) {
 res.status(409).send(`stack "${stackName}" already has update in progress`)
 } else {
 res.status(500).send(e);
 }
 }
};
// deletes a site
const deleteHandler: express.RequestHandler = async (req, res) => {
 const stackName = req.params.id;
 try {
 // select the existing stack
 const stack = await LocalWorkspace.selectStack({
 stackName,
 projectName,
 // don't need a program for destroy
 program: async () => { },
 });
 // deploy the stack, tailing the logs to console
 await stack.destroy({ onOutput: console.info });
 await stack.workspace.removeStack(stackName);
 res.status(200).end();
 } catch (e) {
 if (e instanceof StackNotFoundError) {
 res.status(404).send(`stack "${stackName}" does not exist`);
 } else if (e instanceof ConcurrentUpdateError) {
 res.status(409).send(`stack "${stackName}" already has update in progress`)
 } else {
 res.status(500).send(e);
 }
 }
};
const ensurePlugins = async () => {
 const ws = await LocalWorkspace.create({});
 await ws.installPlugin("aws", "v3.2.1");
};

// install necessary plugins once upon boot
ensurePlugins();

// configure express
const app = express();
app.use(express.json());

Creating routes in Express to the callbacks is straightforward and requires mapping the route to the appropriate callback.

// setup our RESTful routes for our Site resource
app.post("/sites", createHandler);
app.get("/sites", listHandler);
app.get("/sites/:id", getHandler);
app.put("/sites/:id", updateHandler);
app.delete("/sites/:id", deleteHandler);

// start our http server
app.listen(1337, () => console.info("server running on :1337"));

Trying it out

The complete example is available on GitHub, along with other examples in both Nodejs and go. To run the TypeScript example, run:

$ yarn install
$ yarn start

Let’s test it by sending requests to the server on http://localhost:1337.

If you use Postman, you can import a Postman Collection to send requests to the server.

First let’s create our first website which returns the website path (id) and the URL of the static website.

$ curl --header "Content-Type: application/json" --request POST --data '{"id":"hello","content":"hello world\n"}' http://localhost:1337/sites
$ {"id":"hello","url":"s3-website-bucket-db5dae8.s3-website-us-west-2.amazonaws.com"}

We can view the content then update it, then view it again.

$ curl s3-website-bucket-db5dae8.s3-website-us-west-2.amazonaws.com
$ curl --header "Content-Type: application/json" --request PUT --data '{"id":"hello","content":"hello updated world!\n"}' http://localhost:1337/sites/hello
$ curl s3-website-bucket-db5dae8.s3-website-us-west-2.amazonaws.com

Once we’re done, we can delete the website.

$ curl --header "Content-Type: application/json" --request DELETE http://localhost:1337/sites/hello

Pulling it all together

So what can you do with this? We used a static website in this example so that results are immediately available. However, this application does basic CRUD operations, which are the basis of many applications. For example, an application could send photos to a bucket or send data to a database application.

Automation API enables building your custom cloud platforms. This example has shown us how to take our customized unit of infrastructure and expose it to our broader team via a REST interface that we define and control. This interface is familiar to developers and feels more comfortable than using IaC directly. Today, we expose a static website, but tomorrow it could be a Kubernetes application platform with all of our networking and security best practices.

Automation API lets you build and explore new implementation patterns with cloud resources. Check out the other things you can build with Automation API.

The Pulumi Automation API - The Next Quantum Leap in IaC

Evan Boyle — Mon, 19 Oct 2020 00:00:00 +0000

Today’s Infrastructure as Code platforms let organizations build rich, reliable, and complex cloud-based applications and architectures. But as teams move to modern cloud technologies, they continue to search for ways to drive increased software-driven automation. Although modern Infrastructure as Code tools bring key software engineering benefits to cloud engineering, they remain focused on human-driven workflows. For example, a person running pulumi up at their terminal or wiring the Pulumi CLI into their CI/CD system.

To scale up how we deploy and manage cloud infrastructure and ultimately unlock the cloud’s value and agility, we believe it will be critical to build software systems around our Infrastructure as Code platforms - systems that scale with software, not just humans.

Pulumi’s approach of using general purpose languages enables Infrastructure as Software – bringing all of the great lessons learned over decades of building software to your infrastructure (languages, IDEs, testing, debugging, componentization, packaging, versioning, and more).

Today we’re excited to announce the Pulumi Automation API, a robust programmatic layer on top of Pulumi’s declarative Infrastructure as Software. The Automation API exposes Pulumi programs and stacks as strongly-typed and composable building blocks.

With Automation API, Pulumi can be fully embedded inside your software projects - to power a wide range of custom cloud infrastructure automation projects. No CLI, no human-in-the-loop, just code.

Enabling Cloud Engineering

Pulumi customers are already applying the Automation API to an incredible breadth of different scenarios. We continue to be amazed by the creativity that it unlocks as a new fundamental building block for cloud engineering.

So far, our users have applied the Pulumi Automation API to these scenarios:

Platform APIs - Embedding Pulumi opens up a world of possibilities for building internal and public cloud platforms that serve your infrastructure abstractions. It enables new approaches to SaaS, including IaaS and PaaS products and platforms. You can create declarative infrastructure defined by your best practices and expose it behind a REST, gRPC, or Custom Resource API that developers and operators alike can easily consume.
Complex Workflow Orchestration - Distributed systems and microservices incur a significant coordination burden, and dependencies across API boundaries are difficult to track and maintain. Automation API is the layer that orchestrates multi-stack deployments, codifying dependencies, and enabling safe incremental deployment. Networking, database systems, application code, schema migrations, and more can be coordinated in a single strongly typed, familiar programming environment.
Cloud Development Frameworks - Automation API provides building blocks in your favorite language. With them, you can author higher-level frameworks and components that blur the lines between infrastructure and application. We expect to see more frameworks that focus on application code that ambiently creates its own infrastructure.
Supercharged Ops Tooling - Create hardened and tested tooling to automate anything entering your ticketing system. From allocating timebound developer VMs to upgrading TLS certificates and scaling servers, Automation API enables you to build custom CLIs and tools that leverage the full power of Infrastructure as Software underneath custom user interfaces.

The Automation API is a new subpackage in each of Pulumi’s language-specific SDKs that provides APIs to create and manage Stacks and perform lifecycle operations like update, refresh, preview, and destroy. It is a strongly typed and safe way to use Pulumi in embedded contexts such as web servers without having to shell out to a CLI.

You can define a Pulumi program as a function within your codebase rather than in a separate project and use methods to get and set configuration parameters programmatically. The Automation API uses a gRPC interface to execute programs that control and communicate with the core Pulumi engine. It still requires a Pulumi CLI installation, as this is how we bundle and distribute the core engine. Today it’s available in preview for TypeScript/JavaScript, Go and Python, with support for C# under active development.

Platform APIs

For SaaS scenarios like deploying infrastructure in response to user action, relying on shell scripts for online services brings operational challenges with observability and reliability. Automation API is just another package and can be instrumented, profiled, and monitored with your existing APM tools like Honeycomb, Prometheus, and Datadog.

Automation API runs inside your favorite frameworks and interacts with your other packages exactly as you’d expect. We can define custom declarative infrastructure and expose it to end-users behind a familiar REST interface:

// an HTTP handler to create instances of our declarative infra
// each request creates an S3-backed static website
// with content from the POST body
const createHandler: express.RequestHandler = async (req, res) => {
 const stackName = req.body.id;
 const content = req.body.content as string;
 try {
 // create a new stack
 const stack = await LocalWorkspace.createStack({
 stackName,
 projectName,
 // generate our pulumi program on the fly from the POST body
 program: createPulumiProgram(content),
 });
 await stack.setConfig("aws:region", { value: "us-west-2" });
 // deploy the stack, tailing the logs to console
 const upRes = await stack.up({ onOutput: console.info });
 res.json({ id: stackName, url: upRes.outputs.websiteUrl.value });
 } catch (e) {
 // built in support for structured error handling
 if (e instanceof StackAlreadyExistsError) {
 res.status(409).send(`stack "${stackName}" already exists`);
 } else {
 res.status(500).send(e);
 }
 }
};

// an HTTP handler to create instances of our declarative infra
// each request creates an S3-backed static website
// with dynamic content from the POST body
func createHandler(w http.ResponseWriter, req *http.Request) {
 w.Header().Set("Content-Type", "application/json")
 var createReq CreateSiteReq
 _ := json.NewDecoder(req.Body).Decode(&createReq)

 stackName := createReq.ID
 // dynamically define our Pulumi Program
 program := createPulumiProgram(createReq.Content)

 s, err := auto.NewStackInlineSource(ctx, stackName, project, program)
 if err != nil {
 // built in support for structured error handling
 // if stack already exists, 409
 if auto.IsCreateStack409Error(err) {
 w.WriteHeader(409)
 fmt.Fprintf(w, fmt.Sprintf("stack %q already exists", stackName))
 return
 }

 w.WriteHeader(500)
 fmt.Fprintf(w, err.Error())
 return
 }
 s.SetConfig(ctx, "aws:region", auto.ConfigValue{Value: "us-west-2"})

 // deploy the stack
 upRes, err := s.Up(ctx, optup.ProgressStreams(os.Stdout))

 response := &SiteResponse{
 ID: stackName,
 URL: upRes.Outputs["websiteUrl"].Value.(string),
 }
 json.NewEncoder(w).Encode(&response)
}

# an HTTP handler to create instances of our declarative infra
# each request creates an S3-backed static website with dynamic content from the POST body
@app.route("/sites", methods=["POST"])
def create_handler():
 """creates new sites"""
 stack_name = request.json.get('id')
 content = request.json.get('content')
 try:
 def pulumi_program():
 return create_pulumi_program(content)
 # create a new stack, generating our pulumi program on the fly from the POST body
 stack = auto.create_stack(stack_name=stack_name,
 project_name=project_name,
 program=pulumi_program)
 stack.set_config("aws:region", auto.ConfigValue("us-west-2"))
 # deploy the stack, tailing the logs to stdout
 up_res = stack.up(on_output=print)
 return jsonify(id=stack_name, url=up_res.outputs['website_url'].value)
 except auto.StackAlreadyExistsError:
 return make_response(f"stack '{stack_name}' already exists", 409)
 except Exception as exn:
 return make_response(str(exn), 500)

The handlers work with your favorite HTTP frameworks like mux, express and flask:

// install necessary plugins once upon boot
ensurePlugins();

// configure express
const app = express();
app.use(express.json());

// setup our RESTful routes for our Site resource
app.post("/sites", createHandler);
app.get("/sites", listHandler);
app.get("/sites/:id", getHandler);
app.put("/sites/:id", updateHandler);
app.delete("/sites/:id", deleteHandler);

// start our http server
app.listen(1337, () => console.info("server running on :1337"));

func main() {
 ensurePlugins()

 router := mux.NewRouter()

 // setup our RESTful routes for our Site resource
 router.HandleFunc("/sites", createHandler).Methods("POST")
 router.HandleFunc("/sites", listHandler).Methods("GET")
 router.HandleFunc("/sites/{id}", getHandler).Methods("GET")
 router.HandleFunc("/sites/{id}", updateHandler).Methods("PUT")
 router.HandleFunc("/sites/{id}", deleteHandler).Methods("DELETE")

 // define and start our http server
 s := &http.Server{
 Addr: ":1337",
 Handler: router,
 }
 fmt.Println("starting server on :1337")
 log.Fatal(s.ListenAndServe())
}

import Flask

# install necessary plugins once upon boot
ensure_plugins()

# initialize our Flask app
app = Flask(__name__)

# set up our routes
@app.route("/sites", methods=["POST"])
def create_handler():
 ...

@app.route("/sites", methods=["GET"])
def list_handler():
 ...

@app.route("/sites/<string:id>", methods=["GET"])
def get_handler(id: str):
 ...

@app.route("/sites/<string:id>", methods=["UPDATE"])
def update_handler(id: str):
 ...

@app.route("/sites/<string:id>", methods=["DELETE"])
def delete_handler(id: str):
 ...

Check out the full Pulumi over HTTP example in TypeScript, Go and Python.

Here at Pulumi, we used Automation API as the foundation of our Kubernetes Operator that exposes Pulumi stacks as Custom Resources. See how your own platform’s operators can take advantage of the Automation API.

We’re seeing customers reach for Automation API to build internal cloud platforms to empower their application teams, as well as independent software vendors using it to build and sell public cloud offerings like SaaS database products.

Complex Workflow Orchestration

Choosing between small, decoupled units of infrastructure and a maintainable orchestration process is no longer necessary. The Automation API executes just like any other code in your application, meaning that a single process can deploy infrastructure and manage related activities such as database migrations and blue/green deployments.

Like any other Pulumi program, we define our cloud resources. In this case, an AWS RDS database:

const clusterInstance = new aws.rds.ClusterInstance("dbInstance", {
 clusterIdentifier: cluster.clusterIdentifier,
 instanceClass: "db.t3.small",
 engine: "aurora-mysql",
 engineVersion: "5.7.mysql_aurora.2.03.2",
 publiclyAccessible: true,
 dbSubnetGroupName: subnetGroup.name,
});

return {
 host: cluster.endpoint,
 dbName,
 dbUser,
 dbPass
};

_, err = rds.NewClusterInstance(ctx, "dbInstance", &rds.ClusterInstanceArgs{
 ClusterIdentifier: cluster.ClusterIdentifier,
 InstanceClass: pulumi.String("db.t3.small"),
 Engine: pulumi.String("aurora-mysql"),
 EngineVersion: pulumi.String("5.7.mysql_aurora.2.03.2"),
 PubliclyAccessible: pulumi.Bool(true),
 DbSubnetGroupName: subnetGroup.Name,
})
if err != nil {
 return err
}

ctx.Export("host", cluster.Endpoint)
ctx.Export("dbName", dbName)
ctx.Export("dbUser", dbUser)
ctx.Export("dbPass", dbPass)
}

cluster_instance = aws.rds.ClusterInstance(
 "db_instance",
 cluster_identifier=cluster.cluster_identifier,
 instance_class=aws.rds.InstanceType.T3_SMALL,
 engine=aws.rds.EngineType.AURORA_MYSQL,
 engine_version="5.7.mysql_aurora.2.03.2",
 publicly_accessible=True,
 db_subnet_group_name=subnet_group.name
)

pulumi.export("host", cluster.endpoint)
pulumi.export("db_name", db_name)
pulumi.export("db_user", db_user)
pulumi.export("db_pass", db_pass)

But with Automation API, we can perform database migrations and preload rows in the same program:

// create (or select if one already exists) a stack that uses our inline program
const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "dev",
 projectName: "databaseMigration",
 // our pulumi program as an "inline" function
 program: pulumiProgram
});

// run the update to deploy our database
const upRes = await stack.up({ onOutput: console.info });
console.log(`db host url: ${upRes.outputs.host.value}`);
console.info("configuring db...");

// establish mysql client
const connection = mysql.createConnection({
 host: upRes.outputs.host.value,
 user: upRes.outputs.dbUser.value,
 password: upRes.outputs.dbPass.value,
 database: upRes.outputs.dbName.value
});

connection.connect();
console.log("creating table...")

// create our table
connection.query(`
CREATE TABLE IF NOT EXISTS hello_pulumi(
 id int(9) NOT NULL,
 color varchar(14) NOT NULL,
 PRIMARY KEY(id)
);
`, function (error, results, fields) {
 if (error) throw error;
 console.log("table created!")
 console.log('Result: ', JSON.stringify(results));
});

// create our stack with an "inline" Pulumi program (deployFunc)
stack := auto.UpsertStackInlineSource(ctx, stackName, projectName, deployFunc)
// run the update to deploy our database
res, err := stack.Up(ctx, stdoutStreamer)

fmt.Println("Update succeeded!")

// get the connection info
host := res.Outputs["host"].Value
dbName := res.Outputs["dbName"].Value
dbUser := res.Outputs["dbUser"].Value
dbPass := res.Outputs["dbPass"].Value

// establish db connection
db := sql.Open("mysql", fmt.Sprintf("%s:%s@tcp(%s:3306)/%s", dbUser, dbPass, host, dbName))
defer db.Close()

// run our database "migration"
fmt.Println("creating table...")
db.Query(`
CREATE TABLE IF NOT EXISTS hello_pulumi(
 id int(9) NOT NULL,
 color varchar(14) NOT NULL,
 PRIMARY KEY(id)
);
`)

# create (or select if one already exists) a stack that uses our inline program
stack = auto.create_or_select_stack(stack_name=stack_name,
 project_name=project_name,
 program=pulumi_program)

up_res = stack.up(on_output=print)
print(f"db host url: {up_res.outputs['host'].value}")

print("configuring db...")
with connect(
 host=up_res.outputs['host'].value,
 user=up_res.outputs['db_user'].value,
 password=up_res.outputs['db_pass'].value,
 database=up_res.outputs['db_name'].value) as connection:
 print("db configured!")

 # make sure the table exists
 print("creating table...")
 create_table_query = """CREATE TABLE IF NOT EXISTS hello_pulumi(
 id int(9) NOT NULL PRIMARY KEY,
 color varchar(14) NOT NULL);
 """
 with connection.cursor() as cursor:
 cursor.execute(create_table_query)
 connection.commit()

See the full database deployment and migration Automation API program in TypeScript, Go and Python.

Cloud Framework Development

The Automation API removes any distinction between infrastructure and application code. Its unique form factor lends all of the predictability and reliability of immutable infrastructure and the desired state configuration you expect from Pulumi while naturally enabling automation and composition in a way that CLIs and DSLs were never designed to handle.

We prototyped a Heroku-like developer experience called halloumi that demonstrates this. All you worry about is writing HTTP handlers. Halloumi worries about the rest, including orchestrating cross-service dependencies, networking, and deploying docker images to ECS Fargate or serialized JavaScript to AWS Lambda.

// this application will deploy multiple services publicly available in the cloud
const application = new App();

const svc1 = new WebService("hello", (expressApp: express.Express) => {
 // a simple static handler
 expressApp.get('/', (req, res) => res.send("omg i'm alive\n"));
});
application.addService(svc1);


const svc2 = new WebService("world", (expressApp: express.Express) => {
 expressApp.get('/', async (req, res) => {
 // track dependencies across services and make them available at runtime
 const svc1Url = svc1.discover();
 // this handler calls svc1 and passes the result through
 const result = await (await fetch(svc1Url)).text()

 res.send(`this is the world. With a window from hello:\n${result} \n`);
 });
});
application.addService(svc2);

application.run().catch(err => { console.error(err) });

app := app.NewApp("petStore")

// a cloud web service that returns a number [0-100)
randomNumberService := web.NewWebService("randomNumber", func() http.Handler {
 // a normal HTTP handler, halloumi takes care of running this in the cloud
 r := mux.NewRouter()
 handler := func(w http.ResponseWriter, r *http.Request) {
 rand.Seed(time.Now().UnixNano())
 fmt.Fprintf(w, "%d", rand.Intn(100))
 }
 r.HandleFunc("/", handler)
 return r
})
app.Register(randomNumberService)

// a cloud web service that returns N of a random animal.
// this service takes a dependency on the URLs of the previously defined services
nAnimalsService := web.NewWebService("nAnimals", func() http.Handler {
 r := mux.NewRouter()
 // we'll read these variables in via requests to other services
 var num string
 var animal string
 handler := func(w http.ResponseWriter, r *http.Request) {
 // Notice how we have the URL of randomNumberService
 // available to consume here!
 numResp := http.Get(randomNumberService.URL())
 num = readResponse(numResp)

 // Notice how we have the URL of randomAnimalService
 // available to consume here!
 animalResp, err := http.Get(randomAnimalService.URL())
 animal = readResponse(animalResp)

 fmt.Fprintf(w, "Wow, you got %s %ss!", num, animal)
 }
 r.HandleFunc("/", handler)
 return r
})

Check out the full halloumi source in TypeScript and Go.

Like any other framework, you can even set a breakpoint to debug orchestration logic or the pulumi program itself, including Apply callbacks.

Supercharged Ops Tooling

Cloud infrastructure is beyond the days of driving deployments and infrastructure by hand, even though many of our tools are CLIs for manual workflows. Cloud engineering demands automated tools to keep up with growth, not just in terms of pace but also in complexity.

Have a look at our temporary development VM provisioner for Azure vmgr. A simple Cobra CLI command creates a new stack with a VM ready to go:

func NewAddCmd() *cobra.Command {
 return &cobra.Command{
 Use: "add",
 Short: "add deploys an additional vm stack",
 Run: func(cmd *cobra.Command, args []string) {
 stackName := fmt.Sprintf("vmgr%d", rangeIn(10000000, 99999999))
 // create a new stack for our VM with an Inline program.
 // the program is nil for now, but we'll set it later after retrieving some dependent outputs.
 s, err := auto.NewStackInlineSource(ctx, stackName, projectName, nil /* Program */)
 fmt.Println("ensuring network is configured...")
 // lookout network and resource group info
 subnetID, rgName, err := EnsureNetwork(ctx, projectName)
 // set our program for the deployment with the resulting network info
 stack.SetProgram(GetDeployVMFunc(subnetID, rgName))
 fmt.Println("deploying vm...")
 // deploy our VM stack
 stdoutStreamer := optup.ProgressStreams(os.Stdout)

 res, err := s.Up(ctx, stdoutStreamer)
 if err != nil {
 fmt.Printf("Failed to deploy vm stack: %v\n", err)
 os.Exit(1)
 }
 fmt.Printf("deployed server running at public IP %s\n", res.Outputs["ip"].Value)
 },
 }
}

Our vmgr cron environment cleanup job takes advantage of Pulumi’s declarative model. It queries the Pulumi Service for stacks older than five days and removes them along with any child resources:

// query all stacks in the project
stacks := w.ListStacks(ctx)
expiration := time.Now().Add(-1 * FIVE_DAYS)
var stacksToDestroy []string
for _, stackSummary := range stacks {
 lastUpdateTime := time.Parse(time.RFC3339, stackSummary.LastUpdate)
 // mark all stacks past expriation date for deletion
 if lastUpdateTime.Before(expiration) {
 stacksToDestroy = append(stacksToDestroy, stackSummary.Name)
 fmt.Printf("Found expired stack %s last deployed at %s\n", stackSummary.Name, stackSummary.LastUpdate)
 }
}

fmt.Printf("Found %d stacks to clean up\n", len(stacksToDestroy))

success := 0
for _, sName := range stacksToDestroy {
 stack := auto.SelectStack(ctx, sName, w)

 fmt.Printf("destroying stack %s\n", sName)

 // tail logs to stdout
 stdoutStreamer := optdestroy.ProgressStreams(os.Stdout)
 // destroy the expired stack
 stack.Destroy(ctx, stdoutStreamer)
 success++
 fmt.Printf("removing stack %s and all associated config and history\n", sName)
 w.RemoveStack(ctx, sName)
 fmt.Printf("stack %s successfully reaped \n", sName)
}

fmt.Printf("destroyed %d stack(s)\n", success)

Our VM provisioner uses Automation API, backed by Pulumi’s desired state model. This means that there’s no chance of resources leaking or getting lost. Check out the full vmgr example here. For a cloud-native example, take a look at our eksctl inspired aksctl for Azure AKS.

Give it a Try Today!

The Automation API is your tool to tame Cloud Engineering complexity and give your team the leverage to automate your cloud infrastructure. It is completely open source and available today for TypeScript/JavaScript, Go and Python. Want to learn more? Come hang out with us in the #automation-api community slack channel. Download the latest Pulumi release and check out these resources to get started:

Go documentation
JavaScript/TypeScript documentation
Python documentation
The Automation API examples repo
The list of known issues. Please file more as you find them!
The Pulumi Kubernetes Operator
aksctl
Halloumi

Keep an eye out! C# support is under active development and coming soon.

We’d like to give a special thank you to the community members who have consulted with us, chimed in on the original GitHub issue, and have been there for the journey in community slack.

🏭 🏭 🏭 Happy automating! 🏭 🏭 🏭

Pulumi Blog: Automation API

Next-level IaC: How Pulumi Supports Your API Economy Strategy

What is API Economy?

What is Pulumi Automation API in a nutshell?

How to build your own Service API with Pulumi Automation API?

Next Steps

Conclusion

Next Steps

Pulumi Cloud

How Starburst Data Creates Infrastructure Automation Magic With Code

Technical and Business Problems to Solve

Why Starburst Chose Pulumi

Benefits of Using Pulumi

Try Pulumi for Infrastructure as Code Automation

IaC Best Practices: Using Automation API

Managing Complexity in a Multi-Project Pulumi Setup

Simplifying Advanced Deployments with Automation API

Performing Stack Operations Efficiently

Summarizing Best Practices: Automation API

Dependent Stack Updates with Pulumi Deployments

Deployment Webhook Destinations

Expressing Stack Dependencies with Pulumi Auto Deploy

Scale Infrastructure with Software

Review Stacks: Collaborate in the Cloud

Configuring Review Stacks

Pulumi Cloud UI

REST API

Pulumi Cloud Resource Provider

Collaborate in the Cloud

Pulumi Deployments: API-Driven Infrastructure at Scale

Deployment Settings and Triggers

Example: Building RESTful Infrastructure APIs

Pave your Golden Path Today

FinOps With Pulumi

What is FinOps?

Who is Responsible for Cloud Financial Decisions?

Establishing FinOps Practices

FinOps: Reactive vs. Proactive Optimization

FinOps with Pulumi

Best First Action: Implement a Tag Policy

A Basic Tag Policy for FinOps will include:

Pulumi Features in Context

Multi-Language Components

CrossGuard

Automation API

Pulumi FinOps in Practice: Sweeping up Dev and QA

Conclusion

Pulumi Deployments: the Fastest Way to Go from Code to Cloud

Automation API & Deployments - Do More with Less

A Golden Path to Production

git push to Deploy

Click to Deploy

Power-up Your Platform with The Deployments API

Drift Detection

TTL Stacks and Temporary Infrastructure

Managing Scale with Remote Automation API

Centralized Observability

Automation Accelerates the Cloud

How Elkjøp Nordic enables self-service infrastructure for developers

The Problem: Balancing Agility and Governance

The Solution: A Self-Service Infrastructure Platform

How the self-service infrastructure platform was built

Watch the demo

Next steps

Observable Infrastructure as Code

Prerequisites

The Pulumi Automation-API

The Deployment Events

Sending Data to Honeycomb

Honeycomb Events & Traces

Linking up Pulumi & Honeycomb

Getting More Details

Getting Resource Deployment Durations

Getting Errors

Summary

Organizational Patterns: A Developer Portal

Conclusion

Announcing Pulumi 3.0

Pulumi Automation API

Native Providers for Azure and Google Cloud

`git push` to Deploy

Start using Automation API: from `pulumi up` to `stack.up()`