Pulumi Blog: Christian Nunciato

Introducing pulumi do: Direct Resource Operations for Any Cloud

Fraser Waters — Fri, 22 May 2026 00:00:00 +0000

Infrastructure as code is the right model for production systems. State tracking, drift detection, and repeatable deployments all matter when you’re managing real workloads.

But sometimes, you also need a quick, one-off interaction with the cloud: create a bucket or a database, look up a VPC, delete a stray resource.

Today we’re introducing pulumi do, a new command for direct resource operations. With pulumi do, you can create, read, update, delete, and query any cloud resource from the terminal with a single command, across thousands of Pulumi-supported providers — no project, code, or state required.

The problem: Sometimes IaC is more than you need

When you’re managing production workloads, IaC is the proven solution. Code lets you declare complex systems, state tracking catches drift before it becomes a problem, dependency graphs sequence changes safely, and policy keeps everything in bounds. That full lifecycle, especially with the backing of a platform like Pulumi Cloud, is exactly what you want to build systems that scale.

But when you (or your coding agent) need an ad-hoc Postgres database, the simplest path with IaC still takes several steps: make a directory, create a project, configure your credentials, write the code, preview, deploy. It works, but it’s not always necessary for what should be a simple operation. pulumi do collapses all of those steps into one, using the same Pulumi providers, resource model, and ecosystem that powers the core Pulumi platform.

Resource creation is also only part of the problem. As Joe laid out in The Agentic Infrastructure Era, the real challenge for AI agents isn’t with code or CLI commands, it’s with everything else: getting a cloud account, resolving credentials, wiring configuration across multiple services. Agent accounts, also released this week, simplify this by letting an agent provision its own ephemeral Pulumi Cloud account, and Pulumi ESC takes care of consolidating credentials across providers. Together, with pulumi do, agents can now go from zero to deployed infrastructure without requiring a human in the loop — and when that one-off resource needs to grow into a more permanent system, there’s a clear graduation path back to full Pulumi IaC.

What it looks like

As an example, say you wanted to provision an S3 bucket. With the AWS CLI, you’d need to assemble an aws s3api create-bucket invocation with the right set of command-line flags, region constraints, a globally unique name, and so on. With pulumi do, it’s just this:

$ pulumi do aws:s3:Bucket create

That might not look all that different on the surface — but because you’re using the Pulumi engine and resource model, you can provide a minimal set of input properties, take advantage of provider-defined defaults, and use Pulumi’s auto-naming feature to give the bucket a unique name automatically:

$ pulumi do aws:s3:Bucket create

This will create aws:s3/bucket:Bucket with the following inputs:
{
 "bucket": "bucket-279ea56",
 "tagsAll": {}
}

Please confirm that this is what you'd like to do by typing `yes`:

Answer yes (or just pass --yes), and you’re done. To delete the bucket:

$ pulumi do aws:s3:Bucket delete bucket-279ea56 --yes

Need to look up an existing resource? Use a provider function:

$ pulumi do aws:ec2:getVpc --default

{
 "arn": "arn:aws:ec2:us-west-2:663782525873:vpc/vpc-d7b311af",
 "cidrBlock": "172.31.0.0/16",
 "enableDnsHostnames": true,
 "enableDnsSupport": true,
 "enableNetworkAddressUsageMetrics": false,
 "id": "vpc-d7b311af",
 ...
}

Same CLI, same output contract, same provider ecosystem.

The command shape

The do command accepts a Pulumi resource type, or type token, to determine the action to take. Type tokens have the form <package:module:resource>. For example, aws:s3:Bucket refers to the Amazon S3 Bucket resource that belongs to the s3 module of the aws package.

You can also provide a portion of the token to help you find what you’re looking for without ever having to leave the terminal:

$ pulumi do aws:s3

Functions and resources for the s3 module.

Run 'pulumi do <module/resource/function> --help' for more details on usage.

Functions:
 aws:s3:getAccessPoint
 aws:s3:getAccountPublicAccessBlock
 aws:s3:getBucket
 aws:s3:getBucketObject
 ...

Resources:
 aws:s3:AccessPoint
 aws:s3:AccountPublicAccessBlock
 aws:s3:AnalyticsConfiguration
 aws:s3:Bucket
 ...

$ pulumi do aws:s3:Bucket read bucket-d20976f

{
 "arn": "arn:aws:s3:::bucket-d20976f",
 "bucket": "bucket-d20976f",
 "bucketDomainName": "bucket-d20976f.s3.amazonaws.com",
 "bucketNamespace": "global",
 ...
}

The package, module, and resource/function segments all come directly from the Pulumi provider schema, so --help works at every level of the tree. Pass a package name, optional module, and optional function or resource type, and do returns the appropriate level of detail.

You can also provide the input properties of a resource in a YAML or JSON file with the --input option. To create a container service in Google Cloud Run for example:

# service.yaml
location: us-central1
deletionProtection: false
template:
 containers:
 - image: us-docker.pkg.dev/cloudrun/container/hello

$ pulumi do gcp:cloudrunv2:Service create \
 --input yaml \
 --input-file service.yaml

This will create gcp:cloudrunv2/service:Service with the following inputs:
{
 "deletionProtection": false,
 "location": "us-central1",
 "name": "service-b8af752",
 "template": {
 "containers": [
 {
 "image": "us-docker.pkg.dev/cloudrun/container/hello"
 }
 ]
 }
}

The result:

{
 "createTime": "2026-05-22T23:00:22.415839Z",
 ...
 "urls": [
 "https://service-b8af752-921927215178.us-central1.run.app",
 "https://service-b8af752-ctnulmzwoa-uc.a.run.app"
 ]
}

Resource operations

Most resources support the full set of CRUD operations — create, read, update, delete, and list — directly from the CLI. Each operation maps to a provider CRUD method using the same provider logic a full Pulumi program would use, and resources are addressable by their cloud provider IDs:

# Create a resource
$ pulumi do aws:s3:Bucket create --yes | jq -r ".name"
bucket-4f5cb22

# Fetch it
$ pulumi do aws:s3:Bucket read bucket-4f5cb22 | jq -r ".hostedZoneId"
Z3BJ6K6RIION7M

# Update/patch it
$ pulumi do aws:s3:Bucket patch bucket-4f5cb22 --input yaml --input-file tags.yaml

$ pulumi do aws:s3:Bucket read bucket-4f5cb22 | jq ".tags"
{
 "key": "value"
}

# Delete it
$ pulumi do aws:s3:Bucket delete bucket-4f5cb22

Provider configuration

Today, pulumi do resolves provider configuration — for example, applying your AWS credentials — using environment variables or credential files as supported by each individual Pulumi provider. See the Pulumi Registry for provider-specific configuration details.

Designed for humans and agents

We’ve designed pulumi do to serve humans and coding agents equally well, guided by three fundamental ideas:

Consistent command structure across every provider. The do <package:module:type> <operation> pattern is the same for AWS, Azure, Google Cloud, Kubernetes, Cloudflare, Datadog, and every provider, including packages containing higher-level component resources. Once an agent learns that pattern, it applies across the board.
Predictable output contract. JSON on stdout, progress on stderr, consistent exit codes. An agent can parse the result programmatically without scraping human-formatted tables.
A single CLI command that works across every cloud. Many cloud and SaaS providers don’t have a full CLI at all. pulumi do generates commands from the provider schema, so if a Pulumi provider exists for it, the CLI just works. Neither humans nor agents need to install, learn, or even know about cloud provider-specific tooling.

What’s next

Resource operations and provider functions are the foundation. The pulumi do roadmap extends the same direct-operation model with credential management, state tracking, and a path to full IaC.

Unified credentials with Pulumi ESC

One of the hardest parts of multi-cloud operations is credential management. Every provider has its own authentication scheme, environment variables, and session lifecycle. An agent working across AWS, Cloudflare, and Datadog today manages three separate credential mechanisms.

We’re building Pulumi ESC integration into pulumi do so you can manage credentials in one place and resolve them everywhere. ESC handles credential resolution (including OIDC-based dynamic credential generation and short-lived tokens) across all of your providers. Name the credential set, reference it, and ESC does the rest, with rotation, RBAC, and audit built in.

Cross-resource references

Real infrastructure has dependencies — subnets need VPCs, security group rules need their security groups, and so on. When you’re building resources one at a time, those references need to flow between commands somehow.

A future version of pulumi do will let resource inputs reference outputs from previously created resources, allowing the CLI to resolve them automatically and preserve the dependency graph. Later, when the time comes to graduate to a full IaC program, the generated code contains proper resource references rather than hard-coded strings.

Stateful mode and the graduation path

Today, pulumi do is stateless. Each command runs independently. A planned stateful mode will persist resource state across operations, enabling drift detection, lifecycle management, and a graduation path to full infrastructure as code.

Here’s what we’re planning:

Zero setup. Your first pulumi do implicitly creates a project and stack. No manual initialization.
Accumulate resources. Each operation stores resource state. After a few commands, you have a lightweight representation of your infrastructure.
Eject to a full project. When the time comes, generate a Pulumi project in your chosen language with all resources imported and dependency graphs intact.
Connect to Pulumi Cloud. Layer on governance, compliance, team collaboration, and deployment automation through Pulumi Cloud. Resources created via pulumi do can be governed by Pulumi Insights from day one, even before you opt into full IaC.

This path works because pulumi do uses the same providers, resource types, and property schemas as every other pulumi operation. Provisioned cloud resources stay where they are as management capabilities are added as needed.

Get started

pulumi do ships as a research preview in Pulumi CLI v3.242.0 and later. Install or update the CLI, install a provider plugin, and start running commands. The documentation has the full reference.

We can’t wait to hear your feedback. Give it a try today, tell us what works (and what doesn’t), and help shape the CLI that agents and humans both reach for first.

Better CLI Interactions for Agents and Humans

Michael Fallihee — Tue, 19 May 2026 04:00:00 -0700

AI agents do a lot of their work through CLIs. They’re easier to call than HTTP APIs and they produce predictable output. Over the last few months our own CLI traffic has shifted from mostly people typing commands to people and agents running commands together, often in the same session.

Today we’re shipping a release built for both. The Pulumi CLI is reorganized around three ideas: the right command should be the one you can guess, anything you can do in Pulumi Cloud should also be doable from the terminal, and what comes back should be just as readable to an agent as it is to a person.

Designing for guessability

The bar we set was that both developers and coding agents should be able to guess at the right command for a particular task: pulumi env edit to modify an environment, pulumi stack get to see what’s going on with a stack, pulumi org member list to see who’s on the team. If we had to explain which command did what, the usability bar hadn’t been met.

Branches in the tree are now singular nouns like stack, env, org, and deployment. Leaves are now verbs from a canonical vocabulary — list, get, set, new, edit, remove — and they mean the same thing wherever they’re used. edit always means modify an existing thing. Wherever the old vocabulary differed, though, the old name still works: ls, rm, update, and open are all aliased to preserve backward compatibility.

For the most part, product names have also been replaced with familiar nouns. Users (human or otherwise) don’t think in product names; they think in terms of resources, stacks, environments. For example, take Pulumi ESC: the product may be named ESC (and for a while the command was too), but nobody thinks I need to initialize a new ESC — they think I need to create a new environment. The command is therefore pulumi env new, with esc init preserved as an alias to avoid disrupting anyone’s existing workflows.

$ pulumi env new my-project my-env
Environment created.

All of Pulumi Cloud in the terminal

Up to now, most of what you could do with Pulumi Cloud had to be done either in the browser or through direct API calls. Things like reviewing deployments, setting up webhooks, finding non-compliant resources, or managing deployment settings all required you to break out curl and hit the API docs or open a browser and navigate the Pulumi Cloud console.

That changes today. Pulumi Cloud is now fully accessible from the command line through the pulumi CLI, with consistently named nouns and verbs aligned to what you’d expect:

pulumi stack get returns a complete stack overview, metadata, resource list, and more:

$ pulumi stack get \
 --stack cnunciato/chris.nunciato.org/production \
 --output json | jq -r ".resources[].type" | grep "aws:s3"

aws:s3:BucketEventSubscription
aws:s3/bucket:Bucket
aws:s3/bucket:Bucket
aws:s3/bucketPublicAccessBlock:BucketPublicAccessBlock
aws:s3/bucketWebsiteConfiguration:BucketWebsiteConfiguration
aws:s3/bucketOwnershipControls:BucketOwnershipControls
aws:s3/bucketNotification:BucketNotification

… with other stack-related commands like pulumi stack history get events, pulumi stack drift list, pulumi stack schedule new, and pulumi stack webhook new alongside it.

Organizational commands like pulumi org member list, pulumi org role list, pulumi org usage get, and pulumi org audit-log export can help you dig into the details when you need to as well.

Deployment-related commands like pulumi deployment list, get, log, and cancel let you see what’s running, dive into what happened, and take action without having to leave the terminal.

$ pulumi deployment list \
 --stack cnunciato/chris.nunciato.org/production \
 --output table

┌──────────────────────────────────────┬───────────┬─────────┬───────────┬──────────────┬─────────────────────────┐
│ ID │ OPERATION │ VERSION │ STATUS │ INITIATED BY │ MODIFIED │
├──────────────────────────────────────┼───────────┼─────────┼───────────┼──────────────┼─────────────────────────┤
│ 83e44b8c-643c-4e9f-9f36-0c6a81d9db2e │ update │ 140 │ running │ cnunciato │ 2026-05-17 21:26:37.340 │
│ 52a37cbe-b7fd-4027-8e0f-7b4785ab12e8 │ update │ 139 │ succeeded │ cnunciato │ 2026-05-16 23:36:07.999 │
│ 94e04525-b3a4-42b5-9987-e344018a3324 │ preview │ 138 │ succeeded │ cnunciato │ 2026-05-16 23:29:19.709 │
└──────────────────────────────────────┴───────────┴─────────┴───────────┴──────────────┴─────────────────────────┘

And when you need to query across managed (and even unmanaged) resources, pulumi insights resource search and get can help you find what you’re looking for quickly:

$ pulumi insights resource search \
 --query 'type:aws:s3/bucket:Bucket org:cnunciato project:photomap stack:dev' \
 --output table

┌──────────────────────────────────────────────────────────────────────────┬──────────────────────┬───────┬──────────────────────────┐
│ URN │ TYPE │ STACK │ MODIFIED │
├──────────────────────────────────────────────────────────────────────────┼──────────────────────┼───────┼──────────────────────────┤
│ urn:pulumi:dev::photomap::aws:apigateway:x:API$aws:s3/bucket:Bucket::api │ aws:s3/bucket:Bucket │ dev │ 2020-10-31T00:39:47.926Z │
│ urn:pulumi:dev::photomap::aws:s3/bucket:Bucket::images │ aws:s3/bucket:Bucket │ dev │ 2020-10-31T00:39:47.926Z │
└──────────────────────────────────────────────────────────────────────────┴──────────────────────┴───────┴──────────────────────────┘

Showing 2 of 2 resources.

Flags and output formats are consistent across commands (--output table, json), as are the shapes of cross-cutting features like webhooks. If you’ve used pulumi stack webhook, for example, you already know how to use pulumi env webhook and pulumi org webhook, and so on.

Direct access to the Pulumi Cloud API

For any features of Pulumi Cloud that don’t yet have their own commands, you’ve also got pulumi api. It’s a gh api-inspired command designed to give you direct access to the full REST API, without having to manage separate access tokens, auth settings, or request/response payloads. Everything is handled for you through your authenticated pulumi CLI.

There’s even pulumi api list, which enumerates every single endpoint that’s exposed:

$ pulumi api list

┌───────────────┬────────┬───────────────────────────────────────┬──────────────────────────────┐
│ TAG │ METHOD │ PATH │ SUMMARY │
├───────────────┼────────┼───────────────────────────────────────┼──────────────────────────────┤
│ AccessTokens │ GET │ /api/orgs/{orgName}/tokens │ ListOrgTokens │
│ AccessTokens │ POST │ /api/orgs/{orgName}/tokens │ CreateOrgToken │
│ AccessTokens │ DELETE │ /api/orgs/{orgName}/tokens/{tokenId} │ DeleteOrgToken │
│ AccessTokens │ GET │ /api/user/tokens │ ListPersonalTokens │
│ AccessTokens │ POST │ /api/user/tokens │ CreatePersonalToken │
│ AccessTokens │ DELETE │ /api/user/tokens/{tokenId} │ DeletePersonalToken │
...

537 operations. Pass --output=json for a stable, scriptable contract.

To get the details about a particular API, use pulumi api describe:

$ pulumi api describe 'DELETE /api/user/tokens/{tokenId}' # or DeletePersonalToken

DELETE /api/user/tokens/{tokenId}
Tag: AccessTokens
Operation: DeletePersonalToken

DeletePersonalToken

Permanently deletes a personal access token by its identifier. The token is immediately
invalidated and can no longer be used for authentication. Returns 204 on success or 404
if the token does not exist.

Parameters:
 [path] tokenId* (string) — The access token identifier

All requests are made through your authenticated pulumi CLI:

$ pulumi login
Logged in to pulumi.com as cnunciato.

$ pulumi whoami
cnunciato

$ pulumi api /api/user/tokens/2cf15c7d-afad-458f-ace0-fc7ff0512b10 \
 --method DELETE && echo "Token deleted."
Token deleted.

Newly published endpoints are available through pulumi api immediately, so you don’t have to wait for a new CLI release before you can start using them. See the Pulumi Cloud REST API documentation to learn more.

Finding templates in the Pulumi Cloud Registry

Finding out which templates are available to you through your Pulumi organization used to mean having to navigate to the Pulumi Cloud Registry and start searching. The new pulumi template commands make this easier by letting you ask for what’s available right from the shell, either by fetching the full list or filtering with the --name or --search params:

$ pulumi template list --search "container typescript" --org cnunciato

┌─────────────────────────────────────────────┬────────┬────────────┬────────────┐
│ Name │ Source │ Language │ Visibility │
├─────────────────────────────────────────────┼────────┼────────────┼────────────┤
│ pulumi/templates/container-aws-typescript │ github │ typescript │ public │
│ pulumi/templates/container-azure-typescript │ github │ typescript │ public │
│ pulumi/templates/container-gcp-typescript │ github │ typescript │ public │
└─────────────────────────────────────────────┴────────┴────────────┴────────────┘

This is especially useful when you’re working with an agent because it helps the agent discover your org’s approved templates without having to name them. Start with a prompt that tells the agent what you want to build, and let the agent find the right template for you.

Agent-friendly Markdown docs for providers and components

Both humans and agents need to be able to understand what’s inside a Pulumi package before they can use it. And while the Registry is an excellent resource for that, it was mainly designed to deliver HTML — a human-friendly format that agents can certainly use, but that’s much more verbose than they actually need.

With pulumi api, agents can fetch the details about a package from the Registry directly and get back those details either in markdown or json, whichever works best, filtering on properties like language where applicable:

$ pulumi api "/api/registry/packages/pulumi/pulumi/random/versions/4.19.1"

{
 "name": "random",
 "publisher": "pulumi",
 "publisherDisplayName": "Pulumi",
 "source": "pulumi",
 "version": "4.19.1",
 "description": "A Pulumi package to safely use randomness in Pulumi programs.",
 "repoUrl": "https://github.com/pulumi/pulumi-random",
 ...
}

$ pulumi api "/api/registry/packages/pulumi/pulumi/random/versions/4.19.1/docs/random%3Aindex%2FrandomPassword%3ARandomPassword" \
 --output markdown

# RandomPassword

resource `random:index/randomPassword:RandomPassword`

## Example Usage

package main
...

Resources are individually addressable using their URL-encoded Pulumi type tokens — e.g., random:index/randomPassword:RandomPassword — and API endpoints are configured to deliver Markdown when agents ask for it:

$ curl "https://api.pulumi.com/api/registry/packages/pulumi/pulumi/random/versions/latest/readme?lang=python" \
 -H "Accept: text/markdown"

# Installation

The Random provider is available as a package in all Pulumi languages:
...

Even compared to JSON (which is itself a significant improvement over HTML), Markdown is a much more token-efficient format for agents to work with:

Package	Endpoint	JSON	Markdown	Tokens saved
random	`/readme`	10.68 KB	6.04 KB	43%
aws	`/readme`	4.22 KB	2.54 KB	40%
aws	`/nav?depth=full`	204 KB	170 KB	17%
aws	`/docs/{resource type token}`	15.24 KB	11.28 KB	26%
azure-native	`/docs/{resource type token}`	48.13 KB	30.37 KB	37%
aws	`/docs/{function type token}`	2.40 KB	1.46 KB	39%

Learn more about our Registry endpoints in the REST API docs. (Or just ask your agent!)

New to the CLI: Pulumi Neo

When we launched Pulumi Neo last year, the only way to use it was in the Pulumi Cloud Console. But while there’s a ton you can do with Neo in the browser, if you’re an engineer already living in the terminal, chances are that eventually you’re going to wish you had Neo right in the CLI along with you.

Now you do. Running pulumi neo with or without a prompt launches a Pulumi Cloud-connected session that gives Neo access to your local environment just like any other coding agent. Use it on its own to scaffold a new project, understand an existing codebase, or debug a failing deployment — or pull it into an active session with the coding agent you’re already using. Either way, it stays in the shell you’re already working in.

We’ll cover Neo in the CLI in more detail later this week. In the meantime, here’s a peek:

Smaller changes that add up

A long list of smaller changes also runs through this release:

The core loop now speaks JSON end to end, with pulumi up, pulumi destroy, and pulumi import all emitting structured JSON output when called with --output json.
Streams now behave the way scripts expect them to, with data on stdout, progress and diagnostics on stderr.
Exit codes are more consistent across the board. Every failure mode — auth, resource, policy, missing stack, cancellation, timeout, and others — has its own exit code, so agents can branch on the actual cause instead of having to interpret output. The full table is in the docs.
Help text explains why a command exists, not just what it does, and includes at least one concrete example. Examples in --help are one of the most effective ways to improve LLM accuracy on first-try invocations — and it turns out they’re pretty handy for humans, too.

A sneak peek at a new command

Later this week, you’ll get a closer look at pulumi do, a new top-level command that enables direct resource operations like create, read, update, delete, and list across every Pulumi-supported cloud provider and resource, all in one command. A simple example:

$ pulumi do aws getAvailabilityZones

{
 "groupNames": [
 "us-west-2-zg-1"
 ],
 "id": "us-west-2",
 "names": [
 "us-west-2a",
 "us-west-2b",
 "us-west-2c",
 "us-west-2d"
 ],
 "region": "us-west-2",
 "zoneIds": [
 "usw2-az2",
 "usw2-az1",
 "usw2-az3",
 "usw2-az4"
 ]
}

It might look like that’s calling the AWS CLI, but it’s not — it’s using the same AWS provider function a full Pulumi program would use, only without the program, and invoked directly from the CLI.

More on how it works, and what you can do with it, in the days ahead.

Try it yourself

A lot of what makes a developer tool worth using is in the details, and most of what’s in this release is exactly that, across the whole CLI, with humans and agents in mind.

We’d love for you to grab the latest release and give it a try. Tell us what’s now easy, what’s still hard, and what to fix next on GitHub or in the community Slack. The fastest way the CLI gets better is feedback from the humans and agents who live in it.

CDKTF is deprecated: What's next for your team?

Adam Gordon Bell — Thu, 18 Dec 2025 10:00:00 -0800

In July, 2020, CDK for Terraform (CDKTF) was introduced, and last week, on December 10, it was officially deprecated. Support for CDKTF has stopped, the organization and repository have been archived, and HashiCorp/IBM will no longer be updating or maintaining it, leaving a lot of teams out there without a clear path forward.

For most teams, that means it’s time to start looking for a replacement.

It’s an unfortunate situation to suddenly find yourself in as a user of CDKTF, but you do have options, and Pulumi is one of them. In this post, we’ll help you understand what those options are, how Pulumi fits into them, and what it’d look like to migrate your CDKTF projects to Pulumi.

What are the alternatives to CDKTF?

Teams migrating away from CDKTF generally have three options:

Option 1: Fall back to HCL

HashiCorp’s official recommendation is to export your projects to HashiCorp Configuration Language (HCL) and manage them with Terraform. CDKTF even has a command that makes this fairly simple:

cdktf synth --hcl

Of course, if you’re using CDKTF, you probably chose it specifically to avoid HCL. So while possible, this probably isn’t the choice most teams would make unless they had to.

Option 2: Migrate to AWS CDK

If your team is all-in on AWS, another option would be to migrate to AWS CDK. It’s widely used, officially supported, the programming model is similar to CDKTF’s, and both CDK and CDKTF transpile to an intermediate format (CloudFormation YAML and Terraform JSON, respectively) that gets passed on to their underlying tools for deployment.

But while their programming and deployment models are conceptually similar, their resource models and APIs are entirely different. Here’s the code for an S3 bucket written in AWS CDK, for example:

import * as s3 from 'aws-cdk-lib/aws-s3';

const bucket = new s3.Bucket(this, 'my-bucket', {
 bucketName: 'my-example-bucket',
 versioned: true,
 publicReadAccess: false,
});

And here’s the code for a similarly configured bucket in CDKTF:

import { S3Bucket } from '@cdktf/provider-aws/lib/s3-bucket';

const bucket = new S3Bucket(this, 'my-bucket', {
 bucket: 'my-example-bucket',
 versioning: {
 enabled: true,
 },
 acl: 'private',
});

Notice how different these APIs are — and this is just one simple resource with only a few properties; imagine having to rewrite dozens or hundreds of them. Beyond that, there’s also the problem of state: How would you go about translating the contents of a Terraform state file containing hundreds of resources into the equivalent CloudFormation YAML or JSON?

Despite their surface similarities, CDKTF and AWS CDK have little in common. Migration would essentially mean a ground-up rewrite that’d also leave you without the multi-cloud support you already have with CDKTF. For most teams, that makes this option a practical non-starter.

Option 3: Migrate to Pulumi

This is where we should acknowledge our obvious bias — but we genuinely believe that for most users of CDKTF, Pulumi really is the simplest and most broadly compatible alternative.

Like CDKTF, Pulumi lets you build and manage your infrastructure with general-purpose languages like TypeScript, Python, Go, C#, and Java, and it supports organizing your code into higher-level abstractions called components, which you can think of like CDKTF constructs. Both organize cloud resources into stacks (think dev, prod), and both track deployment state similarly, with local, remote, and cloud-hosted options available.

Many of Pulumi’s most popular providers (e.g., the AWS provider) are also built from open-source Terraform schemas, which means their resource models will be nearly identical to what you’re used to with CDKTF. Here’s what an S3 bucket looks like in Pulumi, for example:

import * as aws from '@pulumi/aws';

const bucket = new aws.s3.Bucket('my-bucket', {
 bucket: 'my-example-bucket',
 versioning: {
 enabled: true,
 },
 acl: 'private',
});

You can also use any Terraform provider with Pulumi, and you can even reference Terraform modules directly from within your Pulumi code.

Pulumi is also different from CDKTF in several ways. One is that rather than transpile your source code to a format like JSON as CDKTF does (and then deploying it separately later), Pulumi uses its own declarative deployment engine that resolves the resource graph at runtime and provisions cloud resources directly, which is much faster and more flexible. You can learn more about the deployment model in How Pulumi Works.

Given the API similarities, the support for all Terraform providers and modules, the ability to coexist alongside Terraform-managed projects, and the built-in support for conversion (which we’ll cover next), we think Pulumi is the best alternative for most teams looking to migrate.

Migrate from CDKTF to Pulumi

Keep writing infrastructure in TypeScript, Python, Go, C#, or Java, and use Pulumi’s built-in convert and import commands to bring your CDKTF projects and state across.

Get started

What migrating to Pulumi looks like

Migrating a CDKTF project to Pulumi generally happens in three steps:

Conversion, which translates your CDKTF code into a new Pulumi program
Import, which reads the contents of your CDKTF state into a new Pulumi stack
Refactoring, which brings the code in the new program into alignment with the stack’s currently deployed resources

Conversion and import

Migration starts with exporting your CDKTF project to HCL with cdktf synth. From there, Pulumi’s built-in convert and import commands handle creating the new program and importing your state:

# Export your project to HCL.
cdktf synth --hcl

# Convert the HCL into a new Pulumi project.
pulumi convert --from terraform --language typescript

# Create a new Pulumi stack.
pulumi stack init dev

# Import your CDKTF stack's resources into your new Pulumi stack.
pulumi import --from terraform ./terraform.dev.tfstate

The converter automatically translates Terraform input variables, data sources, resources, and outputs into their Pulumi equivalents. You can read more about how this works in Converting Terraform HCL to Pulumi.

Refactoring

Once you’ve imported your state, you’ll often have to make some adjustments to the code to bring it in line with the new Pulumi stack. For instance, pulumi import marks new resources protected by default, to prevent them from being accidentally deleted — but since the code produced by pulumi convert doesn’t include the protect resource option, you’ll need to add it yourself. Fortunately the import step also emits code that you can copy into your program to make this process a little easier.

Refactoring can get a bit more complicated when custom logic and higher-level abstractions are involved, as fidelity to the original CDKTF code is often lost in the translation to HCL. In these situations, having the help of an LLM to recapture that original logic or translate your CDKTF constructs into Pulumi components can be a big time-saver.

An end-to-end example

The best way to get a feel for how this works, though, is to try it yourself.

The pulumi/cdktf-to-pulumi-example repository on GitHub contains a CDKTF project with multiple stacks written in TypeScript, along with a guide that walks you through the process of migrating that project to Pulumi. The guide covers everything we’ve discussed here so far, including:

Converting the CDKTF project into a new Pulumi project
Importing its actively running resources into Pulumi stacks
Modifying the generated code to align with imported state
Performing an initial deployment with Pulumi to complete the migration process

The walkthrough takes only a few minutes to complete, and it’s a great way to stand up an example of your own to get more familiar with Pulumi.

github.com/pulumi/cdktf-to-pulumi-example

What’s next?

If you’re moving on from CDKTF and looking for an alternative, there are a few possible paths forward. For teams that want to keep using real languages and avoid a ground-up rewrite, Pulumi offers the clearest way forward.

To learn more about how Pulumi works, how it differs from CDKTF and from Terraform, how to handle additional conversion scenarios, and more, we recommend:

Diving into the Pulumi docs to get familiar with core concepts and features of the platform
Reading Migrating from Terraform or CDKTF to Pulumi for more detailed, Terraform-specific migration guidance
Joining us in the Pulumi Community Slack to ask questions and learn from others who’ve successfully made the leap from Terraform and CDKTF to Pulumi
Checking out Pulumi for All Your IaC — Including Terraform and HCL to learn more about Pulumi’s native support for Terraform and HCL

And of course, feel free to reach out! We’d love to help in any way we can.

Next-level IaC: Drop those wrapper scripts and let your language do that for you

Christian Nunciato — Tue, 30 Apr 2024 00:00:00 +0000

Our users are always telling us (particularly the ones who come to Pulumi from other IaC tools) that being able to use general-purpose languages to manage their infrastructure was a game changer for them.

I know it was for me. As a JavaScript developer, when I discovered Pulumi and saw that I could do pretty much everything I was doing with Terraform but with TypeScript, I was immediately hooked; that’s all it took. Just being able to write my resource declarations in a language I knew well (and that my IDE understood) was huge.

But it’s easy to forget, even when you’ve been using Pulumi for a while, that you aren’t just getting the language alone — you’re getting the full capability of the language’s runtime environment also. Not just JavaScript, but all of Node.js. Not just C#, but everything the .NET framework, runtime, and ecosystem have to offer.

This is quite a big deal, actually. Whereas with other tools, you’re often bound by the limits of the tool’s DSL, with Pulumi, you have no such constraints; you can do just about anything your chosen language can do. That means much more than just declaring long lists of cloud resources — it means unlocking all kinds of capabilities that can help bring your infrastructure-management practice to a whole new level.

In this post, the first of a new series we’re calling Next-level IaC, we’ll show you a few small but useful ways you can take advantage of this reality — first, by spotting some opportunities to simplify your deployments.

Run pre-deployment tasks with Pulumi

Fundamentally, a Pulumi program is just a regular program. That means you can take actions programmatically at runtime — before the Pulumi engine gets going.

Suppose you were tasked with writing the code to deploy and manage a static website. Conceptually, you might break this process up into a few separate steps:

Build the website
Deploy the supporting infrastructure (e.g, the cloud storage, maybe a content-delivery network)
Copy the content of the website into the infrastructure

This you might lead you to reach for Bash to write up a shell script like the one below, which uses Hugo (a popular static-site generator) to build the website, Pulumi to provision the infrastructure, and the AWS CLI to push the built website up into the AWS cloud:

#!/bin/bash

# Build the website.
hugo --destination ./public

# Provision the infrastructure.
pulumi up --yes

# Copy the website content into the infrastructure.
aws s3 sync ./public s3://$(pulumi stack output bucketName)

And this would definitely work. But it’s also unnecessary. You can avoid both the Bash wrapper and the dependency on the AWS CLI by using your chosen language’s built-in support for spawning shell processes:

import * as aws from "@pulumi/aws";
import * as childProcess from "child_process";

// Build the website.
const result = childProcess.execSync("hugo --destination ./public", { stdio: "pipe", cwd: "./www" });

// Provision a storage bucket for the website.
const bucket = new aws.s3.Bucket("bucket", {
 website: {
 indexDocument: "index.html",
 },
});

// Copy the website home page into the bucket.
const homepage = new aws.s3.BucketObject(
 "index.html",
 {
 bucket: bucket.id,
 content: fs.readFileSync("./www/public/index.html", "utf-8"),
 contentType: "text/html",
 acl: "public-read",
 },
);

import pulumi_aws as aws
import subprocess

# Build the website.
result = subprocess.run(
 ["hugo", "--destination", "./public"],
 stdout=subprocess.PIPE,
 cwd="./www",
 check=True,
 shell=True,
)

# Provision a storage bucket for the website.
bucket = aws.s3.Bucket(
 "bucket", website=aws.s3.BucketWebsiteArgs(index_document="index.html")
)

# Copy the website home page into the bucket
homepage = aws.s3.BucketObject(
 "index.html",
 bucket=bucket.id,
 content=open("./www/public/index.html", "r").read(),
 content_type="text/html",
 acl="public-read",
)

Running processes like these synchronously, as above, means they’ll complete before Pulumi starts registering resources — important here because the BucketObject resource needs that public folder to exist for the deployment to succeed. Fortunately it does, so all is well:

$ pulumi up
Updating (dev)
Type Name Plan Info
+ pulumi:pulumi:Stack example-dev create 15 messages
+ ├─ aws:s3:Bucket bucket create
+ └─ aws:s3:BucketObject index.html create
Diagnostics:
pulumi:pulumi:Stack (example-dev):
Start building sites ...
| EN
-------------------+-----
Pages | 4
Paginator pages | 0
Non-page files | 0
Static files | 0
Processed images | 0
Aliases | 0
Sitemaps | 1
Cleaned | 0
Total in 11 ms
Resources:
+ 3 created

Also notice how the example captures the output of the process in result and writes it to the terminal. Standard logging like this tends not to be possible with static languages (at least not without some hackery) because there’s no real way to do it programmatically. Similarly, those that do support spawning processes tend to require them to return strictly formatted JSON strings — which, as the example shows, not all processes do. (Forcing you to have to wrap them with yet more shell scripts.)

Simple stuff, but flexible and powerful. With one line, you’ve kept your deployment process simple, dropping two tools you don’t need thanks to the built-in capabilities of Pulumi and your language of choice.

Fetch data to generate resources dynamically

To build on the static-website scenario, suppose you wanted to retrieve some data from an external source and use it to provision some cloud resources dynamically. For example, you might want to let your marketing team create short URLs on their own for social media posts. For this, you might once again turn to Bash and the AWS CLI to pull those URLs from your corporate CMS (e.g., with curl), either before or after running pulumi, and create the corresponding redirects imperatively using a follow-up shell command of some sort.

Again, a totally reasonable approach. But again, why bother, when you have the full capability of Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET at fingertips? Just add a few lines to your program to fetch the data and be on your way:

// ...

// Fetch some redirects from a hypothetical CMS.
const redirects = fetch(`${process.env.CMS_ENDPOINT}/redirects.json`)
 .then(response => response.json())
 .then(items =>
 items.forEach((redirect: any, i: number) => {
 // Create an S3 website redirect for each one.
 new aws.s3.BucketObject(
 `redirect-${i}`,
 {
 bucket: bucket.id,
 key: redirect.from,
 websiteRedirect: redirect.to,
 acl: "public-read",
 },
 { dependsOn: [ownershipControls, publicAccess] },
 );
 }),
 );

#...
import json
import os
import requests

# ...

# Fetch some redirects from a hypothetical CMS.
response = requests.get(f"{os.environ['CMS_ENDPOINT']}/redirects.json")
redirects = json.loads(response.text)

# Create an S3 website redirect for each one.
for i, redirect in enumerate(redirects):
 aws.s3.BucketObject(
 f"redirect-{i}",
 bucket=bucket.id,
 key=redirect["from"],
 website_redirect=redirect["to"],
 acl="public-read",
 )

Notice the example parses the response using the language’s built-in JSON support. If your CMS happened to return data in some other format, or required some additional credentials for authorization, no problem — your language and its package ecosystem are going to be there to help you.

Also notice the example reads the source URL directly from the environment. This is typically more cumbersome (if even possible) with other tools, requiring you to name your environment variables in tool-specific ways and then tack on additional blocks of configuration before you’re able to use them. As you can see, Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET makes this much more convenient.

Finally, notice the redirects are translated into cloud resources using a familiar JavaScript TypeScript Python Go C# Java Pulumi YAML F# VB looping construct. We’ll cover this topic in much more depth in an upcoming post. (Stay tuned!)

Hook the event loop to run tasks after deployment

Finally, you can do nifty things like run tasks programmatically after a Pulumi deployment completes — once again, without ever having to leave the Pulumi program.

If your hypothetical website were sitting behind a CloudFront CDN, for example, you might want some way to clear the CDN’s cache after a deployment. This generally requires some imperative action, like using the AWS CLI to run aws cloudfront create-invalidation sometime later. How might you do something like this with Pulumi, given Pulumi programs exit automatically when their operations complete?

One way — again, since a Pulumi program is just a regular Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET program — would be to write a JavaScript TypeScript Python Go C# Java Pulumi YAML F# VB function to create an invalidation request, and then ask the Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET runtime to run that function before exiting.

The mechanics of this vary slightly by language, but it works, and it’s a very handy way to keep all of your deployment logic in one place — again avoiding the need to relegate anything to a Bash script:

// ...
import * as cloudfront from "@aws-sdk/client-cloudfront";

const config = new pulumi.Config("aws");
const region = config.require("region");

// Create a CloudFront distribution for the website.
const cdn = new aws.cloudfront.Distribution("cdn", {
 enabled: true,
 defaultRootObject: "index.html",
 origins: [{
 originId: bucket.arn,
 domainName: bucket.websiteEndpoint,
 // ...
 }],
 // ...
});

function createInvalidation(id: string) {
 // Only invalidate after a deployment.
 if (pulumi.runtime.isDryRun()) {
 console.log("This is a Pulumi preview, so skipping cache invalidation.");
 return;
 }

 process.on("beforeExit", () => {
 const client = new cloudfront.CloudFrontClient({ region });
 const command = new cloudfront.CreateInvalidationCommand({
 DistributionId: id,
 InvalidationBatch: {
 CallerReference: `invalidation-${Date.now()}`,
 Paths: {
 Quantity: 1,
 Items: ["/*"],
 },
 },
 });

 client
 .send(command)
 .then(result => {
 console.log(`Invalidation status for ${id}: ${result.Invalidation?.Status}.`);
 process.exit(0);
 })
 .catch(error => {
 console.error(error);
 process.exit(1);
 });
 });
}

// Register a function to be invoked before the program exits.
cdn.id.apply(id => createInvalidation(id));

The relevant code is the call to process.on("beforeExit"), which registers a function to be invoked just before the program exits using the resolved ID of the distribution. (For more on how the apply() method works, see Inputs and Outputs.) The function returns early for Pulumi previews — no sense clearing the cache if the site hasn’t changed — and uses the AWS SDK for JavaScript to submit the invalidation request to CloudFront, naming it uniquely with a timestamp and logging the result to the console:

# ...
import boto3
import asyncio

def create_invalidation(id):
 # Don't bother invalidating unless it's an actual deployment.
 if pulumi.runtime.is_dry_run():
 print("This is a Pulumi preview, so skipping cache invalidation.")
 return

 client = boto3.client("cloudfront")
 result = client.create_invalidation(
 DistributionId=id,
 InvalidationBatch={
 "CallerReference": f"invalidation-{time.time()}",
 "Paths": {
 "Quantity": 1,
 "Items": ["/*"],
 },
 },
 )

 print(
 f"Cache invalidation for distribution {id}: {result['Invalidation']['Status']}."
 )


# Register a function to be invoked before the program exits.
cdn.id.apply(lambda id: atexit.register(lambda: create_invalidation(id)))

The relevant code is the call to atexit.register(), which registers a function to be invoked just before the program exits using the resolved ID of the distribution. (For more on how the apply() method works, see Inputs and Outputs.) The function returns early for Pulumi previews — no sense clearing the cache if the site hasn’t changed — and uses Boto3 to submit the invalidation request to CloudFront, naming it uniquely with a timestamp and logging the result to the console:

$ pulumi preview
Previewing update (dev)
Type Name Info
pulumi:pulumi:Stack example-dev 16 messages
Diagnostics:
pulumi:pulumi:Stack (example-dev):
...
This is a Pulumi preview, so skipping cache invalidation.

$ pulumi up
Updating (dev)
Type Name Status Info
pulumi:pulumi:Stack example-dev 16 messages
Diagnostics:
pulumi:pulumi:Stack (example-dev):
...
Cache invalidation of distribution EAJOVC1QNGT13: InProgress.

Techniques like these are a great way to eliminate unnecessary scripts, glue code, and other dependencies while keeping your deployment logic clear, well contained, and amenable to change.

Wrapping up

Hopefully this first post has given you a few new ways to think about how to do more with Pulumi and your language of choice. Over the next few weeks, as this series unfolds, we’ll share a lot more, so keep an eye on this space!

You’ll find the full source for the examples above on GitHub:

Happy coding!

Next Steps

If you haven’t already, install Pulumi today, and follow our self-directed Getting Started guides to learn more about making the most of Pulumi’s next-level infrastructure management features at your organization.

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

The Pulumi Cloud is a fully managed service that helps you adopt Pulumi’s open source SDK with ease. It provides built-in state and secrets management, integrates with source control and CI/CD, and offers a web console and API that make it easier to visualize and manage infrastructure. It is free for individual use, with features available for teams.

Create an Account

Local Testing With Pulumi

Aaron Kao — Fri, 28 Apr 2023 00:00:00 +0000

If you’ve been following along with our IaC Recommended Practices series, then you’re already familiar with Zephyr Archaeotech Emporium, the fictional company at the center of the series. Today, you’ll get an inside look at how Zephyr starts using Pulumi for locally testing the application code for their online store and accelerating the inner dev loop for their development team.

In the event you’re not familiar with the term “inner dev loop,” it’s used to describe the iterative code/build/test loop that every developer goes through to write code and verify that it works before they push their code to version control to be reviewed and merged. Speeding up the inner dev loop makes it possible for developers to iterate on their code more quickly, reducing the time spent waiting on tools or environments.

Structured as a conversation between two employees at Zephyr—Alice, a developer on the team for Zephyr’s online store application, and Bob, a member of Zephyr’s platform engineering team—this post shows one way to use Pulumi to help accelerate the inner dev loop. Let’s listen in…

Listening in at Zephyr HQ

In a conversation at Zephyr HQ between Bob and Alice one day, Bob commented, “I’m really enjoying Pulumi. It’s nice for the platform team to be able to use TypeScript to define all the infrastructure we need.”

“Or to deploy our application onto that infrastructure,” Alice responded. “I just wish there was a way to speed up testing changes to our code.”

“Do you mean testing the Pulumi code?”

“No, no, I was talking about our application code—the code for the online store.”

“Oh, what’s wrong with the current way it’s handled? Is there something we can do better?”

Alice paused. “Well, don’t get me wrong, the way we use per-developer stacks makes it super easy for me to deploy an entire copy of our environment. The problem is that…well, it takes a fair amount of time to create all the AWS stuff. Sometimes I just need something faster, just to test some quick changes to the code before deploying the app to my dev stack.”

Bob nodded thoughtfully. “I see. You know, Pulumi has a Docker provider; in fact, they recently released a new version of their Docker provider.”

Alice’s eyes lit up. “So, you mean I can automate my local Docker instance using Pulumi?”

“Yeah, why?”

Alice smiled. “I have an idea.”

Using Pulumi for the inner dev loop

A couple days later, Alice contacted Bob and said, “I want to show you what I’ve come up with.”

To see the code that Alice created for local testing of the Zephyr online store with Pulumi, visit the zephyr-app repository and look at the blog/inner-dev-loop branch.

When Bob met up with Alice, Alice showed him a TypeScript program she’d written with Pulumi’s Docker provider that automates building and deploying the Zephyr online store to a local Docker daemon.

“With this code,” Alice explains, “I can just run pulumi up and it will deploy the entire Zephyr online store locally, all in just a few minutes. Here, let me just run pulumi up real quick.”

“But why use this instead of Docker Compose?”

“Oh, there’s absolutely nothing wrong with Docker Compose; it’s a great tool!” Alice replied. “But this allows me to work with TypeScript, and to build logic into my Pulumi program. Like this feature here—I can specify whether I want to build a container, or use the released version of a container.”

“See, at the top I have the Pulumi program check for a configuration value that I set using pulumi config set. If the value isn’t set, it defaults to build.”

// Get configuration values
const config = new pulumi.Config();
const srcRepoPath = config.get("srcRepoPath") || "../..";
const assetsFlag = config.get("assetsFlag") || "build";
const cartsFlag = config.get("cartsFlag") || "build";
const catalogFlag = config.get("catalogFlag") || "build";
const checkoutFlag = config.get("checkoutFlag") || "build";
const ordersFlag = config.get("ordersFlag") || "build";

“There’s one for each service in the online store?” Bob asked.

“Right, except for the UI service, which is always built. This is so I can independently test changes to only a specific service. Farther down, the Pulumi code checks for this value and then either builds the container—pulling in whatever changes I’ve made—or uses an already released version of the container.”

// Build the assets image or pull remote image, depending on value of assetsFlag
var assetsImageRef: pulumi.Input<string>;
if (assetsFlag == "build") {
 const assetsImage = new docker.Image("assets-image", {
 build: {
 context: `${srcRepoPath}/src/assets`,
 dockerfile: `${srcRepoPath}/src/assets/Dockerfile`,
 platform: imagePlatform,
 },
 imageName: "zephyr-assets:latest",
 skipPush: true,
 }, { retainOnDelete: true });
 assetsImageRef = assetsImage.id;
} else {
 const assetsRegistryImage = docker.getRegistryImage({
 name: "public.ecr.aws/aws-containers/retail-store-sample-assets:0.2.0",
 });
 const assetsRemoteImage = new docker.RemoteImage("assets-image", {
 name: assetsRegistryImage.then(assetsRegistryImage => assetsRegistryImage.name),
 pullTriggers: [assetsRegistryImage.then(assetsRegistryImage => assetsRegistryImage.sha256Digest)],
 }, { retainOnDelete: true });
 assetsImageRef = assetsRemoteImage.repoDigest;
};

You can see that the released container image is hard-coded in this example. You could also pass this in as a configuration value to allow the Pulumi program to more easily account for new releases.

“By setting the flag for the assets service to build and all the other services to something else—the code only builds if the value is set to build—then I can isolate changes made to that service for testing, to be sure I haven’t broken something elsewhere.”

“That seems useful.”

“Indeed! And you can see here that we are building a Docker network for these containers; this is what enables connectivity between the various parts of the online store when it’s running locally via Docker.”

// Create a network
const network = new docker.Network("network", {
 name: "zephyr-net",
});

// Create an assets container
const assetsContainer = new docker.Container("assets-container", {
 capabilities: {
 drops: ["ALL"],
 },
 envs: [
 "PORT=8080",
 ],
 hostname: "assets",
 image: assetsImageRef,
 memory: 64,
 name: "assets",
 networksAdvanced: [
 {
 name: network.name,
 },
 ],
 restart: "always",
});

Bob nodded. “Yep, this all makes sense. Interesting, I hadn’t really considered Pulumi for this particular use case. Usually when folks mention Pulumi the first thought in my head is provisioning cloud infrastructure.”

“Oh, certainly, and it works really well for that—but it also works really well for this case, too.”

“And since we are already using Pulumi for deploying to Kubernetes in test and production, then this brings some consistency across environments. It’s just a pulumi up regardless of whether you’re running the app locally or publishing to a Kubernetes cluster in your per-developer stack, the test stack, or even the prod stack.”

“Exactly. Oh, look here. It’s done. Looks like it took just over two minutes, and now our online store app is available at this URL on localhost.”

“That’s pretty fast.”

Alice nodded. “Yeah, a lot faster than waiting on a Kubernetes cluster to provision. We still need to run this in our dev stacks to be sure that it runs in an environment that closely mirrors production, but for quick feedback on changes I make this is very useful for me as a developer.”

“So if you make a change to the code and want to see the results, what’s that look like?”

“Here, let’s do it real quick.” Alice switches to her code editor, makes a few changes, and then flips back to the terminal. “I’ll run a pulumi preview so you can see what it’s going to do.”

Alice pointed to the terminal output. “Here, Pulumi has recognized that the source code for this container has changed, and so it is replacing the container image, and restarting all the containers. Let me run pulumi up.”

Bob watches as the operation completes in about 17 seconds.

“Wow, that’s really fast!”

“Exactly, Bob. That’s what makes using Pulumi for this use case so helpful. While it doesn’t eliminate the need for our per-developer stacks, it gives developers a way to quickly see changes we’re making to the source code in very little time. When we are satisfied that the changes are working as we expect, then we can deploy to our dev stack to be sure that the changes are ready to work their way to production.”

Speed up your inner dev loop

Use Pulumi and the Docker provider to build and deploy your application locally, then promote the same program to your cloud stacks. Get started in Pulumi Cloud.

Get started

Try this out yourself

Now that you’ve seen how Alice uses Pulumi to streamline testing the Zephyr online store’s application code locally, feel free to try this out yourself! The code that you saw in this blog post is available in the zephyr-app repository on GitHub. Just select the blog/inner-dev-loop branch and go to the develop/pulumi folder.

IaC Best Practices: Structuring Pulumi Projects

Christian Nunciato — Fri, 17 Mar 2023 00:00:00 +0000

This is the third post in a series of blog posts focused on Zephyr Archaeotech Emporium—our fictional company—and their use of Pulumi to manage their online retail store. In the first post, you saw how Zephyr initially decided to go with a single Pulumi project for managing deployments of their online retail store application. In this post, you’ll see how Zephyr’s use of Pulumi changes as their company grows and evolves.

The first post about organizing code and stacks specifically mentions that the series will show “how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.” While the ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application, it’s important to note that many best practices are “point in time” recommendations: best practices are context-dependent and the recommended practices for a given organization will change as the company and its infrastructure needs also change. This blog post is an example of that evolution in action: the practices that worked well for Zephyr while they were smaller and had smaller infrastructure and less teams working on that infrastructure are no longer ideal as the company has grown.

Growing Development Teams

As a result of launching its online presence, Zephyr saw amazing success. The market for arcane artifacts and novel curiosities is booming, and Zephyr is becoming a leader in this market. Funded by this success, Zephyr has grown rapidly. Along with this growth, Zephyr’s software engineering and IT teams also grew and expanded. Fortunately, Zephyr’s use of per-developer stacks with short-lived ephemeral Git branches positioned the development teams well for this growth.

For a look at the state of Zephyr’s Pulumi code as it stood as of the end of the second post in the series, see the blog/dev-stacks branch of the pulumi/zephyr-app repository on GitHub.

Naturally this growth has resulted in some separation of duties. The IT team now has responsibility for the infrastructure that supports the online retail store. Within that team, some team members are focused on “core” infrastructure, while other team members are focused only on Kubernetes (a natural decision given Zephyr’s use of Kubernetes in their architecture). The application team continues to remain responsible for the online store application itself.

This left Zephyr again asking some questions about the best path forward:

Should they use a single project, or multiple projects? Zephyr had used a single Pulumi project up until now, but growth and organizational change meant they wanted to make sure they were using Pulumi in a way that best reflected their needs. What are the advantages and disadvantages of each approach?
If multiple projects, how many projects? What’s the best way to determine how many projects are needed? What factors play a role in this decision?

Structuring Pulumi Projects

One project, or multiple projects? And if multiple projects, how many projects? These are the core questions the Zephyr needed to resolve. In the end, there are a number of key factors that determine the answers to these questions:

Use case: Zephyr has only a single application (so far!), but in situations where there are multiple applications then keeping those applications (and sometimes their associated infrastructure, depending on other factors) in separate Pulumi projects is generally recommended. In some ways, “use case” is an aggregation of other factors—a new application with a distinct set of resources that do not share a lifecycle with existing resources and are potentially managed by a different team.
Company/department/team structure: How a company/department/team is organized should certainly be taken into account when structuring your Pulumi projects. In Zephyr’s early days with Pulumi, a single team managed everything so a single project was fine. Now with multiple teams, Zephyr will likely find using multiple projects to be more effective.
Security: Closely related to the previous bullet, security is a factor that affects how projects and stacks should be structured. In Zephyr’s case, the ability for one team to affect another team’s resources was present when everything was a single project. With multiple projects, Zephyr can use stack permissions and role-based access control (RBAC) in Pulumi Cloud to appropriately control who can affect what. (The IaC Best Practices blog post on RBAC and security discusses this topic in more detail.)
The relationship between resources: The relationships between resources—such as which resources are dependent upon other resources—is another factor that feeds into your Pulumi project structure. A “shared infrastructure layer” is often best handled in a separate project of its own, but may pull in resources that might otherwise have been kept with an application in its own project. (In some respects, this is a “counter-balance” to the first bullet point.) On the other hand, resources that belong exclusively to a specific application or function might be best bundled in the same Pulumi project.
Resource lifecycle: Do resources have a shared lifecycle, meaning they are always created, updated, or deleted together? If so, then those resources should be placed in the same project. If the resources do have a shared lifecycle, then placing them in the same project simplifies lifecycle management. A great example of resources with a shared lifecycle would be a Kubernetes Deployment and the associated Kubernetes Service.
Change rate of resources: Some resources may change very infrequently. Consider the example of an AWS VPC and associated objects. Once it is created, it may not change for extended periods of time, if ever. On the other hand, some resources may change frequently (in other words, these resources have a high change rate). A fast-moving software development team may revise the Kubernetes Deployment regularly to pull in a new container image with updated code. In a single project, combining resources that have a high change rate with resources that have a low change rate may be suboptimal.
Blast radius: Finally, there is the consideration of what happens in the event of human error, and limiting the “blast radius.” With a single project, all the resources in that project are within the blast radius—all of them could potentially be affected in the event of human error. Using separate Pulumi projects can limit the blast radius. It’s not something that should drive the decision all by itself, but it is a factor to be considered.

Now that you have an idea of some of the factors that affect how users structure their Pulumi projects, what are the advantages and disadvantages of each approach?

For single projects, the key advantage is simplicity. There’s no denying that a single project is simpler and less complex than working with multiple projects. This advantage, though, is also the key disadvantage. A single project does not afford the flexibility to let different teams manage different aspects of the infrastructure, for example, nor does it offer the ability to split resources that don’t have a shared lifecycle. And, as mentioned earlier, blast radius is a concern.

For multiple projects, the key advantage is flexibility. It’s true that the added flexibility comes at the cost of some added complexity (a future post will explore one aspect of that added complexity in the form of StackReferences). However, you gain the ability to separate infrastructure resources according to team ownership/responsibility, relationship to other resources (i.e., grouping resources in a “shared infrastructure layer”), or any of the other dimensions mentioned above.

In some ways, the decision points for single vs. multiple projects are analogous to the decision points for monolith vs. microservice: independent scaling and flexibility versus simplicity.

In general, most customers end up using multiple Pulumi projects per use case. As you’ve seen, though, there are a number of factors that influence that decision and will affect how many projects end up in use.

Moving Forward with Projects

Zephyr had a number of factors in play, not the least of which was the organizational change resulting from their growth and success. In the end, Zephyr decided to restructure their use of Pulumi for the online retail store into three separate projects:

An infrastructure project: This project handled the AWS VPC and related infrastructure resources.
A platform project: This project managed the Kubernetes cluster(s).
An application project: This project handled the actual online retail store itself.

You can use a combination of code refactoring and pulumi import to assist in the process of restructuring Pulumi projects, but it some cases it may be easier to create new infrastructure and migrate over to it.

In the code organization and stacks post, you read about some of the options for where to store Pulumi code relative to the applications it supports. At that time, Zephyr had decided to go with what’s called a monorepo approach, meaning that the Pulumi code was stored in the same repository as the application code. In conjunction with the project reorganization, Zephyr also decided to change how their code was organized:

Code for the infrastructure project moved to a separate “zephyr-infra” repository
A new “zephyr-k8s” repository housed the code for the platform project
The code for the online retail store application itself remained in the same “zephyr-app” repository

Note that relocating code (to a different GitHub repository or different filesystem location within a repository) generally has no impact on Pulumi. As long as the project name and stack name do not change, then changing the filesystem location won’t affect anything—keeping in mind, of course, that filesystem paths and such may need to be adjusted in code to account for any such changes.

After restructuring their projects, Zephyr’s arrangement of Pulumi projects and stacks, along with their associated GitHub repositories, looks something like this:

These repositories are available for you to review on GitHub (zephyr-infra, zephyr-k8s, and zephyr-app); use the branch/tag selector to find the blog/multi-project branch in each repository to see the state of the code as of this blog post. Be aware that the process for standing up the Zephyr online store is now a bit more complex; check the instructions in each repository for more details if you’d like to give these projects a spin.

Summarizing Best Practices: Pulumi Project Structures

This post covered the following recommended practices for working with Pulumi:

Use multiple Pulumi projects when you need the added flexibility that multiple Pulumi projects offer.
Align your Pulumi projects according to external factors like company/department/team structure, resource dependencies, lifecycle, and use case/application.
Change the filesystem location for Pulumi code as needed without impacting the resources managed by that code, as long as the Pulumi project name and stack name don’t change.

The next post will focus on some specific technical details related to Zephyr’s project reorganization; specifically, how they used StackReferences to share information between their Pulumi projects. Stay tuned!

IaC Best Practices: Enabling Developer Stacks & Git Branches

Christian Nunciato — Fri, 10 Mar 2023 00:00:00 +0000

In the first post about code organization and stacks, we introduced Zephyr, a fictional company that uses Pulumi to manage its online retail store. Following on from that post, which discusses code organization and stacks, this post explores two more questions users frequently ask when working with Pulumi in teams — namely, How can I best enable multiple developers to collaborate on a Pulumi project? And how can I use Git and Git branching to support this kind of collaboration? In this post, we’ll provide some guidance and best practices around these topics, using Zephyr and its online store as the use case.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series — not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and its retail application changes to accommodate its growth.

Existing IaC Workflow

When we last met up with the Zephyr team, they were off and running, managing their newly refactored online store, Zephyr Archaeotech Emporium, with a single Pulumi project and two Pulumi stacks — one for development (dev) and another for production (prod). The team had chosen to use one Git repository (a monorepo) to manage the code for the online store and its infrastructure after refactoring the store into a set of containerized microservices deployed with Kubernetes on Amazon EKS.

For a snapshot of the code as it was at the end of the first post in the series, see the blog/original branch of the pulumi/zepyhr-app repository on GitHub.

Up to now, the two-stack approach has worked well for the team. It’s allowed them to focus on building new features, integrating those features into the shared dev environment by developing and testing locally (which we’ll cover in a future post) and merging their feature branches regularly into the base branch, main, using standard GitHub pull-request workflows. Around once a day, as the team’s changes are merged into main, the team lead (and original author of the Pulumi program) deploys them into the dev environment with Pulumi with the following commands:

git pull origin main
cd infra
pulumi stack select zephyr/dev
pulumi up

This builds new container images for any services whose code has changed since the last stack update, pushes those images to Amazon ECR, and updates the EKS cluster to deploy the services automatically.

Then, around once a week, as new features are ready for customers, the team lead deploys the main branch of the repository into production similarly:

git pull origin main
cd infra
pulumi stack select zephyr/prod
pulumi up

The deployment process is still a bit more manual than the team would like — but since updates typically take only a few minutes, the team lead is generally satisfied with the status quo.

A few more details worth noting out about Team Zephyr’s current setup:

Both of the project’s stack-configuration files, Pulumi.dev.yaml and Pulumi.prod.yaml, are checked into version control. This is safe and considered good practice (even with secrets) because it allows the team to review or change the configuration of a given stack easily by viewing or editing the code in each file.
The main branch is the only long-lived branch in the repository; all others (e.g., developers’ feature branches) are deleted once they’re merged into main.
Thanks to this workflow, the application deployed into the dev environment tracks closely with the latest code in the main branch. As such, the dev stack serves as a kind of pre-prod or integration environment that not only allows the team to dogfood new features as they’re added, but also to catch bugs or other regressions before they escape into production. When things do go wrong, the team can review the list of recent stack updates to identify the source of the defect and get it fixed.
Zephyr is using the Pulumi Service backend with a single organization (that everyone on the team belongs to) and separate AWS accounts for the dev and prod environments. Everyone on the team has read/write access to the AWS account used for development, but only the team lead has access to the AWS account used for production.

Growing Pains

Life for Team Zephyr was good — for a while.

Then one day during a particularly active sprint, a developer who’d started working on an improvement to the checkout process realized it might benefit from an updated version of Redis, which appeared to be presently pinned at 6.0 in the Pulumi program:

// ...
const checkoutRedisDeployment = new k8s.apps.v1.Deployment("checkout-redis-deployment", {
 spec: {
 template: {
 spec: {
 containers: [{
 image: "redis:6.0-alpine",
 // ...

A quick glance at the dev and prod stacks in the Pulumi Service confirmed this was the case.

After a quick spike to validate the enhancement locally, the developer felt confident that a Redis upgrade was the way to go — but since the deployed version of Redis would ultimately dictate the code that’d have to be written to finish the feature for real, they couldn’t really be sure without testing the theory with a full deployment. What they really wanted was to be able to deploy a Redis upgrade, along with the associated code changes, into an AWS environment as much like production as possible — the dev environment, say — without having to go through the process of making a pull request, getting it merged, and waiting for the team lead to deploy the dev stack only to realize later that the change didn’t actually work as expected, and be forced to revert.

In situations like these, it can be tempting to suggest that the developer simply pick up and use the dev stack — especially on small, fast-moving teams like Zephyr’s where everyone already has access to the Pulumi stack and the shared cloud environment. Indeed, Pulumi makes this sort of thing easy: to deploy their local changes into the dev environment, our developer need only check out their feature branch, select the dev stack, run pulumi up, and then wait a few moments for the changes to be deployed:

cd infra
pulumi stack select zephyr/dev
pulumi up

This approach definitely works — but it presents at least a few problems:

The team would lose all of the practical benefits of having the dev stack track closely with the latest content in main.
The team lead’s deployment of main into dev later that day would overwrite the developer’s own feature-branch deployment. There’s a good chance the lead would notice the diff of the deployment preview and deduce that someone else on the team had possibly deployed into dev out of band:
```
 kubernetes:apps/v1:Deployment (create):
 spec: {
 template: {
 spec: {
 containers: [
 [0]: {
- image: "redis:6.0-alpine"
+ image: "redis:7.0-alpine"
 ...
 }
 ]
 }
 }
 }
 ...
```
But if they didn’t, the developer’s ad-hoc deployment would be lost. At best, it’d force both team members to drop what they were doing and get in touch to figure out what to do next. At worst, it could lead to confusion and the potential for shipping untested code.
In addition to raising the risk of developers overwriting each other’s work, active collaboration the same stack increases the likelihood of deployment conflicts. On the one hand, this is a good thing in that Pulumi prevents same-stack updates from running concurrently:
```
$ pulumi up
error: [409] Conflict: Another update is currently in progress.
```
But on the other, it’s a not-so-good thing in that it’s a reduction in development velocity: The update that can’t run because another update is in progress is an update that will have to wait until sometime later. The more often this happens, the longer it ultimately takes for the team to get the validation it needs to keep moving forward.

So facing these issues, what should the Zephyr team do? The dev stack had originally been created for rapid development — and that worked well, when there was only one developer working on infrastructure. With several, however, things clearly get complicated. Should the team be using a dev stack at all? If so, how? And if not, what should they be doing instead?

Stepping Back to Reassess

With what we’ve learned so far, we can return to the questions we raised at the beginning of this post and begin to sketch out a few recommendations for Team Zephyr. Those questions, again, were:

How can I best enable multiple developers to collaborate on a Pulumi project? For Zephyr, an ideal answer to this question would give every member of the team the ability to iterate on both application and infrastructure code concurrently, deploying as often as needed without having to worry about disrupting the work or the velocity of anyone else on the team.
How can I use Git and Git branching to support this collaboration? So far, the feature-branch/pull-request workflow has been serving the Zephyr team well. Should they continue on the same path, merging frequently to main and deploying to dev and prod from there? Or is there some alternative workflow that might serve the team better, either now or at some time in the future?

Let’s answer these questions now by taking a look at how the Zephyr team handled them.

Using Developer Stacks

We’ve seen a few ways that having multiple developers share a single stack for ad-hoc deployment can lead to various problems. But we’ve also seen at least one situation in which being able to run ad-hoc deployments — without the risk of colliding with existing environments — can be quite valuable. In fact there are many reasons you might want to do this — and fortunately Pulumi makes it quite easy to do so.

The best practice is to allow developers to create and manage their own Pulumi stacks. In this workflow, developers working on feature branches are able to iterate freely on both application and infrastructure, deploying as often as necessary into a shared cloud environment (typically a sandbox environment, isolated from production) and destroying cloud infrastructure when it’s no longer needed.

For the developer, the typical workflow looks something like this:

Check out a new feature branch
Make changes to application and/or infrastructure code
Select your personal developer stack with pulumi stack select
Deploy your stack into your team’s shared cloud environment with pulumi up
Commit your code and submit a pull request, optionally including a link either to the Pulumi update (e.g., when using the Pulumi Service) or to certain key resources belonging to the stack — for example, an application’s primary HTTP endpoint

The benefits of a workflow like this one are many:

No more resource-naming collisions. Because Pulumi names cloud resources uniquely by default, no two stacks will ever produce cloud resources with the same names.
No more deployment conflicts. When every developer has a stack of their own, they can deploy to the sandboxed environment as often as necessary, without having to wait for another deployment to finish first.
Everyone on the team has their own production-like environment to use for end-to-end testing.

For Zephyr, the quickest path to adopting this workflow was to use the dev stack as a foundation for new developer stacks. To do that, team members simply ran the following Pulumi commands:

pulumi stack init zephyr/developer-name --copy-config-from zephyr/dev
pulumi up

Developer stacks are considered ephemeral, so they’re torn down with pulumi destroy when not in use in order to cut down on cost. Stack configuration files (e.g., Pulumi.developer-name.yaml) are checked into version control and stored alongside other stacks like Pulumi.dev.yaml and Pulumi.prod.yaml.

Keeping Feature-Branch Workflows

We’ve also seen that the Zephyr team’s existing Git workflow (short-lived feature branches merged frequently into a long-lived base branch) has worked well for them — and it turns out this workflow also works well when combined with the use of developer stacks.

For example, after creating and deploying a new stack to validate and test the proposed Redis upgrade, our Zephyr developer simply included a link to the application (deployed with their dev stack) in the associated GitHub pull request. Reviewers were then able to follow the link to the online store to see the effects of the shopping-cart change firsthand, which meant they were able to approve and get the change merged quickly and confidently knowing it’d work when deployed into production.

Naming Stacks Clearly

Lastly, having adopted the use of developer stacks, the Zephyr team realized the name “dev” probably didn’t work quite as well as it once had for the dev stack. Since the stack and its long-running deployment had essentially been serving (and continued to serve) as an environment for testing changes before releasing them into production, the team decided to rename the dev stack test instead. To do that, they simply ran:

pulumi stack select zephyr/dev
pulumi stack rename zephyr/test

Viewing the Next Iteration of Code

You can view the second iteration of Zephyr’s Pulumi and application code — the iteration that corresponds to the decisions described in this blog post — by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to the blog/dev-stacks branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Developer Stacks and Git Branches

In this post, we covered several new guidelines for working collaboratively with Pulumi, all aimed at optimizing for high team and developer velocity:

Use short-lived feature branches and merge them frequently into a single base branch.
Deploy the base branch regularly into a long-running pre-prod environment to continuously test the integration of the components of the system.
Enable developers to create and use dev stacks of their own as sandboxes for development, testing, and collaboration.
Name your stacks clearly and in alignment with the environments they deploy into — test, staging, production, etc.

The next Zephyr blog post will dive into how the Zephyr team structures their Pulumi projects. Stay tuned!

IaC Best Practices: Understanding Code Organization & Stacks

Aaron Kao — Mon, 20 Feb 2023 00:00:00 +0000

This is the first in a series of blog posts that explores how a fictional company—Zephyr Archaeotech Emporium—uses Pulumi to manage their online retail store. This post explores a couple of common questions that users ask when working with Pulumi; specifically, where should I store my Pulumi code? And how do I support multiple environments with Pulumi? This post will provide some guidance and Infrastructure as Code best practices around these topics, using Zephyr and their online store as the use case.

The ultimate goal of this series is to discuss recommended practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series—not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.

Setting Up the Scenario

Zephyr is short for Zephyr Archaeotech Emporium, the fictional company in our scenario. Zephyr is an online retailer that specializes in the sale of “rare arcane artifacts and replicas.” Over the past few years, the company has experienced a significant increase in its online presence, making it easier for customers to purchase unique and mysterious items. Zephyr’s collection includes a variety of rare and unusual objects that are difficult to find elsewhere, making it a popular destination for collectors, enthusiasts, and adventurers.

Moving to a Containerized Architecture

Zephyr’s application is the second generation of their online store. Their first generation was a monolith that they manually deployed. It was sufficient in Zephyr’s early days, but as word about Zephyr spread and the volume of traffic to their online store increased, Zephyr found it increasingly difficult to deploy small, incremental changes to the store’s code. As a result, the velocity of their development efforts slowed. In preparation for the next phase of their growth—driven by robust sales of arcane artifacts—Zephyr’s team evaluated a number of different architectures that would enable them to increase development velocity and give them greater flexibility in scaling different aspects of their online store. In the end, they settled on a containerized architecture deployed to Kubernetes because some of the existing team was already familiar with these technologies.

Here’s a diagram of their second-generation application architecture:

As you can see in the diagram, the application has been broken down into microservices. The UI service communicates with other backend services via HTTP APIs; the various backend services communicate with various databases like MySQL and Redis for data persistence and/or caching. The backend services are responsible for implementing the functionality of a typical e-commerce site like orders, checking out, and managing the inventory catalog. While the choice of Kubernetes does afford Zephyr some much-needed functionality—like the ability to easily scale different services within the application—it also brings with it an added level of complexity that did not exist with their previous monolithic architecture. The Kubernetes manifest to deploy their application tops out at just over 1,000 lines of YAML!

Astute readers may note that Zephyr’s application looks very similar to this application; in truth, this series uses a fork of the AWS Containers retail store example as the application in this fictional scenario. (Many thanks to the AWS Containers DA team for their hard work here!) This is an admittedly over-engineered application, but it does afford the opportunity to address a range of customer scenarios in this blog series.

Managing Infrastructure and Application Deployments

As part of the switch to their new architecture, Zephyr decided they wanted to use Pulumi to manage both the infrastructure and the application deployment. Why Pulumi? Zephyr’s team recognized that moving to microservices on Kubernetes was going to make it more difficult to create reproducible, consistent implementations of their application. Zephyr’s team also knew that adopting infrastructure as code would help them with fast and repeatable deployments with little additional complexity. Another bonus was being able to use programming languages they already knew.

As they prepared to embark on using Pulumi, a couple of questions came up for the Zephyr team:

Where should they store their Pulumi code? The ability for Zephyr’s developers to use a full programming language to manage the infrastructure and deployment of their online store is a huge plus, but should this code go in the same repository as their application code? Or a different repository?
How do they address the need for multiple environments? It’s clear to the Zephyr development team that they’ll need multiple instances of the online store (for things like testing, QA, or development). What’s the best way to handle this when using Pulumi?

Let’s examine each of these issues in a bit more detail, and see how Zephyr chose to proceed.

Storing Pulumi Code

When it comes to answering the question of where to store Pulumi code relative to the application(s) it supports, there are two basic options:

In the same repository as your application code: This approach has the advantage of being simpler, but may not offer the necessary flexibility when dealing with multiple teams or varying levels of access control.
In a separate repository from your application code: Using a separate repository adds some complexity, but does provide greater flexibility to more easily address a variety of organizational requirements and varying levels of access control.

So which approach is best? That will depend on a number of different factors, many of which are outlined in the organizing projects & stacks documentation. Some other factors that users need to take into consideration include:

Who is responsible for maintaining the code? If the application code and the Pulumi code are managed by different teams, then using separate repositories may be the best approach.
What sort of access controls are needed? If the organization needs different access controls for the application code and the Pulumi code, then separate repositories are generally needed.
What is the relationship between the infrastructure resources managed by Pulumi and the applications? If the infrastructure is “shared,” meaning it is used by multiple applications, then keeping the Pulumi code in a separate repository may be preferred. If the infrastructure is dedicated to a specific application, then keeping the application code and the Pulumi code together may be preferred.
What about CI/CD? If an organization will use a CI/CD pipeline, that may affect how you organize your code. (This is something we’ll delve into more deeply later in this series.)

It’s worth stressing that the answers to these questions may change over time as organizations grow and their applications evolve. Addressing that change and its effects on an organization’s Pulumi projects and stacks is something we’ll be discussing later in this series.

The discussion above is working from the assumption that Pulumi users will store their Pulumi code in a version control system, like Git. Strictly speaking, using a version control system isn’t required for Pulumi to function (there is one exception we’ll touch on later in this series when we discuss Pulumi Deployments), but we very strongly recommend that all users use Pulumi in conjunction with version control.

In the case of Zephyr, their team felt like a monorepo approach (storing Pulumi code and application code together) was most appropriate for their specific requirements. Zephyr is a relatively small organization with a single team of developers who are responsible for managing both the cloud infrastructure as well as the development and deployment of the online store application. Zephyr’s team also decided to use a single Pulumi project—for now. (Be sure to follow this blog series to see how that evolves, and why!)

Addressing the Need for Multiple Environments

The second question Zephyr encountered is how to handle the need for multiple instances of their application. In addition to a production instance—which is the instance behind Zephyr’s public-facing online store—Zephyr also felt they needed an environment for the developers to use in testing changes to the online store.

This use case—needing to have multiple, separate instances of the infrastructure and applications created by a single Pulumi program—is exactly what Pulumi stacks were designed to address. Each stack is a separate instance of the resources created by a Pulumi program within a project. Further, each stack has its own independent state, and each stack has its own configuration values. Stacks can be short-lived (meaning the associated resources are also short-lived), or stacks can be long-lived. Aside from the cloud resources created by the Pulumi program, stacks are lightweight and simple to create or delete.

Zephyr decided to initially start with two stacks: a production stack (named “prod”) and a development stack (named “dev”). As you’ll observe throughout this series, this is a decision that is easily adjusted over time as your organization’s requirements change.

Organize your infrastructure with stacks

Model separate development, staging, and production environments as Pulumi stacks in a single project, each with its own state and configuration.

Get started

Viewing the First Iteration of Code

You can view the first iteration of Zephyr’s Pulumi and application code—the iteration that corresponds to the decisions described in this blog post—by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to viewing the blog/original branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Organizing Code and Stacks

This article introduced you to Zephyr, the fictional company for this blog series, and reviewed the application architecture for Zephyr’s online store (a containerized application running on Kubernetes). Throughout the blog series, this application serves as the use case around which various recommended practices are discussed.

In this post the following guidelines were covered:

Use version control for your Pulumi code. Although it isn’t strictly required in many cases, a number of the benefits of adopting infrastructure as code are lost without the use of version control.
Place your Pulumi code into version control repositories in a way that aligns with organizational requirements. There is no “right” or “wrong” approach; choose the approach that enables you to satisfy your organization’s requirements around access control, code/repo ownership, and other organizational requirements.
Leverage Pulumi stacks to create multiple instances of cloud resources with a single Pulumi program. Pulumi stacks were created to address the need for organizations to create multiple instances of cloud resources, each with its own state and its own set of configuration data, from a single Pulumi program.

For now, Zephyr has decided to go with a monorepo approach—a single repository that contains both their application code and the Pulumi code to manage the infrastructure resources. All of the resources are defined in a single Pulumi project, with multiple stacks that correspond to development and production environments. Over the course of this series, we’ll see how Zephyr’s use of Pulumi changes as Zephyr grows and their application evolves.

The next Zephyr blog post will examine how the Zephyr team adjusts their use of stacks and Git branches to support developer productivity. Stay tuned!

How To Build An ETL Pipeline With Amazon Redshift & AWS Glue

Christian Nunciato — Fri, 23 Dec 2022 00:00:00 +0000

In our last episode, Deploying a Data Warehouse with Pulumi and Amazon Redshift, we covered using Pulumi to load unstructured data from Amazon S3 into an Amazon Redshift cluster. That went well, but you may recall that at the end of that post, we were left with a few unanswered questions:

How do we avoid importing and processing the same data twice?
How can we transform the data during the ingestion process?
What are our options for loading data automatically — for example, on a regular schedule?

These are the kinds of questions you’ll almost always have when setting up a data-processing (or ETL) pipeline — and every platform tends to answer them a little differently.

When your platform of choice is Amazon Redshift, those questions will often be answered by pointing you to another Amazon service (actually a collection of services) called AWS Glue. With Glue, you can define processes that monitor external data sources like S3, keep track of the data that’s already been processed (to avoid corruption from duplicate records, as we saw in the previous post), and write code in general-purpose programming languages like Python to process and transform the data on its way into Redshift.

There’s a lot more you can do with AWS Glue than what we need for this project, but as you’ll see, it’s also an excellent fit. So without further ado, let’s finish things off by using AWS Glue (and Pulumi of course) to set up a full ETL pipeline that loads all of our S3 data into Redshift automatically.

But first …

Set Up Redshift & Glue Integration

If you haven’t already, I’d encourage you to read through the previous post to get up to speed on what we’re building and why. In that post, the situation was such that some hypothetical application was generating “events” — little bits of JSON, essentially — and writing them periodically to a text file in S3, and as data scientists, we needed some way to gather up all of these events and load them into a Redshift cluster in order to analyze them later.

When we left off, we’d gotten Redshift up and running, and we were able to pull the data from S3 into Redshift directly (by running a manual query in the Redshift console), but that’s as far as we got — no automation, no protection from duplicate records, just the absolute basics.

So to pick up from there, let’s start by bootstrapping a new project with the code from the previous post. As always, make sure you’ve installed Pulumi and configured your AWS credentials in the usual way first, then run the following commands to get going:

$ mkdir my-data-warehouse-part2 && cd my-data-warehouse-part2
$ pulumi new aws-typescript

$ mkdir my-data-warehouse-part2 && cd my-data-warehouse-part2
$ pulumi new aws-python

Step through the prompts, choosing the AWS region that works best for you. When the new-project wizard completes, replace the contents of index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf with the following code to fast-forward to where we left off:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Import the stack's configuration settings.
const config = new pulumi.Config();
const clusterIdentifier = config.require("clusterIdentifier");
const clusterNodeType = config.require("clusterNodeType");
const clusterDBName = config.require("clusterDBName");
const clusterDBUsername = config.require("clusterDBUsername");
const clusterDBPassword = config.requireSecret("clusterDBPassword");

// Import the provider's configuration settings.
const providerConfig = new pulumi.Config("aws");
const awsRegion = providerConfig.require("region");

// Create an S3 bucket to store some raw data.
const eventsBucket = new aws.s3.Bucket("events", {
 forceDestroy: true,
});

// Create a VPC.
const vpc = new aws.ec2.Vpc("vpc", {
 cidrBlock: "10.0.0.0/16",
 enableDnsHostnames: true,
});

// Create a private subnet within the VPC.
const subnet = new aws.ec2.Subnet("subnet", {
 vpcId: vpc.id,
 cidrBlock: "10.0.1.0/24",
});

// Declare a Redshift subnet group with the subnet ID.
const subnetGroup = new aws.redshift.SubnetGroup("subnet-group", {
 subnetIds: [
 subnet.id,
 ],
});

// Create an IAM role granting Redshift read-only access to S3.
const redshiftRole = new aws.iam.Role("redshift-role", {
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "redshift.amazonaws.com",
 },
 },
 ],
 },
 managedPolicyArns: [
 aws.iam.ManagedPolicy.AmazonS3ReadOnlyAccess,
 ],
});

// Create a VPC endpoint so the cluster can read from S3 over the private network.
const vpcEndpoint = new aws.ec2.VpcEndpoint("s3-vpc-endpoint", {
 vpcId: vpc.id,
 serviceName: `com.amazonaws.${awsRegion}.s3`,
 routeTableIds: [
 vpc.mainRouteTableId,
 ],
});

// Create a single-node Redshift cluster in the VPC.
const cluster = new aws.redshift.Cluster("cluster", {
 clusterIdentifier: clusterIdentifier,
 databaseName: clusterDBName,
 masterUsername: clusterDBUsername,
 masterPassword: clusterDBPassword,
 nodeType: clusterNodeType,
 clusterSubnetGroupName: subnetGroup.name,
 clusterType: "single-node",
 publiclyAccessible: false,
 skipFinalSnapshot: true,
 vpcSecurityGroupIds: [
 vpc.defaultSecurityGroupId,
 ],
 iamRoles: [
 redshiftRole.arn,
 ],
});

import json
import pulumi
from pulumi_aws import ec2, iam, redshift, s3, glue

# Import the stack's configuration settings.
config = pulumi.Config()
cluster_identifier = config.require("clusterIdentifier")
cluster_node_type = config.require("clusterNodeType")
cluster_db_name = config.require("clusterDBName")
cluster_db_username = config.require("clusterDBUsername")
cluster_db_password = config.require_secret("clusterDBPassword")

# Import the provider's configuration settings.
provider_config = pulumi.Config("aws")
aws_region = provider_config.require("region")

# Create an S3 bucket to store some raw data.
events_bucket = s3.Bucket("events", s3.BucketArgs(
 force_destroy=True,
))

# Create a VPC.
vpc = ec2.Vpc("vpc", ec2.VpcArgs(
 cidr_block="10.0.0.0/16",
 enable_dns_hostnames=True,
))

# Create a private subnet within the VPC.
subnet = ec2.Subnet("subnet", ec2.SubnetArgs(
 vpc_id=vpc.id,
 cidr_block="10.0.1.0/24",
))

# Declare a Redshift subnet group with the subnet ID.
subnet_group = redshift.SubnetGroup("subnet-group", redshift.SubnetGroupArgs(
 subnet_ids=[
 subnet.id,
 ],
))

# Create an IAM role granting Redshift read-only access to S3.
redshift_role = iam.Role("redshift-role", iam.RoleArgs(
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "redshift.amazonaws.com",
 },
 },
 ],
 }),
 managed_policy_arns=[
 iam.ManagedPolicy.AMAZON_S3_READ_ONLY_ACCESS,
 ],
))

# Create a VPC endpoint so the cluster can read from S3 over the private network.
vpc_endpoint = ec2.VpcEndpoint("s3-vpc-endpoint", ec2.VpcEndpointArgs(
 vpc_id=vpc.id,
 service_name=f"com.amazonaws.{aws_region}.s3",
 route_table_ids=[
 vpc.main_route_table_id,
 ],
))

# Create a single-node Redshift cluster in the VPC.
cluster = redshift.Cluster("cluster", redshift.ClusterArgs(
 cluster_identifier=cluster_identifier,
 database_name=cluster_db_name,
 master_username=cluster_db_username,
 master_password=cluster_db_password,
 node_type=cluster_node_type,
 cluster_subnet_group_name=subnet_group.name,
 cluster_type="single-node",
 publicly_accessible=False,
 skip_final_snapshot=True,
 vpc_security_group_ids=[
 vpc.default_security_group_id,
 ],
 iam_roles=[
 redshift_role.arn,
 ],
))

At the top of the program, you’ll see that the code assumes you’ve set several Pulumi configuration values — clusterIdentifier, clusterNodeType, and others — so take a moment to do that now with the Pulumi CLI as well:

$ pulumi config set clusterIdentifier my-redshift-cluster
$ pulumi config set clusterNodeType ra3.xlplus
$ pulumi config set clusterDBName dev
$ pulumi config set clusterDBUsername admin
$ pulumi config set clusterDBPassword StrongPass1 --secret

That should be all you need to pick up where we left off. Deploy with an initial pulumi up to bring this project back to life:

$ pulumi up
...
Updating (dev)
Type Name Status
+ pulumi:pulumi:Stack my-data-warehouse-part2-dev created (5s)
+ ├─ aws:ec2:Vpc vpc created (13s)
+ ├─ aws:iam:Role redshift-role created (2s)
+ ├─ aws:s3:Bucket events created (3s)
+ ├─ aws:ec2:VpcEndpoint s3-vpc-endpoint created (6s)
+ ├─ aws:ec2:Subnet subnet created (1s)
+ ├─ aws:redshift:SubnetGroup subnet-group created (1s)
+ └─ aws:redshift:Cluster cluster created (105s)
Resources:
+ 8 created
Duration: 2m7s

When the deployment completes, you should see that eight new resources were created, including a VPC with a private subnet, a Redshift cluster deployed into the subnet, an S3 bucket that we’ll use to hold some raw data, and a VPC endpoint allowing direct access to the bucket over the private network.

Now it’s time to start adding the Glue components to — well, glue it all together.

Automating ETL Jobs in AWS Glue

At a high-level, we’ll need three components to complete our ETL pipeline-to-be:

A Glue crawler. The crawler is responsible for fetching data from some external source (for us, an S3 bucket) and importing it into a Glue catalog. A catalog is a kind of staging database that tracks your data through the ingestion process. Our particular crawler will poll our S3 bucket for new data and import that data into a catalog table.
A Glue job. The job is responsible for running an ETL script (e.g., on a schedule) to process the data imported by the crawler into the catalog.
A Glue script. The script is where the ETL magic happens. Ours will be written in Python and be responsible for extracting data from the catalog, applying some slight transformations to it (mostly just for demonstration purposes), and loading the transformed data into Redshift.

We’ll translate these high-level components into concrete Pulumi resources in the steps ahead.

Step 1: Add a Glue crawler

The first thing we’ll need is a unique name for our Glue catalog database. Rather than hard-code this name into our program code (which would be problematic for projects consisting of multiple stacks), let’s instead use Pulumi to set this catalog name stack-specifically:

$ pulumi config set glueDBName my-glue-db

Next, back in your editor, add the following code to read this new setting and declare the new AWS resources you’ll need to enable the crawler. The comments inline explain each one: there’s a glue.CatalogDatabase to house imported data, a glue.Crawler to poll our bucket every 15 minutes for new data, and an iam.Role granting the Glue service the appropriate set of privileges to read from S3 and write temporary data as needed:

// ...
const glueDBName = config.require("glueDBName");

// ...

// Define an AWS cron expression of "every 15 minutes".
// https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchevents-expressions.html
const every15minutes= "cron(0/15 * * * ? *)";

// Create a Glue catalog database.
const glueCatalogDB = new aws.glue.CatalogDatabase("glue-catalog-db", {
 name: glueDBName,
});

// Define an IAM role granting AWS Glue access to S3 and other Glue-required services.
const glueRole = new aws.iam.Role("glue-role", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "glue.amazonaws.com",
 },
 },
 ],
 }),
 managedPolicyArns: [
 aws.iam.ManagedPolicy.AmazonS3FullAccess,
 "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole",
 ],
});

// Create a Glue crawler to process the contents of the data bucket on a schedule.
// https://docs.aws.amazon.com/glue/latest/dg/monitor-data-warehouse-schedule.html
const glueCrawler = new aws.glue.Crawler("glue-crawler", {
 databaseName: glueCatalogDB.name,
 role: glueRole.arn,
 schedule: every15minutes,
 s3Targets: [
 {
 path: pulumi.interpolate`s3://${eventsBucket.bucket}`,
 }
 ],
});

# ...
glue_db_name = config.require("glueDBName")

# ...

# Define an AWS cron expression of "every 15 minutes".
# https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchevents-expressions.html
every_15_minutes = "cron(0/15 * * * ? *)"

# Create a Glue catalog database.
glue_catalog_db = glue.CatalogDatabase("glue-catalog-db", glue.CatalogDatabaseArgs(
 name=glue_db_name,
))

# Define an IAM role granting AWS Glue access to S3 and other Glue-required services.
glue_role = iam.Role("glue-role", iam.RoleArgs(
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "glue.amazonaws.com",
 },
 },
 ],
 }),
 managed_policy_arns=[
 iam.ManagedPolicy.AMAZON_S3_FULL_ACCESS,
 "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole",
 ],
))

# Create a Glue crawler to process the contents of the data bucket on a schedule.
# https://docs.aws.amazon.com/glue/latest/dg/monitor-data-warehouse-schedule.html
glue_crawler = glue.Crawler("glue-crawler", glue.CrawlerArgs(
 database_name=glue_catalog_db.name,
 role=glue_role.arn,
 schedule=every_15_minutes,
 s3_targets=[
 glue.CrawlerS3TargetArgs(
 path=events_bucket.bucket.apply(lambda name: f"s3://{name}")
 ),
 ],
))

Run a quick pulumi up to deploy these new additions:

$ pulumi up
...

Updating (dev)

 Type Name Status
 pulumi:pulumi:Stack my-data-warehouse-part2-dev
 + ├─ aws:iam:Role glue-role created (2s)
 + ├─ aws:glue:CatalogDatabase glue-catalog-db created (1s)
 + └─ aws:glue:Crawler glue-crawler created (10s)

Resources:
 + 3 created
 8 unchanged

Duration: 15s

Now let’s add the Glue job.

Step 2: Add a Glue job

The job we need is a fairly simple one, but it still requires a handful of AWS resources.

First, we’ll need a glue.Connection to allow Glue to connect to the Redshift database to load the data processed by our TBD Python script. We’ll also need the script itself, which Glue expects to be able to find in an S3 bucket, so we’ll need to define a Bucket and BucketObject for that as well. We’ll need a glue.Job resource to define the compute settings and other parameters with which to run our Python script (passing it various arguments such as which Glue catalog to read from and Redshift database to write to), and then finally, we’ll need a glue.Trigger to kick off the process every 15 minutes. It’s only a few resources, but there’s a fair bit of configuration baked into some of them, so I’d encourage you to spend a little time with the AWS Glue docs to gain a better understanding of what each option does and what else can be done with them.

Believe it or not, that’s all we’ll need for the Pulumi program itself. So to wrap up, add the following code to index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf , taking note of the last line, which exports the name of the S3 bucket to give us something to upload to:

// ...

// Create a Glue connection to the Redshift cluster.
const glueRedshiftConnection = new aws.glue.Connection("glue-redshift-connection", {
 connectionType: "JDBC",
 connectionProperties: {
 JDBC_CONNECTION_URL: pulumi.interpolate`jdbc:redshift://${cluster.endpoint}/${clusterDBName}`,
 USERNAME: clusterDBUsername,
 PASSWORD: clusterDBPassword,
 },
 physicalConnectionRequirements: {
 securityGroupIdLists: cluster.vpcSecurityGroupIds,
 availabilityZone: subnet.availabilityZone,
 subnetId: subnet.id,
 },
});

// Create an S3 bucket for Glue scripts and temporary storage.
const glueJobBucket = new aws.s3.Bucket("glue-job-bucket", {
 forceDestroy: true,
});

// Upload a Glue job script.
const glueJobScript = new aws.s3.BucketObject("glue-job.py", {
 bucket: glueJobBucket.id,
 source: new pulumi.asset.FileAsset("./glue-job.py"),
});

// Create a Glue job that runs our Python ETL script.
const glueJob = new aws.glue.Job("glue-job", {
 roleArn: glueRole.arn,
 glueVersion: "3.0",
 numberOfWorkers: 10,
 workerType: "G.1X",
 defaultArguments: {
 // Enabling job bookmarks helps you avoid loading duplicate data.
 // https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html
 "--job-bookmark-option": "job-bookmark-enable",

 "--ConnectionName": glueRedshiftConnection.name,
 "--GlueDBName": glueDBName,
 "--GlueDBTableName": eventsBucket.bucket.apply(name => name.replace('-', '_')),
 "--RedshiftDBName": clusterDBName,
 "--RedshiftDBTableName": "events",
 "--RedshiftRoleARN": redshiftRole.arn,
 "--TempDir": pulumi.interpolate`s3://${glueJobBucket.bucket}/glue-job-temp`,
 },
 connections: [
 glueRedshiftConnection.name,
 ],
 command: {
 scriptLocation: pulumi.interpolate`s3://${glueJobBucket.bucket}/glue-job.py`,
 pythonVersion: "3",
 },
});

// Create a Glue trigger to run the job every 15 minutes.
const glueJobTrigger = new aws.glue.Trigger("trigger", {
 schedule: every15minutes,
 type: "SCHEDULED",
 actions: [{
 jobName: glueJob.name,
 }],
});

// Export the name of the data bucket.
export const dataBucketName = eventsBucket.bucket;

# ...

# Create a Glue connection to the Redshift cluster.
glue_redshift_connection = glue.Connection("glue-redshift-connection", glue.ConnectionArgs(
 connection_type="JDBC",
 connection_properties={
 "JDBC_CONNECTION_URL": cluster.endpoint.apply(lambda endpoint: f"jdbc:redshift://{endpoint}/{cluster_db_name}"),
 "USERNAME": cluster_db_username,
 "PASSWORD": cluster_db_password,
 },
 physical_connection_requirements=glue.ConnectionPhysicalConnectionRequirementsArgs(
 security_group_id_lists=cluster.vpc_security_group_ids,
 availability_zone=subnet.availability_zone,
 subnet_id=subnet.id,
 ),
))

# Create an S3 bucket for Glue scripts and temporary storage.
glue_job_bucket = s3.Bucket("glue-job-bucket", s3.BucketArgs(
 force_destroy=True,
))

# Upload a Glue job script.
glue_job_script = s3.BucketObject("glue-job.py", s3.BucketObjectArgs(
 bucket=glue_job_bucket.id,
 source=pulumi.asset.FileAsset("./glue-job.py"),
))

# Create a Glue job that runs our Python ETL script.
glue_job = glue.Job("glue-job", glue.JobArgs(
 role_arn=glue_role.arn,
 glue_version="3.0",
 connections=[
 glue_redshift_connection.name,
 ],
 number_of_workers=10,
 worker_type="G.1X",
 default_arguments={
 # Enabling job bookmarks helps you avoid loading duplicate data.
 # https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html
 "--job-bookmark-option": "job-bookmark-enable",

 "--ConnectionName": glue_redshift_connection.name,
 "--GlueDBName": glue_db_name,
 "--GlueDBTableName": events_bucket.bucket.apply(lambda name: name.replace('-', '_')),
 "--RedshiftDBName": cluster_db_name,
 "--RedshiftDBTableName": "events",
 "--RedshiftRoleARN": redshift_role.arn,
 "--TempDir": glue_job_bucket.bucket.apply(lambda name: f"s3://{name}/glue-job-temp"),
 },
 command=glue.JobCommandArgs(
 script_location=glue_job_bucket.bucket.apply(lambda name: f"s3://{name}/glue-job.py"),
 python_version="3",
 ),
))

# Create a Glue trigger to run the job every 15 minutes.
glue_job_trigger = glue.Trigger("trigger", glue.TriggerArgs(
 schedule=every_15_minutes,
 type="SCHEDULED",
 actions=[
 glue.TriggerActionArgs(
 job_name=glue_job.name,
 ),
 ],
))

# Export the name of the data bucket.
pulumi.export("dataBucketName", events_bucket.bucket)

We’re not quite ready to deploy yet though — we need a script!

Step 3: Add a Glue script

To finish things off, create a new file called glue-job.py (alongside index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf ) and paste in the following code. This is the file that’ll be run by the glue.Job we just declared. There’s a lot going on in this script, but the gist is that when it runs, it’ll extract all unprocessed data from our Glue catalog, transform that data using a simple mapping to tweak some field names (again, just to demonstrate how that’s done), and load the resulting data into Redshift. A Glue job bookmark guarantees that we don’t process the same data more than once:

import sys
from awsglue.utils import getResolvedOptions
from awsglue.transforms import ApplyMapping
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext

# Collect the arguments passed in by the glue.Job run.
args = getResolvedOptions(
 sys.argv,
 [
 "JOB_NAME",
 "TempDir",
 "ConnectionName",
 "GlueDBName",
 "GlueDBTableName",
 "RedshiftRoleARN",
 "RedshiftDBName",
 "RedshiftDBTableName",
 ],
)

glueContext = GlueContext(SparkContext.getOrCreate())

job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Extract all unprocessed data from the Glue catalog.
source0 = glueContext.create_dynamic_frame.from_catalog(
 database=args["GlueDBName"],
 table_name=args["GlueDBTableName"],
 additional_options={
 "jobBookmarkKeys": ["id"],
 "jobBookmarkKeysSortOrder": "asc",
 },
 transformation_ctx="source0",
)

# Transform the data (mostly just to show how to do so).
transformed0 = ApplyMapping.apply(
 frame=source0,
 mappings=[
 ("id", "int", "event_id", "int"),
 ("name", "string", "event_name", "string"),
 ],
)

# Load the data into the Redshift database.
glueContext.write_dynamic_frame.from_jdbc_conf(
 frame=transformed0,
 catalog_connection=args["ConnectionName"],
 connection_options={
 "database": args["RedshiftDBName"],
 "dbtable": args["RedshiftDBTableName"],
 "aws_iam_role": args["RedshiftRoleARN"],
 },
 redshift_tmp_dir=args["TempDir"],
)

# Call commit() to reset the job bookmark for the next run.
job.commit()

And that’s it — you’re ready to deploy.

Deploy!

With everything now in place — VPC, private network, Redshift cluster, and the Glue componentry to handle all of the automation (crawler, job, Python ETL script), you’re ready to deploy. Do that now with a final pulumi up:

$ pulumi up
...

Updating (dev)

 Type Name Status
 pulumi:pulumi:Stack my-data-warehouse-part2-dev
 + ├─ aws:s3:Bucket glue-job-bucket created (2s)
 + ├─ aws:glue:Connection glue-redshift-connection created (1s)
 + ├─ aws:glue:Job glue-job created (0.95s)
 + ├─ aws:s3:BucketObject glue-job.py created (0.63s)
 + └─ aws:glue:Trigger trigger created (0.89s)


Outputs:
 + dataBucketName: "events-e713608"

Resources:
 + 5 created
 11 unchanged

Duration: 7s

You should see that the Glue components were created. Now let’s load some sample data.

Load Sample Data into AWS Glue

As before, we’ll simulate this part, since we don’t have an actual application generating real data for us. Run the following command to write a few JSON records to a file called events-1.txt:

$ echo '{"id": 1, "name": "An interesting event"}
{"id": 2, "name": "Another interesting event"}
{"id": 3, "name": "An event of monumental importance"}' > events-1.txt

With the AWS CLI, upload the file into the S3 data bucket using the name we exported at the end of the last deployment:

$ aws s3 cp events-1.txt s3://$(pulumi stack output dataBucketName)

And now … we wait.

Grab a coffee, take a walk, watch an episode of your favorite sitcom, whatever you like — just be gone for a good half-hour or so to give the crawler and job enough time to run so that when you get back, you’ll have something to look at. (Of course, if you’d rather not wait, you can always just log into the AWS console and kick off the crawler and job manually — but it’s definitely more fun to let the glue.Crawler a glue.Trigger do that for you.)

Soon enough, you should be able to verify, by navigating to Crawlers in the AWS Glue console, that the crawler indeed ran, creating a new table in the Glue catalog database:

As well, you should see (at AWS Glue Studio > Jobs) that the job ran successfully as well:

A successful job run, of course, means that our data must’ve made it into Redshift, too — and you can verify by running a query that in the Redshift query editor:

And with that, your work is done. You’re now running a fully-automated, Pulumi-managed ETL pipeline with Amazon Redshift and AWS Glue.

If you like, you can create a second data file — call it events-2.txt:

$ echo '{"id": 4, "name": "A fourth thing"}
{"id": 5, "name": "The fifth and final thing"}' > events-2.txt

upload that file to S3 as well:

$ aws s3 cp events-2.txt s3://$(pulumi stack output dataBucketName)

and sit back while the pipeline ingests this new data, blissfully ignoring the file that’s already been processed (thanks to the Glue job bookmark) and dropping the new records right into Redshift as you’d expect:

Clean Up AWS Resources with Pulumi

When you’re finished experimenting, be sure to tear down the Redshift cluster and all other resources with a single pulumi destroy:

$ pulumi destroy
...

Destroying (dev)

 Type Name Status
 - pulumi:pulumi:Stack my-data-warehouse-part2-dev deleted
 - ├─ aws:glue:Trigger trigger deleted (36s)
 - ├─ aws:glue:Job glue-job deleted (0.32s)
 - ├─ aws:glue:Connection glue-redshift-connection deleted (0.44s)
 - ├─ aws:redshift:Cluster cluster deleted (229s)
 - ├─ aws:redshift:SubnetGroup subnet-group deleted (0.33s)
 - ├─ aws:ec2:VpcEndpoint s3-vpc-endpoint deleted (6s)
 - ├─ aws:s3:BucketObject glue-job.py deleted (0.88s)
 - ├─ aws:ec2:Subnet subnet deleted (1s)
 - ├─ aws:glue:Crawler glue-crawler deleted (1s)
 - ├─ aws:s3:Bucket events deleted (1s)
 - ├─ aws:iam:Role glue-role deleted (1s)
 - ├─ aws:s3:Bucket glue-job-bucket deleted (2s)
 - ├─ aws:ec2:Vpc vpc deleted (1s)
 - ├─ aws:glue:CatalogDatabase glue-catalog-db deleted (1s)
 - └─ aws:iam:Role redshift-role deleted (1s)


Outputs:
 - dataBucketName: "events-e713608"

Resources:
 - 16 deleted

Watch the AWS Glue ETL Pipeline in Action

If you’d like to see a demonstration of what you’ve read in this blog post, then this Modern Infrastructure video covers the whole process:

What’s next?

I hope you learned as much as I did with this post — we covered a lot, and we’ve only scratched the surface of what’s possible with these tools. Once again, I’d encourage you to spend some time digging into the Redshift and Glue documentation to explore both in more depth, as there’s lots more to learn than what we’ve covered so far.

Once you’re up and running with Redshift, you might also want to check out the Metabase package in the Pulumi Registry. The Metabase package makes it easy to deploy Metabase alongside Redshift on AWS, allowing you to build powerful data-analysis and visualization experiences with your data.

You’ll find the full source of this walkthrough (in TypeScript and Python) on GitHub as well.

Happy data-warehousing!

How to Create and Share a Pulumi Template

Christian Nunciato — Mon, 12 Dec 2022 00:00:00 +0000

Last month, we released our first set of architecture templates — configurable Pulumi projects designed to make it easy to bootstrap new stacks for common cloud architectures like static websites, containers, virtual machines, and Kubernetes clusters. Architecture templates are a great way to get a new project up and running quickly, and they’ve already grown quite popular with our users, several of whom have asked if whether it’s possible to create templates of their own.

It’s not only possible, it’s surprisingly easy! So in this post, I’ll show you how. It all starts with a Pulumi project — a template is really just a Pulumi project with a few extra lines of config — so we’ll start with one of those, add a few resources, and turn the project into a template. When it’s ready to go, we’ll test it out locally, and then finally, we’ll publish it so that anyone with Pulumi can use our template to kickstart their own projects.

We’ll do this in five steps:

Create a new project
Design and build the program
Turn the project into a template
Test-drive the template locally
Publish the template to share it with the world

Let’s get to it.

Step 1: Create a new project

If you haven’t already, make sure you’ve installed Pulumi and configured your AWS credentials. For this example, we’ll be using AWS, but the process is the same for any cloud provider, so if you aren’t set up on AWS, you should still be able to follow along anyway.

Create a new project in the usual way using one of our starter templates. The YAML template is a good choice for this walkthrough, as YAML projects are simple and lightweight (no runtime dependencies!), and because they combine both project and program into one file, they’re especially conducive to sharing:

$ mkdir my-template-project && cd my-template-project
$ pulumi new aws-yaml

Step through the prompts, choosing whichever AWS region works best for you. When the new-project wizard finishes, open the generated Pulumi.yaml project file in your editor of choice to start building.

Step 2: Design and build the program

I’m a static-website person, myself — I think pretty much everyone should have one. So in order to realize my grand vision of home page for every human, you and I are going to build a Pulumi project template that lets anyone with the Pulumi CLI deploy a super-simple website of their own on AWS.

As Pulumi template authors, our general goal is twofold:

Make it easy for our users to create deployable projects that address an infrastructural need.
Make it easy to customize and extend those projects after they’ve been created.

A good template, in other words, is one that not only takes you from pulumi new to pulumi up in as few steps as possible, but that also leaves you with an open, extensible program you can use as a foundation to build upon. For us, the goal is to create a template that can be used to provision the minimal set of cloud resources one needs to run a static website on AWS. And done well, our template will also lend itself easily to further development — custom domains, serverless functions, edge caching, and so on.

To that end, let’s start by replacing the contents of Pulumi.yaml with the following program, which defines the core resources we need: an S3 bucket to hold the files of the website and an S3 bucket object (index.html) to serve as its home page. We’ll also export the computed URL of the website as a Pulumi stack output to give us something to navigate to after deployment:

name: my-template-project
description: A simple static website on Amazon S3
runtime: yaml
resources:

 # Create an S3 bucket and configure it as a website.
 bucket:
 type: aws:s3:Bucket
 properties:
 acl: public-read
 website:
 indexDocument: index.html

 # Create a home page for the website.
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${bucket.id}
 acl: public-read
 contentType: text/html
 content: |
 <html>
 <meta name="content" charset="utf-8">
 <title>My Awesome Website</title>
 <body>
 <h1>My Awesome Website</h1>
 <p>Hello, internet person! 👋</p>
 </body>
 </html>

# Export the URL of the website.
outputs:
 url: http://${bucket.websiteEndpoint}

Try running this program now, just to make sure everything works. In a few seconds, you should see that the new resources were created and be able to browse to the new website at the exported url:

$ pulumi up
...
Updating (dev)
Type Name Status
+ pulumi:pulumi:Stack my-s3-website-template-dev created (3s)
+ ├─ aws:s3:Bucket bucket created (2s)
+ └─ aws:s3:BucketObject index.html created (0.37s)
Outputs:
url: "http://bucket-f1bb181.s3-website-us-west-2.amazonaws.com"
Resources:
+ 3 created
Duration: 5s

Good — we have a working program. And looking back at our design goals, you’ll see that this program already meets them pretty well:

It works out of the box. Deployed now, it produces a running website on AWS.
It’s open and extensible. To build onto it — add more pages, attach a custom domain, etc. — you’d simply add more resources alongside those already defined.

Now let’s turn this program into a template.

Before moving on, though, let’s quickly destroy and remove this stack, as we’ll be making a few changes to the project’s settings in the next step, and it’d be better to begin from a clean slate:

$ pulumi destroy --yes --remove

Step 3: Turn the project into a template

The next thing to do is decide which parts of the program should be configurable. Earlier, when you ran pulumi new aws-yaml, you probably noticed you were prompted for an optional AWS region:

$ pulumi new aws-yaml
...
This command will walk you through creating a new Pulumi project.
...
aws:region: The AWS region to deploy into: (us-west-2)

You got this prompt because the authors of the aws-yaml template knew that not every user would want to deploy into the same hard-coded AWS region, so they defined an aws:region setting to make it both configurable and optional, falling back to us-west-2 by default. Users of this template are free to change this value if they like or leave it alone and accept the default.

This is all made possible by the existence of the template block in Pulumi.yaml:

# The `template` block from the aws-yaml template.
template:
 description: A minimal AWS Pulumi YAML program
 config:
 aws:region:
 description: The AWS region to deploy into
 default: us-west-2

Indeed, this block is all that defines aws-yaml as a template; without it, the aws-yaml project would be a regular ol’ Pulumi YAML program like any other.

The template block defines two properties:

An optional description property to give new projects created from the template
A config block that lists the names, descriptions, and default values of any settings that should be configurable for new projects

The template block is essentially where all the new-project magic happens. Any settings you define in this block will prompt users for their values and apply those settings to the project’s initial stack (for example, the dev stack we created in Step 1). By default, these values are captured and applied as plain-text strings, but you can also capture them as encrypted Pulumi secrets by adding secret: true — an appropriate choice for prompting for sensitive data like passwords, API keys, and the like.

Our super-simple website template doesn’t need much in the way of configurability — but let’s define some anyway just to see how it’s done. While we’re at it, we’ll make a few adjustments to make project names and descriptions configurable as well.

Make the changes below to Pulumi.yaml to use the project name and description supplied by the CLI, add a template section to let users configure a home-page title and greeting, and interpolate those values into the body of the home page itself. Here’s the full content of the program for reference:

# Configure project names and descriptions with values obtained from `pulumi new`.
name: ${PROJECT}
description: ${DESCRIPTION}
runtime: yaml

# Define the template's configuration settings.
template:
 description: A simple static website on Amazon S3
 config:
 aws:region:
 description: The AWS region to deploy into
 default: us-west-2
 title:
 description: A title to use for the home page
 default: My Awesome Website
 greeting:
 description: A greeting to show on the home page
 default: Hello, internet person! 👋

resources:

 # Create an S3 bucket and configure it as a website.
 bucket:
 type: aws:s3:Bucket
 properties:
 acl: public-read
 website:
 indexDocument: index.html

 # Create a home page for the website.
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${bucket.id}
 acl: public-read
 contentType: text/html
 content: |
 <html>
 <meta name="content" charset="utf-8">
 <title>${title}</title>
 <body>
 <h1>${title}</h1>
 <p>${greeting}</p>
 </body>
 </html>

# Export the URL of the website.
outputs:
 url: http://${bucket.websiteEndpoint}

That’s all there is to it! The template’s complete and ready to be published. However, before we do that, let’s first give it a try locally just to make sure it delivers the new-project experience we’re looking for.

Step 4: Test-drive the template locally

Back in your terminal (and assuming you’re still in the my-template-project folder), create a new folder alongside my-template-project, then run pulumi new to create a new project, pointing Pulumi to the folder containing your template:

$ cd ..
$ mkdir my-test-website && cd my-test-website
$ pulumi new ../my-template-project

Step through the prompts, which should include each of the configurable settings we defined in the template block earlier, along with their default values:

$ pulumi new ./my-template-project
project name: (my-test-website)
project description: (A simple static website on Amazon S3)
...
aws:region: The AWS region to deploy into: (us-west-2)
greeting: A greeting to show on the home page: (Hello, internet person! 👋) Hi, world!
title: A title to use for the home page: (My Awesome Website) Hey look, it's my website!
Your new project is ready to go! ✨
To perform an initial deployment, run `pulumi up`

Go ahead and deploy the new project with pulumi up, then navigate to the newly deployed website in your favorite web browser:

$ pulumi up --yes
$ open $(pulumi stack output url)

Well done! You’ve written your first Pulumi template.

Let’s finish things off by publishing your template so others can use it. As before, tidy up with pulumi destroy before moving on:

$ pulumi destroy --yes --remove

Step 5: Publish the template

A moment ago, when you tested your first template, you did so by pointing Pulumi to the template’s path on your local filesystem. Before that, when you ran pulumi new aws-yaml, you instructed Pulumi (implicitly) to fetch the source of the template over HTTPS from the official pulumi/templates repository on GitHub. So to make your template available to others, all you have to do publish it to a file path or Git URL that’s accessible by the Pulumi CLI.

You could create a new GitHub repository for this project, then pass the fully-qualified URL of the path containing Pulumi.yaml to pulumi new, like so:

$ pulumi new https://github.com/{your-org}/{your-repo}/tree/main

But since our project consists of just one file, you don’t even need to do that much: you can just paste the contents of the file into a new GitHub gist (since a gist is also a Git repository), name the gist Pulumi.yaml, and point Pulumi to the gist’s URL.

Go ahead and create a gist of your own, then use it to create a new project:

$ mkdir ../my-second-test-website && cd ../my-second-test-website
$ pulumi new https://gist.github.com/cnunciato/b331efae6a4740c237a0364d17fe220f
$ pulumi up

Be sure to tidy up as before with pulumi destroy when you’re done.

Bonus step: Add a Deploy with Pulumi button

In addition to the CLI, your users can also create new projects in the Pulumi Service with the Deploy with Pulumi button. This is a great option for making your project installable from GitHub READMEs and other team docs. Here, for example, is a Deploy button that creates a new project using my version of the gist we created above:

Embedding these buttons yourself is easy — just use one of the snippets below, swapping the values of {template-url} for the URL of your template’s gist or Git repository:

<a href="https://app.pulumi.com/new?template={template-url}">
 <img src="/images/deploy-with-pulumi/dark.svg" title="Deploy with Pulumi">
</a>

[![Deploy with Pulumi](/images/deploy-with-pulumi/dark.svg)](https://app.pulumi.com/new?template={template-url})

Project creators who go down this path will be prompted in the browser for the same configuration values as they would with the CLI, and afterward, they’ll be able to deploy the project either with the Pulumi CLI or with Pulumi Deployments.

See it in action

You can watch this Modern Infrastructure video to see another example of creating and sharing a Pulumi template:

Wrapping up

As you can see, creating a template is both simple and powerful, and I hope this post encourages you to experiment with a few of your own. Feel free to peruse the following links for inspiration:

And as always, be sure to stop by Pulumi Community Slack to let us know know how it goes.

Happy templating!

Deploying a Data Warehouse with Pulumi and Amazon Redshift

Christian Nunciato — Tue, 29 Nov 2022 00:00:00 +0000

It’s fun to think about how much data there is swirling around in the global datasphere these days. However you choose to measure it (and there are various ways), it’s a quantity so massive — hundreds of zettabytes, by some estimates — that it’s kind of a hard thing to quite get your head around.

If you could convert all the world’s data into droplets of water, for instance, at one megabyte per drop, you’d have enough 1MB drops to fill two more Lake Washingtons. If you could store all that data on 3.5" floppies, you’d need more than a hundred quadrillion floppies to capture it all — enough to cover the planet entirely (with much room for overlap) or to pave a nice bridge for yourself from your front porch well into interstellar space. If you could pull all that data into an HD movie, and you sat down to start watching that movie 2.5 million years ago (with your favorite saber-toothed friend, say), you’d still be watching the same movie today.

That’s a lot of data — and there’s more every day.

Of course, the volume of data you care about is probably a lot more modest. But even so, there’s a good chance whatever system you’re building can and will produce useful data of its own, and there’s an equally good chance you’ll want to learn something from it. That data will likely exist in many forms and in multiple places — flat file here, transactional database there, the REST API of some third-party service — so before you can really look at it, you’ll need to figure out how to get it all into one place. And for that, you might want to consider a data warehouse.

A data warehouse is a specialized database that’s purpose built for gathering and analyzing data. Unlike general-purpose databases like MySQL or PostgreSQL, which are designed to meet the real-time performance and transactional needs of applications, a data warehouse is designed to collect and process the data produced by those applications, collectively and over time, to help you gain insight from it. Examples of data-warehouse products include Snowflake, Google BigQuery, Azure Synapse Analytics, and Amazon Redshift — all of which, incidentally, are easily managed with Pulumi.

Today, though, we’re going to focus on Amazon Redshift. Specifically, we’re going to walk through the process of writing a Pulumi program that provisions a single-node Redshift cluster in an Amazon VPC, then we’ll load some sample data into the warehouse from Amazon S3. We’ll load this data manually at first, just to get a sense of how everything works when it’s all wired up, and then later, in a follow-up post, we’ll go a step further and weave in some automation to load the data on a schedule.

Let’s get going!

Create a new project

We’ll begin, as always, by creating a new Pulumi project. (And if you haven’t already, make sure you’ve installed Pulumi and configured your AWS credentials in the usual way.)

$ mkdir my-data-warehouse && cd my-data-warehouse
$ pulumi new aws-typescript

$ mkdir my-data-warehouse && cd my-data-warehouse
$ pulumi new aws-python

Step through the new-project wizard, accepting the defaults if they’ll work for you. For this project, we’ll deploy both cluster and data into the us-west-2 region of AWS, but you can change the target to region you like, as both S3 and Redshift are supported in all AWS regions.

With the project and stack in place, it’s time to start building.

Configure the stack

To launch a new Redshift cluster, you’ll need to give AWS a few details:

A unique (in the scope of your AWS account) name for the Redshift cluster
A name for the cluster’s initial database
A username and password for the cluster’s administrative user
The node type to use for the Redshift cluster

The following configuration settings should work for this walkthrough (but again, feel free to adjust them if you’d like):

$ pulumi config set clusterIdentifier my-redshift-cluster
$ pulumi config set clusterNodeType ra3.xlplus
$ pulumi config set clusterDBName dev
$ pulumi config set clusterDBUsername admin
$ pulumi config set clusterDBPassword StrongPass1 --secret

Compose the program

With these settings in place, you can start writing the Pulumi program.

Open index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf in your editor of choice and replace its contents with the following code, which imports the Pulumi and AWS libraries, reads the configuration values you just set, and provisions an S3 bucket that we’ll use to store some raw, unstructured data later:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Import the stack's configuration settings.
const config = new pulumi.Config();
const clusterIdentifier = config.require("clusterIdentifier");
const clusterNodeType = config.require("clusterNodeType");
const clusterDBName = config.require("clusterDBName");
const clusterDBUsername = config.require("clusterDBUsername");
const clusterDBPassword = config.requireSecret("clusterDBPassword");

// Import the provider's configuration settings.
const providerConfig = new pulumi.Config("aws");
const awsRegion = providerConfig.require("region");

// Create an S3 bucket to store some raw data.
const eventsBucket = new aws.s3.Bucket("events", {
 forceDestroy: true,
});

import json
import pulumi
from pulumi_aws import ec2, iam, redshift, s3

# Import the stack's configuration settings.
config = pulumi.Config()
cluster_identifier = config.require("clusterIdentifier")
cluster_node_type = config.require("clusterNodeType")
cluster_db_name = config.require("clusterDBName")
cluster_db_username = config.require("clusterDBUsername")
cluster_db_password = config.require_secret("clusterDBPassword")

# Import the provider's configuration settings.
provider_config = pulumi.Config("aws")
aws_region = provider_config.require("region")

# Create an S3 bucket to store some raw data.
events_bucket = s3.Bucket("events", s3.BucketArgs(
 force_destroy=True,
))

Next, define a new VPC and the associated network resources for the Redshift cluster. Since the aim is to launch the cluster into a VPC (providing the network isolation we need to keep the cluster from being accessed over the internet), you’ll define the VPC first, then define a private subnet within it, and then finally designate a Redshift subnet group to tell AWS where to provision the cluster:

// ...

// Create a VPC.
const vpc = new aws.ec2.Vpc("vpc", {
 cidrBlock: "10.0.0.0/16",
 enableDnsHostnames: true,
});

// Create a private subnet within the VPC.
const subnet = new aws.ec2.Subnet("subnet", {
 vpcId: vpc.id,
 cidrBlock: "10.0.1.0/24",
});

// Declare a Redshift subnet group with the subnet ID.
const subnetGroup = new aws.redshift.SubnetGroup("subnet-group", {
 subnetIds: [
 subnet.id,
 ],
});

# ...

# Create a VPC.
vpc = ec2.Vpc("vpc", ec2.VpcArgs(
 cidr_block="10.0.0.0/16",
 enable_dns_hostnames=True,
))

# Create a private subnet within the VPC.
subnet = ec2.Subnet("subnet", ec2.SubnetArgs(
 vpc_id=vpc.id,
 cidr_block="10.0.1.0/24",
))

# Declare a Redshift subnet group with the subnet ID.
subnet_group = redshift.SubnetGroup("subnet-group", redshift.SubnetGroupArgs(
 subnet_ids=[
 subnet.id,
 ],
))

Now, on to the cluster itself.

The plan, you’ll recall, is to pull data from an S3 bucket into Redshift (using the Redshift service itself, for now), so you’ll need to give Redshift the appropriate permissions to read from Amazon S3. You can do that with an IAM role and AWS-managed policy granting Redshift read-only access to the S3 service:

// ...

// Create an IAM role granting Redshift read-only access to S3.
const redshiftRole = new aws.iam.Role("redshift-role", {
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "redshift.amazonaws.com",
 },
 },
 ],
 },
 managedPolicyArns: [
 aws.iam.ManagedPolicy.AmazonS3ReadOnlyAccess,
 ],
});

# ...

# Create an IAM role granting Redshift read-only access to S3.
redshift_role = iam.Role("redshift-role", iam.RoleArgs(
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "redshift.amazonaws.com",
 },
 },
 ],
 }),
 managed_policy_arns=[
 iam.ManagedPolicy.AMAZON_S3_READ_ONLY_ACCESS,
 ],
))

That takes care of the permissions part — but there’s one last thing you’ll need to do to make this work. Because the cluster will reside in a private subnet, and that subnet won’t have access to the public internet, you’ll need to give the cluster a way to communicate with S3 without having to leave the VPC network. You can do this by adding a VPC gateway endpoint:

// ...

// Create a VPC endpoint so the cluster can read from S3 over the private network.
const vpcEndpoint = new aws.ec2.VpcEndpoint("s3-vpc-endpoint", {
 vpcId: vpc.id,
 serviceName: `com.amazonaws.${awsRegion}.s3`,
});

# ...

# Create a VPC endpoint so the cluster can read from S3 over the private network.
vpc_endpoint = ec2.VpcEndpoint("s3-vpc-endpoint", ec2.VpcEndpointArgs(
 vpc_id=vpc.id,
 service_name=f"com.amazonaws.{aws_region}.s3",
))

Finally, add the cluster itself, using everything you’ve defined up to now — config settings, network settings, IAM role, and the VPC’s default security group, which enables traffic to and from the cluster on all ports and subnet IPs. Finish things off by exporting the name of the S3 bucket and Redshift ARN, as you’ll need both in order to post and then import the S3 data (both will be auto-named):

// ...

// Create a single-node Redshift cluster in the VPC.
const cluster = new aws.redshift.Cluster("cluster", {
 clusterIdentifier: clusterIdentifier,
 databaseName: clusterDBName,
 masterUsername: clusterDBUsername,
 masterPassword: clusterDBPassword,
 nodeType: clusterNodeType,
 clusterSubnetGroupName: subnetGroup.name,
 clusterType: "single-node",
 publiclyAccessible: false,
 skipFinalSnapshot: true,
 vpcSecurityGroupIds: [
 vpc.defaultSecurityGroupId,
 ],
 iamRoles: [
 redshiftRole.arn,
 ],
});

// Export the bucket name and role ARN.
export const dataBucketName = eventsBucket.bucket;
export const redshiftRoleArn = redshiftRole.arn;

# ...

# Create a single-node Redshift cluster in the VPC.
cluster = redshift.Cluster("cluster", redshift.ClusterArgs(
 cluster_identifier=cluster_identifier,
 database_name=cluster_db_name,
 master_username=cluster_db_username,
 master_password=cluster_db_password,
 node_type=cluster_node_type,
 cluster_subnet_group_name=subnet_group.name,
 cluster_type="single-node",
 publicly_accessible=False,
 skip_final_snapshot=True,
 vpc_security_group_ids=[
 vpc.default_security_group_id,
 ],
 iam_roles=[
 redshift_role.arn,
 ],
))

# Export the bucket name and role ARN.
pulumi.export("dataBucketName", events_bucket.bucket)
pulumi.export("redshiftRoleArn", redshift_role.arn)

And with that, you’re ready to deploy.

Deploy the stack

Back at the command line, deploy the stack in the usual way with pulumi up:

$ pulumi up
...
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack my-data-warehouse-dev created (3s)
 + ├─ aws:s3:Bucket events created (2s)
 + ├─ aws:ec2:Vpc vpc created (12s)
 + ├─ aws:iam:Role redshift-role created (1s)
 + ├─ aws:ec2:Subnet subnet created (0.81s)
 + ├─ aws:ec2:VpcEndpoint s3-vpc-endpoint created (6s)
 + ├─ aws:redshift:SubnetGroup subnet-group created (0.80s)
 + └─ aws:redshift:Cluster cluster created (270s)

Outputs:
 dataBucketName : "events-50d8c96"
 redshiftRoleArn: "arn:aws:iam::616138583583:role/redshift-role-47ad8f0"

Resources:
 + 8 created

Duration: 4m48s

After a few moments (deploying a new cluster takes about five minutes), you should see that the new cluster and all of its resources were created. Again, you won’t be able to access the cluster over the internet, as it’ll have been deployed into an isolated VPC network, but you should be able to browse to the cluster in the AWS Console and start running some queries with it.

Choose Query data > Query in query editor v2 to open the Redshift query editor:

Then, from the query editor, find your newly created cluster in the left-hand pane and connect using the database, username, and password you configured earlier:

Choose Create connection, expand my-redshift-cluster, and you should see the dev database there, ready to go (only without any tables — we’ll get to that next):

Now let’s load some data!

Getting data into Redshift

In a real-world situation, you’d probably already have some data to load — web server logs, for example, or some other sort of raw or unstructured data residing in an S3 bucket, a DynamoDB table, an RDS database, etc. But we’re starting from scratch, so we’ll create and publish that data manually.

Copy and paste the following command to generate a text file containing a few lines of JSON, each representing a fictitious event to be loaded into the data warehouse:

$ echo '{"id": 1, "name": "An interesting event"}
{"id": 2, "name": "Another interesting event"}
{"id": 3, "name": "An event of monumental importance"}' > events-1.txt

Notice there aren’t any commas between these events, so it’s not actually a valid JSON file – it’s just a text file containing some individually valid JSON events, one event per line. Adhering to this format allows Redshift to ingest and map the data to table we create next using one of its built-in parsers.

Upload the file to S3 with the AWS CLI, using Pulumi to supply the name of the destination bucket:

$ aws s3 cp events-1.txt "s3://$(pulumi stack output dataBucketName)"

At this point, you have all you need in terms of infrastructure: You’ve got a single-node Redshift cluster, an empty Redshift database, some unstructured S3 data, and the right set of permissions and network settings to allow the cluster to reach and import the data. All that’s left is to run a couple of SQL queries to make that happen: one to create a new table to hold the data, another to load it.

Back in the Redshift query editor, create the new table by choosing the dev database, pasting the following query into the editor, and hitting Run:

CREATE TABLE events (
 id INT,
 name VARCHAR(255)
);

You should see that the query succeeded and that the table was created:

Now you’re ready to import the data. Replace the contents of the query editor with the following Redshift COPY command, replacing the {bucket-name}, {iam-role}, and {aws-region} placeholders with the corresponding values from your Pulumi stack:

COPY events
FROM 's3://{bucket-name}/events-1.txt'
IAM_ROLE '{iam-role}'
FORMAT AS JSON 'auto'
REGION AS '{aws-region}';

Here’s what I see, for example, after running that command:

After the COPY command completes (which may take a few seconds), you should be able to run a new query the events table and see the new data:

SELECT *
 FROM events
 ORDER BY id;

Huzzah! You’ve now got yourself a working data warehouse.

See it in action

To demonstrate the entire process of building a data warehouse with Redshift and Pulumi, we created this Modern Infrastructure video that shows all of this in action:

Taking stock, and looking ahead

If you’re like me, you’re probably happy to be up and running — but you’ve also got a few questions:

What would happen if you added a few more rows to events-1.txt, then ran that COPY command a second time? Would Redshift ignore the records it’s already processed, and only import the new ones?
Running SQL commands in a browser like this seems less than ideal. Can I run this process automatically somehow — like once a day, or in response to some other event?
My source data’s kind of a mess. Can I process the data somehow when I load it — like drop certain fields, rename columns, split records into multiple tables, and so on?
Can I pull data from other sources that aren’t Amazon S3?

All good questions, and I’m glad that you asked them.

First, as you’ll notice, adding lines to events.txt and running COPY a second time probably doesn’t behave as you might hope:

Duplicate records — the bane of the data scientist.

As of now, your only real recourse is to leave events-1.txt alone, create new data files for each batch of events, and only COPY those files that haven’t been processed already. For small projects that don’t generate much data, this might be a viable option, but over time, it’d probably become pretty tedious, and it’d be all too easy to make a mistake that leaves you with messed-up data. Not ideal.

Instead, what you’ll ultimately want is some process by which your data can be extracted (duplicate-free and from multiple sources), transformed according to rules you specify (keeping what you want, dropping what you don’t), and loaded into the warehouse without your involvement. What you’ll want, in other words, is an ETL pipeline.

The good news is that with the data warehouse up and running, you’re well on your way already. The bad news is that Redshift alone won’t finish the job for you. Redshift is just the destination — the target of your ETL pipeline-to-be. The pipeline itself still needs to be written.

So in the next post, we’ll do that: We’ll take what we’ve done here, add a few more components with Pulumi and AWS Glue, and wire it all up with a few magical lines of Python scripting.

Until then, happy coding — and stay tuned!

Shared configuration stacks with AWS Systems Manager

Christian Nunciato — Thu, 30 Jun 2022 00:00:00 +0000

One thing I love about Pulumi is how easy it is to configure a stack. As a builder mainly of web applications, I’m always thinking about how I’ll configure my apps from one environment to the next, and being able to use Pulumi’s built-in support for configuration and secrets to manage the API keys and database credentials for my dev, staging, and production stacks individually is incredibly convenient.

For larger teams and organizations, though, where multiple applications rely on a set of common configuration settings — dozens of apps, say, depending on the same API service or database — having to keep all of those config settings in sync across all of those individually can become a bit of a pain. When this happens, you may find yourself looking for ways to extract those settings into some sort of a service to allow you to manage them easily in one place, and in a way that allows any application to inherit them automatically.

There are lots of ways of addressing this sort of problem. One that I like is AWS Systems Manager (SSM) — specifically, SSM Parameter Store. With Systems Manager and Parameter Store, you can store plain-text and encrypted secrets and expose them easily to any other resource in your AWS infrastructure. Parameter Store is a powerful tool to have in your cloud-programming toolbox — and combined with Pulumi, and in particular Pulumi stack references, you can use it to build specialized stacks make managing shared configuration simple.

In this post, I’ll show you one way to do this. We’ll start with a simple Pulumi stack that defines a few config settings with Parameter Store, then stand up a couple of other stacks to illustrate how to inherit and use those settings. We’ll also use Pulumi YAML, one of the newest additions to the Pulumi language family, to give you an opportunity to kick the tires on that as well.

Let’s get going.

Sketching it out

The plan is to begin with a Pulumi stack to act as the keeper of our hypothetical organization’s shared configuration. This stack will define two config settings: a message of the day, which we’ll store in plain text, and a secret message of the day (ssh!), which we’ll store as an encrypted secret. Later, we’ll build two stacks to act as downstream consumers of this shared-config stack — one, a static website that’ll use our config values to render some text in the browser, the other a serverless function that’ll return those values as JSON. Both will use Pulumi stack references to obtain their config values from Parameter Store. Here’s a visual:

Let’s start Stack 1 — the shared-config stack.

Building the shared config stack

First, make sure you’ve installed and configured Pulumi for AWS, and then, in your shell of choice, create a new folder to house our three stacks-to-be:

$ mkdir shared-config-with-ssm && cd shared-config-with-ssm
$ mkdir shared-config my-website my-service

This’ll leave you with three folders to work with, one for each stack:

$ tree .
.
├── my-service
├── my-website
└── shared-config

Change to the shared-config folder and generate a new Pulumi YAML project, accepting the defaults to create a new dev stack:

$ cd shared-config
$ pulumi new aws-yaml

When the new-project wizard completes, add two new settings for the dev stack with pulumi config set:

$ pulumi config set motd 'Hello from Pulumi!'
$ pulumi config set motd_secret 'Ssh! This is a secret.' --secret

These are the two values we’re going to make available to Stacks 2 and 3.

Now let’s build out the Pulumi program to provision these items as parameters in AWS Systems Manager. Replace the contents of Pulumi.yaml with the following YAML program, which reads the config values we just set, declares two new aws.ssm.Parameter resources (using fully-qualified paths as a best practice), and exports their parameter names as Pulumi stack outputs:

name: shared-config
runtime: yaml

# The values we're reading from stack configuration.
configuration:
 motd:
 type: String
 motd_secret:
 type: String

# The AWS SSM Parameters managed by this stack.
resources:
 motd_param:
 type: aws:ssm:Parameter
 properties:
 name: /shared-config/dev/motd
 value: ${motd}
 type: String
 motd_secret_param:
 type: aws:ssm:Parameter
 properties:
 name: /shared-config/dev/motd_secret
 value: ${motd_secret}
 type: SecureString

# Outputs are the fully-qualified AWS SSM Parameter names
# that consuming stacks can use to retrieve associated values.
outputs:
 motd_param_name: ${motd_param.name}
 motd_secret_param_name: ${motd_secret_param.name}

At this point, you might be wondering why we’re only exporting our parameters’ names, and not their values.

We certainly could export their values, since we already have them at hand. But if we did, there wouldn’t be much point to using Parameter Store at all, since Pulumi itself is perfectly capable of managing plain-text and encrypted values, and stack references give other stacks an easy way of accessing them programmatically. Why not export the values, then?

For one, it’d be less flexible. If we exposed these values as stack outputs, downstream stacks referencing them would only be able to read them at deploy-time — i.e., during their own Pulumi updates. For shared-config values that change rarely, that might be okay, but for others, it’d be less than ideal in that all downstream stacks would have to be updated immediately whenever one of those values were changed.

Secondly, it keeps secrets a bit more secret. Pulumi is great at protecting the secrets it manages, even across stack boundaries like these, but applications still need a way to get access to the underlying plain-text values of secrets at runtime. Lambda functions, for example, need access tokens and database passwords to do the work we need them to do — but figuring out how to provide them with those raw values safely can be tricky. By requiring client applications to fetch secrets from Parameter Store, we can keep their plain-text values out of the bodies of our Lambda functions and prevent them from surfacing inadvertently in the AWS Lambda Console.

Finally, it leaves room for the occasional out-of-band change. Now and then, you might be faced with having to update a value in Parameter Store directly — e.g., with the AWS Console or CLI. Downstream stacks not reading from Parameter Store, however, wouldn’t have access to these updated values until after the shared-config stack were refreshed and redeployed — and would themselves have to be redeployed in order to use them.

Perhaps that was more explanation than necessary for what amounts to just a few lines of YAML, but hopefully it gives you a sense of the kinds of things to be thinking about as you explore solutions like this on your own. For now, let’s forge ahead and get this first stack deployed.

Back in your terminal, do that with pulumi up. The deployment should take only a few seconds to complete:

$ pulumi up
...

Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack shared-config-dev created
 + ├─ aws:ssm:Parameter motd_param created
 + └─ aws:ssm:Parameter motd_secret_param created

Outputs:
 motd_param_name : "/shared-config/dev/motd"
 motd_secret_param_name: "/shared-config/dev/motd_secret"

Resources:
 + 3 created

Duration: 4s

Notice we’ve named our outputs clearly, and that the secret message is safely encrypted in Pulumi.dev.yaml and in Parameter Store. You can confirm the latter by visiting the AWS Console:

Now let’s put these two new values to practical use.

Fetching configuration at deploy-time

Assuming you’re still in the shared-config folder, change to the my-website folder you created a few moments ago and generate another YAML project, this one for our humble little website:

$ cd ../my-website
$ pulumi new aws-yaml

Once again, follow the prompts to create a new dev stack, then replace the contents of Pulumi.yaml with the program below. Here, we’re fetching the names of the parameters being managed by shared-config, pulling their values from Systems Manager, and creating a one-page website to render those values in the browser. (Be sure to adjust the pulumi:pulumi:StackReferences resource to point to your stack instead of mine.)

name: my-website
runtime: yaml

variables:

 # Get the names of the parameters we care about from the shared-config stack.
 motd_param_ref: ${my-stack-reference.outputs["motd_param_name"]}
 motd_secret_param_ref: ${my-stack-reference.outputs["motd_secret_param_name"]}

 # Fetch (and decrypt) their values from Systems Manager.
 motd_param:
 Fn::Invoke:
 Function: aws:ssm:getParameter
 Arguments:
 name: ${motd_param_ref}
 motd_secret_param:
 Fn::Invoke:
 Function: aws:ssm:getParameter
 Arguments:
 name: ${motd_secret_param_ref}
 withDecryption: true

resources:
 # Create the stack reference
 my-stack-reference:
 type: pulumi:pulumi:StackReference
 properties:
 name: cnunciato/shared-config/dev  # <-- Change this.

 # Create an S3 bucket and configure it as a website.
 my-bucket:
 type: aws:s3:Bucket
 properties:
 website:
 indexDocument: index.html

 # Create a homepage and render the values.
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${my-bucket}
 acl: public-read
 contentType: text/html
 content: |
 <html>
 <head>
 <meta charset="utf-8">
 <style>
 span { display: none; }
 button:active + span { display: inline; }
 </style>
 </head>
 <body>
 <h1>${motd_param.value}</h1>
 <button>Reveal the secret message!</button>
 <span>${motd_secret_param.value}</span>
 </body>
 </html>

outputs:

 # Export the publicly accessible URL of the website.
 url: http://${my-bucket.websiteEndpoint}

Save the file and deploy the stack, which again should take only a few seconds to complete:

$ pulumi up
...

Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack my-website-dev created
 + ├─ aws:s3:Bucket my-bucket created
 + └─ aws:s3:BucketObject index.html created

Outputs:
 url: "http://my-bucket-269e711.s3-website-us-west-2.amazonaws.com"

Resources:
 + 3 created

Duration: 5s

With the exported URL, open the website in your favorite browser, and you should also see the message of the day on the homepage, along with a button you can click to reveal the secret message:

If you’re using the Pulumi Service, which tracks dependencies between stacks, you should see that the my-website stack is now listed as a downstream consumer of shared-config:

Now, when the hypothetical team managing the shared-config stack decides it’s time to update the message of the day, it can do so with a pulumi config set and pulumi up, and my-website will be able to pick up the new message automatically.

However, it’s worth mentioning that that won’t happen immediately, as the my-website stack is configured to pull the message from Systems Manager at deploy-time. If we want to reflect the new value on the homepage, we’ll either need to deploy the my-website again or come up with a way to fetch the message at runtime — which brings us to our third and final stack, in which you’ll learn how to do just that.

Fetching configuration at runtime

The easiest way to see this in action is with AWS Lambda. With Lambda and Pulumi’s support for function serialization, we can capture the parameter names with stack references, pass them into the Lambda at deploy-time (as plain-text strings, which is fine, because they’re just names), and fetch their values from Systems Manager at runtime — i.e., when the Lambda is invoked sometime later — with the AWS SDK for Node.js. This way, any updates made by the shared-config team will be usable by our Lambda function immediately, no redeployment necessary.

So to finish things off, change to the my-service folder you created earlier and generates a third and final project, this one with TypeScript, again accepting the defaults to create a dev stack for it:

$ cd ../my-service
$ pulumi new aws-typescript

Replace the contents of index.ts with the following program. As before, we’re using StackReferences to read our parameter names from the shared-config stack — except here, because StackRerefences are provided as pulumi.Outputs, which can’t be serialized into the Lambda, we’ll need to use .get() to unwrap the parameter name into a string so we can use it at runtime with the AWS SDK:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Get the names of the parameters we care about from the shared-config stack.
const stack = new pulumi.StackReference("cnunciato/shared-config/dev"); // <-- Your name here.
const motdParamRef = stack.requireOutput("motd_param_name");
const motdSecretParamRef = stack.requireOutput("motd_secret_param_name");

const callback = new aws.lambda.CallbackFunction("lambda", {
 policies: [
 aws.iam.ManagedPolicy.LambdaFullAccess,

 // Give the Lambda read-only access to AWS Systems Manager.
 aws.iam.ManagedPolicy.AmazonSSMReadOnlyAccess
 ],
 callback: async () => {

 // Use the AWS SDK for JavaScript to fetch parameter values from Systems Manager.
 const ssm = new aws.sdk.SSM();

 // Fetch the message of the day.
 const motdParam = await ssm.getParameter({
 Name: motdParamRef.get(), // <-- Use .get() for you runtime access to Output values.
 }).promise();

 // Fetch the secret message of the day.
 const motdSecretParam = await ssm.getParameter({
 Name: motdSecretParamRef.get(),
 WithDecryption: true,
 }).promise();

 return {
 statusCode: 200,
 contentType: "application/json",

 // Return the result.
 body: JSON.stringify({
 motd: motdParam.Parameter?.Value,
 motd_secret: motdSecretParam.Parameter?.Value
 }),
 };
 },
});

const lambdaUrl = new aws.lambda.FunctionUrl("lambda-url", {
 functionName: callback.name,
 authorizationType: "NONE",
});

// Export the publicly accessible URL of the service.
export const url = lambdaUrl.functionUrl;

When you’re ready, deploy the my-service stack with pulumi up:

$ pulumi up
...

Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack my-service-dev created
 + ├─ aws:iam:Role lambda created
 + ├─ aws:lambda:Function lambda created
 + ├─ aws:iam:RolePolicyAttachment lambda-b63d666c created
 + ├─ aws:iam:RolePolicyAttachment lambda-b5aeb6b6 created
 + └─ aws:lambda:FunctionUrl lambda-url created

Outputs:
 url: "https://6y3vkpktyybylhua266bu7yplu0dgqgv.lambda-url.us-west-2.on.aws/"

Resources:
 + 6 created

Duration: 19s

Verify the deployment with a request to the Lambda function’s public endpoint:

$ curl $(pulumi stack output url)
{
 "motd": "Hello from Pulumi!",
 "motd_secret": "Ssh! This is a secret."
}

That tells you the runtime lookup is working, which is great.

Now try updating the message of the day. Change to the shared-config folder, use pulumi config set to update the stack’s configured value, and finally, run pulumi up to write the new value to Systems Manager:

$ cd ../shared-config
$ pulumi config set motd 'Hello *again* from Pulumi!'
$ pulumi up
...

Updating (dev)

 Type Name Plan Info
 pulumi:pulumi:Stack shared-config-dev
 ~ └─ aws:ssm:Parameter motd_param updated [diff: ~value]

Resources:
 ~ 1 updated
 2 unchanged

If all goes well, you should be able to invoke the Lambda again and see the new value immediately:

$ cd ../my-service
$ curl $(pulumi stack output url)
{
 "motd": "Hello *again* from Pulumi!",
 "motd_secret": "Ssh! This is a secret."
}

And with that, we’re done. You can tear everything down with pulumi destroy:

$ cd ..
$ pulumi -C my-service destroy
$ pulumi -C my-website destroy
$ pulumi -C shared-config destroy

Wrapping up, and next steps

Managing configuration at scale is a much bigger topic than we could possibly cover in a blog post, but hopefully you’ve learned enough here to fuel some experimentation of your own. You’ll find the full source code for this walkthrough on GitHub as well.

Additionally, as an alternative to Parameter Store, you might also want to check out AWS Secrets Manager. It’s built specifically for managing secrets, and gives you additional capabilities like rotation and auditing, so depending on your use case, it could be a better fit for certain scenarios.

Happy configuring!

Test-Driven Infrastructure Development with Pulumi and Jest

Christian Nunciato — Mon, 13 Jun 2022 00:00:00 +0000

When I was a kid growing up in Southern California, there was a phone number you could call to find out what time it was. It was a local number, 853-1212 (easy to remember as the arrangement of the numbers on the keypad made a capital T), and I used it all the time, to set my watch, adjust the alarm clock, fix the display on the VCR. I don’t recall the last time I used it, probably sometime in the mid ’90s, but I do remember clearly the sound of the voice at the other end of the line.

Services like these were called speaking clocks (although I never called them that; to me, it was just “calling Time”), and they date back to the early 1930s or so. In the U.S., most were powered by a machine called an Audichron that used mechanical drums to play back the time of day, often with a little advertisement or the current temperature to go along with it.

By the mid-2000s, though, most of these locally run services had been shut down — by then, we had clocks baked into our phones — and today, only a handful of Audichrons survive. Thanks to the National Institute of Standards and Technology, however, you can still call a phone number to get the time, and while the voice might not be the same, and long distance rates will apply, it’s definitely there, and you can use it. So on the off chance you happen to find yourself with no idea what time it is and only an analog phone line in reach, fear not — old-school telephone tech has your back. For now. Assuming you remember the number.

Recalling all this stuff did make me wonder, though, what a more modern version of a speaking clock might look like. So in this post, we’re going to build one ourselves. We won’t use an actual phone number, but we will use Pulumi and AWS — and because we want to do it right, we’ll take a test-driven approach to developing the infrastructure with Jest, the JavaScript testing framework. We’ll use TypeScript and Node.js for everything, focus on unit tests, and when we’re done, we’ll have a single, serverless, browser-friendly HTTPS endpoint that returns an MP3 audio stream that speaks the current time.

Let’s get started.

Sketching it out

The first thing we’ll need is a runtime environment — someplace to run some server-side JavaScript that can render and deliver an audio file. Until recently, the easiest way to get an HTTP endpoint up and running on AWS has generally been with AWS Lambda and API Gateway, using Lambda to run the requisite code and API Gateway to expose the Lambda to the internet. Pulumi Crosswalk actually makes this really easy, too — but with the release of AWS Lambda Function URLs this April, we now have another option, one that doesn’t need API Gateway at all.

A Lambda Function URL is just what it sounds like: a URL that exposes a Lambda function. Specifically, it’s an AWS cloud resource that consists of a few properties that tell AWS whether to allow anonymous access to the function or to protect it with AWS IAM, and optionally, you can also provide a few cross-origin resource-sharing (CORS) rules to provide (or restrict) access by websites running on other domains.

For an app like this one, a Function URL is a good fit, as it’s simple, quick to deploy (and update), and it gives us all the control we need to get the job done. Architecturally, then, our little app will look something like this:

That about covers the infrastructure — but what about the sound?

As it happens, Google Translate offers a publicly accessible text-to-speech API that can, as long as you don’t overuse it, render a string to MP3 audio. We can use that API to generate the time-of-day message, tack the sound of a beep onto the end of it (for maximum nostalgic effect), and voilà: instant, infinitely scalable, cloud-native speaking clock.

Plan in place, we can kick things off by creating a new Pulumi project.

Create a new TypeScript project

Start by creating a new AWS TypeScript project in the usual way:

$ mkdir audichron-2022 && cd audichron-2022
$ pulumi new aws-typescript

Step through the prompts to create a new stack (you’ll only need one stack for this project), and when the new-project wizard completes, clear out the contents of index.ts entirely, as we’ll be building this program entirely from the ground up.

Install and configure Jest

Since we’re working with TypeScript, we can use ts-jest, which conveniently brings Jest along for the ride:

$ npm install ts-jest @types/jest --save-dev

Jest requires a bit of configuration, so add a new file at the root of the project, called jest.config.ts, with the following code:

import type { Config } from "@jest/types";

const config: Config.InitialOptions = {
 preset: "ts-jest",
};

export default config;

With Jest now installed and configured, and its helpers (like describe and expect) declared globally, you can create a new spec file to hold your tests. Add another file at the root of the project called index.spec.ts, and give it the following temporary content:

import "jest";

describe("My speaking clock", () => {
 it ("works", () => {
 expect(true).toBeTruthy;
 });
});

Lastly, in package.json, add a scripts block with a helper to make running the tests a little easier:

{
 "name": "audichron-2022",
 "devDependencies": {
 "@types/node": "^14",
 "ts-jest": "^28.0.3"
 },
 ...
 "scripts": {
 "test": "jest"
 }
}

Now run the tests, just to make sure you’ve got everything wired up correctly. If all goes well, you should see that the temporary spec we just added happily passes:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 ✓ works

With that, it’s time to get to start writing some real tests.

Start with some failing tests

As devoted practitioners of test-driven development, we’re going to start by writing some unit tests — specifically, some failing unit tests that we can fix by writing the Pulumi code to make them pass.

Recall that our design requires just two cloud resources: a Lambda function and a Lambda function URL. For the function itself, we’ll use the high-level aws.lambda.CallbackFunction resource, one of my favorites for managing for Lambdas because it requires only one property: an inline JavaScript function to handle the event that triggers the Lambda.

For the URL resource — the eventual triggerer of that event — you’ll use an aws.lambda.FunctionURL configured to make the Lambda publicly accessible (i.e., available to anyone on the internet) and embeddable on any domain, making it easy, for example, to embed the URL as the src attribute of an HTML5 audio element.

So to get things going, we’ll need to:

Configure Pulumi to mock AWS resources. We’re writing unit tests, after all, and we want them to be fast, so we’ll need to prevent those tests from provisioning any real cloud infrastructure.
Import your Pulumi resource declarations — the function and the function URL — into index.spec.ts so you can reference them in tests.

In that light, replace the contents of index.spec.ts with the following code, which, after some setup to switch Pulumi into unit-testing mode, contains just one test: a check to make sure that the function URL is configured with the right authorizationType. I’ve included some comments to help explain what each line is doing:

import * as pulumi from "@pulumi/pulumi";
import "jest";

describe("My speaking clock", () => {

 // Define the infra variable as a type whose shape matches that of the
 // to-be-defined resources module.
 // https://www.typescriptlang.org/docs/handbook/2/typeof-types.html
 let infra: typeof import("./resources");

 beforeAll(() => {

 // Put Pulumi in unit-test mode, mocking all calls to cloud-provider APIs.
 pulumi.runtime.setMocks({

 // Mock requests to provision cloud resources and return a canned response.
 newResource: (args: pulumi.runtime.MockResourceArgs): {id: string, state: any} => {

 // Here, we're returning a same-shaped object for all resource types.
 // We could, however, use the arguments passed into this function to
 // customize the mocked-out properties of a particular resource based
 // on its type. See the unit-testing docs for details:
 // https://www.pulumi.com/docs/iac/concepts/testing/unit
 return {
 id: `${args.name}-id`,
 state: args.inputs,
 };
 },

 // Mock function calls and return whatever input properties were provided.
 call: (args: pulumi.runtime.MockCallArgs) => {
 return args.inputs;
 },
 });
 });

 beforeEach(async () => {

 // Dynamically import the resources module.
 // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#dynamic_imports
 infra = await import("./resources");
 });

 describe("function URL", () => {

 it("is publicly accessible", (done) => {
 infra.timeURL.authorizationType.apply(authType => {
 try {
 expect(authType).toBe("NONE");
 done();
 } catch (err) {
 done(err);
 }
 });
 });
 });
});

We can’t quite run these tests just yet, though, as they rely on a module called resources that we haven’t yet defined. Let’s do that now by creating a new file, resources.ts, containing some stubs for the cloud resources we’ve elected to use:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

export const timeFunction = new aws.lambda.CallbackFunction("time-function", {
 callback: () => {
 // Leave the callback implementation empty for now.
 },
});

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "", // Leave this empty, too. (We want our first test to fail.)
});

Now run the tests, and you should see that the public-accessibility test fails — which shouldn’t be surprising, as we haven’t yet supplied an acceptable value:

$ npm test

FAIL ./index.spec.ts
 My speaking clock
 function URL
 ✕ is publicly accessible

 ● My speaking clock › function URL › is publicly accessible

 expect(received).toBe(expected) // Object.is equality

 Expected: "NONE"
 Received: ""

Now let’s turn that sad-looking ❌ into a ✅.

Make the tests pass

Update the FunctionUrl resource to change its authorizationType input to NONE (meaning for anonymous access, no authorization required):

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
- authorizationType: "",
+ authorizationType: "NONE",
});

Once you do that, you should be able to run the test suite again, and this time, see it pass:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible

Excellent: you’re now well on your way to writing more tests with Jest — and we’ll do that in a moment. But first, let’s digress momentarily to explore a few other ways of writing Jest specs with Pulumi.

Tidy up the specs

If you’ve written JavaScript tests before, particularly with tools like Jest, Jasmine, and Mocha, there’s a good chance you’ve gotten used to writing those tests in a particular way. And if you’re like me, you might’ve raised a bit of an eyebrow when you saw how that first test was written. Here it is again, this time with comments that capture the questions I might’ve had myself if I were seeing this code for the first time:

it("is publicly accessible", (done) => {
 infra.timeURL.authorizationType.apply(auth => { // Why do I have to use .apply() here?
 try { // What happens if I don't try/catch?
 expect(auth).toBe("NONE"); // Is it possible to use expect() by itself?
 done(); // Do I have to call done()? What if I don't?
 } catch (err) { // How can I make these tests more readable?
 done(err); // Is all of this code necessary?
 }
 });
});

This is a perfectly valid test, of course; we know because we just ran it. Still, you might wonder, especially since we’re only checking one property, whether this test might be written a bit more concisely — something more like this, maybe:

it("is publicly accessible", () => {
 expect(infra.timeURL.authorizationType).toBe("NONE"); // Alas, nope.
});

The main reason is the asynchronous nature of the objects we’re testing. An aws.lambda.FunctionUrl, after all, isn’t just a plain ol’ JavaScript object — it’s a Pulumi CustomResource, a representation of an eventual resource running in the cloud, with properties that may not be known until sometime after the resource has been deployed. While it’s true we provided authorizationType as a regular JavaScript string, by the time we attempt to read it in the test, it’s been transformed into a pulumi.Output<string> — an asynchronous value, like a promise, that has to be unwrapped with a call to apply() before it can be read.

However, just adding apply and calling expect from within its callback doesn’t quite work, either:

it("is publicly accessible", () => {
 infra.timeURL.authorizationType.apply(authType => {
 expect(authType).toBe("NONE"); // Also nope.
 });
});

Although that might not be obvious at first, since on the surface, the test appears to pass:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible

It isn’t until you force the test to fail that it becomes clearer something’s gone sideways. Try changing the expected authorizationType to something other than NONE and you’ll see what I mean:

$ npm test

PASS ./index.spec.ts
 My time app
 function URL
 ✓ is publicly accessible

Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 total

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function
without a catch block, or by rejecting a promise which was not handled with .catch(). The promise
rejected with the reason "Error: expect(received).toBe(expected) // Object.is equality

Expected: "AWS_IAM"
Received: "NONE"".] {
 code: 'ERR_UNHANDLED_REJECTION'
}

All is not lost, though. We do have other options, and that UnhandledPromiseRejection offers a bit of a clue.

Turn outputs into promises

Jest actually has built-in support for writing asynchronous tests with promises, and there are a couple of ways you can write specs that use them:

Have the function passed to it (or its alias, test) return a promise containing your expectation
Pass an async function to it and await the promise in the usual way

There’s only one problem: pulumi.Outputs may indeed be promise-like, but they aren’t actual promises.

They can, however, be wrapped in promises pretty easily. I like the way Pulumi community member @aviflax handled this with a little utility function that wraps the Output and preserves the type of its underlying value. Here’s that function, followed by a couple of examples that use it:

// Convert a pulumi.Output to a promise of the same type.
function promiseOf<T>(output: pulumi.Output<T>): Promise<T> {
 return new Promise(resolve => output.apply(resolve));
}

// Our test, rewritten to have it() return a Promise<string>.
it("is publicly accessible", () => {
 return promiseOf(infra.timeURL.authorizationType).then(authType => {
 expect(authType).toBe("NONE");
 });
});

// The same test, rewritten to use async/await.
it("is publicly accessible", async () => { // <-- Don't miss that async!
 const authType = await promiseOf(infra.timeURL.authorizationType);
 expect(authType).toBe("NONE");
});

Either approach works equally well, but I’m slightly more partial to the latter for its conciseness. So for now, update index.spec.ts to add the promiseOf helper and adjust the test accordingly, then run the suite one more time to make sure all’s well:

// ...

// Convert a pulumi.Output to a promise of the same type.
function promiseOf<T>(output: pulumi.Output<T>): Promise<T> {
 return new Promise(resolve => output.apply(resolve));
}

describe("My speaking clock", () => {
 // ...

 describe("function URL", () => {

 it("is publicly accessible", async () => {
 const authType = await promiseOf(infra.timeURL.authorizationType);
 expect(authType).toBe("NONE");
 });
 });
});

Assuming you’re back in ✅ territory, let’s move on so we can finish things off.

Finish the app

We’re in good shape: we’ve got scaffolding for our Jest-powered unit tests and we’re generally happy with how they look, so it’s time to round out the rest of the app and its tests so we can deploy. We’ll go through this last part briskly, as we’ve covered most of the important stuff at this point, and pause only briefly to point out what’s important as we go.

Here’s the plan:

We’ll add one more test to make sure the function URL is CORS-friendly
We’ll add another test to make sure the URL resource is bound to the right Lambda function
We’ll add the Lambda code to generate and return the audio file
We’ll deploy, and bask in the warm glow of nostalgia

Let’s get to it.

Add the remaining tests

To index.spec.ts, add two more tests, one to ensure the function is accessible to browsers running on any domain, the other to verify the function URL resource is exposing the right function (if we didn’t, it’d be easy to change that reference by mistake and accidentally expose the wrong function):

describe("My speaking clock", () => {
 // ...

 describe("function URL", () => {
 // ...

 it("is CORS-friendly", async () => {
 const authType = await promiseOf(infra.timeURL.cors);
 expect(authType).toEqual({
 allowOrigins: ["*"],
 allowMethods: ["GET"],
 });
 });

 it("is bound to the right Lambda function", async () => {
 const timeFuncName = await promiseOf(infra.timeFunction.name);
 const timeURLFunc = await promiseOf(infra.timeURL.functionName);
 expect(timeFuncName).toEqual(timeURLFunc);
 });
 });
});

Run the tests, and you should see one failure:

$ npm test

FAIL ./index.spec.ts (14.256 s)
 My speaking clock
 function URL
 ✓ is publicly accessible (3022 ms)
 ✕ is CORS-friendly (3 ms)
 ✓ is bound to the right Lambda function (1 ms)

 Expected: {"allowMethods": ["GET"], "allowOrigins": ["*"]}
 Received: undefined

Tests: 1 failed, 2 passed, 3 total

A quick addition to the FunctionUrl in resources.ts should get you to green:

// ...
export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "NONE",
+ cors: {
+ allowOrigins: ["*"],
+ allowMethods: ["GET"],
+ },
});

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible
 ✓ is CORS-friendly
 ✓ is bound to the right Lambda function

Tests: 3 passed, 3 total

That’s it for the tests! Now for the final additions.

Produce the audio message

The last thing we’ll need to do is render some audio. For this, we’ll need to communicate with Google Translate, our text-to-speech provider of choice, and that means we’ll need an HTTP client. I generally like node-fetch, so let’s go with that:

npm install --save node-fetch@2

Now, finish the program by importing node-fetch into resources.ts, adding a function to produce a speech-friendly string indicating the current time and updating the CallbackFunction’s callback property to send the string to Google Translate to produce the desired result. Here’s the complete file, with comments, for reference:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const fetch = require("node-fetch");

// Convert the current time into a speech-friendly string.
export function getSpeechText() {
 const now = new Date();
 now.setSeconds(now.getSeconds() + 6);

 const local = new Date(now.toLocaleString("en-US", { timeZone: "America/Los_Angeles" }));
 const localHour = local.getHours();
 const localMinute = local.getMinutes();
 const h = localHour > 12 ? localHour - 12 : localHour;
 const m = localMinute < 10 ? `oh ${localMinute}` : localMinute; // So 2:03 -> "two-oh-three"
 const s = local.getSeconds();

 return `At the tone, the time will be ${h} ${m}. And ${s} seconds.`;
}

export const timeFunction = new aws.lambda.CallbackFunction("time-function", {

 // Update the Lambda callback body to convert the current time into an MP3 file.
 callback: async () => {
 const text = getSpeechText();
 const speechURL = `https://translate.google.com/translate_tts?ie=UTF-8&q=${encodeURIComponent(text)}&textLen=${text.length}&tl=en&client=tw-ob`;
 const beepURL = "https://www.pulumi.com/uploads/beep.mp3";
 const ttsResponse = await fetch(speechURL);
 const beepResponse = await fetch(beepURL);
 const speech = await ttsResponse.arrayBuffer();
 const beep = await beepResponse.arrayBuffer();

 // Tack a beep onto the end of the audio returned from Google Translate, then
 // render the whole thing as a base-64 encoded string.
 const body = Buffer.concat([Buffer.from(speech), Buffer.from(beep)]).toString("base64");

 // Return an appropriately shaped HTTP response.
 return {
 body,
 headers: { "Content-Type": "audio/mpeg" },
 isBase64Encoded: true,
 statusCode: 200,
 };
 },
});

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "NONE",
 cors: {
 allowOrigins: ["*"],
 allowMethods: ["GET"],
 },
});

// Export the public URL of our shiny new service.
export const audioURL = timeURL.functionUrl;

Finally, open index.ts (which should still be empty) and add a couple of lines to import the resources module and export the function URL as a Pulumi stack output:

import "./resources";

export { audioURL } from "./resources";

And that’s it. We’re ready to deploy.

Deploy the service

Doublecheck your AWS credentials are configured, then deploy the new service with a single pulumi up:

$ pulumi up

...
Updating (dev)

View Live: https://app.pulumi.com/cnunciato/audichron-2022/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack audichron-2022-dev created
 + ├─ aws:iam:Role time-function created
 + ├─ aws:lambda:Function time-function created
 + ├─ aws:iam:RolePolicyAttachment time-function-7cd09230 created
 + ├─ aws:iam:RolePolicyAttachment time-function-a1de8170 created
 + ├─ aws:iam:RolePolicyAttachment time-function-1b4caae3 created
 + ├─ aws:iam:RolePolicyAttachment time-function-019020e7 created
 + ├─ aws:iam:RolePolicyAttachment time-function-4aaabb8e created
 + ├─ aws:iam:RolePolicyAttachment time-function-74d12784 created
 + ├─ aws:iam:RolePolicyAttachment time-function-e1a3786d created
 + ├─ aws:iam:RolePolicyAttachment time-function-6c156834 created
 + ├─ aws:iam:RolePolicyAttachment time-function-b5aeb6b6 created
 + └─ aws:lambda:FunctionUrl time-url created

Outputs:
 audioURL: "https://52rjmmybwfwcud3nxxhosznpum0cycbw.lambda-url.us-west-2.on.aws/"

Resources:
 + 13 created

Duration: 20s

When the update completes, open the URL in your browser of choice, and you should hear the message you’ve been waiting for — one that might even make Pat Fleet proud:

$ open $(pulumi stack output audioURL)

When you’re ready, you can tear everything down with a pulumi destroy.

Wrapping up

Beyond just having a nifty (and admittedly rather silly) new way to tell time, you should have a much better sense at this point of how you can use Pulumi with Jest to write better, safer infrastructure code.

From here, there’s a bunch more you might think about next: writing more tests to cover the code we just added, exploring some additional flavors of testing in the docs, or having a look at a few examples. You’ll find the full source for this walkthrough up on GitHub as well.

Happy testing!

Deploy MERN Stack on DigitalOcean with Pulumi [Guide]

Christian Nunciato — Fri, 11 Mar 2022 00:00:00 +0000

As a developer, I get lots of ideas for web apps—little things, mostly: nifty ways to keep track of my kids’ allowances, habit trackers, shopping lists. Most of them, however, never see the light of day, and not just because I’m lazy; I also tend to get hung up trying to decide what to use for the technology stack.

And as a JavaScript developer, I certainly have options—too many, in fact, and that’s part of the problem. Having roughly a hundred and fifty million libraries and frameworks to choose from is definitely better than having none, but at the same time, all that choice can make the actual choosing rather difficult. Which is why, when I just want to get something done, I’ll often reach for a combination of tools known as the MERN stack.

MERN-stack apps are three-tier web apps built with MongoDB, Express, React, and Node.js. You can read all about them in the MongoDB docs, but the gist is that they allow you use one language—JavaScript (or TypeScript, if you like)—to manage all three layers of the application stack: the front end as a single-page app built statically with React, the back end as a REST API managed with Express, and the database as a collection of JSON-like documents with MongoDB. MERN might not always the right tool for the job, but for the kinds of apps I tend to find myself building, it generally works out pretty well.

Still, once I’m finished building my app, I’m often faced with a whole other problem: figuring out how to get the app off of my laptop and onto the web.

The cloud hasn’t made this an easy task for developers. Choosing a cloud provider, deciding which resources to use (and how to use them), setting up networking, debugging permissions, navigating billing, and all the rest, can be overwhelming—and that’s before you’ve given a single thought to anything having to do with automation or infrastructure as code. What we want, I think, is to be able to focus on our apps, and when we’re ready to ship, push our code to a repository and wait patiently for a URL to emerge that we can paste into a browser and have everything just work.

Which is why I was so delighted when I discovered DigitalOcean’s App Platform.

If you’ve used DigitalOcean before, you know it’s all about making infrastructure more accessible to developers. What you may not know, though, or at least I didn’t myself until recently, is that you can do a lot more with DigitalOcean than just virtual machines. A fairly new, fully-managed platform service (think Heroku), App Platform gives you a set of high-level abstractions built to align with the tiers of a typical web application, which means you can focus on what you care about most—your app—and leave the infrastructure and its management to someone else. It’s a compelling option for anyone looking to deploy and manage any web application (MERN or otherwise), and as you’ll see, with Pulumi and a little bit of code, you can easily do so without ever having to leave the comfort of your IDE.

So let’s build ourselves a MERN app and deploy it on DigitalOcean with Pulumi. Given the goal is to focus primarily on the infrastructure and how to code it, we’ll start with a pre-baked web application (a simple grocery list), and we’ll map its tiers to App Platform constructs and wire everything up with Pulumi.

First steps: setting up

The code for this walkthrough is available as a template repository on GitHub, so if you want to follow along (and you should!), you should grab a copy of your own to work with by forking the repository or creating a new one from the template. Once you’ve done that, you should also:

Clone the repository to your local machine.
Install Pulumi and Node.js.
Sign into DigitalOcean and obtain a personal access token with read-write permissions.
Grant DigitalOcean access to your GitHub repository by visiting the Apps page, choosing Create App, and following the steps to install DigitalOcean’s GitHub app.
Optionally, if you’d like to develop the application locally as well, install and configure MongoDB Community Edition.

One thing to note: Since we’ll be provisioning real DigitalOcean resources, there’s a chance you could incur a slight cost for what you use. However, as we’ll be using the least expensive plan settings available, and tearing everything down when we’re through, that cost shouldn’t amount to more than a few pennies or so.

Let’s get started.

Cloning and inspecting the repository

Once you’ve cloned your copy of the template repository and navigated to the root, you’ll see a couple of files and folders that look something like this:

├── frontend
├── backend
└── package.json

The frontend folder contains the React application, and its job is to render the list of groceries and give you something to interact with (to add items, check them off, delete them, and so on). The scaffolding for the app was generated with a tool called Vite, and all of its logic—form fields, click handlers, API calls, etc.—is contained in src/App.tsx.

The backend folder contains the Express application that defines the REST API. It sets up four API routes to handle the CRUD operations you’d expect in an app like this one:

GET /api/items fetches all items from the database and returns them as a JSON array.
POST /api/items accepts a new item and writes it to the database.
PUT /api/items/:id updates an existing item (to toggle its checked/unchecked status).
DELETE /api/items/:id deletes an item from the database.

The back end supports three configurable properties as well, all of which are exposed as optional environment variables:

BACKEND_SERVICE_PORT, which defaults to 8000
BACKEND_ROUTE_PREFIX, which defaults to /api
DATABASE_URL, which defaults to mongodb://127.0.0.1 for local development

Running the application locally (optional)

You don’t have to do this, but if you’d like to, here’s how. After installing MongoDB and starting the service (which should be listening by default on port 27107), you can install all front-end and back-end dependencies and start the development server:

$ npm install
$ npm start

With the development server running, you can browse to http://localhost:3000 and see the app:

The front-end and back-end dev servers are set up to compile your TypeScript to JavaScript automatically, and the front-end server is configured to proxy the back-end service (which runs at http://localhost:8000) at a root-relative path of /api. Proxying the API in this way lets you avoid having to wrestle with CORS-related issues, and as you’ll see when we deploy to DigitalOcean later, App Platform conveniently supports the same configuration out of the box.

Try adding a few items and marking them off, just to make sure everything’s working as expected. If you’ve got a MongoDB client installed as well—I generally use MongoDB Compass—you should be able to find the grocery-list database and see the items collection filling up with delicious foods:

Now let’s have a look at how to go about deploying this stuff.

Charting a course

Earlier I mentioned that every cloud provider handles application deployment a little differently, sometimes in multiple ways, and that’s true for DigitalOcean as well. You could deploy the front end as a DigitalOcean Space, or both the front end and back end (and even the database) as a DigitalOcean Droplet. But given the shape of this application, the best fit is really is App Platform, for several reasons.

One is that because App Platform apps are comprised of these high-level components—abstractions like static site, service, and database—it’s pretty much purpose-built for an application like this one, and DigitalOcean customizes the deployment of each component based on its type. Static websites are distributed and cached on DigitalOcean’s CDN, services are packaged and delivered as containers (with its Kubernetes platform), and databases are deployed as configurable managed services. All of this means you’re not only able to stay focused on the application itself, but you’re able to scale each one of these components up or down however you like, and even delegate your front-end and back-end build processes to DigitalOcean to be handled in response to commits on one or more external Git repositories.

App Platform apps can be configured in one of two ways: manually, by configuring their components individually in the DigitalOcean web console, or programmatically, in the form of an App Platform spec, a JSON document submitted over DigitalOcean’s REST API. In our case, we’ll indirectly go the latter route, using Pulumi with the DigitalOcean provider package to define an app spec comprised of three components:

A staticSite component mapped to the frontend folder
A service component mapped to the backend folder
A database component mapped to a managed MongoDB cluster (which we’ll also configure to be accessible only by the service component)

And once deployed, it’ll all be available at a single DigitalOcean-provided URL.

Let’s begin by creating new Pulumi project.

Creating the project

In the root of the repository, make a new folder called infra, change to it, then run pulumi new using the digitalocean project template:

$ mkdir infra && cd infra
$ pulumi new digitalocean-typescript

$ mkdir infra && cd infra
$ pulumi new digitalocean-python

At the prompts, use the following values:

For project name, use grocery-list
For description, use Deploying a MERN-stack app on DigitalOcean
For stack name, use dev, the default

When the command completes, you’ll have a new Pulumi stack, but you’ll still have a few things to configure. For one, the Pulumi DigitalOcean provider needs to be configured to communicate with DigitalOcean on your behalf (to provision your app and its various resources). For this, you can use the access token you obtained earlier from the DigitalOcean console, and you can apply it by setting a single environment variable:

$ export DIGITALOCEAN_TOKEN="your-access-token"

Also, since one of our goals is to have App Platform deploy automatically on every GitHub commit, you’ll need to tell DigitalOcean where to find the source code for your front- and back-end components. You could bake these settings right into the Pulumi program itself, but it’d be better to apply them as stack-specific configuration settings, as that’d let you deploy to different stacks later (say, in CI) based on the branch of the commit. Everyone does this a little differently, so for now, let’s configure the currently selected stack (which should be dev) to use the default branch of your GitHub repository:

$ pulumi config set repo "your-github-org/your-github-repo" # e.g., cnunciato/fullstack-pulumi-mern-digitalocean
$ pulumi config set branch "your-main-branch" # e.g., main

With these values in place, you’re ready to start writing the program.

App Platform also supports GitLab and other Git-based repositories as well. See the App Specification docs for details.

Writing the program

In your IDE of choice, open index.js index.ts main.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf and replace the sample code with the following lines to import the Pulumi and DigitalOcean SDKs and the configuration values you just set, and add a line to specify the DigitalOcean region to deploy into:

import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";

// Our stack-specific configuration.
const config = new pulumi.Config();
const repo = config.require("repo");
const branch = config.require("branch");

// The DigitalOcean region to deploy into.
const region = digitalocean.Region.SFO3;

import pulumi
import pulumi_digitalocean as digitalocean

# Our stack-specific configuration.
config = pulumi.Config()
repo = config.require("repo")
branch = config.require("branch")

# The DigitalOcean region to deploy into.
region = digitalocean.Region.SFO3

Next, add a few lines to declare the managed MongoDB cluster. We’ll use just one node for now—additional replica nodes can easily be added later by increasing the nodeCount value—and go with the least expensive performance settings:

// ...

// Our MongoDB cluster (currently just one node).
const cluster = new digitalocean.DatabaseCluster("cluster", {
 engine: "mongodb",
 version: "5",
 region,
 size: digitalocean.DatabaseSlug.DB_1VPCU1GB,
 nodeCount: 1,
});

// The database we'll use for our grocery list.
const db = new digitalocean.DatabaseDb("db", {
 name: "grocery-list",
 clusterId: cluster.id,
});

# ...

# Our MongoDB cluster (currently just one node).
cluster = digitalocean.DatabaseCluster("cluster", digitalocean.DatabaseClusterArgs(
 engine = "mongodb",
 version = "5",
 region = region,
 size = digitalocean.DatabaseSlug.D_B_1_VPCU1_GB,
 node_count = 1
))

# The database we'll use for our grocery list.
db = digitalocean.DatabaseDb("db", digitalocean.DatabaseDbArgs(
 name = "grocery-list",
 cluster_id = cluster.id
))

Now for the App Platform spec itself. Notice the digitalocean.App resource takes just one argument, spec, which defines all three of the components of the app: static site, service, and database. Both the static site and the service are configured to use the same GitHub repository (the sourceDir properties indicate their folders within the repository), and both are configured (via the deployOnPush flag) to be rebuilt and redeployed by DigitalOcean on every commit.

The service has a few additional settings that you can use to manage its runtime behavior and deployment topology as well. As in development, we’ll configure the service to listen on port 8000 and be available at /api—the entire app will ultimately be proxied transparently by an App Platform load balancer—and it’ll be powered by just one container instance, again using the least expensive performance settings:

// ...

// The App Platform spec that defines our grocery list.
const app = new digitalocean.App("app", {
 spec: {
 name: "grocery-list",
 region: region,

 // The React front end.
 staticSites: [
 {
 name: "frontend",
 github: {
 repo,
 branch,
 deployOnPush: true,
 },
 sourceDir: "/frontend",
 buildCommand: "npm install && npm run build",
 outputDir: "/dist",
 }
 ],

 // The Express back end.
 services: [
 {
 name: "backend",
 github: {
 repo,
 branch,
 deployOnPush: true,
 },
 sourceDir: "/backend",
 buildCommand: "npm install && npm run build",
 runCommand: "npm start",
 httpPort: 8000,
 routes: [
 {
 path: "/api",
 preservePathPrefix: true,
 },
 ],
 instanceSizeSlug: "basic-xxs",
 instanceCount: 1,

 // To connect to MongoDB, the service needs a DATABASE_URL, which
 // is conveniently exposed as an environment variable thanks to its
 // membership in this app spec (below).
 envs: [
 {
 key: "DATABASE_URL",
 scope: "RUN_AND_BUILD_TIME",
 value: "${db.DATABASE_URL}",
 },
 ],
 },
 ],

 // Include the MongoDB cluster as an integrated App Platform component.
 databases: [
 {
 // The name `db` defines the prefix of the tokens used (above) to
 // read the environment variables exposed by the database cluster.
 name: "db",

 // MongoDB clusters are only available in "production" mode.
 // https://docs.digitalocean.com/products/app-platform/how-to/manage-databases/
 production: true,

 // A reference to the `DatabaseCluster` we declared above.
 clusterName: cluster.name,

 // The engine value must be uppercase, so we transform it with JS.
 engine: cluster.engine.apply(engine => engine.toUpperCase()),
 }
 ]
 },
});

# ...

# The App Platform spec that defines our grocery list.
app = digitalocean.App("app", digitalocean.AppArgs(
 spec = digitalocean.AppSpecArgs(
 name = "grocery-list",
 region = region,

 # The React front end.
 static_sites = [
 digitalocean.AppSpecStaticSiteArgs(
 name = "frontend",
 github = digitalocean.AppSpecJobGithubArgs(
 repo = repo,
 branch = branch,
 deploy_on_push = True
 ),
 source_dir = "/frontend",
 build_command = "npm install && npm run build",
 output_dir = "/dist"
 )
 ],

 # The Express back end.
 services = [
 digitalocean.AppSpecServiceArgs(
 name = "backend",
 github = digitalocean.AppSpecJobGithubArgs(
 repo = repo,
 branch = branch,
 deploy_on_push = True
 ),
 source_dir = "/backend",
 build_command = "npm install && npm run build",
 run_command = "npm start",
 http_port = 8000,
 routes = [
 digitalocean.AppSpecServiceRouteArgs(
 path = "/api",
 preserve_path_prefix = True
 )
 ],
 instance_size_slug = "basic-xxs",
 instance_count = 1,

 # To connect to MongoDB, the service needs a DATABASE_URL, which
 # is conveniently exposed as an environment variable because the
 # database belongs to the app (see below).
 envs = [
 digitalocean.AppSpecServiceEnvArgs(
 key = "DATABASE_URL",
 scope = "RUN_AND_BUILD_TIME",
 value = "${db.DATABASE_URL}"
 )
 ]
 )
 ],

 # Include the MongoDB cluster as an integrated App Platform component.
 databases = [
 digitalocean.AppSpecDatabaseArgs(
 # The `db` name defines the prefix of the tokens used (above) to
 # read the environment variables exposed by the database cluster.
 name = "db",

 # MongoDB clusters are only available in "production" mode.
 # https://docs.digitalocean.com/products/app-platform/concepts/database/
 production = True,

 # A reference to the managed cluster we declared above.
 cluster_name = cluster.name,

 # The engine value must be uppercase, so we transform it with Python.
 engine = cluster.engine.apply(lambda engine: engine.upper())
 )
 ]
 ),
))

Technically that’s all we need to configure the application—but it wouldn’t be a bad idea to add one last thing.

By default, managed MongoDB clusters are configured to be publicly accessible—which is great if you need to be able to connect one yourself, but not so great as a strategy for preventing internet miscreants from doing the same. You can fix this easily by adding a DatabaseFirewall resource to declare the app as a trusted source, thereby rejecting all inbound traffic originating from elsewhere:

// ...

// Adding a database firewall setting grants access solely to our app.
const trustedSource = new digitalocean.DatabaseFirewall("trusted-source", {
 clusterId: cluster.id,
 rules: [
 {
 type: "app",
 value: app.id,
 },
 ],
});

# ...

# Adding a database firewall setting restricts access solely to our app.
trusted_source = digitalocean.DatabaseFirewall("trusted-source", digitalocean.DatabaseFirewallArgs(
 cluster_id = cluster.id,
 rules = [
 digitalocean.DatabaseFirewallRuleArgs(
 type = "app",
 value = app.id
 )
 ],
))

And finally, add one last line to export the app URL, to be generated by DigitalOcean, as a Pulumi stack output:

// ...

// The DigitalOcean-assigned URL for our app.
export const { liveUrl } = app;

# ...

# The DigitalOcean-assigned URL for our app.
pulumi.export("liveUrl", app.live_url)

With that, you’re ready to deploy.

Deploying

Quickly, to recap, here’s what we’ve done so far:

We took a pre-baked MERN app configured to run on localhost.
We mapped the tiers of that app to their corresponding App Platform components.
We wrote a Pulumi program to codify that mapping as an App Platform spec and a managed MongoDB cluster to go along with it.

When you deploy this app in a moment with pulumi up, Pulumi will provision a new MongoDB cluster (which usually takes a few minutes), and then once that’s available, DigitalOcean will take our spec and use it to fetch the components of the app from GitHub at the specified branch and build them. From that point forward, any commit you make to that branch will trigger DigitalOcean to fetch, rebuild, and redeploy the app automatically.

Make sure you’ve installed the DigitalOcean GitHub app as described above—you should see it listed at https://github.com/settings/installations:

Now return to the command line and run pulumi up:

$ pulumi up

Previewing update (dev)

View Live: https://app.pulumi.com/cnunciato/grocery-list/dev/previews/605bf32a-95b1-4221-bc35-0e667b30f38a

 Type Name Plan
 + pulumi:pulumi:Stack grocery-list-dev create
 + ├─ digitalocean:index:DatabaseCluster cluster create
 + ├─ digitalocean:index:DatabaseDb db create
 + ├─ digitalocean:index:App app create
 + └─ digitalocean:index:DatabaseFirewall trusted-source create

Resources:
 + 5 to create

Do you want to perform this update?
> yes

Updating (dev)

View Live: https://app.pulumi.com/cnunciato/grocery-list/dev/updates/1

 Type Name Status
 pulumi:pulumi:Stack grocery-list-dev
 + ├─ digitalocean:index:App app created
 + └─ digitalocean:index:DatabaseFirewall trusted-source created

Outputs:
 + liveUrl: "https://grocery-list-fb9bd.ondigitalocean.app"

Resources:
 + 2 created
 3 unchanged

Duration: 2m30s

Again, it’ll probably take a few minutes to get everything spun up for the first time, but when the process completes, you’ll have a working app at the URL provided by DigitalOcean, emitted as a Pulumi stack output:

$ open $(pulumi stack output liveUrl)

$ open $(pulumi stack output live_url)

You should also be able to explore your shiny new grocery-list app in the DigitalOcean Console, with all three of its components (and their build-time and runtime logs!) now represented:

Now try making a commit to your repository (any commit will do, but ideally one to the frontend or backend folder), and watch as the app redeploys automatically:

And finally, try scaling the service by bumping the instanceCount from 1 to 2 in the code—or better, if you’re up for it, making that value configurable by stack:

 const config = new pulumi.Config();
 const repo = config.require("repo");
 const branch = config.require("branch");
+ const serviceInstanceCount = config.requireNumber("service_instance_count");
 ...
 services: [
 digitalocean.AppSpecServiceArgs(
 ...
+ instanceCount: serviceInstanceCount,

 config = pulumi.Config()
 repo = config.require("repo")
 branch = config.require("branch")
+ service_instance_count = config.requireNumber("service_instance_count");
 ...
 services: [
 digitalocean.AppSpecServiceArgs(
 ...
+ instance_count: service_instance_count,

Set the instance count with pulumi config:

$ pulumi config set service_instance_count 2

Then deploy the new app spec with a final pulumi up:

$ pulumi up

Updating (dev)

 Type Name Status Info
 pulumi:pulumi:Stack grocery-list-dev
 ~ └─ digitalocean:index:App app updated [diff: ~spec]

Outputs:
 liveUrl: "https://grocery-list-fb9bd.ondigitalocean.app"

Resources:
 ~ 1 updated
 4 unchanged

Duration: 1m27s

And when you’re finished experimenting, you can tear everything down in just a few seconds with a pulumi destroy:

$ pulumi destroy

Destroying (dev)

 Type Name Status
 - pulumi:pulumi:Stack grocery-list-dev deleted
 - ├─ digitalocean:index:DatabaseFirewall trusted-source deleted
 - ├─ digitalocean:index:DatabaseDb db deleted
 - ├─ digitalocean:index:App app deleted
 - └─ digitalocean:index:DatabaseCluster cluster deleted

Resources:
 - 5 deleted

Duration: 19s

Wrapping up, and next steps

Hopefully this gives you a sense of the kinds of things you can do with Pulumi and DigitalOcean—and I definitely encourage you to spend a little time with the App Platform docs to dig a bit deeper into some of these concepts and explore a few others we weren’t able to cover. You’ll find the full source for this walkthrough on GitHub, of course, with finished branch containing the completed Pulumi program for reference.

From here, you might think about:

Adding a digitalocean.DnsRecord to give your app a custom domain name.
Creating a second stack with pulumi stack init and adjusting the program to make the source branch configurable—a production stack, say, designed to deploy in response to commits to a release branch.
Using Pulumi’s GitHub Action to run previews and updates as part of a pull-request based workflow.

Happy coding!

Six Things You Might Not Know About the Pulumi Service

Chris Smith — Mon, 24 Jan 2022 00:00:00 +0000

As a reader of this blog, you’ve probably heard of the Pulumi Service, the default state-management backend of the Pulumi CLI. If that’s the case, there’s also a good chance you’ve heard of a number of the Service’s key features, like helping you organize your projects and stacks, collaborate with others with the help of organizations, or handle sensitive data securely with built-in support for encrypted secrets.

What you might not know, though, is that we’re adding new features to the Pulumi Service all the time, and that some of these features can be fairly easy to miss. So in this post, we’ll highlight a handful of the features you might not be aware of, and that we think make it even easier to manage your infrastructure with Pulumi.

Let’s get started!

Feature 1: Stack Tags

The first thing we’ll cover is the Pulumi stack tags feature. Stack tags are key/value pairs that are associated with your Pulumi stack. You can use them for any number of purposes, from keeping things organized to even flagging whether or not it’s okay for a script to reclaim the stack’s resources.

While you can create, edit, and remove stack tags using the Pulumi CLI (see pulumi stack tag), you can also do so in the Pulumi Console:

Once set, you can quickly filter stacks on your organization’s Projects tab as well.

Feature 2: Link Updates to your CI/CD pipeline

Another helpful thing the Pulumi Service can do is link the Pulumi stack update to the CI/CD job or run that was used to perform it, as well as to the specific source commit:

When Pulumi updates a stack, it will store some information about the local machine state, such the current git SHA if the stack resides in a git repository. It is with this data that the Pulumi Service links to relevant services where possible.

If you’re curious to know all of the information Pulumi has for a stack update, you can navigate to the Environment tab for that update:

Feature 3: Custom Update Messages

One of the most useful pieces of additional data stored with each update is the message property. By default, Pulumi will use whatever the latest git commit message is. But did you know you can customize that message from the command line?

By passing the --message (or -m) flag to pulumi up, you can provide a custom update message, which can help explain the context for why you made a particular infrastructure change:

$ pulumi up --message "Release the hounds\!"

Feature 4: Viewing Resource Changes

Another thing you might have missed is that the Pulumi Service supports multiple views for a stack update’s logs.

The standard view mimics the Pulumi CLI’s output. However, you can switch to the Diff or Diagnostic views, too.

The Diff view displays all of the properties of modified resources, along with their previous/updated values. This can be helpful when trying to understand exactly what happened for a stack update.

The Diagnostic view just outputs so-called “diagnostic” messages from the Pulumi update. Most commonly, this is where you will see the output of things like calls to JavaScript’s console.log(...) or Python’s print(...) functions. It’ll also include the output of dependent tools, like the output when building Docker containers.

Feature 5: Stack Resource Visualization

Another cool feature of the Pulumi Service is being able to visualize a stack’s resources and their relationships to one another. This is a really important feature!

One of the advantages of using an infrastructure as code tool like Pulumi is that you aren’t just creating a flat list of resources. Instead, by the virtue of how the code and resource graph is structured, you are building a graph of resources.

This means that Pulumi has a semantic understanding of not only what resources are in your stack but also how they relate to one another.

If you navigate to a stack’s Resources tab, by default you see them listed in a table. You can filter this list by resource type, or by providing a custom filter:

However, in order to see the expanded resource graph, click the Graph View button. This will switch to rendering the resources as an interactive graph.

Notice here that the resources that are part of a component are grouped together. Also, you can double click on those nodes in the graph to expand or collapse them!

Feature 6: Drilling into Resource Properties

But you don’t need to just settle for a “top-level view” of your resources. You can click on any specific resource on the Resources tab, and see all of the properties of that resource.

This allows you to see all of the properties that Pulumi is tracking for that resource. It will also show you any relationships to other resources in the same stack.

The Pulumi Service also links you to the resource’s cloud provider’s console where possible, so you can see even more details.

Is your resource missing a link to the cloud provider console? With new services being launched from the myriad providers Pulumi supports every day, it’s hard to keep up. So if you would like us to add cloud provider links, please file an issue over in the pulumi/console-requests GitHub repository so we can update our database.

There are plenty of other features to check out in the Pulumi Service, and we have exciting improvements on the way! As always, feel free to stop by the Pulumi Community Slack to learn more, ask questions, or share anything cool you’re up to!

API Gateway to EventBridge with Pulumi

Christian Nunciato — Wed, 12 Jan 2022 13:00:00 -0700

If you’re familiar with Amazon API Gateway, you know it’s all about making it easier to provision and manage a web API. Maybe you’ve used it, as I have, with Crosswalk, our AWS extension library, to stand up a REST API and handle requests with AWS Lambda functions:

import * as awsx from "@pulumi/awsx";

// Create a new API Gateway instance.
const api = new awsx.apigateway.API("my-api", {
 routes: [
 {
 // Define an HTTP endpoint.
 method: "GET",
 path: "/things",

 // Handle requests with an AWS Lambda function.
 eventHandler: async (apiGatewayEvent) => {
 return {
 statusCode: 200,
 body: JSON.stringify(["thingOne", "thingTwo"]),
 };
 },
 },
 ],
});

// Export the API's public URL. 🎉
export const apiUrl = api.url;

I love this abstraction, and I use it all the time; it’s an incredibly convenient way to solve a really common problem. But if you examine the code, you’ll notice that it’s making a fairly strong assumption about how you’ll be handling HTTP requests—namely, that you’ll do so with a single Lambda function, and that that function will always return a JavaScript object of a particular shape.

Indeed, this arrangement is the API contract of a Lambda proxy integration—API Gateway integrations come in many shapes and sizes; Lambda integrations just happen to be one of the more popular—and much of the time, an approach like this one works out just fine. But depending on the needs of the application, it might not always be the best fit.

Imagine you were building a print-on-demand service, for example, and you wanted to expose an API to let your customers upload documents and have them converted into orders. On AWS, you might reach for API Gateway, as above, to define an HTTP method and route (POST /uploads, say), wire it up to an AWS Lambda, and have the Lambda parse the upload, write the order to a database, and return a response. Visually, such a design might look something like this:

It’d definitely work, and again, it’s quite common. But at some point, you might find this tight coupling between API Gateway and Lambda too limiting. Say you wanted to be notified whenever a new order was received, with a Slack message, maybe, in one of your team’s shared workspace channels. Under the current design, you’d probably add a few lines of code to the Lambda function to import an HTTP library and make a call to the Slack API to post the message:

That’d work, too—but it’d be less than ideal for a number of reasons. For one, that Lambda would now have two jobs: taking orders and sending Slack notifications. That might be fine for today (it’s only a few lines of code, after all), but over time, those two jobs could easily become three, and then four, and soon enough, that poor Lambda could wind up becoming a lot more difficult to maintain. And given the importance of its main job—capturing orders—it’s not something you’d want to risk failing at runtime because of a random Slack outage or other transient internet mishap. Moreover, with every bit of extra work you tack on to that function, you take a tiny step closer to hitting API Gateway’s 30-second limit.

What you really need, then, is to be able to take multiple independent, possibly long-running actions based on a single API Gateway request. And one way to do that is with Amazon EventBridge.

Hello, EventBridge

Amazon EventBridge (formerly CloudWatch Events) is as a serverless event bus whose job is to receive structured event data—from your own applications, from other AWS Services—and use that data to notify other applications or services using event-handling rules that you specify. With EventBridge, you can build loosely coupled, event-driven systems that take advantage of AWS’s rich support for serverless architectures and that scale gracefully as the needs of those systems change over time.

For this particular application, EventBridge would let you address the multiple-handler problem in a more scalable and easily maintainable way. Rather than have API Gateway invoke Lambda directly, leaving one Lambda responsible for handling multiple tasks, you could have API Gateway publish to EventBridge instead, and let EventBridge invoke as many Lambdas (or other AWS services) as you like, all serverlessly and in parallel—and of course, all easily managed with Pulumi.

Let’s see how. Here’s a revised architecture diagram showing how you might approach building an application like this one with EventBridge positioned between API Gateway and Lambda:

Now let’s have a look at what it’d be like to build it with Pulumi. We won’t build everything in this diagram—things like writing to the database or messaging Slack are left for you to explore—but we will build enough to give you clear picture and a working example of how to connect all of these parts into a working application. Specifically:

an API Gateway instance to act as a container for your public API, along with a stage and a route to handle inbound HTTP requests;
an EventBridge integration (comprised of an event bus and event rule) to handle notifications from API Gateway; and finally,
one or more Lambda functions to be invoked in response to event-rule matches.

Let’s get started.

Create a new project and stack

As always, it’s good practice to begin with a new project and stack:

$ pulumi new aws-typescript

$ pulumi new aws-python

Make sure you configure your AWS credentials as well, and when prompted, choose whatever stack name and AWS region work best for you.

In this post, we’ll be using API Gateway V2 as it’s newer and has a few features (like auto-deploy) that make it easier to work with in this context. The same ideas apply to both V1 and V2, though, so while V2 still lacks full feature parity with V1, the underlying resources we’ll be using—methods, routes, integrations—are the same. You’ll find links to examples of both at the end of this post.

Create the API gateway and stage

Start by replacing the contents of index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf with the following code to declare a new API Gateway API:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create an HTTP API.
const api = new aws.apigatewayv2.Api("api", {
 protocolType: "HTTP",
});

import json
import pulumi
import pulumi_aws as aws

# Create an HTTP API.
api = aws.apigatewayv2.Api("example",
 protocol_type="HTTP"
)

Next, add a stage (feel free to name it whatever you like; I usually use the current stack name for convenience), and set it to deploy automatically whenever a change is made to the API:

// ...

// Create a stage and set it to deploy automatically.
const stage = new aws.apigatewayv2.Stage("stage", {
 apiId: api.id,
 name: pulumi.getStack(),
 autoDeploy: true,
});

# Create a stage and set it to deploy automatically.
stage = aws.apigatewayv2.Stage("stage",
 api_id=api.id,
 name=pulumi.get_stack(),
 auto_deploy=True
);

The next thing to do is register a route on the gateway to give your users a publicly accessible endpoint to upload to. But in order to do that, you’ll need to tell API Gateway what to do when an upload happens. Since the plan of record is to notify EventBridge (using API Gateway’s built-in support for it), you’ll need to declare a few EventBridge things first.

Add an event bus and event rule

Every AWS account gets an event bus by default (one that’s aptly named default), but given how easy it is to create one, we might as well do that for this application. You’ll also need to define an event rule—a resource that watches a specific event bus for events that conform to a particular pattern or shape, and then routes those events to one or more targets (e.g., Lambda functions). Add the following lines to your program for both:

// ...

// Create an event bus.
const bus = new aws.cloudwatch.EventBus("bus");

// Create an event rule to watch for events.
const rule = new aws.cloudwatch.EventRule("rule", {
 eventBusName: bus.name,

 // Specify the event pattern to watch for.
 eventPattern: JSON.stringify({
 source: ["my-event-source"],
 }),
});

# ...

# Create an event bus.
bus = aws.cloudwatch.EventBus("bus")

# Create an event rule to watch for events.
rule = aws.cloudwatch.EventRule("rule",
 event_bus_name=bus.name,
 event_pattern=json.dumps({"source": ["my-event-source"]})
)

The most noteworthy property of the EventRule resource is probably the eventPattern. EventBridge events all conform to a certain schema, and in this case, we’re expressing that this particular event rule should take action on any event that originates from my-event-source. (The source property is just a free-form string that by convention identifies the application or service responsible for the event.)

With the event bus and event rule in place, you’re ready to define the integration itself—the resource responsible for connecting the gateway route (which we’ll get back to shortly) to your newly created event bus. As I mentioned earlier, there are several types of API Gateway integration to choose from, each one suited to a particular purpose. For this example, the AWS_PROXY type is good fit, as it’s simple and requires very little code; it doesn’t give you quite as much control over the API Gateway response as you might like—as a proxy, it just returns to the caller whatever is returned by the backend, in this case EventBridge—but it’s more than enough for the task at hand.

Add the following lines for the integration and route. The comments should explain what each block is doing:

// ...

// Define a policy granting API Gateway permission to publish to EventBridge.
const apiGatewayRole = new aws.iam.Role("api-gateway-role", {
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "apigateway.amazonaws.com",
 },
 },
 ],
 },
 managedPolicyArns: ["arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess"],
});

// Create an API Gateway integration to forward requests to EventBridge.
const integration = new aws.apigatewayv2.Integration("integration", {
 apiId: api.id,

 // The integration type and subtype.
 integrationType: "AWS_PROXY",
 integrationSubtype: "EventBridge-PutEvents",
 credentialsArn: apiGatewayRole.arn,

 // The body of the request to be sent to EventBridge. Note the
 // event source matches pattern defined on the EventRule, and the
 // Detail expression, which just forwards the body of the original
 // API Gateway request (i.e., the uploaded document).
 requestParameters: {
 EventBusName: bus.name,
 Source: "my-event-source",
 DetailType: "my-detail-type",
 Detail: "$request.body",
 },
});

// Finally, define the route.
const route = new aws.apigatewayv2.Route("route", {
 apiId: api.id,
 routeKey: "POST /uploads",
 target: pulumi.interpolate`integrations/${integration.id}`,
});

# ...

# Define a policy granting API Gateway permission to publish to EventBridge.
api_gateway_role = aws.iam.Role("api-gateway-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "apigateway.amazonaws.com",
 },
 },
 ],
 }),
 managed_policy_arns=[
 "arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess",
 ],
);

# Create an API Gateway integration to forward requests to EventBridge.
integration = aws.apigatewayv2.Integration("integration",
 api_id=api.id,

 # The integration type and subtype.
 integration_type="AWS_PROXY",
 integration_subtype="EventBridge-PutEvents",
 credentials_arn=api_gateway_role.arn,

 # The body of the request to be sent to EventBridge. Note the
 # event source matches pattern defined on the EventRule, and the
 # Detail expression, which just forwards the body of the original
 # API Gateway request (i.e., the uploaded document).
 request_parameters={
 "EventBusName": bus.name.apply(lambda name: name),
 "Source": "my-event-source",
 "DetailType": "my-detail-type",
 "Detail": "$request.body",
 },
);

# Finally, define the route.
route = aws.apigatewayv2.Route("route",
 api_id=api.id,
 route_key="POST /uploads",
 target=integration.id.apply(lambda id: f"integrations/{id}"),
);

With that, you’re ready to configure the Lambda.

Add a Lambda function handler

The hard part’s done: you’ve declared an API and route, mapped that route to an integration, configured the integration to publish events to an event bus, and defined an event rule to respond to those events. All that’s left now is to tell the rule how to respond.

So to finish things off, you need:

a Lambda function to handle uploads,
an EventBridge target to bind your event rule to that function, and
an IAM policy granting EventBridge permission to invoke the function.

Add the following lines to your program to complete it:

First, make a new folder and file alongside your Pulumi program, at ./api/handlers.py, and add a Python function to act as your function handler:

// ...

// Create a Lambda function handler with permission to log to CloudWatch.
const lambda = new aws.lambda.CallbackFunction("lambda", {
 policies: [aws.iam.ManagedPolicies.CloudWatchLogsFullAccess],
 callback: async (event: any) => {
 // For now, just log the event, including the uploaded document.
 // That'll be enough to verify everything's working.
 console.log({ source: event.source, detail: event.detail });
 },
});

// Create an EventBridge target associating the event rule with the function.
const lambdaTarget = new aws.cloudwatch.EventTarget("lambda-target", {
 arn: lambda.arn,
 rule: rule.name,
 eventBusName: bus.name,
});

// Give EventBridge permission to invoke the function.
const lambdaPermission = new aws.lambda.Permission("lambda-permission", {
 action: "lambda:InvokeFunction",
 principal: "events.amazonaws.com",
 function: lambda.arn,
 sourceArn: rule.arn,
});

// Export the API Gateway URL to give us something to POST to.
export const url = pulumi.interpolate`${api.apiEndpoint}/${stage.name}`;

# In ./api/handlers.py:
import json

def capture_order(event, context):
 # For now, just log the event, including the uploaded document.
 # That'll be enough to verify everything's working.
 print(json.dumps({ "source": event["source"], "detail": event["detail"] }))

Now, back in index.js index.ts main.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf , add the following lines to your program to complete it:

# ...

# Define a role and policy allowing Lambda functions to log to CloudWatch.
lambda_role = aws.iam.Role("lambda-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "lambda.amazonaws.com"
 },
 "Effect": "Allow"
 }
 ]
 })
)
lambda_role_policy = aws.iam.RolePolicy("lambda-role-policy",
 role=lambda_role.id,
 policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": [
 "logs:CreateLogGroup",
 "logs:CreateLogStream",
 "logs:PutLogEvents"
 ],
 "Resource": "arn:aws:logs:*:*:*"
 }]
 })
)

# Create a Lambda function handler.
lambda_function = aws.lambda_.Function("lambda",
 role=lambda_role.arn,
 runtime="python3.7",
 handler="handlers.capture_order",
 code=pulumi.AssetArchive({
 ".": pulumi.FileArchive('./api')
 })
)

# Create an EventBridge target associating the event rule with the function.
lambda_target = aws.cloudwatch.EventTarget("lambda-target",
 arn=lambda_function.arn,
 rule=rule.name,
 event_bus_name=bus.name,
);

# Give EventBridge permission to invoke the function.
lambda_permission = aws.lambda_.Permission("lambda-permission",
 action="lambda:InvokeFunction",
 principal="events.amazonaws.com",
 function=lambda_function.arn,
 source_arn=rule.arn,
);

# Export the API Gateway URL to give us something to POST to.
pulumi.export("url", pulumi.Output.concat(api.api_endpoint, "/", stage.name))

All Together, Now

Now that the program’s complete, you can run Pulumi to bring it to life:

$ pulumi up
...

Updating (dev)
...

 Type Name Status
 + pulumi:pulumi:Stack eventbridge-v2-dev created
 + ├─ aws:apigatewayv2:Api api created
 + ├─ aws:apigatewayv2:Stage stage created
 + ├─ aws:cloudwatch:EventBus bus created
 ...

Outputs:
 apiURL: "https://geqfietbcl.execute-api.us-west-2.amazonaws.com/dev"

Resources:
 + 15 created

Duration: 31s

When the update finishes, you’ll have a fully functioning API Gateway-EventBridge integration that you can verify with curl:

$ curl --data '{"some-key": "some-value"}' --header "Content-Type: application/json" \
 "$(pulumi stack output url)/uploads"

{"Entries":[{"EventId":"cdc44763-6976-286c-9378-7cce674dff81"}],"FailedEntryCount":0}

Note the response, which comes directly from EventBridge (courtesy of the AWS_PROXY integration), confirming the event was received and written to the event bus.

The final step is to confirm that the request made it all the way to Lambda, which you can easily do by tailing its output with pulumi logs:

$ pulumi logs --follow

Collecting logs for stack dev since 2022-01-06T16:18:48.000-08:00.
...

{
 source: 'my-event-source',
 detail: { 'some-key': 'some-value' }
}

When you’re happy, be sure to tidy up with a pulumi destroy.

What’s next?

There’s a lot more you can do with integrations like this that we didn’t cover: add more Lambda function handlers, have EventBridge target other AWS services (Step Functions might be a good one to try next), validate HTTP request bodies (with API Gateway models, to keep bad data from ever reaching EventBridge), and more. Hopefully this gives you sense of what’s possible, though. And as promised, you’ll find examples that use both versions of API Gateway in our examples repository on GitHub:

API Gateway V2 to EventBridge in TypeScript
API Gateway V2 to EventBridge in Python
API Gateway V1 to EventBridge in TypeScript, with request validation and custom HTTP response mapping

Run Your Own RSS Server on AWS with Pulumi

Christian Nunciato — Mon, 16 Sep 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

It’s been a few years since Google shut down Google Reader, and while a number of nice commercial alternatives have sprung in its wake, none of them has ever been quite the right fit for me personally.

So a while back, after far too much time spent wandering the blogsphere manually, typing URLs into address bars by hand, I decided to go looking to see whether the universe had produced an open-source solution to this problem — and to my surprise and delight, it had! Miniflux is an excellent little open-source RSS server and reader, written in Go and backed by PostgreSQL, that also happens to be packaged as a Docker container. So in this post, I’ll show how easy it is to deploy a Miniflux server of your own on AWS, using only Pulumi and a few lines of TypeScript.

If you’re already comfortable with Pulumi, and you just want to get up and running, I’ve set up a GitHub repo (complete with a Deploy with Pulumi button!) that should have all you need to get going. Just click the button, set a few configs (like your RSS server’s administrative password, which will be stored as an encrypted Pulumi secret, and follow the prompts. Your shiny new server should be up and running within minutes.

Sketching it Out

First, let’s have a look at the app we’ll be building. It’ll consist mainly of two parts:

The Miniflux service, which runs as a web application and API service, and
The Miniflux database, a PostgreSQL instance whose schema and data are fully managed by the Miniflux service.

As I mentioned, the service is packaged as a Docker container that runs on port 8080 by default, and exposes a number of environment variables we can use to configure it, including:

DATABASE_URL, a PostgreSQL connection string used by the service to communicate with the PostgreSQL database,
ADMIN_USERNAME and ADMIN_PASSWORD, which we’ll use to sign into the service for the first time, and
flags for CREATE_ADMIN and RUN_MIGRATIONS, which denote whether to create the administrative user and run database migrations on startup.

We’ll set these properties in the course of developing our program. The service itself, as a container, can easily be run on AWS Fargate, so we can use the @pulumi/awsx package to declare it, and for the database, we’ll use @pulumi/aws to provision a small RDS instance of PostgreSQL. And finally, to make the Miniflux service publicly accessible (initially over HTTP; we’ll switch to HTTPS later), we’ll use an AWS Network Load Balancer, since we’ll only need to be able to route traffic by ports and protocols.

Our architecture is therefore shaping up to look something like this:

Let’s get started!

Setting Up

We’ll start by creating a new Pulumi project. (If you’re completely new to Pulumi, it might be good to begin with our Getting Started guide, which walks you through installing Pulumi and configuring it for your cloud provider.) We’ll use the built-in aws-typescript project template:

$ mkdir miniflux && cd miniflux
$ pulumi new aws-typescript

Once you’ve stepped through the prompts, you should be left with a minimal index.ts file that defines and exports an S3 bucket — but since we won’t be needing an S3 bucket for this project, we can remove that code, and keep only the imports we’ll need:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

Now let’s define a few configuration values for the project.

Configuring the Stack

While we could certainly hard-code all of these values into our program, it’d be better to use Pulumi to set them, since doing so give us the option to vary them by stack (say, if we wanted to run this particular app in multiple environments), but more importantly, to set some passwords for the database user and service administrator. So let’s do that first, so we’ll have them all ready as we develop our program:

$ pulumi config set db_name miniflux
$ pulumi config set db_username miniflux
$ pulumi config set db_password somesupersecretpassword --secret
$ pulumi config set admin_username admin
$ pulumi config set admin_password anothersupersecretpassword --secret

You can change any of these values if you like (and you should definitely change the two passwords, of course), but note that the passwords are set with the --secret flag, which encrypts their values using Pulumi’s built-in support for secrets.

Okay. With our configuration set, we’re ready to start building.

Writing the Code

The Pulumi program we’re writing will:

import our newly created Pulumi configuration, so we can use its values in our program;
create a new DB Subnet Group using the default VPC for your region;
create a new instance of PostgreSQL, with minimal settings, placing it into the subnet group and giving it access to your default ECS Cluster;
create a new Network Listener to define a publicly accessible URL for the Miniflux service;
create a new Fargate service for the Miniflux app, in your default ECS Cluster, passing it the newly created DB connection and Pulumi config settings; and finally,
export the URL as a Pulumi stack Output so we can navigate to the service.

Let’s get coding!

Replace the contents of index.ts with the following code, which comprises the whole program:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Import our Pulumi configuration.
const config = new pulumi.Config();
const dbName = config.require("db_name");
const dbUsername = config.require("db_username");
const dbPassword = config.require("db_password");
const adminUsername = config.require("admin_username");
const adminPassword = config.require("admin_password");

// Get the default VPC and ECS Cluster for your account.
const vpc = awsx.ec2.Vpc.getDefault();
const cluster = awsx.ecs.Cluster.getDefault();

// Create a new subnet group for the database.
const subnetGroup = new aws.rds.SubnetGroup("dbsubnets", {
 subnetIds: vpc.publicSubnetIds,
});

// Create a new database, using the subnet and cluster groups.
const db = new aws.rds.Instance("db", {
 engine: "postgres",
 instanceClass: aws.rds.InstanceTypes.T3_Micro,
 allocatedStorage: 5,
 dbSubnetGroupName: subnetGroup.id,
 vpcSecurityGroupIds: cluster.securityGroups.map(g => g.id),
 name: dbName,
 username: dbUsername,
 password: dbPassword,
 skipFinalSnapshot: true,
});

// Assemble a connection string for the Miniflux service.
const connectionString = pulumi.interpolate `postgres://${dbUsername}:${dbPassword}@${db.endpoint}/miniflux?sslmode=disable`;

// Create an NetworkListener to forward HTTP traffic on port 8080.
const listener = new awsx.lb.NetworkListener("lb", { port: 8080 });

// Create a Fargate service consisting of just one container instance (since that's all we
// really need), passing it the cluster, DB connection and Pulumi config settings.
const service = new awsx.ecs.FargateService("service", {
 cluster,
 desiredCount: 1,
 taskDefinitionArgs: {
 containers: {
 service: {
 image: "miniflux/miniflux:latest",
 portMappings: [
 listener
 ],
 environment: [
 { name: "DATABASE_URL", value: connectionString },
 { name: "RUN_MIGRATIONS", value: "1" },
 { name: "CREATE_ADMIN", value: "1" },
 { name: "ADMIN_USERNAME", value: adminUsername },
 { name: "ADMIN_PASSWORD", value: adminPassword },
 ]
 }
 }
 }
});

// Export the publicly accessible URL.
export const url = pulumi.interpolate `http://${listener.endpoint.hostname}:${listener.endpoint.port}`;

With the program in place, we can bring it to life with pulumi up:

$ pulumi up

In a few minutes, you should see that Pulumi has created 32 new resources:

And if you examine the logs for this stack using pulumi logs -f, you’ll see that the database was created, migrations were run, administrative user was created, and the service is now listening:

Finally, looking back at the output of pulumi up, you’ll see the URL we exported, to which you can navigate, sign in with the administrative user created by our program, and start RSSing:

Excellent! You’ve now got your very own fully functioning RSS service, created with only a few dozen lines of code.

Technically, we’re done — but there are a couple of things we might like to do to make this project a bit more complete.

Running on Port 80

Right now, our service’s load balancer is configured to listen and forward on the same port, 8080. If we wanted to have the load balancer listen on port 80 (or some other port) instead, we’d need to change its declaration a bit, to create the load balancer and target group more explicitly. Let’s do that:

// Create a load balancer, target group and network listener explicitly.
const listener = new awsx.lb.NetworkLoadBalancer("lb")
 .createTargetGroup("group", { port: 8080 })
 .createListener("listener", { port: 80 });

Run pulumi up, and you’ll see that the service is now accessible at a new host and port (though the database was of course left unchanged):

Great! But what if we wanted to run over HTTPS, instead?

Running over HTTPS

In order to run the Miniflux service over HTTPS, you’ll need to have obtained an SSL certificate from AWS Certificate Manager that corresponds with the domain you plan to use for the service.

Provided you’ve obtained that certificate, it’s defined in the same AWS region as the one you’ve configured for your Pulumi stack, and you’re able to make changes to the DNS records for the domain associated with the certificate, then updating the Pulumi program is easy — just change the listener declaration to use TLS and port 443, and add a certificateArn property to apply the certificate:

// Run the service over HTTPS, terminating SSL at the load balancer and forwarding to port 8080 on the container.
const listener = new awsx.lb.NetworkLoadBalancer("lb")
 .createTargetGroup("group", { port: 8080, protocol: "TCP" })
 .createListener("listener", { port: 443, protocol: "TLS", certificateArn: "arn:aws:acm:..." });

Run pulumi up one last time, and you should see your service now running on port 443 — but you won’t be able to access it until you add CNAME record to the DNS settings for your domain (I happen to use Google for mine) mapping a new subdomain to your load balancer’s hostname:

And with that, you should now be able to browse your RSS server securely:

Finishing Up

In this post, we’ve seen how easy it is to run a container as a service connected to an RDS database with Pulumi, and to expose that container securely on the web. If we wanted, we could go even farther — we could refactor the program into Pulumi Components, perhaps (one for the service, one for the database), package it up for sharing on npm, and so on.

But we’ll leave those improvements for another day. For now, let’s enjoy what we’ve created! And start catching up on all that reading we’ve missed.