Pulumi Blog: Joe Duffy

How to Run AI Agents on Kubernetes with Pulumi

Joe Duffy — Tue, 28 Jul 2026 00:00:00 +0000

Kubernetes has become the default place teams run agentic AI workloads: CNCF’s 2026 annual survey found that 66% of organizations hosting generative AI models use Kubernetes to manage some or all of their inference workloads.¹ An entire ecosystem has grown up around that fact — agent runtimes, model servers, GPU schedulers — and most of it assumes the infrastructure underneath is already handled. It usually isn’t. An AI agent is not a stateless web service, and provisioning for one takes more than copying a Deployment YAML and swapping the image.

I spend a lot of my time these days thinking about what changes when the thing consuming your infrastructure isn’t a person or a fixed pipeline, but an agent making its own calls about what to do next — and Kubernetes is where I keep seeing that question show up first, because it’s already where most teams run everything else. This post is about that gap: what changes about Kubernetes infrastructure when the workload is an agent, what the current agentic-Kubernetes ecosystem actually looks like, and how to provision and govern that infrastructure with Pulumi using TypeScript and Python — plus where Pulumi Neo fits once the infrastructure itself has to answer to an agent. If you haven’t read our take on the broader shift toward agentic infrastructure, What Is Agentic Infrastructure? is a good companion piece; this post stays specific to the Kubernetes layer.

What makes agentic AI workloads different from ordinary Kubernetes workloads?

An agent’s resource and governance profile looks nothing like a typical web service, largely because it schedules accelerators, holds state across long sessions, and reaches out to systems a normal Deployment never touches.

Dimension	Typical web service	Agent workload
Compute	CPU, predictable	Often GPU/accelerator, bursty
Session shape	Short-lived, stateless requests	Long-lived sessions, conversational state
Egress	Internal services, one database	Model provider APIs, arbitrary tools, other agents
Credential surface	A handful of scoped secrets	API keys for one or more model providers, plus whatever tools/MCP servers the agent is allowed to call
Access/RBAC	Fixed, narrow	Potentially broad — an agent that can act on infrastructure needs infrastructure-shaped permissions
Cost profile	Roughly linear with traffic	Token- and GPU-time-driven, harder to predict

None of this means Kubernetes is the wrong place to run agents — the opposite: its scheduler, autoscaling, and RBAC primitives are exactly the right building blocks for these problems. It means the infrastructure around an agent deserves more deliberate design than “reuse the Deployment template,” particularly around GPU scheduling, secrets, and the blast radius of what the agent is allowed to do.

What does the agentic Kubernetes stack look like in 2026?

There isn’t one “agents on Kubernetes” product; there’s a stack, and most of it is open source and CNCF-adjacent. Being fair to that ecosystem matters here, because Pulumi’s job is to provision and govern the infrastructure underneath these tools, not to replace them.

Project	What it does	Status
kagent	Framework for running AI agents on Kubernetes, with its own MCP server and CRDs for agents and model providers	CNCF Sandbox (accepted May 22, 2025)²
KServe	Model-serving on Kubernetes	CNCF Incubating (accepted Sept 29, 2025)³
Kueue	Job queueing for batch and ML workloads	CNCF Sandbox
KubeRay	Runs Ray (and Ray Serve) clusters on Kubernetes	Kubernetes SIG / Ray ecosystem project
vLLM + llm-d	High-throughput, Kubernetes-native distributed LLM inference	Red Hat-launched, community project
LeaderWorkerSet / JobSet	Multi-node inference and batch training workload APIs	kubernetes-sigs
Dynamic Resource Allocation (DRA)	Kubernetes-native API for scheduling GPUs and other accelerators	GA in Kubernetes v1.34⁴
Gateway API Inference Extension	Routing and load balancing tuned for LLM inference traffic	GA
NVIDIA GPU Operator	Manages NVIDIA drivers and device plugins on cluster nodes	NVIDIA-maintained

kagent is worth a closer look since it’s the piece most directly aimed at “run an agent on Kubernetes.” As of its current release (v0.9.9), it defines its own custom resources — Agent, ModelConfig, ModelProviderConfig, RemoteMCPServer, and SandboxAgent — and ships an MCP server with built-in tools for Kubernetes, Helm, Istio, Argo, and Prometheus, plus the ability to connect to any external MCP server via the RemoteMCPServer CRD.⁵ It’s a genuinely useful abstraction for defining what an agent can do; it isn’t, on its own, an answer for how the cluster the agent runs on got there, who’s allowed to change it, or how secrets get to it safely. That’s the layer Pulumi sits at.

How do you provision Kubernetes infrastructure for AI agents with Pulumi?

A minimal but representative path from “no cluster” to “an agent running on it, governed” looks like this:

Provision a Kubernetes cluster with a GPU-enabled node group sized for your model workload.
Install the kagent CRDs and controller onto the cluster using a Pulumi Helm chart resource.
Store the model provider’s API key as a Pulumi secret sourced from a Pulumi ESC environment, rather than a plaintext Kubernetes Secret.
Declare the agent itself as a custom resource so it’s versioned and reviewed the same way as everything else in the stack.
Attach a policy pack so specific classes of change — like exposing a service publicly, or granting a service account cluster-admin — require an explicit review.
Roll changes out through CI, or through the Pulumi Kubernetes Operator if you want the cluster itself to reconcile against a Pulumi stack.

Steps 1–4 in TypeScript:

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";
import * as eks from "@pulumi/eks";

// 1. GPU-enabled node group alongside your existing EKS cluster.
const gpuNodeGroup = new eks.ManagedNodeGroup("agent-gpu-nodes", {
 cluster: eksCluster,
 nodeRole: nodeRole,
 instanceTypes: ["g6.8xlarge"],
 gpu: true,
 scalingConfig: { minSize: 1, desiredSize: 2, maxSize: 4 },
});

const k8sProvider = new kubernetes.Provider("k8s", {
 kubeconfig: eksCluster.kubeconfig,
});

// 2. kagent CRDs and controller via the Helm v4 Chart resource.
const kagentCrds = new kubernetes.helm.v4.Chart("kagent-crds", {
 chart: "oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds",
 namespace: "kagent",
}, { provider: k8sProvider });

const kagent = new kubernetes.helm.v4.Chart("kagent", {
 chart: "oci://ghcr.io/kagent-dev/kagent/helm/kagent",
 namespace: "kagent",
 values: {
 providers: { default: "openAI" },
 },
}, { provider: k8sProvider, dependsOn: [kagentCrds] });

// 3. Model provider API key, sourced as a secret from a Pulumi ESC
// environment (declared in Pulumi.<stack>.yaml, not hardcoded here).
const config = new pulumi.Config();
const openAiApiKey = config.requireSecret("openAiApiKey");

const providerSecret = new kubernetes.core.v1.Secret("openai-credentials", {
 metadata: { namespace: "kagent" },
 stringData: { apiKey: openAiApiKey },
}, { provider: k8sProvider });

// 4. The agent itself, as a versioned custom resource.
const agent = new kubernetes.apiextensions.CustomResource("k8s-ops-agent", {
 apiVersion: "kagent.dev/v1alpha2",
 kind: "Agent",
 metadata: { namespace: "kagent" },
 spec: {
 description: "Answers questions about cluster state and proposes fixes",
 modelConfig: "openai-gpt",
 tools: ["kubernetes", "helm"],
 },
}, { provider: k8sProvider, dependsOn: [kagent] });

The same four steps in Python:

import pulumi
import pulumi_eks as eks
import pulumi_kubernetes as kubernetes
from pulumi_kubernetes import apiextensions

# 1. GPU-enabled node group alongside your existing EKS cluster.
gpu_node_group = eks.ManagedNodeGroup("agent-gpu-nodes",
 cluster=eks_cluster,
 node_role=node_role,
 instance_types=["g6.8xlarge"],
 gpu=True,
 scaling_config={"min_size": 1, "desired_size": 2, "max_size": 4})

k8s_provider = kubernetes.Provider("k8s", kubeconfig=eks_cluster.kubeconfig)

# 2. kagent CRDs and controller via the Helm v4 Chart resource.
kagent_crds = kubernetes.helm.v4.Chart("kagent-crds",
 chart="oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds",
 namespace="kagent",
 opts=pulumi.ResourceOptions(provider=k8s_provider))

kagent = kubernetes.helm.v4.Chart("kagent",
 chart="oci://ghcr.io/kagent-dev/kagent/helm/kagent",
 namespace="kagent",
 values={"providers": {"default": "openAI"}},
 opts=pulumi.ResourceOptions(provider=k8s_provider, depends_on=[kagent_crds]))

# 3. Model provider API key, sourced as a secret from a Pulumi ESC
# environment (declared in Pulumi.<stack>.yaml, not hardcoded here).
config = pulumi.Config()
openai_api_key = config.require_secret("openAiApiKey")

provider_secret = kubernetes.core.v1.Secret("openai-credentials",
 metadata={"namespace": "kagent"},
 string_data={"apiKey": openai_api_key},
 opts=pulumi.ResourceOptions(provider=k8s_provider))

# 4. The agent itself, as a versioned custom resource.
agent = apiextensions.CustomResource("k8s-ops-agent",
 api_version="kagent.dev/v1alpha2",
 kind="Agent",
 metadata={"namespace": "kagent"},
 spec={
 "description": "Answers questions about cluster state and proposes fixes",
 "modelConfig": "openai-gpt",
 "tools": ["kubernetes", "helm"],
 },
 opts=pulumi.ResourceOptions(provider=k8s_provider, depends_on=[kagent]))

The Pulumi.<stack>.yaml for the stack pulls the API key from an ESC environment instead of a plaintext value in source control:

environment:
 - agents-team/production

Verify the exact CRD group/version (kagent.dev/v1alpha2 above) against kagent’s own CRD manifests before you ship this in production — CRD APIs move faster than blog posts do.

Why write agent infrastructure in a general-purpose language instead of templated YAML?

Because an agent’s infrastructure has exactly the kind of conditional, repeated, testable shape that general-purpose languages are built for, and templated YAML is not. A for loop that provisions one GPU node pool per region, a function that returns a standard “agent + its guardrail policy” bundle, a unit test that asserts a policy pack actually blocks an over-privileged service account before it ships — all of that is native to TypeScript, Python, Go, C#, and Java, and bolted-on at best in Helm templates or raw manifests.

This isn’t a knock on Helm or Kustomize, which remain genuinely good tools for packaging and distributing charts — including kagent’s own install, which is a Helm chart for a reason. The distinction is what you’re doing with the result. Distributing a reusable, versioned package is a job Helm is built for. Composing that package into your specific cluster, wiring its secrets, gating its rollout with policy, and testing the whole thing before it ships is software engineering, and it benefits from a real language: loops, functions, types, unit tests, and the same CI/CD your application code already runs through.

How do you keep agent workloads governed, least-privileged, and cost-aware?

An agent that can act on infrastructure needs infrastructure-shaped guardrails, not just a service account and a prayer. Three pieces of the Pulumi platform apply directly:

Policy as code. Pulumi Policies let you write rules — in the same general-purpose languages, not a separate policy DSL — that run before a change is applied: block a Deployment requesting cluster-admin, require GPU node pools to carry a cost-center label, or flag any RemoteMCPServer pointed at a host outside an allowlist.
Pulumi ESC for secrets and configuration, so model-provider API keys and other credentials are centrally managed, rotated, and scoped, rather than copy-pasted into Secret manifests across every cluster an agent touches.
RBAC scoped to the agent’s actual job. The same custom-resource pattern used to declare the agent (shown above) is the natural place to also declare its ServiceAccount, Role, and RoleBinding — as code, reviewed the same way as the agent’s own configuration, not layered on afterward.

I want to be precise here about Pulumi Insights, because I see the two products get conflated more than almost anything else we ship: Insights gives you discovery (a searchable inventory of resources across Pulumi, Terraform, CloudFormation, and manually created infrastructure) and policy enforcement, including pre-built compliance packs. Cost-optimization recommendations are a Neo capability, not an Insights one.

Where does Pulumi Neo fit when the infrastructure itself is agentic?

Neo is Pulumi’s own infrastructure agent, and it’s worth being specific about what it actually does today rather than leaning on the general “AI for infrastructure” framing. Documented capabilities include answering questions about your existing infrastructure, proposing changes and opening a pull request for them, running a preview before anything lands, reviewing PRs, and taking on recurring maintenance work using your team’s established Pulumi practices.⁶ The Infrastructure AI docs cover each of these in more depth.

The thing I care about most with Neo is trust, and trust gets earned one reviewed pull request at a time, not granted upfront. That’s why every capability starts human-in-the-loop and only earns more autonomy once the guardrails have proven themselves in practice — the same discipline this post argues for everywhere else an agent touches your Kubernetes infrastructure. Pulumi’s own product blog has also described Neo handling things like Terraform-to-Pulumi migrations and containerizing and migrating a service onto Kubernetes.⁷

The relevant framing for this post: Neo doesn’t replace kagent, KServe, or any other workload-level agent framework running inside your cluster — it operates one layer up, on the infrastructure that provisions and governs the cluster itself. If your team is running kagent-based agents on Kubernetes, Neo is the thing that can propose the PR that adds a new GPU node pool, review the policy-pack change that locks down a RemoteMCPServer, or handle the maintenance toil around keeping that stack current.

Frequently asked questions

How do you deploy AI agents on Kubernetes?

Provision a cluster (with GPU node pools if your agent needs them), install an agent runtime — kagent is one of the more Kubernetes-native options as of 2026 — and declare the agent as a custom resource alongside its model configuration and any MCP tool servers it needs. Doing this with Pulumi means the cluster, the runtime install, the secrets, and the agent definition are all one reviewable, testable stack rather than a mix of kubectl apply commands run by hand.

Do you need GPUs to run AI agents on Kubernetes?

Not always. An agent that only calls a hosted model API (OpenAI, Anthropic, and similar) needs CPU nodes to run its own logic and tool calls; the GPU-heavy work happens on the provider’s infrastructure, not yours. GPUs matter once you’re self-hosting inference — running an open-weight model via vLLM, KServe, or Ray Serve — which is where projects like Kueue, LeaderWorkerSet, and Dynamic Resource Allocation come in.

How is kagent different from running an agent as a plain Deployment?

A plain Deployment gives you a running process; it doesn’t give you a standard way to describe an agent’s model configuration, its tool access, or its connections to MCP servers as first-class, versionable objects. kagent’s CRDs (Agent, ModelConfig, ModelProviderConfig, RemoteMCPServer, SandboxAgent) exist specifically to make those concepts declarative, so a policy engine, a reviewer, or another automation can reason about them the same way it reasons about a Deployment or a Service.

Can Pulumi manage Kubernetes resources that Helm or Terraform already created?

Yes. Pulumi Insights discovers and inventories resources regardless of what provisioned them — Pulumi, Terraform, CloudFormation, or manual changes — which is useful precisely because most clusters running agent workloads already have Helm-installed components like kagent, an ingress controller, or a GPU operator on them. You don’t need to rip those out to bring the rest of the stack under Pulumi; kubernetes.yaml.v2.ConfigFile and ConfigGroup can also adopt existing manifests directly into a Pulumi program.

What are Kubernetes best practices for AI workloads in 2026?

Match node pools to the workload (GPU pools for self-hosted inference, CPU pools for orchestration and tool-calling), use a queueing layer like Kueue for batch and training jobs so they don’t starve latency-sensitive agent traffic, keep model-provider credentials in a secrets manager (not plaintext Secret objects), and scope every agent’s RBAC to the specific tools it’s allowed to call. For general Kubernetes hygiene beyond the AI-specific pieces, see Kubernetes Best Practices I Wish I Had Known Before.

How does Pulumi Neo help manage Kubernetes infrastructure for AI agents?

Neo operates at the infrastructure layer underneath your agent workloads: it can answer questions about the state of your cluster, propose infrastructure changes as a reviewable pull request, run a preview before anything is applied, review incoming PRs, and take on the recurring maintenance that keeps a Kubernetes-plus-agents stack current — all using your team’s existing Pulumi practices rather than a separate workflow.

Where to go next

If you’re starting from zero, the Kubernetes get-started guide and the Kubernetes provider registry cover the provisioning basics this post builds on. If you’re already running agent workloads and want the governance layer, start with Pulumi Policies and Pulumi ESC. And if agent sprawl — not just agentic Kubernetes workloads, but the broader proliferation of agents across your org — is the more pressing problem, Agent Sprawl Is Here. Your IaC Platform Is the Answer. is the companion read.

CNCF Annual Survey Report, January 2026: cncf.io/wp-content/uploads/2026/01/CNCF_Annual_Survey_Report_final.pdf. ↩︎
“kagent was accepted to CNCF on May 22, 2025 at the Sandbox maturity level.” CNCF Projects: cncf.io/projects/kagent/. ↩︎
KServe accepted as a CNCF Incubating project, September 29, 2025: cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/. ↩︎
Dynamic Resource Allocation reached general availability in Kubernetes v1.34 (released August 27, 2025). Kubernetes blog: kubernetes.io/blog. ↩︎
kagent CRD kinds (Agent, ModelConfig, ModelProviderConfig, RemoteMCPServer, SandboxAgent) per the kagent-dev/kagent v1alpha2 API types, and MCP server tooling per the project README: github.com/kagent-dev/kagent. ↩︎
Pulumi Infrastructure AI / Neo documentation: pulumi.com/docs/ai/. ↩︎
“10 Things You Can Do with Neo” and “10 More Things You Can Do with Neo,” Pulumi blog: pulumi.com/blogs/10-things-you-can-do-with-neo/, pulumi.com/blogs/10-more-things-you-can-do-with-neo/. ↩︎

The Agentic Infrastructure Era

Joe Duffy — Tue, 19 May 2026 03:00:00 -0700

The first frontier agents excelled at was coding. The reason is evident: we have billions of lines of self-documenting code available on the internet for the LLMs to learn from. We can measure their performance on coding thanks to linters, type checkers, compilers, and test suites. The most advanced agentic systems to hit product/market fit have been coding-oriented, and it has resulted in an intense velocity increase in how much and how fast code we can write.

But as the AI tsunami whips up reams of code, what happens to it becomes just as critical. As an industry, we’ve moved beyond just coding to engineering, which includes documentation, tests, automation, and, yes, managing the very infrastructure our applications need to run. The deeper into production you go, however, the less good agents naturally are at helping. At Pulumi, we live and breathe infrastructure, and have seen this firsthand. But we’ve also been hard at work building the platform this new era runs on. In this post, I’ll share our point of view, what we’ve built, what we’re launching today, and why all infrastructure is about to be agentic.

See Pulumi Neo in action → Neo is the agent at the center of everything below: it previews changes, checks them against policy, and opens PRs for your review.

LLMs are natural coders

It is remarkable to look back and note that frontier models, less than two years ago, in August 2024, scored just 33% on SWE-bench Verified. Present-day models score 86%, which represents a 4x reduction in the errors models will make when coding. This enables models to solve increasingly difficult coding problems, and humans can lean more heavily on them to offload tasks. Anthropic’s new Mythos model scores 94% and, although it isn’t generally available at the time of this article, there’s no question we’ll close in on 95% by the end of 2026. That is another 2.3x reduction in error rates. This very naturally puts us onto the last mile of fully agentic coding.

This has been the result of code being highly in-distribution combined with the relentless pursuit of solving coding problems from the frontier labs, especially with Anthropic’s Claude Code, but now with OpenAI’s Codex, driving a tight feedback loop that turns into an improvement flywheel.

Given that LLMs are natural coders, most of us simply assumed that the breakout success we’ve seen with agents for coding would automatically translate into new problem domains. And for sure, we have seen some success. But perhaps not as much as we’d like. Not all problem domains are documented equally well so that the models can naturally learn about them.

Andrej Karpathy noted nearly a year ago that, “Building a modern app is a bit like assembling IKEA furniture,” observing that, though writing the code was easy, fun, and fast, the next mile of actually getting the application running in production entailed many things the LLM wasn’t naturally good at, including “services, API keys, configurations, dev/prod deployments.”

At the same time, we’re seeing something magical happen here at Pulumi: LLMs are now doing over 20% of the infrastructure deployments, up from virtually zero a year ago. We expect this to grow to over 50% before the end of this year and well beyond afterwards.

The agentic infrastructure era is here. Today we’re announcing several new platform capabilities to accelerate it further.

Before getting to what’s new, however, why are we seeing this happening in reality?

Turning infrastructure problems into coding problems

We began our journey with our open-source infrastructure-as-code project nearly ten years ago. Having spent much of my career working on programming languages and compilers, I had a strong conviction that the right substrate for infrastructure was the languages developers already knew and loved, not yet another DSL and certainly not piles of YAML. So we focused first on great ergonomics for humans, and for us, that meant letting you use programming languages, tools, and ecosystems that humans already know and love. Languages like Python, TypeScript, Go, C#, Java, and more. Infrastructure as code also, importantly, comes with guardrails to make infrastructure deployments dependable, reviewable, and auditable. Over time we’ve built out an entire platform with security, compliance, and governance capabilities.

One way of thinking about this is we modeled the realm of cloud infrastructure inside the realm of programming languages: cloud resources become objects, configurations are just variables, dependencies between resources are just references, standard blueprints become classes. Doing so turns the cloud into something that is suddenly programmable, and allows us to apply real software engineering patterns and practices to infrastructure.

Here’s the twist we’ve realized this past year. It is a happy accident that the combination of this plus the verifiability of every change is the exact combination that empowers agents to do infrastructure. LLMs are natural coders. By mapping infrastructure space in code space, agents can do what they’re great at – code – and the infrastructure as code engine is the link that maps those code edits back down into infrastructure space. And as the SWE-bench curve plays out, every domain expressed in code rides it. Infrastructure, thanks to the bet we made nine years ago, is now one of them.

This isn’t just about volume of in-distribution code. Most production infrastructure code lives in private repositories. In contrast, public Python and TypeScript include vast amounts of genuinely production-grade open source, demonstrating real patterns at scale. Public infrastructure DSL corpora skew heavily toward tutorial-grade content that doesn’t scale. By modeling infrastructure in real languages, Pulumi inherits at-scale in-distribution directly. Patterns in Python or TypeScript generalize from what the model has practiced, while DSL patterns are idiosyncratic by definition and don’t compound the same way. Code can express everything an engineer can do, which is exactly why models trained on it have become prolific engineers.

What does this look like in practice? For example, an agent can define reusable infrastructure:

import pulumi
import pulumi_coreweave as coreweave
import pulumi_nvidia_aicr as aicr

class TrainingCluster(pulumi.ComponentResource):
 """A reusable training cluster resource."""
 kubeconfig: pulumi.Output[str]
 def __init__(self, name: str, *, nodes: int, accelerator: str):
 super().__init__("training:Cluster", name)
 self.cluster = coreweave.KubernetesCluster(f"{name}-k8s",
 accelerator=accelerator, # gb200/b200/h100
 count=nodes)
 self.stack = aicr.ClusterStack(f"{name}-aicr",
 kubeconfig=self.cluster.kubeconfig,
 accelerator=accelerator,
 intent="training",
 platform="kubeflow")

And then flexibly consume it – all using languages, like Python, it is already fluent in:

import pulumi
from infra import TrainingCluster
from orchestration import RLFleet

# Spin up an RL stack with 1 learner and 8 rollout workers.
learner = TrainingCluster("learner",
 nodes=4, accelerator="gb200")
rollouts = [
 TrainingCluster(f"rollout-{i}",
 nodes=2, accelerator="b200")
 for i in range(8)
]

fleet = RLFleet("ppo",
 learner=learner, workers=rollouts)

pulumi.export("login", fleet.head_node)

Beyond just being in-distribution, infrastructure as code also gives us a critical piece: the ability to diff changes at the infrastructure layer. Just as we wouldn’t vibe code without git showing us the source changes, we shouldn’t vibe infrastructure without a tool that shows what it will do before it does it, and what it has already done in the past. It’s like git diff for your infrastructure.

The combination of languages, type systems, linting, testing, and infrastructure diffs gives agents enough of what they need to create a fully closed, verifiable, self-correcting loop. Verifiability is the thing that ultimately matters. Every action an agent takes can be previewed, policy-checked, reviewed, and audited. By humans, by other agents, by CI/CD automation. Guardrails guide the agent and verifiability proves the end result.

When I say “infrastructure” by the way, I mean something quite broad. It includes public cloud hyperscalers, of course, like AWS, Azure, Google Cloud, and CoreWeave. It includes cloud native technologies like Kubernetes and Helm. It also includes private clouds powered by VMware and other data center technologies. It includes broad cloud platforms like Cloudflare. But, subtly, also any hosted managed infrastructure service, such as Snowflake, MongoDB, Databricks, and more. And increasingly, the observability, security, and operational tooling that wraps everything else, like Datadog, Sentry, PagerDuty, and other systems that give agents the context to reason about what’s actually happening in production. We support thousands of providers with a consistent programming model and agents are increasingly writing those too.

What we’re shipping today

The foundation of expressing infrastructure as code, combined with verifiable changes, makes a 100% agentic infrastructure future even conceivable. But we still have work to realize this vision.

Meeting agents where they work (the CLI)

We know real code is the right substrate for agentic infrastructure – but we also know agents want to start with the smallest possible commitment and progressively level up. So we’ve designed the Pulumi experience as a progression: an agent should be able to sign up, run its first command, and grow into a full infrastructure as code workflow without ever hitting a wall or manual intervention. Here’s what we’re launching today to make that journey work end-to-end:

Agent accounts. Now agents can use free, ephemeral Pulumi Cloud accounts straight from Claude Code, Codex, OpenCode, Copilot, Cursor, and others. This lets an agent do anything a human could with the Pulumi product without a manual signup step, removing friction while letting agents benefit from robust infrastructure management. A human can claim that account at any time to make it permanent and to share with their team.

One-command execution. Today an agent needs to know how to find, download, and install Pulumi, and manage subsequent versioning and upgrades. Anything that adds friction is discouraging to an agent. We now have an npm package that enables npx pulumi <anything>-style commands so agents can run any Pulumi command anywhere without needing to worry about installation or version pinning.

Imperative infrastructure operations. The new pulumi do command enables direct create, read, update, delete, list, and API operations with a single command. For example, pulumi do create eks:Cluster spins up an AWS EKS cluster with built-in best practices, and pulumi do update cloudflare:r2:Bucket --tags '{ "Foo": "Bar" }' updates a bucket’s tags. All thousands of providers and hundreds of thousands of resource types in the Pulumi ecosystem are supported across clouds. As the complexity of your scenario outgrows singular commands, it is easy to eject out into a full-blown infrastructure-as-code project.

Pulumi Cloud in the CLI. Over the years, we’ve added tons of great capabilities to Pulumi Cloud – things like change history, time-to-live stacks, drift detection, resource discovery and search, private registries, IDP, audit logs, secrets management, team-wide policy enforcement – but didn’t add the respective ergonomic commands to the CLI. These are the sort of features that matter at scale. Now it’s all there in your terminal where agents can use them. Over 30 new commands, you can think of this as the equivalent of the gh CLI which agents really like.

Read the blog: Better CLI Interactions for Agents and Humans →

Neo, everywhere you work

Once an agent (or a team) has graduated through these capabilities, what comes next is asynchronous infrastructure work and, increasingly, autonomy. That’s where Neo comes in: Pulumi’s own infrastructure agent, now upgraded with the surfaces, integrations, and cadences teams actually need to run agentic infrastructure in production. We are shipping:

Neo in the CLI. The new pulumi neo command lets you run the same agent that is already in Pulumi Cloud. It even shares the same agentic loop and uses a sophisticated architecture where the agent workstation is your local computer, allowing it to more seamlessly access source context and tools. This allows you to do agentic infrastructure wherever you prefer: Neo in the cloud for asynchronous, autonomous tasks, or your workstation for more intensive deep-dive paired sessions.

Neo GitHub and Slack Apps. Now you can @-mention Neo from GitHub pull requests, and/or straight from Slack, to kick off agentic infrastructure workflows wherever it is most convenient, complete with whatever guardrails you’ve configured already.

Neo Integration Catalog. A new integration catalog lets you configure connectors to other systems that bring valuable infrastructure management context, including Atlassian, Datadog, Honeycomb, Linear, PagerDuty, and Supabase. This lets you tap into additional planning, specification, observability, and live site incident data, expanding what agents can do on Days 0, 1, and especially 2.

Scheduled Tasks and Read-Only Sessions. Now you can automate recurring infrastructure tasks, including confining Neo to read-only operations for extra safety. This opens up scenarios like reporting on infrastructure patterns weekly, automatically cleaning up waste in your dev accounts daily, or scheduled maintenance and upgrades.

Read the blog: 10 More Things You Can Do with Pulumi Neo →

Partnering with the frontier of AI infrastructure

We’re investing in deep partnerships with the companies building the frontier of AI infrastructure, and are shipping two new partner providers in close collaboration with the teams behind them:

CoreWeave provider. This new provider brings the full power of the CoreWeave platform, the leader in GPU infrastructure, to help teams provision AI workloads with ease. This includes support for all of the CoreWeave services including CoreWeave Kubernetes Service (CKS) and includes several examples and templates out of the box for training and inference architectures written in AI-native languages.

NVIDIA AI Cluster Runtime (AICR) provider. This new provider delivers fully functional NVIDIA software stacks atop the underlying cloud provider’s GPU infrastructure. This packages out-of-the-box components like NVIDIA GPU Operator, Kubeflow, NIM Operator, and dozens more otherwise tricky-to-configure cluster components. We’re excited to showcase the new AICR provider in tandem with our new CoreWeave provider: you can now spin up a CKS cluster and configure it with AICR in one program.

Educating and measuring agentic infrastructure intelligence

We’ve also invested in making Pulumi maximally legible to agents, because the best tools only matter if agents can find them, read them, and know which one to reach for and when:

Agent-friendly docs. We now serve our docs website in markdown to agents and did a pass over our documentation and CLI text to ensure it’s maximally useful to agents.

Agent-friendly CLI ergonomics. We have added --json and structured errors across the CLI to help agents parse and react to outputs appropriately.

New skills. We’ve done a pass over our skills and added a new uber-skill that describes how an agent should leverage the full suite of capabilities we now offer, from Level 0 (agent accounts), to Level 1 (pulumi do), to Level 2 (full IaC and pulumi up), to Level 3 (Pulumi Cloud governance and full autonomy with Neo), and beyond. It’s handy for humans too.

InfraBench. We have created a new benchmark to measure how well an agent performs on a wide array of representative infrastructure tasks. This is the infrastructure equivalent to SWE-bench and lets us measure stock agents, agents plus the new tools and skills, and Neo, and how they’re improving over time. Although we are still hard at work on building out the comprehensive suite, and will keep it internally for now, we will begin publishing our progress against InfraBench as we continue tackling new problems and improving agent performance.

What comes next

We’re primarily focused on two pillars as we go forward:

First, we are making agentic infrastructure a reality with increasingly autonomous workflows. The pace and scale of infrastructure in an AI era are beyond anything we’ve seen before, and we need to ensure infrastructure doesn’t become the bottleneck to velocity. We will be shipping steady improvements that help agents score better on InfraBench as a measure of their abilities. This will begin with new building blocks and in-context improvements, but we’re really excited that every improvement the frontier labs ship in general-purpose coding ability is automatically an improvement in agentic infrastructure on Pulumi, at zero cost to us or to our users. The result is that Pulumi-equipped agents will consistently and measurably outperform raw agents on real infrastructure work, and that gap will compound as the substrate and the models both improve.

Next, a world in which agents are provisioning, updating, monitoring, and managing all of our infrastructure changes the human/agent/infrastructure interface greatly. We need better change management, policy enforcement, and observability tools. Going from a single human doing a single infrastructure task at a time, to a single agent doing that task, to teams of agents doing lots of autonomous infrastructure tasks at once, will reveal new bottlenecks needed to fully govern your infrastructure. We have a lot of those facilities in Pulumi Cloud already but we will continue pushing here to improve them and ensure they scale up as fast as they need to. This is the layer that matters even as agents themselves get great at infrastructure. The smartest agent in the world still needs guardrails, audit trails, and policy enforcement to be trusted with production systems at scale, and that layer gets more valuable as agents get more capable, not less.

The combination of these two things will lead to a world where agents are able to manage more infrastructure, and to do so with increasing autonomy, while human operators retain full visibility, understanding, and control over what their teams of agents are doing at all times. This is the full engineering loop applied to infrastructure. Not just code generation, but design, review, deployment, and operation, with agents and humans collaborating across every step.

It’s a new era, and we’re just getting started

It was remarkable to see in practice and at scale that what’s good for humans is good for agents: ergonomic infrastructure as code. We’ve seen every point along the spectrum, ranging from Wiz managing 1 million resources and doing 100k daily deployments, to Compostable.ai already living the 100% agentic infrastructure dream, to a frontier lab who grew their infrastructure footprint by 982% in a year, all using infrastructure modeled in true code. All very different scales and teams, but with one thing in common: it was code all the way down.

Languages and verifiability already provide strong foundations, however, there are still clear areas we need to improve, beginning with making the full suite of infrastructure automation, security, and governance tools more accessible to agents. That’s what today is all about. To deep dive on details of the launches, check out the companion blog posts released today.

Agentic infrastructure is a super exciting one-way door for our industry. We’re pushing hard to take this beyond 20% and toward 100% in the years ahead. This is uniquely enabled by infrastructure as code in languages the models already speak combined with built-in verifiability that makes agentic work automatically safe. There’s more to be done, however, today’s launch will bend the curve even further upwards. We’re still only just getting started.

Get started with Pulumi free → Install the CLI and provision your first agentic-ready stack in minutes.

Happy clouding,

-Joe

Pulumi for All Your IaC — Including Terraform and HCL

Joe Duffy — Thu, 18 Dec 2025 14:00:00 -0800

We work with thousands of customers who prefer Pulumi due to our modern approach to infrastructure that delivers faster time to market with built-in security and compliance. Yet we know many organizations have years of investments into tools like Terraform. At the same time, HashiCorp customers are increasingly telling us about their frustrations post-IBM acquisition: rate increases, loss of open source heritage, overnight rug-pull of CDKTF, … and the hits just keep on coming. Today, we’re excited to announce three new ways Pulumi is enabling customers of HashiCorp, an IBM Company, who want a better, open source friendly, modern solution for their IaC to choose Pulumi. First, Pulumi Cloud will support Terraform and OpenTofu, so you can continue using any Terraform or Pulumi CLI and language with the complete Pulumi Cloud product, including our infrastructure engineering AI agent, Neo. Second, Pulumi’s own open source IaC tool will support HCL natively as one of its many languages, alongside the industry’s best languages including Python, TypeScript, Go, C#, Java, and YAML. Pulumi is multi-language at its core and many organizations are diverse and polyglot—these new capabilities truly make Pulumi the most universal IaC platform with the broadest support. Third, we’re offering flexible financing to make it easy to depart HashiCorp for Pulumi.

The TL;DR

Pulumi Cloud now manages Terraform/OpenTofu with full visibility, governance, and agentic AI included. Pulumi IaC now speaks HCL alongside general purpose languages and YAML. And we’ll cover your costs until your HashiCorp contract ends.

Terraform/OpenTofu in Pulumi Cloud

Pulumi has always been two things: Pulumi IaC, the multi-language, open source infrastructure as code technology, and Pulumi Cloud, our commercial platform that includes infrastructure state management and visibility, our AI agent Neo, secrets management, policy as code, governance, and much more. You can think of it like Git (the tool) versus GitHub (the SaaS).

Historically, we required that you choose Pulumi IaC to benefit from Pulumi Cloud but over time we’ve been moving away from that, such as with our AI and governance features: you point us at any cloud accounts, and we’ll help you make sense of and tame them, regardless of how the accounts’ resources were deployed (Pulumi, CDK, Terraform, … even manual point and click). Today, we are taking that one step further and letting you run Terraform, OpenTofu, or Pulumi IaC as your tool of choice, while still benefitting from Pulumi Cloud’s full suite of capabilities.

The great thing about this is that even if you choose Terraform or OpenTofu IaC on the client, you will still benefit from all of the capabilities of our server, from AI to governance to visibility and more. Terraform statefiles and workspaces will show up effectively the same way Pulumi stacks do, and you’ll get full visibility of who is changing what and when including diffs and logs.

Why would we do such a thing? As we’ve worked with larger and larger companies in our journey to thousands of customers, we’ve seen that there’s significant Terraform out there in the world. Even if a team’s long-term objective is to migrate to 100% Pulumi – reaping the many benefits of modern IaC, like faster time to market by catering better to a polyglot world of developers, infrastructure experts, security engineers, and AI/ML teams – that transition doesn’t happen overnight. Many teams legitimately want a mix of IaC tools. Ensuring all infrastructure is fully automated, secured, and managed is a more righteous outcome to focus on rather than debating one’s choice of IaC tool or language. There are many paths you can take, and now Pulumi can be your one platform to stay on top of all of it and drive towards this outcome.

Support for Terraform/OpenTofu state is generally available today, and we’re working with customers directly to bring their existing state under Pulumi Cloud’s governance, visibility, and Neo integration.

HCL Language Support in Pulumi IaC

At Pulumi, we love our languages. We now support six – depending on how you count: Python, TypeScript, Go, any .NET language (like C#), and any JVM language (like Java itself), and even YAML. Having this broad array of languages is a massive unlock: you suddenly get access to the full ecosystem of tooling and expertise around these languages, including rich syntax (for loops, if statements, functions), IDEs, testing frameworks, true sharing and reuse, and ensuring that LLMs deeply understand your IaC. This choice of language is then married with the best of declarative IaC, so you still get the belts and suspenders safety of a desired state IaC tool.

But we work with customers all the time where some of the team is more comfortable with and/or genuinely prefers HCL. The HCL language was purpose-built for IaC through Terraform and, now, OpenTofu and is easy for simple use cases. We actually shipped YAML support two years ago because it’s an industry standard and we kept hearing about simpler use cases where you didn’t need a full blown language (especially e.g. when code-generating IaC or supporting simple developer self-service CI/CD pipelines where a handful of lines of YAML do the trick). But despite that, there’s a ton of muscle memory with HCL in the IaC community.

We are not dogmatic about languages, we love all of them. The L in HCL and YAML stands for “language” and we’ve always had a “come one, come all” mindset. As soon as we see enough market demand for a given language, we will add it. Well, that time has come for HCL.

The good news is that this is not a bolt on. Just like any of the other Pulumi languages, you have full access to the entirety of the Pulumi ecosystem, including thousands of providers. Thanks to our Terraform bridge, if there’s a Terraform provider out there, it just works. This is explicitly not meant to be a lift-and-shift migration option – we have many other techniques for that, including the new Terraform support in Pulumi Cloud – but is instead for teammates who are more comfortable with or prefer HCL over general-purpose languages, but still want to leverage the more modern Pulumi IaC engine with its advanced multi-language capabilities.

HCL support also integrates with Pulumi’s multi-language technology in a deep way, so that you can author modules in one language and consume them from another. This will let, for example, platform teams author complex components in, say, Go – with the rich facilities offered by the language – and then expose them to teammates who consume them in HCL (or vice versa!)

Similar to Terraform support in Pulumi Cloud, HCL is now generally available as a first-class Pulumi language, alongside Python, TypeScript, Go, C#, Java, and YAML.

Builds on Existing Coexist/Convert Capabilities

These two new technical capabilities augment an existing array of tools that make it easy for you to choose Pulumi as your IaC platform of choice, even in organizations with hybrid client-side IaC tools, and languages may be general-purpose, HCL, or a mixture thereof.

Pulumi IaC supports using any Terraform provider. Many of them are available pre-built in the Pulumi Registry, however, in the event one isn’t pre-published, you can generate one on demand using the “Any Terraform Provider”. Pulumi doesn’t use the Terraform engine, but rather it leverages the resource schemas and create, read, update, and delete methods in them.

We have numerous migration tools. The first is Neo, our infrastructure engineering agent, who has Terraform-specific skills to migrate code and state. Neo leverages a number of building blocks that you can use directly instead. That includes the pulumi convert --from terraform command which understands how to convert Terraform/OpenTofu HCL or CDKTF code into a Pulumi language of your choosing, preserving code structure including migrating modules to Pulumi components. We also support directly importing resources from your cloud accounts directly, either at the CLI or visually in Pulumi Cloud. Read more about migration here.

You can deploy Terraform modules straight off the shelf from your favorite Pulumi language. That works as well for any supported Pulumi language, including HCL, and it shows up as though the underlying module resources were managed natively in Pulumi. This ensures if your team has a collection of battle tested best practices encoded as Terraform modules, and either plan to keep some Terraform or even migrate eventually, you don’t need to migrate right away.

Finally, you can reference Terraform state outputs from Pulumi programs and configurations. Using this you could, for example, keep your network defined in Terraform and define a layer of higher level infrastructure that consumes VPC and subnet IDs from that lower-layer network workspace. This can be useful if there’s short- or long-term coexistence between Terraform and Pulumi IaC programs, such as during an ongoing migration effort or in a hybrid team.

Pulumi Covered Until Your HashiCorp, an IBM Company, Renewal

Even with the new technical capabilities above, we realize many customers already have HashiCorp contracts that may have them locked into a single vendor. The last thing we want is for you to have to pay for two IaC solutions for a period of time. Additionally, we want to ensure that the transition is as smooth as possible and your team is set up for long-term success.

To make it easier to say yes to Pulumi, we are offering three things:

Escape hatch for your current contract. We know paying for two IaC solutions at once is a non-starter, so we’re letting you apply credits purchased from HashiCorp towards your Pulumi usage until your next renewal, avoiding double pay.
Free IaC modernization workshop. Our professional services cloud architects will host a free IaC modernization workshop to review where you’re at with your IaC already and share best practices of how to adopt the Pulumi platform at scale we’ve learned from working with world-class organizations like BMW, NVIDIA, and Supabase. You will leave this session trained up and equipped to succeed with the next phase of your IaC journey.
Return on investment (ROI) calculation. We will show you how the move to Pulumi will not only be spend-neutral thanks to the escape hatch, but how much value and savings you should expect to see, given our experience helping innovators like Snowflake accelerate their time to market – going from code to cloud in weeks to hours.

These ensure there’s no financial penalty for switching, a very clear ROI, and no learning curve. We have always been proud to work with customers of all sizes in all industries, so these offers are available to you whether you’re a Global 2000, startup, or somewhere in between.

Why Snowflake, BMW, and Wiz Switched to Pulumi

We’ve worked with thousands of companies over 8+ years to build what we view as the most innovative infrastructure as code platform on the market. We’re biased, of course, but our customers range from innovative startups, to established public technology companies, to Global 2000, and everywhere in between, and they tell us this all of the time.

For example:

Snowflake radically improved time to market on their path to IPO. “When we demonstrated to people that what used to take a week and a half now, with Pulumi, took under a day, they were shocked.”
Lemonade switched from Terraform to Pulumi so they could embed business logic into infrastructure, share and reuse logic, and scale their lean ops team to support a much larger group of developers. “We’re not limited to one-size-fits-all configurations, but can actually implement environment-specific customizations for our infrastructure.”
BMW was able to establish a center of infrastructure excellence that they call CodeCraft, standardizing all infrastructure delivery, and scaling to support 10,000+ developers.
Supabase was able to scale to meet the heightened demands and pace of AI, saying that “the infrastructure team acts as groundkeepers of our Pulumi practices, not gatekeepers, but promoters for the entire org."
Wiz manages over 1 million resources, tens of thousands Kubernetes clusters, and hundreds of data centers, with over 100,000 daily updates.

Here’s why:

Language choice. Especially with the addition of Terraform/OpenTofu support, and HCL language support natively in Pulumi IaC, the Pulumi platform truly is the most universal IaC platform available. It democratizes access to infrastructure across the entire organization which often has very varied and diverse skillsets. By embracing general-purpose languages, it allows engineering teams to apply rigor to how they manage infrastructure, improving productivity – translating directly to time to market – robustness, and standards compliance.

AI native. We’ve been infusing AI into our platform for over three years now, culminating in the release of our infrastructure engineering agent, Neo. Neo is like Claude Code for your infrastructure and allows you to automate short- and long-horizon infrastructure tasks, like spinning up and scaling infrastructure, upgrading clusters, getting compliant, reducing cloud waste, and so much more. Neo works over any infrastructure no matter how it was provisioned.

Standard enterprise capabilities and controls. Regardless of language or IaC tool choice, you still get one central standardized set of capabilities and controls. This ensures you have one “mission control” from which to standardize, secure, and govern your entire cloud estate, regardless of what client-side tool choice (or even in the presence of click-ops).

Full visibility of what’s happening, when, where, and why. The Pulumi platform always gives you total visibility into your cloud estate so you can quickly understand what is going on. This helps you wrap your hands around your cloud and chart a course to 100% IaC, the table stakes nirvana many organizations are trying to get to, and the choice of language is one small part.

The final point isn’t even a technical one, but we hear it from our customers all the time:

Customer love as our #1 company value. Our first company value is “when the customer is successful, we are too” – and we live it and breathe it daily. I just visited a new customer to review how the project is going yesterday and their head of cloud infrastructure and EVP of product were in the room and both went out of their way to express gratitude for how our team really showed up to help get them on the right track. We don’t view customers as a transaction, we view it as a lifetime partnership, and treat our customers with the respect they deserve. We won’t sleep at night until you’re on the cloud path that you envisioned when selecting Pulumi.

To learn more about our product capabilities, visit these pages:

Get Started Today

If you’d like to get started with the new Terraform/OpenTofu and HCL capabilities, or take advantage of the financial flexibility options, please get in touch.

We will work with you closely on a three-step process to adopt Pulumi: First, the modernization workshop; then a rapid proof of value; and finally an adoption plan that avoids you double paying Pulumi and HashiCorp.

If you’d like to try Pulumi on your own immediately, you can sign up for Pulumi Cloud here, or go through our open source IaC getting started tutorial here.

We can’t wait to earn the right to work together.

The Superintelligence Flywheel: Infrastructure for the AI Era

Joe Duffy — Thu, 11 Dec 2025 00:00:00 +0000

We’ve been in the infrastructure business for nearly a decade, and we’ve never been more excited about, or in awe of, the scale we are seeing as the industry pursues superintelligence. We are now hitting a tipping point that requires entirely different approaches to managing and scaling infrastructure in this new era.

What do we mean by superintelligence? Superintelligence means AI systems that operate with genuine autonomy—planning, reasoning, executing, adapting—at scale, on the path toward human-level and eventually superhuman intelligence. The infrastructure needed to accomplish this is greater than anything we’ve ever seen. Jensen Huang projects $600 billion in AI infrastructure spending this year, scaling to $3-4 trillion by decade’s end. Stargate committed $500 billion to AI infrastructure in the U.S. Microsoft, Meta, and Google are each spending $70-90 billion annually on datacenters. AWS just activated Project Rainier, a data center scaling to one million custom Trainium chips for Anthropic’s frontier models.

Superintelligence is driving the biggest, fastest infrastructure scaling period in the history of computing. This is exciting but comes with challenges: all of that infrastructure has to be managed, secured, scaled, made compliant, and cost effective. Legacy infrastructure tools weren’t built for this reality—they add friction that slows progress or breaks it altogether.

This reveals an important insight:

The infrastructure required to build superintelligence demands superintelligence for infrastructure.

The systems being built today for superintelligence are already straining human platform teams to their limits and yet we’re still only just getting started. To succeed, we will have no choice but to use AI itself to help us manage the infrastructure scaling ahead on the path to superintelligence.

Superintelligence demands more infrastructure, which demands superintelligent approaches to managing and scaling that infrastructure, which leads to faster progress towards superintelligence.

We call this the superintelligence flywheel.

The Flywheel Defined

The superintelligence flywheel describes a self-reinforcing cycle between two forces:

Superintelligence Infrastructure: The massively distributed compute, storage, networking, and orchestration systems required to train and serve frontier AI models.

Infrastructure Superintelligence: AI systems capable of managing infrastructure with increasing autonomy. These systems handle the full spectrum of infrastructure automation, from provisioning to securing to evolving and scaling entire environments.

These reinforce one another. As AI infrastructure grows more complex, human teams will increasingly use AI to manage it. This affects all of us: frontier labs, technology innovators, and enterprises modernizing their products and workforces with an AI-first mindset. Infrastructure agents are simply a requirement for all of us in this new world.

The flywheel accelerates.

The Technical Reality

Frontier labs have the most extreme requirements, but everybody else is somewhere on the curve. You don’t have to be training GPT-5 to feel this pressure. If you’re deploying fine-tuned models, running inference at scale, or just trying to keep up with AI-assisted developers shipping faster—you’re on the same curve, just earlier. To stay competitive and innovative in this new era, access to infrastructure needs to be democratized and inherently automation- and experimentation-friendly.

This can be seen most vividly with frontier lab-scale infrastructure challenges:

Training starts with data preparation. Petabytes of data turns into trillions of curated tokens across thousands of CPU cores. Pre-training runs massive GPU clusters hot for months, at a scale where hardware failures are statistical certainties. Checkpoint management, gradient synchronization, and fault-tolerant scheduling become first-order concerns. Fine-tuning and reinforcement learning add even more coordination complexity: synthetic data generation, AI judges, multiple model copies running simultaneously in tightly coupled loops, hundreds of experiment variants, infrastructure that scales elastically as experiments converge or diverge.

Inference means serving models to an ever-elastic sea of users across massive GPU fleets, many clouds, and distributed globally. ChatGPT, for instance, handles over 2.5 billion queries per day with sub-second time-to-first-token expectations. Traffic patterns shift by geography, time, and viral adoption, and the software must be ready to react. The difference between good and poor GPU utilization at this scale can make or break the user experience and the bank.

These examples share three properties that break traditional infrastructure approaches:

Massive scale: Thousands or millions of coordinated resources with complex architectures across multiple regions and cloud providers.
Extreme dynamism: Infrastructure that must spin up, reallocate, and tear down continuously based on job requirements, spot pricing, capacity availability, and failures.
Velocity beyond human capacity: Infrastructure changes that occur faster than human operators can review, approve, and execute manually, often initiated by AI agents.

Of course, even smaller scale examples share these challenges. Whether you’re using AWS SageMaker or Google Vertex AI to fine-tune models for your enterprise, building out your data engineering function to train custom models, or moving your first ML workloads to production—these infrastructure demands become relevant as you scale.

AI-assisted software development tools are making engineers dramatically more productive. Developers are shipping more code than ever before. But unlike software engineering, legacy infrastructure practices lack inherent guardrails to enable AI-assisted infrastructure engineering: linters, static analysis, testing frameworks, canaries, staged rollouts. The tooling that worked when humans were the limiting factor won’t work when AI agents are generating and deploying infrastructure code at machine speed.

Infrastructure Fungibility: The Holy Grail

Instead of treating each use case as a special snowflake, the holy grail is infrastructure fungibility across many clouds. In this world, infrastructure is dynamically allocated based on availability, demand, and data locality. Inference gets served based on current user demand. Experiments get capacity when it’s available. Long-horizon training runs on whatever’s left over.

This is extraordinarily difficult. But without it, the costs compound: researchers waiting for capacity, GPU-hours sitting idle between workloads, time delays that let competitors ship first. At frontier scale, even small inefficiencies translate to tens of millions of dollars.

This level of dynamism can’t be managed by humans watching dashboards and approving PRs. The infrastructure itself needs to become intelligent—capable of planning, executing, and adapting without waiting for human operators.

The frontier labs building this have learned through direct experience. They have historically had to work around legacy infrastructure tools and slow-moving DevOps teams, taking matters into their own hands. As one infrastructure lead put it: “There are thirty people in the world that know how to do this right now.” It is time for this knowledge to spread faster, in the hands of platform teams, because everyone building with AI is running into some version of the same problem.

Towards a Superintelligence Platform

There are two foundations that form a cohesive superintelligence platform:

Foundation #1: Pulumi as the Infrastructure Automation Substrate

We built Pulumi on three principles that are well-suited to the unique challenges of this era:

Infrastructure as code must be actual code. Domain-specific languages and static configuration formats create a ceiling. Instead, standing on the shoulders of giants with real programming languages provides type systems, conditionals, loops, abstractions, package management, testing frameworks, and more. These capabilities facilitate scale. Just as importantly, your team already knows Python, TypeScript, and Go—why force them to learn yet another proprietary DSL and context-switch constantly between application code and infrastructure configuration?

But there’s a deeper reason this matters now. LLMs have been trained on billions of lines of Python, TypeScript, and Go, and deeply understand how to code correctly. Infrastructure in real programming languages becomes infrastructure that AI systems can reason about, generate correctly, and modify safely. We call this an “LLM-sympathetic architecture”: by projecting cloud infrastructure into code space, LLMs can manipulate infrastructure through coding, something they are already superb at which ensures it can be done repeatably and correctly.

One substrate across all of the clouds. Every cloud, every SaaS provider. Frontier labs run training jobs wherever capacity exists: AWS, Azure, Google Cloud, Oracle Cloud, CoreWeave, Lambda Labs. GPU availability is the constraint; cloud provider preference is now secondary.

A single lab may prefer to train on CoreWeave, overflow to AWS when spot capacity becomes available, and tap Google Cloud when the reservations in other cloud providers are used up. Inference deployments span every region where users exist, across whichever providers offer the best latency and cost characteristics.

Infrastructure platforms that aren’t inherently multi-cloud simply can’t serve these requirements, forcing teams to adopt many fragmented tools and workflows. A platform for the superintelligence era simply must treat multi-cloud as a given with unified resource management, policy enforcement, and state tracking across all providers.

Agent-ready architecture. Human-in-the-loop infrastructure management doesn’t scale to the velocity of the new era. The path forward is progressive autonomy: AI agents that operate infrastructure with increasing independence, bounded by human-defined policies and guardrails.

This requires platforms designed for agent interaction from the ground up, with APIs agents can invoke, state they can reason about, policies that define boundaries, audit trails that maintain accountability. The infrastructure platform becomes the substrate on which agents operate. Infrastructure as code, it turns out, is already very good at this thanks to its declarative core.

Foundation #2: Neo as the Agentic Layer

If managing superintelligence scale infrastructure requires AI agents that can handle infrastructure automation, the critical question is: what should those agents look like?

Neo is our answer. Neo is an AI agent that operates infrastructure like a senior platform engineer, built on top of Pulumi’s foundation. Because Pulumi already provides the guardrails that software engineering takes for granted—previews, policy enforcement, state tracking—Neo can handle infrastructure tasks automatically while validating changes before they hit production.

Uniquely enabled by infrastructure as code. Neo operates on infrastructure code, not opaque configurations or cloud APIs directly. Even for resources not yet managed by Pulumi, Neo seamlessly imports them under code management before making changes. Every action flows through version control, policy checks, and state tracking. This ensures the underlying LLM’s coding smarts apply to infrastructure changes.

Built for progressive autonomy. Organizations configure Neo’s independence by environment and task type. Dev environments might permit fully autonomous operation—daily waste cleanup, weekly drift reconciliation—while production changes may require human approval. When Neo encounters unexpected state or errors, it can self-diagnose or loop in a human for assistance as needed. As confidence builds, the autonomy boundary expands, towards full autonomy.

Inherently understands and respects guardrails. The same velocity that creates opportunity creates risk. An agent that can provision 1,000 resources can also misconfigure 1,000 resources. Pulumi’s infrastructure as code foundation provides the safety mechanisms software engineering takes for granted: previews before deployment, policy as code enforcement, audit trails, human-in-the-loop approvals, and organizational best practices encoded in components. Just as you wouldn’t ship code without version control, you shouldn’t let AI manage infrastructure without these guardrails.

Agentically automate anything: Give Neo a prompt like “Provision a PyTorch training environment for LoRA fine-tuning on GKE with 50 A100s using Nvidia’s GPU Operator and Kueue—then run a smoke test.” Neo generates the project, previews changes, provisions everything, verifies it works. Days become minutes. It’s not just for new infrastructure: one customer used Neo to tackle 300,000+ policy violations—work they estimated would take years is now done in weeks.

Frontier models plus infrastructure smarts: Neo builds on a mixture of frontier models while incorporating nearly a decade of infrastructure expertise. Neo benefits from Pulumi’s nearly one billion deployments, hundreds of thousands of AI interactions and known-to-be-working code examples, and many petabytes of semantically-understood, anonymized infrastructure metadata. This foundation is something that no other company in the world has at their disposal.

Pulumi + Neo: The Superintelligence Platform

Pulumi without Neo is already a powerful infrastructure platform—but one that still requires human operators for every change. Neo without Pulumi would never be production-ready, as it would lack the cloud-to-code projection and guardrails that make Neo so good. Together, humans define intent and policy, and intelligent agents handle execution with autonomy.

This is the foundation for setting the superintelligence flywheel in motion.

Evidence at Scale

The flywheel isn’t theoretical. We’re facilitating it daily with thousands of customers:

Hyper-scale: Wiz manages over one million cloud resources across tens of thousands of Kubernetes clusters in hundreds of data centers across every cloud provider imaginable. They do hundreds of thousands of updates daily across thousands of Kubernetes clusters, leading to 5x faster customer onboarding and entering new markets in days rather than months. They treat infrastructure provisioning as just another service in their distributed system.

Infrastructure democratization: Supabase achieved the infrastructure velocity needed to keep up with the AI era’s demand for their own product, empowering 20x more teammates to scale infrastructure in TypeScript, the same language as their application code, rather than proprietary DSLs that nobody understood. New regions deploy in about a week, with 43,000 new databases daily and 100,000+ API calls per second. According to their platform team, now thanks to Pulumi, “the infrastructure team acts as groundkeepers, not gatekeepers.”

Frontier science: We manage infrastructure for organizations including NVIDIA and multiple frontier labs building the models that will define the next decade. The scale requirements for these workloads exceed any we’ve ever seen and are only getting bigger. Just as many customers democratize infrastructure for their developers, we’re seeing frontier labs use Pulumi to accelerate getting science into production with self-service infrastructure for AI researchers.

The Superintelligence Flywheel is in Motion

AI is driving the largest infrastructure buildout in history—whether you’re training frontier models or deploying your first ML workloads to production. That complexity demands intelligent agents. Organizations that embrace this—building infrastructure for AI while using AI for infrastructure—will define the next era of computing.

The superintelligence flywheel is spinning fast, and it’s only going to get faster from here.

Pulumi provides the infrastructure platform for the superintelligence era: infrastructure as code in general-purpose languages, unified multi-cloud management, and Neo for progressive infrastructure autonomy.

Pulumi Neo ・ Superintelligence Infrastructure ・ Case Studies ・ Contact

Vibe Clouding: Give In, Forget That Cloud Infrastructure Even Exists

Joe Duffy — Tue, 01 Apr 2025 00:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

By this point in 2025, we’ve all heard about “vibe coding”: the AI-fueled craze that enables even my 2 year old nephew to build new applications by simply giving into the vibes, embracing exponentials, and forgetting that the code even exists. Vibe coding enables anybody who can type on a keyboard or speak to a computer to build IPO-worthy software businesses overnight. Today we are excited to introduce vibe coding’s similarly revolutionary close cousin: “vibe clouding”. By giving into the vibes, you can now spin up cloud infrastructure anywhere and everywhere, all by just saying stuff, copy and pasting stuff, and vibing. And even better, it mostly works! Read on to learn more, or just watch the video below.

The Code Vibes

In just two months, vibe coding has transformed the entire coding profession. If you aren’t familiar with the concept, this post by Andrej Karpathy introduces the idea:

Enter the Cloud Vibes

Today we’re announcing an entirely new way of managing your infrastructure, vibe clouding. The vibes are no longer limited to application developers; as a platform engineer, you too can now vibe like the best of them. The combination of natural language support thanks to LLMs and an improved pulumi watch command put you in the flow and the cloud at your fingertips.

Infrastructure as Code in Any Language. No, Really

Today we’re introducing the ability to write declarative IaC in natural language. No code necessary.

IaC is now table stakes for anybody doing anything in the cloud. Pulumi’s unique approach (which is open source, by the way) allows you to use any programming language to declare cloud infrastructure, such as Python or TypeScript, and its declarative model brings you desired goal state into existence as if by magic.

Declaring Cloud Infrastructure in Natural Language

Now you can express infrastructure in any language, programming or otherwise. For example, this IaC “program” written in English creates a Python webserver in AWS that’s accessible over the Internet:

An AWS EC2 instance running a Python web server.
It accepts traffic over the Internet on port 80 and simply responds "Hello, World."
Capture and export its auto-assigned IP address and DNS name.

We save this in a *.pai file (pai = Pulumi AI), and then deploy it with pulumi up like normal. The new language host asks Pulumi’s AI Copilot to figure out the infrastructure we’ve requested and then shows us a preview of what it will do:

$ pulumi up
Previewing update (dev)

 Type Name Plan Info
 + pulumi:pulumi:Stack vibeclouding-dev create 2 messages
 + ├─ aws:ec2:SecurityGroup webSecurityGroup create
 + └─ aws:ec2:Instance webServerInstance create

Diagnostics:
 pulumi:pulumi:Stack (vibeclouding-dev):
 🧠 Thinking...
 🧠 I am happy with this, let's go!

Outputs:
 InstanceDNS: output<string>
 InstanceIP : output<string>

Resources:
 + 3 to create

Do you want to perform this update? [Use arrows to move, type to filter]
 yes
> no
 details

We can select “yes” and it will deploy our cloud stuff:

Updating (dev)
Type Name Status
+ pulumi:pulumi:Stack vibeclouding-dev created (17s)
+ ├─ aws:ec2:SecurityGroup webSecurityGroup created (2s)
+ └─ aws:ec2:Instance webServerInstance created (12s)
Outputs:
publicDns: "ec2-54-189-144-3.us-west-2.compute.amazonaws.com"
publicIP : "54.189.144.3"
Resources:
+ 3 created
Duration: 18s

Seconds later, after it completes, we’ve got a Python webserver on the Internet:

$ curl 54.189.144.3
Hello, World.

Not Just English

This is a new language runtime just like Pulumi’s other supported languages (python, go, java, yaml, etc). We almost called this new language english, except that we can use other languages:

Spanish:

Una instancia de AWS EC2 que ejecuta un servidor web Python.
Acepta tráfico de internet en el puerto 80 y simplemente responde "Hola a todos".
Captura y exporta su dirección IP y nombre DNS autoasignados.

Mandarin Chinese:

运行 Python Web 服务器的 AWS EC2 实例。
它通过端口 80 接受 Internet 上的流量，并简单地响应“Hello, World”。
捕获并导出其自动分配的 IP 地址和 DNS 名称。

Hawaiian:

ʻO kahi hiʻohiʻona AWS EC2 e holo ana i kahi kikowaena pūnaewele Python.
ʻAe ʻo ia i ke kaʻa ma luna o ka Pūnaewele ma ke awa 80 a pane wale ʻo "Aloha, World."
Hopu a hoʻokuʻu aku i kāna helu IP i hāʻawi ʻia a me ka inoa DNS.

The LLM is happy to accept any language known to us humans. So we are simply calling it ai. You just specify runtime: ai in your Pulumi.yaml and all other Pulumi features work as normal:

name: py-webserver
runtime: ai

It Is Feature Rich and Works for Real Workflows

If you change your mind and want it to use a Node.js web server … or listen on port 8080 … or run on Azure instead of AWS … with a different response … Simply change your prompt:

A virtual machine in Azure running a Node.js web server.
It accepts traffic over the Internet on port 8080 and simply responds "Hello, Vibe Coders!"
Capture and export its auto-assigned IP address and DNS name.

We then rerun pulumi up and it knows how to evolve your infrastructure towards the new goal state.

Your prompts can be as imprecise or specific as you’d like. For example, “Run a joke of the day application on Google” needn’t tell the LLM what sorts of infrastructure you want – it will just intelligently decide between a GCE VM, Docker container in Google Cloud Run, GKE app, etc.

You can also spread your prompts across any number of *.pai files: so you can stick your webserver into webserver.pai, your database in database.pai, and so on, and you can even cross-references resources between files, and the LLM will figure it out. Just vibe and out pops cloud infrastructure!

Watch Mode for Rapid Iteration and … Vibing

This is already amazingly cool, but we haven’t begun vibing just yet. We are halfway there.

Today we’re also introducing a vastly improved watch command to properly unlock the vibes.

The little-known pulumi watch command sits there waiting for changes to any parts of your Pulumi IaC and in response to you hitting save will redeploy the changes. It turns out that this combined with native LLM language support yields the magical combination that unlocks vibe clouding.

Let’s take our original example above but use the pulumi watch command to trigger the deployments:

Just write specifications fluently and hit save whenever it’s convenient. Pulumi incrementally deploys what it can, when it can, reports on status interactively, and summarizes what it did afterwards.

If your thought is half-baked, exploratory in nature, or you hit an unexpected hiccup, thanks to the declarative nature of IaC, it can always pick up safely where it left off. Don’t worry, just vibe.

This is Just the Beginning

By now, I hope you’ve realized what date it is today: April 1st. April Fool’s Day. We just couldn’t help ourselves. So this is all just a joke. Sort of.

Much of this is actually becoming reality thanks to the intersection of multiple factors:

Pulumi use of “code” pairs well with the LLMs’ ability to produce correct code.
Pulumi language extensibility enables adding “natural language” as a language provider.
Pulumi Copilot, launched less than a year ago, gives us skills that make the LLMs even better at infrastructure tasks.
The LLM models are getting so darned good … and will get even better still.

The improvements to pulumi watch are real and already in the latest Pulumi release. The pulumi-language-ai language host is, of course, just an early prototype of what’s possible, but we look forward to hearing your feedback if you have the chance to kick the tires. Even more exciting things are on their way throughout 2025 and they won’t be launched on April 1st next time.

Get Started Vibing

If you want to get started, simply:

Download Pulumi
Download the pulumi-language-ai host and ensure it’s on your PATH
Set OPENAI_API_KEY to a valid OpenAI key
Run pulumi watch, begin typing, copy and pasting, and vibing

We highly recommend playing this video on repeat in the background as you do so:

Happy vibe clouding.

Pulumi - Automate, Secure, and Manage Everything You Run in the Cloud

Joe Duffy — Wed, 18 Sep 2024 09:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

At our fourth annual PulumiUP conference, we unveiled a new vision for Pulumi, going beyond infrastructure as code to a comprehensive product suite. Pulumi’s platform now consists of three core products: Pulumi IaC is open source infrastructure as code in any programming language; Pulumi ESC is centralized secrets management that scales; and Pulumi Insights delivers instant visibility, intelligence, and control over all of your infrastructure. Each product can be adopted independently but is better together. In this post, we will share more about how Pulumi is your trusted partner to automate, secure, and manage everything you run in the cloud.

Pulumi - One Unified Platform For All Your Infrastructure Needs

The Pulumi platform solves a broad set of problems that engineering leaders face in the modern cloud era. This spans cloud automation, security, and management, which aligns to our three products: Pulumi IaC, Pulumi ESC, and Pulumi Insights. Our unified approach ensures developers, infrastructure experts, and security teams can collaborate closely and ship faster with high confidence.

In addition to our three products, Pulumi’s platform incorporates three core capabilities that span the entire product suite. Pulumi CrossGuard is our policy as code engine which ensures security, compliance, and best practices; Pulumi Deployments is our deployment workflow technology that lets us orchestrate infrastructure-oriented tasks; and Pulumi Copilot is our generative AI technology that deeply understands the cloud and infrastructure management and operations.

You can see this visually depicted here:

We’ve now worked with 3,000 customers to help with their cloud journey and have listened to and solved the problems we regularly hear about. That includes Nvidia who is at the forefront of the AI revolution as well as BMW who is reimagining their entire engineering mindset in a cloud-first world, both leveraging Pulumi. Unity Games now ships 5x faster from code to cloud, and Tivity Health saved 71% of their cloud spend, automating everything and saving $6.5M annually.

Now let’s dive into the three products and what we announced at PulumiUP:

Pulumi IaC - Infrastructure as Code in Any Programming Language

Pulumi IaC is our flagship open source technology, and it powers virtually everything we do, even the new products. We have seen accelerated adoption of Pulumi IaC. We’ve had over 100M downloads, are about to crack 21K GitHub stars (give us a star and bump us over!), and have seen 167% the contributions of Terraform this past month and 300% of OpenTofu. Not only is Pulumi the most powerful IaC technology, it is becoming more popular and vibrant too.

We are always improving Pulumi IaC and today is no different. We launched a number of new and exciting IaC features. That includes using any Terraform provider, beyond the nearly 200 providers that Pulumi already supports. We also shipped a new Pulumi VS Code Extension that enables rich debugging and easier editing of your IaC programs. Pulumi Copilot can now diagnose and fix your IaC errors. And finally, we’ve improved our Kubernetes support with powerful controls over await logic for more robust cloud native deployments. And we’re not slowing down anytime soon.

Pulumi is open source, of course, but works best with Pulumi Cloud, the easiest, most secure, and most reliable way to adopt Pulumi IaC. Starburst saved 100 days per year by moving from DIY state management to Pulumi Cloud. By using Pulumi Cloud, you also gain access to the other products in this post. Try IaC now and choose the option that suits you best.

Pulumi ESC - Centralized Secrets Management and Orchestration that Scales

Pulumi ESC is a new approach to secrets and configuration management that is scalable, secure automatically with easy dynamic credentials, and tames sprawl with engineering best practices. ESC is not just a feature of our IaC product — it can be adopted entirely independently of it for application and infrastructure secrets. We initially launched ESC in preview nearly a year ago, and since then have had hundreds of customers and thousands of users try it out – many at production scale – who have given us amazing feedback. We are excited to announce that ESC is generally available.

Being generally available is a big deal to us, and means it is ready for robust and secure enterprise-grade workloads. New capabilities available at GA include a new projects system which lets you group your environments into familiar hierarchies. We have added versioning, tagging, and pinning, so that you can institute more robust change management practices. ESC can manage encrypted secrets directly, but we have always supported pulling secrets from other secret stores, like AWS Secrets Manager, Azure KeyVault, Google Secret Manager, and HashiCorp Vault. We have now added 1Password to the list. We now also support secrets sync, which lets you still edit your secrets straight from Pulumi ESC, but have them automatically mirrored back to the backing secrets store. Flexible webhooks now support custom workflows triggered by environment changes. The new Pulumi VS Code Extension helps you edit your environments in code. We’ve launched application SDKs for Node.js, Python, and Go, as well as a Kubernetes Operator to inject secrets into Kubernetes workloads.

We aim for you to “fall into the pit of success” with security, so that it is built-in and automatic, and ESC is a major enabler of that. Try out ESC now, or read more in the launch post.

Pulumi Insights - Visibility, Intelligence, and Controls Over All Infrastructure

Pulumi Insights already delivers visibility, intelligence, and controls over your infrastructure resources. We began with Insights a year and a half ago as a collection of IaC features. We added resource search with structured and natural language search syntax, to help you find your IaC infrastructure no matter where it is running. We added the ability to export your infrastructure data and slice and dice it from data tools like Snowflake. We also added new dashboards to help visualize your infrastructure at scale.

The big news with Pulumi Insights 2.0 is that we’re bringing these capabilities to all of your cloud infrastructure, no matter where it is, and no matter how it was created. You point Pulumi at your cloud accounts for any of nearly 200 supported clouds, and you get an instant asset inventory of everything you have. This is true even for resources created by Terraform, CloudFormation, Azure Resource Manager, cloud CLIs and SDKs, or even by clicking in your cloud console. An improved resource view gives you pivot-table-like capabilities so you can understand everything you have, and new graph visualizations with inferred dependencies help you to make sense of how things relate. Search and Pulumi Copilot work over discovered cloud resources, letting you explore and ask questions interactively. Pulumi Insights leverages Pulumi CrossGuard, our policy as code technology, and can tap into any existing IaC policies to find issues pertaining to security, compliance, cost, and best practices. Pulumi’s compliance-ready policy packs just work. Insights can also leverage CrossGuard’s ability to automate remediations so that Insights not only tells you about your issues, but helps you fix them with the click of a button. Finally, Insights is better together with Pulumi IaC, and enables you to migrate any resources in the cloud to Pulumi IaC – improving the robustness and built-in security of your operations – with a fully interactive import experience.

This marks the start of an ambitious long-term vision for Pulumi Insights. Some of what was announced is actively shipping while other pieces will ship in the coming weeks. Learn more in the launch blog and, get started with Pulumi Insights 2.0.

Thank You

I am incredibly thankful to our community, 3,000 customers, and the partners and speakers who made PulumiUP 2024 a reality. Your feedback and continued belief in Pulumi has fueled the innovations we’re shipping today and into the future.

All of the talks are streaming live here and will be up on YouTube afterwards.

We can’t wait to hear what you think about the new vision for Pulumi’s platform as well as the new IaC, ESC, and Insights capabilities announced at PulumiUP 2024. As always, you can try Pulumi for free here.

Happy cloud spelunking.

-Joe

Announcing New Infrastructure Lifecycle Management Features in Pulumi Cloud

Joe Duffy — Wed, 24 Apr 2024 07:00:05 -0800

Today we announced a new wave of Infrastructure Lifecycle Management capabilities in Pulumi Cloud. These build upon and extend our rich Pulumi Deployments foundation for deployment orchestration and workflow. This is an exciting, customer-driven release, with new features that help with Day 2 operations and management.

These new features are:

Drift Detection and Remediation - Continuously detect and remediate drift
Time-to-Live Stacks - Clean up stale infrastructure automatically
Scheduled Deployments - Automate recurring workflows on a schedule
Free Deployments Minutes - Use all of the above for free, no matter your tier!

We built these features based on customer feedback, and our 2,500+ customers are excited:

“At Oleria, we understand the importance of addressing issues with privacy, security, and data integrity. Earning and maintaining our trust isn’t just a responsibility but a fundamental aspect of our mission,” said Jim Alkove, CEO of Oleria. “Pulumi also understands how these attributes affect cloud infrastructure. Our team is excited to use these new capabilities, giving us even more confidence as we enable our customers to securely manage access to decentralized SaaS applications, adaptively and intelligently.”

These advanced features are available today in the Enterprise and Business Critical editions of Pulumi Cloud. These features together enable an end-to-end experience for managing the lifecycle and consistency of your cloud infrastructure to ensure enterprises have the capabilities needed to operate at scale.

Infrastructure Lifecycle Management Capabilities - Excelling at Day 2 and Beyond

So, why infrastructure lifecycle management?

Infrastructure as Code (IaC) is table stakes for our customers. It helps to ensure Day 1, 2, and beyond cloud practices are robust, secure, and dependable. However, operational realities mean that sometimes teams need more flexibility than what strict IaC and Git-driven deployments offer.

For example, maybe you need to break glass and interact with the AWS console directly to open up SSH to an EC2 instance in order to debug a gnarly issue. Or maybe you need to scale up your Auto Scaling group (ASG) temporarily while a live-site incident of unexpected traffic peaks plays out. Or maybe someone added tags to your resource without knowing it was under the management of Pulumi. This can lead to drift, where your IaC configuration no longer matches reality, and reality no longer matches your IaC. This is where Drift Detection and Remediation comes in to automatically report on and reconcile it.

Alternatively, you may want to enable self-service infrastructure provisioning for your entire team, including developers and data scientists, who are not infrastructure experts. But what if they forget to wind things down when they’re done? You are left spending valuable money and resources managing infrastructure that’s no longer needed. Time-to-Live Stacks automatically reclaim such ephemeral infrastructure, enabling self-serve with confidence.

Next up, we often want to deploy on a Git commit, which Pulumi Deployments already handles. However, what if we want more flexibility around how and when deployments occur? For instance, perhaps you’re a retailer who doesn’t want to deploy on Black Friday. Or maybe you prefer to deploy at set times throughout the week rather than on every Git commit. This is where Scheduled Deployments comes in, to give us flexible cron-based scheduling of deployments.

Finally, we want everyone to be able to kick the tires with all of the above, no commitments needed. That’s why we’re also announcing today that all users – individuals, teams, and enterprises alike – get an allotment of free monthly deployment minutes to use all of this.

Get started now for free at app.pulumi.com. We can’t wait to hear what you think!

Happy cloud spelunking.

10x Your Storage Performance with Amazon S3 Express One Zone and Pulumi

Joe Duffy — Mon, 04 Dec 2023 00:00:00 +0000

At AWS re:Invent 2023, AWS announced the new Amazon S3 Express One Zone storage class. This new service provides incredible performance by collocating your S3 buckets closer to the workloads that use the data. Compared to the S3 Standard storage class, the Express One Zone storage class is up to 10x faster, handles 100,000s of requests per second, offers single-digit millisecond latency, and can reduce request costs by 50%. This can be extremely beneficial for data-intensive workloads such as AI/ML, media, finance, realtime, and high-performance computing scenarios. This blog post shows how to get started with Amazon S3 Express One Zone using Pulumi infrastructure as code.

Introducing Amazon S3 Express One Zone

The new Amazon S3 Express One Zone storage class extends S3 with a new low latency option. This can improve runtime performance by spending less time moving data around. When creating a new bucket, you choose an Availability Zone (AZ) in which that bucket will be placed, which of course should be collocated with the compute workload that will consume that data.

A bucket must be designated as an Express One Zone bucket at creation time. There are a few things that are unique to these buckets that make them different from ordinary S3 buckets:

First, they are called “directory buckets” in AWS tools and in your infrastructure as code. This means how you interact with them sometimes differs from general-purpose S3 buckets.
They have a specific naming convention: [name]--[az]--x-s3. The [name] part is a bucket name with the usual restrictions. [az] is the ID of the AZ in which the bucket will reside. Note that this is an ID (e.g. usw2-az1), not the name (e.g. us-west-1c); read more here about the difference.
These buckets use region-specific endpoints, not the region-agnostic aws-global S3 endpoint scheme that general-purpose S3 buckets utilize. This means you’ll need a new AWS CLI or SDK that understands the new endpoint scheme in order to access these buckets (CLI version 2.14.5 and greater supports them). It also means that sometimes when running aws s3 and aws s3api commands, you will need to specify a --region explicitly. Other aspects of accessing these buckets differ, such as authorization, but the AWS SDK generally takes care of these for you.

This new storage class is not universally available in all regions and AZs yet, although Amazon has plans to expand regions over time. At the time of launch, us-east-1 (US North Virginia), us-west-2 (US Oregon), eu-north-1 (Europe Stockholm), and ap-northeast-1 (Asia Pacific Tokyo) are supported. Finally, because One Zone is explicitly single-AZ, there is always the risk of availability issues or data loss in the event that a single AZ becomes unavailable or is damaged; plan accordingly.

New BucketDirectory Resource Type

You can create a new Amazon S3 Express One Zone bucket using the aws s3api create-bucket CLI command or in the AWS console. For this blog post, however, you’ll use infrastructure as code instead, so that provisioning, updating, and managing our buckets is automated, reliable, and easy to scale.

To do this, you will use Pulumi’s new aws.s3.DirectoryBucket resource type in the AWS package. It takes a few parameters:

bucket, which is the name (conforming to the above-mentioned [name]--[az]--x-s3 convention)
dataRedundancy, which currently just accepts the sole value "SingleAvailabilityZone"
location, which specifies the AZ ID to use
forceDestroy, an optional boolean which, when set to true, allows deleting buckets that still have data in them (by default, only empty buckets can be deleted, to prevent accidental data loss)

Here is a simple program that declares a new Amazon S3 Express One Zone bucket:

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";
import * as random from "@pulumi/random";

// Pick a single AZ for our S3 Express bucket.
const azId = aws.getAvailabilityZones().then(azs => azs.zoneIds.sort()[0]);

// Name our bucket according to S3 Express requirements ([name]--[az]--x-s3).
const bucketName = "my-bucket";
const suffix = new random.RandomString("bucket-name", {
 length: 6, upper: false, special: false });
const bucketFullName = pulumi.interpolate
 `${bucketName}-${suffix.result}--${azId}--x-s3`;

// Provision the bucket and export its ID.
const bucket = new aws.s3.DirectoryBucket(bucketName, {
 bucket: bucketFullName,
 dataRedundancy: "SingleAvailabilityZone",
 location: {
 type: "AvailabilityZone",
 name: azId,
 },
});

export const bucketID = bucket.id;

import pulumi
import pulumi_aws as aws
import pulumi_random as random

# Pick a single AZ for our S3 Express bucket.
az_ids = aws.get_availability_zones().zone_ids
az_ids.sort()
az_id = az_ids[0]

# Name our bucket according to S3 Express requirements ([name]--[az]--x-s3).
bucket_name = 'my-bucket'
suffix = random.RandomString(
 'bucket-name', length=6, upper=False, special=False)
bucket_full_name = pulumi.Output.format(
 '{0}-{1}--{2}--x-s3', bucket_name, suffix.result, az_id)

# Provision the bucket and export its ID.
bucket = aws.s3.DirectoryBucket(bucket_name,
 bucket = bucket_full_name,
 data_redundancy = 'SingleAvailabilityZone',
 location = {
 'type': 'AvailabilityZone',
 'name': az_id,
 },
)

pulumi.export('bucketID', bucket.id)

There are a few things to note here:

We’re picking the first AZ automatically in whatever region we’ve deployed to. Of course, we could hard-code this, or use a different technique for choosing an AZ. This approach makes it trivial to deploy the same project to multiple regions.
We use a random string suffix for our bucket name, just to avoid accidental collisions if we were to pick the same bucket name as another user (or even ourselves when deploying multiple stacks for the same project). Normally Pulumi uses auto-naming to avoid this problem by default, but because the bucket name must conform to a very precise format, it does not for this resource type.

Now let’s say we actually want to populate our bucket. We can use the standard aws.s3.BucketObjectv2 resource to do this. This program adds an object for every file in our project’s data directory, for example:

import * as fs from "fs";
import * as path from "path";

// ...

// Add a bucket object for every data file.
for (const file of fs.readdirSync("data")) {
 new aws.s3.BucketObjectv2(`my-file-${file}`, {
 bucket: bucket.id,
 source: new pulumi.asset.FileAsset(path.join("data", file)),
 }, { parent: bucket });
}

import os

# ...

# Add a bucket object for every data file.
for file in os.listdir('data'):
 path = os.path.join('data', file)
 if os.path.isfile(path):
 aws.s3.BucketObjectv2(f'my-file-{file}',
 bucket = bucket.id,
 source = pulumi.asset.FileAsset(path),
 )

Creating and Accessing an Express One Zone Bucket

Let’s go ahead and spin up a bucket. Assuming we have the above code in a valid Pulumi project (created with pulumi new), let’s ensure it is configured to one of the regions that supports Amazon S3 Express One Zone, such as us-west-2:

$ pulumi config set aws:region us-west-2

Now let’s create some random files in our data directory:

$ mkdir data
$ for i in {0..9}; do \
 dd if=/dev/urandom of=data/file${i}.txt bs=100K count=1; \
 done

Next, we’ll run pulumi up which will show us a preview of the deployment and ask us to confirm, which we’ll do:

$ pulumi up
 Type Name Status
 + pulumi:pulumi:Stack s3-express-one-zone-py-dev created (4s)
 + ├─ random:index:RandomString bucket-name created (0.09s)
 + ├─ aws:s3:DirectoryBucket my-bucket created (2s)
 + ├─ aws:s3:BucketObjectv2 my-file-file2.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file3.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file1.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file4.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file5.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file7.txt created (2s)
 + ├─ aws:s3:BucketObjectv2 my-file-file8.txt created (2s)
 + ├─ aws:s3:BucketObjectv2 my-file-file9.txt created (2s)
 + ├─ aws:s3:BucketObjectv2 my-file-file0.txt created (3s)
 + └─ aws:s3:BucketObjectv2 my-file-file6.txt created (3s)

Outputs:
 bucketID: "my-bucket-z5tn74--usw2-az1--x-s3"

Resources:
 + 13 created

Duration: 11s

And there we go – our new Amazon S3 Express One Zone bucket is available and its ID has been printed. We can check out its contents:

$ aws s3api list-objects-v2 \
 --region=us-west-2 \
 --bucket $(pulumi stack output bucketID)

Note that the storage class of these objects is listed as EXPRESS_ONEZONE:

{
"Contents": [
{
"Key": "my-file-file6.txt",
"LastModified": "2023-12-03T17:18:06+00:00",
"ETag": "\"1254d05e685245ef9fdcca10f3c547bc\"",
"ChecksumAlgorithm": [
"CRC32"
],
"Size": 102400,
"StorageClass": "EXPRESS_ONEZONE"
},
...
{
"Key": "my-file-file2.txt",
"LastModified": "2023-12-03T17:18:04+00:00",
"ETag": "\"d5310d3efd394c96bf5bf4379fc81bff\"",
"ChecksumAlgorithm": [
"CRC32"
],
"Size": 102400,
"StorageClass": "EXPRESS_ONEZONE"
}
]
}

We can also see our bucket and its contents in the S3 Management Console:

General-purpose buckets and directory buckets have different tabs. There are also utilities available to import data from other buckets and perform batch operations.

Benchmarking Express One Zone Performance

To demonstrate how much faster Express One Zone buckets are to access, let’s run a few benchmarks. To do this, we set up two buckets – an Express One Zone and another using Standard storage class – along with an EC2 instance in the same AZ as our One Zone bucket.

For the first test, we’ll allocate 100 files, each 10MB in size, for a total of 1GB of data. We’ll then run an aws s3 sync from our EC2 instance to copy the entirety of the bucket locally. We’ll loop this 10 times and simply measure how long it took for the copy to complete. Obviously this only tests read performance, but it will immediately give us a sense of how much faster Express One Zone can be:

Standard class: 8m5s
Express One Zone class: 2m41s
Speedup: 3.0x

This was a nice bump in performance, however, the new storage class really shines for smaller objects. Storage latency typically impacts small files much more than larger files, as storage overheads and latencies become more of a bottleneck than streaming the data over the network. For the second test, we’ll try 100,000 files, each 100K in size instead – still 1GB total, but comprised of many smaller files:

Standard class: 30m45s
Express class: 5m36s
Speedup: 5.5x

From 3x to more than 5x the performance! With more fine tuning of our sync parallelism and other factors, we could have eked out even more gains, but these simple tests quickly showed us how much faster Express One Zone can be for some very basic examples.

In summary, here are the results:

Conclusion

In this post, we learned all about Amazon S3 Express One Zone, how to manage new buckets and their contents with infrastructure as code, and saw the awesome performance gains in action. To be up and running with Amazon S3 Express One Zone and infrastructure as code in minutes, go ahead and get started with Pulumi for free today. Happy cloud spelunking!

Going Beyond With Advanced Infrastructure as Code Use Cases

Joe Duffy — Tue, 21 Nov 2023 03:53:33 +0000

This is the third of a three-part series originally published on The New Stack. Read Part 1 and Part 2.

Engineers who modernize their Infrastructure as Code with Pulumi get two classes of benefits:

Infrastructure as Code to develop cloud infrastructure with code.
Pulumi Cloud, which tames cloud infrastructure management at scale.

We’ve covered a fair bit of the first above, but have yet to scratch the surface for the second.

Many Infrastructure as Code solutions require that you explicitly manage something typically referred to as “state.” This state is an artifact produced that keeps track of what you think your infrastructure looks like, as defined by your Infrastructure as Code, so that it can be easily compared to what your actual infrastructure looks like. This is how diffs and updates can be done. Every Infrastructure as Code tool stores this infrastructure state, which is really just metadata about all the cloud resources, properties and dependencies. However, how much the tool exposes you to it varies.

Managed services like AWS CloudFormation and Azure Resource Manager (ARM) abstract this state away behind a service. Other tools like Terraform require you to explicitly manage that state by hand, often storing it in an S3 bucket or equivalent on your chosen cloud. The managed service approach tends to be easier to use and also avoids common pitfalls including security ones (like accidentally committing secrets to your Terraform state file in plain text) as well as reliability ones (like a code merge gone wrong that takes out your infrastructure). Pulumi Cloud offers both options, but the default is to use it like a managed service.

Pulumi Cloud is available for unlimited free usage, has a free tier for teams and offers advanced editions for larger teams and enterprises. Pulumi’s SDK is fully open source and can be used without the service, but the cloud service is the easiest, most secure and most reliable way to use Pulumi Cloud.

Pulumi Cloud’s architecture allows your cloud credentials to remain on your client. This when combined with features like identity integration and secrets management, addresses many pressing security concerns, and can mean that using the service is even more secure than managing Infrastructure as Code state on your own.

The service enables integration with your chosen identity provider — such as GitHub, GitLab, Atlassian, Okta, Azure ActiveDirectory or any SAML/SSO provider — and provides a complete visual history of who changed what infrastructure and when, very similar to what GitHub offers for code commits. It also makes it easier to see and control what’s going on in your team with organizations, role-based access controls and policy enforcement. It enables rich workflows via webhooks and gives users the ability to deploy either with the click of a button or straight out of a git repo.

You can get a better sense of what functionality is available in the Pulumi Cloud by reading this overview page. Although the Pulumi Cloud is trusted by many Global 2000 companies, is regularly audited by outside firms for well-architected and security best practices, and is SOC 2 Type II compliant, there is also a self-hosted edition available that can be run anywhere.

Testing Your Infrastructure as Code

A benefit of using general-purpose languages is that we get to test our code. Most of us wouldn’t think of writing application code without testing it, and Infrastructure as Code is no different. It’s not just that we can test it, but we can use standard tools and practices.

Testing can come in many different forms, including unit tests, property tests and integration tests. The following table summarizes the differences between the three approaches:

	Unit Tests	Property Tests	Integration Tests
Provision real infrastructure	No	Yes	Yes
Require the Pulumi CLI	No	Yes	Yes
Time to execute	Milliseconds	Seconds	Minutes
Language	Same as Pulumi program	Node.js or Python	Any language
Validation target	Resource inputs	Resource inputs and outputs	External endpoints

First is basic unit testing which, thanks to using regular programming languages, can be done in the usual way. For instance, the built-in Python and Go test frameworks “just work,” and as do popular third-party frameworks like Mocha (for JavaScript), NUnit (for .NET), JUnit (for Java), and so on. This can be good at ensuring a function that computes CIDR blocks, for instance, is correct. Testing infrastructure itself is a bit more involved, simply because many properties for resources aren’t known a priori until a deployment is done. But when combined with Pulumi Cloud’s infrastructure mocking framework, it’s possible to test infrastructure topologies and properties too. Read more about testing.

Next are assertions. The idea behind this is to embed assertions within your actual program, so that unexpected situations are caught immediately. This can be associated with any program state, including the resources themselves. Read more about assertions.

Next up is integration testing. For Infrastructure as Code, the typical meaning of an integration test is to spin up a dedicated stack solely for purposes of testing, often referred to as an “ephemeral stack.” Because Pulumi Cloud makes it easy to create independent, isolated copies of your infrastructure, thanks to the projects and stack model, these workflows are surprisingly easy to achieve. This can then be integrated into your typical testing processes, whether those are run by hand or automated as part of your CI/CD process. It’s common to spin up an ephemeral stack whose lifetime is tied to a single pull request, for instance, to validate that the changes being proposed are correct: It is created (and tested) when the pull request is opened, updated (and tested) anytime a new push to it is made and destroyed when it’s closed. Read more about integration testing and ephemeral stack techniques.

Finally, lots of more exotic tests are possible. For instance, you can fuzz-test your Infrastructure as Code by adding randomization to ensure the infrastructure behaves correctly in the face of strange inputs. You can chaos test your infrastructure by disabling instances or scaling key services below their typical thresholds to understand how your application behaves. You can stress test your infrastructure by placing unreasonable demands and seeing what breaks. The possibilities here are endless, but by using Infrastructure as Code in a general-purpose language, you have a lot more tools than is typically available to you.

To learn more about testing Infrastructure as Code more generally, with specific code examples, please see the Automated Testing of Pulumi Programs user guide.

Enforcing Security, Compliance, Cost and More with Policy as Code

An adjacent approach to both Infrastructure as Code and testing is something called policy as code. Just like Infrastructure as Code, the idea is to encode definitions in a real code artifact. The difference is that those are policies that analyze and flag problems with your infrastructure. The benefits of expressing these in code, however, are very similar to Infrastructure as Code: You can version policies as your requirements change over time, it’s reviewable by the team, and it’s just more expressive.

These policies can cover literally anything that can be analyzed and checked. This commonly includes security (such as ensuring databases aren’t open to the internet), compliance (that only FIPS endpoints are used for government-critical infrastructure), cost (that a deployment about to be done doesn’t exceed a team’s budget), and best practices (that all resources are tagged according to the team’s conventions). The idea is similar to linters and static analysis, both of which just work with Infrastructure as Code written in general-purpose languages, just purpose-built for infrastructure analysis.

Just as Pulumi Cloud lets you use general-purpose languages for your Infrastructure as Code, it lets you use general-purpose languages for policy as code too. This benefits from the richness of those languages and enables advanced use cases like integrating with cost management or ticketing systems like ServiceNow to enforce IT policies. Pulumi Cloud also supports the Open Policy Agent (OPA) language Rego to make it easier to leverage existing policies.

If we violate a policy, we get an error (for mandatory checks) or warning (for advisory ones) before the mistake even makes its way into production:

This might be “boring” from an engineer’s point of view but tends to be very important for an organization, especially for folks who are building out a platform team.

A benefit of using the Pulumi Cloud is that you can manage and enforce policies across the entire organization without individual engineers needing to even know how to run them. To learn more about policy as code, refer to the user guide for CrossGuard, Pulumi’s policy as code.

Deploying Your Infrastructure as Code Continuously

Most engineering teams ship their applications continuously thanks to CI/CD systems like GitHub Actions, Atlassian BitBucket, GitLab Pipelines, CircleCI, Travis, TeamCity, Spinnaker, or one of the built-in cloud CI/CD services like AWS CodePipeline, Azure DevOps or Google CloudBuild. This unlocks powerful git-driven workflows to push code out into production without needing to deploy things manually. But many teams don’t yet do that for infrastructure.

Adopting Infrastructure as Code helps you to do two things. First, you can verify and ship infrastructure changes the same way you ship application changes, by pushing and reviewing code, a practice we’ve already reviewed as being a core strength of Infrastructure as Code. Second, you can harmonize your application and infrastructure delivery pipelines, so they aren’t as disparate and disconnected. This helps to ship faster while scaling up to many more complex environments as your software sees more success.

The shift to containers, serverless and managed services has blurred the line between “application code” and “infrastructure code.” Is a serverless function on the infrastructure side of the line or application? What about a container registry? A pub/sub-topic? By using entirely different tools and delivery pipelines for applications versus infrastructure, which has historically been true of most infrastructure tools, we introduce a false dichotomy that slows down the team.

Pulumi Cloud offers two solutions for automating infrastructure delivery: First, any of the above CI/CD systems can be used to not just deliver application code, but also your Infrastructure as Code too. Second, Pulumi Cloud offers a built-in system called “Pulumi Deployments” to get up and running and trigger deployments based on git workflows. Read more in the Pulumi Deployments and/or Continuous Delivery user guides.

Advanced IaC Automation with Automation API

What if Infrastructure as Code were a normal library, in your language of choice, rather than a CLI? That was the question we asked before creating the automation API. Why is that an interesting question? It’s interesting because it elevates Infrastructure as Code to a core capability of any larger piece of software, which can be automated, rather than just a tool that needs to be run by hand by humans. We saw above that a small step toward this is to put Pulumi Cloud into your CI/CD, but the automation API takes it to a whole new level.

The scenarios this unlocks are mentioned earlier — custom tools and libraries, internal infrastructure provisioning portals, and even entire SaaS products — all of which need Infrastructure as Code as a central capability. Indeed, some of your favorite SaaS products, and even those that can be managed with Pulumi, use Pulumi on the backend to provision and manage infrastructure across clouds and across clusters.

Interestingly, we have found that, on average, those who leverage the automation API versus traditional Infrastructure as Code use cases are able to manage more than 10 times the infrastructure per engineer. That’s a big deal and the goal of automation: Doing more with less, experiencing less manual toil and just shipping faster with confidence.

Pulumi’s approach of using general-purpose languages made automation API a natural and unique extension of Infrastructure as Code into new realms. To learn more about Automation API, check out this page, which describes some common scenarios with complete examples.

Detecting IaC Drift and Doing Something about It

A problematic scenario with any Infrastructure as Code tool is something called “drift”: this happens when someone manually changes your infrastructure outside of the Infrastructure as Code tool. In that case, the actual infrastructure has “drifted” from what it was supposed to be, given your Infrastructure as Code definitions. Detecting drift is one thing, remediating it is another.

As a concrete example, imagine an engineer on the team is debugging an issue in production. To do so, they crack open port 22 so they can SSH into a Bastion host. (There are probably better approaches but as we know, when there’s a live incident, speed matters and sometimes shortcuts are taken.) Unfortunately, after resolving the incident, the engineer has forgotten to remove port 22 access. This is a potential security risk that we want to catch and resolve.

The two potential remediations to drift are typically:

Reapply the Infrastructure as Code, thereby eliminating the drift, and in this case, closing port 22 access
Incorporate the infrastructure changes into our program, so that the drifted state becomes our last-known-good on a go-forward basis. It’s also sometimes interesting to compare different environments to see if the infrastructure within them has drifted, although thanks to the Pulumi model of projects and stacks, this is uncommon in practice. It’s more of a concern with per-stack configuration, for which the Pulumi Cloud offers features to detect and resolve.

The Pulumi Cloud refresh command can be used to detect drift and then you can decide how to remediate it. All of the building blocks are there to either reapply changes, ingest them back into the program or some combination thereof. The combination of refresh, plus automation API, as well as the Pulumi Deployments capability, enables you to detect and remediate drift on demand or on a scheduled basis. Read more about drift detection and remediation or watch the video.

Learn More about Infrastructure as Code

The Pulumi Getting Started guide is the easiest way to get up and running with Infrastructure as Code. In addition to that, plus exploring the Pulumi Templates and Examples, we recommend some additional resources:

The tutorial series, Infrastructure as Code Recommended Practices, is a great overview of many concerns you will encounter along your Infrastructure as Code journey. This ranges from how to organize your project code and stacks, how to do local development and testing of your code, how to evolve these structures as your needs change, automation scenarios and more.

The Using Pulumi page is a collection of tutorials and getting started guides on specific infrastructure as code topics, such as deeper dives into fundamental concepts, how to simplify your infrastructure projects, and more.

We hope this article has given you a good grasp of the core Infrastructure as Code concepts you’re likely to encounter, no matter the tool you choose, as well as demonstrated some of the unique benefits Pulumi Cloud offers by using general-purpose and popular programming languages.

A Walkthrough of Adopting Infrastructure as Code

Joe Duffy — Tue, 14 Nov 2023 19:54:55 +0000

This is the second of a three-part series originally published on The New Stack.

Following the first piece in this series, Infrastructure as Code in Any Programming Language, this walkthrough will show what it takes to get up and running with Infrastructure as Code. Everything we show will be done with Pulumi’s free and open source Infrastructure as Code SDK. You can also sign up for Pulumi Cloud. After discussing the basics of how to get going, we’ll then dive into some advanced use cases to show what you can do from there.

Migrating to Infrastructure as Code

When you pick an Infrastructure as Code tool, there is a major decision up front: Is this going to be net new infrastructure, or are you adopting existing infrastructure? In the latter case, maybe you clicked to experiment in the AWS console but are now ready to use Infrastructure as Code. Or maybe you started with Terraform and now want to migrate to Pulumi. In either case, there’s no getting around it. This is a trickier situation.

In some cases, it’s better to just toss that infrastructure and start over from scratch. Or perhaps you can focus on succeeding first with a “new” part of the system and integrate with the existing infrastructure until you’re convinced it’s time to migrate. In many cases, however, that’s infeasible: Maybe you actually have a mission-critical service already that you need to bring under the management of Infrastructure as Code. The good news is that Pulumi offers tools to migrate from existing tools like Terraform, AWS CloudFormation/CDK, Azure Resource Manager (ARM), Kubernetes YAML and more, bringing existing infrastructure under the management of Pulumi without any disruption. The user guides in the Migration Hub walk through this process.

Getting Started with Infrastructure as Code

Let’s assume we are starting out with new infrastructure, even though most concepts are relevant even if you’re starting with a migration.

Cloud Infrastructure Resources: Architectures and Topologies

We usually start by provisioning a certain cloud architecture given our unique scenario. Maybe we’re creating a containerized application, a machine learning pipeline, or spinning up a Kubernetes cluster. In all of these cases, it begins by understanding the cloud resources that make up our architecture. In Pulumi, these are the programmable objects you’ll write code against.

The world of cloud infrastructure is a bit daunting. Pulumi supports over 100 clouds. AWS has over 200 services with over 1,000 individual resources and over 300,000 configurable properties across all of them. Pulumi is a multicloud tool, but “multicloud” does not mean “lowest common denominator.” Instead, Pulumi exposes all those individual clouds, resources and properties in their raw, unadulterated form. The benefit of this is that you have the entire capabilities of all of these clouds right at your fingertips. The downside is that to use them you need to understand these clouds and how to use them properly. As a result, you’ll probably quickly find that you want a starting point, rather than a blank page.

Pulumi Templates are a good way to get started. They represent over a dozen of the most common application and infrastructure architectures on the most popular clouds. They were built to be simple enough to be understandable at a glance but complete enough to be useful in practice. That includes containerized services, serverless applications, static websites, virtual machines and Kubernetes clusters across AWS, Azure, and Google Cloud, in addition to common Kubernetes workloads. Each template generates a fully working project that you can kick the tires with or even use as a starting point for your own customized architecture.

Pulumi Examples is an open source collection of over 300 fully working examples covering more clouds and more services. If you didn’t find a good starting point with the templates, there is a good chance you’ll find an approximation of your desired architecture here. If not, there are over 100,000 individual API examples in the Pulumi Registry, such as this one, which demonstrates provisioning a new Snowflake database with replication. Although these aren’t always as complete as examples and templates, they can also provide a great jumping-off point, in addition to serving as great reference material as you create additional code.

Let’s take a quick look at several examples that represent common application architectures.

Architecture 1: Containerized Service

In this example, we build and publish a Docker image running an NGINX web server to a private container registry, and then run that image in a load-balanced service. The load balancer ensures that we can easily scale up the number of replicas on the backend as our workload increases without fundamentally changing how clients can access it. There are many ways to accomplish this depending on the cloud we choose, and the detailed resources will differ.

If our chosen cloud is AWS, we will need an Elastic Container Service (ECS) cluster and registry, with a Fargate service and application load balancer in front of it. This enables our service to be accessed over the internet:

See the template for AWS here.

If we choose Google Cloud, on the other hand, we can either use Google Kubernetes Engine (GKE) or Google Cloud Run. In this example, we use Cloud Run and the built-in artifact repository, since these services are generally easier to use when just getting started:

In both cases, after deploying the infrastructure, we will have a fully functioning microservice environment with one load-balanced service running the NGINX web server. The Infrastructure as Code tool will give us a URL for the internet-accessible load balancer so we can browse to it.

See the template for GCP here.

Architecture 2: Serverless Application

In this example, we will create a serverless application. Unlike containers, which are fairly consistent across clouds (largely thanks to technologies like Docker and Kubernetes), the way serverless works varies greatly across clouds.

For Azure, we’ll need to create an Azure Blob Storage container to host our code, upload the serverless function bundle, and then configure an Azure Function app that will make our function available over HTTPS:

See the template for Azure here.

For AWS, although the architecture is in spirit very similar, the details differ greatly. We will use Amazon API Gateway and Lambda together to allow our function to be accessed over HTTPS:

In both cases, after deploying the infrastructure, we get back a web URL to access it. This example is a good reminder that each of the clouds is very different in its details. Infrastructure as Code doesn’t necessarily shield you from those details, although it is possible to build abstractions that do, which is both a strength but also something you’ll need to navigate.

See the template for AWS here.

Architecture 3: Kubernetes Cluster

For the final example, let’s switch gears and focus on more “foundational” infrastructure than the applications themselves. This is the sort of thing a platform engineer might care more about than, say, a backend systems engineer who is focused mostly on building and shipping services. In this specific case, we’ll spin up a managed Kubernetes cluster in AWS and Azure.

In the case of Azure’s Kubernetes Service (AKS), the design of the service is fairly chunky. So we can simply spin up a managed cluster with the associated networking infrastructure:

See the template for Azure here.

In the case of AWS’s Elastic Kubernetes Service (EKS), on the other hand, the service itself is rather complex to set up, involving many moving parts. That includes the control plane itself, the worker node groups, an autoscaling group, the AWS container network interface plugin to manage pod networking and more:

See the template for AWS here.

You can see here even greater differences between the equivalent cloud services on different cloud providers and AWS’s more “building blocks” style of approach in action. This is also a great example of Infrastructure as Code’s ability to bundle best practices so that they can be reused. This uses the Pulumi EKS Package, which configures all of the above in the standard way so that you don’t need to figure out every little detail when you’re just getting started.

In both cases, we end up with a fully managed Kubernetes cluster that is ready to use. The tool spits out a kubeconfig file so we can immediately deploy into or inspect it.

Infrastructure Projects and Stacks

All of these examples had one thing in common: The Infrastructure as Code program’s job is to declare individual resource objects, their properties and how they relate to each other. That was true no matter the scenario, the cloud or the language we have chosen. Once the program has done that, the Infrastructure as Code tool takes care of the rest.

Pulumi uses the term project to refer to the code that declares all of this, and each instance of said project that is configured and ready to deploy — each “environment,” if you will — is called a stack.

Each project can have many running stacks. That is useful for managing developer stacks (perhaps one per teammate) in addition to staging and production environments. It’s common to have many production environments when scaling a worldwide service. It’s also increasingly common to use short-lived stacks, for example, by creating a temporary stack for the duration of a pull request so pre-commit tests can be run against the hypothetical deployment.

The Pulumi CLI makes it easy to create both. The pulumi new command creates a new project:

$ pulumi new

It will prompt you for the project layout, name and other related settings. A project is just a name, program and associated metadata, and a parent for subsequent stacks.

The new command creates a starter stack, too, often called dev, for early-development purposes when getting going with a new project. However, additional ones can be created and managed using the pulumi stack command:

$ pulumi stack init # create a new stack for this project
$ pulumi stack ls # list stacks in this project
$ pulumi stack select # switch the current stack we're using
$ pulumi stack rm # delete a stack

Everything defined at the project level is shared among all stacks, including the code itself. This is nice because usually we want to keep all of the environments consistent with one another. But clearly sometimes we’ll want certain elements to intentionally diverge. Perhaps we want to deploy different stacks to different regions, use smaller or fewer VMs in our development stack than in production to keep cloud costs down and so on.

This is where Pulumi configuration enters the picture. Using configuration, you can easily set different values for specific settings in one stack versus another, and then the program can change its behavior accordingly. This allows you to keep most of your project’s declared infrastructure consistent across stacks by default but explicitly alter it when needed.

$ pulumi config set aws:region us-west-2

That’s not where configuration ends, however. Some of our configuration may entail using sensitive information, like Stripe access keys, database passwords or other kinds of secret tokens. Although Pulumi treats all state securely with in-transit and at-rest encryption, we never want settings like these to show up in plaintext anywhere. That’s where the Pulumi secrets system comes in handy by providing built-in encryption and secure storage of these settings.

$ pulumi config set dbPassword --secret [redacted]

Finally, Pulumi gives you tools to manage your projects and stacks according to your own requirements, similar to how git can be used for many different source control philosophies such as monorepos. For a description of the tradeoffs involved, see the “Organizing Projects and Stacks” user guide or the IaC Recommended Best Practices blog series. For complex systems, we will likely want a separation of concerns between pieces of our infrastructure, which means it is possible for one stack to depend on another.

For example, there could be a base networking and security layer, a data stack and a Kubernetes cluster that consume information from a lower-level layer but are provisioned separately, with applications above that depending on different assortments of all of the above.

Each stack is a unit of security and deployment concurrency. By layering stacks in this manner, we can let different teammates work in isolation from one another, ensure that the most secure and robust parts of our infrastructure are under lock and key, and just generally let different parts of the team go faster without affecting one another. The feature that enables this is called “Stack References,” and you can read more about how to use them in the documentation.

Doing a Deployment

After we’ve got everything wired up — a project, its code containing the declaration of a desired cloud resource topology, and at least one stack — it’s time to actually get something up and running. Since our code is written in an ordinary language, it’s tempting to think we would just run it outright. (Indeed, we will see later on that with the Automation API, we can do just that.) Infrastructure as Code is special, though; because of its declarative nature, we will run it using the Pulumi CLI, which ensures a declarative Infrastructure as Code workflow.

The pulumi up command is the key behind that workflow in action:

$ pulumi up

The pulumi up command evaluates our program, generates a desired state resource graph and gives us a preview of what will occur should we proceed with our deployment. (We can run the pulumi preview command explicitly if we want to run that as an independent step.) That preview will show any creates, updates or deletes, as well as interesting metadata about our deployment. The first time we ever deploy to a stack, there will be only creates, but after that we will see things like detailed diffs down to the property level for everything that has changed from one up to the next. These changes may arise due to code changes or even configuration updates.

Let’s take a look at all of this in action. First, we’ll create a new stack to run a load-balanced service in AWS (although it’s AWS, the same workflow applies no matter our choice of cloud):

$ pulumi new container-aws-typescript

Now let’s inspect the code that got generated:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const config = new pulumi.Config();
const containerPort = config.getNumber("containerPort") || 80;
const cpu = config.getNumber("cpu") || 512;
const memory = config.getNumber("memory") || 128;

// An ECS cluster to deploy into
const cluster = new aws.ecs.Cluster("cluster", {});

// An ALB to serve the container endpoint to the internet
const loadbalancer = new awsx.lb.ApplicationLoadBalancer("loadbalancer", {});

// An ECR repository to store our application's container image
const repo = new awsx.ecr.Repository("repo", {
 forceDelete: true,
});

// Build and publish our application's container image from ./app to the ECR repository
const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 path: "./app",
});

// Deploy an ECS Service on Fargate to host the application container
const service = new awsx.ecs.FargateService("service", {
 cluster: cluster.arn,
 assignPublicIp: true,
 taskDefinitionArgs: {
 container: {
 name: "app",
 image: image.imageUri,
 cpu: cpu,
 memory: memory,
 essential: true,
 portMappings: [{
 containerPort: containerPort,
 targetGroup: loadbalancer.defaultTargetGroup,
 }],
 },
 },
});

// The URL at which the container's HTTP endpoint will be available
export const url = pulumi.interpolate`http://${loadbalancer.loadBalancer.dnsName}`;

If we deploy this, we will see the full resource graph and the load-balanced service at the end:

That’s what the initial deployment looks like. Now let’s create a database for our application. We’ll first add these code changes:

const basic_dynamodb_table = new aws.dynamodb.Table("ddb-table", {
 attributes: [
 {
 name: "UserId",
 type: "S",
 },
 {
 name: "GameTitle",
 type: "S",
 },
 {
 name: "TopScore",
 type: "N",
 },
 ],
 billingMode: "PROVISIONED",
 globalSecondaryIndexes: [{
 hashKey: "GameTitle",
 name: "GameTitleIndex",
 nonKeyAttributes: ["UserId"],
 projectionType: "INCLUDE",
 rangeKey: "TopScore",
 readCapacity: 10,
 writeCapacity: 10,
 }],
 hashKey: "UserId",
 rangeKey: "GameTitle",
 readCapacity: 20,
 tags: {
 Environment: "production",
 Name: "dynamodb-table-1",
 },
 ttl: {
 attributeName: "TimeToExist",
 enabled: false,
 },
 writeCapacity: 20,
});

And then run pulumi up again. Notice that it shows us the diffs this time, down to individual properties that have changed in addition to what is new:

Note that, thanks to understanding of dependencies between resources, Infrastructure as Code tools will run many operations in parallel. You’ll see that timings are displayed as things are updated. Provisioning cloud infrastructure can be a slow process, but because of parallelism, Pulumi not only knows in what order to run various operations, it can do so as fast as possible.

After all is said and done, we can pulumi destroy our stack, which deletes all infrastructure within it:

$ pulumi destroy

This is clearly a dangerous but necessary operation. Pulumi has two features to help avoid mistakes. First, protecting a resource ensures extra steps are required to delete it, and second, the retain-on-delete option keeps the physical resource around even when the logical resource has been removed from your Pulumi stack. These are good to use with critical and precious resources whose destruction might be catastrophic, like databases — better safe than sorry!

In Part 3 of our series, the final installation, we will discuss advanced IaC use cases and how you can use Pulumi Cloud to develop cloud infrastructure with code and tame cloud infrastructure management at scale. We will be using Pulumi Cloud, which is available here.

Infrastructure as Code in Any Programming Language

Joe Duffy — Tue, 31 Oct 2023 22:36:29 +0000

This is the first of a three-part series originally published on The New Stack.

Infrastructure as Code is a technology for automating the infrastructure for your cloud applications. If you’re an engineer, whether that’s developing a backend service or within a central platform team, it’s not just about writing application code. You’ll need to provision, update and perform other tasks associated with its supporting infrastructure, and that’s where Infrastructure as Code can help. Instead of manually pointing-and-clicking in the cloud console, which is unrepeatable and error-prone, or writing ad-hoc scripts, which can be tedious and hard to scale, Infrastructure as Code lets us, as engineers, use familiar techniques by just writing code.

Not every engineer has a deep infrastructure background and yet needs to get more hands-on with infrastructure these days, which is OK: This three-part series was written from an engineer’s point of view. In it, we will demystify Infrastructure as Code — the why, what, and how — through the lens of Pulumi, a popular Infrastructure as Code tool among engineers.

Why We Need Infrastructure as Code

Modern applications need cloud infrastructure to run. That is equally true for simple monolithic applications running on virtual machines as it is for exotic distributed serverless applications that are fully elastic in scale. The applications themselves need infrastructure that they directly use, like whatever they run within (such as virtual machine, containerized service, serverless function, static website) in addition to any other resources they consume (databases, pub/sub topics, queues, AI/ML services, observability metrics and dashboards), but also depend on more primitive infrastructure to run atop (Kubernetes clusters, security roles and permissions, private networks, load balancers, encryption keys and more).

The phrase “cloud infrastructure” is also broader than it may seem. This phrase evokes immediate thoughts of popular clouds like Amazon Web Services (AWS) , Microsoft Azure, and Google Cloud, as well as more specialized or regional ones like Alibaba Cloud, DigitalOcean and Oracle Cloud. It also quickly leads to cloud native infrastructure like Docker, Kubernetes and Helm. But this phrase also perhaps not obviously applies to modern software-as-a-service (SaaS) infrastructure companies who increasingly are supplying critical pieces of cloud infrastructure, including Confluent, Cloudflare, Databricks, DataDog, Elastic, MongoDB, New Relic and Snowflake.

These companies are essentially specialized clouds providing more specialized services but are increasingly expanding to become clouds of their own. It also applies to private cloud technologies like F5, VMware vSphere and related technologies. Finally, also not obviously, there are SaaS tools that have configurable state that we use every day and may want repeatable management of just like our other cloud assets, including Auth0, GitHub, GitLab and PagerDuty.

Cloud infrastructure’s reach is far and wide! But it also means that there are many complex moving pieces to manage and tame. And with so much innovation happening in cloud capabilities, that complexity is just growing with time. This begs questions like the following:

Where does the infrastructure come from? How do we change it as our requirements evolve? How do we scale it as our needs grow, whether that’s increasing the compute and memory available to our workloads, scaling to many new instances, increasing our availability and reducing latency by deploying to new regions and environments worldwide… or, as is usually the case, a combination of all of these?

How do we ensure our infrastructure practices are repeatable in the event something fails or a mistake is made? How do we capture and reuse best practices? Do the answers to these questions differ across clouds? How do we ensure collaboration can take place safely and our deployments aren’t flaky and prone to colliding? And how do we secure all of it and ensure best practices and policies are enforced at all times?

These are all things that Infrastructure as Code solves, and it starts with code.

Benefits of Infrastructure as Code

The Infrastructure as Code approach provides many benefits, but they fall into two primary categories:

Using code to declare infrastructure
Using a declarative engine to orchestrate infrastructure changes

It is the combination of these two things that leads to the magic of “Infrastructure as Code.”

The Benefits of Code

Encoding your cloud application infrastructure in code results in a durable artifact representing your desired architecture. This can be code-reviewed, committed to source control and versioned in the usual ways. Infrastructure as Code tools not only know how to stand up the initial version of your infrastructure, but can replicate it across many environments (like dev, staging, prod and multiple regions), in addition to upgrading individual environments as your requirements evolve.

In terms of the “code” aspect, the expression of your code varies across the Infrastructure as Code landscape. Some tools support markup languages like JSON or YAML, while others support domain-specific languages (DSLs) that are specific to that tool. In this article, we will use Pulumi, which takes a unique approach to Infrastructure as Code that is well-suited for engineers: namely you use industry-standard general-purpose languages, including C#, Go, Java, JavaScript, Python or TypeScript, to express your code. This approach is great for engineers because it makes infrastructure more accessible and lets you use standard engineering tools and practices that you’re already using to build other software in your team.

Examples of these benefits include having rich constructs like simple if-statements and for-loops, which help avoid repetition and model complex infrastructure needs. Since all of these languages are broadly supported throughout the industry, virtually any editor you pick up will have great support, such as Visual Studio Code, PyCharm, Sublime Text, IntelliJ or even vim or emacs. That means you’ll get interactive statement completion, red squiggles if you make a typo or have a type-checking error, documentation as you hover, right-click to go to definition or refactor, and so much more. It’s easy to take these things for granted, but they are essential for software engineering productivity.

There are other benefits still such as linters, testing tools, the ability to share and reuse with package managers rather than copy-and-pasting and more. Lastly, each of these languages has enormous communities that add up to more than 20 million engineers, which means there’s a wealth of knowledge and support available.

It turns out Pulumi supports YAML too — the L in YAML stands for language, after all — which is a fine choice for simple scenarios, those where you want to machine-generate your Infrastructure as Code, or when engineers want to enable their sysadmins to do Infrastructure as Code too.

The Benefits of Declarative

Code is one major benefit. But in addition to the benefits of code, Infrastructure as Code has another significant advantage: It is “declarative,” even if you’ve chosen an imperative language like Go to express your code.

Infrastructure as Code tools generally work using a concept called “desired state.” The code, when run, produces a picture of the infrastructure your application requires. The Infrastructure as Code tool then understands how to compare the desired state with reality, and plan a course of action based on that information.

If it’s your first time creating a certain environment, something Pulumi calls a stack, then of course all the declared infrastructure will need to be created from scratch. Upon subsequent evaluations, however, that same infrastructure may need to be updated, deleted or even re-created, in addition to new infrastructure that may get spun up when it’s the first time it has been declared. This plan is presented ahead of performing any actions, so you and your team can review it, and if the course of action is wrong, you can correct it first.

One example of this process would be to first create a microservice environment that includes a Layer 4 network load balancer, a containerized cluster and a replicated, containerized service. You could subsequently add a private container registry, switch to a Layer 7 application load balancer, and scale up the service from one to three replicas.

This declarative approach ensures we can preview changes before they are made so we don’t have any unpleasant deployment surprises, gives us a full audit history of exactly what has changed in our actual infrastructure and when, similar to what source control does for our code artifacts, allows us to gate deployments on verification checks such as testing and policy enforcement, and makes it easier to integrate with various automation workflows.

It’s the codification of infrastructure and repeatability of the declarative approach that lets us use Infrastructure as Code in many kinds of automated workflows. That includes running a command-line interface (CLI) manually or as part of a script. Although “manual” may sound bad, the actual deployment is done with all of the above safeguards, so it’s fairly common for an Infrastructure as Code tool to be run this way.

However, most teams will adopt a CI/CD model for their most important environments, like production, which will trigger the actual deployment of code changes off a code commit. This ensures that all changes have been reviewed in the usual ways and go through a standard CI/CD pipeline. That pipeline may also include continuous verification (CV) such as running tests.

Some Infrastructure as Code tools support just one cloud, but our chosen tool, Pulumi, supports many, including all of those mentioned above, so all of these workflows can be standardized across all of the clouds and service providers. It can even track dependencies between cloud services — for example, it would not be strange to provision an Elastic Kubernetes Service cluster in AWS, install Datadog agents on its nodes, deploy some Kubernetes workloads, and place a Cloudflare content delivery network in front of that application, all using a single Infrastructure as Code program.

Pulumi’s unique approach unlocks an even more sophisticated workflow for running your code, using its so-called “Automation API.” This approach embeds Infrastructure as Code workflows right into larger pieces of software so that it can be programmed for highly dynamic scenarios.

This unlocks scenarios like building custom tools and libraries that build on top of and extend Infrastructure as Code, internal infrastructure provisioning portals and even entire SaaS products that need to provision or manage infrastructure as part of delivering their capabilities to their own end users.

In Part 2 of our series, we will take you through the steps needed to set up Infrastructure as Code. In doing so, we will be using Pulumi’s free and open source SDK, which is available here. It’s easy to get started, but you may want to take time now to explore the platform. You might also like to sign up for Pulumi Cloud, which can be done here.

Remediation Policies: Continuous and Automatic Compliance

Joe Duffy — Fri, 20 Oct 2023 00:00:00 +0000

Pulumi’s policy as code engine, CrossGuard, is already very flexible, and can enforce custom or predefined policies across a wide variety of use cases, including security, compliance, cost, and overall best practices. CrossGuard warns or issues errors should a deployment attempt to violate a policy. Last week we announced a new extension to CrossGuard called remediation policies. Remediation policies don’t just check for compliance, they go ahead and actually fix the problems in place. This ensures that every deployment across your entire team conforms, no questions asked, while also not needing to pester end users to remember all of the rules as they write their infrastructure as code, such as tagging resources a specific way. In this post, we will dig deeper into remediation policies and their use cases.

A Refresher on CrossGuard

CrossGuard policies are rules that run during pulumi preview and up to check that resource state conforms to some set of criteria. You can use off-the-shelf policies like AWSGuard and the new Pulumi Compliance-Ready Policies or write your own.

To roll your own, you create a “policy pack” in one of the supported languages (currently JavaScript, Python, TypeScript, or Open Policy Agent (OPA) Rego). A policy pack is a collection of policies which are just objects and functions that are given an opportunity to inspect the resource graph and report whether any policy violations have been detected. They’re packaged natively (such as NPM, PyPI, etc) and Pulumi runs the policies at the right time with the right data to enforce them.

For example, this code enforces that S3 buckets are tagged:

import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy";

new PolicyPack("aws-check-tags", {
 policies: [
 {
 enforcementLevel: "mandatory",
 name: "s3-tags",
 description: "Ensure required tags are present on S3 buckets.",
 validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => {
 if (!bucket.tags || !bucket.tags["Company"] || !bucket.tags["Department"]) {
 reportViolation("S3 Bucket is missing required Company/Department tags");
 }
 }),
 },
 ],
});

from pulumi_policy import (
 EnforcementLevel,
 PolicyPack,
 ResourceValidationPolicy,
)

def s3_check_required_tags(args, report_violation):
 if args.resource_type == "aws:s3/bucket:Bucket":
 if ("tags" not in args.props or
 "Company" not in args.props["tags"] or
 "Department" not in args.props["tags"]):
 report_violation("S3 Bucket is missing required Company/Department tags")

PolicyPack(
 name="aws-check-tags",
 enforcement_level=EnforcementLevel.MANDATORY,
 policies=[
 ResourceValidationPolicy(
 name="s3-tags",
 description="Ensure required tags are present on S3 buckets.",
 validate=s3_check_required_tags,
 ),
 ],
)

In the event of failure, we will see something like the following:

Policies:
 ❌ aws-check-tags@v0.0.1
 - [mandatory] s3-tags (aws:s3/bucket:Bucket: my-bucket)
 Ensure required tags are present on S3 buckets.
 S3 Bucket is missing required Company/Department tags

In this case, the policy is checking a single resource at a time — but stack-wide policies are supported too, where an entire stack’s worth of resources can be checked at once.

There are two ways to tell the Pulumi engine to run a policy pack:

Pass --policy-pack=... when running a pulumi preview or up; this just uses a policy pack locally from wherever the Pulumi CLI is being run.
Publish a policy to your organization using pulumi policy publish and then enable it using pulumi policy enable or in the Pulumi Cloud UI. After this is done, policies automatically run, without any opt-in necessary by the end user (and indeed, opt-out is not even possible). This is much better because you don’t need to worry about distributing policy packs a priori and users can’t forget to enable the right policies.

The first option is available to all state backends while the second requires Pulumi Cloud. The first is a fine option for experimenting, developing your own policy packs, or just getting started.

Finally, there are two levels of enforcement: mandatory violations issue errors that stop the deployment in its tracks; advisory violations issue warnings and are a good way to let a user know something lower priority has been detected and/or to gradually roll out policy enforcement. The same policy can be configured to be either advisory or mandatory when it is enabled depending on the scenario (or even none if you want to skip a policy but run others from the same policy pack).

Introducing Remediation Policies

Policies can now remediate violations to actually fix the problems they find in addition to or instead of reporting them as violations. Just like normal policies, a remediation is given a resource’s state for inspection. If the policy finds a problem, however, it can fix that state in place, and return it, and the engine will use the remediated state in place of the original state produced by the Pulumi program.

For example, this remediates, rather than validates, using the S3 tagging policy logic shown above:

import * as aws from "@pulumi/aws";
import { PolicyPack, remediateResourceOfType } from "@pulumi/policy";

new PolicyPack("aws-check-tags", {
 policies: [
 {
 enforcementLevel: "remediate",
 name: "s3-tags",
 description: "Ensure required tags are present on S3 buckets.",
 remediateResource: remediateResourceOfType(aws.s3.Bucket, (bucket, args) => {
 if (!bucket.tags || !bucket.tags["Company"] || !bucket.tags["Department"]) {
 bucket.tags = bucket.tags || {};
 bucket.tags["Company"] = bucket.tags["Company"] || "ACMECorp";
 bucket.tags["Department"] = bucket.tags["Department"] || "IT0001";
 return bucket;
 }
 }),
 },
 ],
});

from pulumi_policy import (
 EnforcementLevel,
 PolicyPack,
 ResourceValidationPolicy,
)

def s3_check_required_tags(args):
 if args.resource_type == "aws:s3/bucket:Bucket":
 had_violation = False
 if "tags" not in args.props:
 args.props["tags"] = dict()
 had_violation = True
 if "Company" not in args.props["tags"]:
 args.props["tags"]["Company"] = "ACMECorp"
 had_violation = True
 if "Department" not in args.props["tags"]:
 args.props["tags"]["Department"] = "IT0001"
 had_violation = True
 if had_violation:
 return args.props

PolicyPack(
 name="aws-check-tags-py",
 enforcement_level=EnforcementLevel.REMEDIATE,
 policies=[
 ResourceValidationPolicy(
 name="s3-tags",
 description="Ensure required tags are present on S3 buckets.",
 remediate=s3_check_required_tags,
 ),
 ],
)

To enable remediations to take place, the policy is configured with a new enforcement level, remediate, which is stricter than mandatory.

When we run pulumi up, rather than reporting a violation, we will see that it’s automatically remediated:

Policies:
 ✅ aws-check-tags@v0.0.1
 - [remediate] s3-tags (1 resource)

To see the full details, we can select details from the preview menu:

 Policy Remediations:
 aws-check-tags@v0.0.1 [remediate] s3-tags (aws:s3/bucket:Bucket: my-bucket)
 + tags : {
 + Company : "ACMECorp"
 + Department: "IT0001"
 }

This specific policy is written in a way that only works in remediation mode. That means if a different enforcement level, like advisory or mandatory is enabled, the policy will do nothing. It is possible to write a policy that can work in any enforcement level:

import * as aws from "@pulumi/aws";
import { PolicyPack, validateRemediateResourceOfType } from "@pulumi/policy";

new PolicyPack("aws-check-tags", {
 policies: [
 {
 enforcementLevel: /*any enforcement level possible*/,
 name: "s3-tags",
 description: "Ensure required tags are present on S3 buckets.",
 ...validateRemediateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => {
 if (!bucket.tags || !bucket.tags["Company"] || !bucket.tags["Department"]) {
 reportViolation("S3 Bucket is missing required Company/Department tags");
 bucket.tags = bucket.tags || {};
 bucket.tags["Company"] = bucket.tags["Company"] || "ACMECorp";
 bucket.tags["Department"] = bucket.tags["Department"] || "IT0001";
 return bucket;
 }
 }),
 },
 ],
});

from pulumi_policy import (
 EnforcementLevel,
 PolicyPack,
 ResourceValidationPolicy,
)

def s3_check_required_tags(args, report_violation):
 if args.resource_type == "aws:s3/bucket:Bucket":
 had_violation = False
 if "tags" not in args.props:
 args.props["tags"] = dict()
 had_violation = True
 if "Company" not in args.props["tags"]:
 args.props["tags"]["Company"] = "ACMECorp"
 had_violation = True
 if "Department" not in args.props["tags"]:
 args.props["tags"]["Department"] = "IT0001"
 had_violation = True
 if had_violation:
 report_violation("S3 Bucket is missing required Company/Department tags")
 return args.props

PolicyPack(
 name="aws-check-tags-py",
 enforcement_level=# Any enforcement level possible,
 policies=[
 ResourceValidationPolicy(
 name="s3-tags",
 description="Ensure required tags are present on S3 buckets.",
 validate_remediate=s3_check_required_tags,
 ),
 ],
)

Notice in the code that we are both reporting and remediating policy issues at the same time. This might seem not to make sense at first — if we’re remediating the policy error, doesn’t that by definition mean we don’t need to report the violation? Indeed, that is a choice you can make. However, these “dual mode” policies also ensure the same policy can be run with any enforcement level — advisory, mandatory, or remediate — as appropriate.

For instance, in some settings we may prefer to issue an error to the developer so that they fix their code. In other settings, we may prefer that this same remediation happens automatically. The policy remediations feature is very flexible in that it supports a range of policy enforcement.

Configurable Remediations

In the above example, we hard-coded the set of tags we want to check for. For a simple policy for a very targeted use case, that may be ok. However, if we are writing a more general purpose policy, or one that needs to run across many organizations, this hard-coding can be unfortunate.

Policy configuration, on the other hand, enables us to write rules that are parameterized. If we used policy configuration for this example, we could configure the set of tags to check for independent of the policy itself. This would mean that we could write a single tagging policy, once, and then enable/configure it many times. This is useful for enforcement in different organizations and also within the same organization if, say, if our production tags were different than development.

Here’s a rewritten variant of that policy above which uses policy configuration:

import * as aws from "@pulumi/aws";
import { PolicyPack, validateRemediateResourceOfType } from "@pulumi/policy";

new PolicyPack("aws-auto-tagger", {
 policies: [
 {
 enforcementLevel: "remediate",
 name: "s3-tags",
 description: "Ensure required tags are present on S3 buckets.",
 configSchema: {
 properties: {
 requiredTags: {
 type: "object",
 items: { type: "string" },
 },
 },
 },
 ...validateRemediateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => {
 const config = args.getConfig<{ requiredTags?: Record<string, string> }>();
 if (config.requiredTags) {
 let hadViolations = false;
 if (!bucket.tags) {
 bucket.tags = {};
 }
 for (const key of Object.keys(config.requiredTags)) {
 if (!bucket.tags[key]) {
 reportViolation(`S3 Bucket is missing required tag '${key}'`);
 bucket.tags[key] = config.requiredTags[key];
 hadViolations = true;
 }
 }
 if (hadViolations) {
 return bucket;
 }
 }
 }),
 },
 ],
});

from pulumi_policy import (
 EnforcementLevel,
 PolicyConfigSchema,
 PolicyPack,
 ResourceValidationPolicy,
)

def s3_check_required_tags(args, report_violation):
 if args.resource_type == "aws:s3/bucket:Bucket":
 had_violations = False
 config = args.get_config()
 if config and "requiredTags" in config:
 if "tags" not in args.props or not args.props["tags"]:
 args.props["tags"] = dict()
 for tag, value in config["requiredTags"].items():
 if tag not in args.props["tags"]:
 report_violation(f"S3 bucket is missing required tag '{tag}'.")
 args.props["tags"][tag] = value
 had_violations = True
 if had_violations:
 return args.props

PolicyPack(
 name="aws-check-tags-py",
 enforcement_level=EnforcementLevel.REMEDIATE,
 policies=[
 ResourceValidationPolicy(
 name="s3-tags",
 description="Ensure required tags are present on S3 buckets.",
 config_schema=PolicyConfigSchema(
 properties={
 "requiredTags": {
 "type": "object",
 "items": { "type": "string" },
 },
 },
 ),
 validate_remediate=s3_check_required_tags,
 ),
 ],
)

Now when we enable the policy, we can supply a configuration file. Here is a policy.json file:

{
 "s3-tags": {
 "enforcementLevel": "remediate",
 "requiredTags": {
 "Company": "ACMECorp",
 "Department": "IT0001"
 }
 }
}

This can be done at the command line in one of two ways:

If using --policy-pack, we can also pass --policy-config-file=policy.json.
If using organization policies, supply it when enabling your pack, with pulumi policy enable --config=policy.json.

It is also possible to edit or alter the configuration in the Pulumi Cloud console UI:

This Pulumi Cloud console experience makes it very easy to set default policies that apply to all stacks in your organization, hyper-target individual stacks with specific policies, or even group stacks together and enforce policies homogeneously across them, e.g. “all SOC2 stacks.”

Example Use Cases for Remediations

Remediation policies are great for many use cases. Here are a few examples:

Tagging resources with standard, organization-wide tags.
Disabling Internet access for gateways and firewall rules.
Enabling encryption on storage or buckets.
Down-sizing virtual machine configuration to use less expensive machine types.

Clearly this is not an exhaustive list, but it should serve to jog your imagination.

There are certainly use cases where you may elect not to automatically remediate violations. For example, outright misconfigurations are likely better reported to the end user, so that the infrastructure as code author is aware and can fix the source code definitions. It may not always be advisable to silently correct mistakes. And yet, there are clear use cases where it is likely preferable; for instance, rather than pestering every end user in your organization to remember precisely what set of tags and their values you want on all resources, they can just be added automatically. This enhances compliance while also enhancing productivity — a win win!

Sources of Inspiration

Before winding down, it’s worth calling out a few sources of inspiration for this feature. When we initially built policy as code, we were inspired by static analysis more generally which we use all the time with programming languages. These techniques allow us to find and fix errors closer to the source, as we are writing code, rather than after the fact when it’s already too late.

That inspiration included technologies we built during the Trustworthy Computing era at Microsoft, such as PREfix, PREfast, FxCop, SAL, and advanced programming language type systems including Spec#. At the time, security vulnerabilities were running rampant, and the more code can be “correct by construction,” the fewer security mistakes will slip through the cracks and result in costly security flaws. The analogies with the current state of the art in the cloud are abundant.

The Roslyn project took this a step further. As the C# compiler was rewritten in C# itself, the entire compiler object model was suddenly accessible to C# code itself. This facilitated a much easier way to write static analyzers which were paired with code fixes. Code fixes are very similar to policy remediations in that they don’t just report problems, they actually fix them up!

Both Azure and Kubernetes have ways to automatically remediate policy problems also. Azure Policy offers a similar remediations feature, although it is somewhat difficult to extend since it is all expressed in JSON and limited to Azure Resource Manager resources. Kubernetes offers a mutating admission webhook controller that can rewrite Kubernetes manifests on the fly before they are ever actually applied to the desired state that the controller sees.

In some ways, remediation policies are reminiscent of Aspect Oriented Programming (AOP), a popular technique in the Java community in the 2000s. Generally speaking, AOP was seen as too “magical” for widespread use. However, for targeted compliance scenarios like those discussed earlier, this approach strikes the right balance. Furthermore, we took great steps to ensure you can always find out precisely when and why a remediation changed things.

Finally, Pulumi has always had transformations to perform in-memory modifications of resources as they get registered, either one at a time or across the whole stack. These run within a process and can hook resource registrations and mutate their state. The reason remediation policies are so different and interesting, however, is that they can be applied to programs without the program itself even having to opt-in. This is precisely what enables cross-organization enforcement.

Winding Down

Remediation policies take Pulumi’s very powerful policy engine, CrossGuard, to the next level. Using them you can not just detect incompliant resources but actually fix them automatically.

Everything described in this post, except for organizational enforcement, is available in the free Pulumi open source SDK. To get started, just run pulumi policy new in a directory to begin creating your new policy pack. Also, check out the CrossGuard get started guide, which walks through the entire process of creating a new policy pack from scratch and running it. You can read more details about policies and remediations in the documentation.

Check it out and let us know what you think!

A recap of October 2023 - A big month at Pulumi!

Joe Duffy — Mon, 16 Oct 2023 00:00:00 +0000

We’ve been hard at work and are having the biggest month ever at Pulumi. That includes two new products, Pulumi ESC and Pulumi for Platform Teams, and news of our Series C. That’s a boatload of fresh infrastructure as code innovation. And October isn’t even over yet!

Two New Products!

Here’s a recap of the news:

Pulumi ESC is a new approach to cloud environments, secrets, and configuration. It works for applications and infrastructure and cuts down on sprawl while making everything more secure. This tech spans developer desktop scenarios and scales up beyond 1,000s of environments. Read the overview, blog, or get started with the docs. Oh, and don’t forget to star the GitHub repo!

Pulumi for Platform Teams takes what we’ve learned working with over 2,000 customers and 150,000 end users, many building internal developer platforms, and provides common patterns and practices right out of the box. This includes a developer portal, a Backstage plugin, advanced deployment workflows, and the ability to report on and automatically remediate compliance policies. Read the overview, blog, or get started with the docs.

Can we help you build an internal cloud platform? Although the new capabilities are easy to use, we’re happy to give you a demo, discuss architecture, or even help you build out your internal platform. Get in touch.

These launches build on our “multi-product” momentum from our Pulumi Insights product launch earlier this year. Insights applies the power of generative AI to all things cloud giving you an intelligent interface to your infrastructure. We are taking infrastructure as code to many new places. We’ve processed over 130,000 prompts since launch — it’s free to use, so give it a try today!

Series C Fundraise

Last but not least, we announced our Series C. This fundraise will enable us to keep building an incredible team and reimagining how developers and infrastructure teams program the cloud. We’re hiring and have endless fun hard/problems ahead check out and apply to the open positions here.

Let Us Know What You Think

We can’t wait to hear what you think. Get Started with Pulumi, join our Community Slack, and let us know how you’re using these new capabilities and any feedback you have.

From the entire Pulumi team, thank you!

Building the Best Infrastructure as Code with $41M Series C Funding

Joe Duffy — Tue, 03 Oct 2023 00:00:00 +0000

Today we announced a $41M Series C fundraise from Madrona Ventures, NEA, Tola Capital, and Strike Capital. These new funds will help us accelerate momentum, keep innovating with the best infrastructure as code technology on the market, and expand into new product areas to solve even more of our customers’ most pressing cloud challenges. We’ve had a year of exciting milestones – surpassing 150,000 end users in our community, 2,000 customers, and 100 employees – and look forward to many more to come. The future is full of cloud, and yet incredibly bright!

Incredible momentum

Over 2,000 customers have chosen Pulumi Cloud for their infrastructure as code needs. Many more companies build on our open source, including over half of the Fortune 50. And we estimate that our community recently surpassed 150,000 happy and productive end users.

This year we made the top 5% of the Inc. 5000 Fastest Growing Companies in America and were listed in Redpoint’s InfraRed 100, to highlight transformative cloud infrastructure companies. This momentum will continue to propel us even further on our growth trajectory.

The best infrastructure as code

Why have we seen such incredible growth? It’s a combination of things: an amazing, multi-billion-dollar fast-growing market – infrastructure as code – plus a product that is widely recognized as the best product and clearly leading the innovation agenda in that market.

Pulumi’s unique approach to infrastructure as code is to stand on the shoulders of giants by letting you use any language. At the time we began, this was not a popular idea! In fact, a good friend of mine, who is a legend in the cloud native space, bluntly told me, “YAML is good enough – infrastructure as code doesn’t need that good of a developer experience.” We disagreed.

Our thesis was that we went through a one way door with cloud software. We have entered into the era of real distributed applications. And that is super exciting! It means all developers are building cloud software and that infrastructure is central to innovating in the modern cloud.

By applying the industry’s best languages to the infrastructure as code space, we gave the cloud a programmable surface area. We can now treat cloud building blocks as programmable objects and build bigger things out of smaller things. The cloud is the new operating system and this foundational technology lays the groundwork for us to realize our long-term vision.

Fast forward 7 years, and Pulumi is the fastest growing infrastructure as code product and has inspired others, fundamentally reshaping the landscape. Our end users and customers tell us all the time they feel not just 10% more productive, but 10x more productive, thanks to rich, expressive languages, great tools and editors, sharing and reuse, and because infrastructure as code in any language also means infrastructure as code for anybody in the whole team.

We also bet big on open source. Open source is in our entire team’s DNA. We honestly felt it was table stakes that something so fundamental to how we write code and build platforms be open source. This builds trust, helps to foster an ecosystem and community of openness, and enables new scenarios that we couldn’t have even imagined. We paired our open source technology with a great Pulumi Cloud SaaS that 2/3rds of our end users elect to use simply because it’s the easiest, most reliable, and most secure way to adopt Pulumi at scale. Because we were very thoughtful, intentional, and smart about the relationship between our open source and commercial products, we can stand confidently by our open source heritage forever.

Building a platform

Over time, we have expanded beyond infrastructure as code, by solving one problem after another that we see our customers facing. Let there be no question, we are still very much all about infrastructure as code! But as we have worked with our over 2,000 customers, we recognize they have other challenges in areas adjacent to infrastructure as code.

This began with Policy as Code. Policy as code allows you to verify that infrastructure state follows cost, compliance, security, reliability, or best practice guidelines. I ran static analysis efforts at Microsoft for many years during the Trustworthy Computing era and saw just how important and powerful enforcing policy as close to development time as possible is. We are still very early days here but we built Pulumi’s engine to support policy as code right from the outset.

Next came many obvious adjacencies to the infrastructure as code workflow, including Testing and CI/CD Automation. Thanks to our choice of general purpose languages, we decided to tap into software engineering practices too; we sometimes call this “infrastructure as software” because it goes well beyond “just code.” We then built the Automation API to answer the question of, what if infrastructure as code were a library you could link with, and not just a command line interface? The things our customers have built with these capabilities have been mind blowing: self-serve portals, custom CLIs and frameworks, and even entire SaaS products.

Last year, we then began branching out into entirely new product lines:

Pulumi Deployments delivers server-side orchestration of deployment workflows. We frequently heard from customers that no product on the market solved this for them. You can point Pulumi Cloud at a Git repo and get automatic Pull Request validations and deployments upon merges. Or do one-click deployments straight from the Pulumi Cloud console. Pulumi Deployments is highly extensible with a REST API unlocks a world of possibilities for platform teams, including but not limited to drift detection and remediation, blue/green and multi-region rollouts, dependent stack updates, ephemeral stacks, and automatic infrastructure cleanup.
Pulumi Insights provides search, analytics, and intelligence over your infrastructure, across any clouds. Our customers frequently told us they had trouble taming, or even getting visibility into, cloud chaos. The headline feature here is our Pulumi AI coding assistant: if you can dream up an infrastructure as code configuration, it can build it for you in any language. Pulumi AI quickly surpassed 100K prompts within its first couple of months. Insights also enables you to search over any infrastructure anywhere, helping you gain understanding over your team’s use of cloud infrastructure, analyze it further, or even just to find that needle in the haystack.

Pulumi is gradually shifting from a single infrastructure as code product to a complete enterprise platform. As we build out the platform, we do so in a highly integrated way. Pulumi Cloud pulls it all together with a unified data model, REST API, identity model, and AI model. These are not a loosely related assortment of technologies – they are deeply integrated, and can increasingly be adopted independently, but are better together even more than the sum of the parts.

What’s next for Pulumi

It is still day one for us and we’re only just getting started. Our ambitious mission is to democratize the cloud so that every builder, no matter their background, can make the most out of the cloud. And although infrastructure as code in any language makes a giant step in that direction, there is so much more to be done. If the cloud is the new operating system, there are entire application and security models still missing. For our customers, the cloud is a competitive advantage and they’re always wanting to do more, and to do it faster, with the cloud.

As we look to the future, there are several key themes we are excited to tackle:

The best infrastructure as code. We hear all the time from users and customers that they think Pulumi is the best infrastructure as code technology on the market. We want to keep it that way. Best means the most productive, the most fun, the most flexible, and the most enterprise grade. We will continue investing substantially into making this technology even better and more accessible while also growing our community.
Reimagined cloud developer experiences. The modern cloud’s capabilities are endless–and yet that can be daunting. For engineers simply wanting to build a modern cloud application or architecture, the concept count and level of abstraction demanded by the cloud are still often an order of magnitude more complex than they should be. We will continue leveraging our infrastructure as code foundations to deliver revolutionary new developer experiences that make us more productive and impactful in the cloud.
Security and best practices by-construction. Simply put, security should not be an afterthought. No best practice should be. Yet for most cloud platform teams, both still are. We have chipped away at this problem but there is more to be done. Pulumi’s approach enables us to apply tried and true techniques from the programming languages domain to ensure that applications and infrastructure are increasingly correct by-construction. We need to shift from finding things after-the-fact towards stopping security issues dead in their tracks before they ever make their way to production.
AI powers the next wave of cloud innovation. We are big believers in AI. We have already extended large models to deeply understand the cloud infrastructure domain and have realized immense benefits of doing so. We have begun to add AI to our product in areas that improve the user experience or deliver unique capabilities. As we go forward, just as we always relentlessly pursue the best developer experiences for our end users, we expect to ask ourselves, “How can this be better in an AI-first world?”

In short, the new Series C funds will allow us to double down on innovation that is already working, while breaking new ground in pursuit of our mission. Most importantly, we look forward to working with you, our end users and community, to build the best product experiences that solve your hardest and most pressing problems.

Thank you

It is a privilege to count over 2,000 customers, and 150,000 end users, as part of our community. I still remember coding furiously in my basement with Eric and Luke, and going live with our first customer, like it was yesterday. We are just as passionate about our original dream today as we were back then – perhaps even more so, since it is coming true!

We are also so lucky to have such incredible investors at Madrona, NEA, Tola, and Strike who believe in the Pulumi dream too, and are helping us to build an iconic company.

To our end users, we would not be here if it weren’t for you. Whether you are a community member, an enterprise customer, or anywhere in between, your continued feedback, success stories, and belief in the Pulumi vision keep us going and make it all worthwhile.

From the entire Pulumi team, thank you!

Pulumi adoption made easy with the new Migration Hub

Joe Duffy — Tue, 22 Aug 2023 00:00:00 +0000

Today we are launching Pulumi’s new Migration Hub, a comprehensive guide to help you seamlessly adopt Pulumi no matter where you are coming from, whether that’s Terraform, CloudFormation, … or even manually provisioned resources not yet governed by an infrastructure as code solution. Our new Expert Services group is ready to roll up their sleeves to help you adopt Pulumi faster. The Migration Hub also features many commercial offers for open source foundations, startups, and complementary migration, to minimize switching costs and risks. It’s never been easier to adopt Pulumi.

Why Pulumi?

Nearly 2,000 customers and over 130,000 practitioners have chosen Pulumi for their infrastructure as code needs. About half the time, these end users have an existing infrastructure as code solution that has stopped meeting their needs. The other half of the time, infrastructure may have been created in the cloud console (“click-ops”), using a cloud CLI, or some other manual provisioning tool, but now needs to become repeatable and scalable using infrastructure as code.

The most common reasons teams prefer Pulumi include:

Best-in-class productivity
Accessible to the entire organization, thanks to industry standard languages
Ability to deal with complexity at scale, thanks to stacks, config, sharing and reuse, and more
Perfect confidence thanks to policies and testing
Secure by-default, with built-in support for secrets
Highly programmable engine can be embedded anywhere, to build custom platforms and tools
Instant collaborative bridge between developers and infrastructure experts

Thanks to Pulumi, teams are able to do more with less, ship faster with confidence, tackle very challenging cloud infrastructure projects and scale with them, empower more of the team to be self-serve and collaborate more closely, and ultimately turn the cloud into a competitive advantage.

Pulumi is also true Apache 2.0 open source and we welcome anyone who wishes to build atop it.

Migration

Of course, it’s one thing to choose Pulumi, but another thing entirely to adopt it and get all of your infrastructure under its management, especially if you have existing infrastructure. That might include converting existing code, or even just importing existing manually provisioned cloud resources.

There are two major approaches to this challenge.

Self-Serve Migration

Pulumi offers many tools and techniques should you want to perform a migration on your own.

In the most general form, Pulumi provides an import command which can import any existing infrastructure no matter how it was originally created, click-ops included.

Pulumi also offers conversion tools for other popular infrastructure as code solutions, including HashiCorp Terraform, AWS CloudFormation, Azure Resource Manager (ARM), or Kubernetes YAML.

Although the time to migrate existing infrastructure varies wildly based on the size and complexity of your scenario, migrating a single cloud environment can often take as little as a few hours. And thanks to Pulumi being multi-language at its core, these tools will support you the same no matter which language you choose, JavaScript, TypeScript, Python, Go, C#, Java, and YAML alike.

To learn more about performing your own migration, please read this user guide.

Let Us Help You Out with Expert Services!

Although self-serve is an option, there can often still be uncertainty around this approach.

It may not be clear how much time a full migration will take, leaving the possibility that you’ll be left with multiple infrastructure as code solutions – either temporarily or even permanently. Although the tools are a huge help, they don’t always get you 100% of the way there. We also often hear that skilling yourself up on a one-time migration activity takes away from critical time spent on business initiatives.

All of this adds friction, risk, and uncertainty to the Pulumi adoption decision. The last thing we want is for the one-time migration to prevent you from adopting Pulumi. Thankfully, we are here to help!

Today we are announcing the formation of Pulumi’s Expert Services group. This team of industry and infrastructure as code experts is on standby and ready to roll up their sleeves to help you with migration, training, and other services efforts. No project is too small: we are currently working with customers with 100KLOCs of legacy infrastructure as code that needs to be migrated, for example.

Today we are also announcing an extension to our Terraform Migration Offer, in which we will bundle migration of your Terraform projects with your Pulumi Enterprise or Business Critical subscription. Today we are excited to announce that this same package is now available for all migration efforts, including AWS CloudFormation, Azure Resource Manager (ARM), …, or even infrastructure that was manually provisioned. We are also prepared to scale up from there depending on your needs.

Success Stories

Over time, Pulumi is becoming a more comprehensive platform, with new services like Pulumi Deployments for multi-cloud deployment workflows and Pulumi Insights for search, analytics, and AI for your cloud infrastructure. Pulumi also offers policies and secrets as core features of the platform. Enterprises are increasingly betting on Pulumi for their entire cloud management and platform needs.

By moving to Pulumi’s platform, many customers have seen considerable benefits:

Atlassian significantly increased developer velocity, increasing productivity by more than 2x and substantially decreasing the amount of time spent on maintenance tasks.

Mercedes-Benz got the whole team shipping faster and with more confidence by building a platform that helped developers be self-serve with infrastructure while scaling up to 100s of Kubernetes clusters worldwide.

Snowflake rapidly improved time to market by betting on Pulumi, empowering their engineers, and delivering the Data Cloud. Snowflake ultimately beat their IPO deadlines and knew legacy infrastructure as code wouldn’t do it.

Washington Trust Bank modernized their cloud infrastructure and practices, speeding up internal delivery, while also increasing their confidence thanks to policy as code guardrails.

Fauna adopted a born in the cloud mindset and was able to build a multi-cloud, highly scalable, modern SaaS application, actually increasing reliability even in the face of growing complexity.

In each of these cases, the customer evaluated Pulumi against alternative infrastructure as code solutions, but ultimately chose Pulumi due to the aforementioned benefits.

Special Offers

In addition to migration services, Pulumi offers two programs to help get up and running quickly:

Open Source: Any open source foundation or project can use Pulumi Team Edition free of charge.
Startup Discount: Any early stage startup gets $10,000 free credits and access to Pulumi Enterprise.

Calling All Partners!

As Pulumi’s market adoption has skyrocketed, we’ve begun to build out our official partner program. Although we don’t have anything major to announce (yet), we’d love to talk to you if infrastructure as code is an important part of your own customers’ needs. That includes ISV product integrations, global and regional systems integrators and consultants, and more. Note that Pulumi is already available in the AWS and Azure marketplaces. If you are interested in discussing a partnership, please contact us.

Get Started Today

Migrating from an existing solution can seem like a daunting task, but with Pulumi’s new Migration Hub and Expert Services, it has never been easier to adopt Pulumi than it is today. To learn more, visit the Migration Hub or simply contact us – we would love to hear from you.

Pulumi 💜's Open Source

Joe Duffy — Fri, 11 Aug 2023 00:00:00 +0000

Pulumi stands with the open source community.

We are unaffected by HashiCorp relicensing their software yesterday, and express profound support for many of our cloud friends who have been affected.

Pulumi is true open source, uses the Apache 2.0 license, and does not and never will depend on BSL-licensed software in any way, HashiCorp owned or otherwise.

We look forward to continuing to serve our new and loyal customers, always with open source and our amazing, fast-growing community at our core.

Deploy Infrastructure to Multiple Cloud Regions at Once

Joe Duffy — Wed, 03 Aug 2022 08:00:00 -0700

Pulumi makes it easy to flexibly deploy your cloud infrastructure using code. Usually deployments encompass a single slack and a single region in your cloud of choice. If you need to go multi-region, that usually means creating a stack per-region, which Pulumi’s configuration system makes easy. A stack per region isn’t required, though! Sometimes we want a single stack to span regions for performance, scalability, resilience, or just hard requirements. In these cases, Pulumi can seamlessly orchestrate deployments to, or even across, multiple regions, accounts, or clusters. In this article, we’ll see this in action by provisioning an AWS RDS primary database into one region and a read replica in an entirely different region – all from a single Pulumi program, stack, and pulumi up incantation.

Recap of Stack Configuration

Before seeing how to span regions, let’s review how you’d typically configure your region. Every Pulumi project can have any number of stacks. These stacks are often things like dev, test, prod – and sometimes, things like prod-us-east, prod-us-west, prod-euro, and so on, to represent instances of our infrastructure running in different regions. Each stack is isolated from all others and it’s easy to switch between them.

The standard way to vary regions between stacks is to use Pulumi’s configuration system. If you’ve used Pulumi to deploy to AWS, for instance, you’ve probably uttered a CLI command like this:

$ pulumi config set aws:region us-east-1

This isn’t just about regions: it applies to anything that varies between stacks. It might include accounts, secrets like tokens and passwords, sizes or counts of instances, among many other things.

This is normally very convenient and aligns with how we want to configure our projects and stacks. But there are some relatively common circumstances in which it breaks down:

Managing a large number of multi-tenanted resources spanning multiple accounts or clusters.
Cases where a few pieces of infrastructure must run somewhere different than most of it, as is often the case with DNS or SSL/TLS certificate management services.
Situations where we want to provision resources in many places for performance, scalability, or resilience purposes.
Highly programmatic scenarios where some configuration might be fetched dynamically from an external resource.

In all of these cases, thankfully Pulumi has an answer: explicit provider configuration.

How Explicit Provider Configuration Works

Before diving into our example, let’s take a quick look at how explicit provider configuration works. Unlike ordinary configuration, which is set at the CLI, provider configuration is set programmatically inside your Pulumi program. Every Pulumi package exports a Provider resource class which can be instantiated like any other resource, with the sole difference that these resources exist to pass around to configure other resources, and don’t represent infrastructure that is to be provisioned in a cloud of any kind.

For instance, if we wanted to programmatically create an AWS provider that targets a different region than us-east-1, as shown earlier, we can do so as follows:

const awsWest2 = new aws.Provider("aws-west-2", { region: "us-west-2" });

aws_west_2 = aws.Provider('aws-west-2', region='us-west-2')

awsWest2, err := aws.NewProvider(ctx, "aws-west-2", &aws.ProviderArgs{
 Region: pulumi.String("aws-west-2")
})
if err != nil {
 return err
}

var awsWest2 = new Aws.Provider("aws-west-2", new Aws.ProviderArgs
{
 Region = "us-west-2"
});

resources:
 awsWest2:
 type: pulumi:providers:aw
 properties:
 region: us-west-2

This configuration is not used automatically, however. You must pass it to any resources you’d like to use it:

const buck = new aws.s3.Bucket("my-bucket", {
 // ...
}, { provider: awsWest2 });

buck = aws.s3.Bucket('my-bucket',
 # ...
 opts=pulumi.ResourceOptions(provider=aws_west_2)
)

buck, err := s3.NewBucket("my-bucket",
 &s3.BucketArgs{
 // ...
 },
 pulumi.Provider(awsWest2)
)
if err != nil {
 return err
}

var buck = new Aws.S3.Bucket("my-bucket",
 new Aws.S3.BucketArgs
 {
 // ...
 },
 new ResourceArgs
 {
 Provider = awsWest2
 }
);

resources:
 # ...
 buck:
 type: aws:s3:Bucket
 properties:
 # ...
 options:
 provider: ${awsWest2}

This overrides the default of using whatever was set at the CLI, and will instead use the programmatically configured settings. Note also that anything you can set at the CLI is available, not just the region. You can see the full settings available in the package’s registry documentation for its Provider class (for instance, the AWS Provider.

Multi-Region Deployment In Action!

The architecture for our multi-region setup is going to be relatively straightforward so we can focus on the essentials. We will provision a single RDS primary database in one region (us-east-1) and backup to a read replica in an entirely different region – in fact, on an entirely different continent (eu-west-2)! This is relatively straightforward because many scenarios demand that we deploy to a dynamic number of regions, perhaps across dozens or hundreds of resources. By keeping it to just two resources across two regions, we can see the fundamentals in action which can scale easily to more sophisticated use cases.

This is described in AWS’s docs and is depicted in this diagram:

Now let’s dive into some infrastructure as code. There’s a bit of preamble before getting to the meat of the example:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as random from "@pulumi/random";

// Code from below will goes here.

import pulumi
import pulumi_aws as aws
import pulumi_random as random

# Code from below goes here.

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws"
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/rds"
 "github.com/pulumi/pulumi-random/sdk/v4/go/random"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Code from below goes here.
 })
}

using System.Collections.Generic;
using Pulumi;
using Aws = Pulumi.Aws;
using Random = Pulumi.Random;

return await Deployment.RunAsync(() =>
{
 // Code from below goes here.
});

name: aws-multireg-rds
runtime: yaml
description: A multi-region example creating an RDS primary and read-replica in different regions

# Code from below goes here.

Next, we’ll make the retention period configurable, by default it is 30 days. Note how we can still use standard configuration features as well as explicit providers alongside one another:

// Enable backup retention to be configured, but default to 30 days.
const config = new pulumi.Config();
const retentionPeriod = config.getNumber("retentionPeriod") || 30;

# Enable backup retention to be configured, but default to 30 days.
config = pulumi.Config()
retention_period = config.get_float('retentionPeriod') or 30

// Enable backup retention to be configured, but default to 30 days.
cfg := config.New(ctx, "")
retentionPeriod := 30
if param := cfg.GetFloat("retentionPeriod"); param != 0 {
 retentionPeriod = param
}

// Enable backup retention to be configured, but default to 30 days.
var config = new Config();
var retentionPeriod = config.GetNumber("retentionPeriod") ?? 30;

# Enable backup retention to be configured, but default to 30 days.
configuration:
 retentionPeriod:
 type: Number
 default: 30

Next, we’ll create a random password for our database so we don’t need to hard-code one:

// Create a random password for the database's admin user.
const dbPassword = new random.RandomPassword("dbPassword", { length: 12 });

# Create a random password for the database's admin user.
db_password = random.RandomPassword('dbPassword', length = 12)

// Create a random password for the database's admin user.
dbPassword, err := random.NewRandomPassword(ctx, "dbPassword", &random.RandomPasswordArgs{
 Length: pulumi.Int(12),
})

// Create a random password for the database's admin user.
var dbPassword = new Random.RandomPassword("dbPassword", new()
{
 Length = 12,
});

 # Create a random password for the database's admin user.
 dbPassword:
 type: random:RandomPassword
 properties:
 length: 12

Now we’re ready to create the providers. We’ll create one per region that we will deploy into:

// Create two AWS providers, one for each region we'll deploy into.
const euProvider = new aws.Provider("euProvider", { region: "eu-west-2" });
const usProvider = new aws.Provider("usProvider", { region: "us-east-1" });

# Create two AWS providers, one for each region we'll deploy into.
eu_provider = aws.Provider('euProvider', region = 'eu-west-2')
us_provider = aws.Provider('usProvider', region = 'us-east-1')

// Create two AWS providers, one for each region we'll deploy into.
euProvider, err := aws.NewProvider(ctx, "euProvider", &aws.ProviderArgs{
 Region: pulumi.String("eu-west-2"),
})
if err != nil {
 return err
}
usProvider, err := aws.NewProvider(ctx, "usProvider", &aws.ProviderArgs{
 Region: pulumi.String("us-east-1"),
})
if err != nil {
 return err
}

// Create two AWS providers, one for each region we'll deploy into.
var euProvider = new Aws.Provider("euProvider", new()
{
 Region = "eu-west-2",
});

var usProvider = new Aws.Provider("usProvider", new()
{
 Region = "us-east-1",
});

resources:
 # Create two AWS providers, one for each region we'll deploy into.
 euProvider:
 type: pulumi:providers:aws
 properties:
 region: eu-west-2
 usProvider:
 type: pulumi:providers:aws
 properties:
 region: us-east-1

To keep things straightforward, we’ve hard-coded the regions into the program. There are more sophisticated possibilities, however. For example, we could have made the regions themselves configurable too, much like the retention period. Even more powerfully, we could use our programming language’s expressiveness to create them dynamically. For instance, we could have looped over a list of them so that the number of regions isn’t even known in advance. These advanced capabilities are helpful for many real-world scenarios.

Next up, we’ll create the primary database instance in us-east-1:

// Create the primary RDS database instance.
const primaryDb = new aws.rds.Instance("primary", {
 allocatedStorage: 10,
 engine: "mysql",
 engineVersion: "5.7",
 instanceClass: "db.t3.micro",
 username: "admin",
 password: dbPassword.result,
 backupRetentionPeriod: retentionPeriod,
 skipFinalSnapshot: true,
}, {
 provider: usProvider,
});

# Create the primary RDS database instance.
primary_db = aws.rds.Instance('primary',
 allocated_storage=10,
 engine='mysql',
 engine_version='5.7',
 instance_class='db.t3.micro',
 username='admin',
 password=db_password.result,
 backup_retention_period=retention_period,
 skip_final_snapshot=True,
 opts=pulumi.ResourceOptions(provider=us_provider)
)

// Create the primary RDS database instance.
primaryDb, err := rds.NewInstance(ctx, "primary", &rds.InstanceArgs{
 AllocatedStorage: pulumi.Int(10),
 Engine: pulumi.String("mysql"),
 EngineVersion: pulumi.String("5.7"),
 InstanceClass: pulumi.String("db.t3.micro"),
 Username: pulumi.String("admin"),
 Password: dbPassword.Result,
 BackupRetentionPeriod: pulumi.Float64(retentionPeriod),
 SkipFinalSnapshot: pulumi.Bool(true),
}, pulumi.Provider(usProvider))
if err != nil {
 return err
}

// Create the primary RDS database instance.
var primaryDb = new Aws.Rds.Instance("primary", new()
{
 AllocatedStorage = 10,
 Engine = "mysql",
 EngineVersion = "5.7",
 InstanceClass = "db.t3.micro",
 Username = "admin",
 Password = dbPassword.Result,
 BackupRetentionPeriod = retentionPeriod,
 SkipFinalSnapshot = true,
}, new CustomResourceOptions
{
 Provider = usProvider,
});

 # Create the primary RDS database instance.
 primary:
 type: aws:rds:Instance
 properties:
 allocatedStorage: 10
 engine: mysql
 engineVersion: "5.7"
 instanceClass: db.t3.micro
 username: admin
 password: ${dbPassword.result}
 backupRetentionPeriod: ${retentionPeriod}
 skipFinalSnapshot: true
 options:
 provider: ${usProvider}

The key here is that we explicitly passed the provider that was configured to use us-east-1, instructing Pulumi to use that configuration rather than the implicit default configuration set at the CLI as usual.

Now we will create the secondary database instance in eu-west-2:

// Now set up our secondary read replica database in an entirely different region.
const secondaryDb = new aws.rds.Instance("secondary", {
 instanceClass: "db.t3.micro",
 replicateSourceDb: primaryDb.arn,
 backupRetentionPeriod: retentionPeriod,
 skipFinalSnapshot: true,
}, {
 provider: euProvider,
});

# Now set up our secondary read replica database in an entirely different region.
secondary_db = aws.rds.Instance("secondary",
 instance_class="db.t3.micro",
 replicate_source_db=primary_db.arn,
 backup_retention_period=retention_period,
 skip_final_snapshot=True,
 opts=pulumi.ResourceOptions(provider=eu_provider)
)

// Now set up our secondary read replica database in an entirely different region.
secondaryDb, err := rds.NewInstance(ctx, "secondary", &rds.InstanceArgs{
 InstanceClass: pulumi.String("db.t3.micro"),
 ReplicateSourceDb: primaryDb.Arn,
 BackupRetentionPeriod: pulumi.Float64(retentionPeriod),
 SkipFinalSnapshot: pulumi.Bool(true),
}, pulumi.Provider(euProvider))
if err != nil {
 return err
}

// Now set up our secondary read replica database in an entirely different region.
var secondaryDb = new Aws.Rds.Instance("secondary", new()
{
 InstanceClass = "db.t3.micro",
 ReplicateSourceDb = primaryDb.Arn,
 BackupRetentionPeriod = retentionPeriod,
 SkipFinalSnapshot = true,
}, new CustomResourceOptions
{
 Provider = euProvider,
});

 # Now set up our secondary read replica database in an entirely different region.
 secondary:
 type: aws:rds:Instance
 properties:
 instanceClass: db.t3.micro
 replicateSourceDb: ${primary.arn}
 backupRetentionPeriod: ${retentionPeriod}
 skipFinalSnapshot: true
 options:
 provider: ${euProvider}

As expected, we passed the provider configured to use eu-west-2. Notice also that, although these resources are deployed into entirely different regions, they can reference one another! Thanks to Pulumi’s dependency system, it understands that referencing primary.arn means the secondary database depends on the primary, which Pulumi will use to ensure correct ordering of operations.

Finally, we’ll emit the primary and secondary database endpoints plus auto-generated password:

// Output the primary and secondary database ARNs, and the auto-generated admin password.
export const primary = primaryDb.endpoint;
export const secondary = secondaryDb.endpoint;
export const password = dbPassword.result;

# Output the primary and secondary database ARNs, and the auto-generated admin password.
pulumi.export('primary', primary_db.endpoint)
pulumi.export('secondary', secondary_db.endpoint)
pulumi.export('password', db_password.result)

// Output the primary and secondary database ARNs, and the auto-generated admin password.
ctx.Export("primary", primaryDb.Endpoint)
ctx.Export("secondary", secondaryDb.Endpoint)
ctx.Export("password", dbPassword.Result)
return nil

// Output the primary and secondary database ARNs, and the auto-generated admin password.
return new Dictionary<string, object?>
{
 ["primary"] = primaryDb.Endpoint,
 ["secondary"] = secondaryDb.Endpoint,
 ["password"] = dbPassword.Result,
};

# Output the primary and secondary database ARNs, and the auto-generated admin password.
outputs:
 primary: ${primary.endpoint}
 secondary: ${secondary.endpoint}
 password: ${dbPassword.result}

Using these, we can easily access our new instance. Don’t worry, the password is automatically encrypted!

For good measure, since all of our AWS resources are meant to use the explicit providers, and not the ambient default configuration, we can disable default providers altogether:

$ pulumi config set --path pulumi:disable-default-providers[0] "aws"

If we forget to pass an explicit provider, we now get an error:

error: Default provider for 'aws' disabled. 'urn:pulumi:dev::aws-multireg-rds::aws:rds/instance:Instance::primary' must use an explicit provider.

Now that we’ve got everything set up, we run pulumi up which orchestrates everything end-to-end:

$ pulumi up
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack aws-multireg-rds-dev created
 + ├─ pulumi:providers:aws euProvider created
 + ├─ pulumi:providers:aws usProvider created
 + ├─ random:index:RandomPassword dbPassword created
 + ├─ aws:rds:Instance primary created
 + └─ aws:rds:Instance secondary created

Outputs:
 password : [secret]
 primary : "primarybe35b53.c0q7bvujknic.us-east-1.rds.amazonaws.com:3306"
 secondary: "secondaryacb6a3d.c8qninoaapw8.eu-west-2.rds.amazonaws.com:3306"

Resources:
 + 6 created

Duration: 27m0s

Deploy across regions from one stack

Configure explicit providers in your Pulumi program to provision resources across regions, accounts, and clusters from a single stack. Build and ship your multi-region infrastructure with one pulumi up.

Learn more

Winding Down

In this article, you saw some reasons why deploying your infrastructure across multiple regions, accounts, or clusters may be necessary. You saw that Pulumi supports two forms of configuration – implicit configuration set at the CLI with pulumi config as well as explicit provider configuration which is constructed and passed programmatically – and how and when to use them, often in tandem with one another.

The specific example shown to demonstrate multi-region in action deployed an AWS RDS primary database to a US region and a secondary read replica to Europe, which is a common architecture to improve resilience and performance. The power of full programming languages gives you a ton of flexibility in how you orchestrate such complex architectures.

Hopefully this post put a new tool in your infrastructure as code toolbelt! Happy cloud spelunking.

Deploy WordPress to AWS using Pulumi and Ansible

Joe Duffy — Mon, 27 Jun 2022 08:00:00 -0700

There are two primary kinds of infrastructure as code tools: configuration management, like Ansible, Chef, and Puppet, which configure, patch, or upgrade existing servers, and provisioning, like Pulumi, Terraform, and CloudFormation, which create, update, and delete the underlying infrastructure itself. Provisioning has taken over from configuration management as the dominant form of IaC over the past decade as we’ve shifted to modern immutable infrastructure architectures that use containers, serverless, and managed services. And yet, configuration management still remains important and relevant, especially for heritage, stateful, server-centric, and on-prem or hybrid solutions. The good news is that it doesn’t need to be either-or choice: the two approaches are complementary. In this post, you’ll see how and why you might combine them by deploying a WordPress Server to AWS by provisioning infrastructure with Pulumi and configuring the server with Ansible.

Preamble: On Provisioning and Configuration

Before diving in, why might you want to provisioning and configuration IaC tools together?

Provisioning is more common these days because modern workloads using containers, managed services, and serverless architectures are amenable to immutable infrastructure approaches — replacing instead of patching. This entails fewer state transitions and failure modes to consider, making automation less error prone, more repeatable, and more scalable. This streamlines deploying infrastructure from CI/CD or other advanced automation, scaling up to 10s, 100s, or even 1,000s of environments repeatably.

This distinction is sometimes referred to as the difference between “cattle” and “pets”. In this analogy, cattle are given numbers like vm0042.cern.ch, are almost identical to all other cattle, and when they get ill, you get another one. In contrast, pets are given names like myspecialpet.cern.ch, are unique and lovingly hand-raised and cared for, and when they get ill, you nurse them back to health.

To illustrate what these differences mean from an infrastructure as code perspective, let’s say you have a server running an old version of WordPress and want to upgrade to the latest:

With configuration management, it is a pet, so you would patch the server, upgrading it from version X to the new version Y. This is challenging because it requires knowing all possible X=>Y transitions and potential failure modes. Imagine the upgrade from X=>Y fails — how do you recover? Is it a manual recovery? And imagine version Z comes out: is upgrading X=>Z the same upgrade path as Y=>Z? Or is each permutation different? It becomes very difficult to predict and carefully consider all combinations of upgrade paths and failures.
With provisioning, it is cattle, so you don’t worry about such things. You’d place your server, possibly many of them, behind a stable IP address and/or load balancer and, instead of patching, you would provision one or more new servers running version Y (or Z), redirect the IP address and/or load balancer to the new server(s), and then destroy the old server(s) running the old version X after traffic has been redirected. This reduces the space of state transitions to consider and eliminates a whole host of failure recovery scenarios. If something fails, you just fix the problem, and rinse and repeat; provisioning tools are better at picking up where they left off, even from partial states. In fact, this workflow is precisely the way most container-based systems, including Kubernetes, work, generalizing common patterns.

Many heritage workloads still use configuration management tools for mutable infrastructure like virtual machine (VM) servers running in on-prem or hybrid environments. Provisioning can work too, however, many VM-centric tools and practices formed over the years around mutable infrastructure. In some cases with stateful workloads it may legitimitely be preferable to continue using VMs and configuration management. And even if you’re not choosing it for new workloads, configuration management is definitely here to stay — many organizations have mission-critical workloads and procedures built up around this family of tools.

For this post, imagine your organization decided to stick with Ansible configuration management for managing a stateful WordPress server, but is otherwise modernizing by using Pulumi IaC to provision AWS infrastructure.

Pulumi and Ansible Better Together - The Approach

Our goal is to set up WordPress in AWS in an automated and repeatable manner. It should be easy to scale a single environment based on demand, upgrade the infrastructure as requirements evolve, and repeatably set up and tear down entirely new environments. These are all areas where IaC shines.

Here’s a quick sketch of the approach:

Create a Pulumi program in your language of choice (Python, YAML, any Node.js language (such as JavaScript, TypeScript), Go, any .NET language (such as C#), or any JVM language (such as Java, Groovy, Scala, or Kotlin). This program defines the declarative infrastructure specification.
Declare the AWS infrastructure topology using Pulumi’s native programming model. This infrastructure includes a Virtual Private Cloud (VPC), public and private subnets, security groups, a MySQL RDS database that WordPress will store data in, and an EC2 instance that’s accessible over the Internet to run the WordPress server, among a few other supporting bits of infrastructure.

Here is a visual diagram of this AWS architecture:
Declare configuration commands, still all within Pulumi’s programming model, to perform two tasks:
1. Update packages over SSH, using yum to ensure prerequisites like Python are present.
2. Run the Ansible Playbook on the machine running Pulumi as the Ansible control node. This installs and configures WordPress on the new infrastructure.
Run a single command, pulumi up, to orchestrate everything, including provisioning or updating the AWS infrastructure and running commands and Ansible playbooks to perform the on-VM configuration.
Finally, the WordPress server’s IP address is printed, and our server is ready to go!

Now that you’ve got an idea of the overall game plan, let’s get started — first, you’ll create a Pulumi project to house your IaC.

Creating Your Infrastructure as Code Project

To start, you’ll create a Pulumi IaC project, which will declare all of the supporting AWS infrastructure into which you’ll then install WordPress.

Note: If you’re following along, be sure to install the prerequisites:

Install Pulumi

Install Ansible

Configure AWS Credentials

First, create a new Pulumi project in your language of choice:

$ pulumi new aws-typescript

$ pulumi new aws-python

$ pulumi new aws-go

$ pulumi new aws-csharp

$ pulumi new aws-yaml

Next, to make our infrastructure setup easy to vary between environments, the code leverages Pulumi’s “configuration” system — not to be confused with configuration management tools — which just allows you to vary provisioning settings for each stack in case your environments have different needs. For instance, perhaps you have different SSH keys, or want larger instance sizes, in development versus production:

const config = new pulumi.Config();
// A path to the EC2 keypair's public key:
const publicKeyPath = config.require("publicKeyPath");
// A path to the EC2 keypair's private key:
const privateKeyPath = config.require("privateKeyPath");
// The WordPress database size:
const dbInstanceSize = config.get("dbInstanceSize") || "db.t3.small";
// The WordPress database name:
const dbName = config.get("dbName") || "wordpressdb";
// The WordPress database user's name:
const dbUsername = config.get("dbUsername") || "admin";
// The WordPress database user's password:
const dbPassword = config.requireSecret("dbPassword");
// The WordPress EC2 instance's size:
const ec2InstanceSize = config.get("ec2InstanceSize") || "t3.small";

config = pulumi.Config()
# A path to the EC2 keypair's public key:
public_key_path = config.require("publicKeyPath")
# A path to the EC2 keypair's private key:
private_key_path = config.require("privateKeyPath")
# The WordPress database size:
db_instance_size = config.get("dbInstanceSize") or "db.t3.small"
# The WordPress database name:
db_name = config.get("dbName") or "wordpressdb"
# The WordPress database user's name:
db_username = config.get("dbUsername") or "admin"
# The WordPress database user's password:
db_password = config.require_secret("dbPassword")
# The WordPress EC2 instance's size:
ec2_instance_size = config.get("ec2InstanceSize") or "t3.small"

cfg := config.New(ctx, "")
// A path to the EC2 keypair's public key:
publicKeyPath := cfg.Require("publicKeyPath")
// A path to the EC2 keypair's private key:
privateKeyPath := cfg.Require("privateKeyPath")
// The WordPress database size:
dbInstanceSize := cfg.Get("dbInstanceSize")
if dbInstanceSize == "" {
 dbInstanceSize = "db.t3.small"
}
// The WordPress database name:
dbName := cfg.Get("dbName")
if dbName == "" {
 dbName = "wordpressdb"
}
// The WordPress database user's name:
dbUsername := cfg.Get("dbUsername")
if dbUsername == "" {
 dbUsername = "admin"
}
// The WordPress database user's password:
dbPassword := cfg.RequireSecret("dbPassword")
// The WordPress EC2 instance's size:
ec2InstanceSize := cfg.Get("ec2InstanceSize")
if ec2InstanceSize == "" {
 ec2InstanceSize = "t3.small"
}

var config = new Config();
// A path to the EC2 keypair's public key:
var publicKeyPath = config.Require("publicKeyPath");
// A path to the EC2 keypair's private key:
var privateKeyPath = config.Require("privateKeyPath");
// The WordPress database size:
var dbInstanceSize = config.Get("dbInstanceSize") ?? "db.t3.small";
// The WordPress database name:
var dbName = config.Get("dbName") ?? "wordpressdb";
// The WordPress database user's name:
var dbUsername = config.Get("dbUsername") ?? "admin";
// The WordPress database user's password:
var dbPassword = config.RequireSecret("dbPassword");
// The WordPress EC2 instance's size:
var ec2InstanceSize = config.Get("ec2InstanceSize") ?? "t3.small";

configuration:
 # A path to the EC2 keypair's public key:
 publicKeyPath:
 type: String
 required: true
 # A path to the EC2 keypair's private key:
 privateKeyPath:
 type: String
 required: true
 # The WordPress database size:
 dbInstanceSize:
 type: String
 default: db.t3.small
 # The WordPress database name:
 dbName:
 type: String
 default: wordpressdb
 # The WordPress database user's name:
 dbUsername:
 type: String
 default: admin
 # The WordPress database user's password:
 dbPassword:
 type: String
 secret: true
 required: true
 # The WordPress EC2 instance's size:
 ec2InstanceSize:
 type: String
 default: t3.small

After this, you’ll declare a few variables that are fetched dynamically, so you don’t need to hard-code them, and so your stack can be flexibly deployed into different AWS regions. This includes automatically getting the availability zones in the target AWS region as well as the AMI for Amazon Linux 2 which also varies by region:

// Dynamically fetch AZs so we can spread across them.
const availabilityZones = aws.getAvailabilityZones();
// Dynamically query for the Amazon Linux 2 AMI in this region.
const awsLinuxAmi = aws.ec2.getAmi({
 owners: ["amazon"],
 filters: [{
 name: "name",
 values: ["amzn2-ami-hvm-*-x86_64-ebs"],
 }],
 mostRecent: true,
});

# Dynamically fetch AZs so we can spread across them.
availability_zones = aws.get_availability_zones()
# Dynamically query for the Amazon Linux 2 AMI in this region.
aws_linux_ami = aws.ec2.get_ami(owners=["amazon"],
 filters=[aws.ec2.GetAmiFilterArgs(
 name="name",
 values=["amzn2-ami-hvm-*-x86_64-ebs"],
 )],
 most_recent=True)

// Dynamically fetch AZs so we can spread across them.
availabilityZones, err := aws.GetAvailabilityZones(ctx, nil)
if err != nil {
 return err
}
// Dynamically query for the Amazon Linux 2 AMI in this region.
awsLinuxAmi, err := ec2.LookupAmi(ctx, &ec2.LookupAmiArgs{
 Owners: []string{
 "amazon",
 },
 Filters: []ec2.GetAmiFilter{
 ec2.GetAmiFilter{
 Name: "name",
 Values: []string{
 "amzn2-ami-hvm-*-x86_64-ebs",
 },
 },
 },
 MostRecent: pulumi.BoolRef(true),
}, nil)
if err != nil {
 return err
}

// Dynamically fetch AZs so we can spread across them.
var availabilityZones = Aws.GetAvailabilityZones.Invoke();
// Dynamically query for the Amazon Linux 2 AMI in this region.
var awsLinuxAmi = Aws.Ec2.GetAmi.Invoke(new Aws.Ec2.GetAmiInvokeArgs
{
 Owners = { "amazon" },
 Filters =
 {
 new Aws.Ec2.Inputs.GetAmiFilterInputArgs
 {
 Name = "name",
 Values = { "amzn2-ami-hvm-*-x86_64-ebs" },
 },
 },
 MostRecent = true,
});

variables:
 # Dynamically fetch AZs so we can spread across them.
 availabilityZones:
 Fn::Invoke:
 Function: aws:getAvailabilityZones
 # Dynamically query for the Amazon Linux 2 AMI in this region.
 awsLinuxAmi:
 Fn::Invoke:
 Function: aws:ec2:getAmi
 Arguments:
 owners: ["amazon"]
 filters:
 - name: name
 values: ["amzn2-ami-hvm-*-x86_64-ebs"]
 mostRecent: true

Finally, you’ll read in the contents of the public and private key files so they’re ready to use in the infrastructure declarations that are coming soon. Notice that this code uses Pulumi’s built-in secrets feature to ensure the private key remains encrypted at all times:

// Read in the public key for easy use below.
const publicKey = fs.readFileSync(publicKeyPath).toString();
// Read in the private key for easy use below (and to ensure it's marked a secret!)
const privateKey = pulumi.secret(fs.readFileSync(privateKeyPath).toString());

# Read in the public key for easy use below.
public_key = open(public_key_path).read()
# Read in the private key for easy use below (and to ensure it's marked a secret!)
private_key = pulumi.Output.secret(open(private_key_path).read())

// Read in the public key for easy use below.
publicKeyBytes, err := ioutil.ReadFile(publicKeyPath)
if err != nil {
 return err
}
publicKey := pulumi.String(string(publicKeyBytes))
// Read in the private key for easy use below (and to ensure it's marked a secret!)
privateKeyBytes, err := ioutil.ReadFile(privateKeyPath)
if err != nil {
 return err
}
privateKey := pulumi.ToSecret(string(privateKeyBytes))

// Read in the public key for easy use below.
var publicKey = File.ReadAllText(publicKeyPath);
// Read in the private key for easy use below (and to ensure it's marked a secret!)
var privateKey = Output.CreateSecret(File.ReadAllText(privateKeyPath));

variables:
 ...
 # Read in the public key for easy use below.
 publicKey:
 Fn::ReadFile: ${publicKeyPath}
 # Read in the private key for easy use below (and to ensure it's marked a secret!)
 privateKey:
 Fn::Secret:
 Fn::ReadFile: ${privateKeyPath}

Next up, you will start declaring the AWS infrastructure building blocks needed to run your WordPress server.

Provisioning the AWS Network

First comes the network. This is the network into which you’ll deploy the WordPress database and VM. The VM needs to be accessible over the Internet so that you can access WordPress over HTTP(S). But the database clearly must not be! Only the VM will be able to access it. Accomplishing this first entails provisioning a VPC, public and private subnets, an Internet gateway, and route tables:

// Set up a Virtual Private Cloud to deploy our EC2 instance and RDS datbase into.
const prodVpc = new aws.ec2.Vpc("prod-vpc", {
 cidrBlock: "10.192.0.0/16",
 enableDnsSupport: true, // gives you an internal domain name.
 enableDnsHostnames: true, // gives you an internal host name.
 enableClassiclink: false,
 instanceTenancy: "default",
});

// Create public subnets for the EC2 instance.
const prodSubnetPublic1 = new aws.ec2.Subnet("prod-subnet-public-1", {
 vpcId: prodVpc.id,
 cidrBlock: "10.192.0.0/24",
 mapPublicIpOnLaunch: true, // public
 availabilityZone: availabilityZones.then(azs => azs.names[0]),
});

// Create private subnets for RDS:
const prodSubnetPrivate1 = new aws.ec2.Subnet("prod-subnet-private-1", {
 vpcId: prodVpc.id,
 cidrBlock: "10.192.20.0/24",
 mapPublicIpOnLaunch: false, // private
 availabilityZone: availabilityZones.then(azs => azs.names[1]),
});
const prodSubnetPrivate2 = new aws.ec2.Subnet("prod-subnet-private-2", {
 vpcId: prodVpc.id,
 cidrBlock: "10.192.21.0/24",
 mapPublicIpOnLaunch: false, // private
 availabilityZone: availabilityZones.then(azs => azs.names[2]),
});

// Create a gateway for internet connectivity:
const prodIgw = new aws.ec2.InternetGateway("prod-igw", {vpcId: prodVpc.id});

// Create and associate a route table:
const prodPublicRt = new aws.ec2.RouteTable("prod-public-rt", {
 vpcId: prodVpc.id,
 routes: [{
 // associated subnets can reach anywhere.
 cidrBlock: "0.0.0.0/0",
 // use this IGW to reach the internet.
 gatewayId: prodIgw.id,
 }],
});
const prodRtaPublicSubnet1 = new aws.ec2.RouteTableAssociation("prod-rta-public-subnet-1", {
 subnetId: prodSubnetPublic1.id,
 routeTableId: prodPublicRt.id,
});

# Set up a Virtual Private Cloud to deploy our EC2 instance and RDS datbase into.
prod_vpc = aws.ec2.Vpc("prod-vpc",
 cidr_block="10.192.0.0/16",
 enable_dns_support=True, # gives you an internal domain name
 enable_dns_hostnames=True, # gives yoiu an internal host name
 enable_classiclink=False,
 instance_tenancy="default")

# Create public subnets for the EC2 instance.
prod_subnet_public1 = aws.ec2.Subnet("prod-subnet-public-1",
 vpc_id=prod_vpc.id,
 cidr_block="10.192.0.0/24",
 map_public_ip_on_launch=True,
 availability_zone=availability_zones.names[0])

# Create private subnets for RDS:
prod_subnet_private1 = aws.ec2.Subnet("prod-subnet-private-1",
 vpc_id=prod_vpc.id,
 cidr_block="10.192.20.0/24",
 map_public_ip_on_launch=False,
 availability_zone=availability_zones.names[1])
prod_subnet_private2 = aws.ec2.Subnet("prod-subnet-private-2",
 vpc_id=prod_vpc.id,
 cidr_block="10.192.21.0/24",
 map_public_ip_on_launch=False,
 availability_zone=availability_zones.names[2])

# Create a gateway for internet connectivity:
prod_igw = aws.ec2.InternetGateway("prod-igw", vpc_id=prod_vpc.id)

# Create a route table:
prod_public_rt = aws.ec2.RouteTable("prod-public-rt",
 vpc_id=prod_vpc.id,
 routes=[aws.ec2.RouteTableRouteArgs(
 # associated subnets can reach anywhere:
 cidr_block="0.0.0.0/0",
 # use this IGW to reach the internet:
 gateway_id=prod_igw.id,
 )])
prod_rta_public_subnet1 = aws.ec2.RouteTableAssociation("prod-rta-public-subnet-1",
 subnet_id=prod_subnet_public1.id,
 route_table_id=prod_public_rt.id)

// Set up a Virtual Private Cloud to deploy our EC2 instance and RDS datbase into.
prodVpc, err := ec2.NewVpc(ctx, "prod-vpc", &ec2.VpcArgs{
 CidrBlock: pulumi.String("10.192.0.0/16"),
 EnableDnsSupport: pulumi.Bool(true), // gives you an internal domain name.
 EnableDnsHostnames: pulumi.Bool(true), // gives you an internal host name.
 EnableClassiclink: pulumi.Bool(false),
 InstanceTenancy: pulumi.String("default"),
})
if err != nil {
 return err
}
// Create public subnets for the EC2 instance.
prodSubnetPublic1, err := ec2.NewSubnet(ctx, "prod-subnet-public-1", &ec2.SubnetArgs{
 VpcId: prodVpc.ID(),
 CidrBlock: pulumi.String("10.192.0.0/24"),
 MapPublicIpOnLaunch: pulumi.Bool(true), // public
 AvailabilityZone: pulumi.String(availabilityZones.Names[0]),
})
if err != nil {
 return err
}
// Create private subnets for RDS:
prodSubnetPrivate1, err := ec2.NewSubnet(ctx, "prod-subnet-private-1", &ec2.SubnetArgs{
 VpcId: prodVpc.ID(),
 CidrBlock: pulumi.String("10.192.20.0/24"),
 MapPublicIpOnLaunch: pulumi.Bool(false), // private
 AvailabilityZone: pulumi.String(availabilityZones.Names[1]),
})
if err != nil {
 return err
}
prodSubnetPrivate2, err := ec2.NewSubnet(ctx, "prod-subnet-private-2", &ec2.SubnetArgs{
 VpcId: prodVpc.ID(),
 CidrBlock: pulumi.String("10.192.21.0/24"),
 MapPublicIpOnLaunch: pulumi.Bool(false), // private
 AvailabilityZone: pulumi.String(availabilityZones.Names[2]),
})
if err != nil {
 return err
}
// Create a gateway for internet connectivity:
prodIgw, err := ec2.NewInternetGateway(ctx, "prod-igw", &ec2.InternetGatewayArgs{
 VpcId: prodVpc.ID(),
})
if err != nil {
 return err
}
// Create a route table:
prodPublicRt, err := ec2.NewRouteTable(ctx, "prod-public-rt", &ec2.RouteTableArgs{
 VpcId: prodVpc.ID(),
 Routes: ec2.RouteTableRouteArray{
 &ec2.RouteTableRouteArgs{
 // associated subnets can reach anywhere.
 CidrBlock: pulumi.String("0.0.0.0/0"),
 // use this IGW to reach the internet.
 GatewayId: prodIgw.ID(),
 },
 },
})
if err != nil {
 return err
}
_, err = ec2.NewRouteTableAssociation(ctx, "prod-rta-public-subnet-1", &ec2.RouteTableAssociationArgs{
 SubnetId: prodSubnetPublic1.ID(),
 RouteTableId: prodPublicRt.ID(),
})
if err != nil {
 return err
}

// Set up a Virtual Private Cloud to deploy our EC2 instance and RDS datbase into.
var prodVpc = new Aws.Ec2.Vpc("prod-vpc", new Aws.Ec2.VpcArgs
{
 CidrBlock = "10.192.0.0/16",
 EnableDnsSupport = true, // gives you an internal domain name.
 EnableDnsHostnames = true, // gives you an internal host name.
 EnableClassiclink = false,
 InstanceTenancy = "default",
});

// Create public subnets for the EC2 instance.
var prodSubnetPublic1 = new Aws.Ec2.Subnet("prod-subnet-public-1", new Aws.Ec2.SubnetArgs
{
 VpcId = prodVpc.Id,
 CidrBlock = "10.192.0.0/24",
 MapPublicIpOnLaunch = true, // public
 AvailabilityZone = availabilityZones.Apply(az => az.Names[0]),
});

// Create private subnets for RDS:
var prodSubnetPrivate1 = new Aws.Ec2.Subnet("prod-subnet-private-1", new Aws.Ec2.SubnetArgs
{
 VpcId = prodVpc.Id,
 CidrBlock = "10.192.20.0/24",
 MapPublicIpOnLaunch = false, // private
 AvailabilityZone = availabilityZones.Apply(az => az.Names[1]),
});
var prodSubnetPrivate2 = new Aws.Ec2.Subnet("prod-subnet-private-2", new Aws.Ec2.SubnetArgs
{
 VpcId = prodVpc.Id,
 CidrBlock = "10.192.21.0/24",
 MapPublicIpOnLaunch = false, // private
 AvailabilityZone = availabilityZones.Apply(az => az.Names[2]),
});

// Create a gateway for internet connectivity:
var prodIgw = new Aws.Ec2.InternetGateway("prod-igw", new Aws.Ec2.InternetGatewayArgs
{
 VpcId = prodVpc.Id,
});

// Create and associate a route table:
var prodPublicCrt = new Aws.Ec2.RouteTable("prod-public-crt", new Aws.Ec2.RouteTableArgs
{
 VpcId = prodVpc.Id,
 Routes =
 {
 new Aws.Ec2.Inputs.RouteTableRouteArgs
 {
 // associated subnets can reach anywhere.
 CidrBlock = "0.0.0.0/0",
 // use this IGW to reach the internet.
 GatewayId = prodIgw.Id,
 },
 },
});
var prodCrtaPublicSubnet1 = new Aws.Ec2.RouteTableAssociation("prod-crta-public-subnet-1", new Aws.Ec2.RouteTableAssociationArgs
{
 SubnetId = prodSubnetPublic1.Id,
 RouteTableId = prodPublicCrt.Id,
});

resources:
 # Set up a Virtual Private Cloud to deploy our EC2 instance and RDS datbase into.
 prod-vpc:
 type: aws:ec2:Vpc
 properties:
 cidrBlock: 10.192.0.0/16
 enableDnsSupport: true # gives you an internal domain name
 enableDnsHostnames: true # gives you an internal host name
 enableClassiclink: false
 instanceTenancy: default
 # Create public subnets for the EC2 instance.
 prod-subnet-public-1:
 type: aws:ec2:Subnet
 properties:
 vpcId: ${prod-vpc.id}
 cidrBlock: 10.192.0.0/24
 mapPublicIpOnLaunch: true # public subnet
 availabilityZone: ${availabilityZones.names[0]}
 # Create private subnets for RDS:
 prod-subnet-private-1:
 type: aws:ec2:Subnet
 properties:
 vpcId: ${prod-vpc.id}
 cidrBlock: 10.192.20.0/24
 mapPublicIpOnLaunch: false # private subnet
 availabilityZone: ${availabilityZones.names[1]}
 prod-subnet-private-2:
 type: aws:ec2:Subnet
 properties:
 vpcId: ${prod-vpc.id}
 cidrBlock: 10.192.21.0/24
 mapPublicIpOnLaunch: false # private subnet
 availabilityZone: ${availabilityZones.names[2]}
 # Create a gateway for internet connectivity:
 prod-igw:
 type: aws:ec2:InternetGateway
 properties:
 vpcId: ${prod-vpc.id}
 # Create a route table:
 prod-public-rt:
 type: aws:ec2:RouteTable
 properties:
 vpcId: ${prod-vpc.id}
 routes:
 # associated subnets can reach anywhere:
 - cidrBlock: 0.0.0.0/0
 # we use this IGW to reach the internet:
 gatewayId: ${prod-igw.id}
 prod-rta-public-subnet-1:
 type: aws:ec2:RouteTableAssociation
 properties:
 subnetId: ${prod-subnet-public-1.id}
 routeTableId: ${prod-public-rt.id}

Note: All of the AWS building blocks are declared one at a time here to make it entirely transparent what’s going on. Once you’ve set up a VPC a handful of times, however, this can get tedious. The Pulumi Crosswalk for AWS package offers standard VPC configurations out-of-the-box with fewer lines of code, and built-in best practices. Check it out!

Next, to accomplish the stated connectivity goals — EC2 instance available over HTTP(S) and SSH on the Internet, and the RDS database available only to MySQL traffic originating from the EC2 instance itself – you’ll now create two security groups, one each for the public and private subnets respectively:

// Security group for EC2:
const ec2AllowRule = new aws.ec2.SecurityGroup("ec2-allow-rule", {
 vpcId: prodVpc.id,
 ingress: [
 {
 description: "HTTPS",
 fromPort: 443,
 toPort: 443,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 {
 description: "HTTP",
 fromPort: 80,
 toPort: 80,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 {
 description: "SSH",
 fromPort: 22,
 toPort: 22,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
 egress: [{
 fromPort: 0,
 toPort: 0,
 protocol: "-1",
 cidrBlocks: ["0.0.0.0/0"],
 }],
 tags: {
 Name: "allow ssh,http,https",
 },
});

// Security group for RDS:
const rdsAllowRule = new aws.ec2.SecurityGroup("rds-allow-rule", {
 vpcId: prodVpc.id,
 ingress: [{
 description: "MySQL",
 fromPort: 3306,
 toPort: 3306,
 protocol: "tcp",
 securityGroups: [ec2AllowRule.id],
 }],
 // Allow all outbound traffic.
 egress: [{
 fromPort: 0,
 toPort: 0,
 protocol: "-1",
 cidrBlocks: ["0.0.0.0/0"],
 }],
 tags: {
 Name: "allow ec2",
 },
});

# Security group for EC2:
ec2_allow_rule = aws.ec2.SecurityGroup("ec2-allow-rule",
 vpc_id=prod_vpc.id,
 ingress=[
 aws.ec2.SecurityGroupIngressArgs(
 description="HTTPS",
 from_port=443,
 to_port=443,
 protocol="tcp",
 cidr_blocks=["0.0.0.0/0"],
 ),
 aws.ec2.SecurityGroupIngressArgs(
 description="HTTP",
 from_port=80,
 to_port=80,
 protocol="tcp",
 cidr_blocks=["0.0.0.0/0"],
 ),
 aws.ec2.SecurityGroupIngressArgs(
 description="SSH",
 from_port=22,
 to_port=22,
 protocol="tcp",
 cidr_blocks=["0.0.0.0/0"],
 ),
 ],
 egress=[aws.ec2.SecurityGroupEgressArgs(
 from_port=0,
 to_port=0,
 protocol="-1",
 cidr_blocks=["0.0.0.0/0"],
 )],
 tags={
 "Name": "allow ssh,http,https",
 })

# Security group for RDS:
rds_allow_rule = aws.ec2.SecurityGroup("rds-allow-rule",
 vpc_id=prod_vpc.id,
 ingress=[aws.ec2.SecurityGroupIngressArgs(
 description="MySQL",
 from_port=3306,
 to_port=3306,
 protocol="tcp",
 security_groups=[ec2_allow_rule.id],
 )],
 # allow all outbound traffic.
 egress=[aws.ec2.SecurityGroupEgressArgs(
 from_port=0,
 to_port=0,
 protocol="-1",
 cidr_blocks=["0.0.0.0/0"],
 )],
 tags={
 "Name": "allow ec2",
 })

// Security group for EC2:
ec2AllowRule, err := ec2.NewSecurityGroup(ctx, "ec2-allow-rule", &ec2.SecurityGroupArgs{
 VpcId: prodVpc.ID(),
 Ingress: ec2.SecurityGroupIngressArray{
 &ec2.SecurityGroupIngressArgs{
 Description: pulumi.String("HTTPS"),
 FromPort: pulumi.Int(443),
 ToPort: pulumi.Int(443),
 Protocol: pulumi.String("tcp"),
 CidrBlocks: pulumi.StringArray{
 pulumi.String("0.0.0.0/0"),
 },
 },
 &ec2.SecurityGroupIngressArgs{
 Description: pulumi.String("HTTP"),
 FromPort: pulumi.Int(80),
 ToPort: pulumi.Int(80),
 Protocol: pulumi.String("tcp"),
 CidrBlocks: pulumi.StringArray{
 pulumi.String("0.0.0.0/0"),
 },
 },
 &ec2.SecurityGroupIngressArgs{
 Description: pulumi.String("SSH"),
 FromPort: pulumi.Int(22),
 ToPort: pulumi.Int(22),
 Protocol: pulumi.String("tcp"),
 CidrBlocks: pulumi.StringArray{
 pulumi.String("0.0.0.0/0"),
 },
 },
 },
 Egress: ec2.SecurityGroupEgressArray{
 &ec2.SecurityGroupEgressArgs{
 FromPort: pulumi.Int(0),
 ToPort: pulumi.Int(0),
 Protocol: pulumi.String("-1"),
 CidrBlocks: pulumi.StringArray{
 pulumi.String("0.0.0.0/0"),
 },
 },
 },
 Tags: pulumi.StringMap{
 "Name": pulumi.String("allow ssh,http,https"),
 },
})
if err != nil {
 return err
}
// Security group for RDS:
rdsAllowRule, err := ec2.NewSecurityGroup(ctx, "rds-allow-rule", &ec2.SecurityGroupArgs{
 VpcId: prodVpc.ID(),
 Ingress: ec2.SecurityGroupIngressArray{
 &ec2.SecurityGroupIngressArgs{
 Description: pulumi.String("MySQL"),
 FromPort: pulumi.Int(3306),
 ToPort: pulumi.Int(3306),
 Protocol: pulumi.String("tcp"),
 SecurityGroups: pulumi.StringArray{
 ec2AllowRule.ID(),
 },
 },
 },
 // Allow all outbound traffic.
 Egress: ec2.SecurityGroupEgressArray{
 &ec2.SecurityGroupEgressArgs{
 FromPort: pulumi.Int(0),
 ToPort: pulumi.Int(0),
 Protocol: pulumi.String("-1"),
 CidrBlocks: pulumi.StringArray{
 pulumi.String("0.0.0.0/0"),
 },
 },
 },
 Tags: pulumi.StringMap{
 "Name": pulumi.String("allow ec2"),
 },
})
if err != nil {
 return err
}

// Security group for EC2:
var ec2AllowRule = new Aws.Ec2.SecurityGroup("ec2-allow-rule", new Aws.Ec2.SecurityGroupArgs
{
 VpcId = prodVpc.Id,
 Ingress =
 {
 new Aws.Ec2.Inputs.SecurityGroupIngressArgs
 {
 Description = "HTTPS",
 FromPort = 443,
 ToPort = 443,
 Protocol = "tcp",
 CidrBlocks =
 {
 "0.0.0.0/0",
 },
 },
 new Aws.Ec2.Inputs.SecurityGroupIngressArgs
 {
 Description = "HTTP",
 FromPort = 80,
 ToPort = 80,
 Protocol = "tcp",
 CidrBlocks =
 {
 "0.0.0.0/0",
 },
 },
 new Aws.Ec2.Inputs.SecurityGroupIngressArgs
 {
 Description = "SSH",
 FromPort = 22,
 ToPort = 22,
 Protocol = "tcp",
 CidrBlocks =
 {
 "0.0.0.0/0",
 },
 },
 },
 Egress =
 {
 new Aws.Ec2.Inputs.SecurityGroupEgressArgs
 {
 FromPort = 0,
 ToPort = 0,
 Protocol = "-1",
 CidrBlocks =
 {
 "0.0.0.0/0",
 },
 },
 },
 Tags =
 {
 { "Name", "allow ssh,http,https" },
 },
});

// Security group for RDS:
var rdsAllowRule = new Aws.Ec2.SecurityGroup("rds-allow-rule", new Aws.Ec2.SecurityGroupArgs
{
 VpcId = prodVpc.Id,
 Ingress =
 {
 new Aws.Ec2.Inputs.SecurityGroupIngressArgs
 {
 Description = "MySQL",
 FromPort = 3306,
 ToPort = 3306,
 Protocol = "tcp",
 SecurityGroups =
 {
 ec2AllowRule.Id,
 },
 },
 },
 // Allow all outbound traffic.
 Egress =
 {
 new Aws.Ec2.Inputs.SecurityGroupEgressArgs
 {
 FromPort = 0,
 ToPort = 0,
 Protocol = "-1",
 CidrBlocks =
 {
 "0.0.0.0/0",
 },
 },
 },
 Tags =
 {
 { "Name", "allow ec2" },
 },
});

 # Security group for EC2:
 ec2-allow-rule:
 type: aws:ec2:SecurityGroup
 properties:
 vpcId: ${prod-vpc.id}
 ingress:
 - description: "HTTPS"
 fromPort: 443
 toPort: 443
 protocol: "tcp"
 cidrBlocks: [0.0.0.0/0]
 - description: "HTTP"
 fromPort: 80
 toPort: 80
 protocol: "tcp"
 cidrBlocks: [0.0.0.0/0]
 - description: "SSH"
 fromPort: 22
 toPort: 22
 protocol: "tcp"
 cidrBlocks: [0.0.0.0/0]
 egress:
 - fromPort: 0
 toPort: 0
 protocol: "-1"
 cidrBlocks: [0.0.0.0/0]
 tags:
 Name: "allow ssh,http,https"
 # Security group for RDS:
 rds-allow-rule:
 type: aws:ec2:SecurityGroup
 properties:
 vpcId: ${prod-vpc.id}
 # Only allow MySQL ingress from the EC2 VM.
 ingress:
 - description: "MySQL"
 fromPort: 3306
 toPort: 3306
 protocol: "tcp"
 securityGroups: ["${ec2-allow-rule.id}"]
 # Allow all outbound traffic.
 egress:
 - fromPort: 0
 toPort: 0
 protocol: "-1"
 cidrBlocks: [0.0.0.0/0]
 tags:
 Name: "allow ec2"

Now that you have a network, you will deploy the WordPress RDS database and the EC2 instance into it.

Provisioning the AWS EC2 Instance and RDS Database

To start, create a RDS database running MySQL 5.7 so that WordPress can use it. The database is given a specific database name, username, and password, so that WordPress can be subsequently configured to access it using this information. You’ll also create a subnet group to place RDS into the private subnets allocated for it earlier:

// Place the RDS instance into private subnets:
const rdsSubnetGrp = new aws.rds.SubnetGroup("rds-subnet-grp", {subnetIds: [
 prodSubnetPrivate1.id,
 prodSubnetPrivate2.id,
]});

// Create the RDS instance:
const wordpressdb = new aws.rds.Instance("wordpressdb", {
 allocatedStorage: 10,
 engine: "mysql",
 engineVersion: "5.7",
 instanceClass: dbInstanceSize,
 dbSubnetGroupName: rdsSubnetGrp.id,
 vpcSecurityGroupIds: [rdsAllowRule.id],
 dbName: dbName,
 username: dbUsername,
 password: dbPassword,
 skipFinalSnapshot: true,
});

# Place the RDS instance into private subnets:
rds_subnet_grp = aws.rds.SubnetGroup("rds-subnet-grp", subnet_ids=[
 prod_subnet_private1.id,
 prod_subnet_private2.id,
])

# Create the RDS instance:
wordpressdb = aws.rds.Instance("wordpressdb",
 allocated_storage=10,
 engine="mysql",
 engine_version="5.7",
 instance_class=db_instance_size,
 db_subnet_group_name=rds_subnet_grp.id,
 vpc_security_group_ids=[rds_allow_rule.id],
 db_name=db_name,
 username=db_username,
 password=db_password,
 skip_final_snapshot=True)

// Place the RDS instance into private subnets:
rdsSubnetGrp, err := rds.NewSubnetGroup(ctx, "rds-subnet-grp", &rds.SubnetGroupArgs{
 SubnetIds: pulumi.StringArray{
 prodSubnetPrivate1.ID(),
 prodSubnetPrivate2.ID(),
 },
})
if err != nil {
 return err
}
// Create the RDS instance:
wordpressdb, err := rds.NewInstance(ctx, "wordpressdb", &rds.InstanceArgs{
 AllocatedStorage: pulumi.Int(10),
 Engine: pulumi.String("mysql"),
 EngineVersion: pulumi.String("5.7"),
 InstanceClass: pulumi.String(dbInstanceSize),
 DbSubnetGroupName: rdsSubnetGrp.ID(),
 VpcSecurityGroupIds: pulumi.StringArray{
 rdsAllowRule.ID(),
 },
 DbName: pulumi.String(dbName),
 Username: pulumi.String(dbUsername),
 Password: dbPassword,
 SkipFinalSnapshot: pulumi.Bool(true),
})
if err != nil {
 return err
}

// Place the RDS instance into private subnets:
var rdsSubnetGrp = new Aws.Rds.SubnetGroup("rds-subnet-grp", new Aws.Rds.SubnetGroupArgs
{
 SubnetIds =
 {
 prodSubnetPrivate1.Id,
 prodSubnetPrivate2.Id,
 },
});

// Create the RDS instance:
var wordpressdb = new Aws.Rds.Instance("wordpressdb", new Aws.Rds.InstanceArgs
{
 AllocatedStorage = 10,
 Engine = "mysql",
 EngineVersion = "5.7",
 InstanceClass = dbInstanceSize,
 DbSubnetGroupName = rdsSubnetGrp.Id,
 VpcSecurityGroupIds =
 {
 rdsAllowRule.Id,
 },
 DbName = dbName,
 Username = dbUsername,
 Password = dbPassword,
 SkipFinalSnapshot = true,
});

 # Place RDS into the private subnets:
 rds-subnet-grp:
 type: aws:rds:SubnetGroup
 properties:
 subnetIds:
 - ${prod-subnet-private-1.id}
 - ${prod-subnet-private-2.id}
 # Create the RDS instance:
 wordpressdb:
 type: aws:rds:Instance
 properties:
 allocatedStorage: 10
 engine: "mysql"
 engineVersion: "5.7"
 instanceClass: ${dbInstanceSize}
 dbSubnetGroupName: ${rds-subnet-grp.id}
 vpcSecurityGroupIds: ["${rds-allow-rule.id}"]
 dbName: ${dbName}
 username: ${dbUsername}
 password: ${dbPassword}
 skipFinalSnapshot: true

Now that you have a database, you’re ready to provision the EC2 instance and supporting infrastructure. This step comes after the RDS database, since WordPress running on the instance needs to access the database. Pulumi tracks the dependencies between them so it performs operations in the correct order.

Start by creating an EC2 key pair so you can SSH into the machine. This is also required to execute remote commands during the post-provisioning configuration:

// Create a keypair to access the EC2 instance:
const wordpressKeypair = new aws.ec2.KeyPair("wordpress-keypair", {
 publicKey: publicKey,
});

# Create a keypair to access the EC2 instance:
wordpress_keypair = aws.ec2.KeyPair("wordpress-keypair", public_key=public_key)

// Create a keypair to access the EC2 instance:
wordpressKeypair, err := ec2.NewKeyPair(ctx, "wordpress-keypair", &ec2.KeyPairArgs{
 PublicKey: pulumi.String(publicKey),
})
if err != nil {
 return err
}

// Create a keypair to access the EC2 instance:
var wordpressKeypair = new Aws.Ec2.KeyPair("wordpress-keypair", new Aws.Ec2.KeyPairArgs
{
 PublicKey = publicKey,
});

 # Create a keypair to access the EC2 instance:
 wordpress-keypair:
 type: aws:ec2:KeyPair
 properties:
 publicKey: ${publicKey}

Note: Enabling operators to manually SSH into machines is a common practice. Most of us are generally trying to move away from it these days, however. SSH’ing into a machine and manually running commands just leads to headaches: configuration drift, unrepeatable automation, and deployment outages, to name a few. All of these are antithetical to the modern way of doing cloud engineering – but no doubt, it’s unfortunately still relatively commonplace.

Now create the EC2 instance and give it an elastic IP (EIP) address so that it’s stable even if you need to upgrade or even entirely replace the server:

// Create an EC2 instance to run Wordpress (after RDS is ready).
const wordpressInstance = new aws.ec2.Instance("wordpress-instance", {
 ami: awsLinuxAmi.then(awsLinuxAmi => awsLinuxAmi.id),
 instanceType: ec2InstanceSize,
 subnetId: prodSubnetPublic1.id,
 vpcSecurityGroupIds: [ec2AllowRule.id],
 keyName: wordpressKeypair.id,
 tags: {
 Name: "Wordpress.web",
 },
}, {
 // Only create after RDS is provisioned.
 dependsOn: [wordpressdb],
});

// Give our EC2 instance an elastic IP address.
const wordpressEip = new aws.ec2.Eip("wordpress-eip", {
 instance: wordpressInstance.id,
});

# Create an EC2 instance to run Wordpress (after RDS is ready).
wordpress_instance = aws.ec2.Instance("wordpress-instance",
 ami=aws_linux_ami.id,
 instance_type=ec2_instance_size,
 subnet_id=prod_subnet_public1.id,
 vpc_security_group_ids=[ec2_allow_rule.id],
 key_name=wordpress_keypair.id,
 tags={
 "Name": "Wordpress.web",
 },
 # Only create after RDS is provisioned.
 opts=pulumi.ResourceOptions(depends_on=[wordpressdb]))

# Give our EC2 instance an elastic IP address.
wordpress_eip = aws.ec2.Eip("wordpress-eip", instance=wordpress_instance.id)

// Create an EC2 instance to run Wordpress (after RDS is ready).
wordpressInstance, err := ec2.NewInstance(ctx, "wordpress-instance", &ec2.InstanceArgs{
 Ami: pulumi.String(awsLinuxAmi.Id),
 InstanceType: pulumi.String(ec2InstanceSize),
 SubnetId: prodSubnetPublic1.ID(),
 VpcSecurityGroupIds: pulumi.StringArray{
 ec2AllowRule.ID(),
 },
 KeyName: wordpressKeypair.ID(),
 Tags: pulumi.StringMap{
 "Name": pulumi.String("Wordpress.web"),
 },
}, pulumi.DependsOn([]pulumi.Resource{
 // Only create after RDS is provisioned.
 wordpressdb,
}))
if err != nil {
 return err
}

// Give our EC2 instance an elastic IP address.
wordpressEip, err := ec2.NewEip(ctx, "wordpress-eip", &ec2.EipArgs{
 Instance: wordpressInstance.ID(),
})
if err != nil {
 return err
}

// Create an EC2 instance to run Wordpress (after RDS is ready).
var wordpressInstance = new Aws.Ec2.Instance("wordpress-instance", new Aws.Ec2.InstanceArgs
{
 Ami = awsLinuxAmi.Apply(awsLinuxAmi => awsLinuxAmi.Id),
 InstanceType = ec2InstanceSize,
 SubnetId = prodSubnetPublic1.Id,
 VpcSecurityGroupIds =
 {
 ec2AllowRule.Id,
 },
 KeyName = wordpressKeypair.Id,
 Tags =
 {
 { "Name", "Wordpress.web" },
 },
}, new CustomResourceOptions
{
 // Only create after RDS is provisioned.
 DependsOn = { wordpressdb },
});

// Give our EC2 instance an elastic IP address.
var wordpressEip = new Aws.Ec2.Eip("wordpress-eip", new Aws.Ec2.EipArgs
{
 Instance = wordpressInstance.Id,
});

 # Create an EC2 instance to run Wordpress (after RDS is ready).
 wordpress-instance:
 type: aws:ec2:Instance
 properties:
 ami: ${awsLinuxAmi.id}
 instanceType: ${ec2InstanceSize}
 subnetId: ${prod-subnet-public-1.id}
 vpcSecurityGroupIds: ["${ec2-allow-rule.id}"]
 keyName: ${wordpress-keypair.id}
 tags:
 Name: "Wordpress.web"
 options:
 dependsOn:
 # Only create after RDS is provisioned.
 - ${wordpressdb}
 # Give our EC2 instance an elastic IP address.
 wordpress-eip:
 type: aws:ec2:Eip
 properties:
 instance: ${wordpress-instance.id}

Configuring the AWS EC2 Instance Using the Pulumi Command Package

Here’s where things deviate from the normal Pulumi way of doing things: we’ll now configure our server.

Pulumi’s Command package lets you run local or remote commands sequenced amongst other ordinary infrastructure as code actions. For instance, you can copy a local file to a remote server using SCP, run a local command, or run a remote command over SSH, before or after creating or destroying other infrastructure. This is precisely the building block that will allow Pulumi to orchestrate configuration management tasks.

The first thing to do is render the Ansible playbook. Why “render”? The playbook.yml needs information only known at deployment time within your Pulumi program, including the server’s address, the database URL, name, username, password, and so on, all of which are configurable and only known after spinning up the server and database.

The playbook.yml contains everything necessary to get Wordpress up and running. Notice the templated variables like $DB_RDS that need to be substituted with real values:

# An Ansible Playbook that installs Wordpress on a fresh Amazon Linux 2 Server.
# The only prerequisite is that Python 2.7 and Ansible are available on the server.
- name: Install Wordpress on new Amazon Linux 2 Server.
 hosts: all
 become: yes

 vars:
 # Explicitly use Python2.7, since this playbook requires it.
 ansible_python_interpreter: /usr/bin/python

 tasks:

 - name: Setting up variables
 set_fact:
 php_modules:
...
 # RDS MySQL variables populated post-provisioning:
 mysql_rds: $DB_RDS
 mysql_db: $DB_NAME
 mysql_user: $DB_USERNAME
 mysql_password: $DB_PASSWORD

 - name: Yum update
...

Note: The playbook file has been truncated for brevity. The full playbook.yml is available here.

To render the playbook, you’ll just use the command:local:Command resource to execute the standard envsubst UNIX command:

// Render the Ansible playbook using RDS info.
const renderPlaybookCmd = new command.local.Command("renderPlaybookCmd", {
 create: "cat playbook.yml | envsubst > playbook_rendered.yml",
 environment: {
 DB_RDS: wordpressdb.endpoint,
 DB_NAME: dbName,
 DB_USERNAME: dbUsername,
 DB_PASSWORD: dbPassword,
 },
});

# Render the Ansible playbook using RDS info.
render_playbook_cmd = command.local.Command("renderPlaybookCmd",
 create="cat playbook.yml | envsubst > playbook_rendered.yml",
 environment={
 "DB_RDS": wordpressdb.endpoint,
 "DB_NAME": db_name,
 "DB_USERNAME": db_username,
 "DB_PASSWORD": db_password,
 })

// Render the Ansible playbook using RDS info.
renderPlaybookCmd, err := local.NewCommand(ctx, "renderPlaybookCmd", &local.CommandArgs{
 Create: pulumi.String("cat playbook.yml | envsubst > playbook_rendered.yml"),
 Environment: pulumi.StringMap{
 "DB_RDS": wordpressdb.Endpoint,
 "DB_NAME": pulumi.String(dbName),
 "DB_USERNAME": pulumi.String(dbUsername),
 "DB_PASSWORD": dbPassword,
 },
})
if err != nil {
 return err
}

// Render the Ansible playbook using RDS info.
var renderPlaybookCmd = new Command.Local.Command("renderPlaybookCmd", new Command.Local.CommandArgs
{
 Create = "cat playbook.yml | envsubst > playbook_rendered.yml",
 Environment =
 {
 { "DB_RDS", wordpressdb.Endpoint },
 { "DB_NAME", dbName },
 { "DB_USERNAME", dbUsername },
 { "DB_PASSWORD", dbPassword },
 },
});

 # Render the Ansible playbook using RDS info.
 renderPlaybookCmd:
 type: command:local:Command
 properties:
 create: "cat playbook.yml | envsubst > playbook_rendered.yml"
 environment:
 DB_RDS: ${wordpressdb.endpoint}
 DB_NAME: ${dbName}
 DB_USERNAME: ${dbUsername}
 DB_PASSWORD: ${dbPassword}

This command runs locally on the machine running Pulumi (the control node in Ansible parlance), which means we’ll end up with a file, playbook_rendered.yml, very similar to playbook.yml above but with variables substituted, after executing it:

# An Ansible Playbook that installs Wordpress on a fresh Amazon Linux 2 Server.
# The only prerequisite is that Python 2.7 and Ansible are available on the server.
- name: Install Wordpress on new Amazon Linux 2 Server.
 hosts: all
 become: yes

 vars:
 # Explicitly use Python2.7, since this playbook requires it.
 ansible_python_interpreter: /usr/bin/python

 tasks:

 - name: Setting up variables
 set_fact:
 php_modules:
...

 # RDS MySQL variables populated post-provisioning:
 mysql_rds: wordpressdb72d0d1x.fep6qenxwgtk.us-west-2.rds.amazonaws.com:3306
 mysql_db: wordpressdb
 mysql_user: admin
 mysql_password: [secret]

 - name: Yum update
...

Now things get more interesting: you will run a remote command over SSH to run yum commands that update packages and ensure Python is available, since Ansible will need it. This is done with the command:remote:Command and it fully understands how and when to connect remotely to the newly provisioned EC2 instance:

// Run a script to update packages on the remote machine.
const updatePythonCmd = new command.remote.Command("updatePythonCmd", {
 connection: {
 host: wordpressEip.publicIp,
 port: 22,
 user: "ec2-user",
 privateKey: privateKey,
 },
 create: `(sudo yum update -y || true);
(sudo yum install python35 -y);
(sudo yum install amazon-linux-extras -y)
`,
});

# Run a script to update packages on the remote machine.
update_python_cmd = command.remote.Command("updatePythonCmd",
 connection=command.remote.ConnectionArgs(
 host=wordpress_eip.public_ip,
 port=22,
 user="ec2-user",
 private_key=private_key,
 ),
 create="""(sudo yum update -y || true);
(sudo yum install python35 -y);
(sudo yum install amazon-linux-extras -y)
""")

// Run a script to update packages on the remote machine.
updatePythonCmd, err := remote.NewCommand(ctx, "updatePythonCmd", &remote.CommandArgs{
 Connection: &remote.ConnectionArgs{
 Host: wordpressEip.PublicIp,
 Port: pulumi.Float64(22),
 User: pulumi.String("ec2-user"),
 PrivateKey: privateKey,
 },
 Create: pulumi.String("(sudo yum update -y || true);" +
 "(sudo yum install python35 -y);" +
 "(sudo yum install amazon-linux-extras -y)\n"),
})
if err != nil {
 return err
}

// Run a script to update packages on the remote machine.
var updatePythonCmd = new Command.Remote.Command("updatePythonCmd", new Command.Remote.CommandArgs
{
 Connection = new Command.Remote.Inputs.ConnectionArgs
 {
 Host = wordpressEip.PublicIp,
 Port = 22,
 User = "ec2-user",
 PrivateKey = privateKey,
 },
 Create = "(sudo yum update -y || true);"+
 "(sudo yum install python35 -y);"+
 "(sudo yum install amazon-linux-extras -y)"
});

 # Run a script to update packages on the remote machine.
 updatePythonCmd:
 type: command:remote:Command
 properties:
 connection:
 host: ${wordpress-eip.publicIp}
 port: 22
 user: ec2-user
 privateKey: ${privateKey}
 create: >
 (sudo yum update -y || true);
 (sudo yum install python35 -y);
 (sudo yum install amazon-linux-extras -y)

This looks similar to the local command execution, except that it uses the remote:Command type instead of local:Command, and also needs the EC2 key pair’s private key to SSH into the server. Pulumi’s dependency system ensures this action is sequenced correctly after the server is running and information required to perform the command is available.

Configuring WordPress on EC2 Using Ansible

Alas, we now reach the pièce de résistance! In this section, you will configure WordPress on the new EC2 instance using Ansible.

A local command will run the rendered Ansible playbook using the ansible-playbook command, effectively turning the machine running Pulumi into an Ansible control node. Ansible will chug away on the playbook, executing commands as specified within it, thereby installing WordPress on your new server:

// Finally, play the Ansible playbook to finish installing.
const playAnsiblePlaybookCmd = new command.local.Command("playAnsiblePlaybookCmd", {
 create: pulumi.interpolate`ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook \
-u ec2-user -i '${wordpressEip.publicIp},' \
--private-key ${privateKeyPath} \
playbook_rendered.yml`
}, {
 dependsOn: [
 renderPlaybookCmd,
 updatePythonCmd,
 ],
});

# Finally, play the Ansible playbook to finish installing.
play_ansible_playbook_cmd = command.local.Command("playAnsiblePlaybookCmd",
 create=wordpress_eip.public_ip.apply(lambda public_ip: f"""\
ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook \
-u ec2-user \
-i '{public_ip},' \
--private-key {private_key_path} \
playbook_rendered.yml"""),
opts=pulumi.ResourceOptions(depends_on=[
 render_playbook_cmd,
 update_python_cmd,
 ]))

// Finally, play the Ansible playbook to finish installing.
_, err = local.NewCommand(ctx, "playAnsiblePlaybookCmd", &local.CommandArgs{
 Create: wordpressEip.PublicIp.ApplyT(func(publicIp string) (string, error) {
 return fmt.Sprintf("ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook "+
 "-u ec2-user "+
 "-i '%v,' "+
 "--private-key %v "+
 "playbook_rendered.yml", publicIp, privateKeyPath), nil
 }).(pulumi.StringOutput),
}, pulumi.DependsOn([]pulumi.Resource{
 renderPlaybookCmd,
 updatePythonCmd,
}))
if err != nil {
 return err
}

var playAnsiblePlaybookCmd = new Command.Local.Command("playAnsiblePlaybookCmd", new Command.Local.CommandArgs
{
 Create = wordpressEip.PublicIp.Apply(publicIp =>
 $"ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook "+
 $"-u ec2-user "+
 $"-i '{publicIp},' "+
 $"--private-key {privateKeyPath} "+
 $"playbook_rendered.yml"),
}, new CustomResourceOptions
{
 DependsOn =
 {
 renderPlaybookCmd,
 updatePythonCmd,
 },
});

 # Finally, play the Ansible playbook to finish installing.
 playAnsiblePlaybookCmd:
 type: command:local:Command
 properties:
 create: "ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook
 -u ec2-user
 -i '${wordpress-eip.publicIp},'
 --private-key ${privateKeyPath}
 playbook_rendered.yml"
 options:
 dependsOn:
 - ${renderPlaybookCmd}
 - ${updatePythonCmd}

That’s it! The entirety of your infrastructure — from the underlying AWS infrastructure, to the configuration commands to upgrade your server, to running the Ansible playbook which installs WordPress – is declared in a single Pulumi program in your language of choice. The only step left is to deploy it!

Provision and configure in one workflow

Combine Pulumi provisioning with configuration management tools like Ansible to build repeatable environments. Deploy your AWS infrastructure and configure your servers with a single pulumi up.

Get started

Seeing Everything in Action

To deploy this, you just need to do a few things.

You already created a project with pulumi new earlier, but can always create additional stacks using pulumi stack init.

Next, generate the key pair that will be used to access the EC2 instance over SSH:

$ ssh-keygen -f wordpress-keypair

This will create two files: your private (wordpress-keypair) and your public (wordpress-keypair.pub) keys. Keep your private key safe, since anybody with access to it can log into your EC2 machine!

Note that you may choose a different file name if you’re creating multiple stacks, for instance.

After that, simply set the Pulumi configuration values for the various settings. That includes any valid AWS region, the public and private keys, and a database password (which you should encrypt using --secret):

$ pulumi config set aws:region us-east-1 # any valid AWS region
$ pulumi config set publicKeyPath wordpress-keypair.pub # your newly generated public key
$ pulumi config set privateKeyPath wordpress-keypair # your newly generated private key
$ pulumi config set dbPassword Sup45ekreT@ --secret # your RDS database password -- keep it safe!

There are some other optional variables you can set if you choose. Feel free to skip these. If you don’t set them, they’ll receive the default values shown below:

$ pulumi config set dbInstanceSize db.t3.small # the RDS instance size to use
$ pulumi config set dbName wordpressdb # the name of the Wordpress database in RDS
$ pulumi config set dbUsername admin # the name of the Wordpress user that will be used
$ pulumi config set ec2InstanceSize t3.small # the EC2 instance size to provision

Now all you need to do is run pulumi up. This will do all of the magic, and you’ll see various things going on in the output: resources being created in AWS, commands being run locally and remotely, and the Ansible Playbook running and provisioning your Wordpress server:

$ pulumi up
Updating (dev)

 Type Name Status Info
 + pulumi:pulumi:Stack pulumi-ansible-wordpress-dev created
 + ├─ aws:ec2:Vpc prod-vpc created
 + ├─ aws:ec2:KeyPair wordpress-keypair created
 + ├─ aws:ec2:Subnet prod-subnet-private-1 created
 + ├─ aws:ec2:InternetGateway prod-igw created
 + ├─ aws:ec2:Subnet prod-subnet-private-2 created
 + ├─ aws:ec2:Subnet prod-subnet-public-1 created
 + ├─ aws:ec2:SecurityGroup ec2-allow-rule created
 + ├─ aws:ec2:RouteTable prod-public-rt created
 + ├─ aws:rds:SubnetGroup rds-subnet-grp created
 + ├─ aws:ec2:SecurityGroup rds-allow-rule created
 + ├─ aws:rds:Instance wordpressdb created
 + ├─ aws:ec2:RouteTableAssociation prod-rta-public-subnet-1 created
 + ├─ aws:ec2:Instance wordpress-instance created
 + ├─ command:local:Command renderPlaybookCmd created
 + ├─ aws:ec2:Eip wordpress-eip created
 + ├─ command:remote:Command updatePythonCmd created 12 messages
 + └─ command:local:Command playAnsiblePlaybookCmd created

Diagnostics:
 command:remote:Command (updatePythonCmd):
...

Outputs:
 url: "35.83.214.168"

Resources:
 + 18 created

Duration: 8m13s

After a few minutes, your new server will be ready! Notice that Pulumi orchestrates all of it, in the right order, including Ansible.

Now that everything’s up and running, let’s check out WordPress. The server’s security group allows HTTP(S) Internet traffic, so you just need to get its address. You probably noticed it was printed in the example output above. The program exports the resulting auto-allocated EIP address to make fetching it easy:

// Export the resulting wordpress EIP for easy access.
export const url = wordpressEip.publicIp;

# Export the resulting wordpress EIP for easy access.
pulumi.export("url", wordpress_eip.public_ip)

// Export the resulting wordpress EIP for easy access.
ctx.Export("url", wordpressEip.PublicIp)

// Export the resulting wordpress EIP for easy access.
this.Url = wordpressEip.PublicIp;

# Export the resulting wordpress EIP for easy access.
outputs:
 url: ${wordpress-eip.publicIp}

The IP address shows up at the end of pulumi up, but you can also access it with the pulumi stack output command:

$ pulumi stack output url
35.83.214.168

Using this, you can curl the WordPress server and see your handy work in action:

$ curl -L http://$(pulumi stack output url)
<!DOCTYPE html>
<html lang="en-US" xml:lang="en-US">
<head>...</head>
<body class="wp-core-ui language-chooser">
...

Or you can open a web browser to see a more visually pleasing WordPress welcome page:

$ open http://$(pulumi stack output url)

From there, feel free to experiment. You can simply make edits, rerun pulumi up, and it will incrementally update and deploy any changes you have made. After you’re done, you can destroy your stack and remove it, eliminating all AWS resources entirely:

$ pulumi destroy
$ pulumi stack rm

And there you go, cattle and pets, provisioning and configuration infrastructure as code, Pulumi and Ansible — better together!

Winding Down

In this article, you saw that provisioning and configuration don’t need to be an either-or. Indeed, they can be used together quite naturally. This is helpful for many configuration scenarios because infrastructure needs to be created before it can be configured, which is exactly what provisioning does. This can also be a great way to bridge between heritage workloads running on-prem or in hybrid VM environments, as you incrementally modernize your cloud architecture. Patching isn’t generally needed with containers because you just build and publish a new version, and roll out upgrades to it, and there’s a lot to like about that. But there’s no doubt, stateful servers are a reality that will be with us for the foreseeable future, and hopefully this article has given you some ideas how to incrementally modernize one step at a time.

The full code for this post is available on GitHub in all languages:

C#

Go

Python

TypeScript

YAML

Five Years of Infrastructure as Code

Joe Duffy — Tue, 03 May 2022 10:16:25 -0700

Unbelievably, we just celebrated Pulumi’s 5th birthday. To commemorate, we are publishing multi-part series on all things infrastructure as code, starting with why we’re so excited about it:

All Roads Lead Back to Infrastructure as Code
Standing on the Shoulders of Giants
Infrastructure as Code Internals
Bringing Applications and Infrastructure Closer Together
Ringing in the Cloud Engineering Era
(Bonus) The Future of Infrastructure as Code

We published the first post today, and the others will soon follow in the weeks to come. The above hyperlinks will light up as they go live. We have had so much fun creating Pulumi and sharing it with you, and are just as excited about what’s in store during the next five years too!

Announcing the Pulumi REST API

Joe Duffy — Mon, 20 Sep 2021 00:00:00 +0000

Pulumi was designed to be highly extensible from the outset. That includes core languages and cloud providers, of course, but our community is often using Pulumi as a central part of building and connecting their cloud engineering and automation systems, using features like the Automation API. Today we are happy to announce the next major step in enabling these kind of scenarios: the Pulumi REST API. This REST API offers functionality to manage projects and stacks, cloud resources, policies, and more. It has, in fact, been there all along, powering the Pulumi SDK, CLI, and Console behind the scenes, although we haven’t fully documented or supported it until now. That changes today! We’ve already seen some amazing things built with this API and we’re excited to see what you build with it too.

Overview

Pulumi provides a lot of functionality out of the box. But we understand that modern cloud engineering systems are highly interconnected and complex, and that many teams have unique needs. Therefore, we set out to design the Pulumi platform to be extensible in many ways. The Pulumi client (SDK, CLI, etc) was designed to interoperate with the Pulumi server and, though the SDK has a specific set of needs, this protocol has always been designed to be public and easy to consume for custom scenarios and integrations.

Manage Stacks, Updates, Teams, and More

The Pulumi REST API is available at https://api.pulumi.com and includes functionality to interact with and manipulate any kind of metadata managed by Pulumi. That includes:

Projects and Stacks
- List all projects and stacks for a user or organization
- Get metadata about a specific stack, including stack config and resource states
- Transfer a stack between a user and/or organization
- Delete a stack
- Manage stack tags
Previews and Updates
- List all stack previews and updates
- Get the status of an ongoing or past stack preview or update
- List the detailed events occurring during a stack’s update, such as the resource changes, policy checks, or stdout/stderr messages
Organizations
- List or manage users of an organization
- Manage RBAC teams and user roles within an organization
- List or manage access tokens within an organization
Audit Logs
- Get audit log events as JSON
- Export audit log events as CSV or CEF

Authentication and authorization is governed by the Pulumi identity and RBAC systems. As we add functionality to the Pulumi SDK and Service, we are committed to shipping and supporting the underlying REST APIs. If it’s used in our web console or CLI, we want it to be available to so you can add custom features and functionality.

Read complete details about the REST API or keep reading for examples and scenarios.

The REST API for Cloud Engineering

We have already seen Cloud Engineering teams using the Pulumi REST API to build all sorts of custom functionality. These new capabilities are especially powerful when used in combination with the Automation API, which lets you use infrastructure as code as a library embedded into a larger system, and Webhooks, which lets you invoke your own REST API in response to a Pulumi event such as a stack update.

Examples of scenarios we’ve seen include:

Custom dashboards, including “click to deploy”.
Programmatically getting stack outputs to integrate with other automation needs.
Making more metadata-driven Automation API projects that consume project, stack, config, resource, or other service state to make provisioning decisions.
Exposing a full set of projects and stacks into other systems, such as a portal for managed service providers.
Querying resource metadata to find stacks containing resources with certain properties (such as, for example, in response to a security incident).
Managing teams, users, and access tokens for specific automation needs.
Pulling audit logs into other systems (e.g., SIEM).
Enforcing certain policies such as ensuring that stack tags are applied consistently.
Performing automated resource drift and remediation.

With the Pulumi cloud engineering platform, if you can dream it, you can build it!

Example: Finding Stacks with RDS Databases

Let’s take a look at the Pulumi REST API in action. This example will demonstrate querying stacks and resource metadata across an entire organization.

Imagine that there is suddenly a MySQL vulnerability or EOL notice that we need to respond to. We use Amazon RDS and would like to query our entire organization to find projects and stacks that currently have RDS databases deployed using version 5.6 of MySQL. Thankfully the Pulumi REST API makes all metadata about all of my cloud projects and stacks — across all clouds — available for easy querying and inspection. So a dozen lines of bash will do the trick!

The first step is to generate an API access token. The Pulumi Service lets you manage many access tokens (the same as what’s used for pulumi login) and it’s easiest to just create one specific to this project’s purpose. This will let us revoke that token after we’re done and know that there is no lingering access.

To do that, go to https://app.pulumi.com/account/tokens (or click your avatar in the upper right and go to Settings, where you’ll see Access Tokens on the left-hand nav):

Click the Create Token button. That will pop a box that lets us enter a description:

This then gives us the access token text. We need to copy it and put it somewhere for safekeeping, because once we leave this page, we won’t be able to recover it:

We will use this token for all API calls and the token dictates what authorization we will have to perform actions within a given user or organization account. As we will see, all APIs pertain to given permission scopes, much like GitHub, which will determine the permissions granted. If we’re using curl, for example, we will pass the "Authorization: token XYZ" header, with -H, as described in the Authorization section of the REST API documentation.

If we forget to do that, we’ll receive a 4xx error. For instance, 401:

{"code":401,"message":"Unauthorized: No credentials provided or are invalid."}

To test that this is working, let’s curl the /api/user endpoint, which gives us information about the currently “logged in” user (i.e., the user to which the token we’re using belongs):

$ curl \
 -H "Authorization: token pul-fa..REDACTED..fa" \
 https://api.pulumi.com/api/user
{"githubLogin":"joeduffy","name":"Joe Duffy","email":"joe@pulumi.com",...}

OK, looks like everything’s working! Let’s get back to our scenario of finding unpatched RDS instances. To do that, well use a combination of bash, curl, and jq (payloads are JSON).

Here is the full script. We set up the auth header as noted above, enumerate all stacks in the acmecorp organization, export the stack’s metadata including its resources, and then check to see if there are any RDS instances with "mysql" version "5.7" that we need to patch:

# Set up our auth token info.
export AUTH_HEAD="Authorization: token $PAPI_TOKEN"
export ORG_NAME=acmecorp

# Enumerate all stacks in our organization.
for stack in $(
 curl -s -H "$AUTH_HEAD" \
 https://api.pulumi.com/api/user/stacks?organization=$ORG_NAME |
 jq -c '.stacks[]'); do
 ORG=$(echo $stack | jq -r '.orgName')
 PROJ=$(echo $stack | jq -r '.projectName')
 STACK=$(echo $stack | jq -r '.stackName')

 # Query each stack to see if we have RDS instances.
 RDS_INSTANCES=$(curl -s -H "$AUTH_HEAD" \
 https://api.pulumi.com/api/stacks/$ORG/$PROJ/$STACK/export |
 jq -c '.deployment?.resources[]? | select (.type == "aws:rds/instance:Instance")')
 if [[ ! -z $RDS_INSTANCES ]]; then
 for rds in $RDS_INSTANCES; do
 # For each RDS instance, warn on any instance that is 'mysql'
 # version '5.7', and error out so we can patch our instances!
 URN=$(echo $rds | jq -r '.urn')
 ENGINE=$(echo $rds | jq -r '.engine')
 ENGINE_VERSION=$(echo $rds | jq -r '.engineVersion')
 if [[ "$ENGINE"="mysql" && "$ENGINE_VERSION"="5.7" ]]; then
 >&2 echo -e "error: vulnerable MySQL 5.7 RDS instance found:"
 >&2 echo -e " URN=$URN"
 >&2 echo -e " ORG/PROJ/STACK=$ORG/$PROJ/$STACK"
 >&2 echo -e " DETAILS=$rds"
 fi
 done
 fi
done

Now all we need to do is generate a token, set our PAPI_TOKEN environment variable to contain it, and run this script — voila, all of our vulnerable RDS databases will be revealed. Now we can go update those stacks, perhaps programmatically with the Automation API, and then possibly even write a Policy as Code rule to prohibit new 5.6 instances from getting created in the future.

Give It a Try Today

As we have seen, the Pulumi REST API gives us easy access to projects, stacks, cloud resources, teams, and more. It is now generally available to all users of the Pulumi Service, both free and paying customers. To give it a try today, sign up for your free account now, give our Getting Started guide a try, and then check out the Pulumi REST API documentation.

We can’t wait to see the powerful automation you build for your team using the Pulumi REST API! Have fun spelunking in the cloud.

Build and publish container images to any cloud with Infrastructure as Code

Joe Duffy — Tue, 08 Dec 2020 00:00:00 +0000

Going from a containerized application to a service running in the cloud requires a few steps beyond an application’s normal build-and-test cycle. Namely, it means building and publishing a container image in a registry and then consuming that image from your target environment, whether that’s Kubernetes, Amazon ECS, or another container orchestrator. It’s not enough to just write a Dockerfile — you will need to pick a container registry, decide whether that registry should be public or private, authenticate against it, and ideally automate deploying subsequent updates. Infrastructure as code to the rescue! In this article, we’ll see how to build, publish, and consume a simple container image across any cloud, using just a few lines of code.

Approach

The general approach will be to create a new infrastructure as code project that

Prepares a container registry, either public or private
Builds and publishes your container image to that registry
Optionally, consumes the resulting image URL from a containerized task definition in Kubernetes, Amazon ECS, or any other container orchestrator

All told, this will be just a few dozen lines of code. This article demonstrates doing this in AWS, Azure, Google Cloud, DigitalOcean, and Docker Hub, and offers code examples in each supported Pulumi language, namely Python, JavaScript, TypeScript, Go, and C#. These steps will work for any application that has a Dockerfile and is buildable by Docker. In principle, similar steps could be applied if you prefer to build your container image using different means, such as Buildpack.

For purposes of illustration, we’ll create a simple Nginx web server whose Dockerfile contains:

FROM nginx
RUN echo "<h1>Hello, World!</h1>" > \
 /usr/share/nginx/html/index.html

Now, let’s dive in!

Prepare a Container Registry

The first step is to prepare a new container registry. A registry holds one or more repositories, each of which can store and serve many different container images with different tags and versions. Afterward we’ll show how to build and publish to this registry.

The specific details of how to prepare your registry differ by cloud provider, often significantly, particularly when it comes to authenticating. Pick your cloud provider to see the details:

Amazon Elastic Container Registry (ECR)

Amazon Elastic Container Registry (ECR) provides managed Docker container hosting that makes it easy to run containerized applications in your AWS account using Amazon Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS). Each account has a default registry per region, and each registry may have any number of repositories, each for a different Docker image. Each repository can store many versions of that particular image.

Create a New Project

To start, create a new project and ensure it is configured to use your AWS account, and then scaffold your project with the imports and overall program structure that we will fill in one piece at a time:

var aws = require("@pulumi/aws");
var docker = require("@pulumi/docker");

// [Placeholder 1: Create a private ECR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import * as aws from "@pulumi/aws";
import * as docker from "@pulumi/docker";

// [Placeholder 1: Create a private ECR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import base64
import pulumi
import pulumi_aws as aws
import pulumi_docker as docker

# [Placeholder 1: Create a private ECR registry.]

# [Placeholder 2: Get registry info (creds and endpoint).]

# [Placeholder 3: Build and publish the container image.]

package main

import (
 "encoding/base64"
 "errors"
 "strings"

 "github.com/pulumi/pulumi-aws/sdk/v2/go/aws/ecr"
 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Create a private ECR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Aws.Ecr;
using Pulumi.Docker;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Create a private ECR repository.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 }
}

Provision an ECR Repository

Next, declare a new ECR repository resource:

// Create a private ECR repository.
var repo = new aws.ecr.Repository("my-repo");

// Create a private ECR repository.
const repo = new aws.ecr.Repository("my-repo");

# Create a private ECR repository.
repo = aws.ecr.Repository('my-repo')

// Create a private ECR repository.
repo, err := ecr.NewRepository(ctx, "my-repo", nil)
if err != nil {
 return err
}

// Create a private ECR repository.
var repo = new Repository("my-repo");

Authenticate with Temporary ECR Access Token

Next, we will need to generate authentication information to access the repository to build and publish our image. ECR supports this in multiple different ways; however, here, we will demonstrate generating a temporary access token:

// Get registry info (creds and endpoint).
var imageName = repo.repositoryUrl;
var registryInfo = repo.registryId.apply(id => {
 return aws.ecr.getCredentials({ registryId: id }).then(credentials => {
 var decodedCredentials = Buffer.from(credentials.authorizationToken, "base64").toString();
 var [username, password] = decodedCredentials.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: credentials.proxyEndpoint,
 username: username,
 password: password,
 };
 });
});

// Get registry info (creds and endpoint).
const imageName = repo.repositoryUrl;
const registryInfo = repo.registryId.apply(async id => {
 const credentials = await aws.ecr.getCredentials({ registryId: id });
 const decodedCredentials = Buffer.from(credentials.authorizationToken, "base64").toString();
 const [username, password] = decodedCredentials.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: credentials.proxyEndpoint,
 username: username,
 password: password,
 };
});

# Get registry info (creds and endpoint).
def getRegistryInfo(rid):
 creds = aws.ecr.get_credentials(registry_id=rid)
 decoded = base64.b64decode(creds.authorization_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception("Invalid credentials")
 return docker.ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])
image_name = repo.repository_url
registry_info = repo.registry_id.apply(getRegistryInfo)

// Get registry info (creds and endpoint).
imageName := repo.RepositoryUrl
registryInfo := repo.RegistryId.ApplyT(func(id string) (docker.ImageRegistry, error) {
 creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{RegistryId: id})
 if err != nil {
 return docker.ImageRegistry{}, err
 }
 decoded, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
 if err != nil {
 return docker.ImageRegistry{}, err
 }
 parts := strings.Split(string(decoded), ":")
 if len(parts) != 2 {
 return docker.ImageRegistry{}, errors.New("Invalid credentials")
 }
 return docker.ImageRegistry{
 Server: creds.ProxyEndpoint,
 Username: parts[0],
 Password: parts[1],
 }, nil
}).(docker.ImageRegistryOutput)

// Get registry info (creds and endpoint).
var imageName = repo.RepositoryUrl;
var registryInfo = repo.RegistryId.Apply(async (id) =>
{
 var creds = await GetCredentials.InvokeAsync(new GetCredentialsArgs { RegistryId = id });
 var decodedData = Convert.FromBase64String(creds.AuthorizationToken);
 var decoded = ASCIIEncoding.ASCII.GetString(decodedData);

 var parts = decoded.Split(':');
 if (parts.Length != 2)
 {
 throw new Exception("Invalid credentials");
 }

 return new ImageRegistry
 {
 Server = creds.ProxyEndpoint,
 Username = parts[0],
 Password = parts[1],
 };
});

Alternatively, Authenticate with ECR Credential Helper

An alternative approach ECR supports is to use the Docker credential helper from Amazon, which integrates with local IAM settings, adds smart caching, and removes the need for Docker to login. If you elect to go this route, leave out the username/password parts of the docker.ImageRegistry:

// Get registry info (creds and endpoint).
var imageName = repo.repositoryUrl;
var registryInfo = undefined; // use ECR credentials helper.

// Get registry info (creds and endpoint).
const imageName = repo.repositoryUrl;
const registryInfo = undefined; // use ECR credentials helper.

# Get registry info (creds and endpoint).
image_name = repo.repository_url
registry_info = None # use ECR credentials helper.

// Get registry info (creds and endpoint).
imageName := repo.RepositoryUrl
registryInfo := docker.ImageRegistryArgs{} // use ECR credentials helper.

// Get registry info (creds and endpoint).
var imageName = repo.RepositoryUrl;
// Omit registryInfo and use ECR credentials helper.

Learn More About ECR

ECR offers many additional options not shown here. This includes configuring advanced IAM permissions, enabling image vulnerability scanning, managing the lifetime of images so that older, unused images are deleted based on configured policies, encrypting images, and more. For details on these advanced capabilities and more, refer to Pulumi’s ECR user guide, Pulumi’s ECR API reference, or Amazon’s product documentation.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

Azure Container Registry (ACR)

Azure Container Registry (ACR) allows you to build, store, secure, scan, replicate, and manage container images and artifacts using a fully managed and geo-replicated instance of OCI distribution. This makes it easy to deploy containerized workloads to native Azure services like Azure Kubernetes Service (AKS), Red Hat OpenShift, and App Service. Each registry is capable of storing and serving multiple versions of multiple container images.

Create a New Project

To start, create a new project and ensure it is configured to use your Azure account, and then scaffold your project with the imports and overall program structure that we will fill in one piece at a time:

var azure = require("@pulumi/azure");
var docker = require("@pulumi/docker");

// [Placeholder 1: Create a private ACR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import * as azure from "@pulumi/azure";
import * as docker from "@pulumi/docker";

// [Placeholder 1: Create a private ACR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import pulumi
import pulumi_azure as azure
import pulumi_docker as docker

# [Placeholder 1: Create a private ACR registry.]

# [Placeholder 2: Get registry info (creds and endpoint).]

# [Placeholder 3: Build and publish the container image.]

package main

import (
 "github.com/pulumi/pulumi-azure/sdk/v2/go/azure/containerservice"
 "github.com/pulumi/pulumi-azure/sdk/v2/go/azure/core"
 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Create a private ECR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Azure.Core;
using Pulumi.Azure.ContainerService;
using Pulumi.Docker;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Create a private ECR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 }
}

Provision a New ACR Registry

Next, declare a new ACR registry resource. To create a new registry, we need an Azure resource group; here we will create a new one, but feel free to use an existing one by passing its name in the resource group’s place:

// Create a private ACR registry.
var rg = new azure.core.ResourceGroup("myrg");
var registry = new azure.containerservice.Registry("myregistry", {
 resourceGroupName: rg.name,
 adminEnabled: true,
 sku: "Basic",
});

// Create a private ACR registry.
const rg = new azure.core.ResourceGroup("myrg");
const registry = new azure.containerservice.Registry("myregistry", {
 resourceGroupName: rg.name,
 adminEnabled: true,
 sku: "Basic",
});

# Create a private ACR registry.
rg = azure.core.ResourceGroup('myrg')
registry = azure.containerservice.Registry('myregistry',
 resource_group_name=rg.name,
 admin_enabled=True,
 sku='Basic'
)

// Create a private ACR registry.
rg, err := core.NewResourceGroup(ctx, "myrg", nil)
if err != nil {
 return err
}
registry, err := containerservice.NewRegistry(ctx, "my-repo", &containerservice.RegistryArgs{
 ResourceGroupName: rg.Name,
 AdminEnabled: pulumi.Bool(true),
 Sku: pulumi.String("Basic"),
})
if err != nil {
 return err
}

// Create a private ACR registry.
var rg = new ResourceGroup("myrg");
var registry = new Registry("myregistry", new RegistryArgs
{
 ResourceGroupName = rg.Name,
 AdminEnabled = true,
 Sku = "basic"
});

Authenticate with Admin Account

Now we need to gather up the image name and registry authentication information, which has been auto-generated by Azure, in preparation for building and publishing to it. For this simple example, we have enabled admin access, which we will use to authenticate:

// Get registry info (creds and endpoint).
var imageName = registry.loginServer.apply(s => `${s}/myapp`);
var registryInfo = {
 server: registry.loginServer,
 username: registry.adminUsername,
 password: registry.adminPassword,
};

// Get registry info (creds and endpoint).
const imageName = registry.loginServer.apply(s => `${s}/myapp`);
const registryInfo = {
 server: registry.loginServer,
 username: registry.adminUsername,
 password: registry.adminPassword,
};

# Get registry info (creds and endpoint).
image_name = registry.login_server.apply(lambda s: f'{s}/myapp')
registry_info = docker.ImageRegistry(
 server=registry.login_server,
 username=registry.admin_username,
 password=registry.admin_password
)

// Get registry info (creds and endpoint).
imageName := pulumi.Sprintf("%s/myapp", registry.LoginServer)
registryInfo := docker.ImageRegistryArgs{
 Server: registry.LoginServer,
 Username: registry.AdminUsername,
 Password: registry.AdminPassword,
}

// Get registry info (creds and endpoint).
var imageName = Output.Format($"{registry.LoginServer}/myapp");
var registryInfo = new ImageRegistry
{
 Server = registry.LoginServer,
 Username = registry.AdminUsername,
 Password = registry.AdminPassword,
};

Alternatively, Authenticate with ActiveDirectory Service Principal

Although enabling admin access is “easy”, it isn’t the most secure approach. An alternative is to set “admin enabled” to false when declaring our registry, provision or assign an ActiveDirectory service principal with an acrpush role scoped to our registry, and use that for authentication instead:

// New imports:
var azuread = require("@pulumi/azuread");
var random = require("@pulumi/random");

// [As before...]

// Get registry info (creds and endpoint).
var sp = new azuread.ServicePrincipal("mysp", {
 applicationId: new azuread.Application("myspapp").applicationId,
});
var spPassword = new azuread.ServicePrincipalPassword("mysp-pass", {
 servicePrincipalId: sp.id,
 value: new random.RandomPassword("mypass", {
 length: 32,
 }, { additionalSecretOutputs: [ "result" ] }).result,
 endDateRelative: "8760h",
});
var spAuth = new azure.authorization.Assignment("myauth", {
 scope: registry.id,
 roleDefinitionName: "acrpush",
 principalId: sp.id,
});
var registryInfo = {
 server: registry.loginServer,
 username: sp.applicationId,
 password: spAuth.id.apply(_ => spPassword.value),
};

// New imports:
import * as azuread from "@pulumi/azuread";
import * as random from "@pulumi/random";

// [As before...]

// Get registry info (creds and endpoint).
const sp = new azuread.ServicePrincipal("mysp", {
 applicationId: new azuread.Application("myspapp").applicationId,
});
const spPassword = new azuread.ServicePrincipalPassword("mysp-pass", {
 servicePrincipalId: sp.id,
 value: new random.RandomPassword("mypass", {
 length: 32,
 }, { additionalSecretOutputs: [ "result" ] }).result,
 endDateRelative: "8760h",
});
const spAuth = new azure.authorization.Assignment("myauth", {
 scope: registry.id,
 roleDefinitionName: "acrpush",
 principalId: sp.id,
});
const registryInfo = {
 server: registry.loginServer,
 username: sp.applicationId,
 password: spAuth.id.apply(_ => spPassword.value),
};

# New imports:
import pulumi_azuread as azuread
import pulumi_random as random

# [As before...]

# Get registry info (creds and endpoint).
image_name = registry.login_server.apply(lambda s: f'{s}/myapp')
sp = azuread.ServicePrincipal('mysp',
 application_id=azuread.Application('myspapp').application_id,
)
sp_password = azuread.ServicePrincipalPassword('mysp-pass',
 service_principal_id=sp.id,
 value=random.RandomPassword('mypass',
 length=32,
 opts=pulumi.ResourceOptions(additional_secret_outputs=['result'])
 ).result,
 end_date_relative='8760h',
)
sp_auth = azure.authorization.Assignment('myauth',
 scope=registry.id,
 role_definition_name='acrpush',
 principal_id=sp.id,
)
registry_info = docker.ImageRegistry(
 server=registry.login_server,
 username=sp.application_id,
 password=sp_auth.id.apply(lambda _: sp_password.value),
)

// New imports:
import (
 "github.com/pulumi/pulumi-azure/sdk/v2/go/azure/authorization"
 "github.com/pulumi/pulumi-azuread/sdk/v2/go/azuread"
 "github.com/pulumi/pulumi-random/sdk/v2/go/random"
)

// [As before...]

// Get registry info (creds and endpoint).
imageName := pulumi.Sprintf("%s/myapp", registry.LoginServer)
spApp, err := azuread.NewApplication(ctx, "myspapp", nil)
if err != nil {
 return err
}
sp, err := azuread.NewServicePrincipal(ctx, "mysp", &azuread.ServicePrincipalArgs{
 ApplicationId: spApp.ApplicationId,
})
if err != nil {
 return err
}
password, err := random.NewRandomPassword(ctx, "mypass",
 &random.RandomPasswordArgs{
 Length: pulumi.Int32(32),
 },
 pulumi.AdditionalSecretOutputs([]string{"result"}),
)
if err != nil {
 return err
}
spPassword, err := azuread.NewServicePrincipalPassword(ctx, "mysp-pass", &azuread.ServicePrincipalPasswordArgs{
 ServicePrincipalId: sp.ID(),
 Value: password.Result,
 EndDateRelative: pulumi.String("8760h"),
})
if err != nil {
 return err
}
apAuth, err := authorization.NewAssignment(ctx, "myauth", &authorization.AssignmentArgs{
 Scope: registry.ID(),
 RoleDefinitionName: pulumi.String("acrpush"),
 PrincipalId: sp.ID(),
})
if err != nil {
 return err
}
registryInfo := docker.ImageRegistryArgs{
 Server: registry.LoginServer,
 Username: sp.ApplicationId,
 Password: spPassword.Value,
}

// New imports:
using Pulumi.Azure.Authorization;
using AAD = Pulumi.AzureAD;
using Pulumi.Random;

// [As before...]

// Get registry info (creds and endpoint).
var imageName = Output.Format($"{registry.LoginServer}/myapp");
var sp = new AAD.ServicePrincipal("mysp", new AAD.ServicePrincipalArgs
{
 ApplicationId = new AAD.Application("myspapp").ApplicationId,
});
var spPassword = new AAD.ServicePrincipalPassword("mysp-pass", new AAD.ServicePrincipalPasswordArgs
{
 ServicePrincipalId = sp.Id,
 Value = new RandomPassword("mypass",
 new RandomPasswordArgs
 {
 Length = 32,
 },
 new CustomResourceOptions { AdditionalSecretOutputs = { "result" } }
 ).Result,
 EndDateRelative = "8760h",
});
var spAuth = new Assignment("myauth", new AssignmentArgs
{
 Scope = registry.Id,
 RoleDefinitionName = "acrpush",
 PrincipalId = sp.Id,
});
var registryInfo = new Docker.ImageRegistry
{
 Server = registry.LoginServer,
 Username = sp.ApplicationId,
 Password = spAuth.Id.Apply(_ => spPassword.Value),
};

Learn More About ACR

ACR offers advanced functionality not shown here, including configuring retention policies, storage account details, and geo-replication options. For more details on these and more, see Pulumi’s Azure-Native or Azure API documentation, or Azure’s product page.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

Google Container Registry (GCR)

Google Container Registry (GCR) enables you to store, manage, and secure your Docker container images. It includes built-in security vulnerability scanning and fast, high-availability access by storing images in regional private repositories across the world. This makes it easy to deploy custom application images to Google Kubernetes Engine (GKE), App Engine, or Cloud Run. Each account has a default registry per project per region, backed by Google Cloud Storage (GCS), and each registry is capable of storing many images and many versions.

Create a New Project

To start, create a new project and ensure it is configured to use your Google Cloud account, then scaffold your project with the imports and overall program structure that we will fill in one piece at a time:

var gcp = require("@pulumi/gcp");
var docker = require("@pulumi/docker");

// [Placeholder 1: Create a private GCR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import * as gcp from "@pulumi/gcp";
import * as docker from "@pulumi/docker";

// [Placeholder 1: Create a private GCR registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import pulumi
import pulumi_gcp as gcp
import pulumi_docker as docker

# [Placeholder 1: Create a private GCR registry.]

# [Placeholder 2: Get registry info (creds and endpoint).]

# [Placeholder 3: Build and publish the container image.]

package main

import (
 "encoding/base64"
 "errors"
 "strings"

 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi-gcp/sdk/v2/go/gcp/container"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Create a private GCR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Docker;
using Pulumi.Gcp.Container;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Create a private GCR registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 }
}

Ensure Your GCR Registry is Ready

Google Cloud automatically provisions a managed project-wide registry as needed. To ensure our Google Cloud registry is ready, we allocate a registry object, and then fetch the autogenerated repository URL that we can use for images:

// Create a private GCR registry.
var registry = new gcp.container.Registry("my-registry");
var registryUrl = registry.id.apply(_ =>
 gcp.container.getRegistryRepository().then(reg => reg.repositoryUrl));

// Create a private GCR registry.
const registry = new gcp.container.Registry("my-registry");
const registryUrl = registry.id.apply(_ =>
 gcp.container.getRegistryRepository().then(reg => reg.repositoryUrl));

# Create a private GCR repository.
registry = gcp.container.Registry('my-registry')
registry_url = registry.id.apply(lambda _: gcp.container.get_registry_repository().repository_url)

// Create a private GCR registry.
registry, err := container.NewRegistry(ctx, "my-registry", nil)
if err != nil {
 return err
}
registryUrl := registry.ID().ApplyString(func(_ string) (string, error) {
 rep, err := container.GetRegistryRepository(ctx, nil)
 if err != nil {
 return "", err
 }
 return rep.RepositoryUrl, nil
})

// Create a private GCR registry.
var registry = new Registry("my-registry");
var registryUrl = registry.Id.Apply(async _ => {
 return (await GetRegistryRepository.InvokeAsync()).RepositoryUrl;
});

Below we will append the container repository name to this registry URL.

Authenticate with the Google Cloud SDK

Google Cloud offers several mechanisms to authenticate to your registry; however, the most secure and preferred option is to use the Google Cloud CLI, gcloud, as a Docker credential helper. After setting up gcloud on your client, no explicit authentication is required in your code — instead, the ambient gcloud authentication settings will be used instead. As a result, we can leave the authentication information blank:

// Get registry info (creds and endpoint).
var imageName = registryUrl.apply(url => `${url}/myapp`);
var registryInfo = undefined; // use gcloud for authentication.

// Get registry info (creds and endpoint).
const imageName = registryUrl.apply(url => `${url}/myapp`);
const registryInfo = undefined; // use gcloud for authentication.

# Get registry info (creds and endpoint).
image_name = registry_url.apply(lambda url: f'{url}/myapp')
registry_info = None # use gcloud for authentication.

// Get registry info (creds and endpoint).
imageName := pulumi.Sprintf("%s/myapp", registryUrl)
registryInfo := docker.ImageRegistryArgs{} // use gcloud for authentication.

// Get registry info (creds and endpoint).
var imageName = Output.Format($"{registryUrl}/myapp");
// Omit registryInfo and use gcloud for authentication.

Learn More About GCR

GCR uses Google Cloud Storage to store images, which may be configured separately, including configuring IAM and lifecycle policies. For more details on this, please refer to Pulumi’s Google Cloud API or Google’s own Container Registry documentation.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

DigitalOcean Container Registry

DigitalOcean’s Container Registry is an easy way to store and manage private container images for your applications for use with DigitalOcean’s managed Kubernetes service.

Create a New Project

To start, create a new project and ensure it is configured to use your DigitalOcean account and then scaffold your project with the imports and overall program structure that we will fill in one piece at a time:

var digitalocean = require("@pulumi/digitalocean");
var docker = require("@pulumi/docker");
var pulumi = require("@pulumi/pulumi");

// [Placeholder 1: Create a private DigitalOcean container registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import * as digitalocean from "@pulumi/digitalocean";
import * as docker from "@pulumi/docker";
import * as pulumi from "@pulumi/pulumi";

// [Placeholder 1: Create a private DigitalOcean container registry.]

// [Placeholder 2: Get registry info (creds and endpoint).]

// [Placeholder 3: Build and publish the container image.]

import pulumi
import pulumi_digitalocean as digitalocean
import pulumi_docker as docker

# [Placeholder 1: Create a private DigitalOcean container registry.]

# [Placeholder 2: Get registry info (creds and endpoint).]

# [Placeholder 3: Build and publish the container image.]

package main

import (
 "encoding/base64"
 "encoding/json"
 "errors"
 "strings"

 "github.com/pulumi/pulumi-digitalocean/sdk/v2/go/digitalocean"
 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Create a private DigitalOcean container registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.DigitalOcean;
using Pulumi.Docker;
using Newtonsoft.Json;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Create a private DigitalOcean container registry.]

 // [Placeholder 2: Get registry info (creds and endpoint).]

 // [Placeholder 3: Build and publish the container image.]
 }
}

Create a DigitalOcean Container Registry

Next, declare a new DigitalOcean container registry resource:

// Create a private DigitalOcean Container Registry.
var registry = new digitalocean.ContainerRegistry("my-reg", {
 subscriptionTierSlug: "starter",
});

// Create a private DigitalOcean Container Registry.
const registry = new digitalocean.ContainerRegistry("my-reg", {
 subscriptionTierSlug: "starter",
});

# Create a private DigitalOcean Container Registry.
repo = digitalocean.ContainerRegistry('my-reg',
 subscription_tier_slug='starter',
)

// Create a private DigitalOcean Container Registry.
registry, err := digitalocean.NewContainerRegistry(ctx, "my-reg",
 &digitalocean.ContainerRegistryArgs{
 // SubscriptionTierSlug: pulumi.String("starter"),
 })
if err != nil {
 return err
}

// Create a private DigitalOcean Container Registry.
var registry = new ContainerRegistry("my-reg", new ContainerRegistryArgs
{
 SubscriptionTierSlug = "starter"
});

Authenticate with Temporary Credentials

DigitalOcean supports generating temporary read or read/write credentials that Docker can use to authenticate with your new private container registry, which we’ll now use. This resource returns a new credentials file similar to what the docker login command generates, and we can parse it and base64 decode its contents to discover the temporary username and password. Here, we gather the username and password plus the registry’s URL endpoint in preparation for building and publishing the container image to the newly provisioned registry:

// Get registry info (creds and endpoint) so we can build/publish to it.
var imageName = registry.endpoint.apply(s => `${s}/myapp`);
var creds = new digitalocean.ContainerRegistryDockerCredentials("my-reg-creds", {
 registryName: registry.name,
 write: true,
});
var registryInfo = pulumi.all(
 [creds.dockerCredentials, registry.serverUrl]
).apply(([authJson, serverUrl]) => {
 // We are given a Docker creds file; parse it to find the temp username/password.
 var auths = JSON.parse(authJson);
 var authToken = auths["auths"][serverUrl]["auth"];
 var decoded = Buffer.from(authToken, "base64").toString();
 var [username, password] = decoded.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: serverUrl,
 username: username,
 password: password,
 };
});

// Get registry info (creds and endpoint) so we can build/publish to it.
const imageName = registry.endpoint.apply(s => `${s}/myapp`);
const creds = new digitalocean.ContainerRegistryDockerCredentials("my-reg-creds", {
 registryName: registry.name,
 write: true,
});
const registryInfo = pulumi.all(
 [creds.dockerCredentials, registry.serverUrl]
).apply(([authJson, serverUrl]) => {
 // We are given a Docker creds file; parse it to find the temp username/password.
 const auths = JSON.parse(authJson);
 const authToken = auths["auths"][serverUrl]["auth"];
 const decoded = Buffer.from(authToken, "base64").toString();
 const [username, password] = decoded.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: serverUrl,
 username: username,
 password: password,
 };
});

# Get registry info (creds and endpoint).
image_name = registry.endpoint.apply(lambda s: f'{s}/myapp')
def getRegistryInfo(info):
 # We are given a Docker creds file; parse it to find the temp username/password.
 auth_json = info[0]
 auths = json.loads(auth_json)
 server_url = info[1]
 auth_token = auths['auths'][server_url]['auth']
 decoded = base64.b64decode(auth_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception('Invalid credentials')
 return docker.ImageRegistry(server_url, parts[0], parts[1])
creds = digitalocean.ContainerRegistryDockerCredentials('reg-creds',
 registry_name=registry.name,
 write=True,
)
registry_info = pulumi.Output.all(
 creds.docker_credentials, registry.server_url).apply(getRegistryInfo)

// Get registry info (creds and endpoint).
imageName := pulumi.Sprintf("%s/myapp", registry.Endpoint)
creds, err := digitalocean.NewContainerRegistryDockerCredentials(ctx, "my-reg-creds",
 &digitalocean.ContainerRegistryDockerCredentialsArgs{
 RegistryName: registry.Name,
 Write: pulumi.Bool(true),
 },
)
if err != nil {
 return err
}

registryInfo := pulumi.All(creds.DockerCredentials, registry.ServerUrl).ApplyT(
 func(args []interface{}) (docker.ImageRegistry, error) {
 // We are given a Docker creds file; parse it to find the temp username/password.
 authJson := args[0].(string)
 serverUrl := args[1].(string)
 var auths map[string]interface{}
 if err := json.Unmarshal([]byte(authJson), &auths); err != nil {
 return docker.ImageRegistry{}, err
 }
 authMap := auths["auths"].(map[string]interface{})
 authToken := authMap[serverUrl].(map[string]interface{})["auth"].(string)
 decoded, err := base64.StdEncoding.DecodeString(authToken)
 if err != nil {
 return docker.ImageRegistry{}, err
 }
 parts := strings.Split(string(decoded), ":")
 if len(parts) != 2 {
 return docker.ImageRegistry{}, errors.New("Invalid credentials")
 }
 return docker.ImageRegistry{
 Server: serverUrl,
 Username: parts[0],
 Password: parts[1],
 }, nil
 },
).(docker.ImageRegistryOutput)

// Get registry info (creds and endpoint).
var imageName = Output.Format($"{registry.Endpoint}/myapp");
var registryCreds = new ContainerRegistryDockerCredentials("my-reg-creds",
 new ContainerRegistryDockerCredentialsArgs
 {
 RegistryName = registry.Name,
 Write = true,
 });
var registryInfo = Output.All(
 registryCreds.DockerCredentials, registry.ServerUrl).
 Apply(args =>
 {
 var authJson = args[0];
 var serverUrl = args[1];
 dynamic auths = JsonConvert.DeserializeObject(authJson);
 var authToken = auths["auths"][serverUrl]["auth"];
 var decoded = ASCIIEncoding.ASCII.GetString(authToken);

 var parts = decoded.Split(':');
 if (parts.Length != 2)
 {
 throw new Exception("Invalid credentials");
 }

 return new ImageRegistry
 {
 Server = serverUrl,
 Username = parts[0],
 Password = parts[1],
 };
 });

Learn More About DigitalOcean Container Registry

Each DigitalOcean account may have just a single registry, so you are likely to want to provision that in a separate stack for most “real world” examples. Also, this project uses the lowest subscription tier, "starter", which is inexpensive but quite limited (you may only store a single repository of images). If you would like to use an advanced tier or configure any other options, please refer to Pulumi’s DigitalOcean API or DigitalOcean’s product documentation.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

Docker Hub

Docker offers the Docker Hub as an easy way to store and consume public or private container images from a centralized location to any cloud. To start building and publishing to Docker Hub with infrastructure as code, create a new project and scaffold it with the imports and overall program structure that we will fill in one piece at a time:

var docker = require("@pulumi/docker");
var pulumi = require("@pulumi/pulumi");

// [Placeholder 1: Get registry info (creds and endpoint).]

// [Placeholder 2: Build and publish the container image.]

import * as docker from "@pulumi/docker";
import * as pulumi from "@pulumi/pulumi";

// [Placeholder 1: Get registry info (creds and endpoint).]

// [Placeholder 2: Build and publish the container image.]

import pulumi
import pulumi_docker as docker

# [Placeholder 1: Get registry info (creds and endpoint).]

# [Placeholder 2: Build and publish the container image.]

package main

import (
 "encoding/base64"
 "encoding/json"
 "errors"
 "strings"

 "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Placeholder 1: Get registry info (creds and endpoint).]

 // [Placeholder 2: Build and publish the container image.]
 return nil
 })
}

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Docker;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Placeholder 1: Get registry info (creds and endpoint).]

 // [Placeholder 2: Build and publish the container image.]
 }
}

For purposes of this example, you will need to manually provision a repository, and select whether to make it public or private, via the Docker Hub’s UI. Refer to the Docker Hub documentation for details on how to do this. Once you have an account and have set up your registry, create an access token. We will now use your username and this access token to configure authentication so that you can build and publish to your registry.

Because the repository was provisioned outside of the purview of Pulumi, we can jump straight to authenticating. To do this, we will simply use the Pulumi configuration system to store and retrieve authentication information, using the token created above:

// Fetch the Docker Hub auth info from config.
var config = new pulumi.Config();
var username = config.require("dockerUsername");
var password = config.requireSecret("dockerPassword");

// Populate the registry info (creds and endpoint).
var imageName = `${username}/myapp`;
var registryInfo = {
 server: "docker.io",
 username: username,
 password: password,
};

// Fetch the Docker Hub auth info from config.
const config = new pulumi.Config();
const username = config.require("dockerUsername");
const password = config.requireSecret("dockerPassword");

// Populate the registry info (creds and endpoint).
const imageName = `${username}/myapp`;
const registryInfo = {
 server: "docker.io",
 username: username,
 password: password,
};

# Fetch the Docker Hub auth info from config.
config = pulumi.Config()
username = config.require('dockerUsername')
accessToken = config.require_secret('dockerAccessToken')

# Populate the registry info (creds and endpoint).
image_name=f'{username}/myapp',
def get_registry_info(token):
 return docker.ImageRegistry(
 server='docker.io',
 username=username,
 password=token,
 )
registry_info=accessToken.apply(get_registry_info)

// Fetch the Docker Hub auth info from config.
username := config.Require(ctx, "dockerUsername")
password := config.RequireSecret(ctx, "dockerPassword")

// Populate the registry info (creds and endpoint).
imageName := pulumi.String(username + "/myapp")
registryInfo := docker.ImageRegistryArgs{
 Server: pulumi.String("docker.io"),
 Username: pulumi.String(username),
 Password: password.(pulumi.StringOutput),
}

// Fetch the Docker Hub auth info from config.
var config = new Pulumi.Config();
var username = config.Require("dockerUsername");
var password = config.RequireSecret("dockerPassword");

// Populate the registry info (creds and endpoint).
var imageName = $"{username}/myapp";
var registryInfo = new ImageRegistry
{
 Server = "docker.io",
 Username = username,
 Password = password,
};

After writing this code, add your Docker username and password to your Pulumi project:

$ pulumi config set dockerUsername [YOUR USERNAME]
$ pulumi config set --secret dockerAccessToken [YOUR TOKEN]

There are of course other ways to configure this authentication information, however, the Pulumi secrets system ensures that your token is encrypted and safe to use.

To view another cloud provider’s registry details, select a new cloud in the switcher above.

At this stage, we can run pulumi up to check that the program works and to provision any necessary cloud resources.

Build and Publish Your Container

Now we are ready to build and publish your container image to the chosen registry. The Docker provider’s Image component internally uses the Docker engine to perform the necessary steps to carry this out, including building, tagging, capturing, and streaming container build logs as progress is made and pushing the final result.

Simply pass the path to your application’s Dockerfile as the build context, the registry’s URL as the image’s name, and the registry configuration object built up earlier to facilitate authentication:

// [Existing imports ...]
var docker = require("@pulumi/docker");

// [Registry configuration as shown above ...]

// Build and publish the container image.
var image = new docker.Image("my-image", {
 build: "app",
 imageName: imageName,
 registry: registry,
});

// Export the base and specific version image name.
module.exports = {
 baseImageName: image.baseImageName,
 fullImageName: image.imageName,
};

// [Existing imports ...]
import * as docker from "@pulumi/docker";

// [Registry configuration as shown above ...]

// Build and publish the container image.
const image = new docker.Image("my-image", {
 build: "app",
 imageName,
 registry,
});

// Export the base and specific version image name.
export const baseImageName = image.baseImageName;
export const fullImageName = image.imageName;

# [Existing imports ...]
import pulumi_docker as docker

# [Registry configuration as shown above ...]

# Build and publish the container image.
image = docker.Image('my-image',
 build='app',
 image_name=image_name,
 registry=registry_info,
)

# Export the base and specific version image name.
pulumi.export('baseImageName', image.base_image_name)
pulumi.export('fullImageName', image.image_name)

// [Existing imports ...]

import "github.com/pulumi/pulumi-docker/sdk/v2/go/docker"

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Registry configuration as shown above ...]

 // Build and publish the container image.
 image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
 Build: &docker.DockerBuildArgs{Context: pulumi.String("app")},
 ImageName: imageName,
 Registry: registryInfo,
 })

 // Export the base and specific version image name.
 ctx.Export("baseImageName", image.BaseImageName)
 ctx.Export("fullImageName", image.ImageName)
 return nil
 })
}

// [Existing imports ...]

using Pulumi.Docker;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Registry configuration as shown above ...]

 // Build and publish the container image.
 var image = new Image("my-image", new ImageArgs
 {
 Build = new DockerBuild { Context = "app" },
 ImageName = imageName,
 Registry = registryInfo,
 });

 // Export the base and specific version image name.
 return new Dictionary<string, object>
 {
 { "baseImageName", image.BaseImageName },
 { "fullImageName", image.ImageName },
 };
 });
}

Now let’s run pulumi up. If you haven’t already done so, this will provision the cloud resources, as well as build/publish the container image. As the build runs, you will see Docker build output streamed to your terminal:

$ pulumi up
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack project-dev created
 ... cloud-specific resources omitted ...
 + └─ docker:image:Image my-image created

Outputs:
 baseImageName: "...cloud-specific url.../my-repo-dc811b0"
 fullImageName: "...cloud-specific url.../my-repo-dc811b0:78d0fce7c2450c15a6153b6b11208fcb6b9edea7bb7ef3b7b6194f3fc101a170"

Also note the base and versioned image URLs are exported as stack outputs. These are optional but have three benefits:

As we see here, the CLI will print them after each deployment.
The CLI can fetch them on-demand, e.g., ‘pulumi stack output fullImageName’, making it easy to script access to these container images.
The Pulumi StackReference component can be used to depend on this stack from another, allowing us to build higher levels of infrastructure that consume these images.

As an example of (2) in action, let’s run the image locally using the exported image name:

$ pulumi stack output fullImageName
...cloud-specific url.../my-repo-dc811b0:78d0fce7c2450c15a6153b6b11208fcb6b9edea7bb7ef3b7b6194f3fc101a170

Note that this requires that our client can pull from our provisioned registry, which may require cloud-specific authentication first.

To redeploy changes to the Dockerfile or anything that image depends on, including application code, rerun pulumi up. It will detect the differences, rebuild and re-push the minimal layer changes necessary to update and retag the repository image, and then export the resulting image name.

To see this in action, change the Dockerfile’s contents to:

FROM nginx
RUN echo "<h1>Hello, World -- from Pulumi!</h1>" > \
 /usr/share/nginx/html/index.html

And then rerun Pulumi – notice how it detects the change, updates the image, and the versioned image hash changes (indicated by the tilde ~):

$ pulumi up
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack project-dev created
 ~ └─ docker:image:Image my-image updated

Outputs:
 baseImageName: "...cloud-specific url.../my-repo-dc811b0"
 ~ fullImageName: "...cloud-specific url.../my-repo-dc811b0:78d0fce7c2450c15a6153b6b11208fcb6b9edea7bb7ef3b7b6194f3fc101a170"

The Pulumi Docker Image component supports a number of additional options to control its behavior, including passing build arguments, environment variables, extra options for the Docker build, as well as various controls for image caching. For full details on each of these, refer to the image API documentation.

Consume the Container Image

The same container image URLs exported above can be used as inputs to other resources, including infrastructure that will run your container inside of a container orchestration system such as Kubernetes, Amazon ECS, and so on.

This article assumes you already have a containerized environment to deploy to, like a Kubernetes cluster, and have configured your project accordingly. If not, you can provision one using Pulumi first. Pulumi supports many clouds and infrastructure resources, but here are a few starting points to get up and running with:

This example demonstrates deploying our Nginx web server as a load balanced service within Kubernetes. To do so, we’ll declare our Kubernetes configuration, right inside of our existing program defined above, and export its resulting IP address:

// [Previous imports...]
var k8s = require("@pulumi/kubernetes");

// [Registry, build, etc., code from above...]

// Create a load balanced Kubernetes service using this image, and export its IP.
var appLabels = { app: "myapp" };
var appDep = new k8s.apps.v1.Deployment("app-dep", {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 3,
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: "myapp",
 image: image.imageName,
 }],
 },
 },
 },
});
var appSvc = new k8s.core.v1.Service("app-svc", {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: 80, protocol: "TCP" }],
 selector: appLabels,
 },
});

module.exports = {
 // [Previous exports...]
 appIp: appSvc.status.loadbalancer.ingress[0].ip,
};

// [Previous imports...]
import * as k8s from "@pulumi/kubernetes";

// [Registry, build, etc, code from above...]

// Create a load balanced Kubernetes service using this image, and export its IP.
const appLabels = { app: "myapp" };
const appDep = new k8s.apps.v1.Deployment("app-dep", {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 3,
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: "myapp",
 image: image.imageName,
 }],
 },
 },
 },
});
const appSvc = new k8s.core.v1.Service("app-svc", {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: 80, protocol: "TCP" }],
 selector: appLabels,
 },
});
export const appIp = appSvc.status.loadBalancer.ingress[0].ip;

# [Previous imports...]
import pulumi_kubernetes as k8s

# [Registry, build, etc, code from above...]

# Create a load balanced Kubernetes service using this image, and export its IP.
app_labels = { 'app': 'myapp' }
app_dep = k8s.apps.v1.Deployment('app-dep',
 spec={
 'selector': { 'matchLabels': app_labels },
 'replicas': 3,
 'template': {
 'metadata': { 'labels': app_labels },
 'spec': {
 'containers': [{
 'name': 'myapp',
 'image': image.image_name,
 }],
 },
 },
 },
)
app_svc = k8s.core.v1.Service('app-svc',
 metadata={ 'labels': app_labels },
 spec={
 'type': 'LoadBalancer',
 'ports': [{ 'port': 80, 'targetPort': 80, 'protocol': 'TCP' }],
 'selector': app_labels,
 }
)
pulumi.export('appIp', app_svc.status.apply(lambda s: s.loadbalancer.ingress[0].ip))

package main

import (
 // [Previous imports...]
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // [Registry, build, etc, code from above...]

 // Create a load balanced Kubernetes service using this image, and export its IP.
 appLabels := pulumi.StringMap{"app": pulumi.String("myapp")}
 _, deperr := appsv1.NewDeployment(ctx, "app-dep", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{Labels: appLabels},
 Spec: appsv1.DeploymentSpecArgs{
 Selector: &metav1.LabelSelectorArgs{MatchLabels: appLabels},
 Replicas: pulumi.Int(3),
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{Labels: appLabels},
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("myapp"),
 Image: image.ImageName,
 },
 },
 },
 },
 },
 })
 if deperr != nil {
 return deperr
 }
 appSvc, svcerr := corev1.NewService(ctx, "app-svc", &corev1.ServiceArgs{
 Metadata: &metav1.ObjectMetaArgs{Labels: appLabels},
 Spec: &corev1.ServiceSpecArgs{
 Type: pulumi.String("LoadBalancer"),
 Ports: corev1.ServicePortArray{
 corev1.ServicePortArgs{Port: pulumi.Int(80)},
 },
 Selector: appLabels,
 },
 })
 if svcerr != nil {
 return svcerr
 }
 ctx.Export("appIp", appSvc.Status.ApplyT(func(status *corev1.ServiceStatus) *string {
 return status.LoadBalancer.Ingress[0].Ip
 }))

 return nil
 })
}

// [Previous imports...]
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Apps.V1;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;

class Program
{
 static Task<int> Main() => Deployment.RunAsync(async () => {
 // [Registry, build, etc, code from above...]

 // Create a load balanced Kubernetes service using this image, and export its IP.
 var appLabels = new InputMap<string>
 {
 { "app", "myapp" }
 };
 var appDep = new Pulumi.Kubernetes.Apps.V1.Deployment("app-dep", new DeploymentArgs
 {
 Spec = new DeploymentSpecArgs
 {
 Selector = new LabelSelectorArgs
 {
 MatchLabels = appLabels
 },
 Replicas = 3,
 Template = new PodTemplateSpecArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels = appLabels,
 },
 Spec = new PodSpecArgs
 {
 Containers =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "myapp",
 Image = image.ImageName,
 }
 }
 }
 }
 }
 });
 var appSvc = new Pulumi.Kubernetes.Core.V1.Service("app-svc", new Pulumi.Kubernetes.Types.Inputs.Core.V1.ServiceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels = appLabels
 },
 Spec = new ServiceSpecArgs
 {
 Type = "LoadBalancer",
 Ports =
 {
 new ServicePortArgs
 {
 Port = 80,
 TargetPort = 80
 }
 },
 Selector = appLabels
 }
 });

 return new Dictionary<string, object>
 {
 // [Previous exports...]
 { "appIp", appSvc.Status.Apply(status => status.LoadBalancer.Ingress[0].Ip) },
 };
 }
}

Notice here that we are referring to the resulting image name from our service specification. All we need to do run a ‘pulumi up’ and Pulumi will now build, publish, and consume our application’s container image from within our Kubernetes cluster:

$ pulumi up
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack project-dev created
 ... prior output omitted ...
 + ├─ kubernetes:apps/v1:Deployment app-dep created
 + └─ kubernetes:core/v1:Service app-svc created

Outputs:
 appIp : "34.82.32.166"
 baseImageName: "...cloud-specific url.../my-repo-dc811b0"
 fullImageName: "...cloud-specific url.../my-repo-dc811b0:fcbbd958636e4f8ac02568db7dbf8b89a043e210e917432cf12649aaedfa266c"

If this is the first time you’ve run pulumi up, all of the prior resources will be provisioned first, and in any case, the Kubernetes resources will follow.

Now that our deployment has completed, we can curl the endpoint:

$ curl $(pulumi stack output appIp)
<h1>Hello, World -- from Pulumi!</h1>

This image name is the versioned one, meaning that any time a new version is created, Pulumi will detect that it needs to update the Kubernetes deployment which references it, triggering a rollout of the new deployed image. As such, we just need to run pulumi up anytime the application or its Dockerfile changes and Pulumi will detect and trigger a deployment to Kubernetes.

To see this in action, let’s change our Dockerfile’s contents once more:

FROM nginx
RUN echo "<h1>Hello, World -- from Pulumi and Kubernetes!</h1>" > \
 /usr/share/nginx/html/index.html

And then rerun Pulumi – notice that both the image and the Kubernetes deployment has changed this time:

$ pulumi up
Updating (dev)

 Type Name Status Info
 + pulumi:pulumi:Stack project-dev created
 ~ ├─ docker:image:Image my-image updated
 ~ └─ kubernetes:core/v1:Deployment app-dep updated [diff: ~spec]

Outputs:
 appIp : "34.82.32.166"
 baseImageName: "...cloud-specific url.../my-repo-dc811b0"
 fullImageName: "...cloud-specific url.../my-repo-dc811b0:36919b56f6b2dedfa879815443c9964d17a89506caed25025328b1d0204a083f"

Finally, if we re-curl the endpoint, we will now see that the updated container is now live!

$ curl $(pulumi stack output appIp)
<h1>Hello, World -- from Pulumi and Kubernetes!</h1>

And there we go: we have gone from a Dockerfile to a published container image in a container registry of our choosing, which has been used to spin up a load balanced Kubernetes service, and which is now ready for fully automated continuous deployments.

Wrapping Up

In this article, we’ve seen how easy it is to build, publish, and use container images in many popular public and private container registry options. We have seen how to provision new registries using infrastructure as code and how easy it is to trigger deployments of application updates by running a single pulumi up command.

All of these steps were manually run from a CLI; however, a natural next step is to wire the entire process up to a CI/CD system such as GitHub Actions, GitLab Pipelines, Jenkins, Spinnaker, or one of the many available options, so that you can deploy continuously as you merge code. For even more advanced scenarios, we might want to build a custom program such as a CLI that uses the Automation API to perform these actions behind a simpler, purpose-built interface.

Although we’ve shown a very simple set of infrastructure resources, it would be natural to extend these examples by provisioning other ancillary services that your application needs, including databases, pub/sub topics, queues, metrics and dashboards, and more. Using an infrastructure as code approach to building, publishing, and consuming your container images means you can incrementally add on such infrastructure to the base code shown above and reference them from your containers easily with automatic dependency tracking.

The complete examples in this article are available on GitHub. Pulumi is open source and free to get started with — give it a try, and get up and running with infrastructure as code for your containers today!

Pulumi raises Series B to build the future of Cloud Engineering

Joe Duffy — Tue, 27 Oct 2020 00:00:00 +0000

Today I’m thrilled to announce that we’ve raised $37.5 million in Series B funding led by NEA with participation from existing investors, Madrona Venture Group and Tola Capital. We will use this funding to continue serving our fast-growing community of developers and infrastructure practitioners, making Cloud Engineering the new reality for organizations embracing the modern cloud in all aspects of how they ship software.

All software is cloud software,
All developers are cloud developers,
Infrastructure teams enable innovation.

Our vision for the modern cloud era

All software is cloud software, all developers are cloud developers, and infrastructure teams enable innovation. This is Cloud Engineering. The cloud has become a competitive advantage for many companies, enabling new disruptive business models and end user experiences. Pulumi aims to enable this new way of building.

We set out over three years ago to fundamentally rethink how teams create cloud software, with a belief that by standing on the shoulders of giants—leveraging our industry’s best languages, tools, and ecosystems for infrastructure—we could build a platform that uniquely enables developers and infrastructure experts alike to leverage the cloud in new ways. Despite having created languages like C# and TypeScript in our past lives, it struck our founding team that the world didn’t need another programming language, DSL, or YAML templating solution; instead, we could unleash creativity and collaboration by embracing the very languages our industry already knows and loves.

This led to Pulumi’s unique approach to infrastructure as code, something we often call “infrastructure as software.” Bring your own language, fire up your favorite editor, get full access to any or many clouds of your choice, and code away. Use standard libraries, package up and share and reuse your own best practices, or benefit from the community’s shared knowledge. Tap into all the goodness surrounding that language including familiar engineering superpowers like refactoring, interactive documentation and error checking in your IDE, debugging, test frameworks, and tools like code formatters and linters. All the while still getting the belts and suspenders of infrastructure as code: plan deployments by previewing them before they happen, get a full history of who deployed what and when, enforce security, compliance, and cost controls using policy as code, and more, all open source with an easy to use, but optional, SaaS backend.

In addition to turbocharging infrastructure as code, this new approach has enabled new ways to program the cloud. When we were starting, we often heard that infrastructure is “too boring” or that general purpose languages instead of YAML was “too powerful.” On the contrary, the day that AWS put a REST API in front of spinning up a new virtual machine, the world changed irreversibly forever. By having an entire programming and resource model for interacting with cloud infrastructure, we can layer rich abstractions on top, and build bigger things out of smaller things, as we’ve always done with operating systems and the hardware/software interface. As a recent example and preview of things to come, check out our new Automation API, something that simply isn’t possible with traditional YAML, JSON, or DSL-based infrastructure as code approaches.

We also took an approach of integrating with many incredible partners. Pulumi now supports over 50 infrastructure providers including AWS, Azure, Google Cloud, Kubernetes, Alibaba, Auth0, Cloudflare, Datadog, Fastly, GitHub, GitLab, MongoDB, New Relic, and more; over a dozen CI/CD integrations including Azure DevOps, CircleCI, GitHub Actions, GitLab CI, Jenkins, Octopus Deploy, and Spinnaker, for seamless and continuous delivery; and many identity providers, including Atlassian, Azure Active Directory, G Suite, GitHub, GitLab, and Okta. We embraced the Kubernetes ecosystem from day one, with great support for the entire Kubernetes API, managed clusters in all the major clouds (AWS EKS, AKS, GKE, etc.), Helm 2 and 3, Kustomize, Open Policy Agent (OPA)’s Rego language for policy as code, and a Kubernetes Operator, to name a few key features that are helping customers go to production with Pulumi and Kubernetes.

Three years after starting on this journey, hundreds of customers and tens of thousands of end users have used Pulumi to leverage the cloud in transformative new ways, and we regularly hear from developers and infrastructure practitioners about how Pulumi has enabled them to do things previously thought impossible.

Cloud Engineering is here — and growing fast

Cloud Engineering is bigger than just Pulumi. Last month, thousands of us tuned in to learn from industry leaders and expert practitioners at our first annual Cloud Engineering Summit. It is clear that Cloud Engineering is already here in a big way and becoming a standard for the most forward-thinking software teams.

A lot has changed in only three years. At the time we conceived of Pulumi, there was significant fragmentation, uncertainty, and difficulty in adopting Kubernetes, Serverless, Docker, and continuous delivery for production workloads. Developers seldom touched infrastructure. Fast forward to today, and we now have managed container orchestrators, including Kubernetes, across all the major cloud providers, most organizations are shipping daily, and it’s evident that mixed workloads of containers, serverless, VMs, and data services are here to stay. And 6.5 million developers have gone cloud native. Teams are now looking for solutions that help them go from idea to production faster than ever before, and Cloud Engineering is emerging as the best way to do just that.

Cloud Engineering teams are also standardizing on platforms and workflows that go beyond ad-hoc “walls of YAML and mountains of Bash” approaches, taming cloud software complexity while also improving reliability and security with technologies like policy as code. Teams are also increasingly looking for cloud-agnostic platforms due the reality of multi-cloud, whether that’s because they need to ship code to multiple cloud providers, manage Kubernetes resources in addition to cloud infrastructure, manage hosted SaaS infrastructure services by vendors like Cloudflare, Datadog, MailChimp, or MongoDB alongside their cloud infrastructure, or all of the above.

Because all software is cloud software, every software organization in the world needs a Cloud Engineering Platform. We are happy to be that very platform for many of the world’s most innovative companies, and look forward to helping many more customers disrupt their own industries as the cloud continues to transform modern businesses.

What’s next?

We remain focused on Pulumi’s mission to enable Cloud Engineers of all backgrounds, developers and infrastructure experts alike, to collaborate and do their best work together. We have made great progress in three years, laying a foundation that helps to democratize access to cloud infrastructure. But we’re only just getting started.

So, where do we go from here?

We will continue ensuring Pulumi is unquestionably the best infrastructure as code platform available. Features like the Automation API are enabling groundbreaking ways to use cloud infrastructure, powering entire platforms that themselves are powering entire organizations and new cloud products. We are also hard at work creating libraries with built-in best practices, new workshops and educational material, and simpler user experiences, all with the aim of making the cloud easier to use than ever before.

We aren’t going to stop there. This foundation also enables new and exciting product experiences. Although all software is becoming cloud software in a meaningful way, the cloud’s capabilities aren’t yet as easy to harness in our applications as a standard operating system’s are. The next phase of our journey will tackle this problem.

Pulumi is and will continue to be the Cloud Engineering Platform powering the world’s most innovative companies and teams. All software is cloud software, all developers are cloud developers, and infrastructure teams are central to enabling this innovation—and this Series B fundraise will help to ensure we realize the full potential of this vision!

I’d like to take a moment to thank our customers. Our community has grown into an inclusive and collaborative place, reflecting our own company values. One of those values says, “our own success is directly enabled by the success of our customers,” which we remember every day by helping you transform and innovate. We could not have done it without your support and you will continue to be our guiding light for the future.

To the clouds and beyond!

Joe

The Past, Present, and Future of Cloud Engineering

Joe Duffy — Thu, 08 Oct 2020 00:00:00 +0000

All software is cloud software. All modern applications interact with the cloud in some way, whether it’s using cloud for storage, compute capabilities, or with rich cloud services for data, AI/ML, and so much more, to deliver amazing new experiences. As a result, all developers today are cloud developers, and infrastructure teams are key to enabling innovation across the entire organization. I had a great time telling this story at the Cloud Engineering Summit today and wanted to take a moment to put pen to paper.

Introduction

The cloud is the world’s biggest supercomputer. Although the cloud broke our architectures, workflows, and teams, we are now amidst a renaissance of rebuilding new ways to create truly cloud-native applications and experiences.

We conceived of the Cloud Engineering Summit to celebrate this renaissance. The talks throughout the day explore how we build architectures and applications. We also explore how to operate those cloud architectures in a team setting through workflows using automation and processes to build, deploy, and manage those applications and their infrastructure. Finally, teams are working together more closely and in new ways in order to collaborate to deliver applications and infrastructure across all cloud engineering disciplines.

This is what we are calling Cloud Engineering: the next quantum leap in how we build cloud software. Cloud engineering means bringing the cloud closer to application development and infusing its capabilities to create new cloud-enabled experiences. Cloud engineering also means bringing more engineering practices to infrastructure, enabling us to leverage infrastructure as not just code but software. And finally, cloud engineering enables us to unlock collaboration, enabling the entire team to move faster with confidence.

Cloud Engineering in Practice

When the cloud came along, our initial approach was to reuse the same application patterns from the Internet 1.0 era, starting with “lift and shift” and virtual machine-centric architectures. We’ve gone through at least two major sea changes since then, including the introduction of containers and serverless architectures, and if we were to start over with a blank slate today, we’d surely do things very differently.

In fact, the cloud effectively broke our existing approaches for architectures, workflows, and teams.

As the cloud incrementally broke these over the past decade, we invented best practices, tools, and techniques to cope with it. This led to DevOps and a fundamental change to how we worked together to build increasingly cloud-enabled software. DevOps started from a place of collaboration, bringing engineers and operations people closer together and was a very powerful precursor to what is now happening.

This is where cloud engineering now enters the picture. Instead of being reactionary to change, we can now proactively pursue approaches designed for the cloud. Instead of just coping, we can take a step back and reimagine new ways of doing things.

Cloud engineering takes the best practices from DevOps and extends them. There is no question that DevOps, despite its original intent, ended up more “opsy” than “devvy”. With cloud engineering, application developers are now embracing the cloud and infusing cloud capabilities such as microservices and serverless into applications in fundamentally new ways, creating new application experiences for end users.

And at the same time, infrastructure experts, capabilities, and teams are key to enabling innovation more than ever before. During the cloud era, infrastructure became programmable software, which allows us to apply all the software engineering practices we know and love to infrastructure, including testing, refactoring, security, continuous delivery, and agile.

More cloud infrastructure capabilities have moved into application land, while at the same time more application development practices have moved into infrastructure land. What this means is that the whole team is collaborating more closely than ever before. Everyone is empowered to bring their unique skillsets to the table to enable moving faster and deliver amazing software capabilities, regardless of title — developer, DevOps, systems engineer, infrastructure architect, security engineer, etc. This is the modern cloud engineering organization.

Cloud Engineering Architectures

Infrastructure used to be a tax, and IT a cost center, but cloud infrastructure is increasingly a competitive advantage that can transform entire business models. Just think of the many cloud era companies that are enabling new experiences thanks to the cloud — AirBnB, Amazon, Lyft, Spotify, and others. Zoom experienced a 20x increase in usage when COVID started and, because they architected their application for the cloud, they were able to scale and meet demand. Five to ten years ago, without cloud engineering, that wouldn’t have been possible.

Zoom massively scaling is a great example of the difference between just tossing a traditional application into a virtual machine, versus fully embracing cloud capabilities as a foundational piece of the application’s design. The ability to scale up elastically, and pay as you go, is an immense competitive advantage when designing applications to run on the cloud’s planet-scale supercomputer.

This means that cloud infrastructure is a superpower to be wielded and used for great things, not an afterthought. If only teams can figure out how to harness them, they can accomplish new business outcomes and reshape how applications work.

Application models have also evolved in ways that enable this new cloud engineering era. In the 2010s, you had XML web services and SOA, which was the precursor to microservices, and now with modern programming languages, the shift to distributed architectures is a lot easier. Asynchronous programming and concurrency models are pervasive on all of our modern languages and frameworks, and we can use those things to build truly distributed applications that are composed of many smaller pieces that communicate asynchronously and can scale on demand.

Cloud Engineering Workflows

It’s not just the application architectures that have changed; the way that we build, secure, test, ship, and manage those applications and their infrastructures have also fundamentally changed. This is what we call “workflows”: the way in which we operationalize, and often continuously deliver, our cloud applications and infrastructure into our many dev, test, and production environments.

Infrastructure as code lets us declare infrastructure using “code” (although, in many cases, code is really just markup or “text”). We are only just now beginning to apply all the engineering practices that are available to developers when building infrastructure, with technologies like Pulumi that enable infrastructure as software. This includes leveraging great IDEs and tools, enabling higher levels of productivity. Instead of copying and pasting text, we have real patterns and best practices thanks to sharing and reuse.

Among the many things the cloud broke is the inner loop development cycle. We’re often stepping into and out of our IDE, running command lines, pointing and clicking in the cloud console, and so on. The infrastructure as text world often means finding failures 10 minutes into a deployment, when it’s too late, rather than finding errors interactively in your IDE. By embracing infrastructure as software, and the fact that infrastructure is moving closer to application architectures, our inner loop and great productivity is starting to be restored.

Treating infrastructure as software enables us to test infrastructure before it has been deployed, spin up ephemeral environments to exhaustively validate changes before attempting to deploy them, or even apply advanced forms of testing, such as chaos testing, to ensure applications function properly during a service outage, or fuzz testing, to ensure the application works in the face of randomized changes.

Security is a also a key cross-cutting concern but is still often overlooked during the inner loop of developing cloud software. The security perimeter for cloud software is no longer the four walls of the data center but has instead moved to software managed networks and firewalls. This changes the threat model and requires that teams adopt zero trust and principle of least privilege approaches, where applications, infrastructure, and team members have the least permissions possible to get the job done, minimizing the blast zone of possible incidents. We must also adopt approaches that ensure software is secure before deploying, including applying policies that do things like preventing unintended Internet access to services, block S3 buckets and databases open to the Internet, and scan containers for known vulnerabilities.

Once you’ve shipped an application to production, the job has only just begun. Teams must adopt patterns and practices for how to monitor and react to situations that occur in their live environments. That often also includes testing and flighting changes in production. One approach is to adopt an “OODA loop” like approach, where OODA means Observe, Orient, Decide, Act. This is a decision-making framework that allows for rapid decision making with the context of a situation, and good analogy for how we maintain our cloud applications in the real world. How do we know something is wrong? How do we react? The old model of simply relying on performance counters, such as CPU or memory utilization, and highly structured logging, doesn’t really work in the modern cloud era. Instead, teams are adopting observability, and a “log everything” approach, so that unstructured and semi-structured high-cardinality data is available at all times, enabling us to observe, debug problems in realtime as they occur, and make decisions and act when when things go wrong.

Continuous delivery is not a new idea, however, teams are now moving beyond just delivering applications towards shipping entire environments, packaging applications and infrastructure together. This enables all of the above practices, including security and testing, to apply to entire cloud application environments. Continuous delivery is also how application and infrastructure teams work more closer together, enabling teams to go from shipping features shipping quarterly, to monthly, to every week, to daily, to shipping features on every Git commit. Delivery and workflow are intertwined, and our cloud engineering brings them even closer together across the entire team.

Cloud Engineering Teams

Cloud engineering teams are empowered: everybody brings their unique skills, as an expert in their domain, and collaborates to enable 1+1=3 outcomes. Cloud engineering is not about “NoOps” or operations being less important than it was before. It is about recognizing that all of these skills are essential and that a cloud engineering team is the composition of all of these skillsets.

Application developers can focus on what they’re great at: architecting, implementing, and delivering new functionality to their application’s end users. Developers probably aren’t going to want to suddenly go deep in what it takes to create and configure an AWS virtual private cloud, but they sure do want to tap into all of those amazing data and AI/ML services, and bring container and serverless architectures and continuous delivery into the inner loop of how they are building and shipping code. And there’s just no way to ignore the runtime environment when it comes to debugging, performance tuning, scaling the application, and observability, just to name a few important facets.

Infrastructure and DevOps engineers likewise can also focus on what they’re great at: building scalable, reliable, and cost-effective foundations that the entire organization runs on. Infrastructure teams can benefit from great productivity wins of great IDEs, tools, and true sharing and reuse of best practices. That’s not to say suddenly infrastructure engineers need or want to learn everything there is to know about the latest application frameworks or patterns. But many of the foundations of DevOps, where we apply more software engineering and agile practices to how we ship infrastructure, including test driven infrastructure, have reshaped how infrastructure experts operate.

Security engineers are also empowered to be part of this entire lifecycle. This ensures that security isn’t an afterthought, but instead, that security engineers have all the requisite insertion points into the application and infrastructure architectures and workflows, to ensure that solutions are secure on day one, and stay secure. Security engineering is a central part of how cloud engineering teams operate.

Finally, all of these modern cloud engineering practices have helped teams to cope in this suddenly all-remote COVID world. By embracing distributed architectures, automated workflows, and multi-faceted teams that collaborate closely, entire businesses have made the shift to all-remote and online collaboration much faster. Those teams practicing cloud engineering have had an easier time reacting to rapid change.

Towards a Cloud Operating System

Today’s cloud is comprised of 1,000s of building block services across a landscape of multiple cloud providers. AWS alone offers over 200 of these building blocks, and every SaaS infrastructure provider is part of the picture too. As we have seen above, this offers incredible capability, if teams can only figure out how to harness the power of those building blocks. It’s not always easy but the cloud engineering practices above are key to getting started.

It does beg the question, however: If the cloud is a giant supercomputer, where is its operating system? When will we go beyond building blocks? In fact, we are well on our way.

Kubernetes gives us some fundamental operating system-like components — a scheduler, resource model, and a security model, for example — but a complete operating system must go well beyond this, including programming and application models, and many key facets of the cloud, including all those AWS building blocks, still live outside of the Kubernetes cluster (and likely always will).

What’s the user interface to this giant supercomputer supposed to look like? What’s the human experience of using that computer? We’ve extended our previous experience from client-server desktop and Internet applications into the cloud, but it’s just the start of how we interact with the cloud.

What does the developer’s interface to that supercomputer, the programming model, look like? For now, YAML and tools like Terraform give us a way to configure the building blocks using low-level markup languages, but we need to go beyond this and leverage all the things we know and love about other operating system programming models. This includes using great languages and runtimes like TypeScript, Python, .NET, and Go. This is the approach we’ve taken with Pulumi to start creating richer developer experiences with respect to how we stitch together all those building blocks and build bigger things out of smaller things, but it is still early days here.

Finally, where is the application model? How do we reliably and repeatably install, version, secure, work with, and manage these cloud applications? Containers provide a building block for building and deploying applications in the cloud, Helm and ideas like CNAB provide an installation model, but it is a far cry from an “AppStore” experience that we get on other operating systems. The cloud supercomputer interface is very low-level and bottom-up compared to application models which approach the experience from a top-down perspective.

Despite the clear gaps, it is clear we’ve taken some major steps in the direction of a true cloud operating system.

The Future of Cloud Engineering

As we’ve seen, the future of cloud engineering is already here in a very real way. The cloud is being infused into application development, we are increasingly treating cloud infrastructure as software, and teams are embracing new ways of working together.

We hope you enjoyed learning more about cloud engineering architectures, workflows, and teams. Watch the Cloud Engineering Summit Keynote here or check out one of the many amazing Cloud Engineering talks now!

This event would not have been possible without the participation and support of our top sponsors:

Announcing Pulumi 2.0, Now with Superpowers

Joe Duffy — Tue, 21 Apr 2020 00:00:00 +0000

This blog post announces Pulumi 2.0, which has been superseded by newer versions. For information about the latest Pulumi release, please refer to the current documentation and recent release notes.

Today we are excited to announce Pulumi 2.0, the next major stage in our journey as an open source project, company, and community. This release expands on our original vision of using your favorite languages and tools to do all things infrastructure as code, now with new cloud engineering superpowers that will help you and your team adopt modern cloud architectures.

Get Started Learn More

What is Pulumi?

We call Pulumi’s unique approach Modern Infrastructure as Code:

Familiar Languages. Use the best languages and tools to declare infrastructure and gain best-in-class productivity and engineering capabilities. Connect to existing communities and ecosystems and leverage world-class language and tooling innovation.
Multi-Cloud. Adopt consistent workflows across any cloud(s) — public, private, or hybrid — while still using the full breadth and depth of your target cloud’s services.
Modern Architectures. Although Pulumi works great for legacy workloads, it was born in a world of modern container, serverless, and cloud-native architectures. Feel free to mix and match these approaches as you modernize over time.
Cloud Engineering Teams. By choosing Pulumi, you’re betting on your team. The whole team, in fact: developers, infrastructure and operations teams, and security engineers alike. Tackle cloud solutions together without technology or team silos.

Introducing Superpowers!

The main theme of 2.0 is something we’ve taken to calling “superpowers.” This isn’t just a buzzword we came up with — it’s something we keep hearing from happy end users.

What is Pulumi 2.0?

After shipping 1.0 last year, we’ve been hard at work helping customers to succeed at their modern cloud projects. This has taken us beyond the basics to include:

Provisioning. All core language SDKs are now at parity, including all existing languages — Node.js (JavaScript, TypeScript) and Python — as well as the new additions to the family — .NET (C#, F#, etc) and Go. We’ve also significantly expanded upon the supported cloud resource providers in the Registry, totalling above three dozen, and rolled out entirely new API documentation.
Delivery. As we’ve worked with customers to go from development to production, we’ve added more CI/CD integrations as well as environment management capabilities to help at scale, such as project and stack tagging and filtering.
Architecture. Customers are building their own platforms that use abstraction and packaging mechanisms, including our component model, to codify their own best practices. New coexistence and migration options help to build larger systems out of many component parts, including those you haven’t been able to port to Pulumi yet.
Policy. Our new policy as code framework, CrossGuard, lets you define policies using familiar languages and then apply those policies at deployment-time. This prevents mistakes from ever getting out the door, including cost, security, and compliance. Configurable policy packs let you write a single policy and apply it flexibly across your projects.
Testing. Many customers are using a spectrum of techniques to validate their infrastructure in new ways. This includes unit testing using familiar frameworks and new mocking capabilities through integration testing for short- and long-lived environments.

Welcome to the Team, Pulumipus!

We’re also taking the opportunity to have a little fun with the launch and introducing a new mascot, The Pulumipus. You may have met them already at an event or at PulumiHQ, and we’re happy to welcome them as the official mascot going forward.

Expect The Pulumipus to pop up from time to time to help you out with your infrastructure superpowers!

Give it a Try Today

Today is an exciting day for us, our customers, and our community — your passionate support and feedback shaped this release, and we thank you.

To learn more, join us for our launch event next Wednesday, or just download Pulumi and give it a try. If you’re upgrading from Pulumi 1.0, please see our migration guide.

Get Started Learn More

Please tell your friends and colleagues — we will see you in the Community Slack, where we look forward to hearing what you’re doing with your new superpowers!

Joe

Automatically Enforcing AWS Resource Tagging Policies

Joe Duffy — Mon, 06 Apr 2020 00:00:00 +0000

AWS publishes best practices for how to tag your resources for cost tracking, automation, and organization. But how do you enforce that you’re doing it correctly across all of your projects? And is it really necessary to manually track down all those places where you missed a tag and manually patch things up? In this article, we’ll see how to use Policy as Code to enforce your team’s tagging strategies in addition to some powerful Infrastructure as Code techniques to automate applying your tags in a consistent way across all of your projects and resources.

Why Tag Your Resources?

A tag is simply a key/value label that you can apply to your AWS infrastructure resources. Tags enable you to manage, search for, and filter resources. There aren’t any predefined tags — you can use whatever makes sense for your scenario and business requirements.

Amazon recommends many tagging strategies, including technical tags like name and environment, automation tags like dates and security requirements, business tags like owner and cost center, and security tags for compliance. Each of these enables you to apply policies.

Specifying a tag in your Infrastructure as Code is easy. Not all resources are taggable (although the most important ones are); to tag a resource, specify a map of key/values using the tags property. For example, this code declares an S3 Bucket that carries three tags that enable cost allocation reporting: "user:Project", "user:Stack", and "user:Cost Center":

let aws = require("@pulumi/aws");
let pulumi = require("@pulumi/pulumi");

// Create an S3 bucket (with tags).
let config = new pulumi.Config();
let bucket = new aws.s3.Bucket("my-bucket", {
 tags: {
 "user:Project": pulumi.getProject(),
 "user:Stack": pulumi.getStack(),
 "user:Cost Center": config.require("costCenter"),
 },
});

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

// Create an S3 bucket (with tags).
const config = new pulumi.Config();
const bucket = new aws.s3.Bucket("my-bucket", {
 tags: {
 "user:Project": pulumi.getProject(),
 "user:Stack": pulumi.getStack(),
 "user:Cost Center": config.require("costCenter"),
 },
});

import pulumi
import pulumi_aws as aws

# Create an S3 bucket (with tags).
config = pulumi.Config()
bucket = aws.s3.Bucket('my-bucket',
 tags={
 'user:Project': pulumi.get_project(),
 'user:Stack': pulumi.get_stack(),
 'user:Cost Center': config.require('costCenter'),
 },
)

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/go/pulumi"
 "github.com/pulumi/pulumi/sdk/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an S3 Bucket (with tags):
 _, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
 Tags: pulumi.Map({
 "User:Project": pulumi.String(ctx.Project()),
 "User:Stack": pulumi.String(ctx.Stack()),
 "User:Cost Center": pulumi.String(config.Require(ctx, "costCenter")),
 })
 })
 return err
 }
}

using Pulumi;
using Pulumi.Aws.S3;
using System.Collections.Generic;
using System.Threading.Tasks;

class Program {
 static Task Main() {
 return Deployment.RunAsync(() => {
 // Create an S3 Bucket (with tags):
 var config = new Config();
 var bucket = new Bucket("my-bucket", new BucketArgs {
 Tags = {
 { "User:Project", Deployment.Instance.ProjectName },
 { "User:Stack", Deployment.Instance.StackName },
 { "User:Cost Center", config.Get("costCenter") },
 },
 });
 });
 }
}

In this example, we’re using the project and stack names from the current project, and requiring an explicit cost center configured using pulumi config set costCenter 11223344. Of course, this is just for illustration purposes — we can easily use anything for these keys and values.

Suppose our team requires certain tags, though: How do we ensure we don’t forget them?

Enforcing Tagging Strategies

Policy as Code is a way to enforce infrastructure policies, such as ensuring we’re not opening databases to the internet, that we’re leveraging strong encryption in all the right places, that we aren’t violating cost policies, and so on. And it’s a great way to ensure we’re tagging everything.

Imagine we forgot to tag our S3 Bucket from earlier:

// Oops -- no tags!
let bucket = new aws.s3.Bucket("my-bucket");

// Oops -- no tags!
const bucket = new aws.s3.Bucket("my-bucket");

# Oops -- no tags!
bucket = aws.s3.Bucket('my-bucket')

// Oops -- no tags!
_, err := s3.NewBucket(ctx, "my-bucket", nil)
return err

// Oops -- no tags!
var bucket = new Bucket("my-bucket");

By applying the policy defined below, deployments that are missing the required tags will fail:

This approach not only ensures we don’t forget, but if you’re an infrastructure or security engineer, you can now apply this across your team to make sure your team doesn’t forget either.

Defining our Tags Enforcer Policy

This policy is ultimately defined as a simple policy pack as follows:

let policy = require("@pulumi/policy");
let isTaggable = require("../lib/taggable").isTaggable;

new policy.PolicyPack("aws-tags-policies", {
 policies: [{
 name: "check-required-tags",
 description: "Ensure required tags are present on all AWS resources.",
 configSchema: {
 properties: {
 requiredTags: {
 type: "array",
 items: { type: "string" },
 },
 },
 },
 validateResource: (args, reportViolation) => {
 const config = args.getConfig();
 const requiredTags = config.requiredTags;
 if (requiredTags && isTaggable(args.type)) {
 const ts = args.props[tags];
 for (const rt of requiredTags) {
 if (!ts || !ts[rt]) {
 reportViolation(
 `Taggable resource '${args.urn}' is missing required tag '${rt}'`);
 }
 }
 }
 },
 }],
});

import * as policy from "@pulumi/policy";
import { isTaggable } from "../lib/taggable";

new policy.PolicyPack("aws-tags-policies", {
 policies: [{
 name: "check-required-tags",
 description: "Ensure required tags are present on all AWS resources.",
 configSchema: {
 properties: {
 requiredTags: {
 type: "array",
 items: { type: "string" },
 },
 },
 },
 validateResource: (args, reportViolation) => {
 const config = args.getConfig<AwsTagsPolicyConfig>();
 const requiredTags = config.requiredTags;
 if (requiredTags && isTaggable(args.type)) {
 const ts = args.props[tags];
 for (const rt of requiredTags) {
 if (!ts || !ts[rt]) {
 reportViolation(
 `Taggable resource '${args.urn}' is missing required tag '${rt}'`);
 }
 }
 }
 },
 }],
});

interface AwsTagsPolicyConfig {
 requiredTags?: string[];
}

This project defines a policy pack containing a set of policy rules, in this case, just one. That rule takes in a configurable set of tags to ensure they exist on every taggable AWS resource. If a tag is missing, a violation is reported and the deployment fails.

This leverages a library that helps to identify taggable AWS resources (see this repo for the full example code). Although this policy is written in TypeScript, it can be applied to stacks written in any language.

At this point, we can apply our policy pack to our infrastructure project in two ways: at the CLI or in the SaaS web console.

Applying Tags Enforcement in the CLI

This policy pack is configurable so that you can enforce arbitrary tags without needing to change the pack’s code, making it reusable. For the CLI scenario, we will create a policy-config.json file that specifies the same three required tags shown above:

{
 "all": "mandatory",
 "check-required-tags": {
 "requiredTags": [
 "user:Project",
 "user:Stack",
 "user:Cost Center"
 ]
 }
}

Next, we can manually specify that our policy pack is applied using the CLI’s --policy-pack flag, along with --policy-pack-config to point at our configuration file:

$ pulumi up \
 --policy-pack=./policy \
 --policy-pack-config=./policy-config.json

Applying Tags Enforcement in the SaaS Console

It’s nice to be able to use the CLI for this — and all of that is available in open source, no matter whether you’re using the SaaS or not. However, it requires manually distributing policy packs and remembering to pass --policy-pack with the right configuration for every update.

The Pulumi Enterprise SaaS console lets you manage policy packs and apply them to your organization’s stacks in a central way — and then every update that the pack is applied to automatically runs the policy checks. Let’s give it a try.

First, let’s publish the pack:

$ pulumi policy publish
Obtaining policy metadata from policy plugin
Compressing policy pack
Uploading Policy Pack to Pulumi service
Publishing "aws-tags-policies" - version 0.0.1 to "acmepolicy"

Permalink: https://app.pulumi.com/acmepolicy/policypacks/aws-tags-policies/0.0.1

This stores the policy pack in the Pulumi SaaS:

From there, we can enable it in the UI. First, we go to our organization’s “POLICIES” tab:

Every organization gets a default policy group, and that’s what we’ll use here. However, for sophisticated scenarios, you may want multiple policy groups (for instance, to enforce different tags for your production environments than your development ones).

Let’s “ADD” the new pack. This pops a dialog we can use to select the pack, its version, and our desired Enforcement Level (we’ll pick “mandatory”):

Finally, further down in the dialog, we can enter the tags we’d like to enforce:

Now that it is configured, all subsequent updates across the organization will run policy checks.

Enforce policy as code

Write policy as code with Pulumi to catch missing tags, block risky configurations, and keep every deployment compliant across your team.

Get started

Automatically Applying Tags

In all cases, after manually fixing our bucket, and adding the correct tags, the policy will pass:

This is great — we can now rest assured that all taggable AWS resources will be tagged before we provision them. But it sure is tedious to add these tags to every resource and then get policy violations errors anytime we forget. One of the advantages of using Infrastructure as Code is that we can automate the injection of these tags.

To do that, let’s write a function that detects taggable resources and merges in automatic tags:

let pulumi = require("@pulumi/pulumi");
let isTaggable = require("./taggable").isTaggable;

/**
 * registerAutoTags registers a global stack transformation that merges a set
 * of tags with whatever was also explicitly added to the resource definition.
 */
module.exports = {
 registerAutoTags: function (autoTags) {
 pulumi.runtime.registerStackTransformation((args) => {
 if (isTaggable(args.type)) {
 args.props["tags"] = Object.assign(args.props["tags"], autoTags);
 return { props: args.props, opts: args.opts };
 }
 };
 },
};

import * as pulumi from "@pulumi/pulumi";
import { isTaggable } from "./taggable";

/**
 * registerAutoTags registers a global stack transformation that merges a set
 * of tags with whatever was also explicitly added to the resource definition.
 */
export function registerAutoTags(autoTags: Record<string, string>): void {
 pulumi.runtime.registerStackTransformation((args) => {
 if (isTaggable(args.type)) {
 args.props["tags"] = { ...args.props["tags"], ...autoTags };
 return { props: args.props, opts: args.opts };
 }
 return undefined;
 });
}

import pulumi
from taggable import is_taggable

# registerAutoTags registers a global stack transformation that merges a set
# of tags with whatever was also explicitly added to the resource definition.
def register_auto_tags(auto_tags):
 pulumi.runtime.register_stack_transformation(lambda args: auto_tag(args, auto_tags))

# auto_tag applies the given tags to the resource properties if applicable.
def auto_tag(args, auto_tags):
 if is_taggable(args.type_):
 args.props['tags'] = {**(args.props['tags'] or {}), **auto_tags}
 return pulumi.ResourceTransformationResult(args.props, args.opts)

package main

import (
 "reflect"

 "github.com/pulumi/pulumi/sdk/go/pulumi"
)

// registerAutoTags registers a global stack transformation that merges a set
// of tags with whatever was also explicitly added to the resource definition.
func registerAutoTags(ctx *pulumi.Context, autoTags map[string]string) {
 ctx.RegisterStackTransformation(
 func(args *pulumi.ResourceTransformationArgs) *pulumi.ResourceTransformationResult {
 if isTaggable(args.Type) {
 // Use reflection to look up the Tags property and merge the auto-tags.
 ptr := reflect.ValueOf(args.Props)
 val := ptr.Elem()
 tags := val.FieldByName("Tags")

 var tagsMap pulumi.Map
 if !tags.IsZero() {
 tagsMap = tags.Interface().(pulumi.Map)
 } else {
 tagsMap = pulumi.Map(map[string]pulumi.Input{})
 }
 for k, v := range autoTags {
 tagsMap[k] = pulumi.String(v)
 }
 tags.Set(reflect.ValueOf(tagsMap))

 return &pulumi.ResourceTransformationResult{
 Props: args.Props,
 Opts: args.Opts,
 }
 }
 return nil
 },
 )
}

static ResourceTransformation RegisterAutoTags(Dictionary<string, string> autoTags) {
 return args => {
 if (IsTaggable(args.Resource.GetResourceType())) {
 // Use reflection to look up the Tags property and merge the auto-tags.
 var tagp = args.Args.GetType().GetProperty("Tags");
 var tags = (InputMap<object>)tagp.GetValue(args.Args, null) ?? new InputMap<object>();
 foreach (var tag in autoTags) {
 tags[tag.Key] = tag.Value;
 }
 tagp.SetValue(args.Args, tags, null);
 return new ResourceTransformationResult(args.Args, args.Options);
 }
 return null;
 };
}

Now we can go back to our main program, use this new module, remove the explicit tags, and every taggable AWS resource we create will automatically get the tags we’ve specified:

let aws = require("@pulumi/aws");
let pulumi = require("@pulumi/pulumi");
let registerAutoTags = require("./autotag".registerAutoTags);

// Automatically inject tags.
let config = new pulumi.Config();
registerAutoTags({
 "user:Project": pulumi.getProject(),
 "user:Stack": pulumi.getStack(),
 "user:Cost Center": config.require("costCenter"),
});

// Create a bunch of AWS resources -- with auto-tags!

let bucket = new aws.s3.Bucket("my-bucket");

let group = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

let server = new aws.ec2.Instance("web-server-www", {
 instanceType: "t2.micro",
 ami: "ami-0c55b159cbfafe1f0",
 vpcSecurityGroupIds: [ group.id ],
});

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";
import { registerAutoTags } from "./autotag";

// Automatically inject tags.
const config = new pulumi.Config();
registerAutoTags({
 "user:Project": pulumi.getProject(),
 "user:Stack": pulumi.getStack(),
 "user:Cost Center": config.require("costCenter"),
});

// Create a bunch of AWS resources -- with auto-tags!

const bucket = new aws.s3.Bucket("my-bucket");

const group = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

const server = new aws.ec2.Instance("web-server-www", {
 instanceType: "t2.micro",
 ami: "ami-0c55b159cbfafe1f0",
 vpcSecurityGroupIds: [ group.id ],
});

import pulumi
import pulumi_aws as aws
from autotag import register_auto_tags

# Automatically inject tags.
config = pulumi.Config()
register_auto_tags({
 'user:Project': pulumi.get_project(),
 'user:Stack': pulumi.get_stack(),
 'user:Cost Center': config.require('costCenter'),
})

# Create a bunch of AWS resources -- with auto-tags!

bucket = aws.s3.Bucket('my-bucket')

group = aws.ec2.SecurityGroup('web-secgrp',
 ingress=[
 { 'protocol': 'tcp', 'from_port': 22, 'to_port': 22, 'cidr_blocks': ['0.0.0.0/0']},
 { 'protocol': 'tcp', 'from_port': 80, 'to_port': 80, 'cidr_blocks': ['0.0.0.0/0']},
 ],
)

server = aws.ec2.Instance('web-server-www',
 instance_type='t2.micro',
 ami='ami-0c55b159cbfafe1f0',
 vpc_security_group_ids=[ group.id ],
)

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/go/aws/ec2"
 "github.com/pulumi/pulumi-aws/sdk/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/go/pulumi"
 "github.com/pulumi/pulumi/sdk/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Automatically inject tags.
 RegisterAutoTags(ctx, map[string]string{
 "User:Project": ctx.Project(),
 "User:Stack": ctx.Stack(),
 "User:Cost Center": config.Require(ctx, "costCenter"),
 })

 // Create a bunch of AWS resources -- with auto-tags!

 _, err := s3.NewBucket(ctx, "my-bucket", nil)
 if err != nil {
 return err
 }

 grp, err := ec2.NewSecurityGroup(ctx, "web-secgrp", &ec2.SecurityGroupArgs{
 Ingress: ec2.SecurityGroupIngressArray{
 ec2.SecurityGroupIngressArgs{
 Protocol: pulumi.String("tcp"),
 FromPort: pulumi.Int(80),
 ToPort: pulumi.Int(80),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = ec2.NewInstance(ctx, "web-server-www", &ec2.InstanceArgs{
 InstanceType: pulumi.String("t2.micro"),
 Ami: pulumi.String("ami-0c55b159cbfafe1f0"),
 VpcSecurityGroupIds: pulumi.StringArray{grp.ID()},
 })
 return err
 }
})

using Pulumi;
using Pulumi.Aws.Ec2;
using Pulumi.Aws.Ec2.Inputs;
using Pulumi.Aws.S3;
using System.Collections.Generic;
using System.Reflection;

class MyStack : Stack {
 static Config config = new Config();

 public MyStack() : base(
 new StackOptions {
 ResourceTransformations = {
 RegisterAutoTags(new Dictionary<string, string> {
 { "User:Project", Deployment.Instance.ProjectName },
 { "User:Stack", Deployment.Instance.StackName },
 { "User:Cost Center", config.Require("costCenter") },
 }),
 },
 }
 )
 {
 // Create a bunch of AWS resources -- with auto-tags!

 var bucket = new Bucket("my-bucket", new BucketArgs());

 var grp = new SecurityGroup("web-secgrp", new SecurityGroupArgs {
 Ingress = {
 new SecurityGroupIngressArgs {
 Protocol = "tcp", FromPort = 80, ToPort = 80, CidrBlocks = {"0.0.0.0/0"},
 },
 }
 });

 var srv = new Instance("web-server-www", new InstanceArgs {
 InstanceType = "t2.micro",
 Ami = "ami-0c55b159cbfafe1f0",
 VpcSecurityGroupIds = { grp.Id },
 });
 }

 // ...
}

Notice that we didn’t specify any tags by hand for the resource definitions and yet if we run a pulumi preview --diff, we see that the correct tags are applied automatically, thanks to the global stack transformation:

And running an ordinary update now passes:

Try adding some resources of your own — VPCs, EC2 instances, security groups, EKS clusters, anything — and it will automatically get tagged with these same cost center tags.

In Conclusion

In this post, we’ve seen some ways to enforce AWS tagging best practices. This includes manually applying tags using Infrastructure as Code, checking that the desired tags are applied to the relevant resources using Policy as Code, and even using some advanced techniques to automatically tag resources, reducing manual efforts and the chance of human error.

Check out these resources to get started with Pulumi’s open source platform:

Good luck making sure your team’s resources are tagged early and often with less manual effort!

Pulumi’s Coronavirus Plan

Joe Duffy — Wed, 18 Mar 2020 00:00:00 +0000

Together, we’re facing an unprecedented situation with the COVID-19 pandemic. In this note, I’d like to tell you about the steps we’ve taken to ensure the health of our employees, community, and to ensure business continuity throughout.

To our customers

First and foremost, our thoughts are with anyone impacted by the virus either directly or indirectly. We are taking the utmost precautions necessary to ensure the health of our employees and the community.

At Pulumi, we are entirely customer and community-driven and have taken a number of steps to ensure business continuity. We know that stable infrastructure is critical to keeping your own business running smoothly and want you assure you that Pulumi is here for you as you focus on your own response to the situation.

Cloud infrastructure can be an essential component in the fight against this pandemic — something we are keenly aware of as we have government and healthcare customers that must scale up their response. If you or someone you know may be able to benefit from our technology, please contact us directly.

A cross-functional leadership team at Pulumi met early on and acted quickly, moving to remote work over 2 weeks ago. This same team regularly assesses recent developments and monitors local, state, federal, and international guidelines, to ensure that we react appropriately to the ever-changing situation.

Here are some of the concrete steps we have taken.

Remote work

The Pulumi team is fortunate enough to have been remote-friendly since our inception, with many employees already outside of our Seattle HQ, including international employees who are ready to help our customers. Over 2 weeks ago, we made the quick decision to move to 100% remote temporarily so that employees are safe and to support social distancing initiatives.

This shift ensures we are doing our utmost to support the containment of the disease spreading. Moving from remote-friendly to all-remote has required some changes, however, our own team has felt that it has overall pushed us to improve communication and knowledge sharing. GitLab’s all-remote guide has been very helpful in this transition.

Software as a service continuity

From day one, the Pulumi service was architected for continuous delivery without manual processes. We ship new features and improvements multiple times a week, every week, and this situation has not impacted that cadence. We have strong operations policies and procedures to respond to any situations that may arise.

Expect to see the same great dependable service you’ve come to know with a steady stream of new features and improvements in the weeks ahead as we get ready for our major product launch in early April.

Leveraging digital channels

The demand for Pulumi content and education continues to grow and, even in the face of restricted travel, we will ramp it up further. Although many events worldwide we’d planned to sponsor or otherwise participate in have been canceled, we have shifted to working with customers through digital channels wherever we can.

Our blog, Community Slack, and PulumiTV YouTube channel are already great online resources, but we have significant new efforts in the works. We’re starting a weekly live video stream called Modern Infrastructure as Code Wednesday featuring learnings from our deepest technical experts. We are having our first online Hackathon this Thursday. Our major upcoming launch on April 22nd has now turned into a day full of online infrastructure content from our team, customers, and partners. Learn more details about these digital events and more here.

Please drop us a note if you’d like us to participate in an online event, host a joint webinar, or want to see a specific topic covered.

In the weeks ahead

We will continue meeting regularly as a leadership team, and talking to our customers and partners, so that we can evolve our strategy and take new measures for health, safety, and continuity purposes as appropriate. We will keep you updated.

We are confident in what we are doing to keep our employees safe, continue delivering value and reliability in our products, and to keep Pulumi’s business functioning normally for our valued customers. Even as we deal with this pandemic, our entire team remains laser-focused on your success and, as always, if there is anything we can ever to do help with your own ever-changing needs, please get in touch with us.

An Update on our Roadmap

Joe Duffy — Mon, 09 Mar 2020 00:00:00 +0000

2020 is off to a big start for us! The .NET and Go SDKs are now more idiomatic and easier to use, Policy as Code is ready for prime-time, and many other useful features and foundational improvements have been released. I’d like to take a moment to highlight these improvements and ask for feedback — we want to make sure these new features work great for you!

.NET

Our .NET SDK has come a long way. We recently tidied up the API surface area to feel more familiar to .NET developers, added C# getting started guides for AWS, Azure, GCP, and Kubernetes, and added many new examples. We also closed the gap on core Pulumi features missing from the initial release, including aliases, transformations, StackReference, and .NET versions of the Kubernetes, Docker, and Terraform remote state packages.

The remaining areas to improve for the .NET SDK include making the API documentation available in C# (details are below in the “still to come” section), migrating to .NET Core 3.1, and enabling you to test your infrastructure using familiar C# tools more easily.

Go

We completely overhauled our Go SDK to be more idiomatic and easier to use. This blog post provides an overview of this new programming model.

The major change in the Go SDK is that we previously used map[string]interface{} types in many places instead of properly-typed data structures. No more! Instead, you now get strong typing for all Go resource APIs across all providers, including AWS, Azure, GCP, and Kubernetes. The improvements here are huge, as you can see in this before/after PR. Examples and documentation have been updated to this new style.

GoDocs are now here for all of our providers and we’ve also added Go versions to our getting started guides for AWS, Azure, and GCP, and have many more improvements on the way.

We’ve also added support for core Pulumi features, including aliases and transformations. A few remain to flesh out like adding Go support for our Kubernetes, Docker, and Terraform remote state packages. The biggest change to come is retiring Dep in favor of Go modules everywhere.

Policy as Code

CrossGuard, our Policy as Code framework, is now ready to take for a serious test drive. CrossGuard lets you to write policies in real code and enforce them during updates. Policies can check for anything; however, common checks include security, compliance, cost management, and general and team best practices. You can write your policy packages or use off-the-shelf policy packs like our AWSGuard package.

CrossGuard is open source and the functionality is behind the new pulumi policy command, as well as the new --policy-pack flag for the preview and up commands. You can use these features with any edition of Pulumi, without restriction, including the offline backends.

If you choose to use CrossGuard with the Enterprise Edition of Pulumi, however, you’ll get some added functionality. This includes server-side enforcement of policies and organizational policies, including “policy groups” which let you group and apply many policies together to your stacks, such as applying different policies to production than development environments.

We recently added the ability to tag policy packs with a semantic version so that you can easily manage and apply packs using user-friendly versions instead of auto-generated numbers.

The remaining areas of focus between now and launch include:

Enabling policy pack authoring in Python. Policies can already apply to stacks written in any language, no matter the language of the policy pack itself, but today policy packs are authored in Node.js. Python is coming soon, with Go and .NET following afterward.
Allowing policy packs to be configured. Configuring policy packs will let you selectively enable or disable specific rules, and/or set configuration variables on them that influence their behavior, on a per stack or policy group basis.
Making reference policy packs available for AWS, Azure, GCP, and Kubernetes. These packs will be usable out-of-the-box with a collection of some of the most important configurable rules we’ve seen, in addition to being a starting point for creating your packs.

CrossGuard is ready to try today, and we’ve begun blogging about some example use cases that you may find interesting.

Other Goodies

In addition to the major areas above, we’ve continued to ship many other improvements. Here are some of our favorite ones:

Stack export for the CLI and in the service, enabling you to export any past version of a stack’s state file. This helps you debug, restore, or roll back changes, or even just for historical reporting purposes.
Many quality-of-life improvements in the Pulumi SaaS Console, including stack tagging, grouping, and sorting; deep linking into CI/CD systems; infrastructure configuration pretty-printing; reverse stack membership lookups; performance improvements; and more.
Full audit logs for Pulumi Enterprise enable you to go back in time and figure out who, when, and why something about your organization, project, or stack changed.
Pulumi runtime mocking for infrastructure testing, developed in partnership with several of our customers leading the charge on “test-driven infrastructure.” Look for more end-to-end docs and guidance soon!
The ability to generate Kubernetes YAML gives you infrastructure authoring capabilities in your language of choice, while still letting you drive deployments using other YAML-based tools, such as kubectl.
We’ve been steadily adding more and more providers and now have over 30 of them, including Aiven, Cloudflare, Consul, Datadog, DigitalOcean, DNSimple, F5 BIG-IP, Fastly, GitLab, Kafka, Linode, MySQL, New Relic, Okta, Packet, PostgreSQL, RabbitMQ, SignalFX, Vault, and more.

Goodies Still to Come

Our API docs are getting a major overhaul! The new design uses a more resource-oriented view, which will make it much easier to navigate and find what you need. This will also ensure all languages are uniform in terms of how documentation is presented as well as coverage of code samples. Expect to see some of this land and evolve in the days to come.

We continue to make progress on our multi-language library architecture. This will enable us to bring your favorite packages like EKS and AWSX to new languages, including Python, C#, and Go, in addition to eventually letting you write your own. This has been a long journey as we figure out how to correctly build this capability, however we will be bringing EKS to Python first.

Our tf2pulumi tool, which converts any Terraform HCL to Pulumi code, is currently getting a fresh coat of paint. Look for HCL2 support and Python code-generation soon.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

We’ve made some significant performance improvements and have others still in the works. Recent changes have reduced overall network bandwidth used by an average update in some cases by 100x, as well as reducing startup times by multiple seconds. The net result is that you’ll see fewer delays during updates and an overall snappier CLI experience.

Next Steps

A lot of good work has already landed in 2020 — and there is a lot more on its way, including a few surprises!

We appreciate all the feedback you’ve given us that is helping to make sure our work in early 2020 is a major step function in overall features, quality, and capabilities. We’re always looking for feedback so please let us know what you think. Give Pulumi a try today, drop into our Community Slack to share your feedback, and make sure to file any suggestions or improvements we can make on GitHub.

2019 Year at a Glance

Joe Duffy — Tue, 31 Dec 2019 00:00:00 +0000

As we celebrate another incredible year in the books here at Pulumi, I wanted to share some reflections about our most exciting milestones over the past twelve months. The best part has been connecting with more customers worldwide, as we saw more than a 15x growth in our customer base, surely a sign of big things to come in 2020. We couldn’t have done it without our amazing community; thank you deeply for your continued support and passion around Pulumi’s bold mission to empower every engineer to program the cloud — you make it all worthwhile.

It’s impossible to highlight everything, but here are some of our favorite milestones:

Pulumi 1.0. After working with several major organizations to go to production with Pulumi, including multiple Fortune 500 and Global 2000 customers, we felt it was time to focus on a “1.0” release. To us, that meant significant improvements to quality, rounding out any open design issues, and sticking to compatibility. As such, we finished JavaScript, TypeScript, and Python support, made significant improvements to documentation and examples, and landed on a release we were proud to stand behind. Months later, we are up to version 1.8, and have stuck to our promise of “no breaking changes”: we understand how fundamental an infrastructure as code tool is to your organization, and how frustrating breaking changes would be. A solid 1.0 serves as a critical foundation for us going forward.

Global community. Our community of end users grew significantly during 2019, with thousands of end users sharing best practices on our Community Slack, and we expanded our physical presence globally. This included over two dozen events from big to small, including AWS re:Invent, Microsoft Build, Google Cloud Next, KubeCon EU and NA, and many Devopsdays, as well as several community workshops. We’ve also seen an uptick in community-led user groups and presentations, consulting and integration partnerships, and look forward to expanding our efforts here even more significantly next year. (If you ever need support/swag for your event, please contact us!)

Pulumi 2.0 roadmap. After 1.0, we published a roadmap to our next major release, Pulumi 2.0. The two themes for this release are to continue pushing the boundaries of productivity, while also giving you “belts and suspenders” for teams and enterprises. We feel this is the magic of Pulumi: the ability to simultaneously run fast and build out infrastructure productively and using the latest and greatest features in AWS, Azure, Google Cloud, and Kubernetes … while still ensuring that security, compliance, and cost-related requirements and best practices are adhered to.

Simpler, per-user pricing. When we launched, our pricing was based on the number of stacks you have. Although our hearts were in the right place with this approach, we realized after working with customers that this could force architectural choices based on price, not what was in the best interest of teams and their projects. This didn’t feel right — we don’t want to penalize teams for doing the right thing! We rolled out per-user pricing in response and have had success with teams of all sizes — from teams of less than 5 engineers all the way up to teams of hundreds.

Pulumi Crosswalk. We introduced the Pulumi Crosswalk brand for our collection of cloud-specific higher level frameworks that simplify many common infrastructure patterns, while also including built-in best practices. Throughout the year, we’ve rounded out that support, starting with AWS, then Kubernetes, as well as some initial steps towards frameworks for Azure and Google Cloud:

Pulumi Crosswalk for AWS simplifies common AWS patterns, including creating VPCs, ECS, EKS, and Fargate clusters, building and publishing Docker images to ECR registries, and more.
Pulumi Crosswalk for Kubernetes encapsulates battle-tested approaches to running Kubernetes clusters and related infrastructure in production. This includes playbooks for running Kubernetes in AWS, Azure, or Google Cloud, as well as kx, a highly productive library that eliminates the “walls of YAML” boilerplate from your Kubernetes applications configurations.
Although still works in progress, Pulumi Crosswalk for Azure will include simple Azure Functions callbacks and globally distributed applications that auto-scale alongside Cosmos DB, and Pulumi Crosswalk for Google Cloud will include simplified Google Functions.

Policy as Code (“CrossGuard”). We shipped a preview of our new policy as code support, code-named CrossGuard, which allows you to enforce policies at deployment time. CrossGuard brings the same “familiar language” approach we’ve used for infrastructure as code, and applies it to policy as code, enabling you to author your own rich rules to enforce security, compliance, cost, and team best practices. We open sourced previews of out-of-the-box rules by way of AWSGuard and OPA integration. CrossGuard is open source and Pulumi’s Team and Enterprise Editions support advanced organization-wide controls.

.NET. We have added initial preview support for writing your infrastructure as code in any .NET Core language, including C#, F#, and VB. This covers any existing Pulumi resource provider, including AWS, Azure, Google Cloud, as well as support for the entire Kubernetes object model. .NET, along with Go, will transition out of preview into being generally available as part of the 2.0 release.

More identity options. Out of the gate, we supported GitHub as an identity provider when authenticating with the Pulumi service. Since then, we added GitLab, Atlassian, Email, and SAML/SSO, with user guides for Azure ActiveDirectory, Google GSuite, and Okta. These all integrate with features like history/auditing, RBAC, and policy as code.

Enhanced, pluggable secrets. We heard that keeping secrets from leaking in plaintext to your start files is a challenge with current infrastructure as code tools. So, we deepend the way secrets integrate with Pulumi’s engine, including transitive encryption so that, no matter where your secret goes in your program, it won’t leak in plaintext. In addition, we added the ability to use your own secrets provider for encryption, including AWS KMS, Azure KeyVault, Google Cloud KMS, and HashiCorp Vault. This ensures you get great ease-of-use — falling into the “pit of success” with secrets — while still leveraging your existing organization’s approach.

More state storage options. Although the hosted Pulumi SaaS works great for most end users, offering the right mix of convenience and control, we understand it doesn’t work for everyone. It is now possible to store state locally using the pulumi login --local command, or to configure Pulumi to store state in AWS S3, Azure Blob Storage, or Google Cloud Storage. Additionally, Pulumi Enterprise offers a self-hosted option, so that you can still benefit from the Pulumi SaaS’s features around identity, policies, and controls, while hosting it in your own cloud account or behind your firewall.

More CI/CD integrations. In partnership with customers, and in some cases the CI/CD vendors themselves, we have continued adding more CI/CD integrations, to enable automated delivery of your infrastructure. This includes Jenkins, Azure DevOps Pipelines, Octopus Deploy, Codefresh, and revised support for GitHub Actions.

Coexistence and conversion tools. We know that many teams have existing infrastructure in place. We want to make it a easy as possible to choose Pulumi and, as we’ve worked with major customers to transition away from existing infrastructure tools, we’ve built tools to help ease the adoption process. This includes the following approaches:

Coexist with existing infrastructure, including referencing existing Terraform remote state, deploying Kubernetes YAML as-is, deploying Kubernetes Helm charts, deploying AWS CloudFormation stacks, or deploying Azure Resource Manager (ARM) templates — in each case, enabling you to consume output properties from infrastructure deployed via other tools, for either temporary or permanent coexistence of Pulumi with these other tools.
Convert your existing infrastructure so that it comes under the control of Pulumi. This includes adopting existing cloud resources, no matter how they were provisioned (even if done manually by pointing and clicking in your cloud’s UI), as well as the tf2pulumi tool, which converts existing HCL programs to Pulumi, preserving your existing Terraform project and code structure.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

Support for large-scale projects. As we’ve seen bigger and bigger Pulumi projects, we have put together guidance for multi-project approaches. This includes adding the ability for stacks to reference one another, an essential building block when, for instance, one project depends on a piece of infrastructure provisioned by another project.

Lots of providers! We’ve added more than 20 new providers for managing resources across many cloud and service providers. This includes the existing core cloud providers for AWS, Azure, Google Cloud, and Kubernetes, in addition to newly added providers DigitalOcean, Linode, OpenStack, Packet, and vSphere. Modern cloud infrastructures include components managed by SaaS or other service providers alongside and connected to your core cloud infrastructure, and for those, we have added support for Aiven, CloudAMQP, Cloudflare, Consul, Datadog, DNSimple, Docker, Fastly, F5 BigIP, GitLab, Kafka, MySQL, New Relic, Okta, PostgreSQL, RabbitMQ, SignalFX, Spotinst, and Vault.

It’s impossible to overstate how big an impact you, the community, have had on Pulumi. In fact, every feature on this list was delivered in response to a real community, customer, or partner need.

2019 has been a major year for the Pulumi community by any measure. As we look forward, we anticipate even more accelerated growth and capabilities. We wish you all a wonderful New Year, and want to again thank you for the love and support — here’s to an incredible 2020!

— Joe

AWS EKS - How to Scale Your Cluster

Joe Duffy — Thu, 05 Dec 2019 00:00:00 +0000

Looking for information on how to deploy an EKS cluster? Checkout our AWS Elastic Kubernetes Service documentation.

AWS Elastic Kubernetes Service (EKS) provides a range of performance and control for dynamically scaling your Kubernetes clusters, including Managed Node Groups, Fargate, and Manually-Managed Node Groups in EC2. In this post, we’ll see how to use each of these compute options, and when to prefer one over the other in order to maximize productivity, flexibility, and control, based on your needs.

Managing EKS Workers

You now have three primary options for powering your EKS clusters:

Fargate: Let AWS Fargate intelligently manage and scale nodes automatically
Managed Node Groups: Let EKS manage and scale nodes based on declarative specifications
EC2: Manage nodes by hand using explicit groups, EC2 instances, and Auto Scaling Groups

Each option controls how your cluster’s worker nodes are provisioned, managed, and scaled. Worker nodes are what physically run the compute and host the storage used by your Kubernetes pods, and impact the cost and your cluster’s ability to meet workload demands. The spectrum of options available provides a range of flexibility, from least control to most control, respectively, over the configuration and provisioning of these worker nodes.

Abby Fuller from AWS explained these options visually and succinctly on Twitter.

Let’s now see how each of these work. To fully appreciate the new features, we will go in the reverse order — from most control and most difficult, to the least control and easiest, option.

Provisioning EKS Clusters

Running a Kubernetes cluster isn’t easy, but EKS makes the task of doing so much simpler. It offers out-of-the-box integrations with essential AWS services like IAM, EBS, Route53, and CloudWatch, so that your EKS clusters fit with your existing AWS security, storage, and monitoring practices.

It’s great to have all of the building blocks at your fingertips, and the @pulumi/aws package exposes these raw capabilities of EKS to you. We also created an EKS package to simplify common tasks, however, including creating the Kubernetes data plane, configuring VPC/CNI and subnet networking, and managing node groups. This package has been enlightened with the new features, and is what we’ll use in our examples.

Provisioning a new EKS cluster today is already as simple as a dozen lines of infrastructure as code:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");

// Create the EKS cluster itself, including a "gp2"-backed StorageClass.
const cluster = new eks.Cluster("my-cluster", {
 vpcId: vpc.vpcId,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 instanceType: "t2.medium",
 desiredCapacity: 5,
 minSize: 3,
 maxSize: 7,
 storageClasses: "gp2",
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

This program, when deployed with pulumi up, will provision an entire EKS cluster, all of its related infrastructure and internal Kubernetes resources, and then print out the kubeconfig that can be used to access the cluster afterwards:

$ pulumi up
Updating (dev):

 Type Name Status
 + pulumi:pulumi:Stack aws-ts-eks-dev created
 + ├─ awsx:network:Network vpc created
 + │ ├─ aws:ec2:Vpc vpc created
 + │ ├─ aws:ec2:InternetGateway vpc created
 + │ ├─ aws:ec2:Subnet vpc-0 created
 + │ ├─ aws:ec2:Subnet vpc-1 created
 + │ ├─ aws:ec2:RouteTable vpc created
 + │ ├─ aws:ec2:RouteTableAssociation vpc-0 created
 + │ └─ aws:ec2:RouteTableAssociation vpc-1 created
 + └─ eks:index:Cluster cluster created
 + ├─ eks:index:ServiceRole cluster-instanceRole created
 + │ ├─ aws:iam:Role cluster-instanceRole-role created
 + │ ├─ aws:iam:RolePolicyAttachment cluster-instanceRole-e1b295bd created
 + │ ├─ aws:iam:RolePolicyAttachment cluster-instanceRole-3eb088f2 created
 + │ └─ aws:iam:RolePolicyAttachment cluster-instanceRole-03516f97 created
 + ├─ eks:index:ServiceRole cluster-eksRole created
 + │ ├─ aws:iam:Role cluster-eksRole-role created
 + │ ├─ aws:iam:RolePolicyAttachment cluster-eksRole-90eb1c99 created
 + │ └─ aws:iam:RolePolicyAttachment cluster-eksRole-4b490823 created
 + ├─ pulumi-nodejs:dynamic:Resource cluster-cfnStackName created
 + ├─ aws:ec2:SecurityGroup cluster-eksClusterSecurityGroup created
 + ├─ aws:iam:InstanceProfile cluster-instanceProfile created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksClusterInternetEgressRule created
 + ├─ aws:eks:Cluster cluster-eksCluster created
 + ├─ aws:ec2:SecurityGroup cluster-nodeSecurityGroup created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksNodeInternetEgressRule created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksExtApiServerClusterIngressRule created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksClusterIngressRule created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksNodeClusterIngressRule created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksNodeIngressRule created
 + ├─ aws:ec2:LaunchConfiguration cluster-nodeLaunchConfiguration created
 + ├─ pulumi-nodejs:dynamic:Resource cluster-vpc-cni created
 + ├─ pulumi:providers:kubernetes cluster-eks-k8s created
 + ├─ kubernetes:core:ConfigMap cluster-nodeAccess created
 + ├─ kubernetes:storage.k8s.io:StorageClass cluster-gp2 created
 + ├─ aws:cloudformation:Stack cluster-nodes created
 + └─ pulumi:providers:kubernetes cluster-provider created

Outputs:
 kubeconfig: {
 apiVersion : "v1"
 clusters : [
 [0]: {
 cluster: {
 certificate-authority-data: "..."
 server : "https://312E10705C16E8095B2A79E8E76EA00D.gr7.us-west-2.eks.amazonaws.com"
 }
 name : "kubernetes"
 }
 ]
 contexts : [
 [0]: {
 context: {
 cluster: "kubernetes"
 user : "aws"
 }
 name : "aws"
 }
 ]
 current-context: "aws"
 kind : "Config"
 users : [
 [0]: {
 name: "aws"
 user: {
 exec: {
 apiVersion: "client.authentication.k8s.io/v1alpha1"
 args : [
 [0]: "token"
 [1]: "-i"
 [2]: "cluster-eksCluster-560236b"
 ]
 command : "aws-iam-authenticator"
 }
 }
 }
 ]
 }

Resources:
 + 37 created

Duration: 12m12s

As soon as this cluster is up and running, we can interact with it with kubectl, or deploy with Pulumi:

$ pulumi stack output kubeconfig > kubeconfig.yml
$ KUBECONFIG=kubeconfig.yml kubectl get nodes
NAME STATUS ROLES AGE VERSION
aws-ip-10-0-164-133.us-east-2.compute.internal Ready <none> 4m v1.14.8-eks
aws-ip-10-0-233-6.us-east-2.compute.internal Ready <none> 4m v1.14.8-eks

Manually Managing EC2 Node Groups

A fully functioning production cluster usually requires many other considerations. For those, we’ve put together a set of playbooks as part of Pulumi Crosswalk for Kubernetes that walk through how to go to production with EKS specifically, in addition to other managed Kubernetes offerings.

Notice, for instance, that we didn’t need to even provision any worker nodes. This is thanks to the eks.Cluster abstraction creating a default node pool for us, which has an auto-scaling policy that attempts to maintain the desiredCapacity while remaining within the bounds of minSize and maxSize. Often you need more explicit control over the worker nodes, however, for reasons such precise capacity, specializing compute or storage for different workload needs, and so on. This ultimately devolves into managing EC2 instances by hand, which the eks.NodeGroup class supports.

For example, this code creates a new EKS cluster, disabling the default node group — using skipDefaultNodeGroup: true, since we will create our own — and creates a single node group with more specific configuration settings: IAM role, desired instance type, scaling parameters, labels, and so on:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");

// IAM roles for the node group.
const role = new aws.iam.Role("my-cluster-ng-role", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
 Service: "ec2.amazonaws.com",
 }),
});
let counter = 0;
for (const policyArn of [
 "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
 "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
 "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
]) {
 new aws.iam.RolePolicyAttachment(`my-cluster-ng-role-policy-${counter++}`,
 { policyArn, role },
 );
}
const instanceProfile = new aws.iam.InstanceProfile("my-cluster-ng-ip", { role });

// Create an EKS cluster.
const cluster = new eks.Cluster("my-cluster", {
 skipDefaultNodeGroup: true,
 vpcId: vpc.vpcId,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 instanceRoles: [ role ],
});

// Create an AWS node group using a cluster as input.
const nodeGroup = cluster.createNodeGroup("my-cluster-ng", {
 instanceType: "t2.medium",
 desiredCapacity: 1,
 minSize: 1,
 maxSize: 2,
 labels: { "ondemand": "true" },
 instanceProfile: instanceProfile,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

There is even more you can do here, including digging deep into the individual EC2 instances and Auto Scaling Groups powering the node group. Needing to manage workers at this level, and connect them to the right IAM and networking services, can be tedious and difficult to get right. That’s where the new AWS features come into play.

Automatically Managed Node Groups

The new EKS feature Managed Node Groups simplifies the task of managing explicit pools of worker nodes, at the cost of some amount of control. The @pulumi/eks package already had many of these conveniences built-in but this is now an official feature of the AWS platform.

Managed Node Groups will automatically scale the EC2 instances powering your cluster using an Auto Scaling Group managed by EKS. This ASG also runs the latest Amazon EKS-optimized Amazon Linux 2 AMI. This is great on one hand — because updates will be applied automatically for you — but if you want control over this you will want to manage your own node groups. Finally, security groups, IAM roles, and connecting them together is handled for you.

To opt-in to using Managed Node Groups, the raw aws.eks.NodeGroup building block is available. However, we’ve enlightened the EKS package with the eks.Cluster.createManagedNodeGroup function to make it easier and to integrate with cluster provisioning.

The following example creates an EKS cluster with a single Managed Node Group. It looks similar to our explicitly managed node group earlier, but a bit simpler because we can lean on EKS to configure and scale it:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");

// IAM roles for the node group.
const role = new aws.iam.Role("my-cluster-ng-role", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
 Service: "ec2.amazonaws.com",
 }),
});
let counter = 0;
for (const policyArn of [
 "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
 "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
 "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
]) {
 new aws.iam.RolePolicyAttachment(`my-cluster-ng-role-policy-${counter++}`,
 { policyArn, role },
 );
}

// Create an EKS cluster.
const cluster = new eks.Cluster("my-cluster", {
 skipDefaultNodeGroup: true,
 vpcId: vpc.vpcId,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 instanceRoles: [ role ],
});

// Create a simple AWS managed node group using a cluster as input.
const managedNodeGroup = eks.createManagedNodeGroup("my-cluster-ng", {
 cluster: cluster,
 nodeGroupName: "aws-managed-ng1",
 nodeRoleArn: role.arn,
 labels: { "ondemand": "true" },
 tags: { "org": "pulumi" },
}, cluster);

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

This can be provisioned with a single pulumi up, just like before, but instead of the automatically created default node group, our cluster will use a Managed Node Group instead.

Note that it’s possible to mix node groups. So, if you need precise control over some groups, but not others, feel free to call createNodeGroup and createManagedNodeGroup interspersed with one another. The EKS package knows what to do.

For a full list of properties you can configure on your Managed Node Groups, please refer to the createManagedNodeGroup API docs. There are fewer options available than manually managed node groups, such as inability to supply kubelet arguments, for instance. We are giving up some control in exchange for simplicity. For complete information about EKS Managed Node Groups, see AWS’s own product documentation.

Let Fargate Manage It All

Using EKS Managed Node Groups is a great step forward to simplifying how we run our Kubernetes workloads. It still means, however, that we need to declare our compute requirements and think at the level of worker nodes. That’s where AWS Fargate support for EKS offers a huge step function in simplifying our task of managing and scaling our EKS clusters.

AWS Fargate technology is essentially “serverless compute for containers.” Instead of thinking at the level of nodes, we can let Fargate handle provisioning, scaling, and scheduling of our cluster’s worker nodes. This means we can just operate at the level of Kubernetes pods abstractions and let AWS do all of the other hard work for us!

Not only is it simpler, but as Clare Liguori explained at re:Invent, Fargate can react very quickly to load changes. Amazon has put a ton of great effort into projects like AWS Nitro and Firecracker, ultimately leading to lightning fast startup times and impressive auto-scaling capabilities. Fargate for EKS taps into all of this goodness:

Now let’s see how to use it. The key building block that enables Fargate support is something called a Fargate profile. Like eks.NodeGroups above, one of these can be allocated explicitly, if you prefer to program at the level of the raw underlying building blocks.

The EKS package, however, has been enlightened to make allocating a Fargate-powered EKS cluster as simple as saying fargate: true. All we need to do is change our original cluster definition to the following:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");

// Create the EKS cluster itself with Fargate enabled.
const cluster = new eks.Cluster("my-cluster", {
 fargate: true,
 vpcId: vpc.id,
 privateSubnetIds: vpc.privateSubnetIds,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

If we provision this with pulumi up, we’ve got a fully functioning EKS cluster running on Fargate:

$ pulumi up
...
$ pulumi stack output kubeconfig > kubeconfig.yml
$ KUBECONFIG=kubeconfig.yml kubectl get nodes
NAME STATUS ROLES AGE VERSION
fargate-ip-10-0-174-4.us-east-2.compute.internal Ready <none> 3m v1.14.8-eks
fargate-ip-10-0-266-133.us-east-2.compute.internal Ready <none> 2m v1.14.8-eks

All of the hard work around managing and scaling our cluster is left to AWS Fargate, letting us focus on our workload!

In Conclusion

In this article, we’ve seen the full range of EKS cluster management options:

Fargate, for easy auto-scaling of your cluster’s worker compute
Managed Node Groups, enabling more control, while still leveraging EKS to auto-scale
EC2, for full control over your cluster’s compute pools and nodes

We’re excited to offer support for this full range of options the same week of AWS re:Invent, including not just the building block support, but the simpler interface provided by our open source EKS package.

To get started with Pulumi and kick the tires with EKS today, check out our Getting Started guide. There are both AWS and Kubernetes versions available.

Happy Fargating!

Running AWS IAM Access Analyzer at Deployment Time

Joe Duffy — Tue, 03 Dec 2019 00:00:00 +0000

Yesterday AWS announced an exciting new feature — the AWS Identity and Access Management (IAM) Access Analyzer — a service powered by automated reasoning that detects potentially-insecure access to your AWS resources, including S3 Buckets, SQS Queues, Lambdas, and more. At the same time, Pulumi announced a new policy as code solution, CrossGuard, that validates policies at deployment time. The question is: Can IAM Access Analyzer and Pulumi CrossGuard be combined to get the best of both solutions? The answer is Yes!

Fast Forward to Security By Default

Before diving into the details of how this all works, let’s see what this looks like in action.

By default, the IAM Access Analyzer runs in your account, monitoring for new resources and creating alerts as new problems are identified. This approach does not tell you as part of a deployment that a new problem has been introduced, however. In this post, we’ll use Pulumi CrossGuard to flag deployments that have introduced a new problem, so you find out about it right away.

Let’s try to create an S3 Bucket with public-read access:

import * as aws from "@pulumi/aws";
const bucket = new aws.s3.Bucket("my-bucket", {
 acl: "public-read",
});

The acl here means the S3 Bucket is available to the Internet — sometimes what you want, but usually not — and potentially a major security catastrophe waiting to happen! This is a very easy mistake to make. How do we prevent it?

Pulumi CrossGuard can stop such things. In fact, we have an AWSGuard policy pack that will enforce the best practices across all of the AWS services, out of the box, including security, compliance, and cost-based rules. That includes checking for public S3 buckets.

As AWS makes their own platform more secure by default, however, we would love to take advantage of it as soon as new features come out. And IAM Access Analyzer seems to have some very smart smarts behind it, which we would greatly benefit from, as articulated in this research paper.

Thanks to CrossGuard’s use of full-blown programming languages, we can easily tap into the AWS SDK support for IAM Access Analyzer. The result is that if we try to deploy the above program with our custom policy pack — pulumi up --policy-pack ~/iam-analyzer-policy — we get an error:

Nice! Security catastrophe averted! Now let’s dig into how it all works.

IAM Access Analyzer, Meet CrossGuard

First thing’s first: IAM Access Analyzer is new; and Pulumi CrossGuard is new. Both of them were announced just yesterday! So a lot of what we’re going to look at will evolve in the weeks to come. Nevertheless, the result is very exciting — as soon as we saw the IAM Access Analyzer announcement, we knew there could be magic in combining the two ideas.

The approaches are different but complementary:

IAM Access Analyzer uses a single analyzer per-region that validates all resources within it. For each problem it identifies, it creates a so-called “finding” which records what policy is responsible for granting wider access to specific resources than you might want.
CrossGuard, on the other hand, operates by applying individual policy packs to specific stacks of resources. This can be configured to run automatically anytime a deployment occurs in your organization, or explicitly using the --policy-pack argument shown above.

To see how they can work in partnership with one another, we will create a new CrossGuard policy pack that uses the new IAM service. We’ll walk the code step by step, but the full code is also available on GitHub.

First, we created this new policy pack with the following commands:

mkdir ~/iam-analyzer-policy
cd ~/iam-analyzer-policy
pulumi policy new aws-typescript

This populates the ~/iam-analyzer-policy directory with the scaffolding needed to create a policy pack.

By default, the template will have installed the @pulumi/pulumi, @pulumi/policy, and @pulumi/aws packages. We need some others:

npm install aws-sdk uuid

Now let’s import everything we need at the top of our index.ts file, which is where we’ll build out our policy:

import * as AWS from "aws-sdk"
import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";
import { PolicyPack, ReportViolation, StackValidationArgs } from "@pulumi/policy";
import * as uuid from "uuid";

Next we’ll allocate our policy pack and its internal policies. We will only have a single rule at this time:

new PolicyPack("aws-typescript", {
 policies: [{
 name: "iam-access-analyzer",
 description: "Resource has IAM Access Analyzer errors.",
 enforcementLevel: "mandatory",
 validateStack: async (args: StackValidationArgs, reportViolation: ReportViolation) => {
 // interesting part goes here ...
 },
 }],
});

Notice we are using the validateStack callback to validate our resources. The alternative is to use validateResource, which runs on a single resource at a time. In this case, the stack-wide validation is easier, since we’ll be sharing a single IAM analyzer instance.

Let’s start filling out the validateStack function. First, during previews (indicated by the pulumi.runtime.isDryRun() function), we want to skip validation. The reason is that IAM Access Analyzer only runs against real resources already allocated. Perhaps in the future, it will let us query resources before they are created, but for now the best we can do is catch the mistake immediately afterwards. Thankfully, CrossGuard supports running both before provisioning, during previews, in addition to afterwards.

// If we're doing a dry-run, we skip this, because IAM Access Analyzer only analyzes actual resources.
// Ideally in the future, they will support scanning them *before* the problem is actually deployed!
if (pulumi.runtime.isDryRun()) {
 return;
}

Next, now that we know we’re doing an update, we want to create an IAM Access Analyzer SDK object that we can use to call the service:

// Create a fresh analyzer for this run.
const svc = new AWS.AccessAnalyzer({ region: aws.config.region || "us-east-1" });

Now we can provision the temporary analyzer itself that we’ll use for this validation pass:

const analyzerName = `iam-analyzer-${uuid.v4().toString()}`;
const resp = await svc.createAnalyzer({ analyzerName, type: "ACCOUNT" }).promise();
const analyzerArn = resp.arn!;
try {

Notice we’ve wrapped the subsequent code in a try block; you’ll see a finally below that deletes the temporary analyzer object.

Our illustration here creates and uses an IAM Access Analyzer with "ACCOUNT" scope, the only currently supported option at the time of this writing. Because of the restriction that there can only be one analyzer per account, per region, this may cause issues if you already have an analyzer pre-created, or need to run deployments in parallel. Depending on how the IAM Access Analyzer feature evolves, you may want to take a different approach than what is shown here.

Next up, we’ll look for all resource types supported by IAM Access Analyzer, and kick off a resource scan for each one:

 // Now, find all eligible resources, create a map of them, and kick off scans for each one.
 const analyzed: string[] = [];
 const analyzedUrns: {[arn: string]: string} = {};
 for (const res of args.resources) {
 let resourceArn: string | undefined;
 if ([ "aws:iam/role:Role", "aws:kms/key:Key", "aws:lambda/function:Function",
 "aws:lambda/layerFunction:LayerFunction", "aws:s3/bucket:Bucket", "aws:sqs/queue:Queue" ].
 indexOf(res.type) !== -1) {
 resourceArn = res.props.arn;
 }

 if (resourceArn) {
 await svc.startResourceScan({ analyzerArn, resourceArn }).promise();
 analyzed.push(resourceArn);
 analyzedUrns[resourceArn] = res.urn;
 }
 }

The startResourceScan function is asynchronous, so we’ll now need to wait until all of the requested resources have been analyzed. This requires us to poll the AWS API with appropriate pagination. We will wait up to 30 seconds for the results to converge:

 // Wait until all resources have been analyzed.
 const maxRetries = 30;
 for (let i = 0; i < maxRetries; i++) {
 const analyzed: {[arn: string]: boolean} = {};
 let nextToken: string | undefined;
 do {
 const result = await svc.listAnalyzedResources({ analyzerArn }).promise();
 for (const res of result.analyzedResources) {
 analyzed[res.resourceArn] = true;
 }
 nextToken = result.nextToken;
 } while (nextToken);

 let done = true;
 for (const arn of Object.keys(analyzedUrns)) {
 if (!analyzed[arn]) {
 done = false;
 break;
 }
 }
 if (done) {
 break;
 }

 if (i == maxRetries-1) {
 reportViolation(`Timed out waiting for IAM Access Analyzer results; is the region correct?`);
 return;
 } else {
 await new Promise((resolve) => setTimeout(resolve, 1000));
 }
 }

Now that we know the results are available, we simply have to list the detailed findings. In this example, our policy pack will only issue an error for every IAM configuration that leads to potentially unwanted public access. Each finding contains detailed information about the IAM resources causing unwanted access, the ARN of the target resource, and remediation information:

 // Now that the scan is ready, go ahead and list the findings, triggering errors as appropriate.
 let nextToken: string | undefined;
 do {
 const result = await svc.listFindings({ analyzerArn, nextToken }).promise();
 for (let finding of result.findings) {
 if (finding.isPublic && finding.resource) {
 // Only report violations for resources within this stack.
 const urn = analyzedUrns[finding.resource];
 if (urn) {
 reportViolation(
 `${JSON.stringify(finding.principal)} has ${finding.action} access to ` +
 `${finding.resourceType} ${finding.resource} (public) ${finding.error || ""} ` +
 `(message ID ${finding.id})`,
 urn);
 }
 }
 }
 nextToken = result.nextToken;
 } while (nextToken);

And finally, now that all of our scanning and results are available, we just need to clean up our temporary analyzer object:

} finally {
 // Ensure to delete the analyzer before returning.
 await svc.deleteAnalyzer({ analyzerName }).promise();
}

That’s it. Now we have a fully functioning policy pack that can be uploaded to our organization using pulumi policy publish, or run as shown earlier in the post with the --policy-pack flag to pulumi up or pulumi preview. The result?

mandatory: [iam-access-analyzer] Resource has IAM Access Analyzer errors.
{"AWS":"*"} has s3:ListBucket,s3:ListBucketByTags,s3:ListBucketMultipartUploads,s3:ListBucketVersions access to AWS::S3::Bucket arn:aws:s3:::my-bucket-298b5bc (public) (message ID ccaec287-92f3-4136-bb1f-f7a731efdf62)

Trying It Out

In this article, we’ve seen how to enforce AWS’s new IAM Access Analyzer policies at deployment time, using Pulumi’s new CrossGuard feature. This helps to catch security mistakes when you do a deployment — instead of later on, when the cost of a mistake is potentially significantly greater. Although both features are early and will evolve in the weeks to come, it’s nice to see it so easy to quickly combine two features the day after they were both announced, to advance the security of your infrastructure.

To give it a try, download and install Pulumi, grab the code from GitHub, and get started with policy as code, all using free open source.

Pulumi 2.0 Roadmap

Joe Duffy — Mon, 02 Dec 2019 00:00:00 +0000

Today we’ve published Pulumi’s 2.0 roadmap. 2.0 is the next major phase in Pulumi’s journey, and will include better productivity through languages, libraries, and tools, in addition to advanced features for teams in production. And, though we are excited to share our own thoughts, more than anything else, we’d love to hear your feedback to help make sure it’s right.

Since releasing Pulumi 1.0 in September, we have heard loud and clear that you appreciate the commitment to compatibility, as well as the completeness and stability of the platform, and we have been hard at work making sure we honor those promises.

As we look to 2.0, we are aiming to help you be more successful with modern cloud architectures, by implementing features that we’ve heard are missing or could be better, as well as doing better at the fundamentals. Although this is a “2.0” release, we do not plan any major breaking changes, so that we can continue honoring our promise to provide a stable platform to you.

Two Themes for 2.0

The two themes we have chosen for Pulumi 2.0 are:

Best in class productivity. Pulumi’s foundation of general-purpose languages has already enabled great productivity gains thanks to ecosystems of tools, sharing and reuse, and familiarity. But we have ideas for more. This includes more languages, features that bring deployments closer to the inner development loop like watch mode, more libraries of off-the-shelf patterns and practices, and the ability to test your infrastructure more easily.
Belts and suspenders for teams and enterprises. As we work with more teams and enterprises to deliver demanding many-cloud workloads to public, private, and hybrid clouds — increasingly involving Kubernetes, and often spanning developers and operations teams — we have heard about many features that would maximize Pulumi’s effectiveness and utility. This includes advanced security and compliance capabilities, more organization-wide controls and policies, and more flexible self- and on-premises hosting of the Pulumi service.

Several major 2.0 features are available to try out today and we’d love to hear what you think. The 2.0 roadmap is on GitHub so that you can easily comment or follow along as we make progress.

Trying Out 2.0 Preview Features

A number of new features are already available in “preview”; the preview tag simply means we expect possibly-significant changes to them between now and the 2.0 release.

Many features require you to set the PULUMI_EXPERIMENTAL environment variable to true before you can use them. This can be done by running export PULUMI_EXPERIMENTAL=true in your macOS or Linux shell, or by running $env:PULUMI_EXPERIMENTAL = "true" in your Windows PowerShell console.

Better Language Support

Support for Go and .NET Core (C#/F#/VB) languages is currently marked preview because there is known work before we’re comfortable calling them “done.” This includes improving the programming models to be more idiomatic for their respective ecosystems — such as stronger typing in Go instead of interface{} and StackReference support in .NET — as well as improving the documentation, availability of samples, templates, and tutorials, written in those languages.

Expect to see a steady stream of improvements between now and 2.0 being finished; to follow along, see these work items on GitHub: Go and .NET.

Policy as Code (“CrossGuard”)

Today we have released preview support for a new policy as code offering that we call CrossGuard. Just as you can write your infrastructure as code in your favorite language, so too can you now write your policies. These policies run whenever you do a Pulumi deployment, enabling CrossGuard to restrict deployments that violate security, compliance, cost, or stylistic requirements.

You can write custom policies, or use off-the-shelf policies authored by the community, including our preview AWS pack and Open Policy Agent (OPA) integration. Just as you can share your own infrastructure creations using your language’s package manager, you can share your policies too.

CrossGuard is 100% open source, enabled by pulumi up’s new --policy-pack flag, and advanced organization-wide controls are available in the Team and Enterprise Editions. To learn more, read the CrossGuard user guide; or, to dive straight in, try out the getting started tutorial.

Testing

We’ve blogged previously about how to test your infrastructure. However, we hear all the time about new scenarios and ways in which Pulumi could do a better job here, including removing many of the rough edges currently hit when testing Pulumi code.

Pulumi 2.0 will support a spectrum of testing scenarios, including unit testing, integration testing, and post-deployment runtime validation. The addition of CrossGuard also means you can validate your infrastructure configuration integrated into your existing CI/CD processes. As part of the 2.0 release, we will be publishing comprehensive guides and tutorials for testing your infrastructure.

Watch Mode

We recently added the ability to deploy changes to your applications and infrastructure continuously, straight from your IDE. This brings infrastructure as code to your core inner development loop in a major way and is guaranteed to change the way you think about the essential differences between the way you approach engineering modern cloud applications versus the requisite infrastructure.

To give watch mode a try, simply run pulumi watch. Or, to see a demo of watch mode in action, check out this video:

Self-Host and On-Premises

Although we’ve worked hard to make the Pulumi service easy and affordable to use by default, we understand that some customers can’t depend on a multi-tenanted SaaS website for their infrastructure deployments. For these customers, Pulumi already offers custom state hosting in AWS S3, Azure Blob Storage, GCP Cloud Storage, or even as a manually managed JSON file. However, by hosting the state this way, you lose out on what makes the Pulumi service so great: identity, teams, RBAC, policies, webhooks, and more.

Today we are releasing preview support for the self-hosted Pulumi service. This gives you the full capabilities of the Pulumi Enterprise SaaS, while also giving you the flexibility to run it anywhere. Self-hosted Pulumi Enterprise can run on-premises behind your firewall on a VM or in Kubernetes, in your AWS, Azure, or GCP account — virtually anywhere that can run VMs or containers. For more information, see the tutorial about how to configure self-hosted Pulumi Enterprise, or contact us for pricing information and a demo.

A Lot More …

This is just a sampling of what’s already ready to try out. Expect a lot more, including many features and stability improvements that are driven by ongoing customers and community feedback.

From Modern Infrastructure to Modern Architecture

Pulumi 1.0 gave us a solid foundation of modern infrastructure as code. As we see the things you are building with Pulumi, we understand that challenges don’t stop at infrastructure provisioning. Pulumi is also about the applications themselves, as modern cloud architectures demand infrastructure and applications to coexist in harmony and unique ways that Pulumi enables.

In addition to many great features, Pulumi 2.0 will also further advance the Pulumi Crosswalk family of offerings to bring more of these modern application architecture patterns to your fingertips. This includes support for additional languages beyond JavaScript and TypeScript, improved documentation, interactive tutorials, and more extensive libraries of patterns and practices.

Two Asks

Although a lot of 2.0 is already underway, we are still planning out the concrete roadmap and there is a lot of room for feedback. We have two asks for you:

Please give the preview features a try, and let us know what you think.
Please let us know what is missing from our list.

Please give us feedback by commenting on the GitHub tracking issue or in the Community Slack.

Next Steps

1.0 was a major milestone for us and 2.0 is going to be better. 2.0 will make modern cloud architectures even more approachable and productive to create, while also deepening support for modern teams and workloads running in production, often across many-clouds.

We want to thank you for your continued support and feedback — it’s what makes Pulumi tick!

A Year of Helping Build Production-Ready Kubernetes

Joe Duffy — Thu, 14 Nov 2019 00:00:00 +0000

Today we announced Pulumi Crosswalk for Kubernetes, a collection of open source tools, libraries, and playbooks to help developers and operators work together to bring Kubernetes into their organizations. They capture the lessons we learned this past year working with organizations to go from zero to Kubernetes in production for their infrastructure and application workloads. By releasing these as open source, we hope to help everybody be more successful with their Kubernetes projects — as we have learned through experience, it isn’t easy going!

If you want to dive straight in, browse to the documentation. Below is the backstory and why we’re so excited about this release.

Applying infrastructure as code to Kubernetes

A little over a year ago, we launched support for managing Kubernetes resources using Pulumi’s open source infrastructure as code tools. The entire Kubernetes object model across all supported versions is available in your language of choice — including JavaScript, TypeScript, and Python — in addition to the entire ecosystem of Helm Charts. This offers 100% API resource compatibility, and same-day support for newly released Kubernetes versions.

By leveraging general purpose languages, you get rich capabilities like for loops, functions, and classes, the ability to share and reuse best practices — eliminating copy and paste — as well as access to your favorite tools, including editors, test frameworks, and static analysis tools.

This is exciting for any of us who have struggled with “walls of YAML or JSON” — but this alone isn’t the whole story.

A unified approach to infrastructure as code delivers a robust delivery workflow, and enables managing configuration across not only just Kubernetes resources, but also any of the cluster’s underlying public or private cloud infrastructure, including AWS, Azure, Google Cloud, vSphere, and others.

At the time, we didn’t realize how powerful this unified approach would turn out to be. For instance, properly provisioning an Amazon EKS cluster requires not only orchestrating deployments across AWS, but also going back and forth with Kubernetes resources. We designed Pulumi to let you do this using a single consistent platform and workflow, replacing the “mountains of Bash” that are typically required to integrate multiple tools.

This was an exciting foundation to start from, and we had a number of end users and organizations who soon came to our doors to adopt Kubernetes, with our help.

Why Kubernetes in production is hard

If I had to summarize the challenges that we faced and which we worked to help solve, they are:

Kubernetes infrastructure is not an island. Although AWS, Azure, and GCP offer managed Kubernetes services, most organizations with existing infrastructure maturity find it difficult to provision and manage cluster infrastructure. This is often because going to production requires connecting a cluster to a lot of surrounding infrastructure — including IAM, networking, encryption services, private Docker registries, and more. Kubernetes-specific tools only help with a fraction of this problem and create silos.
Empowering an entire organization is tough. Most organizations adopting Kubernetes are attempting to harmonize the “two sides of the house” – developers and operators. The infrastructure operations team is responsible for managing the cluster and underlying infrastructure, while the developers are trying to deliver applications and services to the cluster (often also using other public cloud services, like object stores). Although the Kubernetes architecture is well-suited to this, teams struggle to standardize on tools, workflows, and practices around both infrastructure and application delivery.
Day Two is even harder than Day One. Many teams decide to go “all in” on Kubernetes … only to find out that securing, scaling, and operationalizing Kubernetes is far more of a full-time job than they imagined. Can you perform zero-downtime upgrades? How do you perform failover between regions? How fast can you promote new container images from dev, to staging, to dozens of production clusters?

Adding to the complexity, these challenges tend to differ depending on which cloud environment you’re targeting. Although Kubernetes has helped to standardize the container compute layer, and what it means to scale out a load balanced service, for example, it does not abstract away all of the infrastructure capabilities between the clouds, like IAM, container registries, and data services.

We experienced this first hand with many customers over the past year. There are three specific areas we’ve worked to develop into turnkey solutions for customers:

Infrastructure playbooks
Making Kubernetes more accessible to developers
Operationalizing delivery and observability insights at-scale

Let me tell you about each of these in turn.

Developing playbooks for cluster infrastructures

Most practitioners are left to discover solutions to the hardest problems with Kubernetes infrastructure on their own. After helping dozens of customers go to production with AWS EKS, Azure AKS, Google Cloud GKE, and on-premises Kubernetes infrastructures, we decided to develop best practices into a set of playbooks. This helps organizations to avoid reinventing the wheel, in addition to mitigating potential pitfalls around security, reliability, and maintainability.

These playbooks cover a lot of ground, including topics like:

Control planes. Each Kubernetes cluster has a control plane comprised of controllers, state management, and other centralized services. Each provider offers a distinct and unique way of creating, deploying, and managing the control plane.
Worker nodes. Each cluster also requires worker nodes to actually run compute. Depending on your compute, storage, and security needs, the provisioning and management of these nodes — often using multiple node pools — differs greatly. Getting this right is essential to attaining a good tradeoff between cost and performance.
Identity. All of the above also needs to integrate with your team’s security policies, including Identity and Access Management (IAM) and Role Based Access Controls (RBAC). If you’re in AWS, you undoubtedly want to connect to AWS IAM; if you’re in Azure, you want to leverage existing ActiveDirectory settings; if you’re in GCP, you’ll want to use GCP IAM; and if you’re using a custom configuration, it can be even more complicated. This is an essential part of deploying Kubernetes within an organization.
Cluster services. Each cluster needs cluster-wide services deployed into the cluster, including performance and monitoring services (AWS CloudWatch, Azure Log Analytics and Monitoring, Datadog, Prometheus, etc.), service meshes, container registries, CRDs, and operators.
Application delivery. Applications need to be Dockerized, packaged up and deployed to private container registries, and rolled out to the cluster, often continuously.
Application services. In addition to the applications themselves, often cloud native applications require additional services to come along for the ride. This includes ingress controllers, DNS and certificate management services, and more. Upgrading clusters. Last but not least, cluster upgrades are an essential part of the overall playbook. Kubernetes is fast-moving, so upgrades are a regular occurrence, and yet it’s not always easy to deploy upgrades with confidence, especially when you are worried about zero downtime and persistent workloads.

This layer cake of concerns can be visualized using the following diagram:

We are confident that anybody looking to create production-ready infrastructure in any of the public clouds, or even on-premises, will benefit from this material. It is also open source so we hope to improve and evolve it in partnership with the community in the weeks to come.

Making Kubernetes more accessible to developers

Besides having to navigate walls of YAML, there is a significant amount of repetition on most idiomatic Kubernetes configurations. In fact, it often feels like hand-authoring Kubernetes YAML is akin to writing software in assembly language, rather than higher level languages.

Pulumi exposes 100% of the Kubernetes API, which means you can use modern language practices to start taming some of this low-level chaos. But new library constructs available in Crosswalk take this to another level by eliminating common boilerplate patterns.

For example, this wall of YAML:

apiVersion: v1
kind: ConfigMap
data:
 config: very important data
metadata:
 name: app-config
---
apiVersion: v1
kind: Secret
data:
 app-password: JikkMz9mK2hIRDo3
 database-password: Y29uZmlnLmRhdGFiYXNlUGFzc3dvcmQ=
metadata:
 name: app-secrets
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx
spec:
 replicas: 3
 selector:
 matchLabels:
 app: nginx
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - env:
 - name: APP_CONFIG_PATH
 value: /app/config
 - name: APP_USER
 value: config.user
 - name: APP_PASSWORD
 valueFrom:
 secretKeyRef:
 key: app-password
 name: app-secrets
 - name: APP_DATABASE_PASSWORD
 valueFrom:
 secretKeyRef:
 key: database-password
 name: app-secrets
 image: nginx
 imagePullPolicy: Always
 name: nginx
 ports:
 - containerPort: 80
 name: http
 volumeMounts:
 - mountPath: /app/config
 name: app-config-rsg88x4g
 restartPolicy: Always
 volumes:
 - configMap:
 name: app-config
 name: app-config
---
apiVersion: v1
kind: Service
metadata:
 name: nginx
spec:
 ports:
 - name: http
 port: 80
 targetPort: 80
 selector:
 app: nginx
 type: LoadBalancer

Is distilled down to its essence, with far less ceremony and repetition

// Define the application configuration and secrets.
const configs = new kx.ConfigMap("app-config", {
 data: { "config": "very important data" }
});

const secrets = new kx.Secret("app-secrets", {
 stringData: {
 "app-password": new kx.RandomPassword("app-password"),
 "database-password": config.databasePassword
 }
});

// Define the application Pod.
const appConfigPath = "/app/config";
const appPod = new kx.PodBuilder({
 containers: [{
 image: "app:1.0.0",
 env: {
 "APP_CONFIG_PATH": appConfigPath,
 "APP_USER": config.user,
 "APP_PASSWORD": secrets.asEnvValue("app-password"),
 "APP_DATABASE_PASSWORD": secrets.asEnvValue("database-password"),
 },
 volumeMounts: [
 configs.mount(appConfigPath)
 ],
 }],
});

// Create a Kubernetes Deployment using the previous Pod definition.
const deployment = new kx.Deployment("nginx", {
 spec: appPod.asDeploymentSpec({ replicas: 3 }),
});

// Expose the Deployment with a load balancer using a Kubernetes Service.
const service = deployment.createService({
 type: kx.types.ServiceType.LoadBalancer,
});

These library extensions have been designed to live alongside your existing Kubernetes configuration, so that you can gradually reduce complexity, while the result still feels familiar and integrated.

In addition to making Kubernetes configuration more accessible in these ways, there are other benefits to application configuration that we have seen emerge. This includes:

Data Services. For many of us, managing persistent workloads in Kubernetes is not worth the hassle. Although it’s possible to manage the delicate dance of tainting, draining, and migration, many of our customers rely on hosted offerings for data stores — such as Amazon S3, RDS, Azure Cosmos DB or Google Cloud BigTable — and Kubernetes for stateless compute. Using managed services brings the advantage of intrinsic scaling, backups, and management, without the tedium. Pulumi lets you mix and match Kubernetes and public cloud infrastructure as code.
Container Registries. Most end users need to use private container registries to host application images. Furthermore, most of us want to use our cloud provider’s native offering, such as Amazon Elastic Container Registry, Azure Container Registry, or Google Container Registry, because it integrates with IAM. Using Pulumi, you don’t need separate deployment pipelines for containers compared to the supporting application infrastructure.
Serverless. If you want to use the best and most cost effective serverless capabilities, you likely want to use AWS Lambda, Azure Functions, or Google Cloud Functions. We see end users augmenting container-based applications with serverless extensions all the time.
Architecture as Code. Thanks to Pulumi’s use of general-purpose languages for infrastructure as code, end users have been able to abstract and encapsulate complexity into reusable forms, such as functions and classes. This helps you go beyond even the basic Kubernetes configuration challenges that Kx solves, to codifying recurring patterns in your cloud native architectures.

In addition to all of that, we’ve been hard at work on a new experimental “watch mode” feature which, in conjunction with the above, brings all of Kubernetes to your fingertips in a radically new way. We’re excited to show this off at our booth at KubeCon next week!

Operationalizing delivery at-scale

After you’ve mastered the infrastructure and application authoring challenges, the difficulties move to Day Two and beyond. At this stage, organizations need to figure out how to deliver changes continuously to an ever-increasing number of environments.

Since our initial launch last year, we have added many different CI/CD integrations, including support for GitLab, Codefresh, Azure DevOps, Octopus Deploy, GitHub Actions, and more. This enables delivery at global scale. In fact, one of our customers continuously deploys to 80 different environments across their development and infrastructure operations teams.

Finally, we are excited to release a new technology in preview today, a new query command, and library of Kubernetes queries, that leverages the Pulumi object model to ask operational queries about your cluster and its applications. This includes queries like “How many distinct versions of MySQL are running in my cluster?”, “Which Pods are exposed to the Internet via a load-balanced Service?”, and so on.

For instance, this query shows the distinct versions of MySQL inside your cluster:

import * as kq from "@pulumi/query-kubernetes";

// Find all distinct versions of MySQL running in your cluster.
const mySqlVersions = kq
 .list("v1", "Pod")
 .flatMap(pod => pod.spec.containers)
 .map(container => container.image)
 .filter(imageName => imageName.includes("mysql"))
 .distinct();

mySqlVersions.forEach(console.log);

CQL also supports live streaming queries, which the CLI will display results from in real time. Read more and get started with Pulumi Query for Kubernetes here.

Join a growing community

Everything above is open source and has been battle-tested with many end users and organizations over the last year — all the way from startups to ISVs to the largest enterprises in the world. We are excited to share these best practices in reusable forms, leveraging the foundation uniquely provided by Pulumi’s infrastructure as code platform using familiar languages.

To get started, check out the documentation, or get started with the open source SDK now.

We sincerely hope that your team can be successful in your Kubernetes journey as a result of these lessons learned from others. Additionally, Pulumi offers training and services, so if we can help in any way, please join our entire team and thousands of practitioners on our Community Slack — or simply contact us and we’ll be in touch.

Pulumi 💜 .NET Core

Joe Duffy — Mon, 11 Nov 2019 00:00:00 +0000

Today we are excited to announce the Preview of .NET Core support for all of your modern infrastructure as code needs. This means you can create, deploy, and manage your infrastructure, on any cloud, using your favorite .NET language, including C#, F#, and VB.NET.

This is a heartwarming milestone for the entire Pulumi team. It’s no secret that several of our founding members had a hand in shaping the early .NET project over a decade ago. In fact, before leaving Microsoft to found Pulumi, I had the privilege of helping to kick off the .NET Core effort to take .NET open source and add macOS and Linux support alongside Windows. It’s truly wonderful to build on top of this amazing and vibrant ecosystem, and reconnect with old friends!

Please head on over to the .NET blog to read more, or check out our new .NET Core landing page if you can’t wait to dive in.

True to the spirit of Pulumi, our .NET support and the associated libraries are open source on GitHub. We are releasing .NET support in preview first so we can gather feedback over the next few weeks and make sure everything is just right. Join us in our Community Slack — we can’t wait to hear what you think!

Manage DigitalOcean Kubernetes Clusters and Workloads

Joe Duffy — Mon, 30 Sep 2019 00:00:00 +0000

We recently partnered with DigitalOcean to publish a new tutorial, How to Manage DigitalOcean and Kubernetes Infrastructure with Pulumi. This short tutorial walks you through provisioning a new DigitalOcean Kubernetes cluster, deploying an application to it, and then assigninging a stable domain name to your application’s load balancer — all in a handful of lines of infrastructure as code.

By using infrastructure as code to provision and update your infrastructure, it’s easy to create new environments, modify or scale existing ones, or automate your deployments using continuous delivery. This approach also ensures that deployments are reliable and repeatable, and can even standardize how you deploy code to different cloud providers, including DigitalOcean, AWS, Azure, GCP, and others.

We regularly talk to developers who appreciate the simplicity of working with DigitalOcean. This new tutorial spins up an entire, fully functioning managed Kubernetes environment in just 6 minutes, for instance, giving you more time to focus on application logic and less on your infrastructure configuration. In addition to managing Kubernetes clusters and applications, you can also provision Droplets, LoadBalancers, and any other DigitalOcean resource type, using a similar infrastructure as code approach. For more information on doing so, read Getting Started on DigitalOcean with Pulumi.

Finally, for more detailed information on all of the DigitalOcean resources supported, please consult the DigitalOcean API docs.

Most importantly, have fun getting up and running with DigitalOcean Kubernetes in just a few minutes!

Pulumi 1.0

Joe Duffy — Thu, 05 Sep 2019 00:00:00 +0000

This blog post announces Pulumi 1.0, which has been superseded by newer versions. For information about the latest Pulumi release, please refer to the current documentation and recent release notes.

Today we are excited to announce the general availability of Pulumi 1.0. Pulumi is a modern infrastructure as code tool that lets you declare infrastructure using familiar, general-purpose languages, with a SaaS management console for configuring identities, organizations, and related policies. By using familiar languages, developers and operators are able to work better together, sharing and reusing best practices, accomplishing new levels of automation, and unlocking access to ecosystems of existing tools. The 1.0 release is a significant milestone for us, our community, and our customers, and signals completeness, stability, and compatibility.

What is Pulumi?

Using Pulumi, you:

Declare infrastructure in TypeScript, JavaScript, Python, or Go.
Leverage tools such as libraries, package managers, IDEs, and test frameworks from your language of choice.
Access all of the services from many clouds, including AWS, Azure, GCP, Kubernetes, and DigitalOcean.
Use the Pulumi CLI or CI/CD integrations to perform deployments predictably and reliably.
Build modern, reliable, and scalable applications using a multitude of cloud architectures, including containers, serverless functions, and static websites.
Use higher level frameworks that abstract away tedious aspects of cloud development.

This functionality is delivered through an open source SDK consisting of a CLI and assorted libraries. Pulumi also provides a hosted service for improved usability, reliability, security, and other management features.

Additional Features

In addition to the core infrastructure as code features, Pulumi’s SDK offers:

Stack management: deploy to and promote between multiple environments.
State storage: use the free Pulumi service or choose instead to manually manage state in AWS S3, Azure Storage, Google Cloud Storage, or the filesystem.
Configuration: easily manage different configuration between environments.
Encrypted secrets: use the free Pulumi KMS service to encrypt sensitive configuration, such as passwords or tokens, or plug in AWS KMS, Azure KeyVault, GCP KMS, or HashiCorp Vault.

Using Pulumi in conjunction with the Pulumi service delivers the following additional features:

Pulumi Service: a dashboard for gaining visibility into and managing your deployments.
Identity: user and organization identity provider integrations with GitHub, GitLab, Atlassian, or SAML/SSO, including Active Directory, Okta, Google G Suite, and others.
Role based access controls: team management, also integrated with your identity provider.
REST API and Webhooks: a fully programmable web API for advanced automation scenarios.
CI/CD Integrations: automated deployment integrations with Azure DevOps, AWS Code Services, CircleCI, Codefresh, GitHub, GitLab, Google Cloud Build, Jenkins, Travis, and others.

The Pulumi Service is available as a free Individual Edition as well as advanced Team and Enterprise SaaS editions for larger teams looking for robust enterprise capabilities.

Why Pulumi?

Using familiar languages offers significant advantages for declaring your infrastructure:

Control flow: use if statements, for loops, case statements, etc.
Abstractions and reuse: leverage classes and functions to avoid repeating the same boilerplate over and over again
Package managers: share libraries with your team or the community
IDEs: use your favorite IDE and get statement completion, live error checking, and interactive documentation
Project organization: use modules and well known techniques for managing complex projects
Testing: apply standard test tools, frameworks, and techniques to infrastructure
Code review: use existing code review practices for infrastructure changes

This approach unlocks access to entire ecosystems that aren’t available to markup DSLs. This approach also helps developers and operators work better together using a shared foundation.

How Pulumi Works

Although Pulumi uses general-purpose imperative languages, it is declarative. The program simply declares a set of resources and the Pulumi engine orchestrates all CRUD operations.

Packages in your language of choice provide full access to many clouds, including AWS, Azure, GCP, Kubernetes, DigitalOcean, and more. Each package projects the full set of services from the underlying cloud provider as classes and APIs. Declaring a new S3 bucket is as easy as:

let b = new aws.s3.Bucket("my-bucket", { acl: "private" });

Because of higher level components, even creating a new Kubernetes cluster is this easy:

let cluster = new eks.Cluster("my-cluster");
export const kubeconfig = cluster.kubeconfig;

The pulumi up CLI command runs this program using Pulumi’s multi-language engine, and presents a plan for review — including an overview of the resources to create, update, and/or delete. After you choose to apply those changes, the Pulumi engine coordinates those API calls in parallel to accomplish your deployment.

To see this workflow in action, watch this short 5 minute video:

How to Get It

The Pulumi 1.0 milestone signals completeness, stability, and compatibility, and is the result of two years of work helping 1,000s of customers take modern cloud architectures across many clouds into production. This includes startups and Global 2000s looking to innovate faster using the cloud.

If you’re already a Pulumi user, simply upgrade in the usual ways.

If this is your first time using Pulumi, give it a try today. It is open source and free to use.

For more information about Pulumi Team and Enterprise editions, visit our products page.

We’d like to sincerely thank our community and customers for coming along on this journey with us. Without you, we couldn’t have reached this milestone. We hope you enjoy Pulumi 1.0!

Unit Testing Your Infrastructure with Node.js and Mocha

Joe Duffy — Sat, 13 Jul 2019 00:00:00 +0000

Some parts of this blog post are out-of-date. Please refer to our Testing Guide for the updated overview and tutorials.

Testing your infrastructure using familiar tools like Node.js’s Mocha framework allows you to ensure configuration is correct before provisioning, and that the resulting infrastructure has certain desirable properties afterwards. This can enforce team standards, ensure security guidelines are being followed, and so much more. Because Pulumi uses general purpose languages, you can just embed tests alongside your infrastructure-as-code definitions themselves, using a familiar authoring style and reporting experience. In this post, we’ll explore the ins and outs of unit testing your infrastructure.

Test-Driven Infrastructure

We previously explored many reasons and solutions for testing your infrastructure. In this post, we’ll see a very simple approach that leverages existing test tools and frameworks. For this blog post, we’ll be using Node.js with TypeScript, the Mocha test framework, and the Chai assertion library.

Our example will be testing two things about an Amazon EKS cluster; that it is

Running Kubernetes version 1.13.
Provisioned inside a private VPC, rather than the default one.

In true test-driven development (TDD) form, let’s start with the tests themselves. If you’re familiar with Mocha and Chai, this should look like ordinary testing to you. If you’re not familiar with them, don’t worry – they are quite easy to read and learn: Mocha and Chai.

Here is the contents of our tests/eks.spec.ts test file:

import * as aws from "@pulumi/aws";
import { expect } from "chai";
import * as pulumi from "@pulumi/pulumi";
import { promise } from "./";
import { cluster } from "../";

describe("#Amazon EKS Cluster", () => {
 it("Should use an approved Kubernetes version", async () => {
 const version = await promise(cluster.eksCluster.version);
 expect(version).to.equal("1.13");
 });
 it("Should be provisioned inside a custom, non-default VPC", async () => {
 const vpcId = await promise(cluster.core.vpcId);
 if (vpcId) {
 const defaultVpc = await aws.ec2.getVpc({ default: true });
 expect(vpcId).not.equals(defaultVpc.id);
 } else if (pulumi.runtime.isDryRun()) {
 pulumi.log.warn("Skipped VPC ID check: not known during preview -- will test after apply");
 } else {
 throw new Error("VPC unknown during deployment");
 }
 });
});

This code does a few things worth describing. First, it imports all the packages that we’re going to use. Notably, this includes the normal Chai package for assertions, the AWS and Pulumi SDK packages, the EKS cluster provisioned by this project itself, and a utility function promise that we’ll dive into momentarily. Then it simply uses the Mocha functions describe and it to register test cases that will be run.

The version test fetches a property from our EKS cluster, cluster.eksCluster.version, and then uses the Chai assertion expect(version).to.equal("1.13"). This will fail the test if the version is anything but 1.13 (including if it is unknown).

The VPC test is slightly more complex. For the most part, it is simply checking the cluster.core.vpcId property and ensuring it doesn’t equal the default VPC ID, which we fetch with aws.ec2.getVpc. However, because we might be creating the custom VPC in the program itself, it’s possible we won’t know the ID during previews. (A preview is when Pulumi shows you a dry-run of your deployment without actually doing it yet.) That’s why we check pulumi.runtime.isDryRun and let things slide, with a little warning message. We could always be conservative and fail the test, but in this case, we’ll actually permit the deployment to proceed – but ensure we check it afterwards. This means we might fail the test after the bad cluster has been deployed, but that’s better than not checking at all.

Our Base (Failing) Program

Here is the program we’ll be testing. Keeping with our TDD theme, let’s start with the tests failing to begin with (we are using the default VPC and not specifying a version):

import * as eks from "@pulumi/eks";

// Create a basic EKS cluster.
export const cluster = new eks.Cluster("my-cluster", {
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 storageClasses: "gp2",
 deployDashboard: false,
});

If you want to create a fresh Pulumi project and follow along, simply install the CLI, ensure it is configured for your AWS account, and run pulumi new aws-typescript to create an empty project. Swap out the contents of index.ts with the above.

Before we add our tests, we can run a pulumi preview and all will work just fine:

$ pulumi preview
Previewing update (tdd-infra):

 Type Name Plan
 + pulumi:pulumi:Stack infra-eks-tdd-infra create
 + └─ eks:index:Cluster my-cluster create
 + ├─ eks:index:ServiceRole my-cluster-eksRole create
 + │ ├─ aws:iam:Role my-cluster-eksRole-role create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-eksRole-90eb1c99 create
 + │ └─ aws:iam:RolePolicyAttachment my-cluster-eksRole-4b490823 create
 + ├─ eks:index:ServiceRole my-cluster-instanceRole create
 + │ ├─ aws:iam:Role my-cluster-instanceRole-role create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-03516f97 create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-e1b295bd create
 + │ └─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-3eb088f2 create
 + ├─ pulumi-nodejs:dynamic:Resource my-cluster-cfnStackName create
 + ├─ aws:iam:InstanceProfile my-cluster-instanceProfile create
 + ├─ aws:ec2:SecurityGroup my-cluster-eksClusterSecurityGroup create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksClusterInternetEgressRule create
 + ├─ aws:eks:Cluster my-cluster-eksCluster create
 + ├─ pulumi:providers:kubernetes my-cluster-eks-k8s create
 + ├─ pulumi-nodejs:dynamic:Resource my-cluster-vpc-cni create
 + ├─ kubernetes:core:ConfigMap my-cluster-nodeAccess create
 + ├─ aws:ec2:SecurityGroup my-cluster-nodeSecurityGroup create
 + ├─ kubernetes:storage.k8s.io:StorageClass my-cluster-gp2 create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksExtApiServerClusterIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksClusterIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeInternetEgressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeClusterIngressRule create
 + ├─ aws:ec2:LaunchConfiguration my-cluster-nodeLaunchConfiguration create
 + ├─ aws:cloudformation:Stack my-cluster-nodes create
 + └─ pulumi:providers:kubernetes my-cluster-provider create

Resources:
 + 29 to create

As usual, this shows us all of the resources that Pulumi would provision were we to proceed with a deployment. No tests yet, however.

Adding the Test Harness Boilerplate

Now let’s hook up the tests so they run anytime we do a pulumi preview or pulumi up.

Note that we won’t be running these tests using the mocha CLI. Instead, we will run Mocha programmatically. This lets us run the tests inside of our Pulumi program where all of the necessary functionality is available. It also automatically coordinates test execution with previews and updates.

First, create a test file, tests/index.ts, that runs Mocha. This is basic boilerplate that registers all of the test files in the tests/ directory and will work for any project:

import * as fs from "fs";
import * as Mocha from "mocha";
import * as path from "path";
import * as pulumi from "@pulumi/pulumi";

// runTests executes all test files (*.ts) in the current directory.
export function runTests() {
 // Create a new Mocha test runner (with a long timeout).
 const mocha = new Mocha({ timeout: 1000*60*30 });

 // Only keep the .ts files, and skip this file (index.ts).
 const testDir = __dirname;
 fs.readdirSync(testDir).
 filter(file => file.endsWith(".ts") && file !== "index.ts").
 forEach(file => { mocha.addFile(path.join(testDir, file)); });

 // Now actually run the tests with the desired reporter.
 console.log(`Running Mocha Tests: ${mocha.files}`);
 mocha.reporter("spec").run(failures => {
 process.exitCode = failures ? 1 : 0;
 });
}

// promise returns a resource output's value, even if it's undefined.
export function promise<T>(output: pulumi.Output<T>): Promise<T | undefined> {
 return (output as any).promise() as Promise<T>;
}

The runTests function creates an instance of the Mocha runner, registers all the test files, configures the desired reporter, and runs the tests. We adjust the timeout to be 30 minutes, since provisioning things can take some time and we don’t want Mocha to timeout. For complete information on the options available, see the Mocha documentation; effectively, if you can do it with the mocha CLI, there is a way to do it using the API.

This file also defines the promise function used to access resource properties in our tests. We’ll see why this is needed shortly.

Running the Tests

Now that we have runTests, let’s go back and add a call to it in our primary index.ts file. First we need to import our custom function at the top of the file:

import { runTests } from "./tests";

And now we’ll add a call to runTests at the very bottom of the file:

// Run deployment validation tests.
runTests();

That’s it. Now when we run pulumi up, we will see that our tests are running!

$ pulumi up
Previewing update (tdd-infra):

 Type Name Plan Info
 + pulumi:pulumi:Stack infra-eks-tdd-infra create 1 error; 19 messages
 + └─ eks:index:Cluster my-cluster create
 + ├─ eks:index:ServiceRole my-cluster-eksRole create
 + │ ├─ aws:iam:Role my-cluster-eksRole-role create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-eksRole-4b490823 create
 + │ └─ aws:iam:RolePolicyAttachment my-cluster-eksRole-90eb1c99 create
 + ├─ eks:index:ServiceRole my-cluster-instanceRole create
 + │ ├─ aws:iam:Role my-cluster-instanceRole-role create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-03516f97 create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-e1b295bd create
 + │ └─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-3eb088f2 create
 + ├─ pulumi-nodejs:dynamic:Resource my-cluster-cfnStackName create
 + ├─ aws:iam:InstanceProfile my-cluster-instanceProfile create
 + ├─ aws:ec2:SecurityGroup my-cluster-eksClusterSecurityGroup create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksClusterInternetEgressRule create
 + ├─ aws:eks:Cluster my-cluster-eksCluster create
 + ├─ pulumi:providers:kubernetes my-cluster-eks-k8s create
 + ├─ aws:ec2:SecurityGroup my-cluster-nodeSecurityGroup create
 + ├─ kubernetes:core:ConfigMap my-cluster-nodeAccess create
 + ├─ kubernetes:storage.k8s.io:StorageClass my-cluster-gp2 create
 + ├─ pulumi-nodejs:dynamic:Resource my-cluster-vpc-cni create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksClusterIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeClusterIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeInternetEgressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksExtApiServerClusterIngressRule create
 + ├─ aws:ec2:LaunchConfiguration my-cluster-nodeLaunchConfiguration create
 + ├─ aws:cloudformation:Stack my-cluster-nodes create
 + └─ pulumi:providers:kubernetes my-cluster-provider create

Diagnostics:
 pulumi:pulumi:Stack (infra-eks-tdd-infra):
 Running Mocha Tests: infra-eks/tests/eks.spec.ts
 #Amazon EKS Cluster
 1) Should use an approved Kubernetes version
 2) Should be provisioned inside a custom, non-default VPC
 0 passing (9s)
 2 failing
 1) #Amazon EKS Cluster
 Should use an approved Kubernetes version:
 AssertionError: expected undefined to equal '1.13'
 at Object.<anonymous> (tests/eks.spec.ts:19:35)
 at Generator.next (<anonymous>)
 at fulfilled (tests/eks.spec.ts:4:58)
 2) #Amazon EKS Cluster
 Should be provisioned inside a custom, non-default VPC:
 AssertionError: expected 'vpc-c93b06ae' to not equal 'vpc-c93b06ae'
 + expected - actual
 at Object.<anonymous> (tests/eks.spec.ts:25:38)
 at Generator.next (<anonymous>)
 at fulfilled (tests/eks.spec.ts:4:58)

 error: an unhandled error occurred: Program exited with non-zero exit code: 1

As expected, our tests fail for two reasons:

The Kubernetes version is undefined and not 1.13.
The VPC ID is equal to the default VPC, since the EKS cluster uses that by default.

Fixing Our Program

Now let’s refactor our infrastructure so the tests start passing.

To fix the first problem, we simply need to pass the Kubernetes version explicitly when creating our cluster. That’s as simple as passing a new argument:

export const cluster = new eks.Cluster("my-cluster", {
 ...
 version: "1.13",
});

Note that we could have done what we did with the VPC ID, and change the test to verify that the version is correct after provisioning. It turns out that at the time of this writing, the default version is in fact 1.13, so our test would then pass. But we’d be depending on a brittle default that could change at any moment – so it’s probably better for us to be explicit about this!

If we rerun pulumi up, we’ll see that this test now passes – huzzah:

Diagnostics:
 pulumi:pulumi:Stack (infra-eks-tdd-infra):
 Running Mocha Tests: infra-eks/tests/eks.spec.ts
 #Amazon EKS Cluster
 ✓ Should use an approved Kubernetes version (3704ms)
 1) Should be provisioned inside a custom, non-default VPC
 1 passing (4s)
 1 failing
 1) #Amazon EKS Cluster
 Should be provisioned inside a custom, non-default VPC:
 AssertionError: expected 'vpc-c93b06ae' to not equal 'vpc-c93b06ae'
 + expected - actual
 at Object.<anonymous> (tests/eks.spec.ts:25:38)
 at Generator.next (<anonymous>)
 at fulfilled (tests/eks.spec.ts:4:58)

 error: an unhandled error occurred: Program exited with non-zero exit code: 1

But the VPC test still fails. Let’s fix that one by creating a custom VPC – the awsx.ec2.Vpc component makes this easy – and then passing its resulting ID and subnet IDs as arguments.

const vpc = new awsx.ec2.Vpc("my-vpc");
export const cluster = new eks.Cluster("my-cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.publicSubnetIds,
 ...
});

For reference, here is the complete corrected index.ts file we have now:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import { runTests } from './tests';

// Create a basic EKS cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");
export const cluster = new eks.Cluster("my-cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.publicSubnetIds,
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 storageClasses: "gp2",
 deployDashboard: false,
 version: "1.13",
});

// Run deployment validation.
runTests();

Now if we rerun pulumi up, we will see that the tests pass during the preview, landing us at the prompt asking us whether to deploy our infrastructure:

$ pulumi up
Previewing update (tdd-infra):

 Type Name Plan
 + pulumi:pulumi:Stack infra-eks-tdd-infra create
 + ├─ awsx:x:ec2:Vpc my-vpc create
...
 + └─ eks:index:Cluster my-cluster create
...

Diagnostics:
 pulumi:pulumi:Stack (infra-eks-tdd-infra):
 Running Mocha Tests: infra-eks/tests/eks.spec.ts
 #Amazon EKS Cluster
 ✓ Should use an approved Kubernetes version (839ms)
 ✓ Should be provisioned inside a custom, non-default VPC
 2 passing (928ms)

 warning: Skipped VPC ID check: not known during preview -- will test after apply

Resources:
 + 59 to create

Do you want to perform this update?
 yes
> no
 details

They passed! Notice, however, that the VPC test was skipped. That’s because we can’t know whether the resulting ID is going to end up resolving to the default VPC or not. There are myriad ways that the program could have supplied the ID, including by querying and looking up the default VPC ID dynamically and passing it as an argument.

If we choose yes and deploy the changes, we will a more confidently passing test:

Updating (tdd-infra):

 Type Name Status
 + pulumi:pulumi:Stack infra-eks-tdd-infra created
 + ├─ awsx:x:ec2:Vpc my-vpc created
...
 + └─ eks:index:Cluster my-cluster created
...

Diagnostics:
 pulumi:pulumi:Stack (infra-eks-tdd-infra):
 Running Mocha Tests: infra-eks/tests/eks.spec.ts
 #Amazon EKS Cluster
 ✓ Should use an approved Kubernetes version (666477ms)
 ✓ Should be provisioned inside a custom, non-default VPC (917ms)
 2 passing (11m)

Resources:
 + 59 created

Duration: 12m40s

Voila! A successfully stood up EKS cluster, with built-in TDD security safeguards.

Keeping Track of Failures

Anytime we try to deploy a program that fails, Pulumi is keeping a record of it.

For instance, let’s say somehow the VPC wasn’t known at preview time, and ended up resolving to the default VPC. Even though the deployment occurred, we’d want to know that we now have an issue that we need to go track down.

We can very clearly see such failures in the Pulumi Service:

And if we select it, we’ll see the complete Mocha test output:

We could even use webhooks to fire off a Slack alarm so that nobody misses the issue. This is often very helpful in unattended scenarios, like continuous deployment.

Better to catch these things late than never!

About Resource Properties and Outputs

A brief aside on resource properties is worthwhile, because they are “special” in some important ways when writing test cases.

All properties on resources are of type Output<T>. This behaves a lot like a promise, in that the value isn’t readily available to your program. Instead, you use the apply function to get at it, and Pulumi calls this when your resource’s properties are known.

Sometimes, however, properties cannot be known. A good example is the VPC ID we’re using above. Amazon will automatically assign a VPC ID when the resource is provisioned. That means that during previews, we won’t actually have a known value – it doesn’t even exist yet! We somehow need a way to deal with these situations.

The Output<T>.apply function tries to shield you from these circumstances. It would be maddening to need to check inside of these functions whether a value is known or not, and then you’d have all sorts of conditional code that differs between previews and updates – generally an anti-pattern, because the entire point of a preview is to show you exactly what will be done during an update. Any differences run the risk of showing different results.

Tests are special, however, in that you will need to think about what tests make sense to run before an update, versus those that should run after an update.

That’s why we need the promise<T> function defined earlier. It uses an internal promise() property to get at the raw underlying promise for any resource property, and returns it so you can either await it or call then in the usual promise ways. This does mean we aren’t hidden from the details of whether a preview is running or not.

This function, or functionality like it, will be made available as an official SDK feature as soon as pulumi/pulumi#2287 has been resolved.

By using this function in our tests, we have the following cases to consider:

The value is known, which could be due to the following cases
- It was explicitly supplied in our program
- It was a default populated by the underlying resource provider during a preview
- We are doing an update and the full outputs are available
The value is undefined, which could be due to the following cases
- We are in a preview and the value hasn’t been computed yet
- We have done an update and the property was not made available as an output value

In all cases, the pulumi.runtime.isDryRun function will return true during a preview.

In general, it’s easiest to write tests that just run at deployment time, rather than during a preview, when all of these different states are possible. That has the downside to not catching problems before they get deployed, however. As with many things, this is a tradeoff.

For a more complete overview of Output<T>, please see the Pulumi Programming Model documentation.

Next Steps

In this post, we’ve seen how to test your infrastructure using familiar test driven development (TDD) techniques and frameworks like Node.js’s Mocha and Chai libraries. This is a powerful capability that builds atop Pulumi’s use of general purpose languages, letting you apply existing software development best practices to your infrastructure.

Although we’ve just scratched the surface with our EKS cluster example, this capability can be used to enforce a wide array of team standards, best practices, and security guidelines. If you can express it in your favorite test framework, you can check it!

Everything we’ve seen in this article is open source and free to use – give Pulumi a try today.

ECS vs Fargate vs EKS: The Lowdown on Containers in AWS

Joe Duffy — Thu, 20 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

An updated breakdown of when to prefer Fargate and when EC2 can be found in Fargate vs EC2

Amazon offers multiple solutions for running containers in AWS, through its managed Elastic Container Service (ECS). This includes three major approaches: ECS managed automatically with Fargate, ECS backed by EC2 instances, and Elastic Kubernetes Service (EKS), delivering the full power of Kubernetes. It's not always easy to choose between these, so in this article we provide some basic guidance on the tradeoffs you'll encounter when choosing.

The Options: ECS vs Fargate vs EKS

At Pulumi, we work with customers to deploy AWS applications all the time, from early development to scaling production environments worldwide, using infrastructure as code and often continuous delivery. During this, we’ve run across a breadth of container workloads and requirements, and developed best practices, which this article distills into some basic advice. As with any advice, the right answer depends on your scenario’s specific requirements.

Here are the three major options, roughly speaking from easiest to hardest:

Option 1 - ECS Fargate

Amazon ECS with Fargate launch type allows you to deploy and scale your containerized workloads, automatically, without needing to manage clusters of EC2 instances by hand. You focus on your Dockerized applications, with minimal extra declarative metadata about the requirements of those applications, and not the details of the underlying compute itself. One could say this is “serverless containers.” Of course, there are servers running your containers, however the ECS control plane makes informed decisions on your behalf about when to add capacity, where, and by how much.

To run a workload in ECS, you use a standard Dockerfile to author, build, and publish your container image, and then run that image inside your ECS cluster. Tasks and services are authored using ECS definitions, which specify the container image to run, instance count, and any CPU, memory, networking, and disk requirements your workload has.

Using Pulumi, these definitions can be easily expressed declaratively in code, like so:

import * as awsx from "@pulumi/awsx";

// Create a load balancer on port 80 and spin up two instances of Nginx.
const lb = new awsx.lb.ApplicationListener("nginx", { port: 80 });
const nginx = new awsx.ecs.FargateService("nginx", {
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: "nginx",
 memory: 128,
 portMappings: [ lb ],
 },
 },
 },
 desiredCount: 2,
});

// Export the load balancer's address so that it's easy to access.
export const url = lb.endpoint.hostname;

This example runs 2 load balanced instances of an NGINX web server using ECS Fargate.

ECS Fargate is often a great choice when getting started, when you don’t have stringent requirements around cost, and/or if you’re a small to medium sized organization who prefers simpler solutions. Many teams are successful running it scale too, however Fargate doesn’t give you as much control over persistent storage, privileged containers, custom scheduling, or special VM types like GPUs or FPGAs. For those, EC2 backed ECS is a better choice.

Option 2 - ECS Backed by EC2 Instances

Amazon ECS with EC2 launch type gives you similar capabilities to ECS managed by Fargate – in that you can deploy containerized workloads, declaratively specify their requirements, and let ECS do the placement and overall scheduling of them – but with full control over the compute environment those containers run in. That includes control over the precise numbers of servers, AMIs they are running, and their auto-scaling policies.

To use an EC2 backed cluster, you will need to create a cluster and configure the EC2 launch and autoscaling configurations, and then simply use the awsx.ecs.EC2Service class to create your service instead. Other than these differences, the resulting code would look very similar to Option 1 - ECS Fargate above.

For sophisticated AWS organizations who already know how to fine tune and run compute at scale efficiently and cost effectively, managing your ECS cluster’s EC2 instances explicitly is often a good choice. The good news is that you can easily start with Fargate, and then over time, shift to managing the EC2 compute by hand if you prefer. Managing your cluster amounts to managing fleets of EC2 instances, CloudWatch logging, and standard AWS services.

Option 3 - Elastic Kubernetes Service (EKS)

Amazon Elastic Kubernetes Service (EKS) offers a very different approach to running containerized workloads than ECS Fargate or EC2. EKS is a fully managed offering that runs Kubernetes inside of your AWS account, making it easier to operate Kubernetes, in addition to integrating with many AWS services like ELB for load balancing, IAM for cluster authentication, and CloudWatch for logging and diagnostics. Using EKS, you don’t need to explicitly provision, manage, and upgrade the Kubernetes control plane’s configuration, compute, and storage.

Kubernetes is an open standard for running containers across a variety of public and private cloud environments. Most public cloud vendors now offer an EKS-like managed offering, including Azure (AKS), Google Cloud (GKE), and Digital Ocean, and the availability of on-premises private cloud configurations enables hybrid cloud for large organizations. Kubernetes in inherently more complicated to operate than the equivalent ECS solutions mentioned earlier, because it requires becoming an expert in Kubernetes, which requires more specialization in its tools and management practices. But the potential payoff is larger for organizations wanting to go “all in” on cloud native technologies to facilitate either multi-cloud or hybrid computing – challenges most large enterprises are facing today.

In the EKS case, as with ECS, we continue using standard Dockerfiles, however we use the Kubernetes object model to configure and run workloads. Instead of task definitions, you use Kubernetes objects like Namespaces, Pods, Deployments, and Services. This full API is available to specify in code using Pulumi, like so:

import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster");

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: "nginx",
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the URL for the load balanced service.
export const url = service.status.loadBalancer.ingress[0].hostname;

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Like the ECS example earlier, this spins up 2 load balanced instances of the NGINX web server.

You will find that the entirety of Kubernetes is more feature rich and offers more control over scheduling and scaling your workloads, but with that richness also comes added complexity that will increase training, and time to ramp up, in addition to ongoing maintenance costs. To learn more about Pulumi’s support for Kubernetes, go here.

Private ECR Container Registries

In all of these cases, Amazon Elastic Container Registry (ECR) lets you publish Docker container images to a privately hosted repository inside your AWS account. The benefit of this is that images are stored within your AWS account, where access is governed by IAM and, where they are stored closer to your workloads (maximizing push/pull efficiency). This is an alternative approach to, say, publishing images to the Docker Hub which entails a different authentication and pushing and pulling images over the Internet, where performance can be an issue.

Next Steps

Pulumi Crosswalk for AWS supports three main options for running containers in AWS – ECS Fargate, ECS with EC2 instances, and EKS – each of which integrates deeply with other AWS services like IAM, ELB, and CloudWatch. It’s possible to use any of these services without Pulumi, but there are many benefits to Pulumi’s infrastructure as code.

In short, to get your containers up on AWS, in an easy yet production-ready way, get started for free with Pulumi for AWS today!

After getting started, here are some additional resources to go deeper on specific topics:

Happy Birthday to Pulumi open source!

Joe Duffy — Tue, 18 Jun 2019 00:00:00 +0000

One year ago today – on June 18, 2018 – we open sourced Pulumi, a new approach to multi-cloud infrastructure as code using your favorite languages. And what a year it has been!

Here are some highlights we’ve added in partnership with the community since launching:

Over 100 examples, tutorials, and a brand new Getting Started guide.
A native Kubernetes provider with 100% Kubernetes API/version coverage.
A steady stream of improvements across AWS, Azure, and Google Cloud providers.
Pulumi Crosswalk for AWS, a framework with built-in AWS infrastructure best practices.
Over 20 additional providers, including CloudFlare, Digital Ocean, and MySQL.
Brought our Python 3 SDK to parity with our Node.js-based JavaScript and TypeScript SDKs.
Team and Enterprise SaaS editions for teams managing infrastructure in production.
GitHub, GitLab, Atlassian, and SAML/SSO identity providers.
CI/CD integrations with GitHub, GitLab, Codefresh, CircleCI, major clouds, and more.
Pluggable secrets management and transitive state encryption.
Pluggable state backends for AWS S3, Azure Blob Store, and Google Cloud Store.
Tools for managing complex, multi-stack environments, including Terraform integration.
Numerous engine reliability and performance improvements, including parallelism.
75 blogs, increasingly focused on end to end solutions we see working with customers.

In addition to the steady stream of product improvements, the community has grown fast:

1,000s of cloud engineers trying out Pulumi across over 1,000 companies.
1,000 end users collaborating in our Community Slack – including emerging MVPs!
Open source contributions from over 100 community members.
Nearly 5,000 commits, across over 100 open source Git repositories.

What we’ve enjoyed most is hearing about the innovation you, the end user and customer, have accomplished thanks to Pulumi’s unique approach. These experiences range from enabling better collaboration between application and infrastructure engineers; vastly improved productivity; more secure, robust, and reliable delivery pipelines; and an improved ability to go to production in multiple clouds, including by leveraging the cloud native Kubernetes stack.

A few notable success stories include:

Mercedes-Benz R&D North America used Python infrastructure as code to break down team silos between developers and infrastructure engineers.
Tableau was able to get up and running with Kubernetes in AWS, using Amazon EKS, and make progress on their journey to bringing continuous delivery to their organization.
Learning Machine went from 25KLOC YAML to 500 lines of JavaScript, got containers running in AWS and Kubernetes, and halved their cloud bill along the way. Read more.
Mapbox built a scalable IoT tracking solution with serverless Lambdas. Read more.
Multiple early stage startups, including Jargon, a TechStars company, have used Pulumi to get up and running quickly in the cloud – doing in days what used to take weeks.
Dozens of more great stories that we can’t wait to share in the weeks to come!

We want to sincerely thank our community and our customers and partners – without you, the year wouldn’t have nearly been as successful. The best part is that we’re only just getting started. Expect another year of more innovation, growing community engagement, customer success, and more focus on end to end solutions borne out of experience.

Whether it’s adopting infrastructure as code on your favorite cloud, deploying containers to Kubernetes, or architecting and executing on your multi-cloud journey, Pulumi is here for you!

Sincerely,

Joe and the Pulumi Team

Get Started with Docker on AWS Fargate using Pulumi

Joe Duffy — Tue, 30 Apr 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Update: Check out the Learning Machine Case Study where provisioning went from 3 weeks to 1 hour with Pulumi and AWS.

“The impact of serverless capabilities was also transformative for the Learning Machine business. Pulumi enabled a rapid shift to Amazon ECS, AWS Fargate and AWS Lambda — the net effect of which was a 67% reduction in AWS charges. This enabled the team to spend less time focused on maintaining existing infrastructure and more time deploying new applications on AWS and adding new customers.

Pulumi is the foundational technology that allowed us to transform our organization,” said Hughes. The entire DevOps process was streamlined and in addition to realizing better productivity and higher quality, the team has new insight into their SaaS offering that they never thought possible.”

The Docker Getting Started tutorial shows how to develop, build, and run a modern containerized application, from a single custom Docker container published to the Docker Hub, to a scaled out service with load balancing. But there are challenges: it requires you to program in YAML, run (or script) many CLI commands, and manage your own Swarm or Kubernetes cluster. There is an easier way. By using Pulumi’s infrastructure as code, we can build a custom Docker image, publish it to a private AWS container registry, and spin up an AWS Fargate load balanced service running that container, all in 28 lines of TypeScript code and a single pulumi up command. The result leverages the best of what AWS has to offer, with the entire platform at your fingertips, with a single approach. In this article, we’ll see how.

Defining Everything in Code

The Docker Getting Started tutorial goes from a simple developer desktop scenario to a fully scaled out service in production. There is a lot of ceremony, however: YAML, many CLI commands, running your own Swarm cluster by hand, etc. This is complicated and doesn’t lend itself to repeatable deployments across many environments.

What if we could just write 28 lines of TypeScript instead, and accomplish the same end result, while leveraging the latest innovations in the AWS platform, natively?

In fact, we can. This article will show you how Pulumi’s infrastructure as code solution enables exactly this.

*Note: Everything we’re about to define will be in a single TypeScript file, index.ts. Other languages are available, including using plain Node.js JavaScript. Pulumi also supports other clouds, including Azure and GCP, however to fully exploit the power of the underlying AWS platform, these examples will leverage native capabilities like VPCs, ALBs, and Fargate.

To cut to the chase, here is the entire program. We’ll spend the rest of the post walking through it line by line:

import * as awsx from "@pulumi/awsx";

// Step 1: Create an ECS Fargate cluster.
const cluster = new awsx.ecs.Cluster("cluster");

// Step 2: Define the Networking for our service.
const alb = new awsx.lb.ApplicationLoadBalancer(
    "net-lb", { external: true, securityGroup: cluster.securityGroup });
const web = alb.createListener("web", { port: 80, external: true });

// Step 3: Build and publish a Docker image to a private ECR registry.
const img = awsx.ecs.Image.fromPath("app-img", "./app");

// Step 4: Create a Fargate service task that can scale out.
const appService = new awsx.ecs.FargateService("app-svc", {
    cluster,
    taskDefinitionArgs: {
        container: {
            image: img,
            cpu: 102 /*10% of 1024*/,
            memory: 50 /*MB*/,
            portMappings: [ web ],
        },
    },
    desiredCount: 5,
});

// Step 5: Export the Internet address for the service.
export const url = web.endpoint.hostname;

Step 1. Create a Cluster

The opening stanza imports Pulumi’s open source AWSX NPM package, @pulumi/awsx. It contains high level AWS best practices and patterns, and leverages real languages to eliminate boilerplate YAML templating:

import * as awsx from "@pulumi/awsx";

To provision an entire Elastic Container Service (ECS) cluster, AWS’s native container orchestration technology, we just new one up in a single line of code:

const cluster = new awsx.ecs.Cluster("cluster");

This will use the default Virtual Private Cloud (VPC) in our account. If we want to customize this aspect along with the auto-scaling policies, we can easily override the defaults by passing the appropriate arguments.

Step 2. Define the Networking

Next we need to think about the networking for our desired application. First, we will need a load balancer, as we’re going to create a scaled-out application and want a fixed address. Second, we will want to expose port 80 to the Internet, for incoming traffic. This leads to these two lines:

const alb = new awsx.lb.ApplicationLoadBalancer(
    "net-lb", { external: true, securityGroups: cluster.securityGroups });
const web = lb.createListener("web", { port: 80, external: true });

An Elastic Application Load Balancer (ALB) in AWS is a fully featured Layer 7 load balancer, with advanced features around SSL termination, content based routing, and HTTP/2. The AWSX package enables you to whip together simple routing when appropriate, while still having the ability to dig deeper into the advanced capabilities as you scale your workload to handle more sophisticated scenarios. Simple things simple, hard things possible!

Step 3. Build and Publish the Container Image

The next line declares a Docker image:

const img = awsx.ecs.Image.fromPath("app-img", "./app");

This single line is quite the work horse, and does a lot. Let’s unpack it a bit.

First and foremost, this allocates a private Elastic Container Registry (ECR) repository for your Docker image. This is AWS’s hosted Docker registry solution, privately encapsulated inside your AWS account.

But there is more: this also builds the Dockerfile underneath ./app and publishes the result to said registry, each time you run pulumi up! Any Dockerfile will work. This means you get a single command line to both provision and update your infrastructure, in addition to publishing and rolling out your images.

Step 4. Create a Scale-Out Service

Next up, let’s deploy our published image as a containerized service, behind our load balanced network on port 80, using our ECS Fargate cluster. Fargate is a managed container orchestrator that lets us skip the messy details of installing and managing Swarm on our own. This effectively replaces the docker-compose.yml from the Docker Getting Started tutorial, with a similarly simple sequence of code, and which gives us full access to the AWS platform:

const appService = new awsx.ecs.FargateService("app-svc", {
    cluster,
    taskDefinitionArgs: {
        container: {
            image: img,
            cpu: 128 /*~10%*/,
            memory: 50 /*MB*/,
            portMappings: [ web ],
        },
    },
    desiredCount: 5,
});

Notice that img just refers to the Docker image built in Step 3, and the portMappings structure references our load balancer’s listener defined in Step 2. Pulumi tracks all the dependencies so that it knows how to perform deployment activities in the right order. The task definition structure here exposes all of the underlying AWS ECS capabilities.

Step 5. Deploy the App and Kick the Tires!

On to the final step! Let’s deploy our infrastructure and application, and kick the tires.

The last line of code exposes the load balancer address so it’s easy to access:

export const url = web.endpoint.hostname;

The net result is that, with this project defined with the 28 lines of code above, we can simply run pulumi up; this command will show us a preview of the changes it will make and then, once confirmed, deploy everything to our AWS account, and hand back our service’s hostname:

After deploying everything, you’ve got a fully functioning application. To give it a try, let’s grab the URL exported by our program from the stack output command, and curl it:

$ curl http://$(pulumi stack output url)
<h3>Hello World!</h3>
<b>Hostname:</b> ip-172-31-39-18.ec2.internal<br/>
<b>Visits:</b> <i>cannot connect to Redis, counter disabled</i>

Or open it in our web browser:

$ open http://$(pulumi stack output url)

As an added bonus, the entire time we are doing these deployments, the Pulumi Service is keeping track of deployment state, to ensure everything works reliably and in a team setting, similar to how Git works with GitHub. For example, I can see a history of my deployments:

Congratulations!

In this article, you’ve seen how to take 28 lines of code, and turn them, plus a Dockerfile, into a fully functioning AWS microservice, including the infrastructure and application resources. No YAML required, no complex sequence of CLI commands, and no manual cluster installation.

Pulumi’s “everything is code” approach means deploying everything can be done with a CLI command, unlocking the power of the entire Docker platform with a great inner development loop, that works from the desktop all the way to production. This entire flow can be easily integrated into your favorite CI/CD pipeline, including GitOps workflows.

If we want to augment our service with other AWS resources – like S3 buckets, Aurora or RDS databases, Lambda functions, CloudWatch dashboards, etc – they can be provisioned and managed the same was created our clusters and load balancers: just new them up!

Remember that all of this not only works for native AWS ECS containers, but also Azure and GCP cloud providers, in addition to Kubernetes!

Want to go deeper?

Announcing Per User Pricing and Unlimited Stacks for Teams

Joe Duffy — Fri, 19 Apr 2019 00:00:00 +0000

This blog post is outdated and no longer accurate. We have updated our pricing structure in response to user feedback. Learn all about it in our new blog post: Announcing New Usage-Based Pricing For Your Whole Team. Or visit our Pricing Page for more details about pricing.

Since launching last year, thousands of users and hundreds of companies, from startups to Fortune 500 Enterprises, have chosen Pulumi for cloud applications and infrastructure delivery across AWS, Azure, Google Cloud, and Kubernetes. Today we are announcing important changes to better align our product and pricing with how we’ve heard you want to use Pulumi in production. We’re optimistic that these changes will help companies of all sizes choose Pulumi, enabling their teams to deliver cloud applications and infrastructure faster and more reliably.

The net result is three paid editions:

Team Starter Edition: for teams just getting started with Pulumi
Team Pro Edition: for medium to large teams using Pulumi at scale
Enterprise Edition: for large organizations using Pulumi with advanced needs

Pulumi Community Edition remains unchanged, and is free for unlimited individual use.

In addition to the product edition changes, we are moving away from per stack pricing. We heard clear feedback from customers that per stack pricing made it difficult to plan and budget for future growth, and that it unintentionally penalized certain architectural decisions. The new per user model should be more affordable and predictable.

Here is a summary of the resulting options. For full details, view our pricing page.

TEAM STARTER	TEAM PRO	ENTERPRISE
For teams just getting started with cloud applications and infrastructure.	For medium to large teams using Pulumi for multiple projects or clouds.	For large organizations operating at scale or with advanced custom needs.
Includes Any Cloud Includes 3 users Up to 20 stacks Basic secrets management CI/CD Integrations	Everything in Team Starter plus… Up to 25 users Unlimited projects and stacks Webhooks Advanced secrets management 12x5 support available	Everything in Team Pro plus… Teams, roles, and policies SAML/SSO On-premeses/self-host available 24x7 support available

All editions offer a 14-day free trial, no credit card required. If you need a longer trial period, want to discuss potential proof of concept projects, or are interested in advanced capabilities, such as large numbers of seats, Training, or Support, please contact us.

We remain committed to fostering a vibrant open source community while also delivering a robust production service that teams of all sizes are choosing for their cloud needs. From startups, to medium sized organizations, to the largest Fortune 500 customers, we truly believe that these new editions now offer a great option for everyone.

We hope you agree and look forward to seeing the great things your team builds with Pulumi!

FAQ

Here are some common frequently asked questions about the pricing structure:

How do I get started?

To get started, simply sign up for free using your identity provider of choice (ideally the same one your organization will be backed by). From there, create an organization that your team members will use. To use the free Community Edition, just download the CLI and sign into the free tier of the service when it prompts you.

What is an organization?

An organization is a shared workspace for your business to collaborate across multiple projects at once. Organizations may be backed by your existing source control system (GitHub, GitLab, Atlassian), identity provider (any SAML/SSO provider, including Okta and Active Directory), or managed manually. Organization owners and administrators can manage access to the organization’s projects with security and administrative features.

The key distinction between Pulumi’s free Community Edition and its paid offerings — Team Starter, Team Pro, and Enterprise — is the presence of an organization. The Community Edition is meant for individuals using Pulumi for their private projects, but this tends not to work in a team setting. Most teams want multiple engineers to have access to their projects. This is precisely what organizations deliver to you. The advanced tiers offer even more sophisticated organization management facilities, including RBAC for advanced policy controls.

Can I start small and upgrade later?

Yes! We designed the editions to make it easy to get started with Team Starter, and once you’ve outgrown it, upgrading to Team Pro is a single click away. To upgrade to Enterprise, please contact us.

I’m an existing customer on a per stack plan – what do I do?

We are in the process of reaching out to all existing customers to offer a switch to one of the new plans. If you haven’t heard from us yet, please don’t hesitate to drop us a line. If now isn’t the right time to change for your team, however, don’t worry — we are happy to honor your existing subscription terms.

Do you offer custom pricing for large teams?

We are always happy to discuss the best way to ensure Pulumi can work for your team. To talk with a leader at the company, please simply fill out the contact us form and we’ll be in touch.

What payment options do you accept?

All subscriptions can be paid with a credit card (we use Stripe for processing). The Pulumi Service is meant to be easy to sign up for, so feel free to simply begin your trial, and then you may enter your credit card on your organization’s Settings page.

For annual subscriptions, we also offer invoicing that is payable with bank transfer or check. To discuss those options in more detail, please contact us.

Testing Your Infrastructure as Code with Pulumi

Joe Duffy — Wed, 17 Apr 2019 00:00:00 +0000

Some parts of this blog post are out-of-date. Please refer to our Testing Guide for the updated overview and tutorials.

Using Pulumi and general purpose languages for infrastructure as code comes with many benefits: leveraging existing skills and knowledge, eliminating boilerplate through abstraction, and using the same ecosystem of tools like IDEs and linters that your team already knows and loves. In general, these are all attributes of software engineering, which not only make us more productive, but also improve the quality of our code. It’s only natural, therefore, that using general purpose languages unlocks another important software engineering practice: testing.

In this article, we will see the many ways in which Pulumi lets us test our infrastructure as code.

Why Test Your Infrastructure?

Before jumping into the details, it’s worth asking: Why test your infrastructure, anyway? There are many reasons to do this, however here are a few examples:

Unit testing individual functions or pieces of logic in your program
Ensuring that the program’s desired infrastructure state meets certain constraints
Catching common mistakes, like forgetting to encrypt your storage buckets, or exposing virtual machines unprotected to the open Internet
Ensuring that the provisioned infrastructure has certain desired attributes
Doing runtime testing of the application logic that is running inside of your compute infrastructure, to ensure that it is healthy after your infrastructure has been provisioned

As we see, there’s a broad spectrum of infrastructure testing options available. Pulumi has mechanisms available for testing at each point along this spectrum. Let’s dig in and see how.

Unit Testing

Pulumi programs are authored in a general purpose language like JavaScript, Python, TypeScript, or Go. The full power of each language is available, including access to tools and libraries for that runtime, including testing frameworks. Pulumi is also multi-cloud, meaning the full underlying platform for each cloud is available too, and therefore available for testing.

(Despite being multi-language and multi-cloud, for the purposes of this article let’s assume we are using JavaScript, with Mocha for testing, and targeting AWS. Python unittest, Go’s test framework, or any of your favorite unit test frameworks in your language of choice would do, and of course Pulumi works great with Azure, Google Cloud, and Kubernetes, among others.)

As we’ve seen, there are several reasons you may want to test your infrastructure code. One of them is basic unit testing. Because your code may have functions – say, to compute CIDR blocks, dynamically compute names and tags, etc. – you probably want to test them. Doing this is the same as writing ordinary unit tests against any program in your language as choice.

Going one step further, you may want to test the way your program allocates resources. To illustrate, let’s imagine we want to create a simple EC2-based webserver, but want to ensure:

Instances have a Name tag.
Instances must not use an inline userData script – we must use an AMI (image).
Instances must not have SSH open to the Internet.

Our running example is loosely based on our aws-js-webserver example:

index.js:

let aws = require("@pulumi/aws");

let group = new aws.ec2.SecurityGroup("web-secgrp", {
    ingress: [
 { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
    ],
});

let userData =
`#!/bin/bash echo "Hello, World!" > index.html nohup python -m SimpleHTTPServer 80 &`;

let server = new aws.ec2.Instance("web-server-www", {
    instanceType: "t2.micro",
    securityGroups: [ group.name ], // reference the group object above
    ami: "ami-c55673a0" // AMI for us-east-2 (Ohio),
    userData: userData // start a simple web server
});

exports.group = group;
exports.server = server;
exports.publicIp = server.publicIp;
exports.publicHostName = server.publicDns;

This is a basic Pulumi program: it simply allocates an EC2 security group and instance. Notice, however, that we are violating all three of the rules stated above – let’s write some tests!

Writing the Tests

The overall structure and scaffolding of our tests will look like any ordinary Mocha testing:

ec2tests.js

let assert = require("assert");
let mocha = require("mocha");
let pulumi = require("@pulumi/pulumi");
let infra = require("./index");

describe("Infrastructure", function() {
    let server = infra.server;
    describe("#server", function() {
 // TODO(check 1): Instances have a Name tag.
 // TODO(check 2): Instances must not use an inline userData script.
    });
    let group = infra.group;
    describe("#group", function() {
 // TODO(check 3): Instances must not have SSH open to the Internet.
    });
});

Now let’s implement our first test: ensuring that instances have a Name tag. To verify this we simply need to grab hold of the EC2 instance object, and check the relevant tags property:

// check 1: Instances have a Name tag.
it("must have a name tag", function(done) {
 pulumi.all([server.urn, server.tags]).apply(([urn, tags]) => {
     if (!tags || !tags["Name"]) {
     done(new Error(`Missing a name tag on server ${urn}`));
     } else {
      done();
     }
    });
});

This looks like a normal test, with a few noteworthy pieces:

Since we’re querying resource state before we’ve done a deployment, our tests are always effectively running in “plan” (or “preview”) mode. Thus, there are many properties whose values will simply not resolve yet or will be undefined. This includes any output properties computed by your cloud provider. That’s fine for these tests – we’re checking for valid inputs anyway. We’ll revisit this later when it comes to integration testing.
Because all Pulumi resource properties are “outputs” – since many of them are computed asynchronously – we need to use the apply method to get access to the values. This is a lot like promises and the then function.
Because we are using multiple properties, mostly just so we can show the resource URN in the error message, we need to use the pulumi.all function to join all of them.
Finally, since these outputs are resolved asynchronously, we need to use Mocha’s built-in asynchronous test capability, with the done callback, or by returning a promise.

After we’ve gotten through that setup, we get access to the raw inputs as plain JavaScript values. The tags property is a map, so we just make sure it is (1) not falsey, and (2) not missing an entry for the Name key in the map. This is very basic, but we can check anything!

Now let’s write our second check. It’s even easier:

// check 2: Instances must not use an inline userData script.
 it("must not use userData (use an AMI instead)", function(done) {
 pulumi.all([server.urn, server.userData]).apply(([urn, userData]) => {
 if (userData) {
 done(new Error(`Illegal use of userData on server ${urn}`));
 } else {
 done();
 }
    });
});

And finally, let’s write our third check. It’s a bit more complex because we’re searching for ingress rules associated with a security group – of which there may be many – and CIDR blocks within those ingress rules – of which there may also be many. But it’s still easy:

// check 3: Instances must not have SSH open to the Internet.
it("must not open port 22 (SSH) to the Internet", function(done) {
 pulumi.all([ group.urn, group.ingress ]).apply(([ urn, ingress ]) => {
 if (ingress.find(rule =>
 rule.fromPort == 22 && rule.cidrBlocks.find(block => block === "0.0.0.0/0"))) {
     done(new Error(`Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0) on group ${urn}`));
 } else {
     done();
 }
    });
});

That’s it – now let’s actually run the tests!

Running the Tests

For the most part, we can run the tests in the usual way in your test framework of choice. There is one thing specific to Pulumi we need to cover first, however.

Usually you run your Pulumi programs with the pulumi CLI. That configures the language runtime, monitors execution from the engine so that resource operations can be recorded and turned into a plan, and so on. There is one problem, however: When running under the watchful eye of your test framework, this connection to the CLI, and Pulumi’s engine, will not exist.

To work around this problem, we simply have to mock three pieces of information:

Your project name, which can be supplied with the PULUMI_NODEJS_PROJECT environment variable (or more generally PULUMI_<LANGUAGE>_PROJECT in other languages).
Your stack name, which can be supplied with the PULUMI_NODEJS_STACK environment variable (or more generally PULUMI_<LANGUAGE>_STACK in other languages).
Your stack configuration variables. These can be supplied using the PULUMI_CONFIG environment variable, and its format is a JSON map of configuration key/value pairs.

The program will emit some warning messages indicating the connection to the CLI/engine is not available at runtime. This is important to know, because your program won’t actually be deploying anything – which can come as a surprise if that’s not what you meant to do! To tell Pulumi that’s what you intended, you can set PULUMI_TEST_MODE to true.

Imagine we want a project name of my-ws, stack name of dev, and AWS region of us-west-2. The command line to run your Mocha tests would therefore be:

$ PULUMI_TEST_MODE=true \
 PULUMI_NODEJS_STACK="my-ws" \
 PULUMI_NODEJS_PROJECT="dev" \
 PULUMI_CONFIG='{ "aws:region": "us-west-2" }' \
 mocha tests.js

Running this will tell us that we have three failing tests, as we had hoped!

 Infrastructure
    #server
1) must have a name tag      2) must not use userData (use an AMI instead)
    #group
3) must not open port 22 (SSH) to the Internet
  0 passing (17ms)
  3 failing
 1) Infrastructure
       #server
 must have a name tag:
Error: Missing a name tag on server
urn:pulumi:my-ws::my-dev::aws:ec2/instance:Instance::web-server-www
 2) Infrastructure
       #server
 must not use userData (use an AMI instead):
Error: Illegal use of userData on server
urn:pulumi:my-ws::my-dev::aws:ec2/instance:Instance::web-server-www
 3) Infrastructure
       #group
 must not open port 22 (SSH) to the Internet:
Error: Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0) on group

Let’s fix our program to comply:

"use strict";

let aws = require("@pulumi/aws");

let group = new aws.ec2.SecurityGroup("web-secgrp", {
    ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
    ],
});

let server = new aws.ec2.Instance("web-server-www", {
    tags: { "Name": "web-server-www" },
    instanceType: "t2.micro",
    securityGroups: [ group.name ], // reference the group object above
    ami: "ami-c55673a0" // AMI for us-east-2 (Ohio),
});

exports.group = group;
exports.server = server;
exports.publicIp = server.publicIp;
exports.publicHostName = server.publicDns;

And then rerun our tests:

 Infrastructure
    #server
      ✓ must have a name tag
      ✓ must not use userData (use an AMI instead)
    #group
      ✓ must not open port 22 (SSH) to the Internet
 3 passing (16ms)

They all passed – huzzah! ✓✓✓

Deployment Testing

This style of testing is powerful and lets us do white box testing of the innards of how our infrastructure code works. This style of testing is somewhat limited in what we can check, however. The tests run on an in-memory representation of Pulumi’s deployment plan – before actually performing the deployment – and so cannot test anything the deployment itself. For that, Pulumi has an integration test framework. The two approaches work great together!

Pulumi’s integration test framework is written in Go and is how we test much of our internal code. If the earlier unit test approach took more of a white box testing approach, the integration test framework is very much black box. (There are also options for thorough internals-oriented testing.) This framework has been built to take a directory containing a full Pulumi program and drive various lifecycle operations against it – such as deploying a new stack from scratch, updating it with variations, and tearing it down afterwards, potentially multiple times. We run these in checkin suites, on a recurring basis (such as nightly), and as stress tests.

(We have an open work item which will bring similar integration testing capabilities to each native language SDK. You can use the Go integration test framework no matter the language your Pulumi program is written in, however.)

By running a program through this integration test framework, you can ensure:

Your project’s code is syntactically well formed and runs without errors.
Your stack’s configuration and secrets settings work and are interpreted correctly.
Your project can be successfully deployed to your cloud provider of choice.
Your project can be successfully updated from its starting state to N other states.
Your project can be successfully destroyed and removed from your cloud provider.

As we will see soon, you can also leverage this framework to perform runtime validation.

A Basic Integration Test

To see this in action, we’ll look from the pulumi/examples repo, since our team and community here at Pulumi use that to validate our own examples in pull requests, checkins, and nightlies.

The following is a simplified test of our example that provisions an S3 bucket and some objects in Pulumi:

example_test.go:

package test

import (
    "os"
    "path"
    "testing"

    "github.com/pulumi/pulumi/pkg/testing/integration"
)

func TestExamples(t *testing.T) {
    awsRegion := os.Getenv("AWS_REGION")
    if awsRegion == "" {
 awsRegion = "us-west-1"
    }
    cwd, _ := os.Getwd()
    integration.ProgramTest(t, &integration.ProgramTestOptions{
 Quick: true,
 SkipRefresh: true,
 Dir: path.Join(cwd, "..", "..", "aws-js-s3-folder"),
 Config: map[string]string{
     "aws:region": awsRegion,
 },
    })
}

This test runs through a basic lifecycle of stack creation, updating, and destruction, for the aws-js-s3-folder example. It takes about a minute or so to report success:

$ go test .
PASS
ok ... 43.993s

There are many, many options to control the behavior of these tests. For a full set of options, see the ProgramTestOptions data structure. For instance, you can configure a Jaeger endpoint for tracing (Tracing), tell the harness to expect failure for negative testing (ExpectFailure), apply a series of “edits” to the program for a sequence of update state transitions (EditDirs), and more. Let’s see how to use these to do validate that we deployed what we expected.

Validating the Shape of Resources

All of the above integration ensures that our program “works” – as in, it does not crash. But what if we want to validate properties of the resulting stack? For example, that certain kinds of resources did (or did not) get provisioned, and that they have certain attributes.

The ExtraRuntimeValidation option for ProgramTestOptions allows us to look at the post-deployment state that Pulumi has captured, so that we can do extra validation on it. This includes a full snapshot of the resulting stack’s state, including configuration, exported outputs, all resources and their property values, and all inter-resource dependencies.

To see a basic example of this, let’s check that our program creates a single S3 Bucket:

integration.ProgramTest(t, &integration.ProgramTestOptions{
 // as before...
 ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
     var foundBuckets int
     for _, res := range stack.Deployment.Resources {
 if res.Type == "aws:s3/bucket:Bucket" {
      foundBuckets++
 }
     }
     assert.Equal(t, 1, foundBuckets, "Expected to find a single AWS S3 Bucket")
 },
})

Now when we run go test, it will not only run through the battery of lifecycle tests but will also, after successfully standing up the stack, perform the extra validation against the resulting state.

Runtime Testing

All of the testing thus far has been exclusively about deployment behavior and the Pulumi resource model. What if you want to test that your provisioned infrastructure is actually working, however? For example, that our VM is alive, our S3 bucket contains what we expect, and so on.

You may have already guessed how to accomplish some of this: the ExtraRuntimeValidation option for ProgramTestOptions is the perfect opportunity to do this. At this point, you are running an arbitrary Go test, with access to the full state of your program’s resources. This state includes information such as VM IP addresses, URLs, and everything you need to actually interact with your resulting cloud applications and infrastructure to perform verification.

For instance, our test program above exports the bucket’s websiteEndpoint property, as the stack output named websiteUrl, which is a full URL that we can use to fetch the bucket’s configured index document. Although we could just as well have dug through the state file to find the bucket and read this property directly, many times our stacks will export useful properties like this that we can use conveniently for verification:

    integration.ProgramTest(t, &integration.ProgramTestOptions{
     // as before ...
 ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
     url := "http://" + stack.Outputs["websiteUrl"].(string)
     resp, err := http.Get(url)
     if !assert.NoError(t, err) {
 return
     }
     if !assert.Equal(t, 200, resp.StatusCode) {
 return
     }
     defer resp.Body.Close()
     body, err := ioutil.ReadAll(resp.Body)
     if !assert.NoError(t, err) {
 return
     }
     assert.Contains(t, string(body), "Hello, Pulumi!")
 },
})

Like our previous runtime validation checks, this will run in the harness right after the stack is stood up, all in response to a simple go test invocation. This just scratches the surface of what’s possible – anything you can write in code using Go’s test capabilities is within reach!

Continuous Infrastructure Integration

It’s wonderful to be able to easily run these on demand, on a laptop, when you’re iterating on infrastructure changes and want to verify them before submitting for code review. But we and many of our customers run infrastructure testing at various points in the development lifecycle:

In each pull request opened, to validate before allowing it to be merged.
In response to each commit, to double check it was merged correctly.
On a recurring basis, such as nighty or even weekly for extended testing.
As part of more exotic and thorough test programs, such as performance or stress testing, which typically run over a longer period of time, execute many things in parallel, and/or deploy the same program multiple times.

For each of these, Pulumi supports integration with your favorite continuous integration system. This gives you similar continuous integration testing coverage for your infrastructure as you might already enjoy for your application software or in other programming domains.

Pulumi supports your existing CI systems. Here are a few of those supported:

Please refer to the Continuous Delivery documentation for a more comprehensive guide.

Ephemeral Environments

A very powerful capability this unlocks is the ability to spin up ephemeral environments solely for purposes of acceptance testing. Pulumi’s concept of projects and stacks is designed to make it very easy to stand up entirely isolated and independent environments, and to tear them down, all in either a few easy CLI gestures, or by using the integration testing framework.

If you are using GitHub, Pulumi offers a GitHub App that helps to glue together your Pull Request workflow with this sort of acceptance testing run inside of your CI pipelines. Simply install the App into your GitHub repos, and Pulumi in your CI, and your Pull Requests will light up with infrastructure previews, updates, and test results seamlessly:

By leveraging Pulumi for your core acceptance test workflow, you’ll unlock new automation capabilities that improve your team’s productivity and confidence in the quality of changes.

In Summary

In this post, we saw that by using general purpose programming languages, we unlock many software engineering practices we’ve benefited from in our application development. This includes unit testing, integration testing, and even the intersection of both to do extensive runtime testing. It’s easy to run these on demand and harness them in your CI system of choice.

Pulumi is open source, free to use, and works with your favorite languages and clouds – give it a try today!

Easy Serverless Apps and Infrastructure

Joe Duffy — Mon, 11 Mar 2019 00:00:00 +0000

With Pulumi, you can create, deploy, and manage any cloud resource using your favorite language. This includes application and infrastructure related resources, often in the same program.

One area this gets really fun is serverless computing. Because we’re using general purpose languages, we can create resources, and then wire up event handlers, just like normal event-driven programming. This is the way serverless architecture should be!

In this article, we’ll see how. There’s a broad range of options depending on what you want to do, and how your team likes to operate. We’ll be using Amazon Web Services (AWS) and TypeScript, but other clouds and languages are available.

A Simple Serverless App

In this article, we’ll use a very simple serverless app. We have two S3 buckets: one for TPS reports, and the other to store archived zipfiles of those reports. Anytime a new report comes in, our Lambda will zip and archive it.

Simple enough. This can be visualized as follows:

All source code for this example is available on GitHub.

Event Sources and Event Handlers

Serverless app models today make you think of the event sources – the S3 buckets – and event handlers – the Lambdas and associated code – as very different things – “infrastructure” versus “app code” – managed with distinct tools and workflows. Pulumi, in contrast, gives you a single CLI, pulumi, to manage everything consistently.

You can create event sources and handlers in the same program, but Pulumi’s approach is flexible, allowing you to ease into serverless without boiling the ocean. You can choose to wire functions up to existing resources (say, if your team already created the buckets), or even use functions your team has already made.

Also, because Pulumi provisions and manages resources, updating your functions after creating them is easy. Just edit your code, run pulumi up, and Pulumi will diff and compute the minimal set of changes it can make to upgrade your code – no downtime required. We’ll see this in action shortly. But first let’s create our initial app.

Creating an Event Handler

There are multiple approaches to creating a function. Let’s start with the easiest.

Approach 1: Magic Functions

Our favorite way to create functions is to write them inline. This blurs the line between infrastructure and application logic in a way that makes Pulumi programs feel like real, first class distributed apps.

First, we have our TPS reports and zips buckets (for now we’ll new them up, more on that later):

import * as aws from "@pulumi/aws";
const tpsReports = new aws.s3.Bucket("tpsReports");
const tpsZips = new aws.s3.Bucket("tpsZips");

The bucket objects have many properties, including obvious ones like the ARN, domain name, CORS rules, and so on. But if we look closely, we’ll see there are some onX methods. These register event handlers:

Let’s use onObjectCreated to create a Lambda that will zip up any new reports:

tpsReports.onObjectCreated("zipTpsReports", (e) => {
   const AdmZip = require("adm-zip");
   const s3 = new aws.sdk.S3();
   for (const rec of e.Records || []) {
      const zip = new AdmZip();
      const [ buck, key ] = [ rec.s3.bucket.name, rec.s3.object.key ];
      console.log(`Zipping ${buck}/${key} into ${tpsZips.bucket.get()}/${key}.zip`);
      const data = await s3.getObject({ Bucket: buck, Key: key }).promise);
      zip.addFile(key, data.Body);
      await s3.putObject({
         Bucket: tpsZips.bucket.get(),
         Key: `${key}.zip`,
         Body: zip.toBuffer(),
      }).promise();
   }
});

Most of the code here is the application logic – precisely what we want to be focusing on! In fact, this code looks just like any ordinary JavaScript event-driven program.

A neat thing that Pulumi can do is capture references to other resources. So, notice here, we actually capture a reference to tpsZips from within the event handler, and Pulumi figures out how to serialize the resulting closure. For the most part, any code you write there will work, unlocking all sorts of powerful composition scenarios: multiple functions can be composed together, you can use modules as usual, and so on. Just program like you normally would, without the awkward environment variables and configuration. That’s why we call these magic functions.

Now, let’s deploy the resources to AWS:

After confirming, yes, everything is up and running after just a few seconds:

Note the resource graph. Pulumi uses familiar languages, which allows encapsulation and hiding of unnecessary complexity. So underneath that simple-looking onObjectCreated are many AWS resources, like IAM attachments. This is a lot like how we can ignore all the messy OS API calls when using Node.js abstractions.

Pulumi has many convenience functions built-in, and one of them is pulumi logs, which will let us tail the logs associated with all compute in our stack (including serverless functions):

$ pulumi logs -f Collecting logs for stack dev since 2019-03-10T10:09:56.000-07:00…

Now that we’re tailing the logs, let’s copy a file over using the AWS CLI:

$ aws s3 cp ./tps001.txt s3://$(pulumi stack output tpsReportsBucket) upload: ../tps001.txt to s3://tpsreports-96458ef/tps001.txt

We have used the pulumi stack output command to conveniently fetch the bucket name. After the upload completes, we’ll see that our function comes alive in the pulumi logs command:

2019-03-10T11:10:48.617-07:00[zipTpsReports] Zipping tpsreports-96458ef/tps001.txt into tpszips-edfde11/tps001.txt.zip

And, sure enough, if we list the contents of the zip bucket, we will see our newly added file:

$ aws s3 ls s3://$(pulumi stack output tpsZipsBucket) 2019-03-10 11:10:50 6206 tps001.txt.zip

Voila! A fully functioning serverless application.

Note also that the CallbackFunction class – the powerful abstraction behind all of this – is exported, and offers some knobs, in case you want to do things like reuse existing IAM roles rather than creating new ones. See the pulumi/aws/lambda documentation for details. For instance, say we want to increase the RAM available to our function from 128MB to 256MB:

tpsReports.onObjectCreated(
   "zipTpsReports",
   new aws.lambda.CallbackFunction("zipTpsReportsFunc", {
      memorySize: 256 /*MB*/,
      callback: (e) => { /* same code as before */ },
   },
);

Soon we’ll see how to manage and update our functions, but first let’s see some alternative coding styles.

Approach 2: Manual Function Resources

We just saw that we can write our serverless application logic inside of our Pulumi program, alongside our infrastructure resource definitions. This can feel magical and super productive. But not everybody wants to mix these together. Sometimes you want to stick to the classic way of doing this: keeping your function’s application logic in one place, and your function’s infrastructure definition in another. This aligns more closely with many team workflows.

There is another practical reason to want this: if your functions are written in one language (say, C# or Java) while your Pulumi infrastructure is authored in a different one (say, JavaScript or Python).

Pulumi supports this, by simply defining your Lambda function like any other infrastructure resource:

// First, create some IAM machinery:
const zipFuncRole = new aws.iam.Role("zipTpsReportsFuncRole", {
   assumeRolePolicy: {
      Version: "2012-10-17",
      Statement: [{
         Action: "sts:AssumeRole",
         Principal: {
            Service: "lambda.amazonaws.com",
         },
         Effect: "Allow",
         Sid: "",
      }],
   },
});
new aws.iam.RolePolicyAttachment("zipTpsReportsFuncRoleAttach", {
   role: zipFuncRole,
   policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

// Next, create the Lambda function itself:
const zipFunc = new aws.lambda.Function("zipTpsReportsFunc", {
   environment: {
      variables: {
         "TPS_ZIP_BUCKET": tpsZips.bucket,
      },
   },
   code: new pulumi.asset.AssetArchive({
      ".": new pulumi.asset.FileArchive("./app"),
   }),
   runtime: "nodejs12.x",
   role: zipFuncRole.arn,
});

// Finally, register the Lambda to fire when a new TPS report arrives:
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

Because we’re taking matters into our own hands, we have to create the IAM machinery ourselves. We also need a way to communicate the zip bucket name, so we use an environment variable. This highlights both the benefits and drawbacks to programming at this level – we need to know how to configure all of these ancillary resources, but as a result, the entire power of Lambda is at our fingertips.

Notice that we’ve pointed to our application logic inside of ./app. Pulumi will create the zipfile for you. If we instead wanted to use a zipfile we’ve already packaged, just change code as follows:

// ...
   code: new pulumi.asset.FileArchive("./app.zip"),
// ...

Using Pulumi’s Asset and Archive classes, we can fetch code from anywhere – even the network.

Although managing your functions manually isn’t quite as magical, it is practically useful, and still delivers all the usual infrastructure as code benefits, enabling robust infrastructure management and versioning.

Approach 3: Wire Up an Existing Function

This post is mainly about using serverless functions with Pulumi. But what if you’re using Pulumi mainly for infrastructure, and another solution for the functions themselves? No problem – Pulumi can still glue everything together, even if it’s not provisioning all of the resources.

Every resource type in Pulumi has a static get that looks up an existing resource. These resources aren’t managed by Pulumi, but you can still access their properties and use them. In this case, we’ll just look up an existing Lambda, zipTpsReportsFunc, and register it as an event handler:

const zipFunc = aws.lambda.Function.get("zipTpsReportsFunc", "zipTpsReports-19d51dc");
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

We’ve given the function’s ID, zipTpsReports-19d51dc, which allows Pulumi to locate it in your account and reuse it. This can make it easy to incrementally adopt Pulumi one piece at a time, collaborate between teams, or stitch together resources managed by different stacks.

More About Functions

Now that we’ve seen the basics, let’s look at a few associated function management topics.

Updating a Function

Updating the function is as easy as can be. Imagine we’re running into a problem and want to do some good old fashioned printf debugging inside of our lambda. Maybe we want to log the size of the report and its resulting zipfile. Just add the relevant line:

// ...
zip.addFile(key, data.Body);
console.log(`Report: ${data.ContentLength}, Zip: ${zip.toBuffer().length}`);
s3.putObject(
// ...

And run pulumi up – it will show us what changed:

We can view the full diff, by choosing details, including a Git-like diff of the code changes themselves!

After selecting yes, the function is updated in just a couple of seconds. Similarly, we can update any of the resource definitions, and Pulumi will figure out how to incrementally update them.

Splitting Up your Codebase

A nice middle ground between magic and manual functions is to use your language’s module system to structure code differently. This is similar to how you might structure a typical application: route definitions over here, business logic over there, markup over here, etc. Pulumi can figure out the diffs regardless of how you’ve structured your code, so updates are always based only on what’s changed.

For example, maybe we’ve defined our callback function in ./app:

export async function zipReport(e: aws.s3.BucketEvent): Promise<void> {
   // app code, as shown above, goes here
}

And now we can go back to our infrastructure code, and eliminate the application logic entirely:

import { zipReport } from "./app";
...
tpsReports.onObjectCreated("zipTpsReports", zipReport);

We can take this further and use dynamic package management to split up the code, possibly even spreading pieces of infrastructure and application code across multiple repos and/or packages. This works well for larger teams with independent components versioning at their own pace.

Lastly, it’s possible to use Pulumi stacks to actually break apart your cloud resources and functions into independently deployable pieces. This allows teams to leverage features like RBAC. For instance, it’s common for the DevOps team to manage the physical cloud resources like queues, topics, and buckets, while the development team authors and manages the serverless functions attached to them. Read more about this in the organizaing projects & stacks documentation.

More About Event Sources

We glossed over one of the more interesting points: what resources have event handlers associated with them, and how to get our hands on them.

Provision a New Resource

The simplest answer here is to create a new resource in your Pulumi program using new, as we saw above. Because Pulumi is an infrastructure as code platform, any resources in any cloud are available – AWS, Azure, GCP, Kubernetes, etc. When you new one up, Pulumi understands how to provision and manage it.

We saw simple examples of this earlier:

const tpsReports = new aws.s3.Bucket("tpsReports");

Note that the full set of configuration options are available on each resource. Let’s say we want to turn on server-side encryption of our TPS archive bucket:

const tpsReports = new aws.s3.Bucket("tpsReports", {
   serverSideEncryptionConfiguration: {
      sseAlgorithm: "AES256",
   },
});

Just as with functions, subsequent updates will be diffed and updated in the minimally impactful way.

Connect to an Existing Resource

Although it’s nice we can provision the event sources and targets in a single program, sometimes different members of the team manage different parts of the infrastructure. For example, maybe your DevOps engineers provision resources like buckets, topics, and so on, and your developers wire up the functions.

Not to worry, you can import an existing resource and program against it in the same way:

const tpsReports = aws.s3.Bucket.get("tpsReports", "arn:aws:s3:::tpsReports-4f64efc");

This time, when you run your Pulumi program, you’ll still see this resource but notice it says read instead of create or update. This just means that the resource is read from your account as-is. No matter what you do, the bucket itself will not be modified by Pulumi, other than to subscribe an event.

This, of course, can be combined with the earlier similar functionality for functions, to glue together a bucket and a Lambda, where neither was actually provisioned by Pulumi!

const tpsReports = aws.s3.Bucket.get("tpsReports", "arn:aws:s3:::tpsReports-4f64efc");
const zipFunc = aws.lambda.Function.get("zipTpsReportsFunc", "zipTpsReports-19d51dc");
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

Other Notable Event Sources

Finally, it’s worth noting there are many event handler functions exposed in the AWS package:

apigateway.x.API: create serverless APIs using an Express.js style
cloudwatch.onSchedule: fire a CloudWatch event on a particular schedule, e.g. a cron expression
cloudwatch.Event.onEvent: fire an event when a particular CloudWatch event occurs
cloudwatch.LogGroup.onEvent: fire an event when a CloudWatch logs event occurs
dynamodb.Table.onEvent: fire events for DynamoDB insert, modify, or remove operations
kinesis.Stream.onEvent: fire Kinesis Stream events at particular times or batch sizes
s3.Bucket.onObjectCreated: trigger a function anytime an object is created in an S3 Bucket
s3.Bucket.onObjectRemoved: trigger a function anytime an object is removed from an S3 Bucket
s3.Bucket.onEvent: trigger a function for a wide range of S3 Bucket events
sns.Topic.onEvent: fire SNS Topic events when new messages arrive
sqs.Queue.onEvent: fire SQS Queue events when new messages are enqueued (or on DLQ events, etc)

All of these handlers can be programmed using the spectrum of techniques outlined above.

In Conclusion

In this post, we saw many different options for serverless programming with Pulumi, from magic functions that make serverless apps feel like true event-driven programming, to provisioning and managing infrastructure as code.

We also saw that all of these options can be combined in any way, depending on you and your team’s needs. This is the magic of programming languages in action!

If you’d like to try it out, check out the code for this post on GitHub. Happy serverless hacking!

2018 Year at a Glance

Joe Duffy — Mon, 31 Dec 2018 00:00:00 +0000

As we close out 2018, and enter into a New Year, I was reflecting on our progress here at Pulumi this past year and wanted to share some thoughts. It’s been an incredible year and we are hugely thankful to our passionate community, customers, and partners.

Here are some of the exciting things that happened:

Launching our open source community. After being hard at work for a little over a year, we launched our open source project, with the aim of making it considerably easier and more enjoyable to create and operate cloud software. This was a major moment for us. We had previously only shown Pulumi to a few select friends, family, and private beta users, and the reception was beyond our wildest expectations. We got passionate +1’s from cloud engineers of all roles and responsibilities – developers, infrastructure engineers, operators, and managers alike. The Pulumi community has since grown to thousands of users across hundreds of companies of all sizes spread across the globe. The community is now a warm, welcoming, and collaborative place with significant open source contributions and passionate community leaders emerging who are helping one another while also helping us to make Pulumi even better.

Support for major public, private, and hybrid clouds. We launched with support for the three major public clouds, AWS, Azure, and Google Cloud, and have since added capabilities across all of them, including adding serverless, containers, and infrastructure productivity libraries – it is here where we believe Pulumi’s unique approach of using general purpose languages truly shines. We immediately had a wave of inbound interest in applying the Pulumi approach to infrastructure as code in other areas, and quickly added other providers including OpenStack, VMWare vSphere, Alibaba Cloud, F5 BigIP, and more. In September, we unveiled a native Kubernetes provider with support for the entire API surface area across all versions, plus an operations tool, KubeSpy – both of which continue to lead to significant community growth. Many customers are loving that they can achieve a consistent engineering workflow across multi-cloud environments – often spanning public, private, and/or hybrid clouds.

Launching our commercial SaaS product. In October, we launched our commercial SaaS product, with reasonable pricing so that teams of all sizes are able to get their code to the cloud productively, securely, and collaboratively. If Pulumi is like Git – a local CLI and SDK – then the Pulumi SaaS is like GitHub – a hosted service to ensure you can use Pulumi in a team setting. In addition to the Team Edition, which is meant for the “Fortune 500,000,” we also launched the Enterprise Edition, for the Fortune 500. We already have multiple Fortune 500 customers moving into production with Pulumi, and expect this to be a significant area of future growth. Since launching, we’ve added support for more identity providers – including GitHub, GitLab, and SSO/SAML (for Okta, ADFs, and others) – as well as team management capabilities, such as RBAC, our GitHub App, GitHub Actions partnership, and more.

Closing our Series A funding. Also in October, we secured $15MM in additional funds, which will enable us to scale to meet the incredible opportunity in front of us. The funds will ensure that our open source projects are constantly improving and achieving production quality; that our commercial product can meet the most rigorous business requirements, small or large; and that we can grow our organization in areas necessary to meet our customers’ needs. When we set out to make it 10x easier for teams to do cloud engineering, we saw many facets in need of attention, and these funds will help us take concrete steps towards realizing the vision.

What an incredible year! It can still hardly believe it’s been just six months since launching.

As we look to 2019, we plan to continue engaging with and growing our open source community, built on the same tenets of openness and collaboration – the best part of my job. We have heard loud and clear that what customers need today are solutions, not just raw technologies, and we will continue to deliver commercial value in our Team and Enterprise Edition SaaS products that cloud engineering easier, more secure, and more collaborative.

We are just getting started and we can’t wait to continue on the journey of reimagining cloud engineering together. Thank you for your passionate support – and here’s to a wonderful 2019!

— Joe

Building a future of cloud engineering

Joe Duffy — Mon, 22 Oct 2018 00:00:00 +0000

We founded Pulumi because of a deeply held belief that the cloud promises to change all aspects of software development and that there remains an incredible opportunity to reimagine the entire experience, from idea to creation to delivery to management, with one person in mind: you, the engineer.

Too often, we still think of the cloud as an afterthought, as though it were still 1998 with virtual machines and XML configuration. At Pulumi, we believe instead that the cloud is a first class application architecture concern. We still use thankless tools, markup languages, and brittle, homegrown delivery platforms. At Pulumi, we believe instead in a lovable and productive development experience, with excellent collaboration and reuse.

Since getting started just over a year and a half ago, we’ve stood on the shoulders of giants, tapping into decades of progress our industry has made in programming languages, runtimes, and frameworks, and marrying it with all the amazing innovation in cloud containers, orchestrators, managed data and AI services, and serverless capabilities.

The best part of this journey has been hearing from our passionate community. Thousands of engineers, and hundreds of companies, have created and deployed cloud applications and infrastructure using Pulumi. We are proud to be an open source company, and thrive on the daily collaboration and evolution of the platform.

And we’re just getting started! Today we are thrilled to announce that we’ve raised $15M in additional funding from our partners, Madrona Venture Group and Tola Capital, both here in Seattle. We are honored by their shared belief in our vision and our team. Read more in our press release.

How We Got Here, a Tale of Cloud Evolution

We have seen stunning advancement in cloud platform capabilities. It’s hard to believe that Docker was launched just five years, and AWS Lambda only three, ago! And yet, these technologies change all aspects of how we create, deliver, and manage cloud software. Meanwhile, the cloud has become more programmable with APIs everywhere.

At the same time, “DevOps” as an approach to organizing teams and developing software has matured greatly, and is now the default for many organizations. New practices such as infrastructure-as-code, immutable infrastructure, and Site Reliability Engineering have modernized these roles, giving us approaches that can leverage a more rigorous software engineering practices to tasks that used to be more ad-hoc, manual, and error prone.

But despite these fundamental changes, the way we engineer the cloud hasn’t fundamentally changed. As cloud engineers, we still think of configuration as an afterthought and a task best suited for JSON, YAML, or DSLs. Each individual resource we provision – even fine-grained ones like serverless functions – requires often an equal amount of configuration as there is actual application code. The lack of expressiveness has led us to resort to bolting on templating systems, writing mountains of bash, and overall find ourselves swimming in a sea of complexity.

The problem is, these are crude approximations of familiar, general-purpose languages. They lack real abstraction, sharing, and reuse. You can’t fire up your editor and refactor, test, or analyze your code to find bugs ahead of time. Instead, you find simple syntax errors when it’s too late, after production is on the floor. Developers hate to touch those configuration languages, and yet simultaneously operators want to be more productive and empower their teams to collaborate.

It’s safe to conclude: we have hit the limits of the “afterthought” approach.

Towards Cloud Engineering

We believe in a future where developers and operators alike can meet on common ground to work together. We like to call this combined discipline cloud engineering. It’s not Dev, it’s not DevOps, and it’s not SRE; cloud engineering is the best of all of them. In this new future, our teams will experience more joy, collaboration, and productivity than ever before.

Enter Pulumi.

Using your favorite general purpose programming language – whether it is JavaScript, TypeScript, Python, Go, or something else – you can express the entirety of your cloud software. As a cloud engineer, you can develop it in your favorite editor, with statement completion, built-in documentation, and interactive feedback when you’ve got something wrong. You can test things and benefit from static analysis and linters that enforce best practices. You can reuse packages, publish your own, and refactor your code just like with application code. In fact, after living in this world for long enough, the boundary between infrastructure and application code begins to blur.

This has led to a phrase we use a lot at Pulumi – program the cloud – our rally cry, if you will.

Just like code, we can collaborate with source control systems like Git – code reviewing changes, pushing and pulling code to perform deployments – using easy but robust continuous delivery workflows. Thanks to familiar languages, a cloud object model, and programmable cloud APIs, the system has true semantic understanding of your cloud software, its resources and their relationships, which can be used for rich deployment statuses and to organize and find information more easily. It’s never been such a breeze to go from idea to production.

We’ve become enamored by this new approach to creating software that is truly born in the cloud. But it’s not just our own belief – this new age of cloud engineering is working terrifically in practice. We have worked with organizations of all sizes – small ISVs to Fortune 500 Enterprises – to vastly improve their team efficiency and velocity. We almost always reduce configuration sprawl by an order of magnitude thanks to Pulumi, leading to a level of productivity that these teams previously hadn’t imagined was possible. Developers, operators, and managers alike are loving living in this future of cloud engineering.

Pulumi’s Next Chapter

We’ve only just begun. Pulumi supports AWS, Azure, Google Cloud Platform, and Kubernetes in a big way, with early support for VMWare vSphere and OpenStack. Our delivery platform works great with your favorite CI systems. Expect to see continued growth in all of these areas, in addition to new clouds, languages, and integrations.

Today we launched the Team Edition for our SaaS delivery platform. This includes many features that you, the cloud engineering community, told us you need to better operationalize Pulumi within your teams. We already have many happy customers using these features successfully for organizations big and small. Expect to see a steady stream of improvements, in addition to Enterprise Edition capabilities for the largest of organizations needing on-premises, advanced security, and custom identity. Please always let us know how we can better meet your needs.

This new round of funding will allow us to strengthen our commitment to open source and community, increase our R&D velocity, and also scale our business to enable organizations of all sizes to successfully achieve cloud engineering within their teams. We are hiring in all areas of the company as we enter this next exciting chapter of the company.

Download Pulumi – we can’t wait to see all the great things you build next with Pulumi!

Joe & Eric

Continuous Delivery to Any Cloud using GitHub Actions

Joe Duffy — Wed, 17 Oct 2018 00:00:00 +0000

Today we announced our partnership with GitHub on the new GitHub Actions feature. We are super excited about this bold and innovative technology, especially as it relates to Pulumi, and CI/CD more broadly. We truly believe that Pulumi plus GitHub Actions delivers the easiest, most capable, and friction-free way to achieve continuous delivery of cloud applications and infrastructure, no matter your cloud – AWS, Azure, Google Cloud, Kubernetes, or even on-premises. In this post, we’ll dig deeper to see why, and how to get up and running. It’s refreshingly easy!

GitHub Actions + Pulumi == ❤️

Pulumi lets you write cloud applications and infrastructure in your favorite language. This includes containers, serverless, and even VM-based infrastructure. Combined with GitHub Actions, we can get continuous deployment to any cloud – AWS, Azure, Google Cloud, Kubernetes, or even on-premises – all straight from GitHub.

GitHub Actions gives us a way to run compute in GitHub’s cloud, triggered by any GitHub event – commits, pull requests, and more. This enables us to run Pulumi in response to these events, giving us the benefits of infrastructure as code seamlessly integrated into a lovable workflow. We can edit any files – application- or infrastructure-related – to do end-to-end continuous delivery, without any of the YAML, tedious bash scripting, or manual, error-prone steps.

This capability enables some exciting scenarios:

Deploy cloud changes anytime a commit is made, including
- Provisioning entirely new cloud environments
- Updating existing infrastructure, like scaling things up or modifying config
- Rebuilding and redeploying app containers to your cluster
Preview deployments before they’re actually made, integrated with Pull Requests
Use Git branches to achieve GitOps-style promotion from test/staging to production

To see things in action, check out the video shown at the GitHub Universe keynote today:

In this short video, we see a Ruby on Rails application that is continuously deployed to our Kubernetes cluster (full code available here). It’s just a standard Rails application, with a Dockerfile, and a complete Pulumi infrastructure specification, which are just a collection of TypeScript modules. (We could have used Python or Go.) This includes a fully functioning Google Kubernetes cluster, hosted PostgreSQL database (for easier management!), Kubernetes definitions, and automatic building and publishing of our Rails Docker image to the Docker Hub, which then gets auto-deployed to our cluster. And it works across three entirely separate testing, staging, and production environments!

Notice that it’s all ordinary code, so that it works great with GitHub, in addition to our favorite IDEs and tools.

After committing our changes, Pulumi takes it from there. Deployments can be previewed, diffed, and are recorded so that you’ll always know who changed what, when, and why – all very “Git-like.” Pulumi’s GitHub App adds to this and enables “GitOps” so that teams can propose, approve, and promote code from “staging” to “production” using pull requests (more on that below).

We’ve chosen GKE here as an illustration because it’s amazing how powerful such a simple example can be, but this works just as well if you’re using AKS, EKS, on-premises clusters – and of course works for non-Kubernetes scenarios, like VMs, ECS, and Serverless applications.

All of this with just a handful of TypeScript and GitHub Actions set up to work with Pulumi – despite being quite capable, it’s super easy to get all of this up and running. Let’s see how!

Getting Up and Running

The full sequence of steps is available in our GitHub Actions Getting Started Guide.

In summary, using GitHub Actions with Pulumi is as easy as signing up for Pulumi (if you haven’t already), creating one or more workflow files in your repo (see here for examples), configuring secrets, and adding a Pulumi program. Afterwards, you can simply Commit and Pull Request away to deploy All The Things! GitHub Actions will fire and run Pulumi tasks appropriately; you’ll know it’s working if you see Pulumi actions show up alongside your PRs and commits:

Two sorts of actions, defined the examples referenced above, are

Pulumi Preview runs pulumi preview in response to a Pull Request, showing what would happen if the PR were merged into the target branch.
Pulumi Up runs pulumi up on the target branch, in response to a commit on that branch.

If either of these fail, you’ll see the standard GitHub Checks red “x”, whereas success will give a green “checkmark”; click on either to get the full Pulumi log output:

GitHub Actions’ Logs page, plus Pulumi’s Console, together give you total insight into a deployment’s status, including the Docker build logs, Kubernetes status updates, and more – so you’ll always have the full scoop on what’s going on, good or bad, in your cloud deployments.

Pulumi’s GitHub App

GitHub Actions work with Pulumi without needing any extra configuration beyond the workflow file and GitHub Secrets described above. That’s why it’s so easy!

If you are going to use this setup in a real team setting, you’ll probably also want to use Pulumi’s GitHub App. Simply by installing it into your repo, and combined with the above, you’ll instantly get improved GitHub Checks API integration and, more importantly, context added by the Pulumi bot to your Pull Requests about what a deployment will do before it’s even done, as shown above.

Note that this is optional with GitHub Actions, but worthwhile – it’s much easier for your teams to collaborate on deployments, including knowing whether a change might lead to downtime before it’s even triggered. “GitOps” ahoy!

Learning More

There is so much more fun to have, and we’re just getting started. Look for more in the weeks to come. In the meantime, here are some follow up links in case you want to learn more about GitHub Actions and Pulumi:

To learn more, install Pulumi and then check out our Getting Started guide.
In addition to the keynote video, we have two other videos that you might enjoy watching to learn more:
- In this video, we see the GitOps workflow for promoting from staging to production.
- In this longer-form video, we explore more nuances of how the entire experience works.
The Ruby on Rails App, deployed to Kubernetes, from the keynote, is available for easy forking here.

This is an exciting day for developers, DevOps engineers, and SREs alike. The combination of GitHub Actions and Pulumi gives us all an easy, automated solution for continuous deployment of cloud applications and infrastructure to any cloud, purely using code and Git. What’s best about it is that it’s fun in the same way programming is fun.

If you’re not yet in the GitHub Actions private beta, but want to try Pulumi, head on over to our Pulumi Getting Started guide. You’ll be running CLI commands, but we think that can be a lot of fun too!

We want to thank our GitHub partners for the bold and innovative work that went into GitHub Actions, and for letting us in on the action – we can’t wait to see what you, the Pulumi community, builds using this magical combination!

Program Kubernetes with 11 Pulumi Pearls

Joe Duffy — Wed, 12 Sep 2018 00:00:00 +0000

In this post, we’ll take a look at 11 “pearls” – bite-sized code snippets – that demonstrate using Pulumi to build and deploy Kubernetes applications using cloud native infrastructure as code. These pearls are organized into three categories, each demonstrating a unique scenario:

Config as Code: Use your favorite language for authoring applications and configuration, eliminating toil and YAML.
Multi-Cloud Infrastructure: Mix cloud services alongside Kubernetes resources and manage them using one set of tools and workflows.
Software Delivery as Code: Perform sophisticated continuous delivery of your Kubernetes deployments – including canaries, staged rollouts, leveraging cloud native projects like Envoy and Prometheus – authored in code using familiar languages.

Here is a complete index of the pearls below, in case you want to jump around a little:

Config as Code

Define Kubernetes Applications in a Familiar Language
More Expressiveness, Less Boilerplate
Inject Envoy Sidecars Using Abstraction
Adopt Existing Kubernetes YAML
Programmatically Deploy Helm Charts as Code

Multi-Cloud Infrastructure

Declare Cloud Resources Alongside Kubernetes Ones
Provision and Use Kubernetes Clusters in Any Cloud
Build and Deploy Container Images Alongside Configuration Updates

Software Delivery as Code

Robust and Repeatable Deployments, with a Notion of “Done”
Trigger Cascading Rollouts from Dependent Updates
Staged Application Rollouts Gated by Prometheus Checks

Config as Code

The default authoring experience for Kubernetes applications is in YAML, similar to other infrastructure as code solutions. These configuration files can quickly grow out of hand due to the lack of basic facilities like for loops, functions, and classes, leading to copy-and-paste, lots of boilerplate, and plenty of frustration.

YAML is a fine “assembly language” for computers to read – but it was never meant to be a programming language for humans to write. After even the most basic level of complexity, you’ll soon have accidentally encoded critical software architecture decisions in YAML, where as software developers we prefer to write and maintain such things in code.

Using a familiar language brings productivity and joy to development, but more concretely also unlocks the ability to codify and enforce best practices, and to share and reuse them within your team or the community. This is something that simply isn’t possible with YAML.

Pulumi supports the full Kubernetes API and lets you author and deploy resources in code.

1. Define Kubernetes Applications in a Familiar Language

[ Code | Tutorial ]

Let’s begin by seeing what defining Kubernetes applications in a familiar language looks like. The full Kubernetes API is available, just in code instead of YAML. A simple program looks like so:

import * as k8s from "@pulumi/kubernetes";

const appLabels = { app: "nginx" };
const deployment = new k8s.apps.v1.Deployment("nginx-deployment", {
 spec: {
 replicas: 1,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: { containers: [{ name: "nginx", image: "nginx:1.7.9" }] }
 }
 }
});

Indeed, if you run the pulumi new kubernetes-typescript command to create a basic project from the Kubernetes TypeScript template, this is what you’ll get.

With this program in hand, we can run pulumi up and it will evaluate our program to compute the desired outcome, show us a preview, and then make it so. Remember, Pulumi is based on the concept of infrastructure as code; despite us using an imperative language, the code declares a goal state that Pulumi understands and converges towards, in a reliable way:

Notice we are using TypeScript. This gives us great IDE support and compile-time checking for common mistakes. (Pulumi is multi-language and doesn’t mind what language runs at runtime.) For example, if I accidentally enter a string for the replica count, I get instant IDE feedback:

From here, we can make changes to our program – like changing Nginx from 1.7.9 to 1.8 – and then rerun pulumi up . It will compute the minimal set of changes to bring the running application in line with our new desired state, show us a diff, and ask if we’d like to proceed:

Selecting yes will carry out the update:

At this point, we’ve seen that we can gain some advantages from infrastructure as code (like good IDE support and the use of a variable to avoid repeating ourselves for appLabels ). But let’s now see some more compelling reasons why this is so great.

2. More Expressiveness, Less Boilerplate

[ Code ]

We’ve all seen the canonical Kubernetes guestbook application that uses PHP, Nginx, and Redis. We have the same tutorial written in Pulumi. But the basic conversion from YAML to TypeScript leaves something to be desired; namely, it’s equally as verbose, feels a little too “low level” for an application definition, and misses opportunities to reduce repetition.

This is where languages can rescue us. We can recognize the common pattern of a Deployment for a container followed by a Service to expose that container, optionally behind a load balancer with a stable IP address. This has led us to define the simple ServiceDeployment class.

The definition of this class is interesting to ponder – particularly because, being a language, we can define arbitrary functions and classes to encode our own best practices and common patterns – however, the vast simplification to the guestbook itself is the best part:

import * as k8sjs from "./k8sjs";

let redisMaster = new k8sjs.ServiceDeployment("redis-master", {
 image: "k8s.gcr.io/redis:e2e",
 ports: [ 6379 ]
});

let redisSlave = new k8sjs.ServiceDeployment("redis-slave", {
 image: "gcr.io/google_samples/gb-redisslave:v1",
 ports: [ 6379 ]
 });

let frontend = new k8sjs.ServiceDeployment("frontend", {
 replicas: 3,
 image: "gcr.io/google-samples/gb-frontend:v4",
 ports: [ 80 ],
 loadBalancer: true,
});

export const frontendIp = frontend.ipAddress;

The end result of having three Deployments, three Services, the frontend being load balanced, is the same. But we have gone from the original guestbook YAML of 147 lines down to 20. Not only that, but the code is much simpler to understand, and easier to maintain and evolve.

The deployment workflow is similar to using the raw resources, with a simple pulumi up, and we can see the underlying resources expanded in a tree view during our update:

Notice that the frontend’s allocated IP is printed here thanks to the export line:

$ curl http://$(pulumi stack output frontendIp)
<html ng-app="redis">
<head>
<title>Guestbook</title>
...

3. Inject Envoy Sidecars Using Abstraction

Before moving on, let’s see another powerful example of how abstraction can help us to do useful things. In this pearl, we’ll see how to inject sidecars. A sidecar is a microservices pattern whereby a container runs alongside another collection of containers to add some kind of value, like logging, proxying, or gathering and reporting on performance information. The sidecar pattern is particularly powerful in Kubernetes thanks to its Pod abstraction.

Sidecars, and other best practices and standard patterns, are easy to express in code. This is a simple extension of pearl #2 above, where we can use abstraction to hide complex details.

To see this in action, let’s define a simple EnvoyDeployment class that adds a sidecar to any Kubernetes Deployment object:

export class EnvoyDeployment extends k8s.apps.v1.Deployment {
 constructor(name: string,
 args: k8stypes.apps.v1.Deployment,
 opts?: pulumi.CustomResourceOptions) {
 const pod = args.spec.template.spec;

 // Add an Envoy sidecar container.
 pod.containers = pod.containers || [];
 pod.containers.push({
 name: "envoy",
 image: "lyft/envoy:latest",
 command: ["/usr/local/bin/envoy"],
 args: [
 "--concurrency 4",
 "--config-path /etc/envoy/envoy.json",
 "--mode serve"
 ],
 ports: [{ containerPort: 80, protocol: "TCP" }],
 resources: {
 limits: { cpu: "1000m", memory: "512Mi" },
 requests: { cpu: "100m", memory: "64Mi" }
 },
 volumeMounts: [{ name: "envoy-conf", mountPath: "/etc/envoy" }]
 });

 // Add an associated Volume for Envoy's config, mounted as a ConfigMap.
 pod.volumes = pod.volumes || [];
 pod.volumes.push({
 name: "envoy-conf", configMap: { name: "envoy" },
 });

 super(name, args, opts);
 }
}

Now, anytime we create an EnvoyDeployment , we will automatically get an Envoy sidecar attached to it, and we don’t need to remember every time how to properly configure it:

const appLabels = { app: "nginx" };
const deployment = new EnvoyDeployment("nginx", {
 spec: {
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: { containers: [{ name: "nginx", image: "nginx" }] }
 }
 }
});

Of course, we can do the same with Prometheus or any other project requiring sidecars, and we envision possibly having a more general sidecar injection mechanism, thanks to familiar languages. In fact, we could define a base class that our entire organization uses with certain sidecars baked in, updated easily in one place and rolled out using package managers.

4. Adopt existing Kubernetes YAML

It’s wonderful that we can use familiar languages, but there’s already lots of YAML in the world. Pulumi lets us process that YAML dynamically, and adopt it on the fly to become part of our Pulumi program, without even needing to change it. This enables multi-cloud and advanced delivery scenarios, in addition to being a great way to incrementally convert your existing code.

For instance, let’s imagine we want to deploy the standard Kubernetes Guestbook example directly from the official YAML files, and make its public IP easily accessible:

import * as k8s from "@pulumi/kubernetes";

// Create resources from standard Kubernetes Guestbook YAML example.
const guestbook = new k8s.yaml.ConfigGroup(
 "guestbook", { files: "guestbook/*.yaml" });

// Export the (cluster-private) IP address of the Guestbook frontend.
export const frontendIp =
 guestbook.getResource("v1/Service", "frontend").
 spec.apply(spec => spec.clusterIP);

This slurps up all the YAML files underneath the guestbook directory, creates the desired goal state out of them, and gives the same CLI experience shown earlier with diffs and deployments.

5. Programmatically Deploy Helm Charts as Code

The same is true of Helm Charts, an increasingly popular way to distribute entire Kubernetes applications. Let’s say we want to deploy the latest stable Wordpress Helm Chart, and, again, expose its public IP so that it’s easily accessible to us after deployment:

import * as k8s from "@pulumi/kubernetes";

// Deploy the latest version of the stable/wordpress chart.
const wordpress = new k8s.helm.v2.Chart("wordpress", {
 repo: "stable",
 version: "2.1.3",
 chart: "wordpress"
});

// Export the public IP for Wordpress.
export const frontendIp =
 wordpress.getResource("v1/Service", "wpdev-wordpress").
 status.apply(status => status.loadBalancer.ingress[0].ip);

The resulting Pulumi program can be deployed using pulumi up in the usual way:

An even more exciting example that highlights mixing cloud resources alongside a Helm Chart, check out our example that creates a MEAN stack that uses Azure CosmosDB as the storage layer for an otherwise ordinary Helm Chart, alleviating the burden of needing to worry about persistent volumes, stateful services, and the associated backup and operational difficulties.

For added fun, we can even inspect or rewrite the resulting resource graph on the fly, using the transforms parameter, for both YAML configuration and Helm Charts. This lets us validate conforming configuration, inject sidecars automatically, and more.

Multi-Cloud Infrastructure

Kubernetes applications seldom live on an island by themselves. They typically rely on other cloud infrastructure, as we’ve just seen. This spans a long list of possibilities, for instance:

Data services: AWS S3, Google BigTable, Azure CosmosDB
AI services: speech recognition, image classification, training algorithms
Hosted services: MySQL, Redis, Map/Reduce
Container registries: ECR, ACR, GCR, Docker Hub, JFrog Artifactory

In addition to the very infrastructure hosting the underlying Kubernetes cluster itself, such as the underlying networking and cluster resources (AKS, EKS, GKE), or compute resources for on-premise clusters (VMWare, OpenStack, etc).

Most of us today use different YAML dialects for Kubernetes applications and the other cloud infrastructure, and then end up needing to stitch together different tools using ad-hoc Bash or Python scripts. This is hardly robust and leads to significant toil and tricky failure modes.

Pulumi, on the other hand, can provision and manage the full set of resources mentioned in the above examples, using the same programming model, language, and tools and workflow.

6. Declare Cloud Resources Alongside Kubernetes Ones

Let’s say that we want an AWS S3 Bucket for our Kubernetes application to use. This sounds basic – and indeed happens all the time – but is surprisingly difficult with today’s tools.

Normally we’d find ourselves clicking around in the AWS console to create one, writing AWS CloudFormation YAML and deploying it using AWS, or cobbling together Terraform DSL scripts that are deployed independently from our Kubernetes object definitions themselves. This leaves us having to orchestrate multiple tools and services that were never designed to work together.

Instead of doing that, we can simply author a single TypeScript program that does it all in one go. Pulumi offers packages for AWS, Azure, Google Cloud, and other clouds. We can then use a single CLI invocation, pulumi up, to provision and update all of it:

import * as aws from "@pulumi/aws";
import * as k8s from "@pulumi/kubernetes";

const appName = "nginx";

// nginx config stored in an S3 bucket.
const config = new aws.s3.Bucket(`${appName}-config`);

// nginx container, replicated 1 time.
const appLabels = { app: appName };
const nginx = new k8s.apps.v1beta1.Deployment(appName, {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: {
 initContainers: [nginxConfigPuller(config.bucketDomainName)],
 containers: [{ name: appName, image: "nginx:1.15-alpine" }]
 }
 }
 }
});

Pulumi understands the full dependency graph between resources, and so the deployment of the AWS bucket will be orchestrated properly with respect to the Kubernetes resources. In the event that a subsequent update alters the bucket – for instance, perhaps a new domain name is assigned – Pulumi will know how to cascade those changes to Kubernetes automatically.

7. Provision and Use Kubernetes Clusters in Any Cloud

[ Code ]

In addition to all the usual resources your application might use, you can also provision actual Kubernetes clusters using Pulumi programs. This has become much easier recently thanks to Google’s Kubernetes Engine (GKE), Azure’s Kubernetes Service (AKS), and AWS’s Elastic Kubernetes Service (EKS), which offer hosted solutions at a reasonable cost. Of course, it’s possible to use Pulumi programs to provision custom and on-premise clusters too.

Another thing Pulumi can do is manage an actual cluster and its application resources in a single program. It’s common to separate programs into low-level infrastructure and higher-level applications in the move to production – especially for cost, team roles and responsibilities, etc. – but it’s a game changer for simple cases where as a developer you just want to get up and running quickly without thinking deeply about the clustering underneath. More importantly, you have one set of languages, tools, and workflows for all application infrastructure.

For example, here is a program that deploys a GKE cluster with configurable settings:

import * as gcp from "@pulumi/gcp";
import { nodeCount, nodeMachineType, password, username } from "./config";

const engineVersion = gcp.container.getEngineVersions().then(v => v.latestMasterVersion);
export const k8sCluster = new gcp.container.Cluster("gke-cluster", {
 initialNodeCount: nodeCount,
 minMasterVersion: engineVersion,
 nodeVersion: engineVersion,
 masterAuth: { username, password },
 nodeConfig: {
 machineType: nodeMachineType,
 oauthScopes: [
 "https://www.googleapis.com/auth/compute",
 "https://www.googleapis.com/auth/devstorage.read_only",
 "https://www.googleapis.com/auth/logging.write",
 "https://www.googleapis.com/auth/monitoring"
 ],
 },
});

Imagining that this was defined in a cluster.ts module, we can deploy a canary deployment on top of that cluster using the same program:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import { k8sProvider, k8sConfig } from "./cluster";

// Create a canary deployment to test that this cluster works.
const name = `${pulumi.getProject()}-${pulumi.getStack()}`;
const canaryLabels = { app: `canary-${name}` };
const canary = new k8s.apps.v1beta1.Deployment("canary", {
 spec: {
 selector: { matchLabels: canaryLabels },
 replicas: 1,
 template: {
 metadata: { labels: canaryLabels },
 spec: { containers: [{ name, image: "nginx" }] },
 },
 },
}, { provider: k8sProvider });

// Export the Kubeconfig so that clients can easily access our cluster.
export const kubeConfig = k8sConfig;

The incredible thing about this example is that, with it, we simply run pulumi up, and a little over two minutes later, we have a fully functioning GKE cluster with a canary deployment up and running:

In addition to the GKE example, our examples repo also has fully functioning examples for doing the same thing with Azure’s AKS and AWS’s EKS. A Kubernetes cluster for everybody!

8. Build and Deploy Container Images Alongside Configuration Updates

One scenario we often encounter in practice is the need to coordinate Docker registry provisioning and management (using one set of tools), with building and pushing new Docker images (using another set of tools), with updating Kubernetes configuration to consume said images (using yet another set of tools). Pulumi can handle doing all three things in code, using just the pulumi up command to manage your container registry infrastructure, build and push images, and update your Kubernetes objects to consume those freshly published images.

This example builds and pushes a custom Nginx image to the Docker Hub and then consumes it from a Kubernetes Deployment:

import * as docker from "@pulumi/docker";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();

// Build a Dockerfile located at ./mynginx in my project.
// Publish it to the Docker Hub using the configured username/password.
const mynginx = new docker.Image("mynginx", {
 build: "./mynginx",
 imageName: "hekul/mynginx:v1",
 registry: {
 server: "docker.io",
 username: config.require("dockerUsername"),
 password: config.require("dockerPassword"),
 },
});

// Deploy an app container that consumes the resulting image name.
const appLabels = { app: appName };
const nginx = new k8s.apps.v1beta1.Deployment(appName, {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: {
 // Use the app container at the specific SHA pushed.
 containers: [{
 name: appName,
 image: mynginx.imageName,
 }],
 }
 }
 }
});

Although this example uses the Docker Hub, this technique works great for all of the major cloud providers’ container registries: Amazon ECR, Azure ACR, and Google GCR. Keep an eye out on our examples repo for examples of these soon, or let us know what you’d like to see.

Software Delivery as Code

In this final section, we’ll turn our attention away from authoring, and towards operating. Namely, how to achieve robust continuous deployment (CD) of your Kubernetes applications.

Pulumi takes an “everything is code” approach so, in addition to your infrastructure being written in code, your software delivery pipeline is too. In fact, each Pulumi program really is a deployment workflow expressed in code. The Pulumi CLI is responsible for evaluating programs, producing goal states, diffing states to produce plans, and carrying them out. The Pulumi hosted service at https://app.pulumi.com is used for robust concurrency and state management so that deployments can be done continuously in a team environment, with perfect auditability of who changed what and when, and so that deployments are repeatable and failures recoverable.

Let’s look at a few particularly useful scenarios when using Pulumi for Kubernetes deployments.

9. Robust and Repeatable Deployments, with a Notion of “Done”

[ Code ]

Kubernetes is amazing at container orchestration and running reliable distributed systems. Its primary CLI, kubectl, offers many features for doing basic deployments and rollouts, and the cluster management software itself handles rolling updates. kubectl deliberately leaves more sophisticated deployment “policy” as an exercise outside of the core system, however. This is a noteworthy and agreeable decision, because the foundational CLI must provide robust mechanisms based on eventual consistency and fault tolerance, its most critical underpinnings.

For practical deployments, we often need more opinionated policy on top of these mechanisms, and that is where a tool like Pulumi that does deployment orchestration can help. Pulumi adds the ability to preview updates before applying them (with full diffing); the ability to know when a deployment has succeeded or failed and why; canaries, blue/green, flighting-based rollouts, and more. Pulumi’s CLI has been developed with real world scenarios in mind.

Let’s consider the following program, which deploys Nginx to a Kubernetes cluster using the standard Kubernetes object model, and allocates a public IP address to it:

import * as k8s from "@pulumi/kubernetes";

// nginx container, replicated 1 time.
const appName = "nginx";
const appLabels = { app: appName };
const nginx = new k8s.apps.v1beta1.Deployment(appName, {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: { containers: [{ name: appName, image: "nginx:1.15-alpine" }] }
 }
 }
});

// Allocate a public IP to the nginx Deployment.
const frontend = new k8s.core.v1.Service(appName, {
 metadata: { labels: nginx.spec.apply(spec => spec.template.metadata.labels) },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: 80, protocol: "TCP" }],
 selector: appLabels
 }
});

// When "done", this will print the public IP.
export const frontendIp = frontend.status.apply(
 status => status.loadBalancer.ingress[0].ip);

This code looks a lot like the prior examples, but let’s double click into the CLI experience when deploying it. Out of the box, Pulumi provides several real world niceties.

First, pulumi up will wait for all resources to be initialized before completing. This gives a notion of done, and allows us to see status updates and any errors that result. kubectl intentionally doesn’t do this because it relies on eventual consistency and the idea that deployments will eventually converge. This is a robust and reliable foundation, but as developers, we often want richer monitoring. This leads to bash scripts and frequent polling, with hard to diagnose events.

For instance, if we specify an invalid image, we’ll see fine-grained status errors as they arise:

This allows us to “export” values from the completed resources, which gives a way to elevate the most important output values. In the last line, we see that we’re exporting the public IP address allocated to the Nginx Pod. This allows us to use this IP address as a value:

Finally, in true infrastructure as code fashion, Pulumi works on the notion of goal states. This means it can always diff the desired goal state with whatever is currently deployed to a cluster. That is shown as part of the typical pulumi up sequence, but the pulumi preview --diff command shows a more “Git-like” diff of the full resource graph. In this example, we’ve changed our container image from nginx:15-alpine to nginx:16-alpine:

The ability to look at previews before making an update ensures mistakes don’t get made and that deployments don’t have unanticipated impacts. There is even a Pulumi GitHub App that will show such diffs inside of pull requests before they’ve been deployed, so you know the impact of configuration changes before code even makes its way to master.

10. Trigger Cascading Rollouts from Dependent Updates

[ Code ]

Although it’s not strictly “deployment orchestration”, there are some useful things Pulumi does out of the box when deciding how and when to update resources. Namely, its engine has a full understanding of the resource graph (technically, a DAG), which lets it cascade updates to multiple resources that depend on one another. This is often be as simple as feeding the output of some Kubernetes object as input to another programmatically, so that when you update one you don’t also forget to update the other. Pulumi orchestrates operations in the right order.

In this example, however, we’ll look at a specific example, where this cascading can be useful. Imagine we are using a ConfigMap to store the value for an Nginx Deployment’s config file. There are various ways we could map that ConfigMap and do hot reloads of it, but let’s say we just want to do the simpler thing of triggering a deployment anytime that config changes. Pulumi can handle this automatically, simply based on the dependencies between resources:

import * as fs from "fs";
import * as k8s from "@pulumi/kubernetes";

const appName = "nginx";
const appLabels = { app: appName };

// nginx Configuration data to proxy traffic to `pulumi.github.io`.
// Read from `default.conf` file.
const nginxConfig = new k8s.core.v1.ConfigMap(appName, {
 metadata: { labels: appLabels },
 data: { "default.conf": fs.readFileSync("default.conf").toString() }
});
const nginxConfigName = nginxConfig.metadata.apply(m => m.name);

// Deploy 1 nginx replica, mounting the configuration data into the container.
const nginx = new k8s.apps.v1beta1.Deployment(appName, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: "nginx",
 image: "nginx:1.13.6-alpine",
 volumeMounts: [{
 name: "nginx-configs",
 mountPath: "/etc/nginx/conf.d",
 }].
 }],
 volumes: [{
 name: "nginx-configs",
 configMap: { name: nginxConfigName },
 }],
 },
 },
 }
});

Notice here that we’re just reading a file from our local project workspace to supply as the default.conf data. The end result is that all we need to do is update that config file, rerun pulumi up , and it will detect that the ConfigMap changed, perform those changes, and then cascade the updates to the Deployment such that it ingests those changes:

11. Staged Application Rollouts Gated by Prometheus Checks

[ Code ]

As we start to go deeper into these scenarios, we’re beginning to see that Pulumi is useful not only for expression of application configuration, but also for deployment orchestration. Because a Pulumi program is arbitrary code, we can do useful things in the middle of updates.

In this example, we’ll use this capability to rendezvous with Prometheus to gate our deployment with health checks. In particular, we will first deploy a basic 3-replica canary, ask Prometheus to check health metrics, and then once the P90 response time is looking good, we’ll advance our deployment to a full blown 10-replica deployment.

Let’s check out the code. It looks like a usual Pulumi program, except that we use a promise from the Prometheus health checks as input to the staging ring deployment, which has the effect of making it dependent on the resolution of said health checks:

// Canary ring. Replicate instrumented Pod 3 times.
const canary = new k8s.apps.v1beta1.Deployment(
 "canary-example-app",
 { spec: { replicas: 1, template: instrumentedPod } },
 { dependsOn: p8sDeployment }
);

// Staging ring. Replicate instrumented Pod 10 times.
const staging = new k8s.apps.v1beta1.Deployment("staging-example-app", {
 metadata: {
 annotations: {
 // Check P90 latency is < 20,000 microseconds. Returns a `Promise<string>` with the P90
 // response time. It must resolve correctly before this deployment rolls out. In
 // general any `Promise<T>` could go here.
 "example.com/p90ResponseTime": util.checkHttpLatency(canary, containerName, {
 durationSeconds: 30,
 quantile: 0.9,
 thresholdMicroseconds: 20000,
 prometheusEndpoint: `localhost:${localPort}`,
 })
 }
 },
 spec: { replicas: 1, template: instrumentedPod }
});

If we run this with a pulumi up, the deployment will proceed as usual, but we will see intermediate output telling us at what stage our deployment is at any given time. Most of the time will be spent waiting on Prometheus, but then we’ll see the final deployment scale out – assuming our canary passed the health checks, of course!

Wrapping Up

In this post, we’ve seen some of the exciting things you can do with Pulumi’s Cloud Native SDK for Kubernetes. We’ve been having a great time working with the community over the past couple months to get all of these goodies ready to go, are thrilled to ship, and we look forward to hearing about your ideas and scenarios about where to take things from here.

To get started, head over to the Pulumi Quickstart, meet us over on GitHub where all the goodies are open source https://github.com/pulumi/pulumi, and/or join our Pulumi Community Slack. We can’t wait to hear from you. Happy hacking!

Cloud Native Infrastructure with Kubernetes and Pulumi

Joe Duffy — Wed, 12 Sep 2018 00:00:00 +0000

Kubernetes has quickly become the “gold standard” for running containers in production, spanning public, private, and hybrid cloud scenarios. It’s been remarkable to watch its explosive growth just this past year alone. Every cloud vendor now supports an easy-to-use managed Kubernetes solution — Google GKE, Azure AKS, and AWS AKS — making it easier than ever to start writing and deploying Kubernetes applications.

Pulumi for Kubernetes is a way to create, deploy, and manage Kubernetes applications using your favorite programming languages. that works across AWS, Azure, Google Cloud, OpenStack, and other clouds, now to Kubernetes and cloud native architectures. You can dive right in here and look at some powerful things Pulumi enables here.

Why Pulumi?

Kubernetes is essentially the “POSIX” for the cloud and unlocks the potential for a true cloud native development platform to be built on top. We at Pulumi are tremendously excited by this opportunity – it aligns perfectly with our vision for the future, and our language-driven and true platform approach.

Despite these advances, Kubernetes can still be intimidating to use. This is true for newcomers and experts alike. The use of YAML for configuration has led to the “Wall of YAML” problem, giving way to many quasi-languages, often rudimentary templating systems bolted on top of YAML. The lack of true end-to-end deployment support – including orchestrated rollouts, knowing when a deployment has completed successfully, or even managing resources outside of Kubernetes itself like databases, container registries, and other hosted services – has led to the “Mountains of Bash” problem. All aspects, from creation to deployment to management, can be difficult, particularly at scale.

We envision a future where all developers and DevOps engineers seamlessly leverage Kubernetes to program the cloud. To get there, we need to bring joy, ease of use, and simplicity to the entire Kubernetes software development lifecycle. We must eliminate toil and maximize productivity. This is precisely what Pulumi for Kubernetes aims to do.

Familiar Languages

Our favorite way to program Kubernetes is using TypeScript, as it delivers strong typing where possible and dynamic typing where needed. This delivers great IDE support out of the box, no matter your IDE of choice, complete with auto-completion, interactive documentation and error reporting, refactoring support, and more. The result is a radically improved inner development loop. In addition to TypeScript, JavaScript works too, and Python and Go are on the way.

Perhaps the best part of using a familiar language is instance access to an ecosystem of libraries. Any NPM module is an npm install away. And you can create your own libraries – classes to encapsulate full components, and functions to capture repeated patterns of code – helping to reduce boilerplate, maximize reuse and sharing, and make you more productive. The standard Kubernetes guestbook shrinks to a mere 20 lines of code thanks to abstraction!

Many of our customers have already used this capability to capture best practices within their own team, shrinking 1,000s of lines of copy-and-pasted YAML to just a handful of lines of real code.

We are just getting started with our own Kubernetes abstractions, but we’re most excited to see where you, the community, take things.

End-to-End Deployment Orchestration

Pulumi programs are deployed using a single CLI command

$ pulumi up

This command works the first time – when all resources will be created – in addition to all subsequent updates. Each update will show a full diff of what has changed before you deploy anything:

Because Pulumi also supports resources for all clouds – including AWS, Azure, Google Cloud, OpenStack, and VMWare vSphere – you can mix and match all of your infrastructure needs using one consistent programming language, model, tool, and workflow. This means, for example, that a single program can provision your Kubernetes resources alongside Azure CosmosDB, a Google Container Registry to publish your Docker builds to before pulling them in your cluster, an AWS S3 Bucket for your application, or even CDN and DNS entries for your cluster. Pulumi’s notion of stacks makes it easy to multi-instantiate your application for different environments.

Programs are just code, unlocking sophisticated cloud native scenarios; e.g.

Deploy a canary, wait for Prometheus health checks to pass, and then promote
Create a base class that injects Envoy sidecars automatically for any pod in your org
Manage your Envoy config as code, and version it alongside your Kubernetes config

The Pulumi Delivery Platform works in tandem with the CLI to ensure continuous delivery of your infrastructure. This supports robust deployments in a team environment with state checkpointing and concurrency control. It integrates with any CI system and supports a GitHub App that you can install to get previews of changes inside your Pull Requests. This enables a true GitOps workflow – for both Kubernetes config updates, and your cloud infrastructure.

What’s Next

With Pulumi’s support for Kubernetes and cloud native architectures, we hope that familiar languages will bring you more joy, reduce toil, and ultimately lower the barriers to entry for using Kubernetes with real components and reuse.

We can’t wait to hear what you think. To learn more or give it a try, please check out these resources:

These two tutorials will walk you through your first Pulumi for Kubernetes project:

Program Kubernetes with 11 Pearls: a companion blog post with 11 exciting examples.

Pulumi: A Better Way to Kubernetes: A walkthrough on how Pulumi can make Kubernetes more accessible, using your favorite languages and tools.

If you have any questions or comments, we’d love to hear from you in our Community Slack or over on GitHub.

Happy cloud native hacking!

Program the Cloud with 12 Pulumi Pearls

Joe Duffy — Wed, 25 Jul 2018 00:00:00 +0000

In this post, we’ll look at 12 “pearls” – bite-sized code snippets – that demonstrate some fun ways you can program the cloud using Pulumi. In my introductory post, I mentioned a few of my “favorite things”. Now let’s dive into a few specifics, from multi-cloud to cloud-specific, spanning containers, serverless, and infrastructure, and generally highlighting why using familiar languages is so empowering for cloud scenarios. Since Pulumi lets you do infrastructure-as-code from the lowest-level to the highest, we will cover a lot of interesting ground in short order.

If you want to follow along and try some of this out, Pulumi is open source on GitHub, free to download and use, and the quickstart will acquaint you with the CLI. Most of the examples are directly runnable and available in our examples repo, and are just a pulumi up away, unlike other approaches that require you to point-and-click around in your cloud’s console, and/or author reams of yucky YAML. And you get to use familiar languages!

Here is an index of the pearls in case you want to dive straight into one in particular:

Infrastructure:

Declare cloud infra using a familiar language (with loops!)
Make a reusable component out of your cloud infra

Serverless:

Go serverless without the YAML
Capture state in your serverless funcs, like real lambdas
Simple serverless cron jobs
Run Express-like serverless SPAs and REST APIs at near zero cost

Containers:

Deploy production containers without the fuss
Use containers without Dockerfiles
Invoke a long-running container as a task

General Tips and Tricks:

Use code to avoid hard-coding config
Use config to enable multi-instantiation and code reuse
Give your components runtime APIs

Even if you’re uninterested in low-level infrastructure, it can be fun to work through these examples; it’s “turtles all the way down” with Pulumi and doing so can help understand how the system works. And similarly, it can be fun to see the high-level scenarios these building blocks facilitate, even if you just want to stand up containers and functions.

And with that, let’s dive in.

Infrastructure

Familiar Languages

[ Runnable Example ]

Pulumi gives you a way to express infrastructure configuration using your favorite programming language. In this article, we’ll use TypeScript on Node.js, as it delivers great productivity by blending dynamic and static typing with the NPM ecosystem of reusable packages.

At the lowest layer, there are packages for all major cloud providers – AWS, Azure, Google Cloud – in addition to Kubernetes and OpenStack. So, if you want an EC2 instance, Azure CosmosDB, or Kubernetes Pod, you just new up an object. From there, Pulumi uses an infrastructure-as-code approach similar to using AWS CloudFormation, Azure Resource Manager, Google Resource Manager, Kubernetes or Helm YAML, or Terraform – just without the YAML DSLs.

The strength of using a language truly begins to shine when you go beyond new, however. Using a language gives you control structures, like for loops and if branches. This is easy to take for granted, but compared to existing YAML DSLs, it is very powerful.

For example, let’s create three EC2 VMs and export their IP addresses. Normally you’d need to copy-and-paste, however for loops to the rescue!

index.ts:

import * as aws from "@pulumi/aws";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

let webServers = [];
for (let i = 0; i < 3; i++) {
 webServers.push(new aws.ec2.Instance(`web-server-${i}`, {
 instanceType: "t2.micro",
 ami: "ami-6869aa05", // us-east-1-only
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello, World #${i}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 }));
}
export const publicHostnames = webServers.map(s => s.publicDns);

This provisions three web servers, each of which runs a simple Python webserver.

Of course, this can unlock far more sophisticated scenarios. For example, our own AWS Infrastructure package offers core AWS infrastructure components using AWS’s own best practices, and a community member built an AWS Virtual Private Cloud (VPC) component that automatically distributes subnets across all availability zones in a region, including smartly calculating the CIDR addresses, hiding lots of the messy details of AWS networking.

Reusable Components

[ Runnable Example ]

Speaking of components, this is another benefit of using familiar languages: abstraction. This includes the ability to hide uninteresting details through encapsulation, and create bigger things out of smaller things, a key to how we’ve become so productive with programming languages generally over the years. Applied to infrastructure, this enables some powerful scenarios.

For instance, perhaps you want a standard base class for all servers in your organization, enforcing security policy or even just naming conventions. Or maybe you want to ensure there is an Envoy proxy sidecar for all of your Kubernetes pods. No matter the scenario, we call these things “components” and Pulumi has a rich model that supports parents and children.

To stick with the running example, let’s make a WebServer class that internally creates the EC2 VM so that callers don’t need to know messy details like how to create and configure a SecurityGroup and choosing the right image:

web.ts:

import * as aws from "@pulumi/aws";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

export class WebServer {
 public readonly vm: aws.ec2.Instance;

 constructor(name: string) {
 this.vm = new aws.ec2.Instance(`${name}`, {
 instanceType: "t2.micro",
 ami: "ami-6869aa05", // us-east-1-only
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello from ${name}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 });
 }
}

Now we can go back to our original program and tidy it up a bit:

index.ts:

import { WebServer } from "./web";

let webServers = [];
for (let i = 0; i < 3; i++) {
 webServers.push(new WebServer(`web-server-${i}`));
}
export const publicHostnames = webServers.map(s => s.vm.publicDns);

Not only is the result simpler, but we now have a way to divvy up responsibilities within a team, helping Dev and DevOps to work better together.

This example used purely local components, but you can of course publish and consume any old packages from your favorite package manager – including private registries for your own team.

We will return to these examples later on and improve them to be more reusable through configuration, however let’s first tour some fun serverless and container pearls.

Serverless

No YAML

[ Runnable Example ]

Let’s shift gears and jump into serverless. Pulumi is unique in that its goal is to be a platform for all aspects of modern cloud programming, so that includes serverless, in addition to the abovementioned infrastructure examples and the soon-to-be-mentioned container ones. Most real-world examples we see in fact mix all of these things together in the same program.

Because of the way it uses familiar languages, there are great benefits to serverless programming in Pulumi. No YAML required – just write lambdas in your favorite language, and our compiler/runtime magic will deal with them. But first, let’s start simple.

The AWS Serverless package provides event sources for all of the possible AWS Lambda triggers. This leverages the aws.serverless.Function abstraction that takes a lambda in your language and translates it into a package suitable for uploading to AWS. No need to embed code in YAML files, manually upload to S3, etc. Pulumi handles all of this including versioning.

Let’s look at a simple example that simply posts a Slack message for every new message that arrives in an SQS queue (a new capability recently added to AWS). The way you express serverless functions in Pulumi is by writing ordinary lambdas and the way you hook them up to event sources is by using an event-driven style on your existing cloud resources:

index.ts:

import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

let queue = new aws.sqs.Queue("mySlackQueue");
serverless.queue.subscribe("mySlackPoster", queue, async (e) => {

let slack = require("@slack/client");
let client = new slack.WebClient(config.slackToken);
 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*:n${rec.body}n`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to ${config.slackChannel}`);
 }
});
export const queueURL = queue.id;

The code and instructions for running this example are available in our examples repo.

Serverless programs written in this style feel just like your favorite event-driven frameworks, and – because these are just ordinary lambdas in your language of choice – they compose in all the usual ways, unlocking some fascinating future possibilities (like serverless Spark and/or RxJS). Behind the scenes, Pulumi is doing the hard work to extract out the body of the function, rewrite it minimally but as needed to wire it up to AWS Lambda, and package it up. Subsequent deployments are as easy as the first one, as Pulumi does diffing to compute the minimal deltas.

Not everybody will want to mix infrastructure definitions and runtime code like this. Because these are just ordinary programs, you are free to split code into submodules and structure your workspace as you see fit. In fact, you can create distinct stacks for the different pieces.

This example is AWS-specific, however the Pulumi Cloud Framework offers several multi-cloud serverless (and container) capabilities. We will see some examples of that package in action below, however being able to target your cloud of choice directly means you can use all of its underlying features.

Real Lambdas

[ Runnable Example ]

Pulumi also lets you capture state in your serverless functions. This is one of the major advantages to this approach; normally you’d need to awkwardly arrange for such state to be passed into your function through some sideband mechanism, such as environment variables.

In Pulumi, we just capture state. Just like in your favorite language.

Let’s say we want the Slack information above to be configurable – a good idea, since the Slack token is a secret that should be configured using the –secret flag. To do so, first let’s create a config module:

config.ts:

import * as pulumi from "@pulumi/pulumi";
let config = new pulumi.Config(pulumi.getProject());

export const slackChannel = config.get("slackChannel") || "#general",
export const slackToken: config.require("slackToken");

And now let’s go back to our above program, and use the config module for these properties. Notice that we simply capture a reference to the config module – that’s it!

index.ts:

import * as config from "./config";
import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

let queue = new aws.sqs.Queue("mySlackQueue");
serverless.queue.subscribe("mySlackPoster", queue, async (e) => {
 let slack = require("@slack/client");
 let client = new slack.WebClient(config.slackToken);

 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*:n${rec.body}n`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to ${config.slackChannel}`);
 }
});
export const queueURL = queue.id;

We often call this “capture by serialization” because, of course, the captured state is marshaled into the serverless function’s package, and then unmarshaled at runtime at the time of use. The semantics of capture are specific to each language, but for JavaScript, we do a deep serialization of the object, including its prototype chain, so that the full structure is preserved.

Because of this, functions are even available. We use this capability to let you capture resources by-value that internally have references to a live cloud resource. We will see that next in our serverless cron job example.

Cron Jobs

[ Runnable Example ]

Pulumi’s Cloud Framework has an ultra-straightforward timer module that lets you schedule cron jobs that run serverless functions. This is the easiest way to get up and running with serverless functions, because you don’t even need any other resources to trigger events from.

There are several ways to schedule a timer, depending on your stylistic preferences. These examples simply print the current time to the console on a given interval:

index.ts:

import * as cloud from "@pulumi/cloud";

// Run a timer every minute:
cloud.timer.interval("interval-timer", { minutes: 0 }, () => {
 console.log(`interval-timer: ${Date.now()}`);
});

// Run a timer every minute (cron-style expression):
cloud.timer.cron("cron-timer", "0 * * * * *", () => {
 console.log(`cron-timer: ${Date.now()}`);
});

// Run a timer every day at 7:30 UTC:
cloud.timer.daily("daily-timer", { hourUTC: 7, minuteUTC: 30 }, () => {
 console.log(`daily-timer: ${Date.now()}`);
});

// Run a timer at the 45th minute UTC of every hour:
cloud.timer.hourly("hourly-timer", { minuteUTC: 45 }, () => {
 console.log(`hourly-timer: ${Date.now()}`);
});

To make things slightly more interesting, let’s write a serverless timer that fetches the Hacker News homepage every hour and stashes it into a document database:

index.ts:

import * as cloud from "@pulumi/cloud";

let snapshots = new cloud.Table("snapshots");
cloud.timer.daily("daily-yc-snapshot", { hourUTC: 0, minuteUTC: 0 }, () => {
 let req = require("https").get("https://news.ycombinator.com", (res) => {
 let content = "";

 res.setEncoding("utf8");
 res.on("data", (chunk) => { content += chunk });
 res.on("end", () => {
 snapshots.insert({ date: Date.now(), content: content });
 });
 });
 req.end();
});

We are using the ability to capture a reference to our cloud.Table and call functions on it, like insert. Of course, the body of that function can do absolutely anything that you’d like.

REST APIs

[ Runnable Example ]

Us Node.js programmers love web frameworks, whether that be serving static content using a single page application (SPA) built using React.js, Vue.js, or Angular.js, a program built using dynamic server-side logic using Express.js, or some combination thereof. Serverless offers a way to run these programs at incredible density and low cost, often reducing a VM- or container-based solution to 1/10th its cost (or better). The problem is, this usually means abandoning productive frameworks, and resorting to YAML, hand-authoring Swagger files, and configuring complex API Gateway machinery – all foreign concepts when you just want to run a website.

Pulumi lets you just write code to host both static and dynamic content simply, using its cross-cloud framework’s cloud.API abstraction. It has been designed to have a familiar Node.js feel. The following example serves all static content underneath my www directory at the root, and provides a simple GET endpoint at /hello that returns a { hello: "World!" } JSON object:

index.ts:

import * as cloud from "@pulumi/cloud";
// Create a new API endpoint:
let app = new cloud.API("my-app");
// Serve static files from the `www` folder (using AWS S3):
app.static("/", "www");
// Serve a simple REST API on `GET /hello` (using AWS Lambda):
app.get("/hello", (req, res) => res.json({ hello: "World!" }));
// Export the public URL for the HTTP service:
export const url = app.publish().url;

All of the usual verbs are available – post, patch, etc. – as is support for middleware.

The structure of my local filesystem will demonstrate how some of this ties together:

.
├── Pulumi.yaml
├── index.ts
├── package.json
└── www
├── favicon.png
└── index.html
1 directory, 5 files

Again, all we need to do is pulumi up, and Pulumi will figure out what has changed. So, if we are building an SPA using React.js, etc., then we would simply have a build step that updates the content, and the next time we run pulumi up, it will figure out what has changed and make the minimal set of updates. Similarly, if we edit a dynamic function, it will update just the function.

As you can imagine, things get fun when you start mixing scenarios. Say we wanted a web frontend to our Hacker News timer fetcher above. Thanks to the way we can simply capture a reference to the snapshots database and use it in the body of our API, this is extremely trivial.

Per the earlier conversation around mixing infrastructure and application code, some folks prefer to split the routes from the application code. This is a fine convention, and easy to do:

index.ts:

import * as cloud from "@pulumi/cloud";
import * as routes from "./app/routes";

// Create an API like before, but use a routes module instead of defining inline:
let app = new cloud.API("my-app");
app.get("/hello", routes.hello);

export const url = app.publish().url;

For now, Pulumi only offers an AWS implementation of these APIs, but more clouds are on their way!

Containers

Easy Deployment

[ Runnable Example ]

Let’s shift gears again and take a look at containers. Pulumi supports programming against raw container orchestrators – Amazon’s ECS, Azure’s ACS, Google Cloud’s GKE, Kubernetes directly, and so on – using the cloud resources directly. In this mode, Pulumi is entirely unopinionated about how containers are built, published, and deployed to your orchestrator.

It is important that Pulumi supports that low-level approach, and is what distinguishes it compared to PaaS-like offerings. This entails significant spelunking at a level that is only loosely connected to the task of standing up a container, however, which can be time-consuming and frustrating for many developers. To make this common case easier, Pulumi also offers higher-level abstractions to make these common cases easier, similar in spirit to Docker Compose, except that you can mix these abstractions with all your other cloud resources.

For example, to create a load balanced Nginx container using an image from the Docker Hub:

index.ts:

import * as cloud from "@pulumi/cloud";

let nginx = new cloud.Service("nginx", {
 image: "nginx",
 ports: [{ port: 80 }],
 replicas: 2,
});

export const url = nginx.defaultEndpoint;

This is all you need to run Nginx with 2 load balanced replicas listening on port 80. By default, this will run in Fargate when targeting AWS, which means you can skip the complications of configuring an orchestrator. After running pulumi up, the auto-assigned URL will be printed.

The image directive supports any normal Docker image URL, so private registries work just as well as the default public Docker Hub. And there are a number of other supported properties for memory and CPU constraints, advanced port mapping, and volumes.

Now things get a bit more interesting. Let’s say we wanted to build our own custom container instead. Normally that would mean defining a Dockerfile, provisioning a private registry, and manually building and publishing your image to the registry, and then consuming that container image from the registry. That’s a lot of boilerplate scripting!

Pulumi unifies all container building, publishing, and consumption underneath a simple pulumi up:

index.ts:

import * as cloud from "@pulumi/cloud";

let nginx = new cloud.Service("nginx", {
 build: ".",
 ports: [{ port: 80 }],
 replicas: 2,
});

export const url = nginx.defaultEndpoint;

Dockerfile:

FROM nginx
COPY ./www /usr/share/nginx/html

This is trivial Dockerfile that derives from the nginx base image and copies the ./www directory into the Nginx HTML target so that it will be served up. This is our directory structure:

.
├── Dockerfile
├── Pulumi.yaml
├── index.ts
├── package.json
└── www
├── favicon.png
└── index.html
1 directory, 5 files

Of course, this Dockerfile could be arbitrarily complex, using any Docker image.

No Dockerfiles

[ Runnable Example ]

The above example showed you how to create services that use Docker containers, either via a pre-built image stored in a registry like the Docker Hub, or by building one from a Dockerfile. The cloud.Service class supports a third possibility, which is really fun, as it takes advantage of Pulumi’s serverless programming model earlier to define a container and its service.

Imagine we want to define a simple Node.js webserver rather than using Nginx like this. Perhaps it uses Express.js. For this blog post, however, let’s stick with the most basic http-module-based webserver:

index.ts:

import * as cloud from "@pulumi/cloud";

let www = new cloud.Service("www", {
 function: () => {
 let rand = Math.random();
 require("http").createServer((req, res) => {
 res.end(`Hello, world! (from ${rand})`);
 }).listen(80);
 },
 ports: [{ port: 80 }]
 memory: 128,
 replicas: 2,
});

export const url = www.defaultEndpoint;

This code defines a load balanced service, backed by a container that Pulumi builds on your behalf out of the body of the function provided in the program, all done with a simple pulumi up gesture. This function, of course, can close over other state just like the earlier serverless examples, including cloud resources, making it as easy as can be to connect your microservices.

Long Running Containers as Tasks

[ Runnable Example ]

The above examples are great if what you want is a load balanced, long-running service. Sometimes a container is useful simply for executing a long-running task, or one that has very specific requirements about its execution environment (such as a specific base image).

For those cases, a cloud.Task is just what the doctor ordered. Task supports all the same construction techniques as Service, so you can give it a pre-built image, a build directive, or an inline function. For purposes of illustration, let’s build a custom Dockerfile that performs an FFMPEG transformation, to extract a thumbnail anytime someone adds a video to a bucket:

index.ts:

let cloud = require("@pulumi/cloud-aws");
// A bucket to store videos and thumbnails.
let videos = new cloud.Bucket("bucket");
// A task which runs an FFMPEG transform to extract a thumbnail image.
let ffmpegThumbnailTask = new cloud.Task("ffmpegThumbTask", {
 build: ".",
 memoryReservation: 512,
});

// When a new video is uploaded, run the FFMPEG task on the video file.
videos.onPut(
 "onNewVideo",
 args => {
 let file = args.key;
 ffmpegThumbnailTask.run(
 {
 environment: {
 "S3_BUCKET": videos.bucket.name.get(),
 "INPUT_VIDEO": file,
 "TIME_OFFSET": file.substring(file.indexOf('_')+1, file.indexOf('.')).replace('-',':'),
 "OUTPUT_FILE": file.substring(0, file.indexOf('_')) + '.jpg',
 },
 });
 },
 { keySuffix: ".mp4" });

exports.bucketName = videos.bucket.name;

Dockerfile:

FROM jrottenberg/ffmpeg
RUN apt-get update && apt-get install python-dev python-pip -y && apt-get clean
RUN pip install awscli
WORKDIR /tmp/workdir
ENTRYPOINT
echo "Starting ffmpeg task... " &&
echo "Copying video from s3://${S3_BUCKET}/${INPUT_VIDEO} to ${INPUT_VIDEO}..." &&
aws s3 cp s3://${S3_BUCKET}/${INPUT_VIDEO} ./${INPUT_VIDEO} &&
ffmpeg -v error -i ./${INPUT_VIDEO} -ss ${TIME_OFFSET} -vframes 1 -f image2 -an -y ${OUTPUT_FILE} &&
echo "Copying thumbnail to S3://${S3_BUCKET}/${OUTPUT_FILE} ..." &&
aws s3 cp ./${OUTPUT_FILE} s3://${S3_BUCKET}/${OUTPUT_FILE}

Notice that, just like before, this is backed by an ordinary Dockerfile. This example actually showcases serverless and containers living alongside one another in harmony. Notice how we kick off the task – simply by capturing a reference to the Task from the serverless event handler, and invoking its run method. This leverages Pulumi’s ability to give resources like Tasks custom APIs that can be invoked at runtime, something we will return to again later on.

General Tips and Tricks

Avoid Hard-Coding Config

Let’s return to our webserver example from earlier. Unfortunately, this code will only work in one region, because we hard-coded the "ami-6869aa05" image name (AMI). (Image names in AWS are region-specific.) In general, we prefer to build Pulumi programs that can be multi-instantiated across different regions. Pulumi’s stacks model streamlines this workflow, making it easy to spin up fresh copies of your entire application and/or infrastructure when you need to scale, while also reducing friction during development and testing. Such hard-coding is a definite code smell and impediment, however.

Because we are using a familiar language, we can do anything that general-purpose languages can do. That includes dynamically querying sources of information which, in this case, can be used to avoid hard-coding. We can even use the AWS SDK, which gives us full access to all AWS functionality:

ami.ts:

import * as aws from "@pulumi/aws";

export function getLinuxAmi(): Promise<string> {
 return new Promise<string>((resolve, reject) => {
 let ec2 = new (require("aws-sdk")).EC2({ region: aws.config.requireRegion() });
 ec2.describeImages(
 {
 Owners: [ "amazon" ],
 Filters: [{
 Name: "name",
 Values: [ "amzn-ami-hvm-????.??.?.x86_64-gp2" ],
 }],
 },
 (err: any, data: any) => {
 if (err) {
 reject(err);
 } else {
 let newestImage: string | undefined;
 let newestImageDate: number | undefined;
 for (let image of data.Images) {
 let imageDate = Date.parse(image.CreationDate);
 if (!newestImage || !newestImageDate ||
 imageDate > newestImageDate) {
 newestImage = image.ImageId;
 newestImageDate = imageDate;
 }
 }
 if (newestImage) {
 console.log(`AMI: ${newestImage} (${newestImageDate})`);
 resolve(newestImage);
 } else {
 reject("No Linux AMI found for the current region");
 }
 }
 },
 );
 });
}

I’ll be the first to admit that this is code that only a mother could love. Even better would be to wrap it up in a reusable NPM package that others can use without the boilerplate! Either way, it’s very powerful to have the full capabilities of a language at our fingertips, as this sort of thing simply isn’t possible in other infra-as-code solutions.

Now that we have this function, we can revisit the way we set the AMI property on our VMs:

web.ts:

import * as aws from "@pulumi/aws";
import { getLinuxAmi } from "./ami";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

export class WebServer {
 public readonly vm: aws.ec2.Instance;

 constructor(name: string) {
 this.vm = new aws.ec2.Instance(`${name}`, {
 instanceType: "t2.micro",
 ami: getLinuxAmi(),
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello from ${name}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 });
 }
}

Given this change, we can safely deploy our WebServer to any AWS region.

Multi-Instantiation and Code Reuse

In the webserver example, we used the aws.config.requireRegion() function to obtain the configured AWS region. Using configuration in this manner lets us multi-instantiate stacks; in other words, to easily stand up multiple copies of a service, application, or infrastructure.

We can even make the quantity configurable using Pulumi’s config system:

index.ts:

import { WebServer } from "./web";
import { Config, getProject } from "@pulumi/pulumi";

let config = new Config(getProject());
let webServers = [];
let webServersCount = config.getNumber("count") || 3;
for (let i = 0; i < webServersCount; i++) {
 webServers.push(new WebServer(`web-server-${i}`));
}

export const publicHostnames = webServers.map(s => s.vm.publicDns);

Given this specific program, we can easily change the number of servers created from the default of three, to something larger:

$ pulumi config set count 7

There are plenty of opportunities for configuration like this, like the machine size. By making such settings configurable, different environments can easily have different settings, while still sharing the same code beneath them, eliminating copy-and-paste and its associated pitfalls.

Give Components Runtime APIs

[ Runnable Example ]

As our final example, let’s see how to give your own components APIs that can be invoked at runtime. We’ve already seen this a few times now, such as managing data in a cloud.Table that has been captured in a lambda, and cloud.Task’s run method. Pulumi is built to be extensible, so you can define your own abstractions just like these with APIs too.

For this example, let’s take a simple Redis cache service and turn it into a component. After doing so, we can then add simple get/set APIs on it that let you interact with the cache at runtime in simple ways. By providing this sort of abstraction, we open up interesting possibilities. For instance, our default implementation can use a custom container-backed service, via cloud.Service, but opt to use a simpler hosted solution when running in a public cloud – AWS ElastiCache, Azure Redis Cache, or Memorystore in Google Cloud.

To make our component, we will simply allocate a cloud.Service in the constructor, taking an optional memory parameter to control its memory reservation. We are using configuration to read the Redis password using Pulumi’s built-in secrets mechanism. To give that component APIs, we just define methods on the class that access Redis through its own NPM package, and that’s it! The way lambda capture works will ensure that these functions are accessible:

cache.ts:

import * as cloud from "@pulumi/cloud";
import * as config from "./config";

// A simple cache abstraction that wraps Redis.
export class Cache {
 public readonly get: (key: string) => Promise<string>;
 public readonly set: (key: string, value: string) => Promise<void>;
 private readonly redis: cloud.Service;

 constructor(name: string, memory: number = 128) {
 this.redis = new cloud.Service(name, {
 image: "redis:alpine",
 memory: memory,
 ports: [{ port: 6379 }],
 command: ["redis-server", "--requirepass", config.redisPassword],
 });
 }

 public async get(key: string): Promise<string> {
 let client = await this.createRedisClient();
 return new Promise<string>((resolve, reject) => {
 client.get(key, (err: any, v: any) => {
 if (err) reject(err);
 else resolve(v);
 });
 });
 }

 public async set(key: string, value: string): Promise<void> {
 let client = await this.createRedisClient();
 return new Promise<void>((resolve, reject) => {
 client.set(key, value, (err: any, v: any) => {
 if (err) reject(err);
 else resolve();
 });
 });
 }

 private async createRedisClient(): Promise<any> {
 let ep = (await this.redis.defaultEndpoint).get();
 return require("redis").createClient(
 ep.port, ep.hostname, { password: config.redisPassword });
 }
 }
}

Because this is just a class, we have all the standard programming facilities available to us. If we wanted to specialize it to use our cloud’s hosted Redis services, we could do that using the standard techniques, including conditional code or, slightly more elegantly, using concrete subclasses for the different options.

Now that we’ve defined our cache, we can create an instance, and then use it from within a callback, such as a serverless API implementation. This is a fairly complex application that acts as a URL shortener, with both GET and POST endpoints, but really shows off the power of combining many of the techniques we’ve seen throughout this post:

index.ts:

import * as cloud from "@pulumi/cloud";
import * as cache from "./cache";

// Create a URL shortener app and a table to hold shortened URLs.
let app = new cloud.API("shortener");
let urlTable = new cloud.Table("urls", "name");

// Use our cache component to cache frequently accessed URLs.
let urlCache = new cache.Cache("urlcache");

// GET /url lists all URLs currently registered.
endpoint.get("/url", async (req, res) => {
 try {
 res.status(200).json(await urlTable.scan());
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

// GET /url/{name} redirects to the target URL based on a short-name.
endpoint.get("/url/{name}", async (req, res) => {
 let name = req.params["name"];
 try {
 // First try the Redis cache. If we miss, consult the table.
 let url = await urlCache.get(name);
 if (url) {
 res.setHeader("X-Powered-By", "redis");
 } else {
 let value = await urlTable.get({ name });
 if (value && value.url) {
 url = value.url;
 urlCache.set(name, url); // cache for next time.
 }
 }

 // If we found an entry, 301 redirect to it; else, 404.
 if (url) {
 res.setHeader("Location", url);
 res.status(302);
 res.end("");
 } else {
 res.status(404);
 res.end("");
 }
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

// POST /url registers a new URL with a given short-name.
endpoint.post("/url", async (req, res) => {
 let url = req.query["url"];
 let name = req.query["name"];

 try {
 // Insert into the table and our URL cache.
 await urlTable.insert({ name, url });
 await urlCache.set(name, url);
 res.json({ shortenedURLName: name });
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

export const url = endpoint.publish().url;

Notice that we can capture and use our Cache component just like we can the cloud.Table!

We’ve only scratched the surface of what’s possible with components and custom APIs, however hopefully you’re starting to get a sense of why this is such a powerful capability.

Winding Down

In this post, we’ve seen how Pulumi can define infrastructure, containers, and serverless cloud software, often combining many of the techniques together to build powerful distributed applications. This includes programming to specific resources that your cloud of choice provides, in addition to writing higher level cloud programs that are able to target many clouds (expect more multi-cloud progress here in the weeks to come – including Kubernetes). Each pearl is a topic unto itself; we’d love your feedback on what topics to dig into next.

Happy hacking – and have fun programming the cloud!

Introducing Pulumi, a Cloud Development Platform

Joe Duffy — Mon, 18 Jun 2018 00:00:00 +0000

Ahoy!

Today we launched Pulumi, an open source cloud development platform, providing the cloud’s first true programming model using real programming languages.

We believe that containers, serverless, and hosted cloud services change everything about how we design and build software and applications. All developers these days are cloud developers. We saw a big gap in the way people want to create this software, and how they are required to do it today with configuration YAML and templating languages, and sought to close that gap.

Pulumi brings all the things we know and love with programming languages – IDEs, abstraction, encapsulation, sharing and reuse, and more – to the creation of all aspects of cloud programs and applications, while retaining the full power of immutable infrastructure across Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP), as well as Kubernetes running in any public, private, or hybrid cloud. An open ecosystem of packages promises to eliminate copy-and-paste, and encourage sharing of best practices within teams and the community.

We started on Pulumi a little over a year ago, have built an incredible team, and I’m thrilled to announce its availability. Download or learn more about this unique cloud development platform here, or read more background on Pulumi’s motivation and world-view over on my personal blog.

The team can’t wait to see all the incredible things you will build using Pulumi.