Pulumi Blog: Security

Sign in to Pulumi Cloud with Passkeys

Devon Grove — Mon, 13 Jul 2026 00:00:00 +0000

Pulumi Cloud now supports passkeys for users who sign in with email and password. Select a button, approve with Touch ID, Face ID, Windows Hello, or your hardware key, and you’re signed in.

A passkey is a public-key credential stored on your device: your phone, your laptop, a hardware key (YubiKey, Google Titan, etc.), or your password manager can all function as the authenticator. When you sign in, your device authenticates you locally and signs a challenge from Pulumi Cloud with the private key.

The private key stays on your device — Pulumi Cloud never sees or stores it. Passkeys are built on the WebAuthn standard, so they’re already supported on every major browser and operating system.

Who this is for

This release applies to users who sign in to Pulumi Cloud with an email address and password. If you sign in through an identity provider (IdP), such as GitHub OAuth, GitLab, Bitbucket, Google, or your organization’s SAML SSO, your existing flow is unchanged.

Why passkeys

Passwords have always been the weakest link in account security. Since they are shared secrets, they are vulnerable to phishing attacks, and every place you type one is a place that can be impersonated or a data store that can be leaked. Passkeys swap that out for a per-site key pair that lives on your device:

Phishing-resistant by design. A passkey is bound to the exact origin it was registered for. A look-alike domain can’t trigger your authenticator.
Synced across your devices. Apple iCloud Keychain, Google Password Manager, 1Password, Dashlane, Bitwarden: most credential managers now sync passkeys end-to-end-encrypted to every device you’ve signed in on.
Discoverable. Pulumi Cloud doesn’t need to know which user you are before you authenticate. Just select “Sign in with a passkey” and your device offers the right credential.
Nothing to remember. A passkey lives on your device. There’s no string to memorize, and no sensitive credential stored by us.

Setting up a passkey

The next time you sign in with a password, Pulumi Cloud will ask if you’d like to configure a passkey. It’s a single-step prompt, so you can enroll once and take advantage of passkeys on every subsequent sign-in.

If you dismiss the prompt and would like to add one later, or want to add multiple passkeys, navigate to Account Settings → Passkeys under your user profile. Select Register a passkey, complete the OS-level prompt (Touch ID, Face ID, Windows Hello, or your hardware key), and you’re done. Pulumi Cloud will pick a sensible default name like "iCloud Keychain" or "Chrome on macOS" based on the authenticator, but you can rename it inline anytime under Account Settings → Passkeys.

You can register as many passkeys as you want. Typical setups are one per personal device, or one synced credential plus a hardware key as backup. Removing a passkey takes effect immediately; deleted credentials cannot be used to sign in.

Signing in

Depending on how you arrive:

Explicit sign-in. On the sign-in page, select Sign in with a passkey. Your browser opens the passkey picker, and you authenticate from there.
Autofill. If your browser supports conditional mediation (Chrome, Safari, Edge, recent Firefox), the email field on the sign-in page proactively offers your registered passkeys as autofill suggestions. Pick one and you’re signed in.

What about my existing password and MFA?

Your existing password still works. Passkeys are an additional sign-in option, not a replacement, and they don’t disable password sign-in on your account. If you lose access to every registered passkey, you can still sign in with your email and password as you always have, then register a new passkey from settings.

Other 2FA still applies. If you have TOTP-based MFA enabled on your Pulumi account, a passkey sign-in will still prompt for your second factor. WebAuthn doesn’t reliably tell us how you unlocked the passkey on your device (biometric, PIN, or something weaker), so we can’t safely treat the passkey itself as proof of two factors. Your existing MFA configuration remains the boundary it always was.

Try it out

Passkey support is generally available today for every Pulumi Cloud user who signs in with email and password, at no additional cost and with no configuration. Visit your account settings to register your first passkey, and let us know what you think on our community Slack or GitHub.

Enforce ISO 27001 Across Your AWS Infrastructure

Dan Biwer — Tue, 30 Jun 2026 00:00:00 +0000

ISO/IEC 27001 is the international standard for information security management. Proving you meet it usually means months of mapping abstract security controls to concrete cloud configuration, then authoring custom checks one resource at a time. We’re changing that.

Today we’re shipping a pre-built ISO/IEC 27001:2022 policy pack for AWS, live now in Pulumi Cloud as iso-27001-aws. It encodes the standard’s security expectations as 238 ready-to-run policies, so you can align your AWS estate to ISO 27001 in minutes, not months.

Why ISO 27001 matters

For many companies, ISO 27001 is what stands between them and a customer or a market. The sooner you can reach a certifiable state and prove you stay there, the less compliance slows the business down. The pack collapses months of policy work into something you run continuously, so security keeps pace with growth instead of blocking it.

How the pack maps to ISO 27001

The hard part of ISO 27001 has always been translation: its controls are written in the language of governance and risk management, not in the language of AWS resources. Every team has had to interpret each control and decide what it means for an S3 bucket or an RDS instance.

The pack does that interpretation for you. Its 238 policies are aligned to the relevant ISO 27001 controls, so each result connects back to the standard instead of leaving you to map it yourself. You can browse the full pack in the pack reference.

Audit and prevent

The same pack works two ways, so you can both reach compliance and stay there:

Audit. Scan your existing AWS estate against the pack, including resources that Pulumi doesn’t manage. You get an honest baseline of where you stand against ISO 27001 today, with every finding tied back to the control it affects.
Prevent. Run the same pack as a preventative policy during pulumi up to block non-compliant resources before they’re ever created. New infrastructure is born aligned to the standard.

Audit gets you clean. Preventative policies keep you clean.

A growing library of pre-built packs

ISO 27001 joins a growing library of pre-built packs for AWS, each authored and maintained by Pulumi and kept current with its source standard:

ISO/IEC 27001:2022
CIS Controls v8.1
NIST SP 800-53 Rev. 5
PCI DSS v4.0
HITRUST CSF v11.5
Pulumi Best Practices

Adopting any pack means you skip the authoring work entirely, inherit framework mappings maintained by Pulumi, and apply a consistent baseline across every stack and account.

Get started today

The ISO 27001 pack is available now to every Pulumi Cloud user:

Browse the pack reference to see all 238 policies and how they map to the standard’s controls.
Explore the full pre-built packs index.
Follow the get-started guide to run your first audit.

Try Pulumi policies

Ready to align your AWS infrastructure to ISO 27001? Sign up for Pulumi Cloud and run the pack against your estate, or read the policy get-started guide to dig in.

Need a compliance pack for a framework that isn’t listed here? Open a request in pulumi/pulumi-cloud-requests or come tell us in the community Slack. We’re listening.

Deploy a Private Hermes Agent on Render Securely with Pulumi, Modal, and Tailscale

Engin Diri — Wed, 24 Jun 2026 00:00:00 +0000

Personal AI agents had their breakout this year. OpenClaw crossed 100,000 GitHub stars within months of launching, and self-hosting your own assistant went from a hobbyist trick to something a lot of developers actually do. I wrote up how to deploy that lobster to AWS or Hetzner back when it was everywhere.

The one people are switching to now is Hermes, the open-source runtime from Nous Research, and it caught on just as quickly. The reason shows up in every “I ditched OpenClaw for Hermes” thread: it actually learns, building up memory and writing its own skills as it goes instead of running off a static, human-written list.

Here is the part the launch videos skip. Hermes writes and runs its own code, with no human approving the commands. A model that can write code will eventually write a bad one, and the only thing between that command and your credentials is the sandbox it runs in. That is the box you do not want on the public internet. Researchers found 175,000 exposed Ollama servers sitting open in early 2026, and attackers hijack the ones they find for compute. The fix is not a better lock on the front door. It is to have no front door at all.

So this post deploys a private Hermes agent as one Pulumi program across Render, Modal, and Tailscale. The agent and its chat UI run as Render private services with no public URL, Tailscale puts the UI on your tailnet, and Modal runs the agent’s code in throwaway sandboxes. One pulumi up to stand it up, one pulumi destroy to tear it down, around $50 a month. No Makefile, no CLI escape hatches, and an honest look at the few places “nothing on the public internet” still leaks.

What is Hermes?

Hermes is the open-source, MIT-licensed agent runtime from Nous Research. It runs continuously on a server rather than living in a browser tab, and you talk to it through a chat UI or a messaging platform like Telegram, Discord, Slack, or Signal. A few things make it more than a chatbot:

It is model-agnostic. You point it at any provider. This deployment uses OpenRouter, one key that fronts 400-plus models across providers, with automatic failover when one is down.
It remembers across restarts. Memory persists on disk, so it builds up context over time instead of resetting every session.
It writes its own skills. When it works through a task, it can save that approach as a reusable skill and reach for it next time.
It schedules itself. A built-in cron runs automations on a schedule, driven in plain language.
It runs code in a sandbox. When the agent needs to execute code, it hands that off to an isolated, ephemeral container instead of running it next to its own process.

The difference from a cloud-hosted assistant is the usual self-hosting trade: it runs on your infrastructure with your keys, and your conversation history and memory stay on your disk.

“Hermes” here is the agent runtime from Nous Research, not the Hermes family of language models that share the name. The agent is model-agnostic, and the model it talks to is a separate choice you make through OpenRouter.

Prerequisites

Before getting started, ensure you have:

Pulumi CLI installed and configured
A Pulumi Cloud account
A Render account, an API key, and your workspace owner ID
A Modal account and a token pair (token ID and secret)
An OpenRouter API key with credit on the account
A container registry you can push to; this post uses GitHub Container Registry (a GHCR username and token)
Docker running locally (the image build rides on a local buildx engine), or a Docker-enabled CI runner
A Tailscale account with HTTPS enabled and a reusable auth key
Node.js 18+ (for TypeScript) or Python 3.9+ (for Python), depending on the language you pick

This guide routes models through OpenRouter, but Hermes works with other providers too. The model is just a routing string you pass (gpt-5.5 here), so any model OpenRouter supports works by changing that one value.

Understanding the Hermes architecture

Strip away the dashboard clicks and the architecture is four pieces. One rule governs all of them: what the public internet is allowed to touch.

flowchart LR
User([Your devices]) -->|"tailnet (WireGuard)"| TS["Tailscale<br/>private mesh"]
TS -->|"open-webui.&lt;tailnet&gt;.ts.net"| WebUI["Open WebUI<br/>(Render private service,<br/>joined to your tailnet)"]
WebUI -->|"private network<br/>&lt;hermes-slug&gt;:8642"| Hermes["Hermes gateway<br/>(Render private service)"]
Hermes -->|"Modal SDK at runtime"| Modal["Modal sandbox<br/>isolated code execution"]
Hermes -->|model calls| OR["OpenRouter"]

Component	Port	Description
Hermes gateway	8642	Holds secrets and memory, talks to the model, creates Modal sandboxes. The most dangerous box, so it has no public URL.
Open WebUI	8080	The chat front end. Joins your tailnet and reaches Hermes over Render’s private network.
Modal sandbox	-	Isolated, ephemeral container for running the agent’s code.
Tailscale	-	A private WireGuard mesh that makes the UI reachable to your devices only.

The agent that can run code and read secrets is never routable from the internet, and neither is the UI in front of it. There is no inbound ingress on any box (the outbound calls to the model, to Modal, and to the image registry still cross the internet, as they have to). Expressing “which box can reach which” in code is what makes the boundary auditable: the rule lives in the program rather than in a dashboard.

The four pieces split cleanly in two: two you declare as resources, and two you hand a credential and let run.

Layer	How you provision it	Status
Render services, disks, secrets	official `render-oss` Terraform provider, bridged	Declarative, via one `pulumi package add`.
Container images	`@pulumi/docker-build`	Built and pushed during `pulumi up`. No `docker` CLI.
Modal sandboxes	nothing	No Terraform provider exists. A runtime credential.
Tailnet access (Tailscale)	nothing	No provider needed. A runtime credential, like Modal.

Two of these four layers are credentials, not resources. That distinction often gets missed, and it is worth getting right: a provider can only manage what it can provision, and spinning up a Modal sandbox is a runtime act, not a provisioned resource. We come back to it at each layer.

Setting up ESC for secrets management

Deploying Hermes means handling a pile of credentials: a Render API key, registry credentials, the OpenRouter key, the Modal token pair, and the Tailscale auth key. You don’t want these hardcoded or scattered across environment variables. Pulumi ESC (Environments, Secrets, and Configuration) stores them securely and passes them directly to your Pulumi program.

Create a new ESC environment:

pulumi env init <your-org>/hermes-secrets

Add your secrets to the environment:

values:
 renderApiKey:
 fn::secret: "rnd_xxxxx"
 renderOwnerId: "tea-xxxxx"
 ghcrUser: "your-github-username"
 ghcrToken:
 fn::secret: "ghp_xxxxx"
 openrouterApiKey:
 fn::secret: "sk-or-xxxxx"
 modalTokenId:
 fn::secret: "ak-xxxxx"
 modalTokenSecret:
 fn::secret: "as-xxxxx"
 tailscaleAuthKey:
 fn::secret: "tskey-auth-xxxxx"
 tailnetDnsName: "tailxxxxx.ts.net"
 pulumiConfig:
 renderApiKey: ${renderApiKey}
 ownerId: ${renderOwnerId}
 ghcrUser: ${ghcrUser}
 ghcrToken: ${ghcrToken}
 openrouterApiKey: ${openrouterApiKey}
 modalTokenId: ${modalTokenId}
 modalTokenSecret: ${modalTokenSecret}
 tailscaleAuthKey: ${tailscaleAuthKey}
 tailnetDnsName: ${tailnetDnsName}

To find your tailnet DNS name, go to the Tailscale admin console, look under the DNS section, and find your tailnet name (e.g., tailxxxxx.ts.net). This is the domain suffix used for all machines in your Tailscale network.

Then create a Pulumi.dev.yaml file in your project to reference the environment:

environment:
 - <your-org>/hermes-secrets

This keeps your secrets out of your codebase and hands them to the program at deploy time. The Modal token pair and the Tailscale auth key never become resources; they ride straight through as encrypted env vars on the services that need them.

Generating the Render SDK

Render has no first-party Pulumi provider. What it has is an official, maintained Terraform provider, and Pulumi can turn any Terraform provider into a typed local SDK with Any Terraform Provider:

pulumi package add terraform-provider render-oss/render 1.8.0

That one command generates a render SDK you import like any other package, in whatever language your stack is written in. There is no published @pulumi/render package and no vendor to wait on.

This is also where the provider map shows its edges. The bridge wraps a Terraform provider that already exists, so it works for Render. But Modal has no Terraform provider anywhere, so pulumi package add terraform-provider modal-labs/modal has nothing to fetch. That is not a gap to work around. Modal is not a provisioned resource here; Hermes creates sandboxes at runtime through the Modal SDK, and Pulumi’s only job is to hold the token pair and hand it over. Tailscale lands in the same bucket: a Tailscale provider exists, but it manages your tailnet’s settings, not the act of putting a container on the network, which is a runtime job done inside the image.

Securing with Tailscale

The usual way to make a self-hosted UI reachable is to give it a public hostname and put an identity gate in front: an SSO provider, an access policy, a public DNS record, and a certificate to keep renewing. Tailscale collapses all of it into nothing you declare. There is no gate to bolt in front of a public URL, because there is no public URL. Open WebUI joins your tailnet and answers only to your own devices.

This is a stronger position than a public UI behind a login. Because there is no public URL, there is no login to harden and no certificate to renew, and Open WebUI answers only to your own devices.

Tailscale lives in the Open WebUI image rather than the Pulumi program. The image adds Tailscale and an entrypoint that brings up tailscaled in userspace mode (a container has no /dev/net/tun), joins your tailnet, and runs tailscale serve to expose Open WebUI on it. serve publishes over HTTPS to authenticated members of your tailnet only, where its sibling tailscale funnel would put the same port on the public internet, the one thing this design avoids.

FROM ghcr.io/open-webui/open-webui:main-slim
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates \
 && curl -fsSL https://tailscale.com/install.sh | sh
COPY entrypoint.sh /usr/local/bin/ts-entrypoint
ENTRYPOINT ["/usr/local/bin/ts-entrypoint"]

#!/usr/bin/env bash
set -e
PORT="${PORT:-8080}"
mkdir -p /app/backend/data/tailscale
# Userspace networking: a container has no /dev/net/tun. Keep node state on the disk
# so it survives restarts, and accept-dns=false so Render's DNS still resolves the
# private Hermes hostname.
tailscaled --tun=userspace-networking --statedir=/app/backend/data/tailscale &
n=0
until tailscale up --authkey="${TS_AUTHKEY}" --hostname="${TS_HOSTNAME}" --accept-dns=false; do
 n=$((n + 1)); [ "${n}" -ge 30 ] && { echo "tailnet join failed; check TS_AUTHKEY"; exit 1; }
 sleep 2
done
tailscale serve --bg "${PORT}" # serve Open WebUI on the tailnet over HTTPS
cd /app/backend && exec bash start.sh

The TS_AUTHKEY env var on the service, pulled from ESC, is the only thing Pulumi contributes; everything else is the image doing its own networking.

There is one prerequisite, the same one any Tailscale node needs for serve: enable HTTPS for your tailnet once in the admin console so it can get a certificate. Use a reusable auth key, since the service re-authenticates on every redeploy, and the join loop has a ceiling so a dead key fails the deploy in a minute instead of hanging it.

To generate the reusable auth key:

Go to the Tailscale admin console.
Click Generate auth key.
Enable Reusable, since the service re-authenticates on every redeploy.
Copy the key into your ESC environment as tailscaleAuthKey.

Deploying the agent

This is the part a Makefile usually handles. Two private services, two disks, every secret an encrypted env var, and both images built and pushed during the same pulumi up that creates the services. No CLI runs anywhere, though the build still rides on a local Docker engine through buildx: the docker call is gone, the daemon it talks to is not, so pulumi up needs Docker running.

Create a new Pulumi project (pick the language you prefer; the companion repo has all three):

mkdir hermes-agent-pulumi && cd hermes-agent-pulumi
pulumi new typescript

Install the dependencies and generate the Render SDK:

npm install @pulumi/docker-build @pulumi/random
pulumi package add terraform-provider render-oss/render 1.8.0

Both services need the 2 GB Standard plan. The gateway and Open WebUI each exhaust memory on the 512 MB Starter and OOM on boot.

Two facts make this clean. Render env var values are encrypted at rest, so a secret is only an env var, with no separate resource. And @pulumi/docker-build builds and pushes an image through an embedded buildx during pulumi up, so the image pipeline is declarative too. Render then pulls the pushed image by digest.

The complete program

Hermes comes first: a private Render service with no public URL, a disk for its memory, and every secret as an encrypted env var. The shared bearer key is generated in state with random.RandomBytes, the clean replacement for openssl rand -base64 32; the Modal token pair and the OpenRouter key come from config or Pulumi ESC. Render has no port field, so Hermes binds 0.0.0.0:8642 and Open WebUI reaches it at the service’s read-only slug. Open WebUI is almost the mirror image, private too, except its image joins your tailnet (the tailscaled entrypoint from the previous section) and it carries the operational env vars that keep it lean: no local embedding models, and ENABLE_PERSISTENT_CONFIG=false so your config always wins over a stale copy in its database. One render.Provider carries your renderApiKey and ownerId into every Render resource.

Here is the whole program, copy-paste ready as your index.ts (or __main__.py, or Pulumi.yaml):

import * as pulumi from "@pulumi/pulumi";
import * as dockerbuild from "@pulumi/docker-build";
import * as random from "@pulumi/random";
// Local SDK generated by: pulumi package add terraform-provider render-oss/render 1.8.0
// (no @pulumi/render is published; the SDK is bridged from the official Render provider).
import * as render from "@pulumi/render";

const cfg = new pulumi.Config();

// ---------------------------------------------------------------------------
// Configuration. Non-secret values via `pulumi config set`, secrets via
// `pulumi config set --secret` or a Pulumi ESC environment (see README).
// The Render provider reads renderApiKey/renderOwnerId (or RENDER_API_KEY/RENDER_OWNER_ID).
// ---------------------------------------------------------------------------
const region = cfg.get("renderRegion") ?? "oregon";
const modelDefault = cfg.get("modelDefault") ?? "gpt-5.5";

// Container registry the images are pushed to and Render pulls from (GHCR here).
const ghcrUser = cfg.require("ghcrUser");
const ghcrToken = cfg.requireSecret("ghcrToken"); // GHCR PAT with write:packages

// Long-lived secrets supplied by you (dashboard-minted). Hold in config or ESC.
const modalTokenId = cfg.requireSecret("modalTokenId");
const modalTokenSecret = cfg.requireSecret("modalTokenSecret");
const openrouterApiKey = cfg.requireSecret("openrouterApiKey");

// Tailscale: the only "gateway". Open WebUI joins your tailnet and is reachable
// at https://<hostname>.<tailnet>.ts.net by your devices only — never public.
// Both values are already in your ESC. Enable HTTPS once in the Tailscale admin console.
const tailscaleAuthKey = cfg.requireSecret("tailscaleAuthKey");
const tailnetDnsName = cfg.require("tailnetDnsName"); // e.g. tailc6fb4e.ts.net
const webuiHostname = "open-webui";
const webuiTailnetUrl = `https://${webuiHostname}.${tailnetDnsName}`;

// Render provider auth (from ESC: pulumiConfig.renderApiKey / pulumiConfig.renderOwnerId).
const renderProvider = new render.Provider("render", {
 apiKey: cfg.requireSecret("renderApiKey"),
 ownerId: cfg.require("renderOwnerId"),
});

// ---------------------------------------------------------------------------
// Secrets generated in state (the in-IaC replacement for `openssl rand -base64 32`).
// The shared bearer authenticates Open WebUI -> Hermes; it lands on both services.
// ---------------------------------------------------------------------------
const sharedKey = new random.RandomBytes("hermes-shared-key", { length: 32 });
const webuiSecretKey = new random.RandomBytes("webui-secret-key", { length: 32 });

// ---------------------------------------------------------------------------
// Build and push both images during `pulumi up` (no docker CLI; the provider
// embeds buildx). Render then pulls them by their digest-pinned ref.
// ---------------------------------------------------------------------------
const registryAuth = [{ address: "ghcr.io", username: ghcrUser, password: ghcrToken }];

const hermesImage = new dockerbuild.Image("hermes-image", {
 context: { location: "./hermes" },
 dockerfile: { location: "./hermes/Dockerfile" },
 platforms: ["linux/amd64"],
 tags: [pulumi.interpolate`ghcr.io/${ghcrUser}/hermes-agent:latest`],
 push: true,
 registries: registryAuth,
});

const webuiImage = new dockerbuild.Image("webui-image", {
 context: { location: "./openwebui" },
 dockerfile: { location: "./openwebui/Dockerfile" },
 platforms: ["linux/amd64"],
 tags: [pulumi.interpolate`ghcr.io/${ghcrUser}/open-webui:latest`],
 push: true,
 registries: registryAuth,
});

// Credential Render uses to pull the private GHCR images.
const ghcrCredential = new render.RegistryCredential("ghcr", {
 name: "ghcr",
 registry: "GITHUB",
 username: ghcrUser,
 authToken: ghcrToken,
}, { provider: renderProvider });

// ---------------------------------------------------------------------------
// Hermes: a PRIVATE Render service. No public URL; reachable only over Render's
// private network. A disk holds its memory and workspace. Env values are
// encrypted at rest, which is how secrets ride on Render (no separate flag).
// ---------------------------------------------------------------------------
const hermes = new render.PrivateService("hermes-agent", {
 name: "hermes-agent",
 plan: "standard", // 2 GB; the gateway OOMs on starter's 512 MB
 region: region,
 runtimeSource: {
 // Render wants the bare repo in imageUrl; the tag/digest go in their own field.
 image: {
 imageUrl: pulumi.interpolate`ghcr.io/${ghcrUser}/hermes-agent`,
 digest: hermesImage.digest,
 registryCredentialId: ghcrCredential.id,
 },
 },
 disk: { name: "hermes-data", mountPath: "/opt/data", sizeGb: 5 },
 envVars: {
 PORT: { value: "8642" },
 API_SERVER_ENABLED: { value: "true" },
 API_SERVER_HOST: { value: "0.0.0.0" },
 API_SERVER_PORT: { value: "8642" },
 API_SERVER_MODEL_NAME: { value: modelDefault },
 // Auto-approve the agent's shell/terminal hooks (no TTY in a container),
 // so it can run code in the Modal sandbox without an interactive prompt.
 HERMES_ACCEPT_HOOKS: { value: "1" },
 // Secrets (encrypted at rest on Render):
 API_SERVER_KEY: { value: sharedKey.base64 },
 OPENROUTER_API_KEY: { value: openrouterApiKey },
 // Modal is a runtime credential, not a provisioned resource: Hermes creates
 // sandboxes at runtime with this token pair. No Modal provider, no Modal CLI.
 MODAL_TOKEN_ID: { value: modalTokenId },
 MODAL_TOKEN_SECRET: { value: modalTokenSecret },
 },
}, { provider: renderProvider });

// Open WebUI reaches Hermes at its Render-assigned internal hostname (the read-only
// slug), not the bare service name. Same region + same workspace share the network.
const hermesApiBaseUrl = pulumi.interpolate`http://${hermes.slug}:8642/v1`;

// ---------------------------------------------------------------------------
// Open WebUI: also a PRIVATE Render service, but the container joins your tailnet
// (tailscaled in userspace + `tailscale serve`), so you reach it at
// https://open-webui.<tailnet>.ts.net from your devices. Never public; the gate
// is tailnet membership. It still reaches Hermes over Render's private network.
// ---------------------------------------------------------------------------
const webui = new render.PrivateService("open-webui", {
 name: "open-webui",
 plan: "standard", // 2 GB; Open WebUI OOMs on starter's 512 MB
 region: region, // must match Hermes for private networking
 runtimeSource: {
 image: {
 imageUrl: pulumi.interpolate`ghcr.io/${ghcrUser}/open-webui`,
 digest: webuiImage.digest,
 registryCredentialId: ghcrCredential.id,
 },
 },
 disk: { name: "openwebui-data", mountPath: "/app/backend/data", sizeGb: 5 },
 envVars: {
 PORT: { value: "8080" }, // tailscale serve points at this; Open WebUI binds it
 OPENAI_API_BASE_URL: { value: hermesApiBaseUrl },
 ENABLE_OPENAI_API: { value: "true" },
 ENABLE_OLLAMA_API: { value: "false" },
 DEFAULT_MODELS: { value: modelDefault },
 WEBUI_AUTH: { value: "true" },
 WEBUI_URL: { value: webuiTailnetUrl },
 // Reached only over the tailnet, so the first sign-up is the admin.
 ENABLE_LOGIN_FORM: { value: "true" },
 ENABLE_SIGNUP: { value: "true" },
 DEFAULT_USER_ROLE: { value: "admin" },
 // Always apply env config; otherwise Open WebUI persists the first run's config
 // to its DB and ignores later changes.
 ENABLE_PERSISTENT_CONFIG: { value: "false" },
 // Don't load local embedding/RAG models (a big memory saver; the agent is the brain).
 RAG_EMBEDDING_ENGINE: { value: "openai" },
 BYPASS_EMBEDDING_AND_RETRIEVAL: { value: "true" },
 OFFLINE_MODE: { value: "true" },
 HF_HUB_OFFLINE: { value: "1" },
 // Secrets (encrypted at rest):
 OPENAI_API_KEY: { value: sharedKey.base64 }, // shared bearer to call Hermes
 WEBUI_SECRET_KEY: { value: webuiSecretKey.base64 },
 // Tailscale joins this container to your tailnet and serves Open WebUI on it.
 TS_AUTHKEY: { value: tailscaleAuthKey },
 TS_HOSTNAME: { value: webuiHostname },
 },
}, { dependsOn: hermes, provider: renderProvider });

export const hermesInternalUrl = hermesApiBaseUrl;
// Open this from any device on your tailnet:
export const openWebUiTailnetUrl = webuiTailnetUrl;

import pulumi
import pulumi_docker_build as docker_build
import pulumi_random as random

# Local SDK generated by: pulumi package add terraform-provider render-oss/render 1.8.0
# (no pulumi_render is published; the SDK is bridged from the official Render provider).
import pulumi_render as render

cfg = pulumi.Config()

# ---------------------------------------------------------------------------
# Configuration. Non-secret values via `pulumi config set`, secrets via
# `pulumi config set --secret` or a Pulumi ESC environment (see README).
# The Render provider reads renderApiKey/renderOwnerId (or RENDER_API_KEY/RENDER_OWNER_ID).
# ---------------------------------------------------------------------------
region = cfg.get("renderRegion") or "oregon"
model_default = cfg.get("modelDefault") or "gpt-5.5"

# Container registry the images are pushed to and Render pulls from (GHCR here).
ghcr_user = cfg.require("ghcrUser")
ghcr_token = cfg.require_secret("ghcrToken") # GHCR PAT with write:packages

# Long-lived secrets supplied by you (dashboard-minted). Hold in config or ESC.
modal_token_id = cfg.require_secret("modalTokenId")
modal_token_secret = cfg.require_secret("modalTokenSecret")
openrouter_api_key = cfg.require_secret("openrouterApiKey")

# Tailscale: the only "gateway". Open WebUI joins your tailnet and is reachable
# at https://<hostname>.<tailnet>.ts.net by your devices only -- never public.
# Enable HTTPS once in the Tailscale admin console so `tailscale serve` can get a cert.
tailscale_auth_key = cfg.require_secret("tailscaleAuthKey")
tailnet_dns_name = cfg.require("tailnetDnsName") # e.g. tailc6fb4e.ts.net
webui_hostname = "open-webui"
webui_tailnet_url = f"https://{webui_hostname}.{tailnet_dns_name}"

# Render provider auth (from ESC: pulumiConfig.renderApiKey / pulumiConfig.renderOwnerId).
render_provider = render.Provider(
 "render",
 api_key=cfg.require_secret("renderApiKey"),
 owner_id=cfg.require("renderOwnerId"),
)

# ---------------------------------------------------------------------------
# Secrets generated in state (the in-IaC replacement for `openssl rand -base64 32`).
# The shared bearer authenticates Open WebUI -> Hermes; it lands on both services.
# ---------------------------------------------------------------------------
shared_key = random.RandomBytes("hermes-shared-key", length=32)
webui_secret_key = random.RandomBytes("webui-secret-key", length=32)

# ---------------------------------------------------------------------------
# Build and push both images during `pulumi up` (no docker CLI; the provider
# embeds buildx). Render then pulls them by their digest-pinned ref.
# ---------------------------------------------------------------------------
registry_auth = [{"address": "ghcr.io", "username": ghcr_user, "password": ghcr_token}]

hermes_image = docker_build.Image(
 "hermes-image",
 context={"location": "./hermes"},
 dockerfile={"location": "./hermes/Dockerfile"},
 platforms=["linux/amd64"],
 tags=[pulumi.Output.concat("ghcr.io/", ghcr_user, "/hermes-agent:latest")],
 push=True,
 registries=registry_auth,
)

webui_image = docker_build.Image(
 "webui-image",
 context={"location": "./openwebui"},
 dockerfile={"location": "./openwebui/Dockerfile"},
 platforms=["linux/amd64"],
 tags=[pulumi.Output.concat("ghcr.io/", ghcr_user, "/open-webui:latest")],
 push=True,
 registries=registry_auth,
)

# Credential Render uses to pull the private GHCR images.
ghcr_credential = render.RegistryCredential(
 "ghcr",
 name="ghcr",
 registry="GITHUB",
 username=ghcr_user,
 auth_token=ghcr_token,
 opts=pulumi.ResourceOptions(provider=render_provider),
)

# ---------------------------------------------------------------------------
# Hermes: a PRIVATE Render service. No public URL; reachable only over Render's
# private network. A disk holds its memory and workspace. Env values are
# encrypted at rest, which is how secrets ride on Render (no separate flag).
# ---------------------------------------------------------------------------
hermes = render.PrivateService(
 "hermes-agent",
 name="hermes-agent",
 plan="standard", # 2 GB; the gateway OOMs on starter's 512 MB
 region=region,
 runtime_source={
 "image": {
 "image_url": pulumi.Output.concat("ghcr.io/", ghcr_user, "/hermes-agent"),
 "digest": hermes_image.digest,
 "registry_credential_id": ghcr_credential.id,
 },
 },
 disk={"name": "hermes-data", "mount_path": "/opt/data", "size_gb": 5},
 env_vars={
 "PORT": {"value": "8642"},
 "API_SERVER_ENABLED": {"value": "true"},
 "API_SERVER_HOST": {"value": "0.0.0.0"},
 "API_SERVER_PORT": {"value": "8642"},
 "API_SERVER_MODEL_NAME": {"value": model_default},
 # Auto-approve the agent's shell/terminal hooks (no TTY in a container).
 "HERMES_ACCEPT_HOOKS": {"value": "1"},
 # Secrets (encrypted at rest on Render):
 "API_SERVER_KEY": {"value": shared_key.base64},
 "OPENROUTER_API_KEY": {"value": openrouter_api_key},
 # Modal is a runtime credential: Hermes creates sandboxes at runtime with this pair.
 "MODAL_TOKEN_ID": {"value": modal_token_id},
 "MODAL_TOKEN_SECRET": {"value": modal_token_secret},
 },
 opts=pulumi.ResourceOptions(provider=render_provider),
)

# Open WebUI reaches Hermes at its Render-assigned internal hostname (the read-only slug).
hermes_api_base_url = pulumi.Output.concat("http://", hermes.slug, ":8642/v1")

# ---------------------------------------------------------------------------
# Open WebUI: also a PRIVATE Render service, but the container joins your tailnet
# (tailscaled in userspace + `tailscale serve`), so you reach it at
# https://open-webui.<tailnet>.ts.net from your devices. Never public.
# ---------------------------------------------------------------------------
webui = render.PrivateService(
 "open-webui",
 name="open-webui",
 plan="standard", # 2 GB; Open WebUI OOMs on starter's 512 MB
 region=region, # must match Hermes for private networking
 runtime_source={
 "image": {
 "image_url": pulumi.Output.concat("ghcr.io/", ghcr_user, "/open-webui"),
 "digest": webui_image.digest,
 "registry_credential_id": ghcr_credential.id,
 },
 },
 disk={"name": "openwebui-data", "mount_path": "/app/backend/data", "size_gb": 5},
 env_vars={
 "PORT": {"value": "8080"}, # tailscale serve points at this; Open WebUI binds it
 "OPENAI_API_BASE_URL": {"value": hermes_api_base_url},
 "ENABLE_OPENAI_API": {"value": "true"},
 "ENABLE_OLLAMA_API": {"value": "false"},
 "DEFAULT_MODELS": {"value": model_default},
 "WEBUI_AUTH": {"value": "true"},
 "WEBUI_URL": {"value": webui_tailnet_url},
 # Reached only over the tailnet, so the first sign-up is the admin.
 "ENABLE_LOGIN_FORM": {"value": "true"},
 "ENABLE_SIGNUP": {"value": "true"},
 "DEFAULT_USER_ROLE": {"value": "admin"},
 # Always apply env config; otherwise Open WebUI persists the first run's config.
 "ENABLE_PERSISTENT_CONFIG": {"value": "false"},
 # Don't load local embedding/RAG models (a big memory saver; the agent is the brain).
 "RAG_EMBEDDING_ENGINE": {"value": "openai"},
 "BYPASS_EMBEDDING_AND_RETRIEVAL": {"value": "true"},
 "OFFLINE_MODE": {"value": "true"},
 "HF_HUB_OFFLINE": {"value": "1"},
 # Secrets (encrypted at rest):
 "OPENAI_API_KEY": {"value": shared_key.base64},
 "WEBUI_SECRET_KEY": {"value": webui_secret_key.base64},
 # Tailscale joins this container to your tailnet and serves Open WebUI on it.
 "TS_AUTHKEY": {"value": tailscale_auth_key},
 "TS_HOSTNAME": {"value": webui_hostname},
 },
 opts=pulumi.ResourceOptions(depends_on=[hermes], provider=render_provider),
)

pulumi.export("hermesInternalUrl", hermes_api_base_url)
# Open this from any device on your tailnet:
pulumi.export("openWebUiTailnetUrl", webui_tailnet_url)

name: hermes-agent-pulumi
runtime: yaml
description: Deploy a Hermes agent (Render + Modal + Tailscale) as one declarative Pulumi program.

# The Render SDK is bridged from the official Terraform provider. Run once:
# pulumi package add terraform-provider render-oss/render 1.8.0
# Then `pulumi install` regenerates it on a fresh checkout.
packages:
 render:
 source: terraform-provider
 version: 1.1.4
 parameters:
 - render-oss/render
 - 1.8.0

config:
 renderRegion:
 type: string
 default: oregon
 modelDefault:
 type: string
 default: gpt-5.5
 ghcrUser:
 type: string
 ghcrToken:
 type: string
 secret: true
 modalTokenId:
 type: string
 secret: true
 modalTokenSecret:
 type: string
 secret: true
 openrouterApiKey:
 type: string
 secret: true
 tailscaleAuthKey:
 type: string
 secret: true
 tailnetDnsName:
 type: string
 renderApiKey:
 type: string
 secret: true
 renderOwnerId:
 type: string

variables:
 registryAuth:
 - address: ghcr.io
 username: ${ghcrUser}
 password: ${ghcrToken}
 # Open WebUI reaches Hermes at its private slug; you reach Open WebUI on the tailnet.
 hermesApiBaseUrl: http://${hermes-agent.slug}:8642/v1
 webuiTailnetUrl: https://open-webui.${tailnetDnsName}

resources:
 # Render provider auth (from config or ESC).
 render:
 type: pulumi:providers:render
 properties:
 apiKey: ${renderApiKey}
 ownerId: ${renderOwnerId}

 # Secrets generated in state: the replacement for `openssl rand -base64 32`.
 hermes-shared-key:
 type: random:RandomBytes
 properties:
 length: 32
 webui-secret-key:
 type: random:RandomBytes
 properties:
 length: 32

 # Build + push both images during `pulumi up` (no docker CLI).
 hermes-image:
 type: docker-build:Image
 properties:
 context:
 location: ./hermes
 dockerfile:
 location: ./hermes/Dockerfile
 platforms: [linux/amd64]
 tags:
 - ghcr.io/${ghcrUser}/hermes-agent:latest
 push: true
 registries: ${registryAuth}
 webui-image:
 type: docker-build:Image
 properties:
 context:
 location: ./openwebui
 dockerfile:
 location: ./openwebui/Dockerfile
 platforms: [linux/amd64]
 tags:
 - ghcr.io/${ghcrUser}/open-webui:latest
 push: true
 registries: ${registryAuth}

 # Credential Render uses to pull the private GHCR images.
 ghcr:
 type: render:RegistryCredential
 properties:
 name: ghcr
 registry: GITHUB
 username: ${ghcrUser}
 authToken: ${ghcrToken}
 options:
 provider: ${render}

 # Hermes: a PRIVATE Render service. No public URL; encrypted env vars are the secret mechanism.
 hermes-agent:
 type: render:PrivateService
 properties:
 name: hermes-agent
 plan: standard  # 2 GB; the gateway OOMs on starter's 512 MB
 region: ${renderRegion}
 runtimeSource:
 image:
 imageUrl: ghcr.io/${ghcrUser}/hermes-agent
 digest: ${hermes-image.digest}
 registryCredentialId: ${ghcr.id}
 disk:
 name: hermes-data
 mountPath: /opt/data
 sizeGb: 5
 envVars:
 PORT: { value: "8642" }
 API_SERVER_ENABLED: { value: "true" }
 API_SERVER_HOST: { value: "0.0.0.0" }
 API_SERVER_PORT: { value: "8642" }
 API_SERVER_MODEL_NAME: { value: "${modelDefault}" }
 HERMES_ACCEPT_HOOKS: { value: "1" }
 API_SERVER_KEY: { value: "${hermes-shared-key.base64}" }
 OPENROUTER_API_KEY: { value: "${openrouterApiKey}" }
 MODAL_TOKEN_ID: { value: "${modalTokenId}" }
 MODAL_TOKEN_SECRET: { value: "${modalTokenSecret}" }
 options:
 provider: ${render}

 # Open WebUI: also a PRIVATE Render service, joined to your tailnet by its image.
 open-webui:
 type: render:PrivateService
 properties:
 name: open-webui
 plan: standard  # 2 GB; Open WebUI OOMs on starter's 512 MB
 region: ${renderRegion}  # must match Hermes for private networking
 runtimeSource:
 image:
 imageUrl: ghcr.io/${ghcrUser}/open-webui
 digest: ${webui-image.digest}
 registryCredentialId: ${ghcr.id}
 disk:
 name: openwebui-data
 mountPath: /app/backend/data
 sizeGb: 5
 envVars:
 PORT: { value: "8080" }
 OPENAI_API_BASE_URL: { value: "${hermesApiBaseUrl}" }
 ENABLE_OPENAI_API: { value: "true" }
 ENABLE_OLLAMA_API: { value: "false" }
 DEFAULT_MODELS: { value: "${modelDefault}" }
 WEBUI_AUTH: { value: "true" }
 WEBUI_URL: { value: "${webuiTailnetUrl}" }
 ENABLE_LOGIN_FORM: { value: "true" }
 ENABLE_SIGNUP: { value: "true" }
 DEFAULT_USER_ROLE: { value: "admin" }
 ENABLE_PERSISTENT_CONFIG: { value: "false" }
 RAG_EMBEDDING_ENGINE: { value: "openai" }
 BYPASS_EMBEDDING_AND_RETRIEVAL: { value: "true" }
 OFFLINE_MODE: { value: "true" }
 HF_HUB_OFFLINE: { value: "1" }
 OPENAI_API_KEY: { value: "${hermes-shared-key.base64}" }
 WEBUI_SECRET_KEY: { value: "${webui-secret-key.base64}" }
 TS_AUTHKEY: { value: "${tailscaleAuthKey}" }
 TS_HOSTNAME: { value: "open-webui" }
 options:
 provider: ${render}
 dependsOn:
 - ${hermes-agent}

outputs:
 hermesInternalUrl: ${hermesApiBaseUrl}
 # Open this from any device on your tailnet:
 openWebUiTailnetUrl: ${webuiTailnetUrl}

Neither box is published to the internet. The same code, plus the Dockerfiles and the Tailscale entrypoint, is in the companion repo:

github.com/dirien/hermes-agent-pulumi

Cost

Two details shape the bill. The host starts on Render’s free Hobby workspace, but a private service with a persistent disk runs on paid compute, so you need a card on file the moment you add them. There is no free-tier version of this architecture. Both services also need the 2 GB Standard plan, since the gateway and Open WebUI each OOM on the 512 MB Starter.

	Hermes gateway	Open WebUI
Plan	Standard (2 GB)	Standard (2 GB)
Compute	~$25/mo	~$25/mo
Disk	5 GB	5 GB
Public URL	none	none (tailnet only)

That lands the host around $50/month, in line with what a comparable managed setup runs. Modal and OpenRouter bill by usage on top, so the total tracks how hard the agent works rather than a flat number. Tailscale is free for a personal tailnet, so the private gateway adds nothing to that.

The persistent disk pins each service to a single instance, so there is no autoscaling, and a redeploy drops a few seconds of connectivity while the old instance lets go of the disk. For a single-user agent that is a fine trade, and it is why “stand it up in a second region” is a copy-the-config move and not a live failover.

The OpenRouter Activity dashboard: 56 requests and 738K tokens of gpt-5.5 usage, all routed through one key.

Running the deployment

With your ESC environment referenced in Pulumi.dev.yaml, deploy with:

pulumi up

This generates the Render SDK’s resources, builds and pushes both images through buildx, and creates the two private services with their disks and encrypted env vars, all in one run, with no docker CLI and no secret import hiding in a script. When it finishes, both services show up in the Render dashboard with no public URL.

The hermes-agent and open-webui private services in the Render dashboard, both deployed with no public URL.

Access the agent

Give it a few minutes after pulumi up finishes. The images still have to be pulled and the containers booted, and Open WebUI has to join your tailnet before the URL resolves. Refresh until it loads.

Open WebUI is reachable only from a device on your tailnet, at https://open-webui.<your-tailnet>.ts.net, with no domain to buy, no DNS to point, and no certificate to manage. Open it, sign up so the first account becomes the admin, and ask the agent to run a Python command. It should fire up a Modal sandbox and return the result, with nothing exposed to the public internet at any point.

Open WebUI on the tailnet URL: the gpt-5.5 agent runs sum(i*i for i in range(1, 1001)) in a Modal sandbox and returns 333833500.

When the agent runs that code, Modal builds a throwaway sandbox specifically for it. There is no Modal resource to declare, no Modal CLI to call, and no Modal provider to bridge. The Hermes service carries MODAL_TOKEN_ID and MODAL_TOKEN_SECRET as two encrypted env vars, and that is the whole footprint.

The Modal dashboard: the per-task sandboxes the Hermes agent spun up at runtime to execute code.

Verify the deployment (optional)

If the chat UI does not load, give the services a few minutes to finish their first boot, then check two things:

Render dashboard logs for the hermes-agent and open-webui services, to confirm both started and pulled their images.
Tailscale admin console (Machines), to confirm the open-webui node joined your tailnet. If it did not, the TS_AUTHKEY is likely expired or not reusable.

Keep your AI agent private

Deploy your Hermes agent across Render, Modal, and Tailscale as one Pulumi program, with secrets held in Pulumi ESC and nothing exposed to the public internet.

Get started

Security considerations

Self-hosted AI servers get found fast: scanners like Shodan and Censys enumerate a freshly exposed one within hours to days, and the code-capable ones are already under active attack. Running an always-on agent on a machine you also use for everything else invites prompt injection on top of that. The answer here is to keep the agent unroutable from the internet and to isolate the code it runs. That defeats the scanner: nobody finds an open port to attack. It does less against the agent being turned against you from the inside, which is the harder problem.

One detail makes that isolation essential. In a headless container there is no one at the terminal to approve anything, and the same is true of its cron and CI runs, so the deployment runs Hermes with HERMES_ACCEPT_HOOKS=1 to auto-accept the shell hooks it would otherwise pause on. The practical result is that the agent’s code runs with no human in the loop, which is exactly why the Modal sandbox has to carry the weight: when the model writes a bad command, the throwaway container it runs in is what stands between that command and the gateway’s credentials, not a confirmation prompt.

Concern	Typical self-host	This deployment
Gateway reachability	Public port, or behind a login	Private service, no public URL
Chat UI reachability	Public hostname + auth	Tailnet only (`tailscale serve`)
Code execution	Runs next to the gateway	Isolated Modal sandbox, torn down after
Command approval	Human approves dangerous commands	Headless: isolation replaces the prompt
Secrets in transit	Exposed if accessed over HTTP	Encrypted env vars; tailnet-encrypted access

This is private, which is not the same as fully trusted. A few places the boundary still leaks:

The shared bearer never rotates. Hermes is unroutable from the internet, but it is wide open to Open WebUI, and the two share one static bearer key. Whoever owns the front end owns the gateway behind it. Rotate it periodically.
The tailnet policy is click-ops. The ACLs, the node’s tags, and the one-time HTTPS toggle live in the Tailscale admin console, not in pulumi up, and the reusable key has no tag scoping its blast radius. The Tailscale provider can manage that policy declaratively if you want it in code.
Signup defaults to admin. Fine on a single-user tailnet, the first thing to lock down (ENABLE_SIGNUP, DEFAULT_USER_ROLE) once more than one person can reach the URL.
Every messaging client is another entry point. Each platform you wire up adds another way in, which is why the tailnet is the baseline for securing this, not the full extent of it.
A compromised agent can still reach out. The private core stops external discovery, but it does not contain the agent itself. The gateway holds the Modal and OpenRouter tokens as env vars and can call out freely, so a prompt-injected agent could read its own environment and send them somewhere; the Modal sandbox isolates the code the agent runs, not the credentials it holds. Cap and rotate the OpenRouter key, keep the Modal token off the gateway, and allowlist outbound traffic, which on Render means a forward-proxy sidecar (Render gives you dedicated outbound IPs and inbound rules, but no destination egress filter).

My recommendations:

Keep both services private; never give Hermes a public URL
Run code in Modal sandboxes, since HERMES_ACCEPT_HOOKS=1 means no human approves commands
Rotate the shared bearer key and the Tailscale auth key periodically
Cap and rotate the OpenRouter key; a leaked credential is the likelier failure mode than an open port
Use Pulumi ESC for secrets instead of hardcoding
Lock down Open WebUI signup once more than one person can reach it
Manage Tailscale ACLs and tags deliberately, in the admin console or with the Tailscale provider

What’s next?

The deployment is the foundation the agent builds on. A private agent becomes useful when it remembers earlier context, runs code without a human at the terminal, and initiates contact on its own. The disk is what makes this persist: its memory and its schedule live on that disk, so they survive every restart.

It schedules itself. Hermes ships a first-class cron you drive in plain language, either by asking the agent in the chat window or through its CLI. The schedule takes intervals, five-field cron, or one-shots:

hermes cron create "every 1h" \
 "Scrape <listing-url> for the price, compare it with seen.json in the sandbox, message me on Telegram only if it dropped, and otherwise reply exactly [SILENT]." \
 --deliver telegram

The [SILENT] reply suppresses delivery when nothing changed, so the watcher only speaks up when there is something to say. Scheduled runs skip memory by default, so polling does not pollute what the agent has learned.

It connects where you work, with the same secret pattern. Messaging clients attach to the same gateway, and their tokens ride as encrypted env vars on the Hermes service, the exact shape you used for the Modal pair:

// alongside MODAL_TOKEN_ID on the hermes service:
TELEGRAM_BOT_TOKEN: { value: cfg.requireSecret("telegramBotToken") },

plus a platform block you add to config.yaml (the image ships only the model and terminal blocks; Hermes supports the messaging platforms on top, per its docs):

gateway:
 platforms:
 telegram:
 require_mention: true

With those two pieces, a few things are genuinely doable today:

A price or uptime watcher that only speaks up when something changed, using the [SILENT] and persistent-sandbox dedup pattern above.
A daily briefing that ranks and dedups your feeds and delivers the top items each morning, the documented daily-briefing bot. If you want it to adapt to your taste over time, that is the optional Honcho memory provider’s job; it reads your preferences out of the conversation itself. The built-in file memory has no version of that.
A PR-review teammate that reviews open pull requests against your repo’s AGENTS.md conventions and saves recurring patterns as a skill, so the fifth review needs less hand-holding than the first.

A few limits are worth stating plainly, since demos tend to overstate the capabilities. Hermes does not ship Gmail, Calendar, or market integrations out of the box; those are wired up as MCP servers or skills, which is more work than a launch video implies. It curates its own memory and improves its own skills, but it does not retrain the underlying model or autonomously change its high-level strategy, so “self-learning” here means whatever it writes into memory and skills each run.

Conclusion

Deploying a Hermes agent with infrastructure as code means you can reproduce the setup anytime, version control it, and tear it down with a single pulumi destroy. Keeping both services private, with no public URL and no exposed ports, means the code-executing core never sits on the internet for someone to find.

The part worth carrying to the next service you deploy: not every layer has a provider, and that is fine. The task is to tell a real resource apart from a runtime dependency wearing a costume. Render and the images are resources you declare; Modal and Tailscale are credentials you hand over. Draw that map first, and the program follows from it.

To generate that map instead of writing it by hand, Pulumi Neo can take a target architecture like this one and produce a first draft of the program. The agent side of this shift is covered in How Building AI Agents Has Changed in 2026 and the platform side in Agent Sprawl Is Here. Your IaC Platform Is the Answer..

If you run into issues or have questions, drop by the Pulumi Community Slack or GitHub Discussions. New to Pulumi? Get started here.

How We Eliminated Long-Lived CI Secrets Across 70+ Repos

Boris Schlosser — Tue, 31 Mar 2026 00:00:00 +0000

Supply chain attacks on CI/CD pipelines are accelerating. A growing pattern involves attackers compromising popular GitHub Actions through tag poisoning — rewriting trusted version tags to point to malicious code that harvests environment variables, cloud credentials, and API tokens from runner environments. The stolen credentials are then exfiltrated to attacker-controlled infrastructure, often before anyone notices.

For every engineering organization, the question is no longer if your CI pipeline will encounter a compromised dependency, but what is exposed when it does.

At Pulumi, we asked ourselves that question and decided the answer should be “nothing useful.” Here’s how we got there.

The problem with static CI secrets

Most organizations store long-lived cloud credentials, API tokens, and service account keys as GitHub repository or organization secrets. But this approach has several well-known problems:

Broad availability. Every workflow run on a repository can access every secret stored in that repo. A compromised action in any workflow can read them all.
No expiration. Secrets persist until someone manually rotates them. If exfiltrated, they give attackers persistent access for weeks or months.
No granular audit trail. GitHub tells you a secret was used, but not which workflow, which step, or what it was used for.
Secret sprawl. Across dozens or hundreds of repos, the same credentials are often duplicated, making rotation a coordinated, error-prone effort.

In a supply chain attack scenario, this is exactly what attackers count on: a single compromised action that can dump a trove of long-lived credentials.

Our approach: zero static secrets

We replaced every static GitHub Secret across our CI pipelines with short-lived, dynamically fetched credentials using Pulumi ESC and OpenID Connect (OIDC). The credential flow works in layers, each scoped and ephemeral:

GitHub generates a short-lived OIDC token scoped to the specific workflow run, repository, and branch. This token is cryptographically signed by GitHub’s OIDC provider.
The token is exchanged with Pulumi Cloud for a short-lived Pulumi access token. Pulumi Cloud validates the OIDC claims (organization, repository, branch) against a configured trust policy before issuing the token.
The Pulumi access token opens an ESC environment to retrieve the credentials the workflow needs — cloud provider keys, API tokens, or other secrets.
Cloud credentials themselves are dynamic. ESC environments use OIDC login providers to fetch short-lived credentials directly from AWS, Azure, or GCP. No static keys or cloud credentials are stored anywhere.

The pulumi/esc-action GitHub Action handles this entire flow in a single workflow step.

sequenceDiagram
participant Runner as GitHub Actions Runner
participant GH as GitHub OIDC Provider
participant PC as Pulumi Cloud
participant ESC as Pulumi ESC
participant Cloud as Cloud Provider (AWS/Azure/GCP)
Runner->>GH: Request OIDC token (scoped to workflow run)
GH-->>Runner: Short-lived JWT
Runner->>PC: Exchange JWT for Pulumi access token
PC-->>Runner: Short-lived access token
Runner->>ESC: Open environment with access token
ESC->>Cloud: OIDC login (assume role / federated identity)
Cloud-->>ESC: Short-lived cloud credentials
ESC-->>Runner: Cloud credentials + secrets
Note over Runner,Cloud: Nothing is stored. Everything expires.

What the change looks like

Before this migration, our workflows referenced static secrets stored in GitHub:

env:
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

After the migration, an ESC environment handles credential fetching via OIDC. Here is what the environment definition looks like:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-role
 sessionName: esc-${context.pulumi.user.login}
 # Optional: scope down the session beyond what the role allows
 policyArns:
 - arn:aws:iam::123456789012:policy/ci-build-minimal
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}

The roleArn and optional policyArns make least-privilege straightforward: each login provider assumes a specific role, and policyArns can scope the session down further. You can use multiple login providers in one environment or separate environments per workflow to match permissions to each job’s needs.

The workflow itself becomes minimal — a single step that authenticates via OIDC and injects the credentials:

permissions:
 contents: read
 id-token: write  # Required for OIDC

steps:
 - name: Fetch secrets from ESC
 uses: pulumi/esc-action@v3
 with:
 environment: '<your-organization>/<your-esc-env>'

The static secrets.* references are gone entirely. Every credential is fetched at runtime through ESC.

Scale: 70+ repos, zero static secrets

We didn’t do this for one or two flagship repos; we rolled it out across every Pulumi provider repository: AWS, Azure, GCP, Kubernetes, and over 60 more. The migration was managed centrally through our ci-mgmt tooling, which generates consistent workflow configurations across all provider repos.

The pattern is the same everywhere:

Each repo has a corresponding ESC environment under a github-secrets/ project.
All workflow-level ${{ secrets.* }} references have been removed.
The pulumi/esc-action step with OIDC auth is the single entry point for all credentials.

When every repo follows the same pattern like this, security posture is much more easily verifiable and auditable.

Auditability and centralized control

Beyond eliminating static secrets, this migration gave us centralized visibility and control that GitHub Secrets cannot provide:

Audit logging. ESC records which credentials were accessed, when, and by which workflow. This is a meaningful improvement over GitHub’s binary “secret was used” signal.
Centralized access policies. Access rules are defined once in ESC rather than scattered across individual repository settings pages.
Single-point rotation. Because ESC environments can import other environments, shared credentials live in a common base that all 70+ repo environments are composed of. Update it once, and every repo picks up the change on its next run.
Dynamic credentials by default. For cloud providers like AWS, Azure, and GCP, ESC fetches credentials via OIDC at open time. There is nothing to rotate because nothing is stored.

What happens if a GitHub Action is compromised

With this architecture in place, here is what an attacker gets if a compromised GitHub Action runs in our CI:

No GitHub Secrets to dump. The repository settings page has no stored secrets for a malicious action to exfiltrate.
OIDC tokens are scoped and short-lived. The GitHub-issued JWT is valid only for the specific workflow run and expires within minutes.
Cloud credentials are ephemeral. Any AWS, Azure, or GCP credentials fetched through ESC are short-lived and scoped to the role assumed during that run.
No persistent access. There are no long-lived tokens to reuse hours or days later.

Compare this to the traditional model, where a single compromised action could exfiltrate AWS access keys that remain valid until someone manually rotates them — which could be weeks or months.

The goal is not to prevent every possible attack. It is to make the blast radius as small as possible when something goes wrong.

Get started

If you want to adopt the same pattern in your own CI pipelines:

Tutorial: Using ESC with GitHub Actions — Step-by-step setup guide.
Announcing the Pulumi ESC GitHub Action — Full feature overview and capabilities.
Configuring OIDC for ESC — Set up OIDC trust between ESC and your cloud providers.
Pulumi ESC documentation — Full documentation for environments, secrets, and configuration.

Your CI secrets do not have to be a liability. With OIDC and Pulumi ESC, they do not have to exist at all.

Pulumi IAM Expands: Manage Access at Scale with Tags, Roles, and Teams

Devon Grove — Thu, 19 Mar 2026 10:30:00 -0700

Since the launch of Pulumi IAM with custom roles and scoped access tokens, organizations have been using fine-grained permissions to secure their automation and CI/CD pipelines. As teams scale to hundreds or thousands of stacks, environments, and accounts, the next challenge is applying those permissions efficiently.

Today, we’re introducing three new capabilities to help you manage permissions more dynamically at scale: tag-based access control, team role assignments, and user role assignments.

Why tag-based access control?

With custom roles, you can define granular permissions using fine-grained scopes. However, applying those roles still requires selecting individual stacks, environments, or accounts one by one. For organizations managing a large number of Pulumi entities, this means either granting overly broad access or spending significant time on manual configuration. Tag-based access control solves this problem.

What’s new?

Tag-based access control

You can now create rules within a custom role that dynamically grant permissions based on entity tags. This works across IaC stacks, ESC environments, and Insights accounts. For example, when a new stack is created and tagged env:prod, anyone with a role containing a matching tag-based rule automatically gets the right permissions. No manual assignment required.

A single role can include multiple tag-based rules, and they are evaluated with OR logic. If an entity matches any of the rules, the permissions are granted. Within a single rule, you can combine multiple key-value conditions with implicit AND logic for precise targeting. For example, a rule with conditions env:prod and team:payments ensures access is granted only to production resources owned by the payments team.

Team role assignments

Custom roles can now be assigned directly to teams within your Pulumi organization. When an engineer joins a team, whether manually or via SCIM provisioning, they automatically inherit the permissions defined in the team’s assigned roles.

Teams support both inline permissions (ad-hoc access to specific stacks, environments, or accounts) and role-based permissions simultaneously. You can assign multiple roles to a single team, giving you full flexibility to compose access from reusable building blocks while retaining the ability to grant one-off access where needed. If you have existing workflows built around ad-hoc assignments to teams, those continue to work exactly as before. You can adopt roles incrementally or mix both approaches on the same team.

Team admins (or users with the team:update scope) can continue to manage their team’s inline permissions as they do today. However, assigning organization-level custom roles to a team requires additional permissions: role:read and role:update.

User role assignments

Custom roles can also be assigned directly to individual organization members. This is useful for users whose responsibilities span multiple teams or require permissions beyond the existing org-level Admin, Member, and Billing Manager roles.

How permissions work together

Permissions in Pulumi IAM are additive. A user receives the union of all permissions granted to them, including permissions from roles assigned directly to them as a user and permissions from roles assigned to any team they belong to. A user on both the “SRE” and “Security” teams inherits permissions from both team roles, plus any role assigned to them individually.

How to get started

Configuring tag-based access control and role assignments is done through the Pulumi Cloud console and REST API.

1. Create a custom role with tag-based rules

In Pulumi Cloud, navigate to Settings > Access Management > Roles and create a new custom role. In the role configuration, add tag-based rules that define which entities the role should apply to.

For example, to create a role that grants write access to all production stacks:

Click Create custom role and give it a descriptive name (e.g., “Production Deployer”)
Add a permission set (e.g., Stack Write) to the role
Under entity selection, choose Tag-based rule
Set the condition: tag key env equals prod
Save the role

2. Assign the role to a team

Go to Settings > Access Management > Teams, select a team, and assign your custom role. All team members immediately inherit the defined permissions.

3. Assign a role to an individual user

For users with unique access requirements, go to Settings > Access Management > Members, select a user, and assign a custom role directly.

Enforce tagging standards with Pulumi Policy

Tag-based access control relies on consistent tagging. If a stack is missing a tag or has an incorrect value, permissions won’t be applied as expected. Pulumi Policy closes this gap by letting you enforce tagging standards as a preventative policy group, so any pulumi up on a stack with missing or invalid tags is blocked before deployment. This ensures your tag-based RBAC rules always grant the correct permissions. Policy enforces the standard, RBAC enforces the access.

To learn how to write policies that validate stack tags, see Using stack tags in policies.

Pulumi Policy currently supports tag enforcement for IaC stacks. For ESC environments and Insights accounts, tags are managed through the Pulumi Cloud console or REST API.

Availability

Tag-based access control, team role assignments, and user role assignments are available today for customers on the Pulumi Enterprise and Pulumi Business Critical plans. Check out our pricing page for more details on editions and what’s included.

Conclusion

With custom roles providing fine-grained permissions, tag-based rules enabling dynamic access policies, and the ability to assign roles directly to teams and users, Pulumi IAM now provides everything you need to implement automated, least-privilege access control at scale. We’re excited to see how you leverage these new capabilities to secure and streamline your cloud operations.

Explore the IAM documentation to get started, and share your feedback in our GitHub repository.

Learn more

Pulumi Cloud Now Supports Google Sign-In

Pablo Seibelt — Tue, 10 Mar 2026 00:00:00 +0000

Many developers and platform engineers already use Google accounts daily for email, cloud console access, and collaboration. Until now, signing in to Pulumi Cloud required a GitHub, GitLab, or Atlassian account, or an email/password combination. Today, we’re adding Google as a first-class identity provider, so you can sign in to Pulumi Cloud with the same Google account you already use for everything else.

Adding Google as an identity provider brings several benefits:

Use the account you already have. If your team already lives in Google Workspace, you can sign in to Pulumi Cloud with a single click, no new credentials required.
Inherit your existing security policies. If you’ve already configured two-factor authentication, device management, and other protections in a Google Workspace, you can carry them over to Pulumi Cloud automatically.

How it works

Signing up or signing in

On the Pulumi Cloud sign-in page, you’ll see a new Sign in with Google button alongside the existing GitHub, GitLab, and Atlassian options. If you are a new user, select it, authenticate with your Google account, and you’re in.

If you already have an existing Pulumi Cloud account, make sure to associate to your existing account as described in the next section.

Connecting Google to an existing account

If you already have a Pulumi Cloud account, you can link your Google identity from your account settings:

Navigate to your Account Settings.
Scroll to the Identity providers section.
Under Available identities, select Connect Google.

Once connected, you can use Google to sign in to your existing Pulumi Cloud account.

Google sign-in lets you authenticate with Pulumi Cloud using your individual Google account. It does not enable Google as a single sign-on (SSO) identity provider for your Pulumi Cloud organization.

If your team uses Google Workspace and needs centralized membership governance for Pulumi Cloud, configure SAML SSO with Google Workspace instead. SAML SSO is available on Pulumi Enterprise and Business Critical editions.

Get started

Google sign-in is available now for all new and existing Pulumi Cloud users:

New users: Sign up with Google on the Pulumi Cloud sign-up page.
Existing users: Connect your Google account in your account settings.

For more details, see the Pulumi Cloud accounts documentation.

We’d love to hear your feedback. Join the conversation in the Pulumi Community Slack or open an issue on GitHub.

Passwordless PostgreSQL: IAM Authentication with Pulumi

Elisabeth Lichtie — Fri, 13 Feb 2026 00:00:00 +0000

Managing database credentials is one of the persistent challenges in cloud infrastructure. Passwords need to be rotated, secrets need to be stored securely, and access needs to be carefully controlled. AWS IAM authentication for RDS offers a better way: instead of managing long-lived passwords, your applications authenticate using short-lived tokens generated from IAM credentials. This approach is more secure, eliminates password rotation overhead, and integrates seamlessly with your existing IAM policies. With Pulumi, you can set up this entire system using reusable components that make IAM authentication a standard part of your infrastructure.

Why IAM authentication for PostgreSQL?

Traditional database authentication relies on usernames and passwords. These credentials need to be stored somewhere secure, rotated regularly, and distributed to applications that need them. Each of these steps introduces complexity and potential security risks.

IAM authentication changes this model fundamentally. Instead of passwords, your applications generate authentication tokens on demand using their IAM credentials. These tokens are valid for only 15 minutes, eliminating the need for password rotation. Access control happens through IAM policies, which you’re already using to manage other AWS resources. The result is a more secure system that’s easier to maintain and audit.

The benefits become even more pronounced when you componentize this setup with Pulumi. Rather than repeating the same configuration steps for each database, you can build reusable components that handle IAM authentication setup automatically. This turns a complex, multi-step process into a simple, repeatable pattern that your entire team can use. In this post, we’ll cover an example that sets up the complete flow: an RDS cluster with IAM authentication, the necessary IAM roles and policies, and a Kubernetes application that connects using IAM tokens.

Architecture overview

This example sets up a complete environment with IAM-authenticated database access from a Kubernetes application. The architecture includes:

VPC with public and private subnets across multiple availability zones
EKS cluster for running containerized applications
Aurora PostgreSQL cluster with IAM authentication enabled
IAM roles and policies that connect Kubernetes service accounts to database access
A demo application that authenticates to the database using IAM tokens

The key integration point is IAM Roles for Service Accounts (IRSA), which allows Kubernetes pods to assume IAM roles. This means your application doesn’t need any AWS credentials stored in environment variables or mounted secrets. The pod’s service account automatically has the permissions it needs to generate database authentication tokens.

Setting up the RDS cluster with IAM authentication

The foundation of this setup is an RDS cluster configured to accept IAM authentication. With Pulumi’s componentized approach, you can encapsulate all the RDS configuration in a reusable component:

const rdsCluster = new RdsCluster("iam-postgres", {
 vpcId: vpc.vpc.id,
 vpcCidrBlock: vpc.vpc.cidrBlock,
 subnetIds: vpc.publicSubnets.map((s) => s.id),
 databaseName: DATABASE_NAME,
 masterUsername: MASTER_USERNAME,
 masterPassword: dbMasterPassword,
 instanceClass: "db.t4g.medium",
 engineVersion: "17.4",
 iamDatabaseUser: IAM_DB_USERNAME,
});

Inside the component, the critical configuration is iamDatabaseAuthenticationEnabled:

this.cluster = new aws.rds.Cluster(
 `${name}-cluster`,
 {
 engine: "aurora-postgresql",
 engineVersion: engineVersion,
 databaseName: args.databaseName,
 masterUsername: args.masterUsername,
 masterPassword: args.masterPassword,
 dbSubnetGroupName: subnetGroup.name,
 vpcSecurityGroupIds: [this.securityGroup.id],
 iamDatabaseAuthenticationEnabled: true,
 skipFinalSnapshot: true,
 tags: { Name: `${name}-cluster` },
 },
 { parent: this }
);

This single flag tells RDS to accept authentication tokens in addition to traditional passwords. The cluster still needs a master password for initial setup and administrative tasks, but your applications can use IAM authentication instead.

Creating the database user with IAM permissions

Enabling IAM authentication on the cluster isn’t enough. You also need to create a database user and grant it the special rds_iam role that allows IAM authentication. This is where Pulumi’s PostgreSQL provider comes in:

const dbSetup = new DbSetup(
 "iam-postgres",
 {
 dbEndpoint: rdsCluster.cluster.endpoint,
 dbName: DATABASE_NAME,
 masterUsername: MASTER_USERNAME,
 masterPassword: dbMasterPassword,
 iamUsername: IAM_DB_USERNAME,
 },
 { dependsOn: [rdsCluster.instance] }
);

The DbSetup component handles all the database-level configuration. It creates the IAM user and grants the necessary permissions:

// Create IAM-enabled database user
this.iamRole = new postgresql.Role(
 `${name}-iam-user`,
 {
 name: args.iamUsername,
 login: true,
 },
 { parent: this, provider: this.provider }
);

// Grant rds_iam role to enable IAM authentication
new postgresql.GrantRole(
 `${name}-grant-rds-iam`,
 {
 role: this.iamRole.name,
 grantRole: "rds_iam",
 },
 { parent: this, provider: this.provider }
);

The component also grants the necessary database privileges (CONNECT, USAGE, CREATE, and table operations). By encapsulating this in a component, you ensure that every IAM-authenticated database user is set up consistently with the correct permissions.

Configuring IAM policies for database access

The next piece is the IAM role and policy that allows applications to generate authentication tokens. This involves two parts: the role that the application assumes, and the policy that grants database access.

For Kubernetes applications, this uses IRSA (IAM Roles for Service Accounts). The RdsCluster component includes a method that creates the appropriate IAM role:

const rdsIamRole = rdsCluster.createIamRole(
 eksCluster.oidcProvider,
 NAMESPACE,
 SERVICE_ACCOUNT_NAME
);

This creates an IAM role with a trust policy that allows the specified Kubernetes service account to assume it:

assumeRolePolicy: pulumi
 .all([oidcProvider.arn, oidcProvider.url])
 .apply(([arn, url]) =>
 JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: { Federated: arn },
 Action: "sts:AssumeRoleWithWebIdentity",
 Condition: {
 StringEquals: {
 [`${url}:sub`]: `system:serviceaccount:${namespace}:${serviceAccountName}`,
 },
 },
 },
 ],
 })
 ),

The role is then granted the rds-db:connect permission for the specific database user:

{
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Action: ["rds-db:connect"],
 Resource: `arn:aws:rds-db:${region}:${accountId}:dbuser:${resourceId}/${iamDatabaseUser}`,
 },
 ],
}

This policy is scoped to exactly one database user on one cluster. This level of granularity is one of the security advantages of IAM authentication: you can control database access with the same precision you use for other AWS resources.

Connecting from Kubernetes with IRSA

The application running in Kubernetes needs a service account annotated with the IAM role ARN:

this.serviceAccount = new k8s.core.v1.ServiceAccount(
 `${name}-sa`,
 {
 metadata: {
 name: args.serviceAccountName,
 namespace: args.namespace,
 annotations: {
 "eks.amazonaws.com/role-arn": args.iamRoleArn,
 },
 },
 },
 { parent: this, provider: args.provider }
);

When a pod uses this service account, EKS automatically configures the pod with temporary AWS credentials for the associated IAM role. The application can then use these credentials to generate database authentication tokens.

How IAM authentication works at runtime

The authentication flow happens at connection time:

%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#FF9900','primaryTextColor':'#232F3E','primaryBorderColor':'#232F3E','lineColor':'#666','secondaryColor':'#527FFF','tertiaryColor':'#fff'}}}%%
sequenceDiagram
autonumber
participant App as Application Pod
participant SDK as AWS SDK
participant STS as AWS STS
participant IAM as AWS IAM
participant RDS as RDS Aurora
rect rgb(255, 249, 230)
Note over App,SDK: Phase 1: Token Generation
activate App
App->>+SDK: generate_db_auth_token()
SDK->>+STS: Get temporary credentials (IRSA)
STS-->>-SDK: Return AWS credentials
SDK->>SDK: Sign auth request
SDK-->>-App: Return IAM token (15 min TTL)
deactivate App
end
rect rgb(235, 245, 255)
Note over App,RDS: Phase 2: Database Connection
activate App
App->>+RDS: Connect (user + IAM token)
activate RDS
RDS->>+IAM: Validate token & permissions
IAM-->>-RDS: Token valid + rds-db:connect allowed
RDS->>RDS: Verify rds_iam role
RDS-->>-App: Connection established
deactivate App
end
Note over App,RDS: 💡 Token expires after 15 minutes - generate new token per connection

The application uses the AWS SDK to call generate_db_auth_token(), passing the database endpoint, port, and username
The SDK uses the pod’s IAM credentials (provided automatically by IRSA) to sign the request and generate a token
The application connects to PostgreSQL using the IAM username and the token as the password
RDS validates the token against IAM, verifying that the caller has rds-db:connect permission for that database user
If the token is valid, the connection is established

Here’s what this looks like in Python:

def get_iam_token():
 """Generate an IAM authentication token for RDS"""
 client = boto3.client('rds', region_name=AWS_REGION)
 token = client.generate_db_auth_token(
 DBHostname=DB_ENDPOINT,
 Port=DB_PORT,
 DBUsername=DB_USER,
 Region=AWS_REGION
 )
 return token

def get_db_connection():
 """Create a database connection with IAM auth"""
 token = get_iam_token()
 return psycopg2.connect(
 host=DB_ENDPOINT,
 port=DB_PORT,
 database=DB_NAME,
 user=DB_USER,
 password=token,
 sslmode='require',
 sslrootcert=RDS_CA_CERT
 )

The token is valid for 15 minutes. Applications should generate a new token for each connection or implement token caching with refresh logic.

Testing the setup with the demo application

The example includes an interactive demo application that lets you test the IAM authentication setup. Once deployed, the application provides a web interface for creating tables, adding messages, and querying the database:

Deploy the infrastructure and access the demo app:

pulumi up
export APP_URL=$(pulumi stack output appUrl)
echo "Demo app: http://$APP_URL"

The application shows the database endpoint and IAM username it’s using, and provides buttons to create tables and add data. Behind the scenes, every database operation authenticates using IAM tokens, demonstrating that the entire authentication flow is working correctly.

The full code for this example is available at https://github.com/pulumi-demos/examples/tree/main/typescript/aws-iam-for-postgres. The example includes all the component code referenced in this post.

Make passwordless access your default

Wrap this IAM authentication setup in a Pulumi component and apply the same passwordless pattern to every RDS cluster your team runs.

Get started

Production considerations

This example demonstrates IAM authentication in a working environment, but you’ll want to make several adjustments for production use:

Network security: The example places RDS in public subnets to allow the PostgreSQL provider to connect during deployment. In production, place RDS in private subnets and ensure your deployment environment (like GitHub Actions or Pulumi Cloud) can reach the database for initial setup, either through a bastion host or VPN.

Connection pooling and performance: IAM tokens expire after 15 minutes. If you’re using connection pooling, you’ll need logic to refresh tokens before they expire. AWS recommends using IAM authentication only when your application creates fewer than 200 new connections per second. For higher connection rates, consider using Amazon RDS Proxy, which manages connection pooling and can reduce the overhead of IAM token generation.

Master password management: You still need a master password for database administration and for the initial user setup. Store this in AWS Secrets Manager or Pulumi ESC, and restrict access to it carefully.

Monitoring and auditing: Token generation via the AWS SDK is logged in CloudTrail. Note that CloudWatch and CloudTrail do not log database authentication attempts themselves, so you’ll need to rely on PostgreSQL’s native logging for connection monitoring.

Multi-region considerations: If your application runs in multiple regions, ensure that IAM policies and database access work correctly across regions. Token generation and validation must happen in the same region as the RDS cluster.

Cost: IAM authentication itself is free, but be aware that cross-AZ data transfer costs still apply for database connections.

Conclusion

IAM authentication for PostgreSQL transforms database credential management from a security burden into a seamless part of your infrastructure. By eliminating long-lived passwords, you reduce your attack surface and remove the operational overhead of password rotation. By integrating with IAM, you gain fine-grained access control and comprehensive audit logs.

Pulumi makes this setup practical and repeatable through componentization. The components in this example encapsulate the complexity of IAM authentication setup, making it easy to apply the same pattern across multiple databases and applications. This turns security best practices into your team’s default approach.

Whether you’re building a new application or improving the security posture of existing infrastructure, IAM authentication for RDS deserves consideration. The investment in setup pays dividends in security, maintainability, and operational simplicity.

Deploy OpenClaw on AWS or Hetzner Securely with Pulumi and Tailscale

Engin Diri — Mon, 26 Jan 2026 00:00:00 +0000

Update (January 2026): The lobster has molted into its final form! From Clawdbot to Moltbot to OpenClaw. With 100k+ GitHub stars and 2M visitors in a week, the project finally has a name that’ll stick. The CLI command is now openclaw and the new handle is @openclaw. Same mission: AI that actually does things. Your assistant. Your machine. Your rules. See the official getting started guide for updated installation instructions.

Update (April 2026): Refreshed for OpenClaw 2026.4.27. Upstream now recommends Node 24, but the cloud-init script in this post still installs Node 22 — both work. If you’d like Node 24, change the nvm install 22 lines to nvm install 24.

The short version: Deploy OpenClaw to AWS or Hetzner with a Pulumi TypeScript program that provisions the VM, installs Docker, Node, and OpenClaw, then joins the instance to your Tailscale network so the gateway and browser ports stay private. One pulumi up to deploy, one pulumi destroy to tear down. Total cost: about $33/month on AWS or $7/month on Hetzner.

OpenClaw is everywhere right now. The open-source AI assistant gained 9,000 GitHub stars in a single day, received public praise from former Tesla AI head Andrej Karpathy, and has sparked a global run on Mac Minis as developers scramble to give this “lobster assistant” a home. Users are calling it “Jarvis living in a hard drive” and “Claude with hands”—the personal AI assistant that Siri promised but never delivered.

The Mac Mini craze is real: people are buying dedicated hardware just to run OpenClaw, with some enthusiasts purchasing 40 Mac Minis at once. Even Logan Kilpatrick from Google DeepMind couldn’t resist ordering one. But here’s the thing: you don’t actually need a Mac Mini. OpenClaw runs anywhere: on a VPS, in the cloud, or on that old laptop gathering dust.

With all this hype, I had to try it myself. But instead of clicking through the AWS console or running manual commands on a VPS, I wanted to do it right from the start: infrastructure as code with Pulumi. Why? Because when I inevitably want to tear it down, spin up a new instance, or deploy to a different region, I don’t want to remember which buttons I clicked three weeks ago. I want a single pulumi up command.

Dan got the assignment right:

In this post, I’ll show you how to deploy OpenClaw to AWS or Hetzner Cloud (if you want European data residency or just want to spend less). We’ll use Pulumi to define the infrastructure and Tailscale to keep your AI assistant off the public internet.

What is OpenClaw?

OpenClaw is an open-source AI assistant created by Peter Steinberger that runs on your own infrastructure. It connects to WhatsApp, Slack, Discord, Google Chat, Signal, and iMessage. It can control browsers, generate videos and images, clone your voice for voice notes, and run scheduled tasks via cron. There’s a skills system for extending functionality, and you can run it on pretty much anything: Mac Mini, Raspberry Pi, VPS, laptop, or gaming PC.

The difference from cloud-hosted AI? OpenClaw runs on your server, not Anthropic’s. It’s available 24/7 across all your devices, can schedule automated tasks, and keeps your entire conversation history locally. Check the official OpenClaw documentation for the full feature list.

Prerequisites

Before getting started, ensure you have:

Pulumi CLI installed and configured
A Pulumi Cloud account
AWS account (for AWS deployment)
Hetzner Cloud account (for European deployment)
Anthropic API key
Node.js 22+ installed (Node 24 recommended; the cloud-init script in this post installs Node 22 inside the VM)
Tailscale account with HTTPS enabled (one-time setup in admin console)

This guide uses Anthropic’s API, but OpenClaw works with other providers too. Check the providers documentation if you’d rather use OpenAI, Google Gemini, or a local model via Ollama.

Understanding OpenClaw architecture

OpenClaw uses a gateway-centric architecture where a single daemon acts as the control plane for all messaging, tool execution, and client connections:

Component	Port	Description
Gateway	18789	WebSocket server handling channels, nodes, sessions, and hooks
Browser control	18791	Headless Chrome instance for web automation
Docker sandbox	-	Isolated container environment for running tools safely

The Gateway connects to messaging platforms (WhatsApp, Slack, Discord, etc.), the CLI, the web UI, and mobile apps. The Browser component lets OpenClaw open web pages, fill forms, scrape data, and download files. Docker sandboxing runs bash commands in isolated containers so your bot can execute code without risking your host system.

Setting up ESC for secrets management

Deploying OpenClaw means handling sensitive credentials: API keys, auth tokens, cloud provider secrets. You don’t want these hardcoded or scattered across environment variables. Pulumi ESC (Environments, Secrets, and Configuration) stores them securely and passes them directly to your Pulumi program — the same pattern Pulumi uses to eliminate long-lived CI secrets across 70+ repos.

Create a new ESC environment:

pulumi env init <your-org>/openclaw-secrets

Add your secrets to the environment:

values:
 anthropicApiKey:
 fn::secret: "sk-ant-xxxxx"
 tailscaleAuthKey:
 fn::secret: "tskey-auth-xxxxx"
 tailnetDnsName: "tailxxxxx.ts.net"
 hcloudToken:
 fn::secret: "your-hetzner-api-token"
 pulumiConfig:
 anthropicApiKey: ${anthropicApiKey}
 tailscaleAuthKey: ${tailscaleAuthKey}
 tailnetDnsName: ${tailnetDnsName}
 hcloud:token: ${hcloudToken}

Then create a Pulumi.dev.yaml file in your project to reference the environment:

environment:
 - <your-org>/openclaw-secrets

This approach keeps your secrets out of your codebase and passes them directly to OpenClaw during automated onboarding. To prevent stack-level overrides of these values, see locking down ESC values with fn::final.

Securing with Tailscale

By default, deploying OpenClaw exposes SSH (port 22), the gateway (port 18789), and browser control (port 18791) to the public internet. This is convenient for testing but not ideal for production use.

Tailscale creates a secure mesh VPN that lets you access your OpenClaw instance without exposing unnecessary ports publicly. When you provide a Tailscale auth key, the Pulumi program:

Removes gateway and browser ports from public access
Keeps SSH as fallback for debugging if Tailscale setup fails
Installs Tailscale on the instance during provisioning (after other dependencies)
Enables Tailscale SSH so you can SSH via Tailscale without managing keys
Joins your tailnet automatically using the auth key

The Pulumi program installs Docker, Node.js, and OpenClaw first, then configures Tailscale last. This ensures that even if the Tailscale auth key is invalid or expired, you can still SSH in via the public IP to troubleshoot.

To generate a Tailscale auth key:

Go to Tailscale Admin Console
Click “Generate auth key”
Enable “Reusable” if you plan to redeploy
Copy the key and add it to your ESC environment

Deploying to AWS

Let’s walk through the complete AWS deployment. Create a new Pulumi project:

mkdir openclaw-aws && cd openclaw-aws
pulumi new typescript

Install the required dependencies:

npm install @pulumi/aws @pulumi/tls

Do not use t3.micro instances for OpenClaw. The 1 GB memory is insufficient for installation. Use t3.medium (4 GB) or t3.large (8 GB) instead.

The Pulumi program

Running OpenClaw on AWS means setting up a VPC, subnets, security groups, an EC2 instance, SSH keys, and a cloud-init script that installs everything. That’s a lot of clicking in the AWS console. The Pulumi program below defines all of it in code.

Replace the contents of index.ts with the following:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as tls from "@pulumi/tls";

const config = new pulumi.Config();

const instanceType = config.get("instanceType") ?? "t3.medium";
const anthropicApiKey = config.requireSecret("anthropicApiKey");
const model = config.get("model") ?? "anthropic/claude-sonnet-4";
const enableSandbox = config.getBoolean("enableSandbox") ?? true;
const gatewayPort = config.getNumber("gatewayPort") ?? 18789;
const browserPort = config.getNumber("browserPort") ?? 18791;

const tailscaleAuthKey = config.requireSecret("tailscaleAuthKey");
const tailnetDnsName = config.require("tailnetDnsName");

// Generate a random token for gateway authentication
const gatewayToken = new tls.PrivateKey("openclaw-gateway-token", {
 algorithm: "ED25519",
}).publicKeyOpenssh.apply(key => {
 // Create a deterministic token from the public key (take first 48 hex chars)
 const hash = require("crypto").createHash("sha256").update(key).digest("hex");
 return hash.substring(0, 48);
});

const sshKey = new tls.PrivateKey("openclaw-ssh-key", {
 algorithm: "ED25519",
});

const vpc = new aws.ec2.Vpc("openclaw-vpc", {
 cidrBlock: "10.0.0.0/16",
 enableDnsHostnames: true,
 enableDnsSupport: true,
 tags: { Name: "openclaw-vpc" },
});

const gateway = new aws.ec2.InternetGateway("openclaw-igw", {
 vpcId: vpc.id,
 tags: { Name: "openclaw-igw" },
});

const subnet = new aws.ec2.Subnet("openclaw-subnet", {
 vpcId: vpc.id,
 cidrBlock: "10.0.1.0/24",
 mapPublicIpOnLaunch: true,
 tags: { Name: "openclaw-subnet" },
});

const routeTable = new aws.ec2.RouteTable("openclaw-rt", {
 vpcId: vpc.id,
 routes: [
 {
 cidrBlock: "0.0.0.0/0",
 gatewayId: gateway.id,
 },
 ],
 tags: { Name: "openclaw-rt" },
});

new aws.ec2.RouteTableAssociation("openclaw-rta", {
 subnetId: subnet.id,
 routeTableId: routeTable.id,
});

const securityGroup = new aws.ec2.SecurityGroup("openclaw-sg", {
 vpcId: vpc.id,
 description: "Security group for OpenClaw instance",
 ingress: [
 {
 description: "SSH access (fallback)",
 fromPort: 22,
 toPort: 22,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
 egress: [
 {
 fromPort: 0,
 toPort: 0,
 protocol: "-1",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
 tags: { Name: "openclaw-sg" },
});

const keyPair = new aws.ec2.KeyPair("openclaw-keypair", {
 publicKey: sshKey.publicKeyOpenssh,
});

const ami = aws.ec2.getAmiOutput({
 owners: ["099720109477"],
 mostRecent: true,
 filters: [
 { name: "name", values: ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"] },
 { name: "virtualization-type", values: ["hvm"] },
 ],
});

const userData = pulumi
 .all([tailscaleAuthKey, anthropicApiKey, gatewayToken])
 .apply(([tsAuthKey, apiKey, gwToken]) => {
 return `#!/bin/bash
set -e

export DEBIAN_FRONTEND=noninteractive

# System updates
apt-get update
apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker
usermod -aG docker ubuntu

# Install NVM and Node.js for ubuntu user
sudo -u ubuntu bash << 'UBUNTU_SCRIPT'
set -e
cd ~

# Install NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

# Load NVM
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

# Install Node.js 22
nvm install 22
nvm use 22
nvm alias default 22

# Install OpenClaw
npm install -g openclaw@latest

# Add NVM to bashrc if not already there
if ! grep -q 'NVM_DIR' ~/.bashrc; then
 echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc
 echo '[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"' >> ~/.bashrc
fi
UBUNTU_SCRIPT

# Set environment variables for ubuntu user
echo 'export ANTHROPIC_API_KEY="${apiKey}"' >> /home/ubuntu/.bashrc

# Install and configure Tailscale
echo "Installing Tailscale..."
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --authkey="${tsAuthKey}" --ssh || echo "WARNING: Tailscale setup failed. Run 'sudo tailscale up' manually."

# Enable systemd linger for ubuntu user (required for user services to run at boot)
loginctl enable-linger ubuntu

# Start user's systemd instance (required for user services during cloud-init)
systemctl start user@1000.service

# Run OpenClaw onboarding as ubuntu user (skip daemon install, do it separately)
echo "Running OpenClaw onboarding..."
sudo -H -u ubuntu ANTHROPIC_API_KEY="${apiKey}" GATEWAY_PORT="${gatewayPort}" bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw onboard --non-interactive --accept-risk \
 --mode local \
 --auth-choice apiKey \
 --gateway-port $GATEWAY_PORT \
 --gateway-bind loopback \
 --skip-daemon \
 --skip-skills || echo "WARNING: OpenClaw onboarding failed. Run openclaw onboard manually."
'

# Install daemon service with XDG_RUNTIME_DIR set
echo "Installing OpenClaw daemon..."
sudo -H -u ubuntu XDG_RUNTIME_DIR=/run/user/1000 bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw daemon install || echo "WARNING: Daemon install failed. Run openclaw daemon install manually."
'

# Configure gateway for Tailscale Serve (trustedProxies + skip device pairing + set token)
echo "Configuring gateway for Tailscale Serve..."
sudo -H -u ubuntu GATEWAY_TOKEN="${gwToken}" python3 << 'PYTHON_SCRIPT'
import json
import os
config_path = "/home/ubuntu/.openclaw/openclaw.json"
with open(config_path) as f:
 config = json.load(f)
config["gateway"]["trustedProxies"] = ["127.0.0.1"]
config["gateway"]["controlUi"] = {
 "enabled": True,
 "allowInsecureAuth": True
}
config["gateway"]["auth"] = {
 "mode": "token",
 "token": os.environ["GATEWAY_TOKEN"]
}
with open(config_path, "w") as f:
 json.dump(config, f, indent=2)
print("Configured gateway with trustedProxies, controlUi, and token")
PYTHON_SCRIPT

# Enable Tailscale HTTPS proxy (requires HTTPS to be enabled in Tailscale admin console)
echo "Enabling Tailscale HTTPS proxy..."
tailscale serve --bg ${gatewayPort} || echo "WARNING: tailscale serve failed. Enable HTTPS in your Tailscale admin console first."

echo "OpenClaw setup complete!"
`;
 });

const instance = new aws.ec2.Instance("openclaw-instance", {
 ami: ami.id,
 instanceType: instanceType,
 subnetId: subnet.id,
 vpcSecurityGroupIds: [securityGroup.id],
 keyName: keyPair.keyName,
 userData: userData,
 userDataReplaceOnChange: true,
 rootBlockDevice: {
 volumeSize: 30,
 volumeType: "gp3",
 },
 tags: { Name: "openclaw" },
});

export const publicIp = instance.publicIp;
export const publicDns = instance.publicDns;
export const privateKey = sshKey.privateKeyOpenssh;

// Construct the Tailscale MagicDNS hostname from the private IP
// AWS private IPs like 10.0.1.15 become hostnames like ip-10-0-1-15
const tailscaleHostname = instance.privateIp.apply(ip =>
 `ip-${ip.replace(/\./g, "-")}`
);

export const tailscaleUrlWithToken = pulumi.interpolate`https://${tailscaleHostname}.${tailnetDnsName}/?token=${gatewayToken}`;
export const gatewayTokenOutput = gatewayToken;

Deploying to Hetzner

Hetzner Cloud is a solid choice if you need European data residency or want to spend less money. Spoiler: it’s a lot less money.

Hetzner has similar concepts to AWS but different names. EC2 instances become Servers. Security groups become Firewalls. Same idea, different provider. The resource types come from @pulumi/hcloud.

Create a new project for Hetzner:

mkdir openclaw-hetzner && cd openclaw-hetzner
pulumi new typescript

Install the Hetzner provider:

npm install @pulumi/hcloud @pulumi/tls

The default server type cax21 is an ARM-based (Ampere) instance with 4 vCPUs and 8 GB RAM. ARM instances cost less for the same compute. If you need x86 architecture, use ccx13 or similar CCX series instead.

The Hetzner Pulumi program

Replace index.ts with the following:

import * as pulumi from "@pulumi/pulumi";
import * as hcloud from "@pulumi/hcloud";
import * as tls from "@pulumi/tls";

const config = new pulumi.Config();

const serverType = config.get("serverType") ?? "cax21";
const location = config.get("location") ?? "fsn1";
const anthropicApiKey = config.requireSecret("anthropicApiKey");
const model = config.get("model") ?? "anthropic/claude-sonnet-4";
const enableSandbox = config.getBoolean("enableSandbox") ?? true;
const gatewayPort = config.getNumber("gatewayPort") ?? 18789;
const browserPort = config.getNumber("browserPort") ?? 18791;

const tailscaleAuthKey = config.requireSecret("tailscaleAuthKey");
const tailnetDnsName = config.require("tailnetDnsName");

// Generate a random token for gateway authentication
const gatewayToken = new tls.PrivateKey("openclaw-gateway-token", {
 algorithm: "ED25519",
}).publicKeyOpenssh.apply(key => {
 const hash = require("crypto").createHash("sha256").update(key).digest("hex");
 return hash.substring(0, 48);
});

const sshKey = new tls.PrivateKey("openclaw-ssh-key", {
 algorithm: "ED25519",
});

const hcloudSshKey = new hcloud.SshKey("openclaw-sshkey", {
 publicKey: sshKey.publicKeyOpenssh,
});

const firewallRules: hcloud.types.input.FirewallRule[] = [
 {
 direction: "out",
 protocol: "tcp",
 port: "any",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound TCP",
 },
 {
 direction: "out",
 protocol: "udp",
 port: "any",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound UDP",
 },
 {
 direction: "out",
 protocol: "icmp",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound ICMP",
 },
 {
 direction: "in",
 protocol: "tcp",
 port: "22",
 sourceIps: ["0.0.0.0/0", "::/0"],
 description: "SSH access (fallback)",
 },
];

const firewall = new hcloud.Firewall("openclaw-firewall", {
 rules: firewallRules,
});

const userData = pulumi
 .all([tailscaleAuthKey, anthropicApiKey, gatewayToken])
 .apply(([tsAuthKey, apiKey, gwToken]) => {
 return `#!/bin/bash
set -e

export DEBIAN_FRONTEND=noninteractive

# System updates
apt-get update
apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker

# Create ubuntu user (Hetzner uses root by default)
useradd -m -s /bin/bash -G docker ubuntu || true

# Install NVM and Node.js for ubuntu user
sudo -u ubuntu bash << 'UBUNTU_SCRIPT'
set -e
cd ~

# Install NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

# Load NVM
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

# Install Node.js 22
nvm install 22
nvm use 22
nvm alias default 22

# Install OpenClaw
npm install -g openclaw@latest

# Add NVM to bashrc if not already there
if ! grep -q 'NVM_DIR' ~/.bashrc; then
 echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc
 echo '[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"' >> ~/.bashrc
fi
UBUNTU_SCRIPT

# Set environment variables for ubuntu user
echo 'export ANTHROPIC_API_KEY="${apiKey}"' >> /home/ubuntu/.bashrc

# Install and configure Tailscale
echo "Installing Tailscale..."
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --authkey="${tsAuthKey}" --ssh || echo "WARNING: Tailscale setup failed. Run 'sudo tailscale up' manually."

# Enable systemd linger for ubuntu user (required for user services to run at boot)
loginctl enable-linger ubuntu

# Start user's systemd instance (required for user services during cloud-init)
systemctl start user@1000.service

# Run OpenClaw onboarding as ubuntu user (skip daemon install, do it separately)
echo "Running OpenClaw onboarding..."
sudo -H -u ubuntu ANTHROPIC_API_KEY="${apiKey}" GATEWAY_PORT="${gatewayPort}" bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw onboard --non-interactive --accept-risk \
 --mode local \
 --auth-choice apiKey \
 --gateway-port $GATEWAY_PORT \
 --gateway-bind loopback \
 --skip-daemon \
 --skip-skills || echo "WARNING: OpenClaw onboarding failed. Run openclaw onboard manually."
'

# Install daemon service with XDG_RUNTIME_DIR set
echo "Installing OpenClaw daemon..."
sudo -H -u ubuntu XDG_RUNTIME_DIR=/run/user/1000 bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw daemon install || echo "WARNING: Daemon install failed. Run openclaw daemon install manually."
'

# Configure gateway for Tailscale Serve (trustedProxies + skip device pairing + set token)
echo "Configuring gateway for Tailscale Serve..."
sudo -H -u ubuntu GATEWAY_TOKEN="${gwToken}" python3 << 'PYTHON_SCRIPT'
import json
import os
config_path = "/home/ubuntu/.openclaw/openclaw.json"
with open(config_path) as f:
 config = json.load(f)
config["gateway"]["trustedProxies"] = ["127.0.0.1"]
config["gateway"]["controlUi"] = {
 "enabled": True,
 "allowInsecureAuth": True
}
config["gateway"]["auth"] = {
 "mode": "token",
 "token": os.environ["GATEWAY_TOKEN"]
}
with open(config_path, "w") as f:
 json.dump(config, f, indent=2)
print("Configured gateway with trustedProxies, controlUi, and token")
PYTHON_SCRIPT

# Enable Tailscale HTTPS proxy (requires HTTPS to be enabled in Tailscale admin console)
echo "Enabling Tailscale HTTPS proxy..."
tailscale serve --bg ${gatewayPort} || echo "WARNING: tailscale serve failed. Enable HTTPS in your Tailscale admin console first."

echo "OpenClaw setup complete!"
`;
 });

const server = new hcloud.Server("openclaw-server", {
 serverType: serverType,
 location: location,
 image: "ubuntu-24.04",
 sshKeys: [hcloudSshKey.id],
 firewallIds: [firewall.id.apply(id => Number(id))],
 userData: userData,
 labels: {
 purpose: "openclaw",
 },
});

export const ipv4Address = server.ipv4Address;
export const privateKey = sshKey.privateKeyOpenssh;

// Construct the Tailscale MagicDNS hostname from the server name
// Hetzner servers use their name as the hostname
const tailscaleHostname = server.name;

export const tailscaleUrlWithToken = pulumi.interpolate`https://${tailscaleHostname}.${tailnetDnsName}/?token=${gatewayToken}`;
export const gatewayTokenOutput = gatewayToken;

You can find both programs in the Pulumi examples repo under openclaw/:

github.com/pulumi/examples

Cost comparison

Before deploying, let’s compare the costs between AWS and Hetzner for running OpenClaw 24/7:

	AWS (t3.medium)	Hetzner (cax21)
vCPUs	2	4
Memory	4 GB	8 GB
Storage	30 GB gp3 (+$2.40/mo)	80 GB NVMe (included)
Traffic	Pay per GB	20 TB included
Architecture	x86 (Intel/AMD)	ARM (Ampere)
Hourly price	$0.0416	€0.0104 (~$0.011)
Monthly price	~$33 (with storage)	€6.49 (~$7)
Annual cost	~$396	~$84

Hetzner gives you double the vCPUs, double the RAM, at less than a quarter of the price. The trade-off? ARM architecture instead of x86. But OpenClaw doesn’t care - it’s just Node.js and Docker.

Prices are for on-demand instances as of January 2026. AWS prices are for us-east-1; Hetzner prices exclude VAT. Both include standard networking and storage. Check AWS EC2 pricing and Hetzner Cloud pricing for current rates.

Running the deployment

With your ESC environment configured in Pulumi.dev.yaml, deploy with:

pulumi up

After deployment completes, you’ll see outputs similar to:

Outputs:
 gatewayTokenOutput : "786c099cc8f8bf20dbebf40b8b51b75cf5cdab25..."
 privateKey : [secret]
 publicDns : "ec2-x-x-x-x.compute-1.amazonaws.com"
 publicIp : "x.x.x.x"
 tailscaleUrlWithToken: "https://ip-10-0-1-x.tailxxxxx.ts.net/?token=786c099..."

The tailscaleUrlWithToken output provides the complete URL with authentication token. Copy and paste it into your browser to access the OpenClaw web UI.

Output names vary slightly between providers: AWS uses publicIp and publicDns, while Hetzner uses ipv4Address. The Tailscale hostname is derived from the instance’s private IP (AWS) or server name (Hetzner).

Automated onboarding

The Pulumi program runs OpenClaw’s non-interactive onboarding during instance provisioning. It uses your Anthropic API key from ESC, binds the gateway to loopback with Tailscale Serve as the HTTPS proxy, generates a secure gateway token (exported in Pulumi outputs), installs the daemon as a systemd user service, and configures trustedProxies and controlUi.allowInsecureAuth to skip device pairing when accessed via Tailscale.

The cloud-init script runs openclaw onboard --non-interactive with all necessary flags, then configures the gateway for secure Tailscale access. Your instance is ready as soon as provisioning finishes.

Access the web UI

Please wait 3-5 minutes after pulumi up completes. The cloud-init script runs after the instance launches and needs time to install Docker, Node.js, OpenClaw, and Tailscale, then run the onboarding process and start the daemon. Periodically refresh the page until it loads. If the page still doesn’t load after 5 minutes, see Verify the deployment to troubleshoot.

The easiest way to access the OpenClaw web UI is to use the tailscaleUrlWithToken output from Pulumi:

# Get the full URL with token
pulumi stack output tailscaleUrlWithToken

Copy and paste this URL into your browser. The URL includes both the Tailscale MagicDNS hostname and the authentication token, so you can access the web UI directly.

Finding your tailnet DNS name: Go to the Tailscale admin console and look under the DNS section for your tailnet name (e.g., tailxxxxx.ts.net). You can also find your machines and their MagicDNS hostnames in the Machines tab.

Token-based authentication provides an additional layer of security on top of Tailscale’s network-level authentication. Only devices on your tailnet can reach the URL, and the token prevents unauthorized access if someone gains access to your tailnet.

From the web UI, you can connect messaging channels (WhatsApp, Discord, Slack), configure skills and integrations, and manage settings.

Verify the deployment (optional)

If you encounter issues accessing the web UI, you can SSH into your instance to troubleshoot:

# Check your Tailscale admin console for the new machine
ssh ubuntu@<tailscale-ip>

# Check OpenClaw gateway status
systemctl --user status openclaw-gateway

Test your assistant

Open the gateway dashboard using the tailscaleUrlWithToken output and use the built-in chat to test your assistant:

Your personal AI assistant is now running 24/7 on your own infrastructure, accessible securely through Tailscale.

Redeploy OpenClaw with one command

With your infrastructure defined in Pulumi, you can move OpenClaw to a new region, rebuild it, or tear it down with pulumi destroy, without remembering which console buttons you clicked.

Get started

Security considerations

When self-hosting an AI assistant, security matters. OpenClaw’s rapid adoption meant thousands of instances spun up in days, and not everyone locked them down. The community noticed:

The tweet isn’t exaggerating. A quick Shodan search shows exposed gateways on port 18789 with shell access, browser automation, and API keys up for grabs:

Don’t let your instance be one of them.

Concern	Without Tailscale	With Tailscale
SSH access	Public (port 22 open)	Public fallback + Tailscale SSH
Gateway access	Public (port 18789 open)	Private (Tailscale only)
Browser control	Public (port 18791 open)	Private (Tailscale only)
API keys in transit	Exposed if gateway accessed over HTTP	Protected by Tailscale encryption
Attack surface	3 open ports	1 open port (SSH fallback)

SSH remains accessible as a fallback even with Tailscale enabled. This allows you to troubleshoot if Tailscale fails to connect. Once you’ve confirmed Tailscale is working, you can manually remove the SSH ingress rule from your security group for maximum security.

My recommendations:

Always use Tailscale for production
Rotate your auth keys periodically
Use Pulumi ESC for secrets instead of hardcoding
Enable Tailscale SSH to avoid managing keys manually
Monitor your Tailscale admin console for unauthorized devices
Remove the SSH fallback after confirming Tailscale works if you want zero public ports

What’s next?

Now that OpenClaw is running, you can install skills (voice generation, video creation, browser automation), set up scheduled tasks with cron, invite colleagues to your tailnet for shared access, or connect additional channels like WhatsApp and Discord.

For a deeper case study of running an AI assistant on Pulumi-managed infrastructure, see how we built Platybot, or read our KubeCon EU 2026 recap on the year AI moved into production on Kubernetes.

Conclusion

Deploying OpenClaw with infrastructure as code means you can reproduce your setup anytime, version control it, and tear it down with a single command. Adding Tailscale keeps it private - no exposed ports, no hoping you configured your firewall correctly at 2am.

If you run into issues or have questions, drop by the Pulumi Community Slack or GitHub Discussions.

New to Pulumi? Get started here.

Pulumi IAM Now Available for Self-Hosted Pulumi Cloud

Davide Massarenti — Wed, 14 Jan 2026 11:18:00 +0000

We’re excited to announce that Pulumi Identity and Access Management (IAM) is now available for self-hosted instances of Pulumi Cloud. This foundational security capability brings the same enterprise-grade access management we launched for Pulumi Cloud SaaS to organizations running Pulumi on their own infrastructure.

Enterprise Security for Self-Hosted Deployments

Self-hosted Pulumi Cloud customers can now leverage the full power of Custom Roles and Granular Access Tokens to implement Zero Trust security principles and least privilege access controls within their own environments. This means you can:

Define Custom Permissions with fine-grained scopes (e.g., stack:delete, environment:read) tailored to your organization’s security requirements.
Create Custom Roles by combining these permissions with specific Pulumi entities (Stacks, Environments, etc.).
Generate Scoped Organization Access Tokens that are precisely limited to the permissions defined in their associated roles, dramatically reducing the blast radius if credentials are compromised.

Secure Automation for On-Premises Infrastructure

This release is powerful for self-hosted Pulumi Cloud deployments where security and compliance requirements are often even more stringent. You can now:

Implement Least Privilege CI/CD: Scope pipeline tokens to only the actions and resources they absolutely need, ensuring your automation follows the same security standards as your infrastructure.
Enhance Compliance Posture: Demonstrate precise, auditable control over programmatic access to auditors and security teams.
Reduce Operational Risk: Limit the potential impact of compromised tokens by restricting them to specific roles and permissions.

Getting Started

Self-hosted customers can access IAM features through the same intuitive interface available in Pulumi Cloud SaaS. Navigate to Settings -> Access Management -> Roles to begin creating Custom Permissions and Custom Roles, then generate scoped Organization Access Tokens from Settings -> Access Management -> Access Tokens.

For detailed information about Pulumi IAM capabilities, including step-by-step guides and best practices, see our comprehensive announcement blog post.

Learn More

Explore the IAM & RBAC documentation to get started:

We’re committed to bringing enterprise-grade security features to all Pulumi deployments, whether in the cloud or on-premises. If you have questions or feedback, please reach out through your account representative or our GitHub repository.

Native OIDC Token Exchange for Pulumi CLI

Boris Schlosser — Tue, 16 Dec 2025 00:00:00 +0000

Managing credentials in CI/CD pipelines has always involved tradeoffs. Long-lived access tokens are convenient but create security risks when they leak or fall into the wrong hands. Short-lived credentials are more secure but require additional tooling to obtain and manage. Today, we’re eliminating this tradeoff with native OIDC token exchange support in the Pulumi CLI.

The Pulumi CLI now includes built-in support for exchanging OIDC tokens from your identity provider for short-lived Pulumi Cloud access tokens. This means you can authenticate to Pulumi Cloud directly from CI/CD environments like GitHub Actions, GitLab CI, or Kubernetes without storing any long-lived Pulumi credentials as secrets.

Why OIDC token exchange matters

Most CI/CD workflows authenticate to Pulumi Cloud using personal access tokens or organization tokens stored as secrets. While this approach works, it comes with significant security concerns:

Credential exposure: If a token is accidentally committed to a repository or logged in CI output, attackers gain long-term access to your infrastructure
Rotation complexity: Rotating tokens requires updating secrets across multiple CI/CD systems
Over-privileged access: Tokens often have broader permissions than needed for specific workflows
Audit trail gaps: Difficult to trace which workflow run used which credentials

With OIDC token exchange, you eliminate these risks by leveraging short-lived tokens that your CI/CD platform or identity provider already issues. No long-lived secrets to manage, rotate, or secure.

How it works

The pulumi login command now accepts OIDC tokens directly:

pulumi login --oidc-token <token> --oidc-org <org-name>

The CLI exchanges your OIDC token for a short-lived Pulumi Cloud access token, which is then used for all subsequent operations. Tokens expire after 2 hours by default, though you can customize this with the --oidc-expiration flag.

You can scope tokens to specific teams or users:

# Scope to a team
pulumi login --oidc-token <token> --oidc-org my-org --oidc-team platform-team

# Scope to a user
pulumi login --oidc-token <token> --oidc-org my-org --oidc-user alice

The --oidc-token flag accepts either a raw token string or a file path prefixed with file://, making it easy to integrate with various token delivery mechanisms.

Example: Kubernetes (EKS)

For workloads running in Kubernetes, you can use service account tokens and exchange them for Pulumi access tokens. The following example uses a Pulumi program to define a Kubernetes Job resource.

import * as kubernetes from "@pulumi/kubernetes";

const script = new kubernetes.core.v1.ConfigMap("script", {
 data: {
 "entrypoint.sh": `#!/bin/bash
EKS_ID_TOKEN=$(cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)
pulumi login --oidc-token $EKS_ID_TOKEN --oidc-org MY_ORG_NAME
pulumi whoami
`
 }
});

const job = new kubernetes.batch.v1.Job("runner", {
 metadata: {},
 spec: {
 template: {
 spec: {
 serviceAccountName: "pulumi-service-account",
 containers: [{
 name: "runner",
 image: "pulumi/pulumi:latest",
 command: ["/bin/entrypoint.sh"],
 volumeMounts: [
 {
 name: "script",
 mountPath: "/bin/entrypoint.sh",
 readOnly: true,
 subPath: "entrypoint.sh",
 },
 ],
 }],
 restartPolicy: "Never",
 volumes: [
 {
 name: "script",
 configMap: {
 defaultMode: 0o700,
 name: script.metadata.name,
 },
 },
 ],
 },
 },
 backoffLimit: 0,
 },
});

export const jobName = job.metadata.name;

This approach works with any Kubernetes cluster that supports service account token projection, including EKS, GKE, and AKS. The example uses EKS’s default token location at /var/run/secrets/eks.amazonaws.com/serviceaccount/token, but you can adapt the token path for other Kubernetes distributions.

Prerequisites

Before using OIDC token exchange with the Pulumi CLI, you need to:

Register your OIDC provider as a trusted issuer in your Pulumi organization settings
Configure authorization policies that specify which tokens can be exchanged and what permissions they receive
Ensure your CI/CD system or identity provider is configured to issue OIDC tokens with the appropriate audience claim

Get started

Native OIDC token exchange is available now in the latest version of the Pulumi CLI. To get started:

Update to the latest Pulumi CLI version
Configure your OIDC provider and authorization policies in Pulumi Cloud
Update your CI/CD workflows to use pulumi login --oidc-token

For complete documentation, including setup guides for specific identity providers, see:

We’re excited to see how this feature helps you build more secure infrastructure automation workflows. If you have questions or feedback, join us in the Pulumi Community Slack.

Future of the Cloud: 10 Trends Shaping 2026 and Beyond

Sara Huddleston — Thu, 04 Dec 2025 07:56:40 +0000

In 2026, several trends will dominate cloud computing, driving innovation, efficiency, and scalability. From Infrastructure as Code (IaC) to AI/ML, platform engineering to multi-cloud and hybrid strategies, and security practices, let’s explore the 10 biggest emerging trends.

On This Article

1. Cloud Will Become a Business Necessity by 2028
2. Hyperscalers Accelerate AI-Driven Cloud Expansion
3. Hybrid and Multi-Cloud to Drive Innovation
4. Enterprises Rebuild Their Cloud Foundations to Operationalize AI
5. IaC Drives Scalable Cloud, Multi-Cloud, and AI Operations
6. DevSecOps Evolves Into AI-Integrated Security
7. Platform Engineering and Internal Developer Platforms
8. AIOps Matures Into a Cloud Operations Standard
9. Kubernetes Dominance and Increased Complexity
10. AI Code Assistants in the Enterprise
The Future of Cloud: Reinvented for an AI-First Decade

1. Cloud Will Become a Business Necessity by 2028

According to Gartner, by 2028 the cloud will be the key driver for business innovation, and estimates that over 95% of new digital workloads will be deployed on cloud-native platforms.

The future of cloud computing. Credit: Gartner

According to McKinsey & Company’s “In search of cloud value” report:

Cloud value is driven by innovation, worth 5x more than cost savings.
EBITDA uplift of 20–30% by 2030 for high-performing organizations.
Asia leads in projected cloud value, followed by the US and Europe.
High-ROI organizations excel by aligning cloud strategy with business priorities, building strong cloud foundations, and using modern operating models.

Teams succeeding in this transition increasingly use Infrastructure as Code, automation, and unified governance frameworks like Pulumi Insights + Policies to operationalize this value.

2. Hyperscalers Accelerate AI-Driven Cloud Expansion

Hyperscalers are making the largest infrastructure investments in cloud history — nearly all centered on AI workloads, inference, and high-performance compute.

AWS has integrated Anthropic’s Claude 3 and Claude 4 models into Amazon Bedrock for enterprise LLM workflows. “Claude Opus 4 and Claude Sonnet 4 are available today in Amazon Bedrock, enabling customers to build agents with stronger reasoning, memory, and tool use.” — AWS, May 2025
Microsoft Azure revenue rose 33% year-over-year in Q3 (ended March 31), outperforming estimates of ~29.7%. AI contributed 16 percentage points to this growth, up from 13 points in the prior quarter. “Microsoft is on track to invest approximately $80 billion to build out AI-enabled datacenters to train AI models and deploy AI and cloud-based applications around the world,” said Brad Smith, the Microsoft Vice Chair and President.
Google Cloud is committing $25 billion over two years for data center and AI infrastructure expansion across the PJM grid, with total capital expenditure for 2025 ranging from $75–85 billion. “As our CEO has said, in these early days of a very transformative technology, the risks of under-investing are dramatically higher than the risks of over-investing,” said Eunice Huang, Head of AI and Emerging Tech Policy.
Oracle anticipates 15–20% cloud revenue growth in FY 2026–2027 attributable to AI infrastructure demand, tied to its partnership in the Stargate initiative.

As hyperscalers integrate AI deeper into their service layers, engineering teams must adapt with IaC-driven automation, reusable patterns, and policy controls to deploy cloud and AI infrastructure consistently. See how organizations deploy AWS infrastructure at the speed of AI with Pulumi and Pulumi Policies.

3. Hybrid and Multi-Cloud to Drive Innovation

Hybrid and multi-cloud strategies are now mainstream:

Hybrid cloud will grow from $130B to $310–330B by 2030 (ResearchAndMarkets).
87% of enterprises run workloads across multiple clouds (Mordor Intelligence).
Gartner predicts that 40% of enterprises will adopt hybrid compute architectures in mission-critical workflows by 2028 (up from 8%).

Credit: Cloud Worldwide Service, Forbes

As AI and regulatory requirements grow, organizations must deploy workloads across AWS, Azure, Google Cloud, on-prem, and edge — while maintaining consistent security, compliance, and configuration.

Modern cloud teams use:

Infrastructure as Code for consistent multi-cloud provisioning and environment standardization, forming the backbone of AI infrastructure orchestration
Reusable components and internal platforms to define scalable architecture patterns and accelerate delivery across Kubernetes, AI/ML pipelines, and hybrid environments
Policy-driven guardrails to maintain cost, security, and compliance across environments, supporting cloud governance automation and modern cloud cost governance

Pulumi enables all three through its multi-cloud IaC model, Pulumi Policies, and internal developer platform capabilities.

4. Enterprises Rebuild Their Cloud Foundations to Operationalize AI

While hyperscalers are transforming the global cloud platform, enterprises face a different challenge: adapting their own cloud foundations to support AI at scale. Organizations are moving beyond prototypes and integrating AI into core products, internal workflows, and customer-facing systems, requiring new levels of automation, governance, and AI infrastructure orchestration.

According to Gartner, global AI infrastructure spending is expected to surpass $2 trillion in 2026. IDC predicts that by 2027, more than 50% of enterprises will use AI agents to drive core workflows, which requires scalable, secure, and automated cloud architectures to support model execution and orchestration.

To enable this transition, enterprises are investing in:

GPU provisioning and orchestration, data pipelines, vector databases, feature stores, and LLM infrastructure needed for real-time AI workloads.
Data pipelines, vector databases, and feature stores needed for real-time AI workloads
Model-serving infrastructure, including gateways, inference routers, and autoscaling layers
Strong identity, secrets, and access controls as AI systems increase security exposure
Automation through Infrastructure as Code to ensure reproducibility and reduce drift
Policy-driven governance to secure cost, compliance, and architectural consistency

As AI becomes deeply embedded across engineering organizations, teams are increasingly using software engineering approaches such as Infrastructure as Code, reusable components, platform engineering, and policy automation to standardize how AI infrastructure is deployed, scaled, and secured across clouds.

To support this shift, Pulumi’s perspective on Superintelligence Infrastructure explains why AI workloads, from pre-training to inference at massive scale, require dynamic infrastructure orchestration rather than static configuration.

Pulumi users increasingly rely on:

Pulumi IaC for standardized AI infrastructure
Pulumi ESC to manage all secrets and configuration at scale
Pulumi Insights for visibility and misconfiguration analysis
Pulumi Policies for AI-specific guardrails in code, cost detection, and to provide automated compliance protections

5. IaC Drives Scalable Cloud, Multi-Cloud, and AI Operations

As cloud environments expand and AI workloads demand highly dynamic infrastructure, Infrastructure as Code (IaC) is becoming the foundation for scaling reliably across all environments. Organizations are increasingly adopting IaC in general-purpose languages to unify development and infrastructure workflows, reduce configuration drift, and deliver cloud resources at speed.

Modern Infrastructure as Code is advancing far beyond simple provisioning:

Standardizing multi-cloud and hybrid patterns so teams can deploy consistently across AWS, Azure, Google Cloud, on-prem, and edge environments.
Integrating seamlessly with cloud providers and third-party services, including data platforms and messaging systems like CockroachDB, Confluent Cloud, and Kafka.
Providing deeper validation and type-safety, ensuring parameters, dependencies, and security controls are correct before deployment.
Improving cloud resource efficiency and visibility with tools like Pulumi Insights Discovery.
Embedding security and compliance through Policy as Code, enforcing guardrails, cost controls, and regulatory requirements automatically, enabling truly policy-driven cloud management.
Enabling intelligent automation, from unit and integration tests to auto-remediation policies and policy-driven approvals.
Incorporating AI-driven optimization and insights, helping teams detect misconfigurations, analyze usage patterns, and generate infrastructure updates with tools like Pulumi Neo and Pulumi Policies.

As organizations scale both traditional cloud workloads and AI-driven systems, IaC has become critical for achieving secure, repeatable, and high-velocity operations across every environment.

6. DevSecOps Evolves Into AI-Integrated Security

As AI becomes embedded across cloud-native systems, DevSecOps is entering a new era. Gartner predicts that by 2028, over 50% of enterprises will use AI security platforms to protect their AI investments. Below are the 3 key predictions for the future of DevSecOps:

AI-driven security automation: Teams will increasingly rely on AI to detect threats, enforce policies, and generate secure infrastructure patches. See Pulumi’s capabilities in AI-powered remediation.
More focus on secrets management: With AI systems accessing more sensitive data, secure secret storage will be essential. Pulumi ESC helps teams centralize and govern credentials, keys, and tokens safely.
Greater cross-team collaboration: Dev, Sec, and Ops workflows will converge under shared frameworks: IaC, policy automation, runtime scanning, and GitOps.

As organizations increase their use of AI across cloud-native systems, the need for tightly aligned security, governance, and cloud governance automation becomes even more urgent. At the Gartner Data & Analytics Summit in Sydney, Carlie Idoine, VP Analyst at Gartner, emphasized this growing dependency:

"[AI]… it doesn’t deliver value on its own – AI needs to be tightly aligned with data, analytics, and governance to enable intelligent, adaptive decisions and actions across the organization."

This perspective mirrors what we’re seeing across modern DevSecOps practices: AI can amplify security, but only when paired with strong foundations in secrets management, governance, and cross-team collaboration.

Build for an AI-first cloud

Pulumi gives teams infrastructure as code, reusable components, and policy guardrails to deliver consistently across every cloud and AI workload.

Get started

7. Platform Engineering & Internal Developer Platforms (IDPs)

According to Gartner, by 2026, 80% of large software engineering organizations will establish platform engineering teams as internal providers of reusable services, components, and tools for application delivery. Platform engineering will ultimately solve the central problem of cooperation between software developers and operators.

Mid-size to large companies will begin or continue to invest in implementing platform engineering practices, with large tech companies as first adopters. They will provide Internal Developer Platforms (IDP) to elevate the Developer Experience (DX, sometimes referred to as DE or DevEx), helping them work faster, like abstracting the complexities of configuring, testing, and validation, deploying infrastructure, and scanning their code for security.

Internal developer platform-in-a-box. Credit: Pulumi

IDPs are reshaping how developers interact with cloud infrastructure, bringing together platform engineering, automation, and emerging AI platform engineering practices.

8. AIOps Matures into a Cloud Operations Standard

AIOps is becoming mainstream, helping teams predict failures, auto-scale infrastructure, and resolve incidents with minimal manual effort. As AI and automation continue to evolve, the fusion of these technologies will enable organizations to achieve unprecedented levels of efficiency and scalability.

Proactive Operations: AI-powered tools will assist teams in foreseeing issues with greater accuracy, minimizing downtime, and reducing the firefighting nature of incident management. These tools will automatically detect anomalies, optimize performance, and trigger remediation actions.
Intelligent Automation: Routine operational tasks like patching, monitoring, and resource scaling will be fully automated. AI-driven decision-making will allow for smarter resource allocation and optimization, dynamically adjusting infrastructure and workloads in response to real-time demands and predictions.
Data-Driven Insights: AIOps will analyze vast amounts of operational data and provide actionable insights, enabling teams to focus on high-impact tasks such as improving system architecture and user experience. The AI-powered insights will also inform better strategic decisions, helping teams to continuously evolve their DevOps practices.
Collaboration Across Teams: AIOps will bridge the gap between DevOps, SecOps, and IT operations by bridging monitoring and automation. Cross-team collaboration will improve as AI systems consolidate and interpret data from various departments, allowing for a more cohesive approach to system management.

AIOps features include observability, automation, and real-time analytics to bridge DevOps, SRE, and IT operations.

9. Kubernetes Dominance and Increased Complexity

Kubernetes will continue its ascent in 2026. According to Research & Markets, the global Kubernetes market was valued at USD 2.3 billion in 2024 and is projected to reach USD 8.2 billion by 2030, with a CAGR of 23.8% over the forecast period.

The CNCF Annual Survey shows AI/ML workloads rapidly moving onto Kubernetes — including batch pipelines, model experimentation, real-time inference, and data preprocessing — even as only 41% of ML/AI developers are cloud-native today. This shift is accelerating as teams need flexible GPU scheduling, distributed pipelines, and portable execution environments.

Kubernetes is also evolving in response to AI demands. Inference workloads, powered by LLMs and GPUs, now require low-latency execution closer to the user. This shift is pushing organizations to build intelligent orchestration layers that schedule AI pipelines across edge and core clusters, often leveraging Kubernetes as the common control plane for AI cluster orchestration.

As we move into 2026, three patterns are becoming clear:

Kubernetes is evolving to support AI through GPU-aware scheduling, Kubernetes GPU scheduling optimizations, and more advanced workload orchestration.
Governance and consistency matter more than ever, as teams struggle to secure and manage multi-cluster, multi-cloud environments.
Platform engineering is essential, providing curated patterns and automation rather than raw YAML to reduce cognitive load.

Kubernetes will remain a strategic foundation — but operating it effectively now depends on robust automation, strong security controls, and standardized delivery models that scale across clouds, clusters, and AI pipelines.

10. AI Code Assistants in the Enterprise

AI-powered coding assistants like GitHub Copilot, Claude Code, Cursor, and others are rapidly becoming part of modern software development.

By 2027, the use of AI assistants will dramatically increase developer velocity to meet functional business requirements for 70% of new digital solutions in production (source: IDC).

The value of AI code assistants. Credit: Gartner

According to Gartner, by 2028, 75% of enterprise software engineers will use dedicated AI code assistants, and 63% of organizations are currently piloting, deploying or beginning to use AI code assistants.

As enterprise adoption of AI assistants increases, expectations are rising: they must not only generate code but also understand the state of infrastructure, configurations, and security posture. That means being able to answer questions about environments, surface misconfigurations, or act directly on infrastructure.

One of the newest developments is the release of Pulumi Agent Skills, a collection of infrastructure expertise packaged for use in AI coding assistants. These skills teach tools such as Claude Code, Cursor, or Gemini CLI to reason about Pulumi projects, reducing hallucination and improving outputs based on real infrastructure conventions and practices.

Combined with infrastructure access via tools like Pulumi’s Remote MCP Server, teams can build secure, AI-driven workflows where assistants provide insights and Pulumi Neo safely executes actions with previews, policies, and orchestration.

AI code assistants are no longer experimental; they’re fast becoming a competitive advantage in cloud software development.

The Future of Cloud: Reinvented for an AI-First Decade

Cloud infrastructure is entering its most transformative era since the rise of Kubernetes. The trends shaping 2026 reveal a clear pattern: AI is no longer a workload — it’s becoming the organizing principle of cloud strategy.

AI-native cloud architectures that require elastic compute, GPU orchestration, fast data access, and governance built into every layer
Infrastructure as Code as the operational backbone, standardizing deployments across AI, cloud, and hybrid environments
Platform engineering and IDPs to enable self-service, gold-standard patterns, and automated guardrails
Security integrated into every pipeline, with AI-assisted threat detection, strong secrets management, and policy-driven compliance
AIOps and intelligent automation are becoming standard for scaling modern cloud systems
Kubernetes evolving for AI, driving new orchestration patterns across edge, core, and inference clusters
Multi-cloud and hybrid ecosystems accelerating to support interoperability, resilience, and global workload placement

Taken together, these shifts point to a new model of cloud operations that is intelligent, automated, policy-aware, and built on software engineering principles rather than manual configuration.

Organizations that invest now in modern IaC, unified governance, reusable components, and policy frameworks — all core capabilities of the Pulumi Cloud platform — will be positioned to lead in an AI-first world. The gap between teams that modernize and those that do not will widen rapidly in 2026 and beyond.

Try Pulumi for Free

New Compliance Packs for CIS, NIST, and PCI DSS

Luke Ward — Wed, 05 Nov 2025 00:02:00 +0000

Achieving compliance with industry standards such as CIS, NIST, or PCI DSS is a foundational step for every organization. Yet for many teams, it’s often a manual, months-long process that involves interpreting controls, authoring custom policies, and validating configurations across multiple clouds. These challenges often slow progress toward a known and secure cloud state.

We’re changing that. To simplify this journey, Pulumi launched a new suite of pre-built compliance policy packs for CIS Controls v8.1, NIST SP 800-53 Rev. 5, and PCI DSS v4.0.

These packs are your accelerator for the “Get Clean” journey, allowing you to enforce critical security and compliance baselines across your cloud infrastructure in minutes, not months.

More Than Just Detection: The Complete Governance Lifecycle

Traditional security tools are reactive, scanning for problems after resources have been deployed. With Pulumi, these new compliance packs are the engine for an end-to-end governance lifecycle that integrates directly into your cloud operations.

Audit for Full Coverage: Run these packs in audit mode to scan your entire cloud estate, including resources managed by Pulumi and those created through other means. This gives you an instant, comprehensive view of your current compliance posture.
Triage and Remediate: When a pack finds a violation, the finding appears in the new Policy Findings hub. From there, your team can triage, assign, and track the issue through its entire lifecycle. And with our new AI-powered capabilities, you can assign the issue to Pulumi Neo to automatically generate a pull request with the fix.
Prevent Non-Compliance: Once your environment is clean, you use these same packs as preventative guardrails. By running them during pulumi up, you block non-compliant resources before they are ever created, ensuring you “Stay Clean.”

This tri-modal capability—Audit, Remediate, and Prevent—is uniquely powerful, allowing you to fix existing issues while stopping new ones from being introduced.

New and Expanded Compliance Packs

Our new policy packs provide extensive, out-of-the-box coverage for some of the most widely adopted security frameworks. They are authored and maintained by Pulumi experts and join our existing library to provide a comprehensive toolkit for cloud governance.

Framework	AWS	Azure	Google Cloud
CIS Controls v8.1	✅	✅	✅
NIST SP 800-53 Rev. 5	✅
PCI DSS v4.0	✅
HITRUST CSF v11.5	✅	✅	✅
Pulumi Best Practices	✅	✅	✅

Benefits of Pre-Built Packs

Accelerate Compliance: Implement comprehensive governance controls in minutes without authoring hundreds of policies from scratch.
Leverage Expert Knowledge: Packs are authored and maintained by Pulumi, incorporating deep expertise in cloud and the nuances of each framework.
Codify Controls for Audits: Demonstrate to auditors that specific compliance controls are consistently enforced through code, providing a clear evidence trail.
Reduce Risk Proactively: Catch common security risks and compliance violations before deployment, drastically reducing your organization’s exposure.

Get Started Today

These policy packs are available now and are the perfect way to begin your governance journey with Pulumi.

To get started, head to the Policies page in your Pulumi Cloud organization and click on the All tab to find these new packs. Add them to an Audit Policy Group and run a scan. Within minutes, you’ll see a complete picture of your compliance posture in the Policy Findings hub, ready for triage and remediation.

Need a compliance pack for a standard that isn’t listed here? Please let us know by raising a request on our GitHub repository.

Try Pulumi Policies

New to Pulumi? Start your governance journey today.

Sign up for Pulumi Cloud and start a compliance task with Neo
Read the Get Started guide to apply and manage policies across your cloud infrastructure
Join the Community Slack to share feedback on the new features

For complete documentation, visit our Policies documentation.

How to Implement Robust Security Guardrails Using Policy as Code

Adam Gordon Bell — Tue, 30 Sep 2025 00:00:00 +0000

Welcome to the third post in our IDP Best Practices series, where we explore how to implement policy as code with Pulumi CrossGuard to create deployment guardrails that make self-service infrastructure both powerful and safe.

Platform engineering presents a fundamental tension: we want to enable developer velocity while maintaining security and compliance. Every platform team faces the same question: how do you give teams the freedom to deploy infrastructure quickly without compromising on safety, security, or organizational standards? The answer isn’t to choose between speed and safety, but rather to embrace automated guardrails powered by policy as code that make both possible simultaneously.

Want hands-on experience? Access the complete demo code and policy examples from this workshop.

The Platform Engineering Challenge: Speed vs. Safety

Let me tell you a story that perfectly captures the platform engineering dilemma. Statsig, a fast-growing feature flag platform processing an incredible 2 trillion events daily, had one infrastructure engineer named Jason who handled all infrastructure requests. Everything worked smoothly until Jason went on parental leave. Suddenly, the entire engineering organization ground to a halt. Infrastructure requests piled up, deployments slowed to a crawl, and the team faced a stark realization: their entire infrastructure capability depended on a single person.

This crisis forced them to confront a fundamental question that every growing engineering team eventually faces: How do you enable self-service infrastructure without sacrificing security, compliance, or operational stability? The answer wasn’t to hire more Jasons or to lock down infrastructure even tighter. Instead, they discovered that deployment guardrails with policy as code could transform their infrastructure conversation from “talk to the infrastructure person” to “ship with confidence, knowing guardrails will catch any issues.”

Think of it like building a system of roads with guardrails. When you create well-designed infrastructure components with proper safety barriers, teams can drive fast and confidently, knowing they’re protected from going off the cliff. The guardrails don’t slow them down; they enable speed by removing fear.

Understanding Platform Engineering Layers

Before we dive into implementing guardrails, we need to understand the architecture of a modern platform and where policies fit within it. Think of your platform as having three distinct layers, each with different security and access requirements.

At the foundation, you have your Layer 1: Foundational Infrastructure, which includes security controls, shared networking, and identity management systems like OIDC and IAM. Platform teams typically manage this layer with strict access controls because it forms the security backbone of your entire infrastructure.

Building on that foundation is Layer 2: Shared Infrastructure, which encompasses VPCs, compute platforms, and load balancers. These are standardized components that teams can use but with some room for customization based on specific needs.

The most dynamic layer is Layer 3: The Workloads Layer, where deployable artifacts, pipelines, and operability tools live. This is where the rubber meets the road, where most self-service activity happens and, consequently, where guardrails become absolutely critical. Developers need maximum freedom at this layer to innovate and ship quickly, but this freedom also introduces the most risk. It’s precisely here that policy-driven guardrails prove their worth, enabling that freedom while automatically preventing dangerous configurations.

What Are Deployment Guardrails?

Deployment guardrails are automated policies that act as your infrastructure’s safety net. Rather than relying on manual reviews or hoping developers remember all the security requirements, guardrails automatically prevent misconfigurations before they reach production. They enforce security standards without human intervention, guide developers toward best practices through immediate feedback, and enable safe self-service by catching issues at the earliest possible moment.

A helpful analogy is to think of guardrails like type checking in programming languages. Just as TypeScript doesn’t restrict your ability to write JavaScript but rather catches type errors before runtime, deployment guardrails don’t limit your infrastructure creativity. They simply ensure you’re following secure patterns and catch potentially dangerous configurations before they cause problems in production.

Introducing Pulumi CrossGuard: Policy as Code

Pulumi CrossGuard is Pulumi’s policy as code framework that brings the same engineering rigor to compliance and security that you apply to your application code. Instead of maintaining policy documents in wikis or relying on manual reviews, you can write policies in familiar programming languages like Python, TypeScript, or Go. These policies then enforce themselves across all your cloud resources and providers, running at different stages of the deployment lifecycle and integrating seamlessly with your CI/CD pipelines for automated enforcement.

Key Policy Types

CrossGuard supports two fundamental types of policies, each serving different validation needs:

Resource Policies: Validate individual resources

def restrict_dangerous_ports(args: ResourceValidationArgs, report_violation: ReportViolation):
 if args.resource_type == "aws:lb/targetGroup:TargetGroup":
 port = args.props.get("port")
 dangerous_ports = ["22", "23", "3389"]
 if str(port) in dangerous_ports:
 report_violation("Dangerous port detected. Avoid using SSH, Telnet, or RDP ports.")

Stack Policies: Validate relationships across resources

def validate_microservice_encryption(args: StackValidationArgs, report_violation: ReportViolation):
 microservice_components = []
 s3_buckets = []

 for resource in args.resources:
 if resource.resource_type == "custom:infrastructure:Microservice":
 microservice_components.append(resource)
 elif resource.resource_type == "aws:s3/bucket:Bucket":
 s3_buckets.append(resource)

 if microservice_components and s3_buckets:
 for bucket in s3_buckets:
 encryption = bucket.props.get("serverSideEncryptionConfiguration")
 if not encryption:
 report_violation("S3 bucket must have encryption enabled when used with microservice components.")

Building Practical Guardrails: Real-World Examples

Let’s walk through building guardrails for a microservice platform, starting with the component from our previous workshop.

Example 1: Port Security Policy

Problem: Developers might accidentally expose services on dangerous ports like SSH (22) or RDP (3389).

Solution: A policy that blocks dangerous port configurations.

import pulumi_policy as policy

def restrict_dangerous_ports_validation(args, report_violation):
 """Prevent services from running on dangerous ports."""
 if args.resource_type == "aws:lb/targetGroup:TargetGroup":
 port = args.props.get("port")
 dangerous_ports = ["22", "23", "3389", "5432", "3306"]

 if str(port) in dangerous_ports:
 report_violation(
 f"Port {port} is not allowed. This port is commonly used for "
 f"administrative services and should not be exposed via load balancer."
 )

restrict_dangerous_ports = policy.ResourceValidationPolicy(
 name="restrict-dangerous-ports",
 description="Prevent services from using dangerous ports",
 validate=restrict_dangerous_ports_validation,
 enforcement_level=policy.EnforcementLevel.MANDATORY
)

Example 2: Resource Limits Policy

Problem: Teams might request oversized resources, leading to cost overruns.

Solution: A policy that enforces reasonable resource limits with advisory warnings.

def limit_memory_validation(args, report_violation):
 """Limit memory allocation for microservices."""
 if args.resource_type == "custom:infrastructure:Microservice":
 # Extract memory from component tags or properties
 memory_str = args.props.get("memory", "512Mi")

 # Parse memory value (assuming format like "2Gi", "1024Mi")
 if memory_str.endswith("Gi"):
 memory_gb = float(memory_str[:-2])
 if memory_gb > 1:
 report_violation(
 f"Memory allocation of {memory_str} exceeds recommended limit of 1Gi "
 f"for microservices. Consider optimizing your application or "
 f"contact the platform team for exceptions."
 )

limit_memory = policy.ResourceValidationPolicy(
 name="limit-microservice-memory",
 description="Enforce reasonable memory limits for microservices",
 validate=limit_memory_validation,
 enforcement_level=policy.EnforcementLevel.ADVISORY
)

Example 3: Cross-Resource Security Policy

Problem: When microservices use S3 buckets, encryption should be mandatory.

Solution: A stack policy that validates encryption across related resources.

def microservice_s3_encryption_validation(args, report_violation):
 """Ensure S3 buckets used with microservices have encryption enabled."""
 microservice_resources = []
 s3_buckets = []

 # Collect relevant resources
 for resource in args.resources:
 if resource.resource_type == "custom:infrastructure:Microservice":
 microservice_resources.append(resource)
 elif resource.resource_type == "aws:s3/bucket:Bucket":
 s3_buckets.append(resource)

 # If we have both microservices and S3 buckets, check encryption
 if microservice_resources and s3_buckets:
 for bucket in s3_buckets:
 sse_config = bucket.props.get("serverSideEncryptionConfiguration")
 if not sse_config:
 report_violation(
 f"S3 bucket '{bucket.name}' must have server-side encryption "
 f"enabled when used with microservice components."
 )

microservice_s3_encryption = policy.StackValidationPolicy(
 name="microservice-s3-encryption",
 description="Ensure S3 buckets used with microservices are encrypted",
 validate=microservice_s3_encryption_validation,
 enforcement_level=policy.EnforcementLevel.MANDATORY
)

Policy Enforcement Models

Pulumi CrossGuard supports multiple enforcement models to fit different workflows, and understanding when to use each model is crucial for effective policy implementation.

The Preventative Model

The most common approach is the preventative model, where policies run before resources are created. This gives you fast feedback and prevents bad deployments from ever reaching the cloud. When you run pulumi preview, policies are evaluated immediately, and if violations are found during pulumi up, the deployment is blocked. This model works particularly well for validating port configurations, resource limits, and other settings that you can check without needing to see the actual deployed resource.

# Policies run automatically during preview and deployment
pulumi preview # Policies evaluated here
pulumi up # Policies block deployment if violations found

The Detective Model

Sometimes you need to validate the actual cloud state, including auto-generated values like ARNs or IP addresses. That’s where the detective model comes in. These policies run after resources are created, making them perfect for compliance scanning and drift detection. While they can’t prevent initial deployment of non-compliant resources, they excel at catching configuration drift and validating properties that only exist after deployment.

CI/CD Integration

The third model integrates policies directly into your deployment pipeline. This ensures consistent enforcement across all teams and creates natural deployment gates. For example, you might configure GitHub Actions to run policy validation on every pull request, blocking merges if violations are found. This approach combines the best of both worlds: early feedback during development and guaranteed enforcement before production.

# GitHub Actions example
- name: Run Policy Validation
 run: |
 pulumi preview --policy-pack ./policies
 if [ $? -ne 0 ]; then
 echo "Policy violations found. Deployment blocked."
 exit 1
 fi

Policy Remediation: Beyond Detection

Modern policy frameworks don’t just detect violations; they can automatically fix them:

def auto_tag_resources(args, report_violation):
 """Automatically add required tags to resources."""
 if args.resource_type == "aws:s3/bucket:Bucket":
 tags = args.props.get("tags", {})

 if "Department" not in tags:
 # Instead of just reporting, fix the issue
 if not hasattr(args, 'tags'):
 args.props["tags"] = {}
 args.props["tags"]["Department"] = "Engineering"
 return args.props # Return modified properties

 return None # No changes needed

auto_tag_policy = policy.ResourceValidationPolicy(
 name="auto-tag-resources",
 description="Automatically add required tags",
 validate=auto_tag_resources,
 enforcement_level=policy.EnforcementLevel.REMEDIATE
)

Server-Side Policy Enforcement

For enterprise deployments, Pulumi provides server-side policy enforcement that ensures policies can’t be bypassed. The process starts by publishing your policies to your Pulumi organization with pulumi policy publish ./my-policies. Once published, you can create policy groups that combine multiple policies with specific enforcement levels, targeting particular stacks or environments while configuring exceptions for special cases. The beauty of this approach is that policies run automatically without requiring CLI flags, providing consistent governance across your entire organization without relying on developers to remember to include policy packs in their commands.

Compliance-Ready Policies

While custom policies address your specific organizational needs, compliance requirements often follow industry standards. Pulumi provides hundreds of pre-built policies for common compliance frameworks:

import { PolicyManager } from "@pulumi/policy";
import { policyManager } from "@pulumi/compliance-ready-policies/policy-manager";

new PolicyPack("aws-compliance-ready-policies-typescript", {
 policies: [
 ...policyManager.selectPolicies({
 vendors: ["aws"],
 services: ["ec2", "s3"],
 severities: ["medium", "high", "critical"],
 topics: ["encryption"],
 frameworks: ["pcidss"],
 }, "mandatory"),
 ],
});

This automatically includes policies for major compliance frameworks like PCI DSS for payment card industry standards, SOC 2 for security and compliance controls, ISO 27001 for information security management, and CIS Benchmarks for security configuration standards.

Add guardrails with policy as code

Write policies in Python or TypeScript, enforce them across every deployment, and give teams self-service infrastructure that stays secure and compliant.

Get started

Best Practices for Policy Implementation

After implementing policies at dozens of organizations, we’ve learned that successful policy adoption follows predictable patterns. The key is to adopt a product mindset: think of your policies as a product serving your development teams, not as bureaucratic rules imposed from above. Start small and iterate based on real feedback rather than trying to implement a comprehensive policy framework from day one.

Start Small and Iterate

Begin with just two or three critical policies that address your most pressing risks. Use advisory enforcement initially, which warns developers about violations but doesn’t block deployments. This gives your team time to understand and adapt to the policies. Only after gathering feedback and refining the policies should you graduate to mandatory enforcement.

Provide Clear, Actionable Error Messages

Nothing frustrates developers more than cryptic policy violations. Your error messages should explain not just what’s wrong, but how to fix it:

def good_error_message(args, report_violation):
 if violation_detected:
 report_violation(
 "S3 bucket encryption is required for compliance. "
 "Add serverSideEncryptionConfiguration to fix this. "
 "See: https://docs.company.com/s3-encryption"
 )

Embrace Progressive Enforcement

Think of enforcement levels as a dial, not a switch. Start with advisory mode to warn about issues, move to mandatory to block deployments, and eventually implement remediation to automatically fix common issues. This progression gives teams time to adapt while gradually raising the security bar.

Test Your Policies Thoroughly

Policies are code, and like any code, they need testing. Write unit tests for your policies to ensure they catch violations correctly and don’t produce false positives:

# Test policies with unit tests
def test_dangerous_port_policy():
 # Mock resource with port 22
 args = create_mock_args("aws:lb/targetGroup:TargetGroup", {"port": 22})
 violations = run_policy(restrict_dangerous_ports, args)
 assert len(violations) == 1
 assert "dangerous port" in violations[0].message.lower()

Document Escape Hatches

No matter how well-designed your policies are, there will always be edge cases. Rather than forcing teams to work around policies in dangerous ways, provide clear, documented escape hatches. Create processes for requesting exceptions, handling emergency exemptions, and appealing for policy changes. As one platform engineer put it, you need “guardrails outside the guardrails” because platform adoption will never be 100%, and that’s okay. This transparency builds trust and ensures policies remain effective enablers rather than bureaucratic obstacles to circumvent.

Real-World Success: Statsig’s Transformation

Remember Statsig’s infrastructure crisis when Jason went on parental leave? Let’s look at how they transformed their platform using the guardrails approach we’ve been discussing. After implementing policy-driven guardrails, their infrastructure story completely changed.

As Tyrone Wong, Infrastructure Engineer at Statsig, explains: “We had a single infra owner. Parental leave forced us to build self-service, and that’s when everything clicked. Developers open a PR, Pulumi previews the change right in the PR, and CI blocks risky changes. That’s how you move fast safely. The magic was turning ’talk to the infra person’ into ‘ship with guardrails.’”

The results speak for themselves. Deployment times dropped from 1.5 weeks to mere minutes. The infrastructure ticket queue that once haunted their Slack channels disappeared entirely. Security actually improved through automated policy enforcement, eliminating the human error factor. Most importantly, developer satisfaction soared as teams gained the autonomy to ship infrastructure changes without friction or fear.

When asked if developers were actually using the self-service platform, Tyrone’s response was telling: “Yeah, people are actually using it because there’s changes they wanted to get done. Previously it became this giant effort with lots of meetings, people being concerned, and Google docs about what the changes would be. Now people can just put together a PR, we have our discussions there, see if anything gets flagged, and move forward.”

Building Your Policy Strategy

Implementing deployment guardrails isn’t a big-bang transformation; it’s a journey that unfolds in phases. Based on patterns we’ve seen across successful implementations, the first week or two should focus on assessment. Start by understanding your current state, auditing existing infrastructure patterns to identify what teams are actually deploying. Look for common misconfigurations that have caused incidents or near-misses, and document your security and compliance requirements as specific, enforceable rules. Most importantly, survey your developers to understand their pain points with the current infrastructure process, as this assessment forms the foundation for policies that solve real problems rather than creating new ones.

In weeks three and four, build your foundation by implementing three to five core policies that address your most critical risks. Set up CI/CD integration so policies run automatically on every pull request, starting with advisory enforcement to gather feedback without blocking deployments. Create clear documentation and runbooks that explain not just what the policies do, but why they exist and how to work with them.

By the second month, you’re ready to expand. Add compliance-specific policies for regulatory requirements and implement server-side enforcement to ensure policies can’t be bypassed. Create formal processes for policy exemptions and exceptions, and begin measuring policy effectiveness through metrics like violation rates and remediation times.

Remember that policy implementation is never “done.” Continuously monitor violation patterns to identify areas where policies might be too strict or too lenient. Refine policies based on developer feedback and incident data, add automated remediation for common violations to reduce manual fixes, and gradually expand coverage to new services and teams using lessons learned from early adopters.

Measuring Policy Success

You can’t improve what you don’t measure, and policy effectiveness is no exception. Successful teams track metrics across three critical dimensions to validate their policy strategy.

From a security perspective, start by monitoring your policy violation rate: the percentage of deployments that trigger policy warnings or blocks. This tells you whether your policies are catching real issues or creating unnecessary friction. Track time to remediation to understand how quickly teams fix violations when they occur. Most importantly, measure the reduction in security incidents caused by misconfigurations. If your policies aren’t reducing incidents, they’re not addressing the right risks.

Security is only half the equation, though. The developer experience dimension is equally important. Monitor deployment velocity to ensure policies aren’t slowing teams down. Regular developer satisfaction surveys reveal whether teams see policies as helpful guardrails or annoying obstacles. Track support ticket volume; successful policy implementation should reduce infrastructure-related requests as teams become more self-sufficient.

Finally, don’t forget to measure the operational health of your policy system itself. Policy coverage shows what percentage of your resources are protected by policies, while the exemption rate reveals whether policies are practical or if teams constantly need exceptions. Track remediation automation to see how many violations are fixed automatically versus requiring manual intervention. Together, these metrics paint a complete picture of your policy program’s health and help you continuously improve your approach.

Common Pitfalls and How to Avoid Them

Even with the best intentions, policy implementations can go wrong. Through years of helping organizations implement policy as code, we’ve identified four critical pitfalls that can derail your efforts.

The first and most common trap is over-engineering policies. It’s tempting to create sophisticated policies that handle every edge case, but overly complex policies are hard to understand and maintain. Teams won’t trust what they don’t understand. Instead, start with simple, clear policies that address specific risks. Use descriptive naming and provide examples of both compliant and non-compliant configurations. Remember, a simple policy that everyone follows is better than a perfect policy that everyone circumvents.

Equally dangerous is falling into “policy theater”: implementing policies because you think you should, not because they address real risks. This creates friction without improving security. Every policy should trace back to an actual incident, security requirement, or compliance need. If you can’t explain why a policy exists with a concrete example, it probably shouldn’t exist.

Poor developer experience is another common failure point. Nothing kills policy adoption faster than cryptic error messages and unclear remediation steps. When a developer hits a policy violation at 5 PM on a Friday, they need clear guidance on how to fix it. Invest heavily in documentation, helpful error messages, and examples. Consider your policies’ error messages as part of your product’s user experience.

Finally, inadequate testing can erode trust quickly. Policies that break legitimate use cases will be worked around or disabled. Before enforcing any policy, test it thoroughly with real scenarios from your production workloads. Run policies in advisory mode first to catch false positives, and have teams attempt real deployments with policies enabled to ensure they don’t block valid configurations.

The Future of Policy as Code

As infrastructure becomes increasingly complex and distributed, policy as code is evolving beyond simple rule enforcement to become an intelligent layer that understands context and intent. Three major trends are shaping this evolution.

First, we’re seeing the emergence of AI-enhanced policies with smart detection systems that use machine learning to identify anomalies that rule-based policies might miss. These systems learn from your infrastructure patterns and can provide contextual recommendations that adapt based on actual usage. Even more exciting is predictive enforcement: imagine policies that can identify risky patterns before they become violations, guiding developers away from problems they haven’t encountered yet.

Second, the multi-cloud reality is driving the need for universal governance. Organizations need policies that work seamlessly across AWS, Azure, and GCP, with federated enforcement that maintains consistency across multiple cloud accounts and regions. Compliance automation is also maturing, with systems that automatically collect evidence for audits and generate compliance reports without manual intervention.

Finally, policy enforcement is moving closer to where developers actually work. IDE integration will soon provide real-time policy feedback as you write infrastructure code, catching issues before you even attempt to deploy. Self-service exemption workflows will let developers request and receive policy exceptions through automated approval processes. Perhaps most intriguingly, we’re seeing the development of learning policies: systems that improve and adapt based on developer feedback and usage patterns, becoming more helpful over time rather than more restrictive.

Conclusion: Enabling Safe Self-Service at Scale

We started this post with a fundamental tension in platform engineering: the need for both speed and safety. Through the lens of Statsig’s transformation and the technical deep-dive into Pulumi CrossGuard, we’ve seen that this isn’t actually a tension that needs resolving. It’s a false dichotomy that policy as code eliminates entirely.

The key insight from successful platform teams like Statsig is that guardrails don’t restrict freedom; they enable it. When developers know that automated policies will catch dangerous configurations, they gain the confidence to move fast and experiment. When platform teams know that policies automatically enforce security and compliance standards, they can focus on building better platforms instead of reviewing every change. This is the magic of policy as code: it transforms infrastructure governance from a bottleneck into an accelerator.

But perhaps the most important lesson is that policy as code isn’t about saying “no” to developers. It’s about intelligent automation that makes the secure path the path of least resistance. It’s about catching mistakes before they become incidents. It’s about encoding your organization’s hard-won knowledge into systems that help every developer benefit from that experience.

As you embark on your own journey to implement deployment guardrails, remember that perfection isn’t the goal; progress is. Start small, iterate based on feedback, and gradually expand your coverage. Your developers will thank you for the clarity and confidence that comes with well-designed guardrails, and your security team will sleep better knowing that policies are enforced automatically and consistently.

The path from manual reviews to automated guardrails is well-traveled and well-documented. Our complete policy examples provide real-world implementations you can adapt to your needs, while the CrossGuard documentation offers deep technical details for advanced use cases. If you’re on AWS, AWSGuard’s pre-built policies offer immediate value, and our compliance-ready policy catalog addresses specific regulatory requirements.

The future of infrastructure management isn’t about choosing between developer autonomy and operational control. It’s about using policy as code to achieve both, creating platforms that are simultaneously powerful and safe, flexible and compliant, fast and secure.

In our next post, we’ll explore Day 2 Platform Operations, diving into how to maintain infrastructure compliance after deployment and automatically remediate configuration drift. Because getting to production is just the beginning; keeping your infrastructure secure and compliant over time is where the real challenge lies.

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

Secrets Management Tools: The Complete 2025 Guide

Asaf Ashirov — Thu, 24 Jul 2025 00:00:00 +0000

Every modern application depends on secrets to function: database passwords, API keys, certificates, and configuration values that enable secure communication between services. But here’s the challenge: as your infrastructure grows, managing these secrets becomes exponentially more complex.

The numbers tell a stark story. According to the Akeyless State of Secrets Management Report, 96% of organizations struggle with secrets sprawl such as credentials scattered across code repositories, configuration files, and deployment scripts. The consequences are severe: Verizon’s 2025 Data Breach Investigations Report found that 88% of data breaches involved compromised credentials, with IBM’s research showing the average breach now costs organizations $4.88 million.

The path forward requires more than just storing secrets securely, it demands intelligent orchestration across your entire infrastructure. But with dozens of solutions available, from traditional enterprise vaults to modern orchestration platforms, finding the right approach can feel overwhelming.

This guide cuts through the complexity. We’ll examine the leading secrets management tools across multiple categories, helping you understand not just what they do, but when and why to use them. Whether you’re architecting your startup’s first production environment or modernizing enterprise legacy systems, you’ll find actionable insights to make the right choice for your specific needs.

Secrets Management Tools Overview

The Evolution Beyond Simple Secret Storage

What started as simple password vaults has evolved into something far more sophisticated. Traditional secret storage addressed only part of the challenge: keeping credentials secure. But modern infrastructure demands orchestration across complex, distributed environments where applications need dozens of different secrets available across multiple cloud providers and deployment stages.

The fundamental shift happened when teams realized that managing secrets effectively requires treating them as infrastructure components, not just security artifacts. Your applications don’t just need database passwords stored securely—they need those credentials dynamically generated, automatically rotated, and consistently available across development, staging, and production environments without manual intervention.

Today’s leading platforms function as intelligent orchestrators rather than simple storage systems. They aggregate secrets from multiple existing sources, provide configuration-as-code capabilities that enable version control and reproducible deployments, and generate dynamic credentials with automatic expiration. This orchestration approach means you can work with your existing secret stores while gaining centralized management and modern capabilities like hierarchical organization that reduces duplication across environments.

The best solutions recognize that secrets management is fundamentally an infrastructure orchestration problem. They provide unified access patterns that work seamlessly across multi-cloud environments, comprehensive audit logging for compliance requirements, and fine-grained access controls that adapt to your organizational structure all while maintaining compatibility with the tools and workflows your teams already use.

Most Popular Secrets Management Tools

Secrets Orchestration Platforms

This represents the newest evolution in secrets management: platforms that connect and coordinate multiple secret sources rather than forcing you to abandon your existing infrastructure. Instead of requiring wholesale migration to yet another secret store, these intelligent brokers provide unified interfaces that work with your current tools while adding modern capabilities like dynamic credential generation and configuration-as-code workflows.

Pulumi ESC (Environments, Secrets, and Configuration)

Pulumi ESC takes a fundamentally different approach to secrets management. Rather than forcing you to migrate from your existing secret stores, it orchestrates them connecting HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, and over 20 other providers through a unified interface. This means you can modernize your secrets workflow without abandoning your current infrastructure investments.

The platform’s strength lies in its configuration-as-code approach. You define hierarchical YAML environments that cascade from base configurations through development, staging, and production. This eliminates the manual configuration copying that leads to secrets sprawl: define common settings once, inherit them everywhere they’re needed, while maintaining appropriate security boundaries between environments.

Enterprise-Grade Security and Compliance: ESC provides comprehensive audit logging with detailed access trails, including user identity, timestamp, access method, and resource details. The platform’s audit capabilities support compliance requirements for frameworks like SOC 2, GDPR, and industry-specific regulations by providing immutable audit logs and real-time monitoring of secret access patterns.

Performance and Scalability: ESC delivers enterprise-scale performance with low-latency secret retrieval, high availability across multiple regions, and the ability to handle thousands of concurrent requests. The architecture is designed to scale automatically with organizational growth while maintaining consistent sub-second response times for secret operations.

Dynamic credential generation sets Pulumi ESC apart from traditional storage-focused tools. Instead of storing long-lived AWS access keys or Azure service principal secrets, it automatically provisions short-lived OIDC tokens on-demand. This dynamic approach eliminates entire categories of security risk by ensuring credentials expire automatically and can’t be misused over extended periods.

Advanced Key Management: ESC provides advanced key management capabilities with support for industry-standard encryption algorithms, automated key rotation, and secure key derivation. The platform offers envelope encryption and fine-grained access controls that ensure encryption keys are managed securely and in accordance with regulatory requirements.

The platform maintains a zero vendor lock-in philosophy through its open-source engine and comprehensive multi-language SDK support for TypeScript, Python, Go, and .NET. Teams can integrate ESC into existing workflows through familiar CLI tooling while maintaining full visibility into orchestration logic through transparent, auditable code.

Cost-Effective Enterprise Solution: Pulumi ESC offers significant cost advantages through its unified platform approach, reducing the need for multiple point solutions and simplifying operational overhead. ESC’s integration with existing infrastructure workflows eliminates the need for separate tools and reduces training costs, while its automated rotation and compliance features reduce manual administrative tasks and associated labor costs. For organizations with strict data residency requirements, Pulumi Service offers self-hosting options that provide additional cost predictability and security control.

Pulumi ESC offers a generous free tier with unlimited environments, making it accessible for teams to experiment with configuration-as-code principles. Usage-based pricing for advanced features scales naturally with organizational growth, making it particularly attractive for multi-cloud environments and teams prioritizing systematic approaches to eliminating secrets sprawl.

Doppler

Doppler focuses on developer experience with an intuitive interface and extensive integrations for modern development workflows. The platform provides branch-based environment management and real-time secret synchronization across connected services, with plans starting at $3 per user per month.

Infisical

Infisical combines open-source transparency with modern secrets management, offering both self-hosted and cloud options. The platform includes built-in secrets scanning capabilities and comprehensive SDK support, with cloud service starting at $8 per user per month.

Enterprise Secrets Vaults

Traditional enterprise vaults focus on maximum security and compliance, offering comprehensive secret storage with advanced features designed for large organizations with sophisticated security requirements.

HashiCorp Vault

HashiCorp Vault remains the established standard for enterprise secrets management, offering unmatched flexibility and a comprehensive feature set for complex environments. The platform provides dynamic secrets generation with automatic expiration for over 50 different systems, enabling organizations to move away from static, long-lived credentials toward more secure, just-in-time access patterns.

Vault’s architecture supports over 100 authentication methods and secret engines, providing flexibility to integrate with virtually any existing infrastructure or identity system. Multi-region clustering with strong consistency guarantees ensures high availability and data integrity across geographically distributed deployments. Extensive audit logging and compliance reporting capabilities help organizations meet regulatory requirements while maintaining operational visibility.

The platform’s cloud-agnostic deployment options support both Kubernetes and traditional VM-based infrastructure, making it suitable for organizations with diverse technical environments. Vault’s open-source core provides transparency and community-driven development, while commercial enterprise features starting at $2 per node per hour add advanced management, replication, and support capabilities.

HashiCorp Vault excels in organizations requiring maximum flexibility and customization, complex multi-cloud environments, and teams with strong DevOps expertise capable of managing sophisticated infrastructure platforms.

CyberArk Conjur

CyberArk Conjur focuses on enterprise security with advanced compliance features and integration with broader privileged access management systems. The platform provides enterprise-grade role-based access control with comprehensive audit capabilities, enabling organizations to implement fine-grained security policies across their entire infrastructure.

Modern versions of Conjur incorporate AI-powered anomaly detection and threat analysis, helping security teams identify unusual access patterns that may indicate compromise or policy violations. Automated compliance reporting supports SOC 2, PCI-DSS, and other regulatory frameworks, reducing the operational overhead of maintaining compliance documentation.

Conjur’s integration with CyberArk’s broader privileged access management suite provides a comprehensive approach to credential security that extends beyond simple secret storage to include session monitoring, just-in-time access provisioning, and threat detection. The platform’s advanced policy engine enables fine-grained access controls that can adapt to changing organizational and regulatory requirements.

With enterprise licensing based on scale and deployment requirements, CyberArk Conjur targets regulated industries, enterprises with strict compliance requirements, and organizations prioritizing maximum security over operational simplicity.

Akeyless

Akeyless offers a modern, cloud-native approach to enterprise secrets management without the operational complexity of traditional self-hosted solutions. The platform’s SaaS-first architecture eliminates infrastructure management overhead while providing enterprise-grade capabilities including just-in-time access and automated credential rotation.

The platform implements zero-knowledge encryption with client-side key management, ensuring that even Akeyless cannot access customer secrets. This approach addresses concerns about cloud-based secret storage while maintaining the operational benefits of a managed service. A comprehensive API and integration ecosystem enables connection with existing tools and workflows.

Akeyless provides a generous free tier supporting up to 5 clients, with usage-based pricing that scales naturally with organizational growth. This pricing model, combined with the elimination of operational overhead, makes Akeyless attractive for organizations seeking enterprise features without the complexity of self-managed infrastructure, particularly cloud-first teams prioritizing rapid deployment and scaling.

Cloud-Native Secrets Managers

Cloud-native solutions provide tight integration with specific cloud platforms, offering optimized experiences for organizations committed to particular cloud ecosystems. These platforms leverage native cloud services and identity systems to provide seamless integration with existing infrastructure.

AWS Secrets Manager

Amazon’s native secrets management service provides deep integration with the AWS ecosystem and automatic rotation capabilities specifically designed for AWS services. The platform offers native integration with over 50 AWS services including RDS, Lambda, ECS, and EKS, enabling seamless credential management across the entire AWS service portfolio.

Automatic rotation capabilities for supported databases and AWS services eliminate the operational overhead of manual credential management while improving security through regular rotation cycles. Cross-region replication with automatic failover ensures high availability for critical secrets, while fine-grained IAM permissions with resource-based policies provide precise access control aligned with AWS security best practices.

VPC endpoint support enables private network access without internet routing, addressing security requirements for air-gapped or highly secure environments. The service’s pricing model of $0.40 per secret per month plus $0.05 per 10,000 API calls provides predictable costs that scale with usage.

AWS Secrets Manager excels for AWS-native organizations, applications requiring automatic credential rotation, and teams leveraging extensive AWS service portfolios where tight integration provides operational efficiency and security benefits.

Azure Key Vault

Microsoft’s comprehensive platform manages secrets, keys, and certificates within the Azure ecosystem while providing strong compliance features particularly valued by government and enterprise customers. The platform integrates seamlessly with Azure services and Active Directory authentication, leveraging existing identity infrastructure to provide consistent access control across the Azure ecosystem.

FIPS 140-2 Level 2 validated Hardware Security Modules provide the highest levels of cryptographic security, making Azure Key Vault suitable for government contractors and organizations with stringent security requirements. Managed HSM support extends these capabilities for scenarios requiring maximum security control and audit capabilities.

Certificate lifecycle management with automatic renewal reduces operational overhead while ensuring consistent security posture across web applications and services. Virtual network integration and private endpoint support enable secure access patterns that align with enterprise network security architectures.

Azure Key Vault’s standard tier pricing of $0.03 per 10,000 operations provides cost-effective secret management for Azure-centric organizations, government contractors requiring FIPS compliance, and enterprises with Microsoft-heavy infrastructure environments.

Google Secret Manager

Google Cloud’s scalable secrets management service is optimized for high-volume, global deployments with strong integration across GCP services. The platform provides automatic global replication across all GCP regions, ensuring low-latency access to secrets regardless of geographic distribution.

IAM integration supports fine-grained, condition-based permissions that can adapt to complex organizational structures and security requirements. Secret versioning with automatic rollback capabilities provides operational safety for configuration changes, while integration with Cloud Build, GKE, Cloud Run, and App Engine enables seamless secret management across Google Cloud’s entire service portfolio.

The platform’s high-performance API with global edge caching minimizes latency for high-frequency secret retrieval operations, making it suitable for applications with demanding performance requirements. Pricing of $0.06 per secret version per month plus $0.03 per 10,000 API calls provides cost-effective scaling for high-volume usage patterns.

Google Secret Manager serves Google Cloud-native organizations, applications requiring massive scale, and global deployments where low latency and high availability are critical operational requirements.

Developer-Focused Tools

These tools prioritize user experience and ease of adoption, often serving as excellent entry points for teams beginning their secrets management journey. They emphasize intuitive interfaces and efficient workflows that reduce barriers to adoption.

1Password Secrets Automation

1Password has successfully expanded beyond personal password management to offer comprehensive business secrets management while maintaining their renowned user experience. The platform implements zero-knowledge architecture with client-side encryption, ensuring that even 1Password cannot access customer secrets while providing the usability that made their consumer products successful.

The intuitive interface remains accessible to both technical and non-technical users, addressing the common challenge of secrets management tools that require specialized expertise. Comprehensive SDK and CLI support enables developer workflows while maintaining the simplicity that drives organization-wide adoption. Integration with popular CI/CD platforms and development tools provides the automation capabilities modern development teams require.

Advanced sharing capabilities include granular, time-limited permissions that enable secure collaboration while maintaining audit trails and access control. This combination of security and usability makes 1Password particularly effective in mixed environments where both technical and business users need access to secrets.

At $8 per user per month with volume discounts available, 1Password Secrets Automation targets teams prioritizing usability over advanced features, mixed technical/non-technical environments, and organizations already invested in the 1Password ecosystem.

Bitwarden Secrets Manager

Bitwarden leverages their password management expertise to provide cost-effective secrets management specifically designed for development teams. The platform includes machine account support for automated CI/CD workflows, addressing the common requirement for non-human access to secrets in modern development processes.

CLI and SDK integration covers major programming languages while maintaining the straightforward approach that characterizes Bitwarden’s products. Comprehensive event logging and audit trails provide the visibility required for security and compliance, while two-person integrity controls add an extra layer of protection for sensitive operations.

The platform’s competitive per-user pricing of $3 per month with transparent costs makes it particularly attractive for small to medium teams, organizations seeking simple and cost-effective solutions, and existing Bitwarden users looking to extend their investment into development workflows.

Application Security & Scanning

These tools focus on preventing secrets leakage by detecting credentials in code repositories, CI/CD pipelines, and other development artifacts. Rather than managing secrets after they’re created, these platforms prevent security incidents by identifying and remediating credential exposure.

GitGuardian

GitGuardian provides comprehensive secrets detection and remediation across the entire software development lifecycle with high accuracy and low false positives. The platform offers real-time scanning of commits, pull requests, and issues across all major Git platforms, ensuring that credential exposure is identified and addressed before code reaches production environments.

Advanced machine learning algorithms detect over 350 secret types while minimizing false positives that can lead to alert fatigue. Developer-friendly remediation workflows provide guided resolution steps that help development teams address issues quickly without disrupting their normal workflows. Historical repository scanning capabilities enable comprehensive security assessment of existing codebases.

Integration with GitHub, GitLab, Bitbucket, and Azure DevOps ensures coverage across diverse development environments. The platform’s free tier supports small teams, while enterprise plans starting at $18 per developer per month provide advanced features and support for larger organizations.

GitGuardian excels in organizations with large codebases, teams implementing DevSecOps practices, and compliance-focused environments where preventing credential exposure is critical to maintaining security posture.

TruffleHog

TruffleHog offers powerful open-source secrets detection with both community and enterprise versions for comprehensive credential scanning. The platform uses high-performance scanning algorithms that combine advanced entropy analysis with pattern matching to identify credentials across diverse file types and encoding formats.

Support for over 700 credential types includes custom patterns that can adapt to organization-specific secret formats. CI/CD integration with GitHub Actions, GitLab CI, and Jenkins enables automated scanning as part of existing development workflows. Historical git repository scanning provides commit-level analysis that can identify when and how credentials entered the codebase.

The active open-source community ensures regular updates and improvements while providing transparency into detection algorithms. Enterprise features and support options are available for organizations requiring commercial backing and advanced capabilities.

TruffleHog serves open-source advocates, security-conscious development teams, and organizations with strong technical security expertise capable of implementing and maintaining open-source security tools.

Specialized & Integration Tools

These tools address specific use cases or provide bridge functionality between secrets managers and deployment platforms, filling gaps in comprehensive secrets management architectures.

External Secrets Operator

The External Secrets Operator provides Kubernetes-native integration with external secrets management systems, enabling GitOps workflows with secure secret handling. The platform uses Kubernetes Custom Resource Definitions to provide native resource management that integrates seamlessly with existing Kubernetes operational patterns.

Multi-provider support includes Vault, AWS, Azure, GCP, and numerous other secret sources, enabling organizations to maintain consistent Kubernetes secret management regardless of their underlying secret storage choices. Automatic secret synchronization with configurable refresh intervals ensures that Kubernetes secrets remain current with external sources while minimizing API load.

Template-based secret transformation and formatting capabilities enable adaptation of external secret formats to Kubernetes requirements. As an active CNCF community project with growing enterprise adoption, the External Secrets Operator benefits from diverse contributions and broad compatibility testing.

This free and open-source tool excels in Kubernetes-heavy environments, teams adopting GitOps practices, and organizations requiring cloud-agnostic secret synchronization across diverse infrastructure platforms.

Berglas

Google’s Berglas provides optimized secrets injection for serverless and container environments, particularly Google Cloud Run and similar platforms. The platform’s serverless-optimized secret injection minimizes cold start impact, addressing one of the key performance challenges in serverless architectures.

Deep integration with Google Cloud services and IAM leverages existing cloud infrastructure to provide secure, efficient secret access. Container-native deployment patterns use init container approaches that work seamlessly with existing containerized application architectures. Minimal runtime overhead and memory footprint ensure that secret management doesn’t impact application performance.

As an open-source project backed by Google’s engineering team, Berglas benefits from direct integration with Google Cloud services and ongoing development aligned with Google Cloud platform evolution. The free and open-source nature makes it accessible for teams of all sizes.

Berglas serves Google Cloud serverless deployments, container-native applications, and teams prioritizing minimal overhead in performance-sensitive environments.

Confidant

Lyft’s open-source Confidant offers a production-proven approach to secrets management with focus on developer experience and AWS optimization. Battle-tested at Lyft’s scale with millions of requests daily, Confidant provides real-world validation of its architecture and performance characteristics.

AWS-optimized architecture leverages KMS and DynamoDB to provide secure, scalable secret storage using proven AWS services. Developer-centric design emphasizes simple API and CLI interfaces that reduce friction in day-to-day development workflows. The blind credentials feature provides enhanced security by ensuring that even system administrators cannot access certain types of sensitive information.

IAM integration with fine-grained permissions leverages AWS’s native access control systems to provide consistent security policies across the entire infrastructure. As a free and open-source solution, Confidant enables organizations to implement enterprise-grade secret management without licensing costs.

Confidant excels in AWS-heavy environments, teams with strong technical capabilities, and organizations seeking proven open-source solutions with demonstrated scale and reliability.

Chamber

Segment’s Chamber provides elegant secrets management using AWS Parameter Store, optimizing for simplicity and cost-effectiveness. The platform leverages existing AWS infrastructure to minimize complexity while providing essential secret management capabilities.

Cost-effective operations use Parameter Store’s generous free tier to provide secret storage without additional infrastructure or licensing costs. Simple deployment requires minimal operational overhead, making it suitable for teams with limited DevOps resources. Version control integration supports infrastructure as code practices by enabling secret management through familiar development workflows.

Namespace organization with hierarchical parameter structure provides logical organization that scales with application complexity while maintaining simplicity of operation. As a free and open-source tool with only AWS Parameter Store costs, Chamber provides extremely cost-effective secret management.

Chamber serves AWS-native deployments seeking simplicity, teams with cost constraints, and organizations preferring minimal tooling complexity over advanced feature sets.

Top Features to Look for in Secrets Management Tools

Secrets Orchestration and Multi-Source Integration

The most advanced secrets management platforms function as intelligent orchestrators, connecting multiple secret sources and providing unified access patterns. This capability is crucial for organizations with heterogeneous environments where secrets are distributed across multiple systems. Rather than forcing migration to a single secret store, orchestration platforms work with existing infrastructure investments while providing centralized management and consistent access patterns.

Effective orchestration platforms can aggregate secrets from existing enterprise vaults like HashiCorp Vault and CyberArk, cloud-native services including AWS Secrets Manager, Azure Key Vault, and Google Secret Manager, legacy systems with database configuration and file-based secrets, and third-party services like 1Password, Bitwarden, and specialized tools. Pulumi ESC exemplifies this orchestration approach, providing a unified interface across over 20 secret sources while maintaining compatibility with existing infrastructure investments.

This orchestration capability reduces migration risk and preserves existing investments while enabling gradual adoption of modern secret management practices. Organizations can maintain their current secret stores while gaining the benefits of centralized management, consistent access patterns, and advanced features like dynamic credential generation.

Configuration as Code and Environment Management

Modern secrets management extends beyond simple credential storage to comprehensive configuration management. The best platforms support configuration-as-code principles that bring software engineering discipline to environment management and secret distribution.

Version control integration with full change tracking and rollback capabilities ensures that configuration changes undergo the same scrutiny as application code. Hierarchical organization reduces duplication through inheritance and composition, enabling teams to define common configurations once and apply them consistently across multiple environments. Reproducible deployments ensure consistency across environments while code review processes apply software engineering practices to configuration changes. Automated testing capabilities validate configuration changes before deployment, preventing configuration errors that could lead to security vulnerabilities or application failures.

This approach transforms secrets from scattered, manual processes into systematic, auditable infrastructure that follows established software engineering best practices. Teams can apply familiar development workflows to secret management, improving both security and operational efficiency.

Dynamic Credential Generation

Static secrets pose inherent security risks due to their long-lived nature and potential for unauthorized sharing. Dynamic credential generation addresses these challenges by creating credentials on-demand with automatic expiration, significantly reducing the attack surface associated with credential compromise.

Effective dynamic credential systems provide short-lived tokens with automatic expiration and renewal, eliminating the need for manual rotation while ensuring credentials cannot be misused over extended periods. Just-in-time access generates credentials only when needed, reducing the window of opportunity for credential misuse. OIDC integration leverages identity providers for cloud access, enabling seamless authentication without storing long-lived cloud credentials.

Database credentials with automatic rotation for major database systems ensure that application access remains current while removing the operational burden of manual credential management. API keys with configurable lifespans and scope limitations provide fine-grained control over third-party service access. This dynamic approach significantly improves security posture while reducing operational overhead.

Multi-Platform Integration and SDK Support

Your secrets management platform should integrate seamlessly with your existing development and deployment infrastructure, providing native support for the tools and workflows your teams already use. Effective integration reduces friction in adoption while ensuring that secret management becomes a natural part of existing processes rather than an additional burden.

Development integration should include CI/CD platforms like GitHub Actions, GitLab CI, Jenkins, and Azure DevOps, enabling automated secret management as part of deployment workflows. Container orchestration platforms including Kubernetes, Docker Swarm, and Nomad should provide native secret injection capabilities. Infrastructure as Code tools like Terraform, Pulumi, and CloudFormation should support secret management through familiar provisioning workflows. Local development integration should include environment synchronization and IDE integration to provide consistent experiences across development and production environments.

Runtime integration capabilities should extend to application frameworks like Spring Boot, Django, and Express.js, enabling developers to access secrets through familiar programming patterns. Cloud platforms including AWS Lambda, Azure Functions, and Google Cloud Run should provide optimized secret injection with minimal performance impact. Monitoring tools, service meshes, and other infrastructure components should integrate seamlessly to provide comprehensive observability and management capabilities.

Comprehensive Audit and Compliance Capabilities

Security and compliance requirements demand detailed visibility into secret access and management activities. Effective audit capabilities provide the transparency required for security monitoring, compliance reporting, and incident investigation while enabling organizations to demonstrate adherence to regulatory requirements.

Complete audit trails with detailed attribution, timestamps, and context enable security teams to understand exactly who accessed which secrets when and from where. Change tracking for all secret modifications, rotations, and access grants provides visibility into the complete lifecycle of credential management. Compliance reporting supporting SOC 2, ISO 27001, PCI-DSS, and other regulatory frameworks automates the documentation required for compliance audits.

Anomaly detection capabilities identify unusual access patterns that may indicate compromise or policy violations, enabling proactive security response. Integration with SIEM systems provides centralized security monitoring and alerting, ensuring that secret management activities are included in overall security operations. These capabilities transform secret management from a potential compliance liability into a security asset that enhances overall organizational security posture.

Performance and Scalability

Consider both current requirements and future growth when evaluating secrets management platforms. API performance with low latency for high-frequency secret retrieval ensures that secret management doesn’t become a bottleneck in application performance. Geographic distribution supporting global deployments with regional access provides consistent performance regardless of deployment location.

High availability with automatic failover and disaster recovery ensures that secret management doesn’t become a single point of failure in critical applications. Horizontal scaling capabilities handle increased load without performance degradation as organizations grow. Intelligent caching strategies optimize performance while maintaining security, reducing API load while ensuring secrets remain current.

Effective performance and scalability ensure that secret management enhances rather than hinders application performance, enabling organizations to scale their security practices alongside their infrastructure growth.

Try secrets orchestration

Pulumi ESC connects your existing secret stores behind a single interface, adding configuration as code and dynamic, short-lived credentials.

Get started

Key Considerations for Choosing Secrets Management Tools

Orchestration vs. Replacement Strategy

The fundamental choice between orchestration and replacement approaches significantly impacts both implementation complexity and long-term flexibility. Orchestration platforms like Pulumi ESC work with existing infrastructure, providing unified interfaces without requiring complete migration. This approach offers reduced migration risk with gradual adoption, preserved investments in existing secret stores and processes, flexibility to use best-of-breed solutions for different use cases, and simplified operations through centralized management of distributed secrets.

Replacement approaches require migration of all secrets to a single platform. While potentially simpler architecturally, this approach involves higher migration costs and extended transition periods, vendor lock-in that makes future changes more difficult, all-or-nothing adoption requiring complete workflow changes, and potential loss of functionality from specialized existing systems.

Organizations with heterogeneous environments, multiple cloud providers, or significant existing investments in secret management infrastructure typically benefit from orchestration approaches that preserve flexibility while providing centralized management capabilities.

Cloud Strategy Alignment

Your choice of secrets management platform should align with your overall cloud strategy and deployment patterns. Organizations with multi-cloud or hybrid strategies benefit from cloud-agnostic solutions that provide consistent experiences across platforms. Pulumi ESC, HashiCorp Vault, and other orchestration platforms excel in these environments by providing unified interfaces regardless of underlying infrastructure.

Organizations committed to specific cloud platforms can leverage native services like AWS Secrets Manager, Azure Key Vault, and Google Secret Manager for optimal integration and cost efficiency within their chosen ecosystem. These platforms provide deep integration with cloud-native services while optimizing costs through native billing and resource management.

Migration flexibility becomes important as organizational requirements evolve. Orchestration platforms provide natural migration paths between different underlying secret stores as requirements change, while cloud-native solutions may require more significant changes if cloud strategy evolves. Consider solutions that maintain flexibility for future changes while meeting current operational requirements.

Team Structure and Expertise

The technical complexity and operational requirements of different secrets management platforms vary significantly, making team structure and expertise critical factors in platform selection. High-complexity tools like HashiCorp Vault and CyberArk Conjur suit organizations with dedicated security or platform teams capable of managing sophisticated infrastructure. These platforms offer maximum flexibility and features but require significant expertise to implement and operate effectively.

Medium-complexity tools like Pulumi ESC and Doppler balance advanced features with operational simplicity, making them suitable for organizations with strong technical teams that prefer managed services or simplified operations. These platforms provide enterprise-grade capabilities without the operational overhead of self-managed infrastructure.

Low-complexity tools like 1Password and Bitwarden prioritize ease of use over advanced capabilities, making them suitable for organizations with limited technical resources or mixed teams where non-technical users require access to secrets. These platforms excel in environments where adoption and usability take priority over advanced features.

Organizational structure also influences platform choice. Centralized teams can manage sophisticated platforms and enforce consistent policies across the organization, while distributed teams benefit from self-service capabilities and intuitive interfaces that enable independent operation. Mixed environments require tools accessible to both technical and non-technical users, emphasizing usability and clear documentation.

Security and Compliance Requirements

Regulatory and security requirements significantly influence platform selection, with different industries and use cases demanding specific capabilities and certifications. Government and defense organizations may require FIPS 140-2 Level 2 validation provided by platforms like Azure Key Vault and CyberArk, while financial services often need comprehensive audit trails and segregation of duties capabilities.

Healthcare organizations require HIPAA-compliant secret handling and access controls, while European operations must consider GDPR implications for secret storage and processing. Understanding these requirements early in the selection process helps narrow platform choices and avoid costly compliance gaps.

Security model preferences also vary significantly between organizations. Zero-knowledge encryption models where service providers cannot access secrets appeal to security-conscious organizations, while Hardware Security Module support provides the highest levels of cryptographic security for sensitive applications. Network isolation capabilities become important for air-gapped or highly secure environments, while multi-factor authentication integration with existing identity systems ensures consistent security policies.

Cost Optimization

Understanding the total cost of ownership for different secrets management approaches requires consideration of both direct and indirect costs. Per-user pricing models suit teams with predictable user counts but can become expensive as organizations scale. Usage-based pricing scales naturally with actual consumption but requires careful monitoring to avoid unexpected costs. Per-secret pricing works well for applications with many users but few secrets, while enterprise licensing may provide cost-effective solutions for large deployments.

Total cost of ownership extends beyond licensing to include operational overhead for self-hosted versus managed solutions, training and certification requirements for specialized platforms, integration costs including development time and ongoing maintenance, and migration expenses from existing solutions. Managed services may appear more expensive initially but often provide better total cost of ownership when operational overhead is considered.

Consider both current costs and future scaling requirements when evaluating pricing models. Platforms that provide cost-effective entry points while supporting growth can provide better long-term value than solutions optimized only for current requirements.

The Future of Secrets Management: Orchestration and Configuration as Code

The secrets management landscape is undergoing a fundamental shift from simple storage systems to sophisticated orchestration platforms. This evolution reflects the reality of modern infrastructure—applications span multiple clouds, integrate with dozens of services, and require dynamic adaptation to changing requirements.

From Storage to Orchestration

Traditional secret management models rely on centralized vaults where all secrets are stored in a single system, requiring applications to understand specific retrieval mechanisms and access patterns. This approach often creates vendor lock-in and forces organizations to migrate from existing secret stores, disrupting established workflows and creating migration risk.

The emerging orchestration model uses intelligent brokers that aggregate secrets from multiple sources—existing vaults, cloud services, legacy systems—presenting them through unified, consistent interfaces. This orchestration approach provides investment protection by working with existing secret stores, gradual migration without disruptive all-or-nothing transitions, best-of-breed integration leveraging specialized solutions where appropriate, and operational simplification through centralized management of distributed secrets.

Orchestration platforms enable organizations to maintain their current infrastructure investments while gaining the benefits of centralized management, consistent access patterns, and modern capabilities like dynamic credential generation. This approach reduces both risk and cost while providing a clear path toward modern secret management practices.

Configuration as Code Revolution

The convergence of secrets and configuration management represents the next major evolution in infrastructure management. Modern applications require more than isolated credentials—they need comprehensive environment configuration including secrets, feature flags, service endpoints, and deployment parameters. Managing these elements separately creates complexity and increases the risk of configuration drift between environments.

Configuration-as-code principles bring software engineering discipline to environment management by providing version control with complete change history and rollback capabilities, code review processes that ensure changes undergo appropriate scrutiny, automated testing that validates configuration changes before deployment, and reproducible environments that eliminate configuration drift between stages.

Pulumi ESC pioneered this approach, demonstrating how configuration-as-code can systematically address secrets sprawl while improving security, reliability, and developer productivity. This convergence transforms environment management from manual, error-prone processes into systematic, auditable infrastructure that follows established software engineering best practices.

Dynamic Credentials and Zero-Trust Architecture

The shift toward dynamic, short-lived credentials aligns with zero-trust security principles that assume compromise and limit blast radius through minimal access grants. Rather than managing long-lived secrets that require careful rotation and access control, modern platforms generate credentials on-demand with automatic expiration.

This approach provides reduced attack surface through minimal credential lifetime, simplified rotation with automatic credential lifecycle management, enhanced audit trails with precise attribution and timing, and improved compliance through automated policy enforcement. Dynamic credentials eliminate many of the operational challenges associated with traditional secret management while significantly improving security posture.

Organizations adopting zero-trust architectures benefit from secret management platforms that support dynamic credential generation, fine-grained access controls, and comprehensive audit capabilities. These platforms become integral components of zero-trust implementations rather than separate security tools.

AI and Machine Learning Integration

Advanced platforms are incorporating AI and machine learning capabilities to enhance security and operational efficiency. Anomaly detection algorithms identify unusual access patterns that may indicate compromise or policy violations, enabling proactive security response before incidents escalate. Intelligent rotation systems optimize credential lifecycles based on usage patterns and risk assessment, balancing security requirements with operational efficiency.

Automated compliance capabilities ensure policies are consistently applied across all environments while predictive security features identify potential vulnerabilities before they’re exploited. These AI-powered capabilities transform secret management from reactive operations into proactive security systems that enhance overall organizational security posture.

Choosing the Right Path Forward

The secrets management landscape has fundamentally shifted from simple storage to intelligent orchestration. Your choice today will determine not just your current security posture, but your ability to scale and adapt as your infrastructure grows more complex.

The decision framework is clearer than you might expect. If you’re managing multi-cloud environments or have significant existing secret store investments, orchestration platforms like Pulumi ESC provide the most strategic value. They let you modernize your approach without disrupting current workflows, while adding configuration-as-code capabilities and hierarchical environment management that systematically address secrets sprawl.

Organizations committed to maximum flexibility and customization will find HashiCorp Vault’s extensibility unmatched, though it requires substantial technical expertise to operate effectively. Cloud-native teams can leverage AWS Secrets Manager, Azure Key Vault, or Google Secret Manager for seamless integration within their chosen ecosystem, optimizing both cost and operational complexity.

Teams prioritizing rapid adoption and developer experience should consider Doppler, Infisical, or 1Password Secrets Automation. These platforms excel at driving organization-wide adoption through intuitive interfaces while providing essential capabilities without overwhelming complexity.

Here’s the critical insight: secrets management isn’t just a security decision—it directly impacts your development velocity and operational efficiency. Platforms that embrace orchestration, configuration-as-code, and dynamic credentials enable faster development cycles while reducing the operational overhead that comes from managing scattered, manual processes.

The cost of maintaining ad-hoc secrets management extends far beyond security incidents. Developer productivity losses from hunting down credentials, operational complexity from manual rotation processes, and the constant risk of configuration drift between environments create technical debt that compounds over time. The investment in systematic solutions pays dividends immediately through reduced operational overhead and improved security posture.

The industry’s evolution toward zero-trust architectures and configuration-as-code practices isn’t slowing down. Organizations that adopt systematic approaches today position themselves to handle tomorrow’s infrastructure complexity with confidence, while those that delay face increasingly difficult migration challenges as their systems grow more distributed and interdependent.

Ready to revolutionize your secrets and configuration management? Explore Pulumi ESC and discover how secrets orchestration can transform your development workflows while eliminating sprawl across your entire infrastructure.

Announcing Pulumi Identity and Access Management (IAM)

Germán Lena — Mon, 09 Jun 2025 00:00:00 +0000

Cloud development is accelerating at an unprecedented pace, fueled by AI and the relentless drive for innovation. But this incredible speed demands unwavering trust in your security posture. How do you empower teams to deploy rapidly and frequently without opening doors to risk or violating compliance mandates? Today, we’re thrilled to answer that critical challenge by introducing Pulumi Identity and Access Management (IAM) – a foundational new capability designed to embed robust, granular security directly into your cloud development lifecycle, enabling you to innovate both quickly and safely with Pulumi. Pulumi IAM provides the unified framework for fine-grained authorization needed to confidently manage modern cloud infrastructure and applications across the entire Pulumi Cloud platform.

Our Vision for Pulumi IAM

Pulumi IAM is a foundational investment, delivering enterprise-grade access management through a phased approach. Today’s release marks the beginning, with much more planned:

Phase 1: Granular Access Tokens & Custom Roles (Available Today)
- Define custom, reusable Permissions with fine-grained scopes (e.g., stack:delete only).
- Create Custom Roles by combining Permissions with specific Pulumi Entities (Stacks, Environments, etc.).
- Generate Organization Access Tokens scoped precisely to these Custom Roles, perfect for secure automation.
Phase 2: User & Team Role Assignment (Coming Soon)
- Leverage OIDC configuration to dynamically assume Custom Roles for secure, tokenless authentication from CI/CD systems like GitHub Actions, GitLab, and more.
- Assign these powerful Custom Roles directly to individual users and teams within your Pulumi organization.
- Implement a complete overhaul of user and team access management, moving beyond the basic Admin/Member distinctions, and enabling reusability of custom building blocks permissions and roles that work for your organization
Phase 3: Advanced Authorization & Scalability (Future Release)
- Introduce Attribute-Based Access Control (ABAC), allowing policies based on tags or other attributes of Pulumi Entities (e.g., “grant ‘dev-role’ access to all stacks tagged ’env:dev’”).
- Enable the creation of Custom RBAC Policies with conditional logic for highly specific access scenarios and reuse them
- Provide mechanisms to manage permissions across hundreds or thousands of Pulumi Entities efficiently.

A Foundation for Zero Trust & Unified Security

Pulumi IAM isn’t just another feature; it’s a foundational pillar underpinning security across the entire Pulumi Cloud ecosystem, enabling organizations to implement Zero Trust principles for their infrastructure management. Modern security models assume breaches will happen and demand rigorous verification for every access request. Static, organization-wide roles no longer suffice where separation of duties, least privilege, and compliance are paramount.

Pulumi IAM addresses these challenges by providing:

Least Privilege Enforcement: Define precisely who can do what on which specific resources, minimizing the potential impact if credentials or accounts are compromised. This is core to Zero Trust – grant only the minimum necessary access, verified at the point of action.
Granular Control Across Pulumi:
- Infrastructure as Code (IaC): Apply fine-grained controls over Pulumi Stacks
- Secrets Management: Define specific access levels for Pulumi ESC Environments.
- Insights: Manage permissions for Pulumi Insights account settings.
Secure Automation: Provide secure, least-privilege tokens and OIDC integration for CI/CD pipelines and automation, drastically reducing the risk associated with over-privileged service accounts.
Unified, Scalable Governance: Establish a consistent authorization model that simplifies administration and scales from small teams to complex enterprise environments, ensuring security doesn’t hinder velocity.

Launching Today: Granular Access Tokens via Custom Roles

This vision begins today with the initial phase of Pulumi IAM, enabling you to define Custom Roles built from fine-grained Permissions and apply them specifically to Organization Access Tokens. This initial step provides immediate, significant security benefits, particularly for automation:

True Least Privilege for CI/CD: Scope pipeline tokens to only the actions (e.g., pulumi up) and Entities (e.g., stack: myapp-prod) they absolutely need.
Reduced Blast Radius: If a scoped token is compromised, the potential damage is limited strictly to the permissions defined in its associated role.
Enhanced Compliance: Demonstrate precise control over programmatic access to auditors.

How to Get Started with Granular Access Tokens

Configuring and using Custom Roles for scoped tokens is done via the Pulumi Cloud console:

1. Define a Custom Permission (Optional)

Create reusable sets of fine-grained scopes.

As an admin, navigate to Organization Settings -> Roles -> Permissions
Follow instructions for creating a custom permission.

2. Create a Custom Role

Combine permissions with specific resources.

As an admin, navigate to Organization Settings -> Roles
Follow instructions for creating a custom role.

3. Generate a Scoped Organization Access Token

Generate an organization access token with narrowed scope.

As an admin, navigate to Organization Settings -> Access Tokens -> Organization Access Tokens.*
Click “Create token”. Provide a description. Select your Custom Role from the “Role” dropdown. Generate the token.

New Scenarios Unlocked Today

This release immediately enables more secure and compliant workflows:

Secure Multi-Environment CI/CD: A single pipeline can use different tokens based on the target environment (dev, staging, prod), each assuming a role with appropriately restricted permissions (e.g., read-only for prod dependencies, write for the target stack).
Restricted Operational Scripts: An automation script designed only to read audit logs can use a token tied to a role granting only audit_log:read permission, preventing accidental or malicious modifications.
Safer ChatOps & Tooling: Integrations like ChatOps bots can operate with tokens scoped down to only necessary actions (e.g., triggering a pulumi preview on specific stacks).

Available Today: Custom Permissions, Custom Roles, and the ability to scope Organization Access Tokens using these roles, is available now for customers on the Pulumi Enterprise and Pulumi Business Critical tiers. Explore these features in your Pulumi Cloud organization settings!

Conclusion: Building a More Secure Future

Pulumi Identity and Access Management (IAM) represents a fundamental advancement in securing cloud development lifecycles managed by Pulumi, providing the controls needed to confidently embrace speed and scale. Today’s launch of Granular Access Tokens via Custom Roles provides immediate security improvements for automation and programmatic access, laying the vital groundwork for our comprehensive IAM vision rooted in Zero Trust principles.

This empowers platform and security teams with the fine-grained control needed to implement least privilege, enhance compliance, and scale Pulumi usage securely without sacrificing velocity.

We encourage our Enterprise and Business Critical customers to explore Custom Roles and Granular Access Tokens today. Dive into the documentation and start building roles tailored to your security requirements. We welcome your feedback and feature requests in our GitHub repository. Join us as we build a more secure foundation for cloud engineering!

Learn More

Learn more about Pulumi Cloud’s new IAM & RBAC features:

Announcing Snowflake Dynamic and Rotated Credentials with Pulumi ESC

Claire Gaestel — Thu, 01 May 2025 00:00:00 +0000

Snowflake is the data cloud powerhouse for countless businesses, critical for everything from customer dashboards to billing pipelines. The stakes are immense: this data must be strictly secured and always available. But managing this with static credentials or manual key rotation creates persistent security vulnerabilities and introduces operational instability, risking disruptions during clumsy updates. Pulumi ESC eliminates this dilemma with two purpose-built Snowflake integrations:

snowflake-login: Provides dynamic, short-lived OIDC tokens for temporary authentication to Snowflake.
snowflake-user: Automates the rotation of RSA keypair secrets for Snowflake users, essential for secure key-pair authentication.

Pulumi ESC allows you to securely manage and consume these Snowflake credentials in your applications, development workflows through ESC SDK, Pulumi CLI, Kubernetes integrations, and more!

Use the snowflake-login provider when you need temporary, just-in-time credentials to interact with Snowflake, such as from local development environments, CI/CD pipelines, or scripts. It leverages OIDC to mint short-lived access tokens, eliminating the need to store static tokens.

This approach significantly reduces the attack surface by removing long-lived credentials required for authentication.

Setup:

Configure OIDC in Snowflake: Create a Security Integration in Snowflake to trust the Pulumi OIDC provider (https://api.pulumi.com/oidc), map the appropriate user claims and create a Snowflake user. Check out the docs] for more details
Configure ESC Environment: Define the snowflake-login provider in your ESC environment, specifying the Snowflake account and the user configured for OIDC.

# my-org/logins/snowflake
values:
 snowflake:
 login:
 fn::open::snowflake-login:
 oidc:
 account: myorganization-account
 user: ESC_LOGIN_USER
 role: ESC_ROLE  # Optional

When this environment is opened, ESC securely handles the OIDC flow and makes the temporary token available under snowflake.login.token.

snowflake-user: Automated Rotation for RSA Key Secrets

For applications or services using Snowflake’s key-pair authentication, maintaining the security of those RSA keys is crucial. The snowflake-user rotator automates the lifecycle management of these keys as rotated secrets within ESC. This rotater uses the same two-secret rotation strategy we use in rest of ESC’s rotated secrets providers. This eliminates manual rotation toil and ensures keys are regularly refreshed according to policy, enhancing the security of the user secret itself. Check out the rotated secrets blog post to learn more about its benefits.

Setup:

Prepare Snowflake:
- Create the target user (e.g., MY_APP_SNOWFLAKE_USER) whose keys need rotation.
- Set up a dedicated rotation role with OWNERSHIP permission over the users you wish to rotate, so it can modify their keys.
- Create a service user using the rotation role, and setup OIDC to allow ESC to assume the role for rotation.
Rotation Environment: Define the rotation, importing the managing credentials and specifying the target user.

# Environment: my-org/rotators/snowflake-app-key
values:
 rotatedKey:
 fn::rotate::snowflake-user:
 inputs:
 login: ${environments.logins.snowflake.snowflake.login} #reference credentials created using `snowflake-login`
 targetUser: ESC_ROTATION_DEMO_USER # User whose keys rotate

After setup, use the triple-dot menu -> “Rotate Secrets” in the Pulumi Cloud UI and ensure rotation happens. Once you rotate a couple of time and ensured everything works well, navigate to the “Secret Rotation” tab for your Rotation environment in the Pulumi Cloud console and define the rotation schedule.

Feature	`snowflake-login` (Dynamic OIDC Login)	`snowflake-user` (Rotated Secrets)
Credential Type	Short-lived OIDC access token	Long-lived RSA Private Key secret (managed by ESC) + Public Keys (in Snowflake)
Primary Use Case	Temporary authentication to Snowflake (CLI, scripts, dev, CI/CD).	Key-pair authentication for long-lived applications/services connecting to Snowflake.
Ideal Scenario	Short-lived tasks; frequent, temporary access needs.	Applications needing persistent connections; keys required at deploy time; apps sensitive to auth blips.
Lifecycle	Generated on-demand, expires quickly (minutes/hours).	Rotated automatically on a schedule (e.g., days/months).
Connection Impact	Requires re-authentication or new token fetch when token expires.	Rotation is typically seamless; two-key strategy avoids connection drops if apps update keys promptly.
Benefit Focus	Eliminates static authentication tokens/credentials.	Automates lifecycle management of user key-pair secrets, ensures key freshness.

Conclusion

Pulumi ESC’s new Snowflake integrations provide robust solutions for modern credential management challenges. Use snowflake-login for secure, temporary OIDC-based access and snowflake-user to automate the rotation of critical RSA key secrets. By adopting these capabilities, you can significantly enhance your Snowflake security posture, reduce operational overhead, and simplify compliance.

Explore the detailed documentation for snowflake-login and snowflake-user, and see how Pulumi ESC can centralize and secure your configuration and secrets across all your environments. Join the conversation in the Pulumi Community Slack or our GitHub repository.

Introducing Automated Database Credential Rotation for PostgreSQL and MySQL in Pulumi ESC

Sean Yeh — Wed, 30 Apr 2025 00:00:00 +0000

Securing access to critical data stores is paramount in today’s cloud-native world. Yet, managing database credentials often involves static, long-lived passwords – a significant security blind spot. These static secrets, frequently embedded in application configurations or accessible to multiple team members, represent a prime target for attackers. Manually rotating these credentials is a cumbersome, error-prone task that’s often neglected, leaving databases vulnerable for extended periods. Building on our commitment to robust secrets management, we are excited to launch Automated Database Credential Rotation for PostgreSQL and MySQL in Pulumi ESC!

The Persistent Challenge of Static Database Secrets

Relying on static database credentials introduces substantial risks, which automated rotation helps mitigate:

Security Vulnerabilities & Exposure: Static credentials, if compromised through leaks, phishing, or unauthorized access, provide long-term access to attackers. Automated rotation significantly shrinks this window of opportunity.
Operational & Compliance Burdens: Manually rotating database credentials is complex, error-prone, and requires careful coordination to avoid downtime. These manual rotations are hard to audit and demonstrate compliance with regulations like SOC 2, GDPR, or HIPAA (which often mandate rotation).
Network Complexity & The DIY Burden: Databases in private networks (like AWS VPCs) require secure access for rotation. We provide an open-source Lambda connector for AWS VPCs. While homegrown solutions can bridge the network gap, building and maintaining the actual rotation logic (state, error handling, two-secret strategy) is complex. Crucially, DIY approaches typically lack integrated revision history, centralized auditing, and unified management provided by platforms like Pulumi ESC, increasing operational overhead.
Tooling, Ecosystem, and Cloud-Native Connectivity: Despite cloud IAM authentication options, many standard tools (GUIs, BI, ETL), legacy apps, and even certain cloud-native configurations (multi-cloud, specific Kubernetes setups) still rely on username/password. Direct IAM integration isn’t always feasible or practical, making automated rotation of traditional credentials essential for broad compatibility and security.

Pulumi ESC addresses these challenges by automating the secrets rotation process, including connectivity options for databases in private AWS networks using our Lambda connector, and allows you to consume them in your applications through ESC’s various developer-friendly methods including ESC SDK, Pulumi CLI, Kubernetes External Secrets Operator, CSI Driver, etc.

Introducing ESC Rotated Secrets for Databases

Pulumi ESC’s Rotated Secrets capability now extends to your critical databases, bringing automation and enhanced security to credential lifecycle management. We are adding support for the following databases and connectivity options:

Database	Publicly Accessible	AWS VPCs	Azure VNETs	GCloud VPCs
PostgreSQL	✅	✅ (via Lambda)	Planned (upvote issue)	Planned (upvote issue)
MySQL	✅	✅ (via Lambda)	Planned (upvote issue)	Planned (upvote issue)

Key benefits include:

Automated Rotation Schedules: Effortlessly define rotation frequencies (daily, weekly, monthly) that align with your security policies, eliminating manual toil and ensuring consistency.
On-Demand Rotation: Instantly rotate credentials in response to security events, personnel changes, or policy updates with a single click or command.
Seamless ESC Environment Integration: Manage rotated database credentials alongside all your other application configurations within ESC Environments, leveraging composition and environment inheritance.
Two-Secret Strategy: ESC maintains both the current and previous valid credentials simultaneously. This prevents application downtime during rotation, as instances can gracefully transition to the new credential.
Auditing and Tracking: Maintain a complete audit trail of credential rotations, including who initiated the rotation and when, simplifying compliance and governance.
Admin and Creator Control: Securely configure rotation using privileged database “managing user” credentials stored within ESC, keeping them separate from the rotated credentials consumed by applications.
Configure Webhooks for Automation: Trigger notifications, CI/CD pipelines (e.g., Pulumi Deployments to update application stacks), or custom workflows upon successful rotation, keeping dependent systems synchronized.

The following setup guide provides a detailed walkthrough for rotating PostgreSQL credentials for an AWS RDS instance hosted within a private VPC, which requires the AWS Lambda-based connector. For publicly accessible databases (PostgreSQL or MySQL), you can use the simpler Direct Connect method documented on the respective integration pages. Please refer to the MySQL documentation page for specific instructions on MySQL rotation (both Direct Connect and Lambda Connector).

How to Set Up and Use Rotation for AWS RDS PostgreSQL running in VPC

Setting up rotation for AWS RDS or Aurora PostgreSQL databases running inside an AWS VPC involves creating the required users within your database, deploying the AWS Lambda connector infrastructure via a Pulumi template, configuring ESC environments, and scheduling the rotation.

Understanding the AWS Lambda Connector: The open-source ESC Lambda connector acts as a secure proxy for database credential rotation within your VPC. It enables Pulumi ESC to safely rotate database credentials without needing direct network access from ESC to your database and without exposing your database to external networks. This connector is currently required to rotate credentials for PostgreSQL or MySQL databases hosted within an AWS VPC. Note: The AWS Lambda Connector is only available to Enterprise and Business Critical customers.

Step 1: Prepare Your PostgreSQL Database Users

You’ll need two types of users configured, resulting in three specific database user accounts:

Managing User: A privileged user that ESC will use (via the Lambda connector) to perform the password rotations. This user needs permissions to change the passwords of the application users.
Application User 1 and Application User 2: The two application users whose passwords ESC will manage. ESC rotates between these two users, ensuring two users are always active as applications gradually phase out the previous user credentials.

Connect to the database as a superuser and run the following SQL commands:

Create the application users whose passwords you want ESC to rotate automatically:

```sql
CREATE USER user1 WITH PASSWORD 'initial_password';
GRANT SELECT, INSERT, UPDATE ON yourDatabase TO user1;
CREATE USER user2 WITH PASSWORD 'initial_password';
GRANT SELECT, INSERT, UPDATE ON yourDatabase TO user2;
```

Create the designated “managing user” account with privileges to modify passwords for the application users:

```sql
CREATE USER managing_user WITH PASSWORD 'manager_password';
ALTER USER managing_user WITH CREATEROLE;
GRANT user1 TO managing_user WITH ADMIN OPTION;
GRANT user2 TO managing_user WITH ADMIN OPTION;
```

Step 2: Bootstrap Rotation Infrastructure (Lambda Connector) and Create ESC Environments

To rotate credentials for PostgreSQL databases within AWS VPCs securely, you need to configure an AWS Lambda function (the connector) within your VPC and create the corresponding ESC environments.

If your PostgreSQL instance is running on AWS RDS, we recommend using our Pulumi Template to simplify the process of creating the Lambda function, associated AWS IAM Roles, Security Groups, and the basic ESC environment structure. You can start it from the Pulumi console or by running the following command in your terminal:

Run pulumi new esc-connector-lambda-typescript in your terminal.
This command initializes a new Pulumi project designed to provision the necessary AWS Lambda connector resources and create the ESC environments.
Follow the prompts. You will first provide a Pulumi project name (e.g., my-db-rotation-infra). Then, you will be prompted for configuration values like:
- Your target AWS Region (aws:region)
- The RDS Database Identifier (rdsDbIdentifier)
- The specific name for the ESC rotation environment (e.g., postgresrotation/Rotator). This environment will contain the two application users’ credentials and configuration for rotation.
- The specific name for the ESC managing credentials environment (e.g., postgresrotation/ManagingCreds).
Run pulumi up to deploy the AWS resources and create the ESC environments using the names you provided.

Step 3: Configure ESC Environments and Test

The Pulumi template creates two ESC environments with most of the configuration pre-filled. You need to update them with your specific PostgreSQL user credentials and database name.

Managing Credentials Environment (e.g., postgresrotation/ManagingCreds): Navigate to this environment in the Pulumi Cloud console. Update the managing user’s username and password. The awsLogin section, which provides credentials for the Lambda connector to assume, should be pre-filled by the template.

# postgresrotation/ManagingCreds
values:
 managingUser:
 username: managing_user # Replace with your managing user's name
 password:
 # Securely replace the placeholder with your managing user's password using fn::secret
 fn::secret:
 ciphertext: ZXNjeA... # Replace with your encrypted password
 awsLogin:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::616138583583:role/PulumiEscSecretRotatorLambda-InvocationRole-7e4646f # Example Role ARN populated by template
 sessionName: pulumi-esc-secret-rotator-${esc.context.environment} # Example Session Name populated by template

Rotation Environment (e.g., <project-name>/Rotator): Navigate to this environment. Most fields related to the Lambda connector (Lambda ARN, host, port, references) are pre-filled by the template outputs. You must update the database name and the application usernames mapping. Note the connector.awsLambda block indicating the use of the Lambda connector.

# postgresrotation/Rotator
values:
 dbRotator:
 fn::rotate::postgres:
 inputs:
 database:
 # Configure the AWS Lambda connector for private VPC access
 connector:
 awsLambda:
 login: ${environments.postgresrotation.ManagingCreds.awsLogin}
 lambdaArn: arn:aws:lambda:us-west-2:616138583583:function:PulumiEscSecretRotatorLambda-Function-d9932fe # Example Lambda ARN populated by template
 # Database connection details (used by the Lambda connector)
 database: mydb # Replace with your DB name
 host: tf-20250428212643894700000001.cluster-chuqccm8uxqx.us-west-2.rds.amazonaws.com # Example Host populated by template
 port: 5432 # Example Port populated by template
 managingUser: ${environments.postgresrotation.ManagingCreds.managingUser}
 # List of application users whose passwords will be rotated
 rotateUsers:
 username1: user1 # Replace DB username if different from logical name
 username2: user2 # Replace DB username if different from logical name

 # state section is automatically managed by ESC after first rotation
 # Do not provide state unless migrating existing, known credentials.
 # state:
 # current:
 # password:
 # fn::secret:
 # ciphertext: ZXNjeAAAA...
 # username: user1
 # previous:
 # password:
 # fn::secret:
 # ciphertext: ZXNje21AA...
 # username: user2

Test with Manual Rotation:
- Use the triple-dot menu -> “Rotate Secrets” in the Pulumi Cloud UI for the Rotator environment.
- Alternatively, use the Pulumi CLI: pulumi env rotate <your-org>/<project-name>/Rotator> (e.g., pulumi env rotate myorg/postgresrotation/Rotator). Remember to replace <your-org> and <project-name>.
- Perform a rotation. If you didn’t provide the state block, this first rotation generates a new password for the user designated as current (e.g., user1 associated with username1) and updates it in the database via the Lambda connector. The dbRotator value in ESC will now show this user/password under current, and the state block will be populated.
- Perform a second rotation. Observe how the previous current user (user1/username1) and its password become the previous entry in the ESC value. The system now rotates the password for the other user (user2/username2), makes it the new current, and updates the database accordingly. Subsequent rotations cycle between user1 and user2.
- Check the Environment Revision History in the Pulumi Cloud console to track the changes to the current and previous state fields after each rotation.

Step 4: Schedule Automated Rotations

Once you are satisfied that manual rotations via the Lambda connector are working correctly, navigate to the “Secret Rotation” tab for your Rotator environment (e.g., postgresrotation/Rotator) in the Pulumi Cloud console.
Define your desired automated rotation schedule (e.g., every 30 days). ESC will automatically trigger rotations according to this schedule, ensuring your credentials stay fresh without manual intervention.

What’s Next?

This launch significantly enhances database security posture through automation for PostgreSQL and MySQL, but we’re just getting started. We plan to expand our rotated secrets capabilities:

More Databases: Support for MongoDB, Microsoft SQL Server, Oracle, and others.
Databases in Private Networks on Other Clouds: Rotation solutions for databases hosted in private networks within Azure, Google Cloud, and more.

Have a specific database or integration in mind? Upvote existing issues or open a new request in our GitHub repository!

Conclusion

Pulumi ESC’s Automated Database Credential Rotation drastically simplifies a critical security task. By automating the rotation of PostgreSQL and MySQL credentials, ESC reduces operational overhead, minimizes the risk associated with static secrets, and helps you meet compliance requirements more easily. Whether your databases are publicly accessible (using Direct Connect) or in private AWS networks (using the Lambda connector), ESC provides a unified way to manage their lifecycle securely.

We encourage you to explore the Pulumi ESC documentation and try out rotated database secrets for PostgreSQL and MySQL. Your feedback is invaluable – please share your experiences and suggestions on our GitHub repository. Secure your databases with automated rotation and build with confidence!

Why Every Cloud Engineer Needs Pulumi ESC for Secrets Management

Sara Huddleston — Mon, 17 Mar 2025 07:28:56 +0000

Managing secrets is one of the most critical responsibilities in cloud engineering. Secrets like API keys, database credentials, and encryption tokens are the backbone of secure and seamless cloud operations. Yet they are so often an afterthought. They get replicated across cloud-specific secrets managers and stuffed in GitHub secrets, compromising security for the sake of simplicity. ¿Por que no los dos? Why can’t secrets management be secure and simple?

Enter Pulumi ESC (Environments, Secrets, and Configuration)—a breakthrough in taming secrets sprawl and streamlining configuration management across infrastructure. Let’s explore why Pulumi ESC is a necessity for cloud engineers, helping make secrets management secure while keeping it simple.

The Challenge of Secrets Management in Modern Cloud Environments

Secrets management has changed significantly over the past decade. Gone are the days when secrets could be hardcoded, manually maintained in static configuration files, or left floating around in plaintext on dev machines. Nowadays, cloud environments demand a higher level of sophistication:

Distributed Systems: Even in multi-cloud and hybrid setups, secrets must be accessible across varied platforms without exposing vulnerabilities.
Dynamic Infrastructure: Kubernetes, serverless architectures, and ephemeral environments must have secrets that adapt dynamically.
Security Risks: Hardcoded or poorly managed secrets can lead to catastrophic breaches, costing companies millions in data recovery and compliance penalties.
Operational Burdens: Manual secrets management is error-prone, inefficient, unscalable, and needs to align with DevOps and DevSecOps best practices.

Pulumi ESC addresses these issues head-on, redefining how cloud engineers manage secrets efficiently, securely, and at scale.

Jk Jensen, Team Lead at Mysten Labs, said:

“Pulumi ESC has been a lifesaver for us. It’s nice to throw everything behind an ESC environment and eliminate one-off granting/IAM permissions and other issues related to static credentials. It gives us peace of mind knowing that we can grant permissions quickly and revoke easily limiting blast radius for any access.”

What is Pulumi ESC?

Pulumi ESC simplifies how organizations manage secrets and configurations. It is designed to secure sensitive configurations across modern cloud environments and supports seamless integration, enabling engineers to:

Access, share, and manage secrets, passwords, API keys, and configuration settings like network and deployment options.
Synchronize secrets and configuration from any secrets manager to any app, tool, or CI/CD platform.
Access secrets securely through CLI, API, Kubernetes operator, the Pulumi Cloud UI, and in-code with Typescript/Javascript, Python, and Go SDKs.
Connect to cloud providers and secrets stores via OIDC to generate dynamic, short-lived, auto-expiring credentials.
Set role-based access controls (RBAC), making securing secrets and configurations easy by assigning permissions to users based on their roles.

Whether integrated with Pulumi’s Infrastructure as Code (IaC) platform or used as a standalone service, Pulumi ESC enables cloud engineers to streamline secrets management with centralized control.

Key Features of Pulumi ESC

1. Seamless Integration with External Platforms

Pulumi ESC integrates with popular secrets providers, including AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, 1Password, and HashiCorp Vault, making it adaptable for multi-cloud and hybrid cloud architectures.

2. Dynamic Secrets Synchronization with ESO

Partnering with the External Secrets Operator, Pulumi ESC synchronizes secrets securely into Kubernetes clusters. This eliminates hardcoding secrets into manifests or relying on unsecured manual processes.

3. Safely Roll Back to a Previous Version

Pulumi ESC Versioning gives you unprecedented control over your secrets and configuration. Every change is captured in an immutable revision history, allowing you to audit modifications, compare versions, and safely roll back.

4. Secure by Design

Pulumi ESC follows a “secure by default” model, employing encryption, access control, and detailed audit trails. Engineers can meet compliance regulations effortlessly while gaining full visibility into secret access patterns.

5. Automated Rotation and Expiry

Pulumi ESC minimizes security risks by automating the rotation of secrets. This feature aligns secrets management with CI/CD processes for cloud engineers focused on DevOps, ensuring credentials remain valid only when needed.

6. Configuration-as-Code, Automation, and Integration Everywhere

Pulumi ESC embraces an “as-code” approach, enabling configuration and secrets management using TypeScript, JavaScript, Go, Python, or YAML. The ’esc’ CLI and API support automation in CI/CD environments, reducing credential duplication and ensuring a single source of truth.

7. Dev Tools Integrations

Pulumi ESC’s metadata and support for popular configuration formats enable seamless integration with tools like Direnv, Docker, and GitHub, allowing easy management of environment variables, secrets, and configurations.

8. Infrastructure Tools Integrations

Pulumi ESC extends its capabilities beyond Pulumi IaC by integrating with other infrastructure tools such as Cloudflare, Terraform, and OpenTofu. These integrations enable seamless provisioning of cloud credentials and input variables directly from ESC environments.

Why Cloud Engineers Need Pulumi ESC

Enhanced Security and Compliance

Cloud engineers often have to balance security and operations. Pulumi ESC centralizes secrets in a secure vault, ensuring encrypted storage, access control, and audit visibility. It also complies with industry standards like SOC 2, PCI-DSS, and MTCS.

Streamlined Multi-Cloud Management

Managing secrets across AWS, Azure, and GCP can quickly become chaotic. Pulumi ESC’s cross-platform integration simplifies this process, ensuring engineers can manage and sync secrets from a single interface.

Simplified Kubernetes Secrets

Kubernetes’ default secrets offer limited security and scalability. Pulumi ESC with ESO and Secrets Store CSI Driver overcomes these limitations by securely synchronizing and managing secrets within Kubernetes clusters. This prevents unauthorized access and provides automated updates.

Zero Downtime Through Automation

Manual secrets management often leads to errors such as expired credentials or outdated tokens. Pulumi ESC automates the entire lifecycle of secrets—creation, rotation, replication, and expiry—guaranteeing uninterrupted services.

Developer-Friendly Workflows

Tools should make engineers’ lives easier, not harder. Pulumi ESC’s CLI, SDKs, and API provide intuitive ways to integrate into existing workflows. For cloud engineers leveraging Infrastructure as Code with Pulumi, managing secrets alongside the stack becomes effortless.

Thomas Meckel, Senior Cloud Architect at Ophios GmbH, explains:

“Pulumi’s built-in encryption of secrets sets it apart as a leader in secure infrastructure management. Unlike other tools that leave sensitive data exposed or require complex configurations, Pulumi ensures secrets are encrypted by default, leveraging flexible options like Azure Key Vault or AWS KMS. This eliminates the risk of plaintext exposure and simplifies secure deployments. For any organization serious about cloud security, Pulumi’s approach to secrets management is a critical differentiator, combining ease of use with robust protection against breaches.”

Kubernetes Secrets Management

For secrets management in Kubernetes environments, Pulumi ESC becomes even more powerful when paired with External Secrets Operator (ESO) or the Secrets Store CSI Driver. Here’s how it works:

Using Pulumi ESC with External Secrets Operator

External Secrets Operator is an open-source Kubernetes operator that syncs secrets from external providers (like Pulumi ESC) into Kubernetes as native secrets.

Centralized Storage: Store secrets securely in Pulumi ESC and synchronize them with Kubernetes across multiple clusters.
Dynamic Updates: Whenever a secret is updated in ESC, ESO automatically replicates the changes into Kubernetes, eliminating manual intervention.
Granular Management: Define which secrets are synchronized and to which namespaces, ensuring tight access controls and minimizing risk.
Automated Secret Rotation: By leveraging ESC’s rotation capabilities, ESO ensures Kubernetes receives refreshed credentials without requiring downtime.

Learn how to manage Kubernetes secrets with Pulumi ESC and External Secrets Operator: The Perfect Solution for Today’s Cloud-Native Secret Management.

Using Pulumi ESC with Secrets Store CSI Driver

The Secret Store CSI Driver mounts secrets, certificates, and keys from external stores into Kubernetes pods as volumes.

Automated Secrets Injection: Secrets Store CSI Driver enables Pulumi ESC to automatically inject secrets into Kubernetes pods as mounted volumes or environment variables. This reduces the manual overhead of managing secrets directly in Kubernetes.
Secure and Dynamic Secret Access: By leveraging Pulumi ESC, Kubernetes applications can securely fetch secrets from external providers, ensuring dynamic access without exposing credentials in pod specifications.
Streamlined Operations: This integration simplifies the process of synchronizing secrets from Pulumi ESC with Kubernetes-native constructs. The automation reduces errors and boosts efficiency.

Learn how to Master Kubernetes Secrets with Pulumi ESC + Secrets Store CSI Driver.

The Future of Secrets Management with Pulumi ESC

Pulumi ESC is more than just a secrets management tool — it’s the foundation of secure, scalable, and agile cloud operations. For cloud engineers grappling with multi-cloud complexity, Kubernetes deployments, and compliance concerns, it provides a streamlined, automated approach that enhances efficiency and security.

With rising cyber threats and stricter data regulations, adopting tools like Pulumi ESC is no longer optional. It’s a competitive advantage for cloud engineers and organizations aiming to lead in today’s cloud-native world.

Next Steps to Secure Your Secrets

Secrets management doesn’t have to be a headache! With Pulumi ESC, you can safeguard your infrastructure, ensure compliance, and eliminate manual errors—all while enhancing security and agility.

Get started with Pulumi ESC today and experience the future of secrets management. Take control of your cloud environment—securely and effortlessly.

Integrating DevOps and Security in Platform Engineering

Sara Huddleston — Wed, 11 Dec 2024 07:41:06 +0000

Platform engineering has become essential for mid-to-large organizations, moving beyond a DevOps trend. Gartner predicts that by 2026, 80% of software companies will have internal platform services to streamline development. The goal is to empower developers with self-service tools while maintaining security, compliance, and reliability through DevSecOps practices.

At PulumiUP Europe 2024, experts shared insights on aligning DevOps with security to build scalable, secure platforms:

Jess Mink, Sr. Director of Platform Engineering at Honeycomb
Kief Morris, Global Head of Infrastructure Engineering at ThoughtWorks
Lindsay Jack, VP of Engineering & Security at Snyk
Nariman Aga-Tagiyev, Application Security Architect at WiseFrog Security
Komal Ali, Engineering Manager at Pulumi

The panel discussed key strategies, challenges, and pillars of successful platform engineering.

The Core Pillars of Platform Engineering

Platform engineering teams comprise multiple professionals with many responsibilities and focus areas. According to our panel of experts, the core pillars of platform engineering include:

Developer Experience (DevEx): Provide developers with the tools, frameworks, and abstractions they need to be productive and proactive without getting stuck in infrastructure or operational concerns.
Reliability and Scalability: Ensure that the platform and infrastructure can support the organization’s needs, with the ability to scale up or down as required.
Security and Compliance: Embed robust, accessible security and compliance frameworks into the development lifecycle while making it easy for developers to adhere to these policies.
Automation and Tooling: Leverage Infrastructure as Code (IaC) and automation to enforce standardized processes and consistency and reduce cognitive load and manual effort.
Observability and Monitoring: Provide visibility into the platform’s health and performance, delivering actionable insights that allow teams to identify and resolve issues quickly.

These pillars work together to create a platform that empowers developers to innovate and deliver value to the organization and customers while maintaining the necessary controls and safeguards.

As Jess Mink, Director of Platform Engineering at Honeycomb, explains:

“The goal of platform engineering is to help the company run smoother and faster and unlock things people didn’t know were possible […] We tend to focus on tools and software, but it’s really about people, processes, and tools. If you consider this, platforms are responsible for social and technical support across the organization. A common pitfall is building tools no one uses because you didn’t meet people where they are.”

Aligning DevOps and Security for Secure Platform Engineering

Integrating security into platform engineering ensures it becomes a proactive part of the development lifecycle and ensures that “DevSecOps” is not an afterthought but a core consideration.

Key best practices shared by the panel include:

Shift Left Security

One of the fundamental principles of DevSecOps is to “shift left” - that is, to integrate security earlier in the development process rather than waiting until the end. This can involve activities such as:

Incorporating security requirements and threat modeling into the initial design phase
Automating security scans and tests as part of the continuous integration (CI) pipeline
Providing developers with secure coding guidelines and tools to identify and remediate vulnerabilities

By addressing security concerns upfront, organizations can reduce the time and cost of remediating issues later in the development lifecycle.

Embrace Automation and Standardization

Consistency is key. Platform engineering teams should leverage automation and built-in safeguards and security processes. This may include:

Automating the provisioning of secure infrastructure and application environments
Implementing Infrastructure as Code (IaC) to define and manage infrastructure in a declarative, version-controlled manner
Standardizing security controls, policies, and configurations across the platform

By automating these tasks, platform engineering teams can reduce the risk of human error, improve visibility and auditability, and free up developers to focus on building features rather than managing infrastructure.

Prioritize Observability and Monitoring

Effective security requires visibility into the health and performance of the platform. Platform engineering teams should invest in robust observability and monitoring solutions to:

Detect and respond to security incidents and anomalies in real-time
Gain insights into the behavior and usage patterns of the platform
Identify and address vulnerabilities or misconfigurations proactively

By integrating security-focused monitoring and alerting into the platform, organizations can quickly identify and mitigate threats while also providing developers with the necessary context to understand and address security-related issues.

Foster a Culture of Collaboration

Platform engineering is often referred to as being the practical application of DevOps practices. Integrating security practices often requires a cultural shift towards collaboration and shared responsibility between all teams in development, operations, and security, thus the name DevSecOps. Platform engineering teams can facilitate this by:

Treating Platforms as Products
Involving security stakeholders in the design and planning of platform initiatives
Providing security training and education to developers to empower them to make informed decisions
Establishing communication channels and feedback loops between teams

As Kief Morris, Global Head of Infrastructure Engineering at ThoughtWorks, explains:

“There is a new way of thinking that is trying to avoid that “build it, and they will come mentality,” which leads to building it and nobody using it. One of the trends we are seeing is product thinking […]—using techniques like creating user personas of different types of users in the organization, conducting research to understand their needs, and talking with them.”

Breaking silos and fostering communication helps organizations build secure, scalable platforms that support the needs of developers, platform engineers, architects, and security teams.

Challenges of Integrating Security in Platform Engineering

While the benefits of integrating DevOps and security in platform engineering are clear, the journey has expected challenges. Our panel of experts highlighted several key obstacles that organizations may face:

Balancing Autonomy and Control

The primary goal is to empower developers to be more productive and proactive. However, this autonomy needs to be balanced with necessary security controls and governance measures.

Jess Mink points out the importance of achieving harmony between developer autonomy and operational control, stating:

“It’s a delicate balance - you want to make things easy for developers, but you also need to maintain the right level of control and security. It’s about finding the right abstractions and interfaces that give developers the freedom they need while still ensuring the platform remains secure and compliant.”

Driving Adoption and Changing Mindsets

Integrating security into the platform engineering workflow can often be met with resistance from developers accustomed to moving quickly and may view security as an obstacle to their productivity.

As Nariman, a Software Security Architect, notes:

“The challenge is not just about the tools or the technology - it’s about changing the mindset and getting people to understand the importance of security. You need to find ways to motivate developers and make them feel like they’re part of the solution rather than just imposing more rules and processes.”

Effective communication, education, and a focus on the business value of security are key to driving adoption and fostering a culture of shared responsibility.

Adapting to Evolving Needs and Technologies

As organizations grow and their technology stacks evolve, the demands on the platform engineering team can shift rapidly. Keeping up with these changes, while maintaining a secure and reliable platform, can be a significant challenge.

Lindsay Jack, VP of Engineering for the Platform Division at Snyk, explains:

“You might have a platform team that’s really good at a certain set of technologies, but then the organization starts moving in a new direction, and suddenly those skills don’t match up anymore. It’s about being agile and adaptable and making sure you have the right mix of skills and expertise to support the organization’s evolving needs.”

Fostering internal mobility, continuous learning, and a flexible, modular platform architecture can help platform engineering teams navigate these changes more effectively.

Measuring Success in Secure Platform Engineering

Measuring the success of a platform engineering initiative can be complex as it involves balancing a range of technical, operational, and business-oriented metrics but also considers human factors. According to our panel, some key metrics to include:

Developer Experience: Metrics such as developer satisfaction surveys, time-to-onboard new developers, and the number of self-service platform capabilities can provide insights into the effectiveness of the platform in supporting developer productivity and autonomy.
Reliability and Scalability: Monitoring service-level objectives (SLOs), incident response times, and the ability to handle increased traffic or user demands can help assess the platform’s reliability and scalability.
Security and Compliance: Tracking the number of security incidents, the ratio of security issues found during threat modeling versus post-deployment, and the adoption of security best practices can indicate the platform’s security posture.
Automation and Tooling: Metrics like the percentage of infrastructure provisioned through code, the frequency of platform updates, and the time saved through automation can demonstrate the platform’s operational efficiency.
Observability and Monitoring: Measuring the effectiveness of observability tools, the time to detect and resolve issues, and the quality of incident reports can provide insights into the platform’s overall health and performance.

As Jess Mink emphasizes, it’s important not just to collect these metrics but to use them to drive meaningful action and improvement:

“We look at all of those [metric categories] every quarter and write summaries that go up to the executive level. This creates visibility and a shared understanding of problems so that there’s room for movement and change in the right ways.”

The Future of Secure Platform Engineering

Software development and infrastructure management are evolving, and the role of platform engineering will only become more critical to support it. By integrating DevOps and security practices, platform engineering teams can create scalable, secure platforms that empower developers to be more productive and innovate, delivering business value.

Learn how Pulumi customers build secure, scalable platforms and empower their development teams:

Tivity Health: DevSecOps Game-Changer: Security Automation That Delivers Business Results
BMW Group: Unified and Programmatic Approach to Infrastructure Management at BMW Using Pulumi
Lemonade: How the DevOps team supports a much larger group of developers using by Pulumi to standardize infrastructure components and enforce best practices.

Discover platform engineering best practices in The Guide to Platform Engineering: 7 Steps to Get It Right.

Build secure, scalable platforms with confidence—get started with the Platform Engineering Workshop Series & Course

DevSecOps Game-Changer: Security Automation That Delivers Business Results

Sara Huddleston — Thu, 21 Nov 2024 07:16:30 +0000

Organizations are under constant pressure to deliver new products and features faster than ever. But speed alone isn’t enough—businesses must also navigate the complex challenges of ensuring security and managing infrastructure costs effectively.

Enter DevSecOps - the strategic integration of security practices into the DevOps workflow. By automating security processes, organizations can achieve improved speed, scalability, and business impact, all while ensuring security remains a priority.

Tivity Health, a leading health and fitness solutions provider, has embraced this DevSecOps approach using Pulumi, a modern infrastructure as code (IaC) platform. During PulumiUP 2024, David Giambruno, Tivity Health’s VP of Engineering and DevOps, shared how, by leveraging Pulumi, he led the transformation that continuously drives remarkable results in speed, cost savings, and security.

On this article:

The Beginning: From Data Center to the Cloud

Tivity Health’s journey began with a strategic decision to transition from a traditional data center environment to a cloud-native architecture. Rather than opting for a “lift and shift” approach, they made the bold choice to go directly to a cloud-native model, embracing the principles of DevSecOps along the way.

The need to automate infrastructure deployment and management was fundamental. Giambruno explained, “If you can’t automate it, we don’t need it.” This philosophy drove the team to seek out a solution that would not only streamline their operations but also empower their developers to focus on building innovative products for their customers.

Embracing Pulumi: The Power of Automation, Productivity, and Security

Tivity Health’s search for the right tool led them to Pulumi, a unified platform for all the infrastructure needs that allows teams to use general programming languages, such as TypeScript, Python, Java, and Go, to define and manage their cloud infrastructure. Giambruno and his team immediately recognized Pulumi’s ability to deliver on their key requirements:

Automation: Pulumi’s infrastructure as code (IaC) approach enabled Tivity Health to automate the deployment and management of its cloud environments, reducing the time and effort required for these tasks.
Productivity: using Pulumi’s general-purpose programming languages allowed developers to define, deploy, and manage infrastructure within their existing tools.
Security: Pulumi’s security features allowed Tivity Health to integrate security practices into its infrastructure deployment processes, reducing the risk of security breaches and ensuring compliance.
Cost Optimization: Pulumi’s ability to automate the spin-up and teardown of cloud environments on demand has led to significant cost reductions for Tivity Health.

Giambruno describes Pulumi as a “three-dimensional” tool, offering a versatile set of capabilities that have been instrumental in transforming Tivity Health’s operations. “The ability to use those dimensions in lots of different ways to do the automation is what really makes a difference to the teams,” he says.

Driving Business Impact Through Security Automation

By adopting Pulumi and DevSecOps automation, Tivity Health realized significant business benefits:

Speed and Agility: By automating its infrastructure deployment and management processes, Tivity Health has dramatically reduced the time required to spin up new environments or change existing ones. “We run it through automation and boom, it’s out, it’s done,” Giambruno says. This newfound speed and agility have empowered Tivity Health’s developers to focus on building products and features rather than getting bogged down in infrastructure-related tasks.
Cost Optimization: Tivity Health’s cloud-native approach and Pulumi’s automation capabilities have resulted in significant cost savings. The company estimates that its annual cloud spend has decreased from $9.5 million in its data center days to just $2 million—a staggering 79% reduction. These cost savings have allowed Tivity Health to redirect resources towards more strategic initiatives that drive business growth.
Improved Security and Compliance: Tivity Health’s DevSecOps strategy, anchored by Pulumi, has strengthened its security posture and compliance efforts. By integrating security directly into its infrastructure workflows, Tivity Health has improved its security posture. Automation ensures that security measures are enforced consistently across their cloud environments, reducing risks and improving compliance.

Fostering DevOps Culture Through Cross-Functional Collaboration

Using Pulumi provided more than technical benefits. It also enabled better collaboration between the development, security, and operations teams. By providing a common language and framework for infrastructure management, Pulumi has helped break down silos and align these traditionally disparate groups towards a shared goal of delivering secure, high-quality products faster.

Lessons Learned: Navigating the Cultural Shift

Transitioning to DevSecOps and cloud-native practices required a cultural shift at Tivity Health. Giambrono acknowledges that this cultural shift was not without its challenges. He emphasizes the importance of addressing the human element of change, offering the following advice for organizations embarking on a similar journey:

Start with a proof of concepts: Giambruno recommends beginning with a small-scale proof of concepts to demonstrate the capabilities and benefits of the new technologies and processes to help alleviate fears and build confidence among team members.
Empower developers and make them feel safe: By involving developers in the process and ensuring they feel comfortable with the new tools and workflows, Tivity Health gained user buy-in and overall support for the new DevSecOps approach.
Secure buy-in from business and financial stakeholders: Address the concerns of business and financial stakeholders early on, such as the impact on costs and the ability to deliver tangible results. This is crucial for securing the necessary support and resources.
Celebrate successes and build momentum: Giambruno highlights the importance of celebrating the team’s achievements along the way, even when there are bumps on the road, as this helps build enthusiasm and keep the momentum going.

The Future of DevSecOps and Pulumi at Tivity Health

As Tivity Health continues to refine its DevSecOps strategy, Giambruno is optimistic about the future. He envisions a world where the company’s ability to deploy new products and features rapidly will give Tivity Health a significant competitive edge, allowing it to better serve its customers and drive business growth.

Looking ahead, Giambruno is particularly enthusiastic about the potential of Pulumi’s AI-powered capabilities, which he believes will further streamline and optimize the company’s infrastructure management processes. “I’m super looking forward to the tests we’re going to do, like when we acquire someone and then taking them in, ‘Borg-ing’ them into our automation and seeing how much we can take out of their operating cost as fast as possible,” he says.

Tivity Health’s journey with Pulumi is a powerful example of how the right tool, DevSecOps strategy, and automation can drive tangible business results. By focusing on automation, security, and collaboration, organizations can achieve faster, cheaper, and better cloud deployments—putting them on the path to long-term success in an increasingly competitive, cloud-native world.

To learn more about Pulumi and how it can transform your software development and infrastructure management:

Get started with Pulumi Tutorials
Attend an upcoming workshop
Try out the Pulumi Neo code assistant to accelerate your infrastructure as code journey

Pulumi ESC and External Secrets Operator: The Perfect Solution for Today's Cloud-Native Secret Management

Engin Diri — Fri, 04 Oct 2024 00:00:00 +0000

Managing secrets in a cloud-native environment can be challenging, but it is crucial for ensuring the security and integrity of any application or infrastructure. We encounter a lot of different types of secrets, from API keys, database passwords, and certificates to tokens and passwords. These secrets need to be stored securely and accessed by different services in a secure way without exposing any sensitive information to unauthorized users.

Here is where Pulumi ESC and External Secrets Operator come into play by providing a secure and efficient solution for cloud-native secret management.

Pulumi ESC (Environments, Secrets, and Configuration)

Pulumi ESC is our new secrets management & orchestration service for environments, secrets, and configurations. It can be used as part of your Pulumi IaC (Infrastructure as Code) projects using pulumiConfig to expose the stored values to your Pulumi stacks.

But it can also be used as a standalone service to manage secrets and configurations. Pulumi ESC provides support for applications through its dedicated SDKs built for different programming languages. It can also be managed from the command line with the pulumi env commands in the Pulumi CLI, which can inject secrets and configuration as a context for your CLI tools (e.g. terraform).

For more information on Pulumi ESC, you can check out the official documentation.

External Secrets Operator (ESO)

The Open Source project External Secrets Operator (ESO) is a Kubernetes operator that helps manage secrets in a highly secure and scalable way. ESO synchronizes secrets stored in external secret management systems like Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and more into Kubernetes secrets. This approach allows us to control secret sprawl and centralize the management of secrets in a single place while providing a secure and controlled way to access them.

Why Kubernetes secrets are not enough?

Kubernetes secrets are a great way to store sensitive information in a Kubernetes cluster. Kubernetes secrets are created and stored as a Kubernetes resource object and then used by pods or other resources inside the cluster.

Here is an example of a Kubernetes secret manifest file:

apiVersion: v1
kind: Secret
metadata:
 name: mysecret
type: Opaque
data:
 username: YWRtaW4= # admin
 password: YWRtaW4= # admin

or using kubectl:

kubectl create secret generic mysecret --from-literal=username=admin --from-literal=password=admin

Kubernetes secrets have several limitations. Here are the top ones:

They are stored by default in etcd in base64 encoded format, which is not secure.
They are created either manually by using kubectl commands or by using a manifest file, making them hard to manage at scale.
Hard to manage and synchronize secrets across different environments and clusters.
There is no default way to rotate secrets automatically.

ESO shines by tackling all the challenges listed and providing a way to manage secrets in a more secure and efficient manner. Benefits include:

Mitigate security risks as secrets are stored in a secure external secret management system like Pulumi ESC, which provides more robust security features like encryption, access control, and auditing.
Eliminate manual management of secrets by synchronizing them from external secret management systems into the Kubernetes cluster, reducing the need for manual intervention and minimizing the risk of human error.
ESO provides a unified interface, making it easy to manage secrets across clusters and multi-cloud environments.
ESO integrates with external secret management systems which ensure that access to secrets is tracked and an audit trail is maintained. This is crucial for security compliance and adhering to regulatory standards.
Automate secrets rotation when the external secret management system has support it.

External Secrets Operator architecture

The central part of the ESO architecture is the extension of the Kubernetes API with a new Custom Resource Definition (CRD), which defines where the secrets are stored and how they are synchronized into Kubernetes secrets. The ESO controller then retrieves the secrets from the external secret management system and creates or updates the Kubernetes secrets. Every time a secret is updated in the external secret management system, the ESO controller will synchronize the secret in the Kubernetes cluster and update the secret accordingly.

Core resources of ESO

(Cluster)SecretStore: Defines the external secret management system where the secrets are stored. It contains the necessary information to connect to the external secret management system, such as the endpoint, authentication credentials, and other configuration options. The SecretStore is namespaced while the ClusterSecretStore is cluster wide, which allows you cross-namespaced access to the secrets.
(Cluster)ExternalSecret: Defines the secret that needs to be synchronized into the Kubernetes cluster. It contains the reference to the SecretStore and the secret key and how the data should be transformed and saved as a Kubernetes secret.
PushSecret: With the PushSecret resource, you can push secrets from the Kubernetes cluster to the external secret management system. The Pulumi ESC provider for ESO supports this feature, which allows you to push secrets from the cluster into Pulumi ESC. This feature is useful when you create secrets through other means and want to synchronize them into Pulumi ESC to manage them centrally.

How to Use Pulumi ESC and ESO Together

To demonstrate how to use Pulumi ESC and ESO together, we will store some secrets and configuration values in Pulumi ESC, then synchronize them into a Kubernetes cluster using ESO.

To follow along, you’ll need the following prerequisites:

A Kubernetes cluster (I will be using a local KinD cluster, but you can use any Kubernetes cluster)
Pulumi CLI installed. You can use the pulumi env commands to manage secrets and configurations.
kubectl CLI installed for some debugging

Step 1: Install ESO and Link to Pulumi ESC

We will use Pulumi with the pulumi-kubernetes provider to deploy ESO into the Kubernetes cluster. The power of Pulumi ESC integration comes in two ways: First, to supply the Pulumi access token to ESO via Pulumi IaC, allowing the new ESO instance to access Pulumi ESC later. Second, to provide configuration and secrets from Pulumi ESC, via ESO, to the PodInfo application within the Kubernetes cluster.

pulumi new kubernetes-typescript --name eso-pulumi-esc

You should see the following output:

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

Project description (A minimal Kubernetes TypeScript Pulumi program):
Created project 'eso-pulumi-esc'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
Stack name (dev):
Created stack 'dev'

The package manager to use for installing dependencies npm
Installing dependencies...


added 310 packages, and audited 311 packages in 12s

44 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

Head to Pulumi Cloud Console and create a new Pulumi ESC project with the name eso-esc-demo:

And the environment eso-esc-dev:

In the editor add the following yaml into the Environment definition:

values:
 pulumiConfig:
 pulumi-pat:
 fn::secret: <your-pulumi-pat>

Don’t know how to create a PAT? Check out the official Pulumi documentation.

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/eso-esc-demo/eso-esc-dev

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-esc-demo/eso-esc-dev

Now, we need to link the Pulumi ESC project to the Pulumi IaC project. To do this, we need to add the following to your Pulumi.dev.yaml:

environment:
- eso-esc-demo/eso-esc-dev

Now that all the wiring is done, we can deploy the ESO instance into the Kubernetes cluster. Open the index.ts file and add the following code:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();

// Create a Kubernetes namespace
const ns = new k8s.core.v1.Namespace("external-secrets", {
 metadata: {
 name: "external-secrets",
 },
});

// Deploy a Helm release into the namespace
const externalSecrets = new k8s.helm.v3.Release("external-secrets", {
 chart: "external-secrets",
 version: "0.10.4", // Specify the version of the chart
 namespace: ns.metadata.name,
 repositoryOpts: {
 repo: "https://charts.external-secrets.io",
 },
});

// Deploy a secret into the namespace
const mySecret = new k8s.core.v1.Secret("my-secret", {
 metadata: {
 namespace: ns.metadata.name,
 name: "pulumi-access-token",
 },
 stringData: {
 "PULUMI_ACCESS_TOKEN": config.require("pulumi-pat"),
 },
 type: "Opaque",
});

// Export the namespace name
export const namespaceName = ns.metadata.name;

// Export the Helm release name
export const helmReleaseName = externalSecrets.name;

Deploy the stack by running:

pulumi up

And you should see that the secret was created in the Kubernetes cluster and the ESO instance was deployed successfully.

kubectl get secret -n external-secrets pulumi-access-token -o jsonpath='{.data.PULUMI_ACCESS_TOKEN}' | base64 -d

Step 2: Create an External Secret

Now, we will create a secret in the Pulumi ESC project and synchronize it into the Kubernetes cluster by creating an ESO reference to the external secret.

Create a new ESC environment called esp-esc-app in the eso-esc-demo ESC project.

Add the following YAML into the Environment definition:

values:
 app:
 hello: world
 hello-secret:
 fn::secret: world

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/eso-esc-demo/esp-esc-app

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-esc-demo/esp-esc-app

Either way, you should see following environment configuration in the Pulumi Cloud Console:

We can now create ClusterExternalSecret in the index.ts and reference the ESC secret.

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// cut for brevity

const crd = new k8s.apiextensions.CustomResource("cluster-secret-store", {
 apiVersion: "external-secrets.io/v1beta1",
 kind: "ClusterSecretStore",
 metadata: {
 name: "secret-store",
 },
 spec: {
 provider: {
 pulumi: {
 organization: pulumi.getOrganization(),
 project: "eso-esc-demo",
 environment: "esp-esc-app",
 accessToken: {
 secretRef: {
 name: mySecret.metadata.name,
 key: "PULUMI_ACCESS_TOKEN",
 namespace: mySecret.metadata.namespace,
 },
 },
 },
 },
 },
});

We can check that the secret was successfully synchronized by running:

kubectl get clustersecretstore secret-store -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

Step 3: Deploy an Application and Reference the Secret

Now, we can deploy an application that references the secret from the Kubernetes cluster. I am going to deploy the app called PodInfo, which is an application that exposes the pod information as an API. Add the following code to the index.ts:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// cut for brevity
const podInfo = new k8s.helm.v3.Release("podinfo", {
 chart: "podinfo",
 version: "6.7.0",
 namespace: "podinfo",
 createNamespace: true,
 repositoryOpts: {
 repo: "https://stefanprodan.github.io/podinfo",
 },
 values: {
 extraEnvs: [
 {
 name: "FROM_ESC_VIA_ESO",
 valueFrom: {
 secretKeyRef: {
 name: "esc-secret-store",
 key: "hello-secret",
 }
 }
 },
 {
 name: "FROM_ESC_VIA_ESO_2",
 valueFrom: {
 secretKeyRef: {
 name: "esc-secret-store",
 key: "hello",
 }
 }
 }
 ]
 },
});

const externalSecretPodInfo = new k8s.apiextensions.CustomResource("external-secret-podinfo", {
 apiVersion: "external-secrets.io/v1beta1",
 kind: "ExternalSecret",
 metadata: {
 name: "esc-secret-store",
 namespace: podInfo.namespace,
 },
 spec: {
 dataFrom: [
 {
 extract: {
 conversionStrategy: "Default",
 key: "app",
 }
 }
 ],
 refreshInterval: "1h",
 secretStoreRef: {
 kind: clusterSecretStore.kind,
 name: clusterSecretStore.metadata.name,
 }
 },
});

This will deploy the PodInfo application and create an ExternalSecret resource that references the secret from the Pulumi ESC project. We mount the secret into the pod as an environment variable by using the extraEnvs field in the Helm chart.

In a real-world application, you might use alternate ways to reference the secret, like mounting it as a volume or using fromEnv field in the Helm chart. It depends on your application and how you want to consume the secret.

After deploying the stack, you should see that the PodInfo application is running. To access the application, we can port-forward the service by running:

kubectl port-forward svc/podinfo 9898:9898 -n podinfo

Our PodInfo demo application provides a large selection of different endpoints to interact with. We are interested in the /env endpoint to check if the secret was successfully mounted into the pod. Let’s try that:

curl -s localhost:9898/env | grep FROM
 "FROM_ESC_VIA_ESO=world",
 "FROM_ESC_VIA_ESO_2=world",

And here we see secret values from the ESC project, proxied by ESO, and passed to the PodInfo application as environment variables. That means the secret was successfully mounted into the pod and the application is running as expected.

Step 4: Clean Up

After you are done with the demo, you can clean up the resources by running:

pulumi destroy

Conclusion

The combination of Pulumi ESC and External Secrets Operator provides a secure and efficient way to manage secrets in a cloud-native environment, improving security, efficiency, and compliance when consuming secrets in your Kubernetes cluster. By using Pulumi ESC, you can manage secrets and configurations in a centralized way, and with ESO, you now have a streamlined process to synchronize secrets into your Kubernetes cluster.

Start using Pulumi ESC and ESO today by creating an account on the Pulumi Cloud Console and begin managing your secrets in a secure and efficient way.

Create an Account

Announcing OIDC Support for Pulumi Azure Providers

Thomas Kappler — Thu, 01 Jun 2023 00:00:00 +0000

We are happy to announce the delivery of Azure OIDC authentication, one of the most requested features for the Pulumi Azure Native Provider. With the v1.100.0 release, OpenID Connect (OIDC) authentication is now fully supported in both the Azure Native and Azure Classic providers. Let’s dig in to learn what it is, how it works, and why it’s useful.

Pulumi announced OIDC support for Deployments in January. This article is about the same OIDC protocol, used to authenticate your Pulumi program with Azure instead of Pulumi Deployments.

The What and Why of OIDC

OIDC stands for OpenID Connect, a standardized protocol for federated identity. This means that if you maintain an identity with two service providers, you can tell one of them to trust the other. Then, you can use credentials for the trusted service to get credentials for the other.

As an example, we’ll use a Pulumi program running in CI on GitHub Workflows, creating Azure resources. Without OIDC, this program would need a secret to authenticate with Azure - a client secret or a certificate. With OIDC, your program doesn’t need a secret. It can send its GitHub token to Azure instead. Based on the trust relationship between the two, Azure will exchange the GitHub token for an Azure token.

Secrets need to be safeguarded, rotated, and possibly revoked. If you’ve ever managed a system, whether production or CI/CD, the possibility of managing fewer secrets will be enticing. So how do we go about implementing this?

Enabling OIDC

There are two parts to enabling your Pulumi program to authenticate via OIDC. One is establishing the trust relationship between Azure and the other service that’s involved, like GitHub. This is a one-time operation. The second one is providing your program with the necessary configuration to perform the OIDC token exchange at runtime.

Fortunately, Microsoft has the first step covered with their establishing the trust relationship guide. The idea is that an Active Directory app registration holds the necessary federated credentials. Your program will then use this app’s tenant id to request the token exchange. The setup depends on whether you run on GitHub Actions, Kubernetes, or other providers.

On to the second step, configuring your Pulumi program. First, tell the Pulumi provider to use OIDC by setting the environment variable ARM_USE_OIDC or the Pulumi configuration azure-native:useOidc to true. Next, you’re in luck if your program runs on GitHub Actions: you’re done. GitHub exports the necessary values in form of variables that Pulumi understands.

For GitHub, pay close attention to the “entity type” of your Azure Active Directory credential as documented in the Azure guide. It can be a Git branch or tag, a pull request, or an environment as specified in the GitHub workflow. When your Action requests the OIDC token exchange, one of these needs to match exactly. This means that the only way to configure OIDC authentication for all branches or tags is to use environments in your workflows.

For other providers, you need to provide your Pulumi program with two more settings. Environment variable ARM_OIDC_REQUEST_TOKEN or configuration azure-native:oidcRequestToken is your provider’s token to exchange for an Azure token. ARM_OIDC_REQUEST_URL or azure-native:oidcRequestUrl is the URL to contact to initiate the token exchange.

A complete example

Let’s take a look at how the CI end-to-end tests of the Pulumi Azure Native provider are set up.

Configuration

First, we have an Active Directory App creatively named oidc-test. We simply copy its client and tenant ids and set them in the CI workflow, along with our subscription:

env:
ARM_CLIENT_ID: 11223344-****-****-****-************ [redacted]
ARM_SUBSCRIPTION_ID: 55667788-****-****-****-************ [redacted]
ARM_TENANT_ID: 99aabbcc-****-****-****-************ [redacted]
ARM_USE_OIDC=true

Note how there is no secret or certificate present.

We also give the required permissions to the workflow step where the authentication happens:

permissions:
id-token: write # required for OIDC auth

In Azure, we added the required credentials to the Active Directory App. You’ll find this under “Certificates & secrets” -> “Federated credentials” on the app’s page. Let’s look at two of the credentials. Both enable OIDC for all pull requests on the repository, one for each Azure provider.

End to End Flow

Now, when our GitHub workflow for CI tests runs, the Azure provider knows to authenticate via OIDC because ARM_USE_OIDC is set. It performs the following steps:

Call GitHub’s OIDC provider, found at the URL that GitHub automatically exports as ACTIONS_ID_TOKEN_REQUEST_URL. For authentication, it includes a token GitHub generates for each workflow run and exports as ACTIONS_ID_TOKEN_REQUEST_TOKEN. The response is an OIDC token in form of a JSON Web Token (JWT).
Send the OIDC token to Azure’s Active Directory endpoint. Include the client and tenant ids of our Active Directory App that we configured via ARM_CLIENT_ID and ARM_TENANT_ID. Active Directory looks up the trust relationship we configured and validates the token, and responds with a valid access token for Azure.
The Pulumi provider can now access Azure resources.

That’s it! Your CI job now runs without maintaining an Azure secret in GitHub, or wherever your Pulumi program runs.

Wrapping up

This article only demonstrated the basics. You can also customize your OIDC token’s claims to control access to Azure in a more fine-grained manner.

I hope this helps you simplify your Azure authentication. Please try it out and let us know if you hit any issues in the Pulumi Community Slack!

IaC Best Practices: Implementing RBAC and Security

Scott Lowe — Tue, 23 May 2023 00:00:00 +0000

This post continues our series of blog posts focused on IaC best practices. In earlier posts, we introduced Zephyr Archaeotech Emporium, the fictional company that sits at the center of this series, and discussed Zephyr’s primary use case for Pulumi: managing their online retail store. You read how Zephyr’s initial use of Pulumi changed to incorporate the use of short-lived per-developer stacks. Later, as Zephyr continued to grow, you saw how Zephyr restructured their Pulumi projects and stacks, and incorporated the use of Stack References. This post is a complement to the post on structuring Pulumi projects, concentrating on the use of role-based access control (RBAC) and security in Zephyr’s multi-project configuration.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. You’ve seen these practices emerge over the course of the series—not all immediately, and not all right away. Structuring the blog series in this way is a deliberate decision; many best practices are “point in time” recommendations: they are context-dependent and the recommendations for any given company, like Zephyr, may change as the company and its needs change.

Evolving Project Structure to Enhance Security and Scalability

So far, you’ve read about how Zephyr started out with a single project and a single Git repository. Adding short-lived per-developer stacks, as described in the post on best practices for developer stacks and Git branches, didn’t really impact this structure. However, as the company grew, Zephyr’s project and Git repository structure evolved into its current form:

The “zephyr-infra” project handles base infrastructure (VPC and related constructs).
The “zephyr-k8s” project automates the Kubernetes platform layer, building atop the base infrastructure components supplied by the “zephyr-infra” project.
Finally, the “zephyr-app” project handles the deployment of the online retail store application onto Kubernetes, layering on top of the platform provided by the “zephyr-k8s” project.

You can review the reasons for this structure in the post on best practices for structuring Pulumi projects. One of the reasons listed in the third Zephyr blog post was security:

…security is a factor that affects how projects and stacks should be structured. In Zephyr’s case, the ability for one team to affect another team’s resources was present when everything was a single project. With multiple projects, Zephyr can use stack permissions and role-based access control (RBAC) in Pulumi Cloud to appropriately control who can affect what.

The last part of that quote—the part about being able to “appropriately control who can affect what”—is another way of describing what’s known as the principle of least privilege. If you’re unfamiliar with this term, we recommend following the link and getting more information, because the principle of least privilege stands at the core of all the recommended practices you’ll read in this post.

Pulumi supports a number of self-managed backends, including Amazon S3 (and S3-compatible servers like Minio or Ceph), Azure Blob Storage, and Google Cloud Storage. The RBAC functionality described in this post applies only to the managed Pulumi Cloud backend.

Understanding the RBAC Building Blocks in Pulumi Cloud

To understand how to secure your Pulumi projects and stacks in Pulumi Cloud, it is important to first explain the basic building blocks you’ll use:

Organization: An organization represents a customer/company within Pulumi Cloud. Generally speaking, a single company like Zephyr is represented by a single organization.
Member: Each member represents a user account that is part of an organization. In Zephyr’s case, each Zephyr employee that needs to interact with Pulumi is a member of Zephyr’s organization in Pulumi Cloud.
Team: A team is a grouping of members, and is the primary component involved in assigning per-stack permissions in the Pulumi Cloud RBAC model. How you structure teams within your organization will depend on a great many different factors we’ll explore in this post.
Stack: A stack is a separate instance of the resources created by a Pulumi program within a project. Each stack has its own independent state and its own configuration values. With respect to RBAC, each stack is a separate entity to which one or more teams can be granted permission. In Zephyr’s case, there is a long-lived production stack for each of the separate projects, and a persistent (or long-lived) test stack for each of the separate projects. There are also the per-developer stacks, which are typically short-lived.

This diagram, also used in the post on best practices for stack references, shows the stacks and projects for Zephyr’s online store:

Now, how are these different components assembled together? At its most basic level, you would use teams to group members together, and then assign stack permissions to the team. This diagram helps illustrate the idea:

A couple of important points emerge from the above diagram:

The relationship between stacks and teams is many-to-many. That is, any given stack can have many teams assigned permissions, and any given team may be assigned permissions to many stacks.
Similarly, the relationship between teams and members is also many-to-many (any given team can—and almost always will—have multiple members, and any given member can be part of multiple teams at the same time).

Assembling Security for Stacks

The first, and perhaps the most important, best practice with regards to securing your stacks is to actually use Pulumi Cloud’s RBAC functionality. While this may seem obvious, there are practical implications to this statement that are useful to explore.

Pulumi Cloud has the ability to set a default stack permission for all members of an organization. In the screenshot below, you can see that the default permission has been set to Write:

In this configuration, you are effectively disabling Pulumi Cloud’s RBAC functionality. Permissions in Pulumi Cloud are additive; that is, permissions only state what you can do, not what you can’t do. Therefore, if the default stack permission is set to Write, then assigning a team read-only permissions to a stack has no effect—because members of that team can read from the stack (by virtue of their inclusion in a team) and can write to the stack (by virtue of the default stack permission). The team members’ read-only permission is added to the default stack permission for their effective permission.

For this reason, Pulumi recommends setting the default stack permission to Read or None. Setting it to Read means that all members of the organization can see and read from stacks that aren’t their own, but can’t change or modify the others stacks. Setting it to None means that members have permissions to their own stacks by default—gaining any form of access to stacks other than their own requires being added to a team that has permissions to those stacks.

When it comes to assigning a team access to a stack, there are three options:

Stack reader (read-only access; also called “Stack reader access” in some areas of the Pulumi Cloud UI)
Stack editor (read/write access, but can’t delete the stack; also called “Stack write access” in the Pulumi Cloud UI)
Stack admin (full access; you may also see it called “Stack admin access”)

When the “Allow organization members to create stacks and transfer stacks to this organization” is checked, like in the screenshot above, then the user (organization member) that creates a stack automatically becomes the stack admin for that stack. The user’s ability to delete the stack, however, is affected by the “Allow stack admins to delete stacks” setting.

Full details on what each of these permissions grants is detailed in the projects and stacks documentation (see the “Stack Permissions” section). In accordance with the principle of least privilege, Pulumi recommends you grant the minimum level of access necessary in order for a user (organization member) to meet their job responsibilities. You accomplish this by adding the member to a team, and then granting the team the minimum level of necessary access. You can’t assign stack permissions directly to users.

Because team membership is central to the use of stack permissions, Pulumi has several ways to make it easier to manage the relationship between teams and stacks:

You can use the Pulumi Cloud user interface to assign a team access to a stack.
You can use Pulumi Cloud’s REST API to control team access to stacks.
You can use the Pulumi Cloud provider for Pulumi to programmatically manage team access to a stack using Pulumi itself.
You can use the --teams flag to pulumi stack init to assign stack editor access to one or more teams when you create the stack.

In general, Pulumi recommends declaratively managing your Pulumi Cloud RBAC configuration with the Pulumi Cloud provider (formerly known as the Pulumi Service provider), unless you have specific requirements that would prevent it. However, be aware that the Pulumi Cloud provider doesn’t allow you to change a member’s role in an organization (every member is either a “member” or an “admin” within an organization); that would have to be done via the Pulumi Cloud user interface or REST API.

Pulumi also recommends using the --teams flag with pulumi stack init to ensure that new stacks are properly assigned team permissions upon creation. You can then go back and fine-tune those team permissions—which are set to “Stack editor” or “Stack write access” when you use the --teams flag—using a Pulumi program with the Pulumi Cloud provider. Alternately, teams can use a team access token when creating a new stack, and that will also grant the team permission to the newly-created stack.

Finally, Pulumi recommends allowing users to create their own stacks. Otherwise, only an organization admin would be allowed to create new stacks, and this is likely to cause a bottleneck in provisioning cloud infrastructure—one of the very issues that infrastructure as code tools like Pulumi are attempting to solve! (It may be necessary in some cases to restrict this functionality, such as in certain highly regulated industries or verticals.) Preventing users from creating their own stacks would also prevent the use of per-developer stacks like Zephyr uses, which as you’ve seen can be enormously helpful in accelerating development and deployment velocity.

Inspecting RBAC Implementation

Zephyr’s RBAC implementation in Pulumi Cloud is represented in part by a diagram that you also saw earlier in this post:

Zephyr set the organization default stack permission to Read. This enabled teams to read all stacks, and enabled the continued use of stack references without any additional configuration. Teams were created to grant “Stack editor” (also called “Stack write access”) to stacks based on the overall organizational structure of the company. Thus, the Platform team gained access to the non-production zephyr-infra and zephyr-k8s stacks, while the Application team was given access to the non-production zephyr-app stacks. However, only members of the Production team were given access to the production stacks in all three projects, ensuring that access to update those stacks—and thus the actual resources running Zephyr’s online store—remained relatively controlled.

Summarizing Best Practices: RBAC & Security

The post covered the following guidelines for securing your stacks in Pulumi Cloud:

Use Pulumi Cloud’s RBAC functionality by setting your organization’s default stack permissions to Read or None. This is generally the safest way to implement the principle of least privilege, unless you have a good use case for allowing all organization members full write access to all stacks.
When assigning team permissions, grant the minimum level of access necessary. If a user isn’t responsible for making updates to a stack, then give that user stack reader permission through a team membership (remember that you can’t assign stack permissions directly to organization members, you have to use a team). Keep in mind, though: if the organization level stack permission has been set to Read, then members already have the equivalent of stack reader permissions on all stacks, so creating a team to grant stack reader permission is unnecessary and superfluous.
Declaratively manage your Pulumi Cloud RBAC configuration where possible using the Pulumi Cloud provider from within a Pulumi program.
Use the --teams flag with pulumi stack init to ensure that teams have stack permissions upon stack creation. You can also use a team access token to accomplish the same result.
Allow users to create their own stacks. Generally, turning this functionality off (found in the Settings > Access Management screen of Pulumi Cloud) will end up creating a bottleneck, as only organization admins would be allowed to create new stacks.

In the next IaC Best Practices post, the focus is on Automation API. Stay tuned to see how Zephyr uses Automation API to further streamline and automate the deployment of their cloud resources!

Using Pulumi Securely

Tushar Shah — Thu, 13 Oct 2022 00:00:00 +0000

Cloud computing’s greatest strength and weakness is the proliferation of a massive number of services globally. To adequately assess and mitigate the inherent risks for your company, customers, and employees, cloud architects are typically responsible for a vast surface area of potential endpoints and vectors of attack.

Cloud Security Posture

AWS likes to talk about a Shared Responsibility Model, their friendly way of saying, “we’ve got our stuff covered, and if you’re compromised, it’s almost certainly your fault.” They have built up practices and recommendations, like the AWS Well-Architected Framework. Their pillars of architecture philosophy give you your best chances to securely use their resources (following these recommendations is easy with Crosswalk for AWS from Pulumi). Increasing high-profile attacks against private and open-source software supply chains have developers increasingly on high alert and conscious of incorporating more security into their DevOps.

Risks associated with maintaining a good cloud security posture extend to your infrastructure as code with Pulumi and it’s important to leverage the features and capabilities of the Pulumi platform along with good industry practices to secure your code, configuration and secrets.

Pulumi’s Hierarchy

Pulumi has three levels to consider access for: users, teams and organizations.

Organizations are a space for you to collaborate on shared projects and stacks. You may have multiple cloud accounts from a single cloud provider or multiple cloud provider accounts tied to an organization in any combination. It’s recommended that you segment organizations and cloud accounts to limit access and the “blast radius” of security or financial events that may occur within them.

Teams are groups of users within an organization with specific permissions to stacks. Consider using these to break up access to environments such as dev, staging and production. You should look to mirror your code repository or IAM teams/groups as much as possible.

Users are tied to personal identities, often with an identity provider and should be used wherever a human is using Pulumi. These identities should not be used for any systems, tools or pipeline-based provisioning.

Security Best Practices with Pulumi

The following recommendations should help you get started and prepare you to scale your company, application and cloud engineering team long term.

Manage Federated Identity

Connect your user and Pulumi organization with an Identity Provider such as Github, Gitlab or BitBucket, your Single Sign On (SSO) system and/or another System for Cross-Domain Identity Management (SCIM). Your code repositories hold your Infrastructure as Code, and most organizations are motivated to keep access tightly controlled. SSO improves this process across tools and puts more control into your organization’s hands. SCIM enables you to manage your users and groups centrally in your Identity Provider (IdP) and then synchronize those users and groups to the Pulumi Service. Leveraging your existing onboarding and offboarding process will help limit the risk of bad actors in Pulumi itself.

Build Pulumi Teams

Teams improve on basic role-based access control (RBAC) for administrators and users by allowing you to group users and their access to specific stacks in your organization. For example, a team called productionreadonly could be created and scoped so that all members have only limited access to read the production stacks, while another team, productionadmins, would retain full privileges to those stacks.

Import Manually Created Resources

Manually created cloud resources are error-prone and difficult to audit, iterate and improve on. If you’ve created manual resources, consider importing them to Pulumi programs.

Tag your Stacks

We have previously covered Tag Policies as a best practice for the cloud resources themselves. Still, you can also tag your Pulumi stacks to help organize and visualize your application resources in logical groupings that will help you assess, respond and automate incident remediation. Here are two examples of using tagging your stacks automatically: in Python and TypeScript.

Deploy from Pipelines

Automate every step of your deployment process for anything beyond local development by creating CI/CD pipelines. Manual approval steps in your pipeline should be avoided. Consider creating automated checks to ensure your infrastructure is created only with approved Pulumi Packages and automatically ensure your infrastructure is in compliance by using CrossGuard.

Leveraging Automation API as the programmatic interface for running Pulumi programs without the Pulumi CLI is a strongly typed and safe way to use Pulumi in embedded contexts such as web servers without having to shell out to a CLI. For example, Elkjop Nordic uses Automation API to provide a self-service portal that serves as a secure IT vending machine full of their application and infrastructure building blocks.

Log Everything

Track key system events such as user and pipeline activity, attempted and restricted activity, and identity and access control changes. Use Pulumi audit logs to simplify this process out of the box. If applicable, we would suggest automatically exporting Pulumi Audit Logs to your systems.

Use Tokens

Organization Access Tokens, Team Access Tokens and Personal Access Tokens securely connect your automation pipelines and development environments with Pulumi without the risks of association user/password combinations. Machines talk to Pulumi with tokens of various types, and it is always advisable to use Tokens over Users where possible. More on the types of tokens is below.

Organizational Access Tokens

These are scoped to the entire Pulumi organization. Use these for tooling with broad access across stacks and resources deployed in that organization. For example, if your production environment runs in an isolated cloud account and organization, then an Organization Access Token is likely appropriate for your CI/CD pipeline that deploys production.

Team Access Tokens

Scoped to the stack access of a Pulumi team. Use these when resources associated with different environments or services are commingled within a single cloud account or organization. In general, it is often recommended to create a Team Access Token for each CI/CD pipeline, for example, dev vs production.

Personal Access Tokens

Scoped to individual users. Used whenever a developer deploys from their local machine. Be sure your users are part of a Pulumi team to simplify their role-based access within the platform.

Rotate the Tokens

Add Pulumi to your list of tokens to rotate on a regular basis.

Assessing and Implementing

Even if you’ve been using Pulumi for a while, we recommend you regularly take these three steps to continuously audit and improve your cloud security posture as it relates to your infrastructure as code:

Confirm your users’, teams’, and organization access are in line with either or both your code access or cloud access policies; make any necessary updates to access, permissions or scope of privilege.
Audit any pipeline that uses Pulumi and ensure that they use fresh, rotated tokens.
Review your audit logs regularly, noting any abnormal or unexpected activity. It is advisable to export these audit logs and process them with a security event management system.

Some of the features mentioned, such as teams, SSO/SCIM and audit logs, are only available to Enterprise and/or Business Critical Edition users and those on our 14-day trial. If you would like to implement and/or test any of these features please start a trial or contact us for access.

Accessing Secrets Safely in Lambda Functions

Piers Karsenbarg — Wed, 12 Oct 2022 17:19:53 +0100

The subject of how to make use of secrets in Lambda Functions comes up a fair bit, and although there seems to be a lot of discussion on where you should store them, the one thing that comes up is that you should never store the plain text values of secrets in the Lambda Function’s environment variables. One such discussion I was having with a customer made me think about how it should be possible to take the secrets that you’ve got on your stack config file and then use them to configure your Lambda Function, with the plain text values going into the Function’s environment variables and the encrypted secret values going into AWS’ Secrets Manager.

Getting started

I’m going to assume you’ve already got Pulumi and AWS set up, you’re logged into your backend, and you’ve already created a project and stack. I’m also using the Pulumi Service as my secrets provider.

Although Lambda Functions are free to deploy, and you get a generous allowance as part of AWS’ free tier, storing and accessing secrets is not. You can view the cost on the AWS Secrets Manager Pricing page.

Structured configuration

We’re going to start by adding settings, both secret and plaintext, to our stack configuration, in a structured way. We’re going to add these settings as a dictionary so we can use the names as the variable names in our Lambda Function.

First we add two settings in plaintext that are going to be used as environment variables in our lambda:

pulumi config set --path 'lambdawithsecrets.envvars["envvar1"]' envvar1value
pulumi config set --path 'lambdawithsecrets.envvars["envvar2"]' envvar2value

and then we add the secret settings:

pulumi config set --path 'lambdawithsecrets.secrets["secret1"]' secretvalue1 --secret
pulumi config set --path 'lambdawithsecrets.secrets["secret2"]' secretvalue2 --secret

If we take a look at the stack config (in my stack named dev it’s the Pulumi.dev.yaml file):

config:
 lambda-secrets:lambdawithsecrets:
 envvars:
 envvar1: envvar1value
 envvar2: envvar2value
 secrets:
 secret1:
 secure: AAABALZIAzDaE45MbdO08w/QMHK6CE4hssidMK70pyYt6hIFHpKeq0mBNic=
 secret2:
 secure: AAABAI+y9HpdgQ1u1rSPKGZeBkAIIbZfki84zmRrcWVltIMI5SGWZbYOyfQ=

there are several things I want to point out:

The entire config item is named lambdawithsecrets (this is how we’ll retrieve the configuration later)
There are two sections, envvars and secrets. These are how I’ll refer to the dictionaries later on in the code, but you can name these in any way you want.
We’ll use the dictionary keys (envvar1, envvar2, secret1, secret2) to refer to the variables in the Lambda Function so we know what they’re named and can easily access them. Again, you can name these to be whatever values you want.

Writing our Pulumi program

Next we’re going to start updating our index.ts file. Everything from now on can be copied and pasted in order into the file, and I’ll post the entire contents at the end.

As we’re only going to use the Pulumi and the AWS NPM packages you can replace everything in the file with the following:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

A warning about accessing the configuration

Before we continue, it’s worth pointing out the following (from the Pulumi documentation on secrets ):

On pulumi up, secret values are decrypted and made available in plaintext at runtime. These may be read through any of the standard pulumi.Config getters. While it is possible to read a secret using ordinary non-secret getters, this is almost certainly not what you want. Use the secret variants of the configuration APIs instead since this ensures that all transitive uses of that secret are themselves also marked as secrets.

Now that’s out the way, let’s continue.

Accessing the configuration

Since we’re using structured configuration to store the values, we can access them in a strongly-typed way. We’re going to create an interface to hold the structure:

interface LambdaConfig {
 envvars: string[];
 secrets: pulumi.Output<string>[];
}

You’ll notice that the property names match the dictionary names in the stack config and that the secrets property is of type pulumi.Output<string>[] instead of string[]. This is because they’re Pulumi secrets, and we want to handle them in the correct way.

Next we’ll access the config and store the values in a variable:

const config = new pulumi.Config();
const lambdaconfig = config.requireObject<LambdaConfig>("lambdawithsecrets");

Storing the secret values

As I said in the beginning, we’re going to store the secrets from the stack config in AWS Secrets Manager, and then the ARN of each of the secrets will be stored in the Lambda Function’s environment variables.

We’re going to create a couple of arrays to store (non-sensitive) information about the Secret resources that we’re going to create to use later on. One to hold the Secret resources and one to hold the ARNs of the secrets. Due to the fact that it’s the SecretVersion where we store the secret value, it’s pretty safe to store the Secret.

Next we’ll loop through the config’s secrets dictionary, grab the values and for each one create a Secret and a SecretValue resource, and then populate the two arrays with the relevant values.

const secretsArnArray: pulumi.Output<string>[] = new Array();
const secretArray: aws.secretsmanager.Secret[] = new Array();
for (const key in lambdaconfig.secrets) {
 if (lambdaconfig.secrets.hasOwnProperty(key)) {
 const secret = new aws.secretsmanager.Secret(`${key}`);
 const secretVersion = new aws.secretsmanager.SecretVersion(`secretversion-${key}`, {
 secretId: secret.id,
 secretString: lambdaconfig.secrets[key],
 });
 secretArray.push(secret);
 secretsArnArray[key.toLocaleUpperCase()] = secret.id;
 }
}

We’re not actually doing anything with the SecretVersion resources, so that’s why we’re not assigning them to any variables.

Identity and Access Management

There are a few resources we’re going to need to create and assign to the Lambda Function to give it access to the Secret resources:

A role that we can assign to the Lambda Function
A Policy that is going to give that role read access to the values of the secrets we’ve created
A couple of RolePolicyAttachment resources which is how we attach policies to the above role

Firstly, the role:

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(aws.iam.Principals.LambdaPrincipal),
});

We’re using the assumeRolePolicyForPrincipal function because doesn’t that look better than this:

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [{
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Sid: "",
 Principal: {
 Service: "lambda.amazonaws.com",
 },
 }],
 }),
});

(either of the above Role resources can be used, but I’m using the top one because I think it’s neater).

Next we create a policy document and a policy. We’re going to use the secretArray variable that we created earlier to allow us to only allow the Lambda Function access to the secrets that we’ve created:

const secretManagerPolicyDoc = aws.iam.getPolicyDocumentOutput({
 statements: [
 {
 effect: "Allow",
 actions: [
 "secretsmanager:GetSecretValue",
 ],
 resources: secretArray.map(x => pulumi.interpolate`${x.arn}`),
 },
 ],
});

const secretManagerPolicy = new aws.iam.Policy("secretsPolicy", {
 policy: secretManagerPolicyDoc.apply(doc => doc.json),
});

Finally we’re going to attach the above policy and the AWS Managed AWSLambdaBasicExecutionRole policy to the IAM role we created:

const rpaBasic = new aws.iam.RolePolicyAttachment("rpa-basic", {
 role: role,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
});

const rpaSecrets = new aws.iam.RolePolicyAttachment("rpa-secrets", {
 role: role,
 policyArn: secretManagerPolicy.arn,
});

The aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole is one of our enums that points at the relevant ARN for this policy.

Finally, the Function

There are two parts to this: the Lambda Function resource and the code we’re going to deploy as part of the function.

The Resource

The function resource itself is pretty simple. The interesting bit is how we take the array of the ARNs we created earlier, combine it with the array of environment variable values we have in the config and assign them to the environment input:

const lambdaEnvVars: pulumi.Input<{
 [key: string]: pulumi.Input<string>;
}> = {};

for (const key in secretsArnArray) {
 if (secretsArnArray.hasOwnProperty(key)) {
 lambdaEnvVars[key] = secretsArnArray[key];
 }
}

for (const key in lambdaconfig.envvars) {
 if (lambdaconfig.envvars.hasOwnProperty(key)) {
 lambdaEnvVars[key.toLocaleUpperCase()] = lambdaconfig.envvars[key];
 }
}

The reason that we’re using the .toLocaleUpperCase() function is because as a rule, the usual way of defining environment variable names is using upper case letters.

Once we’ve done this we can put together the function resource:

const lambda = new aws.lambda.Function("lambdaWithSecrets", {
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./app"),
 }),
 role: role.arn,
 handler: "index.handler",
 runtime: aws.lambda.Runtime.NodeJS16dX,
 environment: {
 variables: lambdaEnvVars,
 },
 timeout: 15,
});

export const lambdaName = lambda.name;

There’s nothing complicated about the function. We’re using the role from earlier, setting the environment variables to be the object we created just above and pointed the code input at a local folder called app, which we’ll look at next. The lambda name being exported at the end is so that we can use the Pulumi CLI to get this value and invoke the function using the AWS CLI.

The Code

First we need somewhere to put the code:

mkdir app && cd app

Then we create a new NPM application and install the AWS client-secrets-manager package that we’ll use to access the secrets in the code:

npm init -y && npm install @aws-sdk/client-secrets-manager

We’re also going to update the type setting in the package.json file to be module so that we can have a top level async function. Here’s what our file looks like:

{
 "name": "app",
 "version": "1.0.0",
 "description": "",
 "main": "index.js",
 "scripts": {
 "test": "echo \"Error: no test specified\" && exit 1"
 },
 "keywords": [],
 "author": "",
 "license": "ISC",
 "dependencies": {
 "@aws-sdk/client-secrets-manager": "^3.183.0"
 },
 "type": "module"
}

The index.js file for the lambda isn’t exciting. We’re going to make the function more efficient by making the calls to the secrets manager API outside of the main handler code, so it doesn’t get run for all invocations (just the ones during a cold start):

import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager"; // ES Modules import
const client = new SecretsManagerClient();

const getSecretForLambda = async (secretId) => {
 const input = { "SecretId": secretId };
 const command = new GetSecretValueCommand(input);
 console.log('Retrieving secret during top-level await');
 return await client.send(command);
}

const secret1 = getSecretForLambda(process.env.SECRET1);
const secret2 = getSecretForLambda(process.env.SECRET2);

export async function handler() {
 console.log('Using secret in handler');
 return {
 secret1: (await secret1).SecretString,
 secret2: (await secret2).SecretString,
 };
};

The above JavaScript code will return the secret values from the function and is purely for demo purposes. In the Real World^TM you definitely shouldn’t do this.

Deployment time

Now that we’ve got everything together we can deploy this:

pulumi up

and you should see something like the following:

Previewing update (dev)

View Live: https://app.pulumi.com/acmecorp/lambda-secrets/dev/previews/209f07fb-190d-4596-af9f-d2d72e9c5cc9

 Type Name Plan
 + pulumi:pulumi:Stack lambda-secrets-dev create
 + ├─ aws:secretsmanager:Secret secret1 create
 + ├─ aws:secretsmanager:Secret secret2 create
 + ├─ aws:iam:Role roleLambdaWithSecrets create
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret2 create
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret1 create
 + ├─ aws:iam:Policy secretsPolicy create
 + ├─ aws:iam:RolePolicyAttachment rpa-basic create
 + ├─ aws:iam:RolePolicyAttachment rpa-secrets create
 + └─ aws:lambda:Function lambdaWithSecrets create

Outputs:
 lambdaName: "lambdaWithSecrets-b8ac5e9"

Resources:
 + 10 to create

Do you want to perform this update? yes
Updating (dev)

View Live: https://app.pulumi.com/acmecorp/lambda-secrets/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack lambda-secrets-dev created
 + ├─ aws:secretsmanager:Secret secret1 created
 + ├─ aws:secretsmanager:Secret secret2 created
 + ├─ aws:iam:Role roleLambdaWithSecrets created
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret2 created
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret1 created
 + ├─ aws:iam:Policy secretsPolicy created
 + ├─ aws:iam:RolePolicyAttachment rpa-basic created
 + ├─ aws:lambda:Function lambdaWithSecrets created
 + └─ aws:iam:RolePolicyAttachment rpa-secrets created

Outputs:
 lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 + 10 created

Duration: 24s

Let’s see what this has done. If we go to look at the Lambda Function in the console, click on the Configuration tab and select Environment variables you’ll see the two plaintext environment variables and their values along with the two secret environment variables containing the ARNs to the relative secret in the Secrets Manager. Since we know the environment variable key (the uppercase version of what’s in the stack config), we can reliably use that same name in the function code (the two lines where we call the getSecretForLambda function).

For me, it looks something like this:

Finally, let’s run the function via the AWS CLI and see what we get:

aws lambda invoke --function-name $(pulumi stack output lambdaName) /dev/stdout

You should see something like the following (depending on what values you added to the stack config at the beginning):

{"secret1":"secret1value","secret2":"secret2value"}

You can also use the test function in the AWS Console and see the results there.

Tidying up

As stated earlier, it costs to store and access secrets so we need to make sure that we remove the infrastructure once we’ve completed this demo. As ever, you can do this by running pulumi destroy:

Previewing destroy (dev)

View Live: https://app.pulumi.com/pierskarsenbarg/lambda-secrets/dev/previews/f111f92e-1dbb-4615-9871-30de417bf8f1

 Type Name Plan
 - pulumi:pulumi:Stack lambda-secrets-dev delete
 - ├─ aws:iam:RolePolicyAttachment rpa-secrets delete
 - ├─ aws:iam:RolePolicyAttachment rpa-basic delete
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret1 delete
 - ├─ aws:iam:Policy secretsPolicy delete
 - ├─ aws:lambda:Function lambdaWithSecrets delete
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret2 delete
 - ├─ aws:secretsmanager:Secret secret2 delete
 - ├─ aws:iam:Role roleLambdaWithSecrets delete
 - └─ aws:secretsmanager:Secret secret1 delete

Outputs:
 - lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 - 10 to delete

Do you want to perform this destroy? [Use arrows to move, enter to select, type
Do you want to perform this destroy? [Use arrows to move, enter to select, type
Do you want to perform this destroy? yes
Destroying (dev)

View Live: https://app.pulumi.com/pierskarsenbarg/lambda-secrets/dev/updates/64

 Type Name Status
 - pulumi:pulumi:Stack lambda-secrets-dev deleted
 - ├─ aws:iam:RolePolicyAttachment rpa-secrets deleted
 - ├─ aws:iam:Policy secretsPolicy deleted
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret1 deleted
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret2 deleted
 - ├─ aws:lambda:Function lambdaWithSecrets deleted
 - ├─ aws:iam:RolePolicyAttachment rpa-basic deleted
 - ├─ aws:secretsmanager:Secret secret2 deleted
 - ├─ aws:secretsmanager:Secret secret1 deleted
 - └─ aws:iam:Role roleLambdaWithSecrets deleted

Outputs:
 - lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 - 10 deleted

Duration: 9s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
If you want to remove the stack completely, run `pulumi stack rm dev`.

And to complete the tidying up, let’s delete the stack and all the relevant history as well:

pulumi stack rm dev

Complete code

If you just want to copy and paste the entire index.ts file without worrying which section goes where, I got you covered:

// Copyright 2016-2022, Pulumi Corporation. All rights reserved.

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

interface LambdaConfig {
 envvars: string[];
 secrets: pulumi.Output<string>[];
}

const config = new pulumi.Config();
const lambdaconfig = config.requireObject<LambdaConfig>("lambdawithsecrets");

const secretsArnArray: pulumi.Output<string>[] = new Array();
const secretArray: aws.secretsmanager.Secret[] = new Array();
for (const key in lambdaconfig.secrets) {
 if (lambdaconfig.secrets.hasOwnProperty(key)) {
 const secret = new aws.secretsmanager.Secret(`${key}`);
 const secretVersion = new aws.secretsmanager.SecretVersion(`secretversion-${key}`, {
 secretId: secret.id,
 secretString: lambdaconfig.secrets[key],
 });
 secretArray.push(secret);
 secretsArnArray[key.toLocaleUpperCase()] = secret.id;
 }
}

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(aws.iam.Principals.LambdaPrincipal),
});

const rpaBasic = new aws.iam.RolePolicyAttachment("rpa-basic", {
 role: role,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
});



const secretManagerPolicyDoc = aws.iam.getPolicyDocumentOutput({
 statements: [
 {
 effect: "Allow",
 actions: [
 "secretsmanager:GetSecretValue",
 ],
 resources: secretArray.map(x => pulumi.interpolate`${x.arn}`),
 },
 ],
});

const secretManagerPolicy = new aws.iam.Policy("secretsPolicy", {
 policy: secretManagerPolicyDoc.apply(doc => doc.json),
});

const rpaSecrets = new aws.iam.RolePolicyAttachment("rpa-secrets", {
 role: role,
 policyArn: secretManagerPolicy.arn,
});

const lambdaEnvVars: pulumi.Input<{
 [key: string]: pulumi.Input<string>;
}> = {};

for (const key in secretsArnArray) {
 if (secretsArnArray.hasOwnProperty(key)) {
 lambdaEnvVars[key] = secretsArnArray[key];
 }
}

for (const key in lambdaconfig.envvars) {
 if (lambdaconfig.envvars.hasOwnProperty(key)) {
 lambdaEnvVars[key.toLocaleUpperCase()] = lambdaconfig.envvars[key];
 }
}

const lambda = new aws.lambda.Function("lambdaWithSecrets", {
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./app"),
 }),
 role: role.arn,
 handler: "index.handler",
 runtime: aws.lambda.Runtime.NodeJS16dX,
 environment: {
 variables: lambdaEnvVars,
 },
 timeout: 15,
});

export const lambdaName = lambda.name;

Summary

This post describes one of the ways that you can securely access secrets in your Lambda Functions by adding secrets to your stack config, uploading them to AWS Secrets Manager and then accessing them in your functions, all without exposing the plaintext values. There are other ways, but I think this is one of the neatest.

Of course you can use other tools to manage your secrets, such as Azure Key Vault, Google Cloud Secret Manager or Hashicorp Vault, getting the path to the key back from the resource and using it in the same way as we used the ARN from the Secret.

If you’d like more information on how you can use Pulumi to securely and easily deploy your cloud infrastructure using modern programming languages, get in touch and ask us how.

Preview of the Manage Track at Cloud Engineering Summit 2021

Matty Stratton — Mon, 13 Sep 2021 08:00:00 -0500

The Cloud Engineering Summit 2021 is coming up fast, and the speakers are out! To get you ready to attend, let’s take a look at the sessions for the Manage track.

The Cloud Engineering Summit’s three tracks are built around three concepts: Build, Manage, and Deploy. I’m Matt Stratton, and I’m your charismatic track chair for Manage. For us, that means managing cloud applications and infrastructure with Policy as Code, visibility, and access controls. For example, managing infrastructure with policies that detect configuration drift, enforce best practices, and even prevent compliance violations before deployment. It means building visibility across your cloud infrastructure so that you always understand its current and past states, including detailed audit history. Finally, you ensure the right guardrails and controls are set in place so that distributed teams can securely develop.

Without further ado, let’s take a look at each of the talks I’ve selected for you!

Overwhelmed by permissions and authorization? Me, too. Fortunately, we have Joy Ebertz presenting Authorization: Ensuring Only Ada Can Access Her Files. Joy is going to cover the various authorization models available to us, with use-cases for each. Authorization can be overwhelming if you don’t have a good grasp of the options available to your organization and an understanding of what works when, so I’m excited for this one.

Taking a higher-level view, Quintessence Anx will be presenting DevSecOps and Secure Incident Response. The introduction of security to the DevOps equation means some shifts in the way we think about the software development lifecycle, and Quintessence is uniquely qualified to talk about why that’s true. A look inside PagerDuty’s 14 Step Secure Incident Response process promises to be interesting for anyone involved in either development or operations.

I don’t think we have the ability to predict the future, but J. Paul Reed wonders if we can develop the ability to forecast incidents just as we have learned to have staggeringly accurate weather short-term forecasts. With Forecasting the Future: Creating a Radar for Risk, Paul will share with us how Netflix uses Risk Radar to collect and make sense of risk in their sociotechnical systems.

When we talk about infrastructure as code, there’s more to it than just the code! Fariba Khan and Stephen Van Gordon will take us on a journey with Lifecycle of a Pulumi Program to share how they leverage a custom state backend with SSO, RBAC, and programmatically configurable pipelines powered by CI/CD tooling and the Pulumi Automation API to drive IaC at their organization.

Snowflake’s Charles Xu will be here to talk about Day-two Operation of Multi-cloud Kubernetes and Vault, a very cool overview of Snowflake’s Kubernetes infrastructure and how Pulumi helps, from blue-green updates, to managing Hashicorp Vault, to rotating TLS certificates. This is a pretty complex topic, so I’m very interested to see where it goes!

Gremlin’s Jason Yee comes to us with Trailblazers: exploration, discovery, & navigating failure - a set of topics that I personally find fascinating and essential! I think we are constantly in a state of discovery and understanding of our complex systems, and learning how to explore them - and work through the inevitable failures - helps us build and operate even more reliable systems.

Rounding things out is the always incredible Tim Banks, with Toxic Velocity: Speed Kills. In this talk, Tim is going to cover the various stressors that put pressure on both organizations and individuals, how they affect us, and what it looks like when things start to spiral out of control. I’m excited for this one because while crashes do happen, we’d all like them to happen less often (or not at all!), and being able to identify some of these things before they become a real problem is better for our businesses and our people.

Explore all of the rest of the great sessions, and don’t forget to register for the cloud engineering summit!

Easily bring your team to Pulumi with SAML SSO and SCIM

Alex Mullans — Tue, 20 Apr 2021 00:00:00 -0700

Pulumi’s Cloud Engineering Platform helps teams of all sizes deliver and manage cloud apps and infrastructure. In the Pulumi Service, everyone on the team can see the infrastructure the team is responsible for, when it was last deployed, how it’s configured, and more. You can see a full breakdown of the infrastructure and understand how the team brings together individual cloud services to create applications. When you bring your teams together on Pulumi, you get a “single pane of glass” over all the cloud applications and infrastructure managed with Pulumi.

Most teams larger than a few people define their team members, and the groups they’re a part of, using an Identity Provider (IdP) like Okta, Microsoft’s Azure Active Directory, or Google Cloud Identity & Access Management. Defining groups allows you to set policies based on logical groupings of teammates, such as giving different capabilities to individual developers versus infrastructure administrators. Pulumi works seamlessly with these IdPs (and many more) by providing Single Sign-On with SAML and user and group synchronization via SCIM 2.0. When you use both SAML SSO and SCIM, you can manage the users who should have access to Pulumi and the Pulumi teams they belong to entirely from your IdP. Managing users and teams this way helps keep your organization more secure by ensuring a single point of control over the users who have access to manage and update your infrastructure. It also saves you time by centralizing all of your identity and access management workflows in your identity provider.

SAML SSO and SCIM support are only available in Pulumi Enterprise. See pricing for more details.

SAML SSO

In a world of many cloud applications and service providers, IT departments can use single sign-on with SAML to make it easier for their employees to sign in and be productive. In addition, if an employee leaves, all of that employee’s access can be disabled from a single pane of glass, rather than having to manually update each application and service provider separately. Pulumi’s support for single sign-on with SAML provides this functionality. We support any IdP that implements the SAML 2.0 specification, with specific integrations for Azure Active Directory, Okta, and GSuite.

To configure single sign-on with SAML for your Pulumi organization, use the guide that matches your IdP:

After configuring your IdP, any and all changes, either to your cloud applications and infrastructure, or your account’s settings, will be tracked and associated with the user who performed that activity. The end result is full end-to-end visibility and attestation for important changes to your Pulumi organization.

User and group synchronization via SCIM

After you’ve configured single sign-on with SAML, you can take it one step further by synchronizing your users and groups using SCIM. Synchronization extends the power of single sign-on by enabling organization administrators to use their IdP to manage the users who have access to Pulumi and which teams each user belongs to. Granting and revoking user access is seamless and automatically provides the right level of permissions to critical infrastructure.

To configure user and group synchronization via SCIM, use the guide that matches your IdP:

Distributed, least privilege access permissions model

In addition to support for standards like SAML and SCIM, the Role-Based Access Control (RBAC) permissions model enables you to set up a least privilege access model for your infrastructure resources. In such a model, all access to infrastructure is specifically granted to individual users by creating Teams that have the appropriate level of access (Read, Write, or Admin) to a set of Stacks, then adding users to those teams. Team membership can be managed manually by Team Admins or automatically using SCIM group synchronization.

Learn how to create Teams in the Teams docs.

Next steps

The Pulumi Cloud Engineering Platform’s support for the SAML and SCIM standards for SSO, user synchronization, and group synchronization makes it easy to ensure every team member has the right access and the right permissions, allowing you to empower the right teammates while still applying the principle of least privilege. Here’s how to get started:

If you’re an existing Pulumi customer, you can enable SAML in your organization settings.
If you’re new to Pulumi, start your trial now or contact sales to request a demo.

Keeping Your Secrets Secret

Sophia Parafina — Tue, 23 Mar 2021 00:00:00 +0000

Companies that have suffered data breaches are, unfortunately, frequently in the news. A data breach is when information that should be private, such as credit card numbers or even trade secrets, is stolen. These thefts can be because of an actual cyber-attack, but they can also be due to simple carelessness, such as disposing of computer equipment without taking proper precautions.

As an example, Bitcoin miners have perpetrated many data breaches. Suppose you accidentally push a dotfile with an aws_access_key_ID to GitHub. Bitcoin miners have a host of tools that automatically scrape GitHub for exactly that sort of information. Once they have your AWS credentials, they lock you out, spin up machines and start mining. This isn’t as common as it once was since cloud providers are better at detecting these sorts of incursions, but you can see how catastrophic (and costly) such a data breach can be.

Of course, you can protect yourself and your company. In this blog post, we’re going to give a quick rundown of some actions you can take to make sure your secrets stay secret.

What Is a Secret?

We’re going to define secrets as small, critical pieces of information. They’re no larger than a few KBs, and they’re necessary to run your infrastructure. If they’re leaked, your infrastructure is compromised and bad things can happen. Some examples of secrets are passwords, API keys, and SSH keys.

Remove All Plaintext Secrets

Step one is to search through your existing codebase and remove any secrets that are in plaintext. A simple approach is to manually grep or search through your files, looking for words such as username and password. However, there are also tools available that will comb through your Git history and commits, looking for sensitive information. Many of these tools are open source.

Build detection into your CI/CD pipeline

Making secrets detection a part of your CI/CD pipeline is a great way to ensure you’re not pushing confidential information to a public repo. There are tools that integrate into your automated workflow. You can configure those tools to stop the pipeline if they detect secrets such as keys or passwords.

Don’t Forget about Git Commits

It’s easy to forget that Git commits can contain sensitive information. For example, someone might talk about a security issue and how they fixed it. Instead, give a link to your ticketing system and document the problem there.

Encrypt and Manage Your Secrets

After you’ve stripped out your plaintext secrets from your codebase, you still need to to store and manage them securely. Encrypting secrets rather than storing them as plaintext is a good start.

Manual encryption

There are command-line tools for encrypting secrets, such as GPG (also called GnuPG) and PGP, which stands for Pretty Good Privacy. These are free, but many people find them difficult to use, and standard security practices such as rotating your keys can be challenging to implement.

Secret servers

There are many servers that store secrets securely available on the market. Secret servers encrypt your secrets and usually have many other features that encourage best security practices, such as access control lists and the ability to audit who’s accessing your keys.

Cloud-provided services

Given that many companies have their infrastructure in the cloud, it makes sense to consider using the secrets services that your cloud provider offers. To mention a few, Google Cloud has Secret Manager, AWS has the AWS Secrets Manager, which relies on the AWS Key Management Service (KMS), and Microsoft Azure has Key Vault. These services have many built-in capabilities that enable you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.

Have a Plan

Mistakes happen and, even if you’re careful, leaks can occur. It’s important to have a plan already in place if you’ve detected a data breach. Here are some of the things you should do.

Reset passwords and create new keys.
Monitor your systems to find anomalies and unusual usage.
Destroy as many machines as you can and spin up new instances with new credentials.
Keep track of what you’ve done so that you can have a meaningful post-mortem once you’ve put out the fires.

If you haven’t done so already, consider installing intrusion-detection systems. There are many on the market, including open-source tools. There’s no single answer for what to do when leaks happen, but it’s important that all the stakeholders get together and put a process in place.

Practice makes perfect

Once you have a plan, consider staging a data breach and follow that plan to see how well it works. For example, you might simulate a spoofing attack or have someone “steal” a laptop and see how much damage they can do.

Move Security to the Left

Security should be as much a part of your workflow as any other phase of the development process. Just as you start testing your code early to find bugs, you should test early to find security problems. Security shouldn’t come at the end of the development cycle when you think you’re production-ready. Just as with any other problem, security issues are much easier to fix early in development.

Learn More

Pulumi takes security risks seriously. That’s why our platform is secure by default. It protects all secret data using Pulumi’s built-in secret storage, a secret store that “just works.” We also support all the major cloud-provided secret services. Want to learn more? Visit pulumi.com or get started for free today.

Easily bring your team to Pulumi with SAML SSO and SCIM

Alex Mullans — Wed, 24 Feb 2021 00:00:00 +0000

The Pulumi Service helps teams of all sizes deliver and manage cloud apps and infrastructure. In the console, everyone on the team can see the infrastructure the team is responsible for, when it was last deployed, how it’s configured, and more. They can see a full breakdown of the infrastructure as well, so they can understand how the team brings together individual cloud services to create their applications. When you bring your teams together on the Pulumi Service, you can provide a “single pane of glass” over all the infrastructure that you manage with Pulumi.

Most teams larger than a few people define their team members, and the groups they’re a part of, using an Identity Provider (IdP) like Okta, Microsoft’s Azure Active Directory, or Google Cloud Identity & Access Management. The Pulumi Service works seamlessly with these IdPs (and many more) by providing Single Sign-On with SAML and user and group synchronization via SCIM 2.0.

When you use both SAML SSO and SCIM, you can manage the users who should have access to Pulumi and the Pulumi teams they belong to entirely from your IdP. Managing users and teams this way helps keep your organization more secure by ensuring a single point of control over the users who have access to manage and update your infrastructure. It also saves you time by centralizing all of your identity and access management workflows in your identity provider.

SAML SSO

In a world of many cloud applications and service providers, IT departments can use single sign-on with SAML to make it easier for their employees to sign in and be productive. In addition, if an employee leaves, all of that employee’s access can be disabled from a single pane of glass, rather than having to manually update each application and service provider separately. The Pulumi Service’s support for single sign-on with SAML provides this functionality. We support most IdPs that implement the SAML 2.0 specification.

To configure single sign-on with SAML for your Pulumi organization, use the guide that matches your IdP:

User and group synchronization via SCIM

After you’ve configured single sign-on with SAML, you can take it one step further by synchronizing your users and groups using SCIM. Synchronization extends the power of single sign-on by enabling organization administrators to use their IdP to manage the users who have access to the Pulumi Service and which Pulumi teams each user is a member of. Granting and revoking user seat licenses is seamless and easily provides the right level of permissions to critical infrastructure.

To configure user and group synchronization via SCIM, use the guide that matches your IdP:

Next steps

The Pulumi Cloud’s support for the SAML and SCIM standards for SSO, user synchronization, and group synchronization make it easy to ensure every team member has the right access and the right permissions. Here’s how to get started:

If you’re an existing Pulumi customer, you can enable SAML in your organization settings.
If you’re new to Pulumi, start your trial now or contact sales to request a demo.

Keep your secrets secure, by default

Alex Mullans — Fri, 19 Feb 2021 00:00:00 +0000

An unauthorized user gaining access to your infrastructure can be catastrophic: data can be stolen or leaked, security holes can be exploited, and more. That risk makes it critical to keep the infrastructure secrets—the passwords, access tokens, keys, and so on—well-protected. This is particularly true in automated systems, like continuous integration and delivery and infrastructure-as-code systems.

Pulumi’s built-in secret storage

Pulumi takes these risks seriously. That’s why we built the Pulumi platform to be secure by default. It protects all secret data using Pulumi’s built-in secret storage. This secret store “just works”, regardless of whether you use our hosted Pulumi Service, the Self-Hosted Pulumi Service, or the self-managed backend. The Pulumi Service protects secrets automatically when you create a stack with pulumi new or pulumi stack init. If you use a self-managed backend like AWS S3 or Google Cloud Storage, secrets are protected by a passphrase that you choose when you start a new stack with pulumi new. In all cases, every Pulumi stack gets a unique encryption key.

What secrets are protected

Pulumi protects all of the state metadata that it knows about your infrastructure with encryption: encryption in transit when it’s sent to your backend and encryption at rest (if you’re using the hosted or self-hosted Pulumi Service). This base layer of protection applies to all state, including both non-secret and secret metadata. Secret storage is the next level of protection. It ensures that secret values are never exposed in plaintext in any file that Pulumi writes to your disk. Secrets are identified in a few ways:

All Pulumi providers—both native providers like azure-native and Terraform-based providers like Datadog—know which of their outputs should be secret and mark them as such. This ensures that you’re secure by default when you start using a new provider. There’s no need to pick which outputs are secret or worry that you’ve missed one.
You can mark other Pulumi outputs as secret.
In addition to Pulumi inputs and outputs, you can also mark Pulumi configuration values as secret.

Any secret identified by you or by Pulumi is fully protected in the secret storage and not stored in plaintext on disk. This prevents leaks of secrets when files are committed to a source code repository or uploaded to a continuous integration and delivery system.

Integration with other secret stores

For smaller organizations, the secret storage built into Pulumi is usually sufficient. In larger organizations however, it’s likely that a centralized secret management system is already in use, like AWS Key Management System, Azure Key Vault, Google Cloud Key Management, or Hashicorp Vault. Pulumi integrates with each of these systems to make it easy to protect secrets in the way required by the organization.

Next steps

It’s easy to get started with secrets management in Pulumi, because there’s nothing extra you need to do. Begin with any one of our get started tutorials and Pulumi will automatically protect your secrets along the way.

Learn more about Pulumi’s secret storage in our docs

Rotating Secret Providers

Paul Stack — Wed, 05 Aug 2020 00:00:00 +0000

Customers and users have asked for the ability to change the secrets manager associated with their stacks. This would allow a user to rotate their secrets providers when people leave their organization or even to be able to migrate to another secret manager of their choice. The v2.8.0 release of Pulumi adds support for this specific feature.

Let’s have a look at how to change a secrets provider for an existing stack:

pulumi stack change-secrets-provider "awskms://alias/ChangeSecretsProvider?region=us-west-2"

This command will change the secrets provider for the currently selected stack. It will ensure that the configuration values in Pulumi.<stack>.yaml are re-encrypted with the new secrets provider and that the latest checkpoint (state file) also has any secrets held within are migrated with the new secrets provider.

Creating New Stacks based on existing stacks

When creating a new stack, we added the ability to create that new stack based on the configuration of an existing stack. This means that when you need to create a new developer stack or a new stack for a different environment, then you can create that stack as follows:

pulumi stack init test --copy-config-from dev

This flag creates a new test stack and initiates the default secrets provider for the stack. It will then get the existing configuration from the dev stack and re-encrypt it to the test stack’s default secret provider. We can, of course, pass a secrets provider to this new stack:

pulumi stack init test --secrets-provider="awskms://alias/ChangeSecretsProvider?region=us-west-2" --copy-config-from dev

Copying Config Between Stacks

When working on a Pulumi project that has multiple stacks, there are times that we create a configuration in one of the stacks that need to be copied to the other stacks in the project. Rather than selecting each stack individually, we can now programmatically copy the configuration between stacks. Let’s take an example where a new database password was added to the dev stack. We can copy that password to the production stack as follows:

pulumi config cp MyDatabasePassword --dest production

This ensures that the MyDatabasePassword configuration will be re-encrypted into the production stack. If there’s a situation where we want to copy all of the configuration between the dev and production stacks, then we omit the key (MyDatabasePassword):

pulumi config cp --dest production

We can iterate over all of the stacks that we need to copy the configuration to and the new configuration values will be encrypted as per the new stack.

Learn More

If you’d like to learn about Pulumi and manage your infrastructure through code, get started today. Pulumi is open source and free to use.

As always, you can check out our code on GitHub, follow us on Twitter, subscribe to our YouTube channel, or join our Community Slack channel if you have any questions, need support, or just want to say hello.

Managing AWS Credentials on CI/CD: Part 3

Chris Smith — Thu, 14 May 2020 00:00:00 +0000

This article is the third part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we cover the last leg of the continuous delivery process to update your AWS resources and how to store sensitive data using Pulumi securely.

Recap

In part 1, we created a dedicated IAM User with limited privileges. Then in part 2, we set up a simple, serverless Pulumi program that periodically rotated the User’s access keys and updated the CI/CD system.

In this final post, we will use those AWS credentials to update cloud resources as part of a CI/CD workflow using Pulumi.

We will cover AWS’s IAM Roles and how they can be used to safely manage the access Pulumi has to your AWS account. We will also cover how sensitive data in your stack is stored within the Pulumi Service.

Assuming IAM Roles for Performing Updates

In the previous post, we automatically create and rotate the AWS access keys for an IAM User. So it seems natural to think that we should just use that same AWS access key to gain access to AWS resources. However, using the access keys for an IAM User has a few major drawbacks.

First, those access keys are long-lived. If we didn’t create that Pulumi program to rotate them automatically, the key would last forever. (Meaning that a bad actor, perhaps elsewhere, or far in the future, could take advantage of a compromised key.) Second, that access key could be used to do anything that the IAM User could. So you cannot differentiate the set of capabilities based on the task at hand. (For example, if updating the Pulumi stack website/staging, we might need a different set of permissions than if we were updating the stack website/production.)

The preferred alternative to using IAM User access keys is to “assume” an IAM Role. An IAM Role is similar to an IAM User in that it grants a set of policies and permissions. However, unlike an IAM User, an IAM Role does not have a password, nor can it be added to a Group and inherit any policies. Also, the credentials for using an IAM Role are always short-lived. (Lasting no more than 12 hours, and likely much, much less.)

Instead, an IAM Role defines a set of capabilities that can be performed by any principal capable of assuming it. So in other words, you can exchange your limited access IAM User’s credentials for a specialized IAM Role. The IAM Role then can be specifically tailored to a specific— wait for it— role to perform.

Don’t worry if that doesn’t all make sense at first; let’s look at a concrete example.

Desired End State

In our situation, we want to create an IAM Role WebsiteStackUpdaterRole that grants permissions necessary for Pulumi to update the website/production stack. We will custom tailor this role to only grant access to the set of resources and operations we know the stack needs.

We also want to restrict who can assume the WebsiteStackUpdaterRole, so that only specific users/accounts can access it. e.g., the IAM User cicd-bot but not IAM User dwight-over-in-sales.

And just to show a more real-world example, we’ll do this using multiple AWS accounts, where the website/production stack’s resources are housed in a “production” AWS account, and the cicd-bot IAM User is defined in a bastion AWS account.

Creating the IAM Role

The following snippet creates an IAM Role resource using Pulumi. The most important input property of which is the assumeRolePolicy document, which defines who can assume this role.

This is a situation where Pulumi’s ability to declare the policy document in code is super-useful, as it makes it much easier to reference other resources (e.g., another AWS account’s ID) as well as supporting inline comments.

// IAM Role used by cicd-bot to update the website stack.
export const websiteStackUpdaterRole = new aws.iam.Role("WebsiteStackUpdaterRole", {
 name: "WebsiteStackUpdaterRole",
 description: "Broad access for updating Pulumi stacks in the 'production' AWS account.",
 maxSessionDuration: 30 * 60, // 30 minutes
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: {
 // We grant the "bastion account", which defines IAM Users the ability to assume this role.
 // The IAM Policies in that other AWS account will then determine which users, groups, etc.
 // will be able to use the `sts:AssumeRole` action on this IAM Role.
 "AWS": `arn:aws:iam::${awsAccountIds.bastionAccount}:root`,
 },
 // But as an additional precaution against misconfiguration in the bastion account, we add
 // some additional checks to determine who can access this policy.
 Condition: {
 "ForAllValues:StringEquals": {
 "aws:PrincipalTag/access-to-production": "true",
 "sts:ExternalId": [
 "contoso/website/production",
 "contoso/website/staging",
 ],
 },
 },
 Action: "sts:AssumeRole",
 },
 ],
 },
});

There’s a lot packed into that IAM Role’s configuration.

First, the “principal” (who can assume the role) references a specific AWS account. This is an example of how you can associate one AWS account with another. (And is best practice for isolating data between different departments, etc.) We’ll cover how to configure the AWS bastion account side to use this role later.

The next thing to call out is the use of the Condition element of the policy document. To assume the role, we require two checks to be performed.

That the principal assuming the role (e.g., IAM User) has the tag "access-to-production" with value "true." This way, we will reject any IAM User in the bastion account who isn’t tagged with "access-to-production" from assuming this role.
We also verify that the “external ID” (an optional parameter passed when assuming an IAM Role) matches a well-known Pulumi stack name. This way, we can double-check that the entity using the IAM Role is doing so to use a specific stack. (This is less of a security measure and more of a general hygiene type requirement.)

IAM Policy Document

Now we have an IAM Role; we need to attach a policy document that defines what permissions are granted.

This is the most important step since it defines precisely what the CI/CD bot can able to do.

If we define the policy too broadly, then it is a general risk. For example, what if a compromised NPM package in your Pulumi program did something nefarious. If the policy is defined too narrowly, then your pulumi up might fail because the necessary permissions weren’t available. When in doubt, err on the side of having a more restrictive policy.

It can be advantageous to have an identical “staging” and “production” instance of the same Pulumi stack, so you can detect any permission issues before you update the user-facing production stack.

It can take some time to winnow down the specific set of permissions necessary since that is ultimately specific to the Pulumi program. For example, if the stack governs the networking configuration, then you can simply not deny to the s3:* actions.

Unfortunately, Pulumi doesn’t currently support generating a reasonable IAM Policy document for your stack. For a good introduction to some of the “worst-case scenarios” that could arise from a misconfigured IAM Policy, see this blog post from BishopFox.

The following is a hypothetical IAM policy that restricts the operations performed to just those for the AWS CloudFront and S3 products. (Which might be sufficient for updating a CDN-based website hosted on AWS, but likely not much else.)

// Policy document defining the permissions granted to the WebsiteStackUpdaterRole.
export const websiteUpdaterPolicyDocument: aws.iam.PolicyDocument = {
 Version: "2012-10-17",
 Statement: [
 // Grant access to all AWS S3 actions, but scoped to just the specific S3 buckets
 // used for the given Pulumi stack.
 {
 Sid: "GrantS3Access",
 Effect: "Allow",
 Action: "s3:*",
 Resource: [
 // Content bucket.
 "arn:aws:s3:::website-bucket",
 "arn:aws:s3:::website-bucket/*",
 // S3 bucket which stores the access logs.
 "arn:aws:s3:::website-logs-bucket",
 "arn:aws:s3:::website-logs-bucket/*",
 ],
 },
 // Grant access to update a CloudFront distribution. For more information, see:
 // https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/access-control-managing-permissions.html
 {
 Sid: "GrantCloudFrontAccess",
 Effect: "Allow",
 Action: [
 "cloudfront:CreateDistribution",
 "cloudfront:DeleteDistribution",
 "cloudfront:GetDistribution",
 "cloudfront:GetDistributionConfig",
 "cloudfront:ListDistributions",
 "cloudfront:UpdateDistribution",
 "cloudfront:ListCloudFrontOriginAccessIdentities",
 ],
 // The specific CloudFront distribution used in the stack.
 Resource: "arn:aws:cloudfront::0123456789:distribution/A123B1CABWZZ",
 },
 ],
};

const websiteUpdaterPolicy = new aws.iam.Policy("WebsiteUpdaterRolePolicy", {
 name: "WebsiteUpdaterRolePolicy",
 description: "Policy granting the permissions needed for updating the website/production Pulumi stack",
 policy: websiteUpdaterPolicyDocument,
}, resourceOptions);

To actually associate the IAM Policy document with the new IAM Role, we need to create a PolicyAttachment associating the two.

const rolePolicyAttachment = new aws.iam.RolePolicyAttachment("WebsiteStackUpdaterRolePolicyAttachment", {
 role: websiteStackUpdaterRole,
 policyArn: websiteUpdaterPolicy.arn,
});

Reminder, we defined the IAM Role in our production AWS account, which allows any principal in the AWS bastion account (with the right tag, external ID) to assume that role. We then need to update the bastion account to grant the sts:AssumeRole action on that IAM Role so it can be used! That isn’t covered in this post.

Assuming the IAM Role

Finally! With everyone in place, we can now use the IAM Role.

The following is a set of commands you can run on your CI/CD worker to exchange the low-privilege IAM User credentials for the specific IAM Role we just created in the production AWS account. But there are many alternatives for programmatically assuming an IAM Role, such as using the AWS_PROFILE environment variable) or the assume-role tool.

# Use the AWS credentials passed to the CI system and available as environment variables.
# These are the low-privilege credentials that automatically get rotated.
aws configure set aws_access_key_id ${AWS_ACCESS_KEY_ID}
aws configure set aws_secret_access_key ${AWS_SECRET_ACCESS_KEY}
aws configure set region "${AWS_REGION:-us-west-2}"

# Assume the desired IAM Role. Note how we pass the stack name in the --external-id parameter,
# as required to assume the role.
readonly CREDS_JSON=$(aws sts assume-role \
 --role-arn "${ARN_OF_IAM_ROLE_TO_ASSUME}" \
 --role-session-name "<useful description of the context, such as CI/CD job ID>" \
 --external-id "${TARGET_STACK}")

# The `aws sts assume-role` command just returns a JSON blob. So we now tease out the actual
# access key and set the corresponding environment variables.
export AWS_ACCESS_KEY_ID=$(echo "${CREDS_JSON}" | jq ".Credentials.AccessKeyId" --raw-output)
export AWS_SECRET_ACCESS_KEY=$(echo "${CREDS_JSON}" | jq ".Credentials.SecretAccessKey" --raw-output)
export AWS_SESSION_TOKEN=$(echo "${CREDS_JSON}" | jq ".Credentials.SessionToken" --raw-output)

# If everything worked correctly, then running the get-caller-identity command will refer to
# the IAM Role and not our IAM User.
echo "Assumed AWS identity:"
aws sts get-caller-identity

Storing Sensitive Data using Pulumi

As a developer, the most critical security threat is access to your cloud account since that’s ultimately where your or your customer’s data is stored. But beyond that, there are places where Pulumi stores sensitive data.

Pulumi goes to great lengths to safeguard your data. Here we review where Pulumi interacts with your sensitive data and what you can do to safeguard it better.

Secrets in Configuration

A Pulumi stack can have configuration data associated with it, which allows you to parameterize cloud resources. For example, a machine-size configuration key might determine which class of virtual machine to use. A dev stack might only need a "t2.small" but your production instance might need something more powerful.

Configuration data is stored in a file like Pulumi.website-production.yaml should be checked in along with the source code for a stack, so that builds and stack updates are reproducible.

However, what if those configuration settings contain secrets? Like you need to store an API key to use a 3rd party API.

Pulumi supports encrypting sensitive configuration data. You just need to add the --secret flag.

# Securely add an API key to the stack's configuration.
pulumi config set api-key "hunter2" --secret

For stacks hosted on the Pulumi Service, the default is that your configuration data is encrypted using a key specific to your stack. (So the ciphertext stored in the Pulumi.yaml file is safe to check into your source tree, since it cannot be copied/decrypted for another stack.)

Secrets in Checkpoint Files

Pulumi keeps track of your cloud resources in a something called a checkpoint file, and that too might contain sensitive information. For example, a Pulumi resource might have a "password" output property.

Pulumi has support to mark that resource output as “secret” and make sure that it is encrypted within the checkpoint file. (So if you were to look at the checkpoint file contents via pulumi stack export, you would not be able to recover the data.

Just like for secret configuration values, the default for stacks hosted on the Pulumi Service is to encrypt this data using a key that Pulumi manages and is specific to your stack.

Custom Secrets Providers

We at Pulumi take great care in safeguarding your data, especially configuration or checkpoint data that is marked as sensitive. However, you might feel more comfortable if your stack’s data were encrypted using a key that you controlled. (So even if the data stored in the Pulumi Service were available, it would be useless without your specific key.)

If that’s the case, then you can use a configurable secrets provider, and swap out the default “Pulumi Service managed” encryption scheme for your own. (And we won’t take it personally, promise.)

When you create a new stack using pulumi stack init, you can optionally specify a --secrets-provider flag. That will determine where and how secrets get managed on your stack.

For example, to use your own KMS key for encrypting data, you can pass the secrets provider "awskms://alias/ExampleAlias?region=us-east-1". The Pulumi command-line will work and behave the same. Except that whenever it needs to encrypt or decrypt some data, it will refer to the custom provider.

As you would expect, if you do use a custom secrets provider and lose access to your encryption key, then there is no way to recover the encrypted data stored on your stack.

For more information on custom secret providers, see Peace of Mind with Cloud Secret Providers or Managing Secrets with Pulumi.

Wrapping Up

Securely managing any sensitive data is challenging to get right, especially if there are many moving parts, such as when coordinating access between your AWS account, CI/CD provider, and your Pulumi stack.

In this blog series, we walked through the best practices in creating, securing, and managing access to an AWS account using IAM Users and Roles. We then reviewed the specifics of where and how Pulumi stores sensitive data, in case you want to take more control over your data.

If you have any other questions, corrections, or other ideas you’d like to discuss about Pulumi, Continuous Delivery, or security, please join the conversation over on the Pulumi Community Slack.

Peace of Mind with Cloud Secret Providers

Lee Briggs — Thu, 07 May 2020 00:00:00 +0000

The secrets in your infrastructure are a vital part of your security model, and provisioning infrastructure is an inherently privileged process. Previously we introduced secret encryption and started encrypting secret configuration values inside the Pulumi state so that users could be confident their passwords, tokens, and other secret values were viewable only by them while managing their infrastructure. Our first iteration of the encryption used either a passphrase for encrypting the secret or encryption via the Pulumi service backend. However, these options didn’t meet the needs of our users who needed more control over their data. That’s why we also added support for “Cloud Secret Providers,” giving users full confidence that their sensitive values are for their eyes only.

Pulumi supports encryption via the Pulumi service, AWS KMS, Azure KeyVault, Google Cloud KMS and HashiCorp Vault. This post shows you one example of using a cloud secret provider in a Pulumi stack using AWS KMS.

Create a KMS Key

First, create a KMS key. We also can set an alias on the key to make it easier to reference later:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create a new KMS key
const key = new aws.kms.Key("stack-encryption-key", {
 deletionWindowInDays: 10,
 description: "KMS key for encrypting Pulumi secret values",
});

// Create a new alias to the key
const alias = new aws.kms.Alias("alias/stack-encryption-key", {
 targetKeyId: key.keyId,
});

// Export the arns
export const keyArn = key.arn
export const aliasArn = alias.arn

"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");

// Create a nw KMS Key
const key = new aws.kms.Key("stack-encryption-key", {
 deletionWindowInDays: 10,
 description: "KMS key for encrypting Pulumi secret values"
});

// Create an alias to the key
const alias = new aws.kms.Alias("alias/stack-encryption-key", {
 targetKeyId: key.keyId
});

// Export the arns
exports.keyArn = key.arn;
exports.aliasArn = alias.arn;

import pulumi
from pulumi_aws import kms

# Create a new KMS key
key = kms.Key("stack-encryption-key",
 deletion_window_in_days=10,
 description="KMS key for encrypting Pulumi secret values"
)

// Create an alias to the key
alias = kms.Alias("alias/stack-encryption-key",
 target_key_id=key.key_id
)

# Export the arns
pulumi.export('key_arn', key.arn)

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v2/go/aws/kms"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 // Create a new KMS key
 key, err := kms.NewKey(ctx, "stack-encryption-key", &kms.KeyArgs{
 Description: pulumi.String("KMS key for encrypting Pulumi secret values"),
 DeletionWindowInDays: pulumi.Int(10),
 })
 if err != nil {
 return err
 }

 // Create an alias to the key
 alias, err := kms.NewAlias(ctx, "alias/stack-encryption-key", &kms.AliasArgs{
 TargetKeyId: key.KeyId,
 })
 if err != nil {
 return err
 }

 // Export the arns
 ctx.Export("keyArn", key.Arn)
 ctx.Export("keyAlias", alias.Arn)
 return nil
 })
}

using Pulumi;
using Kms = Pulumi.Aws.Kms;

class KeyStack : Stack
{
 public KeyStack()
 {
 // Create a new KMS key
 var key = new Kms.Key("stack-encryption-key", new Kms.KeyArgs
 {
 DeletionWindowInDays = 10,
 Description = "KMS key for encrypting Pulumi secret values"
 });

 // Create a new alias to the key
 var alias = new Kms.Alias("alias/stack-encryption-key", new Kms.AliasArgs
 {
 TargetKeyId = key.KeyId
 });

 this.KeyArn = key.Arn;
 this.AliasArn = alias.Arn;
 }

 [Output("keyArn")] public Output<string> KeyArn { get; set; }
 [Output("aliasArn")] public Output<string> AliasArn { get; set; }
}

Creating the key is enough to allow us to start using it for our encryption provider. However, we need to consider who has access to the key before we start encrypting our sensitive values with it in our Pulumi programs.

Scoping Permission to our Key

Generally, in AWS, you scope access to resources using IAM roles. However, for sensitive values like KMS keys, IAM roles alone aren’t enough to provide the security you might need. As an example, if someone in your AWS account has an IAM role with the following policy attached:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "Stmt1588215924595",
 "Action": "kms:*",
 "Effect": "Allow",
 "Resource": "*"
 }
 ]
}

That user gets access to every KMS key in your account, which would also mean they could decrypt any secret in your Pulumi stack.

To rectify this, we need to attach a Key Policy to the key. We can do this by updating our previous code:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const config = new pulumi.Config();
const iamRole = config.require("iamRole");
const accountID = config.require("accountID");


const keyPolicy = {
 Version: "2012-10-17",
 Id: "policy",
 Statement: [
 // This statement allows all users to view the key in the console
 {
 Sid: "AllowGetKeys",
 Effect: "Allow",
 Action: ["kms:Describe*", "kms:Get*", "kms:List*"],
 Principal: {
 "AWS": [`arn:aws:iam::${accountID}:root`]
 },
 Resource: "*",
 },
 // This is a configurable statement that we can use to allow access to an IAM arn
 {
 Sid: "AllowIAMUserAccessKeys",
 Effect: "Allow",
 Action: ["kms:*"],
 Principal: {
 "AWS": iamRole,
 },
 Resource: "*",
 }
 ]
}

// Create a new KMS key
const key = new aws.kms.Key("stack-encryption-key", {
 deletionWindowInDays: 10,
 description: "KMS key for encrypting Pulumi secret values",
 policy: JSON.stringify(keyPolicy),
});

// Create a new alias to the key
const alias = new aws.kms.Alias("alias/stack-encryption-key", {
 targetKeyId: key.keyId,
});

// Export the arns
export const keyArn = key.arn
export const aliasArn = alias.arn

"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");

let config = new pulumi.Config();
let iamRole = config.require("iamRole");
let accountID = config.require("accountID");

const keyPolicy = {
 Version: "2012-10-17",
 Id: "policy",
 Statement: [
 // This statement allows all users to view the key in the console
 {
 Sid: "AllowGetKeys",
 Effect: "Allow",
 Action: ["kms:Describe*", "kms:Get*", "kms:List*"],
 Principal: {
 "AWS": [`arn:aws:iam::${accountID}:root`]
 },
 Resource: "*",
 },
 // This is a configurable statement that we can use to allow access to an IAM arn
 {
 Sid: "AllowIAMUserAccessKeys",
 Effect: "Allow",
 Action: ["kms:*"],
 Principal: {
 "AWS": iamRole,
 },
 Resource: "*",
 }
 ]
}

// Create a nw KMS Key
const key = new aws.kms.Key("stack-encryption-key", {
 deletionWindowInDays: 10,
 description: "KMS key for encrypting Pulumi secret values",
 policy: JSON.stringify(keyPolicy),
});


// Create an alias to the key
const alias = new aws.kms.Alias("alias/stack-encryption-key", {
 targetKeyId: key.keyId
});


// Export the arns
exports.keyArn = key.arn;
exports.aliasArn = alias.arn;

});


// Export the arns
exports.keyArn = key.arn;
exports.aliasArn = alias.arn;

import pulumi
import json
from pulumi_aws import kms

config = pulumi.Config()
iam_role = config.require("iamRole")
account_id = config.require("accountID")


key_policy = {
 "Version": "2012-10-17",
 "Id": "policy",
 "Statement": [
 # This statement allows all users to view the key in the console
 {
 "Sid": "AllowGetKeys",
 "Effect": "Allow",
 "Action": ["kms:Describe*", "kms:Get*", "kms:List*"],
 "Principal": {
 "AWS": ["arn:aws:iam::{}:root".format(account_id)]
 },
 "Resource": "*",
 },
 # This is a configurable statement that we can use to allow access to an IAM arn
 {
 "Sid": "AllowIAMUserAccessKeys",
 "Effect": "Allow",
 "Action": ["kms:*"],
 "Principal": {
 "AWS": iam_role,
 },
 "Resource": "*",
 }
 ]
}
# Create an AWS KMS key
key = kms.Key("stack-encryption-key",
 deletion_window_in_days=10,
 description="KMS key for encrypting Pulumi secret values",
 policy=json.dumps(key_policy)
)

alias = kms.Alias("alias/stack-encryption-key",
 target_key_id=key.key_id
)

# Export the name of the bucket
pulumi.export('key_arn', key.arn)
pulumi.export('alias_arn', alias.arn)

package main

import (
 "encoding/json"
 "fmt"

 "github.com/pulumi/pulumi-aws/sdk/v2/go/aws/kms"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {

 type Principal struct {
 AWS string `json:"AWS"`
 }
 type Statement struct {
 Sid string `json:"Sid"`
 Effect string `json:"Effect"`
 Principal Principal `json:"Principal"`
 Action []string `json:"Action"`
 Resource string `json:"Resource"`
 }

 type KeyPolicy struct {
 Version string `json:"Version"`
 ID string `json:"Id"`
 Statements []Statement `json:"Statement"`
 }

 pulumi.Run(func(ctx *pulumi.Context) error {

 config := config.New(ctx, "")

 iamRole := config.Require("iamRole")
 accountID := config.Require("accountID")

 rootIamRole := fmt.Sprintf("arn:aws:iam::%s:root", accountID)

 rawKeyPolicy := &KeyPolicy{
 Version: "2012-10-17",
 ID: "policy",
 Statements: []Statement{
 {
 Sid: "AllowGetKeys",
 Effect: "Allow",
 Action: []string{
 "kms:Describe*", "kms:Get*", "kms:List*",
 },
 Resource: "*",
 Principal: Principal{
 AWS: rootIamRole,
 },
 },
 {
 Sid: "AllowIAMUserAccessKeys",
 Effect: "Allow",
 Action: []string{"kms:*"},
 Resource: "*",
 Principal: Principal{
 AWS: iamRole,
 },
 },
 },
 }

 keyPolicy, err := json.Marshal(rawKeyPolicy)

 if err != nil {
 panic("Error formatting keypolicy")
 }

 // Create an AWS KMS key
 key, err := kms.NewKey(ctx, "stack-encryption-key", &kms.KeyArgs{
 Description: pulumi.String("KMS key for encrypting Pulumi secret values"),
 DeletionWindowInDays: pulumi.Int(10),
 Policy: pulumi.String(string(keyPolicy)),
 })
 if err != nil {
 return err
 }

 // Create an alias to the key
 alias, err := kms.NewAlias(ctx, "alias/stack-encryption-key", &kms.AliasArgs{
 TargetKeyId: key.KeyId,
 })
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("keyArn", key.Arn)
 ctx.Export("keyAlias", alias.Arn)
 return nil
 })
}

using Pulumi;
using Kms = Pulumi.Aws.Kms;

class KeyStack : Stack
{
 public KeyStack()
 {
 var config = new Config();
 var iamRole = config.Require("iamRole");
 var accountID = config.Require("accountID");

 var keyPolicy = $@"{{
 ""Version"": ""2012-10-17"",
 ""Id"": ""policy"",
 ""Statement"": [
 {{
 ""Sid"": ""AllowGetKeys"",
 ""Effect"": ""Allow"",
 ""Action"": [""kms:Describe*"", ""kms:Get*"", ""kms:List*""],
 ""Principal"": {{
 ""AWS"": [""arn:aws:iam::{accountID}:root""]
 }},
 ""Resource"": ""*""
 }},
 {{
 ""Sid"": ""AllowIAMUserAccessKeys"",
 ""Effect"": ""Allow"",
 ""Action"": [""kms:*""],
 ""Principal"": {{
 ""AWS"": ""{iamRole}""
 }},
 ""Resource"": ""*""
 }}
 ]
 }}";

 // Create a new KMS key
 var key = new Kms.Key("stack-encryption-key", new Kms.KeyArgs
 {
 DeletionWindowInDays = 10,
 Description = "KMS key for encrypting Pulumi secret values",
 Policy = keyPolicy
 });

 // Create a new alias to the key
 var alias = new Kms.Alias("alias/stack-encryption-key", new Kms.AliasArgs
 {
 TargetKeyId = key.KeyId
 });

 this.KeyArn = key.Arn;
 this.AliasArn = alias.Arn;
 }

 [Output("keyArn")] public Output<string> KeyArn { get; set; }
 [Output("aliasArn")] public Output<string> AliasArn { get; set; }
}

In this example, we’ve created a key policy which allows full access to a defined IAM role, and we’ve also given read permissions to the key to everyone in this AWS account. We’ve made the IAM role configurable, so let’s set the IAM role now so we can be sure we can use this key for our next Pulumi stack:

# Get the current AWS account ID and set it as a config variable
aws sts get-caller-identity | jq .Account -r | pulumi config set accountID
# Get the current IAM role we're using as set it as a config variable
aws sts get-caller-identity | jq .Arn -r | pulumi config set iamRole

Initialize a New Stack

Now our key has been created and is adequately scoped; we can create a new stack and use the secrets-provider flag on creation to specify the KMS key to encrypt our secrets.

# We need to first retrieve the stack encryption key from our previous stack
KEY_ALIAS=$(pulumi stack output aliasArn | cut -d/ -f2)
# Note, we need to set the region we're deploying to
pulumi new aws-<language> -n <projectname> -s <stackname> -d "An example stack encrypted with AWS KMS" --secrets-provider="awskms://alias/${KEY_ALIAS}?region=us-west-2" --config aws:region=us-west-2 --dir $HOME/git/new-stack

You can check inside your Pulumi.<stackname>.yaml for which key you’re using to encrypt your secrets:

secretsprovider: awskms://alias/<stack-encryption-key-alias>?region=us-west-2
encryptedkey: AQICAHjqW3rb5Hw5Vpxi0c1sayz52VXj7yn20WVwsVILJSBU8wFDjvGuox3wDCJX99TxZFzAAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMWNeFRZIg8kVXMxrUAgEQgDtz6zV0aqegeAbmaQUNllMp8PQJa1qbjBH813I/XH6LbfynxZO9NE3sYPG89G0u/ltYsADiUAFS0bnadQ==
config:
 aws:region: us-west-2

It’s worth noting here that the “encryptedkey” field here is an encrypted version of the data key in AWS KMS. You can read more about this here.

Use your Encrypted Secret

Now we’ve initialized our stack we can add a secret configuration value using the secret flag and see the encrypted config value in the stack configuration:

pulumi config set --secret supersecret correct-horse-battery-stable

Verify the encryption

To verify that the secret is indeed only accessible to the KMS key we created earlier, we can remove access to the key temporarily and try to perform a Pulumi operation. There are a few ways to remove your access, but first, let’s use the secret in our stack:

import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();
export const superSecret = config.requireSecret("supersecret");

"use strict";
const pulumi = require("@pulumi/pulumi");

const config = new pulumi.Config();
exports.superSecret = config.requireSecret("supersecret");

import pulumi
config = pulumi.Config()

pulumi.export('superSecret', config.require_secret("supersecret"))

package main

import (
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 config := config.New(ctx, "")
 superSecret := config.RequireSecret("supersecret")

 ctx.Export("superSecret", pulumi.Sprintf("%s", superSecret))
 return nil
 })
}

using Pulumi;

class AnotherStack : Stack
{
 public AnotherStack()
 {
 var config = new Config();
 this.SuperSecret = config.RequireSecret("supersecret");
 }

 [Output("superSecret")] public Output<string> SuperSecret { get; set; }
}

Now we need to verify if the value is actually encrypted. An easy way to do that is to try and export the secret value without access to the key. How this is done depends on your AWS configuration, however in my setup, I use the AWS_PROFILE environment variable which refers to a named profile to configure access to AWS. If I unset this environment variable, I will no longer be using the AWS credentials that have access to this KMS key. Let’s unset the AWS_PROFILE environment variable and then rerun pulumi up:

unset AWS_PROFILE
pulumi up
error: getting secrets manager: secrets (code=Unknown): InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
 status code: 400, request id: 9cf7508d-a7d5-40bb-b40f-98f68e82ac74

Excellent! We can’t read these values without access to this KMS key. We can be safe in the knowledge our secret values are only readable by us.

Examine the Pulumi State

The final part is to make sure our values are stored encrypted inside the Pulumi state. To do this, we need to use our secret value from earlier and use it in a resource. Let’s create an s3 bucket, and write out super secret value to a file in the bucket. Update the Pulumi program you used before like so:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const config = new pulumi.Config();

const superSecret = config.requireSecret("supersecret");

// Create a private bucket
const bucket = new aws.s3.Bucket("bucket", {
 acl: "private",
});

// Create an object from the secret value
const superSecretObject = new aws.s3.BucketObject("secret", {
 bucket: bucket.id,
 key: "secret",
 content: superSecret, // use our secret value as the content
})

"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");

const config = new pulumi.Config();

const superSecret = config.requireSecret("supersecret");

// Create a private bucket
const bucket = new aws.s3.Bucket("bucket", {
 acl: "private",
});

const superSecretObject = new aws.s3.BucketObject("secret", {
 bucket: bucket.id,
 key: "secret",
 content: superSecret, // use our secret value as the content
});

import pulumi
from pulumi_aws import s3

config = pulumi.Config()

superSecret = config.require_secret("supersecret")

# Create a private bucket
bucket = s3.Bucket('bucket', acl="private")

# Create an object from the secret value
bucketObject = s3.BucketObject("secret", bucket=bucket.id, key="secret", content=superSecret)


package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v2/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 config := config.New(ctx, "")

 superSecret := config.RequireSecret("supersecret")

 // Create a a private bucket
 bucket, err := s3.NewBucket(ctx, "bucket", &s3.BucketArgs{Acl: pulumi.String("private")})
 if err != nil {
 return err
 }

 _, err = s3.NewBucketObject(ctx, "secret", &s3.BucketObjectArgs{
 Bucket: bucket.ID(),
 Key: pulumi.String("secret"),
 Content: pulumi.Sprintf("%s", superSecret),
 })

 ctx.Export("superSecret", pulumi.String(superSecret))
 return nil
 })
}

using Pulumi;
using Pulumi.Aws.S3;

class AnotherStack : Stack
{
 public AnotherStack()
 {
 var config = new Config();
 this.SuperSecret = config.RequireSecret("supersecret");

 var bucket = new Bucket("bucket", new BucketArgs {
 Acl = "private",
 });

 var bucketObject = new BucketObject("secret", new BucketObjectArgs {
 Bucket = bucket.Id,
 Key = "secret",
 Content = SuperSecret

 });
 }
 [Output("superSecret")] public Output<string> SuperSecret { get; set; }
}

We now need to look inside our Pulumi statefile to verify the value is encoded there. We can do this with pulumi stack export and some JSON manipulation magic using jq:

pulumi stack export | jq '.deployment.resources[].outputs | select(.content).content'

The result should look like this:

{
 "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
 "ciphertext": "v1:cQ5qr21hTdb2PTM5:ZwEW8pN1kC6fUlppi1eS84D/lodoe54wV2dgEsqu0csu2VyTQg0wTf8Qv7axCQ=="
}

Wrap up

This example showed how to use client-side encryption with AWS KMS. Pulumi, as mentioned before, has support for Azure KeyVault, Google Cloud KMS, and HashiCorp Vault for storing your keys. You can find examples of how to use these encryption methods in our examples repo and take a look at our secrets provider documentation.

We hope your next compliance audit is more relaxed with this feature available!

Managing AWS Credentials on CI/CD: Part 2

Chris Smith — Thu, 26 Mar 2020 00:00:00 +0000

This article is the second part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we go in-depth on providing AWS credentials securely to a 3rd party and introduce a Pulumi program to automate rotating access keys.

NOTE: These recommendations do not apply if you are running a CI/CD system within your AWS account, e.g., running a Jenkins server on EC2 or using AWS CodeDeploy. In those cases, please refer to AWS’s documentation for how to assume IAM Roles when running on an EC2 instance instead.

Provide IAM credentials to your CI/CD system

In the first post in our series, we created a dedicated IAM User to perform updates to AWS resources within your CI/CD system. The next step is to pass the AWS access keys for that user to your CI/CD system.

We need to take great caution. AWS’s documentation states, “Do not provide your access keys to a third party”.

Providing AWS access keys to any system is dangerous because of the risk that a bad actor could obtain those keys and do something nefarious. Even if your CI/CD service takes great care to protect your secrets, those AWS credentials could be inadvertently exposed in debugging output, system logs, or in some other way.

This is one of the main advantages of performing continuous deployment from within your AWS account. For example, running Jenkins on an EC2 instance you manage, or using the AWS CodeDeploy service. Because when performing a deployment from within an AWS compute environment, you can use built-in mechanisms to obtain credentials securely.

However, there are many reasons to use a hosted service for performing your CI/CD, such as developer productivity, ease of use, performance, or simply because you don’t want to recreate your existing deployment workflows.

Ultimately, it is essential to understand the risks involved and how to mitigate them.

Properly Storing Credentials

If you choose to provide your CI/CD with credentials, the most important thing is to mark them as secret. Your CI/CD provider typically has built-in support for handling “secure variables” or other sensitive information. As opposed to general, configuration data or environment variables.

User-supplied configuration values are typically write-only and only accessible by the CI/CD worker jobs at runtime. For example, Travis CI has encrypted environment variables or secure environment variables in CircleCI.

Please consult your CI/CD provider’s documentation for how to pass and store sensitive information appropriately.

Why not use a “secrets manager” service?

A common question when discussing is how to pass access keys to a CI/CD system securely:

Rather than giving your CI/CD provider AWS credentials, why not have your CI/CD system obtain credentials from a specialized “secrets manager” service?

In other words, if you choose not to trust your CI/CD system with this data, can you instead trust some other system dedicated for securely storing and retrieving sensitive information?

It sounds like a good idea on the surface but doesn’t seem to make your data any more secure.

Abstractly, the difference here is that the credentials are provided on-demand, rather than being available to the CI/CD job when it starts (and stored via the CI/CD provider). Instead, your CI/CD job would obtain credentials from the “secrets manager” only when needed.

There are some advantages to this approach, such as providing a clear audit trail for access and more control over the distribution of sensitive information.

However, the secrets provider system needs to be presented with some form of credentials. And those credentials need to be available to your CI/CD environment. So using a secrets manager leaves you in the same place you started, i.e., needing to provide sensitive data to your CI/CD provider.

Also, by adding a dependency on a secrets manager, you introduce additional risks. Not only do you need to be even more security-conscious about that secrets manager, but it also needs to be highly available. Any outage for that service would mean that you would be unable to perform deployments!

So it does not seem that using a secrets manager to dole out AWS credentials to your CI/CD system is a good practice to follow. Or, at the very least, makes some tradeoffs without fundamentally making your approach to CI/CD any more or less secure.

Instead, it is better to focus on other ways to reduce the risk if those AWS credentials do get exposed.

Automate Rotating and Revoking AWS Credentials

When providing credentials to a 3rd party, rather than hoping it will be 100% secure forever (which is impossible), we can make those credentials volatile. If we regularly invalidate and rotate the credentials supplied to the CI/CD system, we can dramatically reduce the impact of any accidental disclosure. Even if the IAM User’s credentials were leaked, by the time they were discovered and used, they would no longer be valid.

The AWS security blog describes how to rotate access keys for IAM Users, by using the following steps:

Create a second access key in addition to the one in use.
Update all your applications to use the new access key.
Change the state of the previous access key to inactive.
Verify everything is still working as expected.
Delete the inactive access key.

That all sounds simple enough, but it’s certainly tedious. And if you need to repeat the process across dozens, if not hundreds of different CI/CD pipelines, you need a better solution. Fortunately, Pulumi provides an excellent and extensible way for writing a serverless program to automate rotating credentials.

The next few sections describe a simple infrastructure application for automating AWS IAM credential rotation. You can see the full application on GitHub at chrsmith/pulumi-aws-travis-cicd-demo.

AWS Credential Rotator 9000

The affectionally titled AWS Credential Rotator 9000 is a simple, serverless Pulumi application for rotating AWS access keys.

It creates an AWS lambda function that’s triggered on a regular schedule, e.g., every 30 minutes, and performs the next step in the sequence for rotating access keys as outlined above.

First, it creates a new access key and pushes the new value out. On the next iteration, it marks the older access key as “inactive.” On the following iteration, the inactive key is deleted. The process repeats, generating a new key and removing the inactive key.

Periodically Invoking an AWS Lambda

The heart of the application is triggering it to execute on a fixed interval. Thankfully this is super-easy to do using Pulumi since it allows you to seamlessly blend your “cloud infrastructure” with “code” in a natural way. The user guide for Pulumi Crosswalk for AWS has more information on serverless eventing if you would like to learn more.

The following snippet is the core part of the credential rotator app. We define a function to handle the logic of key rotation in rotateIAMUserKeys. Then we create an AWS Lambda resource named lambda. Finally, the triggerSchedule resource invokes our lambda on a fixed schedule, thereby ensuring that the key rotation process goes on indefinitely. You can see the full code here.

async function rotateIAMUserKeys(iamUser: aws.iam.User, onNewCredentials: (key: string, secret: string) => Promise<void>) {
 ...
}

const lambda = new aws.lambda.CallbackFunction<aws.cloudwatch.EventRuleEvent, void>(
 "keyRotatorLambda",
 {
 callback: async (e) => {
 await rotateIAMUserKeys(userToRotate, async (newKey: string, newSecret: string) => {
 console.log("onNewCredentials callback called!");
 console.log(`A new AWS access key "${newKey}" has been created.`);
 });
 },

 role: lambdaRole,
 runtime: "nodejs10.x",
 });

const triggerSchedule = aws.cloudwatch.onSchedule(
 "keyRotatorScheduler", "rate(1 hour)", lambdaCallbackFn);

When writing reusable infrastructure components in Pulumi however, it is helpful to organize things into a custom resource.

For example, we can bundle together the AWS Lambda, CloudWatcn schedule, and the associated IAM policies into a single conceptual resource AccessKeyRotator. Bundling resources allows for the code reuse.

const rotator = new AccessKeyRotator("rotator", {
 interval: config.require("rate"),
 user: user,
 credentialPusher: demoTravisCIPusher,
})

Pushing Credentials

With the mechanics of updating an AWS access key out of the way, the next step is to notify dependent systems what the new access key should be. The key rotator app has a CredentialPusher abstraction to provide a pluggable way for you to send credentials to where they need to be. The example on GitHub only supports updating a Travis CI project (See credential-pusher-travis.ts.), but it could be easily extended to support other CI/CD systems.

The following snippet shows how the CredentialPusher is configured in the key rotator app. We load a Travis CI access token from the Pulumi stack’s configuration, and hard-code the specific set of projects and encrypted environment variables to store the new access key.

// A "credential pusher" is the component that pushes new AWS IAM credentials out to 3rd parties
// as the older ones get rotated. For demonstration purposes, this updates the Travis CI settings
// for the chrsmith/pulumi-aws-travis-cicd-demo repo. But you can imagine another implementation
// that pushes the new IAM credentials to GitLab CI, or updating multiple CI/CD pipelines.

const demoTravisCIPusher = new CredentialPusher(
 new TravisCIPusher(),
 {
 accessKey: config.require("travis-ci-token"),
 projects: [
 {
 project: "chrsmith/pulumi-aws-travis-cicd-demo",
 // In the Travis CI configuration for that GitHub repo, there are two
 // environment variables for storing the AWS credentials. So whenever the
 // AWS credentials get rotated, the job's configuration settings will be
 // updated to reflect the new values.
 accessKeyIDLocation: "AWS_ACCESS_KEY_ID",
 secretAccessKeyLocation: "AWS_SECRET_ACCESS_KEY",
 }
 ],
 });

Demo

To demonstrate the access keys rotation, we can examine the log files generated from AWS Lambda. These can be accessed from the command-line using the pulumi logs command.

Here’s a summary of the output for clarity:

 START RequestId: 9406913e-a9d0-46a4-b861-2efdee508b2b Version: $LATEST
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO IAM User has 2 keys:
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO - AKIASHIVKXX3SA2K3ME7 [Inactive] Tue Mar 24 2020 09:37:03 GMT+0000 (Coordinated Universal Time)
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO Deleting older, inactive access key AKIASHIVKXX3SA2K3ME7 [Inactive]
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO Key rotation step complete.
 END RequestId: 9406913e-a9d0-46a4-b861-2efdee508b2b

 START RequestId: 05644e4a-81a4-4197-8ad7-fc3517998270 Version: $LATEST
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO IAM User has 1 keys:
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Created new key key AKIASHIVKXX3YX4IXRVH
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Pushing out the new key to 3rd party services...
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Pushing new credentials to Travis CI project 'chrsmith/pulumi-aws-travis-cicd-demo'
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updating env var 'AWS_ACCESS_KEY_ID' (b246ab3b-c4cb-e76f-a6fc-50f4cbf451c0)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updated AWS access key ID. Got response code (200)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updating env var 'AWS_SECRET_ACCESS_KEY' (ba9ac4be-fbb3-4da9-c5b8-35a1afe9e02b)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updated AWS secret access key. Got response code (200)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Key rotation step complete.
 END RequestId: 05644e4a-81a4-4197-8ad7-fc3517998270

 START RequestId: ca204959-32a7-44a8-ba20-738d139542ba Version: $LATEST
 ca204959-32a7-44a8-ba20-738d139542ba INFO IAM User has 2 keys:
 ca204959-32a7-44a8-ba20-738d139542ba INFO - AKIASHIVKXX3YX4IXRVH [Active] Tue Mar 24 2020 15:37:03 GMT+0000 (Coordinated Universal Time)
 ca204959-32a7-44a8-ba20-738d139542ba INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 ca204959-32a7-44a8-ba20-738d139542ba INFO Invalidating older access key AKIASHIVKXX3QEQNZNX5
 ca204959-32a7-44a8-ba20-738d139542ba INFO Key rotation step complete.
 END RequestId: ca204959-32a7-44a8-ba20-738d139542ba

 START RequestId: de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 Version: $LATEST
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO IAM User has 2 keys:
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO - AKIASHIVKXX3YX4IXRVH [Active] Tue Mar 24 2020 15:37:03 GMT+0000 (Coordinated Universal Time)
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO - AKIASHIVKXX3QEQNZNX5 [Inactive] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO Deleting older, inactive access key AKIASHIVKXX3QEQNZNX5 [Inactive]
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO Key rotation step complete.
 END RequestId: de753fb4-0720-4f4f-a7aa-ad6b9d4c8832

As you can see, the AWS credentials for the IAM User are automatically updated every few hours. And whenever a new access key is created, the value is pushed out to the impacted Travis CI projects automatically.

Wrapping Up

In this post, we covered some of the things to consider when providing AWS credentials to your CI/CD system. (In short, be cautious and follow best practices.)

We then showed how relatively easy it is to stand up a Pulumi infrastructure application for automatically rotating AWS access keys and updating your CI/CD system. By rotating credentials, you can limit the impact if the value is inadvertently disclosed.

At this point, we’ve now given access keys to a CI/CD system for a low-privilege AWS IAM User account. But we still can’t do anything with it. The IAM User whose credentials we have given to the CI/CD system doesn’t have access to your production AWS account.

In the next post, we’ll go into the details about IAM Roles, and they can be used to securely and temporarily gain access to additional resources. This is how our low-privilege IAM User can access needed to update production data for your CI/CD pipelines.

Running AWS IAM Access Analyzer at Deployment Time

Joe Duffy — Tue, 03 Dec 2019 00:00:00 +0000

Yesterday AWS announced an exciting new feature — the AWS Identity and Access Management (IAM) Access Analyzer — a service powered by automated reasoning that detects potentially-insecure access to your AWS resources, including S3 Buckets, SQS Queues, Lambdas, and more. At the same time, Pulumi announced a new policy as code solution, CrossGuard, that validates policies at deployment time. The question is: Can IAM Access Analyzer and Pulumi CrossGuard be combined to get the best of both solutions? The answer is Yes!

Fast Forward to Security By Default

Before diving into the details of how this all works, let’s see what this looks like in action.

By default, the IAM Access Analyzer runs in your account, monitoring for new resources and creating alerts as new problems are identified. This approach does not tell you as part of a deployment that a new problem has been introduced, however. In this post, we’ll use Pulumi CrossGuard to flag deployments that have introduced a new problem, so you find out about it right away.

Let’s try to create an S3 Bucket with public-read access:

import * as aws from "@pulumi/aws";
const bucket = new aws.s3.Bucket("my-bucket", {
 acl: "public-read",
});

The acl here means the S3 Bucket is available to the Internet — sometimes what you want, but usually not — and potentially a major security catastrophe waiting to happen! This is a very easy mistake to make. How do we prevent it?

Pulumi CrossGuard can stop such things. In fact, we have an AWSGuard policy pack that will enforce the best practices across all of the AWS services, out of the box, including security, compliance, and cost-based rules. That includes checking for public S3 buckets.

As AWS makes their own platform more secure by default, however, we would love to take advantage of it as soon as new features come out. And IAM Access Analyzer seems to have some very smart smarts behind it, which we would greatly benefit from, as articulated in this research paper.

Thanks to CrossGuard’s use of full-blown programming languages, we can easily tap into the AWS SDK support for IAM Access Analyzer. The result is that if we try to deploy the above program with our custom policy pack — pulumi up --policy-pack ~/iam-analyzer-policy — we get an error:

Nice! Security catastrophe averted! Now let’s dig into how it all works.

IAM Access Analyzer, Meet CrossGuard

First thing’s first: IAM Access Analyzer is new; and Pulumi CrossGuard is new. Both of them were announced just yesterday! So a lot of what we’re going to look at will evolve in the weeks to come. Nevertheless, the result is very exciting — as soon as we saw the IAM Access Analyzer announcement, we knew there could be magic in combining the two ideas.

The approaches are different but complementary:

IAM Access Analyzer uses a single analyzer per-region that validates all resources within it. For each problem it identifies, it creates a so-called “finding” which records what policy is responsible for granting wider access to specific resources than you might want.
CrossGuard, on the other hand, operates by applying individual policy packs to specific stacks of resources. This can be configured to run automatically anytime a deployment occurs in your organization, or explicitly using the --policy-pack argument shown above.

To see how they can work in partnership with one another, we will create a new CrossGuard policy pack that uses the new IAM service. We’ll walk the code step by step, but the full code is also available on GitHub.

First, we created this new policy pack with the following commands:

mkdir ~/iam-analyzer-policy
cd ~/iam-analyzer-policy
pulumi policy new aws-typescript

This populates the ~/iam-analyzer-policy directory with the scaffolding needed to create a policy pack.

By default, the template will have installed the @pulumi/pulumi, @pulumi/policy, and @pulumi/aws packages. We need some others:

npm install aws-sdk uuid

Now let’s import everything we need at the top of our index.ts file, which is where we’ll build out our policy:

import * as AWS from "aws-sdk"
import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";
import { PolicyPack, ReportViolation, StackValidationArgs } from "@pulumi/policy";
import * as uuid from "uuid";

Next we’ll allocate our policy pack and its internal policies. We will only have a single rule at this time:

new PolicyPack("aws-typescript", {
 policies: [{
 name: "iam-access-analyzer",
 description: "Resource has IAM Access Analyzer errors.",
 enforcementLevel: "mandatory",
 validateStack: async (args: StackValidationArgs, reportViolation: ReportViolation) => {
 // interesting part goes here ...
 },
 }],
});

Notice we are using the validateStack callback to validate our resources. The alternative is to use validateResource, which runs on a single resource at a time. In this case, the stack-wide validation is easier, since we’ll be sharing a single IAM analyzer instance.

Let’s start filling out the validateStack function. First, during previews (indicated by the pulumi.runtime.isDryRun() function), we want to skip validation. The reason is that IAM Access Analyzer only runs against real resources already allocated. Perhaps in the future, it will let us query resources before they are created, but for now the best we can do is catch the mistake immediately afterwards. Thankfully, CrossGuard supports running both before provisioning, during previews, in addition to afterwards.

// If we're doing a dry-run, we skip this, because IAM Access Analyzer only analyzes actual resources.
// Ideally in the future, they will support scanning them *before* the problem is actually deployed!
if (pulumi.runtime.isDryRun()) {
 return;
}

Next, now that we know we’re doing an update, we want to create an IAM Access Analyzer SDK object that we can use to call the service:

// Create a fresh analyzer for this run.
const svc = new AWS.AccessAnalyzer({ region: aws.config.region || "us-east-1" });

Now we can provision the temporary analyzer itself that we’ll use for this validation pass:

const analyzerName = `iam-analyzer-${uuid.v4().toString()}`;
const resp = await svc.createAnalyzer({ analyzerName, type: "ACCOUNT" }).promise();
const analyzerArn = resp.arn!;
try {

Notice we’ve wrapped the subsequent code in a try block; you’ll see a finally below that deletes the temporary analyzer object.

Our illustration here creates and uses an IAM Access Analyzer with "ACCOUNT" scope, the only currently supported option at the time of this writing. Because of the restriction that there can only be one analyzer per account, per region, this may cause issues if you already have an analyzer pre-created, or need to run deployments in parallel. Depending on how the IAM Access Analyzer feature evolves, you may want to take a different approach than what is shown here.

Next up, we’ll look for all resource types supported by IAM Access Analyzer, and kick off a resource scan for each one:

 // Now, find all eligible resources, create a map of them, and kick off scans for each one.
 const analyzed: string[] = [];
 const analyzedUrns: {[arn: string]: string} = {};
 for (const res of args.resources) {
 let resourceArn: string | undefined;
 if ([ "aws:iam/role:Role", "aws:kms/key:Key", "aws:lambda/function:Function",
 "aws:lambda/layerFunction:LayerFunction", "aws:s3/bucket:Bucket", "aws:sqs/queue:Queue" ].
 indexOf(res.type) !== -1) {
 resourceArn = res.props.arn;
 }

 if (resourceArn) {
 await svc.startResourceScan({ analyzerArn, resourceArn }).promise();
 analyzed.push(resourceArn);
 analyzedUrns[resourceArn] = res.urn;
 }
 }

The startResourceScan function is asynchronous, so we’ll now need to wait until all of the requested resources have been analyzed. This requires us to poll the AWS API with appropriate pagination. We will wait up to 30 seconds for the results to converge:

 // Wait until all resources have been analyzed.
 const maxRetries = 30;
 for (let i = 0; i < maxRetries; i++) {
 const analyzed: {[arn: string]: boolean} = {};
 let nextToken: string | undefined;
 do {
 const result = await svc.listAnalyzedResources({ analyzerArn }).promise();
 for (const res of result.analyzedResources) {
 analyzed[res.resourceArn] = true;
 }
 nextToken = result.nextToken;
 } while (nextToken);

 let done = true;
 for (const arn of Object.keys(analyzedUrns)) {
 if (!analyzed[arn]) {
 done = false;
 break;
 }
 }
 if (done) {
 break;
 }

 if (i == maxRetries-1) {
 reportViolation(`Timed out waiting for IAM Access Analyzer results; is the region correct?`);
 return;
 } else {
 await new Promise((resolve) => setTimeout(resolve, 1000));
 }
 }

Now that we know the results are available, we simply have to list the detailed findings. In this example, our policy pack will only issue an error for every IAM configuration that leads to potentially unwanted public access. Each finding contains detailed information about the IAM resources causing unwanted access, the ARN of the target resource, and remediation information:

 // Now that the scan is ready, go ahead and list the findings, triggering errors as appropriate.
 let nextToken: string | undefined;
 do {
 const result = await svc.listFindings({ analyzerArn, nextToken }).promise();
 for (let finding of result.findings) {
 if (finding.isPublic && finding.resource) {
 // Only report violations for resources within this stack.
 const urn = analyzedUrns[finding.resource];
 if (urn) {
 reportViolation(
 `${JSON.stringify(finding.principal)} has ${finding.action} access to ` +
 `${finding.resourceType} ${finding.resource} (public) ${finding.error || ""} ` +
 `(message ID ${finding.id})`,
 urn);
 }
 }
 }
 nextToken = result.nextToken;
 } while (nextToken);

And finally, now that all of our scanning and results are available, we just need to clean up our temporary analyzer object:

} finally {
 // Ensure to delete the analyzer before returning.
 await svc.deleteAnalyzer({ analyzerName }).promise();
}

That’s it. Now we have a fully functioning policy pack that can be uploaded to our organization using pulumi policy publish, or run as shown earlier in the post with the --policy-pack flag to pulumi up or pulumi preview. The result?

mandatory: [iam-access-analyzer] Resource has IAM Access Analyzer errors.
{"AWS":"*"} has s3:ListBucket,s3:ListBucketByTags,s3:ListBucketMultipartUploads,s3:ListBucketVersions access to AWS::S3::Bucket arn:aws:s3:::my-bucket-298b5bc (public) (message ID ccaec287-92f3-4136-bb1f-f7a731efdf62)

Trying It Out

In this article, we’ve seen how to enforce AWS’s new IAM Access Analyzer policies at deployment time, using Pulumi’s new CrossGuard feature. This helps to catch security mistakes when you do a deployment — instead of later on, when the cost of a mistake is potentially significantly greater. Although both features are early and will evolve in the weeks to come, it’s nice to see it so easy to quickly combine two features the day after they were both announced, to advance the security of your infrastructure.

To give it a try, download and install Pulumi, grab the code from GitHub, and get started with policy as code, all using free open source.

GitHub & Pulumi Join Forces To Ensure Pulumi Tokens Are Safe

Praneet Loke — Mon, 19 Aug 2019 00:00:00 +0000

We are very excited to announce that we have partnered with GitHub to offer our users better protection for their Pulumi Access Tokens.

By default, Pulumi users manage the state of their cloud infrastructure deployments using https://app.pulumi.com. This service provides state storage, concurrency control, audit history and access controls for both individuals and teams working with Pulumi. Each user and service account can generate one or more Pulumi Access Tokens to be used to authenticate with this service. These access tokens can be used on both local development machines, as well as in CI/CD systems for automated infrastructure deployments. These access tokens are sensitive secrets which should never be shared publicly, and in particular should never be committed to source control.

How GitHub Token Scanning Works for Pulumi Users

With the recent changes to integrate Pulumi with GitHub Token Scanning, GitHub now scans each commit you push to a public repo on github.com for any files containing these tokens. When GitHub’s Token Scanning Service finds a potential match for our tokens, the service will call ours to have us verify the token. Only Pulumi Access Tokens generated after June 28, 2019 that contain the prefix pul- will be matched by the scanning service.

During the processing of the requests from the Token Scanning Service, if we find that the token belongs to a Pulumi account, we will use the primary email on the account to notify the user. We will not, however, revoke the token automatically. So ensure that your account’s email address is up-to-date by going to your profile in the Pulumi Service.

The whole process, from the check-in to the email notification, typically takes just a few seconds, minimizing the window of an accidental breach. Of course, best practice is to avoid checking in the token at all!

Avoid Checking-In The Token In The First Place

Pulumi Access Tokens grant access to your stacks. They allow you to login into the CLI non-interactively and thus allow the CLI to access a stack’s configuration, which may in-turn contain other secrets. The token is typically used inside CI/CD environments. So you should never have to check-in a token to your source tree (public or otherwise.)

Most CI/CD services have a way for you to specify secret environment variables that allow you to safely inject them into your build environment. We have written-up quite a few guides on how to integrate Pulumi into your CI/CD. If you don’t see a guide for a CI/CD system that you are using, feel free to create an issue here, or even better, raise a PR for it. We love community contributions! :)

7 Ways to Deal with Application Secrets in Azure

Mikhail Shilkov — Fri, 26 Jul 2019 00:00:00 +0000

Every non-trivial application relies on configuration values that may depend on the current execution environment. Some of these values contain sensitive information that shouldn’t be shared publicly. In general, the fewer parties that have access to those secret values, the safer the application will be—in fact, in an ideal world, no one would be granted direct access to those secrets.

Examples of secret configuration values include:

A connection string to a message bus or a database
A SAS Token to an Azure Storage account
An access key for a third-party service

There’s no one universal way to manage secrets, as a lot depends on the context in which they are used. In this article, I go through seven ways to use secret values in a .NET Core application running in Azure. I start with naively hard-coded strings and build up from there to more secure options.

While the concepts are universally applicable, my code samples focus on a .NET application running in Azure App Service and configured with Pulumi. Consider them best practices for managing client secrets in Azure.

1. Hard-Coded Application Secrets

Whenever you want to try a new API requiring a secret access token, it’s natural to copy-paste that secret into your code and run it—simply to make sure that the setup works.

For example, you are testing integration with a payment service:

var apiKey = "payment-service-key";
var paymentService = new PaymentService(apiKey);

or you want to start sending messages to an Azure Storage Queue:

var connectionString = "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key";
var storageAccount = CloudStorageAccount.Parse(connectionString);
var queueClient = storageAccount.CreateCloudQueueClient();

Both snippets are fine for a Hello-World application with a lifespan of two hours. However, I would strongly discourage doing so in any code that can potentially be checked into a source control system, even for a 10-minute experiment. Don’t copy-paste secrets into files which are part of a git repository. One accidental git commit & git push—and the secrets are compromised.

Avoid the “Secrets as Code” practice: there are bots scanning your GitHub repository for those.

2. Configuration Files

Traditionally, putting secrets in a configuration file is considered more secure. For classic .NET applications, this would be an app.config or a web.config file. The idea is that the values on a developer machine are different from the values in the production environment. Non-sensitive development settings are kept on disk and maybe in source control, while the real secrets are only injected by a deployment script or a CI/CD system, so they are not exposed publicly.

In the .NET Core world, a configuration is usually stored in an appsettings.json file:

{
 "MyConfig": {
 "PaymentApiKey": "payment-service-key",
 "StorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key"
 }
}

which is then mapped to a plain C# object in code:

public class MyConfig
{
 public string PaymentApiKey { get; set; }
 public string StorageConnectionString { get; set; }
}

With some configuration not shown here, the properties are filled at startup time and can be used in the code as a plain object:

var apiKey = config.PaymentApiKey;
var paymentService = new PaymentService(apiKey);
// ...

var connectionString = config.StorageConnectionString;
var storageAccount = CloudStorageAccount.Parse(connectionString);
var queueClient = storageAccount.CreateCloudQueueClient();

In my experience, such setup still poses substantial risks. appsettings.production.json may still be accidentally checked into the source control. Developers tend to use real cloud resources for their local and test environments, which do contain sensitive information that can be exploited when leaked.

On top of that, it doesn’t feel right to mix configuration—how many threads to run, or how big the message batches should be—with secret connection strings and API keys in the same file. These are separate kinds of configuration, and they warrant different workflows.

3. Environment Variables and Application Settings

One alternative approach is to read the secret values from environment variables. .NET Core configuration system can parse environment variables instead of, or in addition to the settings files. Likewise, one could read such values with a one-liner in C#:

var apiKey = Environment.GetEnvironmentVariable("PAYMENT_API_KEY");
// ...

var connectionString = Environment.GetEnvironmentVariable("STORAGE_CONNECTION_STRING");
// ...

Your CI/CD system should inject those values as part of the deployment pipeline.

The App Service gives you the ability to set environment variables via Application Settings. Here is a snippet of a Pulumi program that passes both secret values to appSettings:

const cfg = new pulumi.Config();
const paymentApiKey = cfg.requireSecret("paymentApiKey");

const storageAccount = new azure.storage.Account("storage", {
 resourceGroupName: resourceGroup.name,
 accountReplicationType: "LRS",
 accountTier: "Standard",
});

const app = new azure.appservice.AppService("app", {
 resourceGroupName: resourceGroup.name,
 appServicePlanId: appServicePlan.id,
 appSettings: {
 "PAYMENT_API_KEY": paymentApiKey,
 "STORAGE_CONNECTION_STRING": storageAccount.primaryConnectionString,
 },
});

The Storage connection string is produced by the Pulumi program directly, so it doesn’t have to be placed anywhere outside the program itself.

The payment service key is provided by a third party, so its encrypted value is stored in Pulumi configuration. Read Managing Secrets with Pulumi to learn about security options available for secrets in Pulumi config.

4. Azure Key Vault

In the previous example, both secrets end up in Application Settings. Every person with sufficient permissions may go to the App Service and see them in clear text. While this can be restricted, it’s a good idea to grant full read access to the application developers and operators of the App Service to give them the full picture when troubleshooting issues.

In some cases, compliance to a certain standard may require the use of a certified key management service offering enhanced security for sensitive secrets.

Azure has a dedicated service for storing secrets, Azure Key Vault. You can create and populate a Key Vault with all the secrets from the same Pulumi program:

const vault = new azure.keyvault.KeyVault("vault", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: "standard",
 },
 tenantId: tenantId,
 accessPolicies: [{
 tenantId,
 // The current principal has to be granted permissions to Key Vault so that it can actually add and then remove
 // secrets to/from the Key Vault. Otherwise, 'pulumi up' and 'pulumi destroy' operations will fail.
 objectId: currentPrincipal,
 secretPermissions: ["delete", "get", "list", "set"],
 }],
});

const secret = new azure.keyvault.Secret("paymentApiKey", {
 keyVaultId: vault.id,
 value: paymentApiKey,
});

Additionally, one or many Service Principals (SP) should be configured to access the Key Vault. One SP which has been granted secret management access deploys the infrastructure by running the Pulumi program. That’s why the snippet above assigns an access policy to currentPrincipal.

Another SP is used by the application itself to read the secret values. The Client ID and Client Secret parameters are placed in the application configuration. Finally, .NET Core configuration gets hooked to the Key Vault at startup:

builder.AddAzureKeyVault(
 $"https://{config["azureKeyVault:vault"]}.vault.azure.net/",
 config["azureKeyVault:clientId"],
 config["azureKeyVault:clientSecret"]
);

This solution is not entirely satisfying though, since we’ve traded storing the secrets for storing SP credentials. Is that a big enough win? Luckily, it’s easy to get rid of those credentials with Managed identities.

5. Accessing Key Vault with Managed Identities

With Managed identities, Azure takes care of creating a Service Principal, passing the credentials, rotating secrets, and so on. Enabling a managed identity on App Service is just an extra option:

const app = new azure.appservice.AppService("app", {
 resourceGroupName: resourceGroup.name,
 appServicePlanId: appServicePlan.id,
 // A system-assigned managed identity
 identity: {
 type: "SystemAssigned",
 },
});

On top of that, the managed principal must be granted access to the Key Vault:

const principalId = app.identity.apply(id => id.principalId);

// Grant App Service access to KV secrets
new azure.keyvault.AccessPolicy("app-policy", {
 keyVaultId: vault.id,
 tenantId: tenantId,
 objectId: principalId,
 secretPermissions: ["get"],
});

Now, the configuration block in the .NET Core app doesn’t need to retrieve any secrets. AzureServiceTokenProvider helps with the authentication process:

var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVaultClient = new KeyVaultClient(
 new KeyVaultClient.AuthenticationCallback(
 azureServiceTokenProvider.KeyVaultTokenCallback));
builder.AddAzureKeyVault(
 keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());

That’s quite a bit of a boilerplate, but there is a way to get rid of it.

6. Accessing Key Vault from Application Settings

App Service has a neat feature of integrating its Application Settings with Key Vault. It allows us to combine #3 and #5’s approaches and get the best of both:

// Produce a URI of the KV secret defined above
const secretUri = pulumi.interpolate`${secret.vaultUri}secrets/${secret.name}/${secret.version}`;

const app = new azure.appservice.AppService("app", {
 resourceGroupName: resourceGroup.name,
 appServicePlanId: appServicePlan.id,
 // A system-assigned managed identity
 identity: {
 type: "SystemAssigned",
 },
 appSettings: {
 // The setting points directly to the KV setting
 "PAYMENT_API_KEY": pulumi.interpolate`@Microsoft.KeyVault(SecretUri=${secretUri})`,
 }
});

With that, the API key is loaded into the App Service environment variable without its value being publicly exposed anywhere!

7. Role-Based Access Control

What is the most secure way to deal with secrets? Have no secrets. The Storage Account connection string is a great example when it’s possible to avoid storing and reading the sensitive value altogether.

Role-Based Access Control (RBAC) of Azure Active Directory (AAD) is a great tool to manage permissions in a declarative way. Let’s assume our application only needs to send messages to one Storage Queue. Instead of storing a full connection string with an access token, the connection string should point to the account, and the identity behind the App Service should be granted write permissions to the required Storage Queue:

const permission = new azure.role.Assignment("send", {
 principalId,
 scope: pulumi.interpolate`${storageAccount.id}/queueServices/default/queues/${queue.name}`,
 roleDefinitionName: "Storage Queue Data Message Sender",
});

We can put the queue URL into Application Settings because there’s nothing secret in it:

const queueUri = pulumi.interpolate`${storageAccount.primaryQueueEndpoint}${queue.name}`;

const app = new azure.appservice.AppService("app", {
 resourceGroupName: resourceGroup.name,
 appServicePlanId: appServicePlan.id,
 identity: {
 type: "SystemAssigned"
 },
 appSettings: {
 "STORAGE_QUEUE_URL": queueUri,
 },

It takes a bit of C# boilerplate to send a message with role-based authorization:

var provider = new AzureServiceTokenProvider();
string accessToken = await provider.GetAccessTokenAsync("https://storage.azure.com/");
var tokenCredential = new TokenCredential(accessToken);
var storageCredentials = new StorageCredentials(tokenCredential);
var url = Environment.GetEnvironmentVariable("StorageBlobUrl");
var queue = new CloudQueue(new Uri(url), storageCredentials);
queue.AddMessage(new CloudQueueMessage("Hello"));

Let’s hope another quality-of-life improvement is on the way.

Role of Infrastructure as Code

While security practices may vary depending on application requirements, Pulumi plays an essential role in the appropriate setup of service configuration and environment:

It links an output of one resource to another one’s input, avoiding the need to store the values.
It provides a built-in mechanism to manage external secrets.
It is a great way to take advantage of Azure features like Managed identities and RBAC in a cohesive way.

Infrastructure as Code helps make your applications secure and reliable. Refer to the full example of using Key Vault, Managed identities, RBAC with App Service and Pulumi.

How to Manage Secrets with Pulumi

Matt Ellis — Fri, 17 May 2019 00:00:00 +0000

We recommend Pulumi ESC for managing secrets with Pulumi. Since this blog post, we have launched Pulumi ESC which offers first class support for secrets, getting dynamic short-term credentials from cloud providers, simplifying your configurations with hierarchical environments, and much more.

We’ve had a 1st class concept of encrypted secrets configuration ever since first releasing Pulumi. Customers have told us they love having such a simple and easy way to ensure safe management of tokens, database passwords, and more. Since launching, however, we’ve also heard that you’d like more control over encryption and to see this protection expanded to cover not just configuration, but all of the secret data within their Pulumi deployments.

To support this, we’ve added two new features to Pulumi in our latest 0.17.12 release:

Automatic tracking of secret values throughout a Pulumi program to ensure that all such values are always encrypted in the resulting state, no matter how they are used.
A new option to use custom client-side encryption, instead of the default of using the Pulumi backend for encryption, to have full control over the secrets encryption and decryption.

Together, these features provide you with complete control over how secrets are managed within Pulumi deployments. We have worked with customers with advanced security and compliance needs while developing this feature, enabling them to use our online hosted SaaS with even greater confidence.

Why Secrets Management Matters in Cloud Infrastructure

Like many infrastructure as code systems, Pulumi uses a state file to describe the current state of your infrastructure. When you run pulumi up, Pulumi takes your existing state file, runs your program to compute a new desired state and compares the two states. It makes updates to the current state so it matches the desired state and updates its state file as it does so. As part of this, Pulumi needs to retain all the input values passed to resources in the state file, so it can detect if they have changed from run to run.

While Pulumi has allowed you to pass --secret to force configuration values to be encrypted before being stored in a stack’s configuration file, if you used these configuration values as inputs to resources, they would be stored in plain text in the state file. While the state file itself is stored securely (we encrypt all state files in transit and at rest), anyone with access to the state file itself would be able to see the plain text for all of these secrets.

By adding first class support for secrets with Pulumi, we are now able to automatically track secrets across your program’s execution and ensure that secret values are encrypted in the state file. This means you can use secrets confidently without worrying about accidentally leaking plain text values. Let’s take a look at how it works!

How Pulumi Encrypts & Manages Secrets Automatically

To start, let’s talk a bit about Output, one of the centerpieces of the Pulumi programming model. Output<T> ties together a value (which may not be ready yet, since it could depend on some data from a cloud resource that is still being created) and resources that the output depends on. When you create a resource with Pulumi, the properties of that resource are Output’s that you can pass to other resources. Pulumi uses the information tracked by Output to understand dependencies between different resources. For example when an Output<string> is used to construct a resource, Pulumi knows this resource depends on any resources that were used to generate that output. The underlying value that the Output wraps is what we store in the state file as an input for this new resource.

With 0.17.11 of Pulumi, we now have Output<T> track if it contains secret data. If it does, we ensure that the data is encrypted before we store it in the state file. There are few ways to create secret Outputs today:

By fetching values from the Config object in the JavaScript and Python SDKs, using the newly added getSecret or requireSecret (JavaScript) and get_secret or require_secret (Python), as well as some type specific overloads. These methods fetch the requested value from the configuration bag and then wrap it up in an Output which is marked as a secret. By using pulumi.secret (JavaScript) or pulumi.Output.secret (Python) to take an existing value and wrap it up in an Output which is marked as a secret. These behave the same way as pulumi.output (JavaScript) and pulumi.Output.from_input (Python) except they also mark the returned output as a secret. By retrieving an output that is marked as a secret from a resource.

When constructing a resource that has one or more secret inputs for a property, the entire corresponding output property of the resource is marked as a secret as well. In addition, as you combine outputs together, via all or apply, the resulting output is marked as a secret if any of the inputs values where themselves secrets. This means that just like dependency information, the “secret-ness” of an output flow naturally as you combine it with other data.

Let’s take a look at a small program which creates an AWS Systems Manager parameter, based on a secret configuration value.

Here’s the program we’ll be using:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const cfg = new pulumi.Config();
const secretMessage = cfg.requireSecret("secretMessage");

const param = new aws.ssm.Parameter("secretParameter", {
 type: "SecureString",
 value: secretMessage.apply(s => s.toUpperCase())
});

export const paramId = param.id;
export const paramValue = param.value;

In the above code sample, we’re using the new requireSecret method to pull out a configuration value as a secret. In addition, we use an apply to transform the string to all uppercase before creating our SSM Parameter. Because secretMessage was marked as a secret, this new value is also marked as a secret. It’s important to note the function that runs during the apply has access to the unencrypted value, so you need to be sure that your code inside the apply does not cause the secret to leak (for example, don’t write it to a text file!)

For a demo, let’s create a new stack, target us-west-2 in AWS and set a secret message:

$ pulumi stack init dev
$ pulumi config set aws:region us-west-2
$ pulumi config set --secret secretMessage "it's a secret to everybody"

Now, when we run pulumi up after creating a new stack, we’ll see the following preview:

Previewing update (dev):
Type Name Plan
+ pulumi:pulumi:Stack secrets-blog-dev create
+ └─ aws:ssm:Parameter secretParameter create
Resources:
+ 2 to create

If we look at the details for this deployment (before actually running the update), we can see that the value of this resource has been marked as a secret:

+ pulumi:pulumi:Stack: (create)
[urn=urn:pulumi:dev::secrets-blog::pulumi:pulumi:Stack::secrets-blog-dev]
+ aws:ssm/parameter:Parameter: (create)
[urn=urn:pulumi:dev::secrets-blog::aws:ssm/parameter:Parameter::secretParameter]
name : "secretParameter-1d79dca"
type : "SecureString"
value : "[secret]"

Once we’ve deployed our program, we can use pulumi stack export to look at the state file for our deployment, we see that the value is encrypted there as well (I’ve removed some uninteresting fields here, for clarity):

{
"urn": "urn:pulumi:dev::secrets-blog::aws:ssm/parameter:Parameter::secretParameter",
...snip...
"inputs": {
...snip...
"name": "secretParameter-56f0ffb",
"type": "SecureString",
"value": {
"4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
"ciphertext": "AAABAMo1ZLFpKzoHxUkGPXsUMjLBANri5fkPiveYUrjuMzsqONi2U1LnZSPxsN1vvFTs50skEru+Ff6N"
}
},
"outputs": {
...snip...
"name": "secretParameter-56f0ffb",
"type": "SecureString",
"value": {
"4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
"ciphertext": "AAABAOjpCFOLHMzP4G9OXc4r+mQs6/4DJv2aWO+vX0LyYHLjfawAHWFlRmv3dErda6Ip48pRB19bBL9t"
}
}
...snip...
}

As you can see, the value is encrypted in the state file for this resource! Also note that because the value was marked as a secret input, the corresponding copy in the output section of the state file was also marked as a secret. Pulumi ensures that any outputs with the same names as inputs which had secret data are also considered secrets. If there are additional outputs you want to set as secrets, you can pass the additionalSecretOutputs (JavaScript) or additional_secret_outputs (Python) resource option when constructing a resource to provide a list of other property names you want treated as secrets including computed output properties of a resource which might be sensitive, like generated passwords or access credentials.

Manage secrets with Pulumi ESC

Store encrypted secrets, pull short-lived cloud credentials, and share configuration across your stacks and environments from a single source with Pulumi ESC.

Get started

Configuring Your Secrets Provider

You might be wondering how these values are actually encrypted. We use the same encryption that we have always used for our configuration system. This means when storing state with Pulumi Cloud, we use a key managed by the app.pulumi.com service, specific to your stack, to encrypt everything. Some users have asked for more control over what key is used (and the ability to use a key not managed by Pulumi at all!)

When creating a new stack (via pulumi stack init or pulumi new), you may now pass --secrets-provider passphrase to specify that both configuration secrets and secrets stored in the state file should be encrypted using a key derived from a passphrase (if you’ve used Pulumi’s local state storage mode, this will be familiar to you). When you use a passphrase, we use PBKDF2 to derive a 32 byte encryption key, which we then use with the AES-256-GCM encryption algorithm to encrypt your value (using a random 12 byte nonce per value encrypted). Let’s run through deploying the same code but using the passphrase secret provider:

First, I create a new stack, setting the secrets provider to passphrase:

$ pulumi stack init dev --secrets-provider passphrase
Enter your passphrase to protect config/secrets:
Re-enter your passphrase to confirm:
Created stack 'dev'

As part of creating the stack, I had to enter a passphrase, which I’ll have to use during future updates. This passphrase is used to derive the key used for both configuration and state management. I can now configure my stack as I please:

$ pulumi config set aws:region us-west-2
$ pulumi config set secretMessage --secret "it's a secret to everybody"
Enter your passphrase to unlock config/secrets
(set PULUMI_CONFIG_PASSPHRASE to remember):

Note that to set the secret value, I had to provide my passphrase (since it is needed to generate the key that is used to encrypt the value).

Finally, I can run pulumi up, here I’m prompted to enter my passphrase again. I could also set PULUMI_CONFIG_PASSPHRASE in my environment. You might do this locally as part of your local development loop (so you don’t have to type your passphrase over and over) or in your CI system (where you’d be unable to type your passphrase in interactively).

$ pulumi up
Enter your passphrase to unlock config/secrets
(set PULUMI_CONFIG_PASSPHRASE to remember):

If we use pulumi stack export again to examine the state file, we can see that the structure of the ciphertext has changed:

"value": {
"4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
"ciphertext": "v1:qdfpSdF8vCWRJIDa:4gPQAMRSXi+5ap0koiZBsSVRnqzbp79cSEyWnLYkD9M5S/oI8qhgy521IBA="
}

The change is because we are no longer using the Pulumi service to encrypt or decrypt this data, instead the encryption and decryption happens locally, the data never leaves your machine. So while I get to continue to use <app.pulumi.com> to store state for my stack, I don’t have to worry about my secrets being encrypted with a key managed by a third party.

Support for changing the secrets provider for an existing stack is on its way. To track progress on this feature, please see GitHub issue pulumi/pulumi#481.

What’s Next for Secrets Management

In addition to passphrase based encryption, we plan to add support for encrypting using AWS KMS, Azure KeyVault and GCP KMS in the coming weeks.

The whole team is super excited about this feature and we love how nicely we were able to integrate it into our overall programming model. With these two new features, Pulumi users gain full control over how their secrets are managed, but without sacrificing usability and productivity. We’re excited for you all to start playing around with it! Pulumi is open source, free to use, and works today with variety of clouds. Try it today!

Pulumi Blog: Security

Sign in to Pulumi Cloud with Passkeys

Who this is for

Why passkeys

Setting up a passkey

Signing in

What about my existing password and MFA?

Try it out

Enforce ISO 27001 Across Your AWS Infrastructure

Why ISO 27001 matters

How the pack maps to ISO 27001

Audit and prevent

A growing library of pre-built packs

Get started today

Try Pulumi policies

Deploy a Private Hermes Agent on Render Securely with Pulumi, Modal, and Tailscale

What is Hermes?

Prerequisites

Understanding the Hermes architecture

Setting up ESC for secrets management

Generating the Render SDK

Securing with Tailscale

Deploying the agent

The complete program

Cost

Running the deployment

Access the agent

Verify the deployment (optional)

Security considerations

What’s next?

Conclusion

How We Eliminated Long-Lived CI Secrets Across 70+ Repos

The problem with static CI secrets

Our approach: zero static secrets

What the change looks like

Scale: 70+ repos, zero static secrets

Auditability and centralized control

What happens if a GitHub Action is compromised

Get started

Pulumi IAM Expands: Manage Access at Scale with Tags, Roles, and Teams

Why tag-based access control?

What’s new?

Tag-based access control

Team role assignments

User role assignments

How permissions work together

How to get started

1. Create a custom role with tag-based rules

2. Assign the role to a team

3. Assign a role to an individual user

Enforce tagging standards with Pulumi Policy

Availability

Conclusion

Learn more

Pulumi Cloud Now Supports Google Sign-In

How it works

Signing up or signing in

Connecting Google to an existing account

Google sign-in vs. SAML SSO

Get started

Passwordless PostgreSQL: IAM Authentication with Pulumi

Why IAM authentication for PostgreSQL?

Architecture overview

Setting up the RDS cluster with IAM authentication

Creating the database user with IAM permissions

Configuring IAM policies for database access

Connecting from Kubernetes with IRSA

How IAM authentication works at runtime

Testing the setup with the demo application

Production considerations

Conclusion

Deploy OpenClaw on AWS or Hetzner Securely with Pulumi and Tailscale

What is OpenClaw?

Prerequisites

Understanding OpenClaw architecture

Setting up ESC for secrets management

Securing with Tailscale

Deploying to AWS

The Pulumi program

Deploying to Hetzner