Pulumi Blog: Esc

Preview ESC Changes with Environment Overrides

Sean Yeh — Thu, 23 Jul 2026 00:00:00 +0000

Pulumi ESC makes it easy to store configuration and secrets for your Pulumi programs, and with Approvals for ESC you can review and approve changes before they go live. The new --override-env flag lets you preview any environment change, including an unapproved draft, to see exactly how it would affect your stack before it becomes the latest version.

Example scenario

Your team stores production app configuration in ESC and has enabled Approvals to keep bad values out of critical infrastructure. But one important question remains: how can reviewers properly validate a configuration change before approving it?

Introducing: draft references and the –override-env flag

We are introducing a new --override-env flag to the Pulumi CLI that works with every Pulumi operation that can consume ESC environments: preview, up, refresh, and destroy. This makes it easy to test configuration changes on the fly, giving your team the confidence to validate them as part of the review process.

Example usages

The basic usage is --override-env <env>=<replacement>, and you can override multiple environments at once! In this example, we are running a pulumi preview with environments app/myenv and app/myenv2 replaced by their draft versions (denoted by @draft:<draft-id>).

pulumi preview \
--override-env "app/myenv=app/myenv@draft:123e4567-e89b-12d3-a456-426614174000" \
--override-env "app/myenv2=app/myenv2@draft:123e4567-e89b-12d3-a456-426614174000"

The --override-env flag not only works with draft references, but for any environment too! Here is an example of deploying your stack with your AWS test environment:

pulumi up --override-env "aws-login/prod=aws-login/testing"

Note: --override-env can also override environments that are imported (directly or indirectly) from the ESC environment in your stack config. This allows you to override any environment in the import chain for full flexibility.

Ways to use it

Validate a draft ESC environment with your stack before approving
One-off debugging without editing stack config

Get started

Draft references and --override-env are available today in the Pulumi CLI. Upgrade to the latest release and try it out! To learn more:

The Standalone ESC CLI Retired: Use pulumi env

Pablo Terradillos — Wed, 22 Jul 2026 00:00:00 +0000

Pulumi ESC CLI v0.26.0 is the latest standalone release. We encourage users to use the Pulumi CLI instead.

The ESC repository has been archived and the code now lives under pulumi.

Why are we making this change

Pulumi ESC is the best way to store and manage configuration and secrets in your Pulumi programs and while you can certainly use ESC to store secrets and configurations for your applications or to manage your AI agents’ credentials, it’s still a core feature of Pulumi Cloud, and we want to make sure we deliver a consistent experience across our entire suite.

In addition to consistency, ESC and the Pulumi CLI share a lot of capabilities to interface to Pulumi Cloud and we want to make it easier and simpler for our customers: it’s now easier to understand which Pulumi organization you are working with, which user is logged in, etc.

Playing nicer with humans and AI agents

A single CLI means one binary to install, one login to manage, and one place to look things up. This simplicity is beneficial whether the operator is a person or an agent.

AI agents love code, and they love CLIs. By unifying the Pulumi CLI and ESC we offer a single place to look up operations around your Pulumi programs. Agents can now reason better about where to store configuration and secrets, without being confused by multiple CLI options that interface against the same service.

Faster improvements

With a shared CLI, any improvement to account management is automatically delivered to both: Pulumi and ESC users. At the same time, both benefit from ESC improvements and new features from day 1.

ESC is a core component of our platform; a separate interface no longer makes sense.

The same code all along

If you’re wondering what actually changes under the hood: nothing about how the commands work. The pulumi env commands were never a reimplementation — the Pulumi CLI has built them from the ESC CLI’s own code since ESC’s first release, which is why the same command surface has always been available under pulumi env. What we’re retiring is a second way to ship and install that code, not the code itself.

Do I need to update?

If you are using the ESC CLI, you should switch as soon as possible, as the standalone binary will not get the latest features. Don’t worry — the mapping is mechanical, and the migration guide covers it command by command:

esc env <command> becomes pulumi env <command>. For example, esc env ls becomes pulumi env ls.
The esc open and esc run shortcuts become pulumi env open and pulumi env run.
esc login, esc logout, and esc version become pulumi login, pulumi logout, and pulumi version.

If you are already using the Pulumi CLI, you have nothing to worry about; this has no effect since ESC commands were already available under the pulumi env sub-command.

Ready to switch? Install the Pulumi CLI if you don’t have it yet, then run pulumi env ls to see your environments. Every command is documented in the pulumi env reference.

Connect Your Cloud Accounts to Pulumi in Minutes

Levi Blackstone — Wed, 15 Jul 2026 00:00:00 +0000

Pulumi Insights gives you visibility and governance across your entire cloud footprint, but that visibility is only as complete as the set of accounts you’ve connected. Until now, connecting an account meant repeating a manual setup for each one: OIDC configuration, hand-written Pulumi ESC environments, and per-account scan and policy setup. For an organization with dozens or hundreds of AWS accounts, Azure subscriptions, or Google Cloud projects, that per-account friction was the biggest obstacle to getting a complete picture. Today, the new Connect cloud accounts wizard removes it: discover every account in your cloud organization, select the ones you want, and onboard them all in a single guided flow.

From hours of setup to a single flow

The time savings are real: connecting a batch of accounts across AWS, Azure, and Google Cloud takes about three minutes end to end. Setting up those same accounts with the manual per-account workflow would take hours to days.

The wizard handles the entire onboarding lifecycle for AWS, Microsoft Azure, and Google Cloud:

Bulk discovery: Authenticate once with your cloud organization and the wizard lists every account, subscription, or project you can access. Accounts that are already connected are recognized and skipped.
Automatic OIDC setup: The recommended flows configure each account with short-lived credentials based on OpenID Connect (OIDC) and workload identity federation. No long-lived cloud secrets are stored in Pulumi Cloud.
ESC environments, created for you: The wizard generates Pulumi ESC environments that follow the best practices from the manual OIDC guides — environments that previously had to be authored one by one.
Scans and policies from day one: By default, scheduled discovery scans and a pre-built policy pack are applied to every account as part of setup: the Pulumi Best Practices pack on the Team and Enterprise editions, or a compliance pack (CIS, or NIST 800-53 for AWS and Google Cloud) on Business Critical.

How it works

The wizard walks you through five steps: choose a provider, authenticate, select accounts, configure discovery and policy, and review the results. You can open it from Management > Accounts in the Pulumi Cloud console, or from the Get to know Pulumi card on the home dashboard.

Authentication uses each provider’s native federation mechanism: IAM Identity Center (SSO) for AWS, Microsoft Entra ID workload identity federation for Azure, and Workload Identity Federation for Google Cloud. After you sign in, the wizard discovers the accounts in your organization and pre-selects everything that isn’t already connected. You can search, select all, or toggle individual accounts.

Choose the right access level for your security posture

Not every team wants to grant write access on day one. The wizard offers two access levels, and you can set them per account:

Build & Manage (read and write): Enables the full platform: Pulumi Neo, infrastructure as code, deployments, and policies that remediate issues automatically (Business Critical).
Discovery & Policy (read-only): Limited to discovery scanning and inventory. Pulumi can’t modify your infrastructure.

If your security review requires it, start everything read-only and raise access for specific accounts later.

Everything set up, nothing hidden

When setup completes, the summary shows exactly what was created: the ESC environments grouped by access level, the state of discovery scanning, and the policy pack applied. If any account fails to connect, the summary lists it with the specific error so you can fix the cause and re-run the wizard. Accounts that connected successfully are skipped on retry.

For security reviewers, the docs include a full accounting of what the wizard creates in your cloud provider and in Pulumi Cloud: the IAM roles, app registrations, and service accounts on the cloud side, and the ESC environments and Insights accounts on the Pulumi side.

Get started

The Connect cloud accounts wizard is available now for all Pulumi Cloud organizations. To connect your first accounts:

Navigate to Management > Accounts in the Pulumi Cloud console and select Connect cloud accounts.
Follow the guided flow for AWS, Azure, or Google Cloud.
Explore your discovered resources and policy findings.

To learn more:

Connect cloud accounts documentation — prerequisites, each wizard step in detail, and troubleshooting
Insights & Governance overview — full documentation for discovery and policy capabilities
Pulumi ESC — how the generated environments manage cloud credentials with OIDC

Introducing ESC Secret Rotation Webhooks

Sean Yeh — Fri, 26 Jun 2026 00:00:00 +0000

Pulumi ESC centralizes your secrets and configuration, and it can automatically rotate secrets on a schedule so credentials never go stale. But a rotation is only useful if the systems that depend on it know it happened. ESC secret rotation webhooks close that gap by notifying you the moment a secret rotates.

Introducing secret rotation webhooks

With ESC webhooks, you can react to rotations automatically. When ESC rotates an environment’s secrets, a webhook can be configured to trigger on either success or failure. Use it to notify your team in Slack, refresh services that hold the old credential, or catch a failed rotation before it causes an outage.

How to configure

Pulumi Cloud Console

Using the Pulumi Cloud Console, you can now configure webhooks for “Environment rotation succeeded” and “Environment rotation failed” in your ESC Environment’s Settings page (under Settings -> Notifications).

Pulumi Service Provider

You can use the Pulumi Service Provider in your Pulumi program to configure webhooks. Here is an example in TypeScript:

const environmentWebhook = new service.Webhook("env-webhook", {
 active: true,
 displayName: "env-webhook",
 organizationName: "my-org",
 projectName: environment.project,
 environmentName: environment.name,
 payloadUrl: "https://example.com",
 filters: [WebhookFilters.EnvironmentRotationSucceeded, WebhookFilters.EnvironmentRotationFailed],
})

Pulumi CLI

You can also use the Pulumi CLI to configure the webhook:

pulumi env webhook new my-org/project/env env-webhook \
--url https://example.com \
--event environment_rotation_succeeded \
--event environment_rotation_failed

Get started

Secret rotation webhooks are available now for all Pulumi ESC environments. See the webhooks documentation to get started, and share your feedback on our GitHub repository.

Seven Rules for Building an AI-Native Software Factory

Ewan Dawson — Thu, 21 May 2026 06:00:00 -0700

Ewan Dawson is CTO of Compostable AI, where five engineers run an AI-native software factory: nineteen clients, custom AWS deployments, most of them shipped within a day of contract signing. This article is adapted from his recent Pulumi webinar, and covers rules in more depth than we had time for on stage.

For the past twenty years, I’ve viewed software development as a craft. The best engineers drew on decades of experience to get every function right.

But two years into the agentic AI revolution, I realised software is going to look more like a factory than a craft. The economics have changed. We can’t treat code as bespoke anymore. To scale, we have to think industrial — use the tools to ship more value with fewer engineers.

I joined Compostable AI soon after it was founded 2.5 years ago, and I built the engineering org AI-native from day one. The technology has come a long way since then, and so has my understanding of what AI-native actually means. Here are seven rules I keep coming back to.

An AI software factory is a software operation where autonomous agents write and ship most of the code. The engineers stop writing it by hand and spend their time deciding what gets built and talking to customers. The rules below are our rules for building and running an AI-native software factory.

1. Transform, don’t enhance

Going AI-native isn’t an upgrade to your existing process. If you treat AI as a way to hand your developers smarter tools, you leave most of the value on the table. You get the leverage by rebuilding how you write software — and the culture and processes around it.

I know that’s a tall order for a large, mature engineering org. My advice: start small. Pick one team or one business area and run it as a fully AI-native function. Take what you learn and roll it out from there. And do the political work early, especially with your Governance, Risk, and Compliance function. Get GRC on your side early. Otherwise AI becomes a compliance fight instead of a structural advantage.

Don’t bolt AI onto your existing workflow. Redesign the workflow around what agents can do.

Most of the leverage in this technology comes from rebuilding around it. The tool change is the small part.

2. Remove the problem, don’t solve it

Going AI-native flips which problems are hard and which are easy. The right move often isn’t to engineer a solution. It’s to reframe the problem so it goes away.

Here’s an example. Serving multiple clients with agents writing the code, blast radius wasn’t a hypothetical. One bad agent run could trash a customer’s database, or leak one client’s data into another’s. Our instinct was to build a secure multi-tenant sandbox with guardrails, approvals, rollback. But every version we tried still had agents loose in a shared environment, one bug away from making one customer’s data visible to another’s. So we removed the problem: every client gets two dedicated AWS accounts, one for production and one “digital twin” staging account. Agents iterate on staging until the work checks out. Only then does it ship to production. We have nineteen accounts now, one per client.

Managing nineteen AWS accounts with five engineers used to be an administrative nightmare. When code is cheap, infrastructure-as-code tools like AWS Control Tower and Pulumi make it the easier path.

Remove the problem before you try to solve it.

It’s cheaper to reframe the problem than to engineer your way through it.

3. Pick tools your agents can drive

Removing problems is the process side. The other side is tooling. If you want an automated factory, your tech stack has to be something agents can drive. This overlaps a lot with tools that have great developer experience. If a tool has a robust API plus a clean CLI, agents can drive it. If it’s heavy click-ops around a web UI, agents stop there.

We didn’t get there first try. Our first IaC tool worked fine when we had a couple of clients. As we added more, accounts drifted, deployments slowed, retries got complicated. We needed something built for where we were heading.

I went looking, and Pulumi fit. We express infrastructure as type-safe code — TypeScript, in our case, rather than HCL — and agents are good at writing it. Pair that with Pulumi Neo — pre-loaded with domain-specific Pulumi skills — and we ship infrastructure that follows best practices. One of my colleagues put it: “The scary thing about Neo is it just seems to know everything about what we do.” Pulumi IaC plus Pulumi ESC for configuration beats stitching tools together. And TypeScript lets us build higher-level abstractions that keep the AWS account fleet tractable.

“I don’t actually care if it’s HCL or TypeScript, as long as my software development agents can write it. And they do a better job with TypeScript than HCL.”

Tools have to share your AI-native mindset. If they don’t integrate deeply, the human becomes the glue.

If part of your stack still requires a human to click through a web UI to provision an account, your agents stop there.

4. Don’t let one agent do everything

When I first started with agents, I reached for a god prompt: one massive system prompt meant to guide a single agent through the whole software lifecycle. It didn’t work. Agents struggle when you give them multiple goals. The writer is lenient on its own work — it won’t catch what it just shipped. You don’t want it reviewing the code, checking for security flaws, or hunting bugs.

We get better results from a constellation of specialized agents, each handling one part of the line. Pulumi Neo handles infrastructure. Alongside it sit agents specialized in:

Code implementation
Code review and testing
Security auditing
Internal standards compliance
Documentation updates

Tasks pass down the line. Clean code comes out the other end, with almost no human involved.

Don’t let any agent mark its own homework. Specialize by job.

Treat agents the way you’d treat a team. The one who writes the code shouldn’t be the one signing it off.

5. Measure human hours per unit of value

Once we had agents writing and agents reviewing, throughput went up — but the bottleneck moved past the PR. Engineering hours were still the most expensive thing in the building, so my core metric is human hours per unit of value produced. Minimize that.

That means hunting for every step that still goes through a person — especially the mid-pipeline steps between ideation and production. Automate the human touchpoints along that line, and the factory runs 24/7.

Pushing automation this hard also forces good engineering. A chaotic, undocumented process is impossible to automate. Good engineering is still good engineering, AI or not. Agents won’t fix a weak process.

Measure human hours per unit of value. Treat every one as a bottleneck to remove.

You can’t automate what you can’t describe. Every human in the pipeline marks a piece that hasn’t been described yet.

6. Design for convergence, not one-shot correctness

Even with the human touchpoints removed, the agents don’t ship right the first try. Once you embrace the factory pipeline, you stop needing them to. We design for convergence instead — a system that lands on the right answer through automated iteration.

The loop we run looks like this:

Refinement: agents iterate on the Product Requirements Document until the problem is clear.
Planning: agents draft multiple technical approaches, and evaluation agents pick the best one.
Implementation: coding agents write the software.
Review: specialized checking agents look for bugs, API misuse, and security flaws.

If the checkers find a problem, they hand it back to the implementation agent. The loop repeats until the tests pass and the agents agree on a clean PR. Once it converges, we merge and deploy to staging.

Two things have to be true. You need a way to evaluate the output. Without that, you don’t know when to stop. And the loop has to converge — each pass has to get closer. A checker that fails every PR for a different reason isn’t helping — it just keeps the work going in circles. The feedback has to narrow the search, not widen it.

Once it converges, the question moves on. How cheap can we make it? Lower the time to PR, reduce token count, drop the overall cost. The optimization never really ends.

Don’t aim for one-shot correctness. Design for convergence.

It doesn’t matter how many tries it takes, as long as the loop closes without a human in it. Get convergence first. The optimization comes after.

7. Run the factory in the cloud, not on a laptop

Even a converged factory has to live somewhere. Try running a fully automated factory on individual developers’ laptops, and it falls apart. Laptops are highly trusted machines. Put autonomous agents on them and your security posture drops, fast. And the factory has to run 24/7. Events come from elsewhere — PR comments, Slack threads, errors in test environments.

Cloud also kills configuration drift across a dozen developer machines. The same prompts run against different model versions, and env vars sit half-set on half the laptops. The thing you’re trying to optimize lives in different states across the team. Cloud isn’t just where the factory runs; it’s the only place a team can iterate on it together. Keep everything in one place — AWS, Pulumi Cloud, GitHub. The specific stack matters less than the principle of one place.

And the part that matters most: the factory keeps running, testing, and deploying long after we’ve closed our laptops and gone to sleep — the dark factory pattern, where the line keeps producing with the lights off.

Build the factory somewhere you can work on it — not just somewhere it can run.

A factory scattered across laptops can’t be improved as a system. Cloud keeps it in one shape, 24/7, and lets the team iterate together.

How to build an AI-native software factory

There’s no single blueprint for how to build a software factory, but these seven rules are the software factory model we run at Compostable AI: transform the workflow rather than bolt AI onto it, remove problems instead of engineering around them, pick tools your agents can drive, split work across specialized agents, measure human hours per unit of value, design for convergence over one-shot correctness, and run the factory in the cloud so it operates 24/7.

Closing thought

I’ve shipped more code in the last two years than I did in the fifteen before that. Most of it in languages I couldn’t write by hand. And that’s after a stretch in leadership where I wrote almost none.

If you’re where I was two years ago: don’t ask how AI fits into what you already do. The factory is built one rule at a time, and it’s not a template — it’s the practice of finding where you’re taking advantage of the new economics and where you’re not, where your practices still need an update. The leverage is in finding these places and improving them.

Frequently asked questions

What is a software factory?

A software factory runs software delivery as a repeatable industrial process instead of hand-crafting each feature, the way a physical factory turns out products on a line. The idea predates AI. An AI-native software factory is the version where agents do most of the building and the engineers steer.

How is a software factory different from traditional software development?

Traditional development treats code as a craft, each function written and reviewed by hand. A software factory treats software delivery as an industrial process: agents handle the writing and checking through an automated, converging loop, and the expensive human hours move to defining the work and deciding what to build. The goal is to minimize human hours per unit of value shipped.

How do you keep AI agents from breaking things across customers?

Rather than build a shared multi-tenant sandbox, Compostable AI removes the problem: every client gets two dedicated AWS accounts, one for production and a “digital twin” staging account. Agents iterate on staging until the work checks out, and only then does it ship to production. Infrastructure-as-code tools like AWS Control Tower and Pulumi make running that account fleet tractable for a small team.

What tools does an AI-native software factory need?

Tools your agents can actually drive, anything with a solid API and a clean CLI, rather than click-ops around a web UI. Compostable AI expresses infrastructure as type-safe TypeScript with Pulumi, pairs it with Pulumi Neo for domain-specific infrastructure skills, and uses Pulumi ESC for configuration. If part of your stack still requires a human to click through a UI, your agents stop there.

Watch the original Pulumi webinar. Read the Compostable AI case study, and learn more about Compostable AI and Pulumi Neo.

Neo Integrations: MCP Servers and Cloud CLIs

Pulumi Neo Team — Wed, 20 May 2026 00:00:00 +0000

Pulumi Neo already understands your infrastructure: your code, your stacks, your state. Today we’re launching new capabilities that extend Neo’s reach in two directions: into the third-party systems your team uses to plan and observe, and out to the cloud CLIs that actually drive your infrastructure.

The first half is MCP integrations: connections to Atlassian, Datadog, Honeycomb, Linear, PagerDuty, and Supabase that show up as tools Neo can call during a task. The second half is CLI integrations: scopable access to aws, gcloud, az, and kubectl. Both are configured once at the org level and available to every Neo task in the organization.

Integrations in action

A PagerDuty alert just fired: RDS storage on payments-prod is at 90% and climbing. You want to know how fast, and whether you can buy yourself any runway before it fills.

You: Neo, RDS storage on payments-prod just paged at 90%. How fast is it growing, and what do we have configured?

Neo pulls the active incident from PagerDuty, decides on its own to check Datadog for the storage-utilization curve over the last 30 days, and runs aws rds describe-db-instances --db-instance-identifier payments-prod through your production-aws CLI integration (the name your org gave its production AWS credentials). The database has been growing about 5 GB a day. The instance has AllocatedStorage at 200 GB and MaxAllocatedStorage also at 200, so storage autoscaling is effectively disabled. At current growth, the disk fills in three days.

You: Bump max allocated storage to 500. Open a PR.

Neo edits the payments stack’s Pulumi program to raise maxAllocatedStorage from 200 to 500 on the RDS instance, runs pulumi preview to confirm the change is scoped to that one resource, and opens a pull request with the diff, the preview output, and links to the PagerDuty incident and the Datadog graph. You review the PR and merge it. Pulumi applies the change, and Neo posts the resolution back to PagerDuty.

With three integrations and one conversation, the change is reviewed, shipped, and the alert resolved a few minutes later.

MCP integrations: context from your existing tools

The launch catalog covers six services that show up most often in infrastructure investigations: Atlassian for Jira issues and Confluence runbooks, Datadog for metrics and logs, Honeycomb for traces, Linear for issue tracking, PagerDuty for incidents and on-call schedules, and Supabase for managed database changes. Each connects Neo to a remote MCP server hosted by the provider, so the agent has access to the full set of tools the vendor chooses to expose.

Integrations can be enabled by organization administrators on the Neo Settings page. Once configured, they’re available to every Neo task in your organization.

CLI integrations: live cloud insights

CLI integrations cover what MCP doesn’t reach: live cloud insights. With AWS, GCP, Azure, or Kubernetes connected, Neo can check live database utilization, look up the current state of a running service, verify a service quota before scaling, or reach into resources that aren’t managed by any Pulumi stack.

An admin enables a CLI integration the same way as an MCP one, from your org’s Neo settings. Each integration gets a name your team chooses, like production-aws or staging-gcloud, and tasks reference that name to tell Neo which environment to reach into. You can connect multiple instances of the same CLI (for example, production-aws and staging-aws) so Neo can investigate staging without touching production. Credentials are backed by Pulumi ESC environments your org owns; the CLI integrations docs walk through setup.

Per-task control and failure handling

Both surfaces default to org-wide availability, with per-task overrides. Before starting a task, you can toggle individual MCP integrations off. The toggles only affect that task; the org-level configuration is unchanged.

Failures behave the same way for both. If an integration can’t be reached, Neo logs a warning, skips it, and continues with the rest. A single broken integration doesn’t stop a task. CLI integration connect and disconnect events go to your organization’s audit log, and Neo’s individual CLI calls appear in the task transcript alongside its other tool calls.

Try it out

Both MCP and CLI integrations are available now for Neo-enabled organizations. Open your org’s Neo settings, connect the MCP server or CLI of your choice, and let Neo do the next investigation against the tools you already use. The MCP integrations docs and CLI integrations docs walk through credential setup for each one, and the Neo integrations hub ties it all together.

Today’s launch is part of a bigger story. Read our launch-day piece on the agentic infrastructure era for the broader vision, and the Neo CLI launch post for Neo’s new home in the terminal.

As always, we’d love to hear what you think — and if you have any suggestions for integrations that’d make Neo even better, file an issue in pulumi-cloud-requests.

Policy Packs Can Now Access Pulumi ESC Environments

Dan Biwer — Thu, 23 Apr 2026 00:00:00 +0000

Policy authors who need external credentials or environment-specific configuration have had to hardcode values or manage them outside of Pulumi. Policy packs can now reference Pulumi ESC environments, bringing centralized secrets and configuration management to your policies.

The problem

Pulumi policy packs let you enforce rules across your infrastructure, but some policies need more than just the resource inputs they evaluate. A policy that validates resources against an external compliance API needs an API token. A cost-enforcement policy might need different spending thresholds for development and production environments. An access-control policy might need to reference an internal service registry.

Until now, these values had to be hardcoded in your policy group configuration or managed through a separate process entirely. This created several problems:

Security risk: Credentials stored in plain text in policy group config
Operational burden: Updating a credential meant touching every policy group that used it
No environment separation: The same values applied everywhere, with no way to vary configuration across environments

What’s new

Policy packs can now reference ESC environments, just like stacks already do. When you attach an ESC environment to a policy pack in a policy group, the values from that environment are available to your policies at runtime — whether you’re running preventative or audit policies.

This means your policy packs can use ESC for:

Secrets: API tokens, service credentials, and other sensitive values managed through ESC’s secrets management, including dynamic credentials from providers like AWS, Azure, and GCP
Configuration: Environment-specific thresholds, allowed regions, service allowlists, and other policy parameters that vary across environments

How it works

You configure ESC environment references on a policy pack within a policy group. At runtime, the values from those environments are resolved and made available to your policies through the policy pack’s configuration.

Here’s an example ESC environment that provides configuration to a compliance policy pack:

values:
 compliance:
 apiToken:
 fn::secret: xxxxxxxxxxxxxxxx
 costThreshold: 5000

 policyConfig:
 cost-compliance:
 maxMonthlyCost: ${compliance.costThreshold}
 apiEndpoint: https://compliance.example.com
 apiToken: ${compliance.apiToken}

The policyConfig property works just like pulumiConfig does for stacks. Values nested under each policy name are made available as configuration to that policy at runtime. Secrets remain encrypted and are only decrypted when the environment is resolved.

You can also use the environmentVariables property to inject values as environment variables into the policy runtime, following the same pattern as stack environment variables.

Example: compliance API validation

Consider a policy that validates every new resource against an external compliance API before it can be provisioned. The API requires an authentication token and returns whether the resource configuration meets your organization’s compliance standards.

Before, the API token lived in the policy group configuration in plain text. Rotating the token meant updating every policy group. There was no audit trail for who accessed the credential, and no way to use different API endpoints for staging and production compliance checks.

After, the API token lives in an ESC environment. You get:

Centralized rotation: Update the token in one place and every policy group that references the environment picks up the change
Access controls: ESC’s role-based access controls govern who can view or modify the credential
Audit trail: Every access to the environment is logged
Environment separation: Use different ESC environments for different policy groups, so staging policies validate against a staging compliance endpoint while production policies use the production endpoint

Get started

To start using ESC environments with your policy packs:

Create an ESC environment with your policy configuration and secrets
Attach the environment to a policy pack in your policy group through the Pulumi Cloud console
Update your policies to read from the configuration values provided by the environment

To learn more:

Automate Azure App Secret Rotation with ESC

Sean Yeh — Mon, 06 Apr 2026 00:00:00 +0000

Microsoft Entra ID (formerly Azure Active Directory) is Azure’s identity and access management service. Any time your application needs to authenticate with Entra ID, you create an app registration and give it a client secret that proves its identity. But those secrets expire, and if you don’t rotate them in time, your app loses access.

If you or your team manages Azure app registrations, you know that keeping track of client secrets is a constant hassle. Forgetting to rotate them before they expire can lead to broken authentication and unexpected outages. With Pulumi ESC’s azure-app-secret rotator, you can automate client secret rotation for your Azure apps, so you never have to worry about expired credentials again.

Setup

Prerequisites

An Azure App Registration
An azure-login environment
- Note for OIDC users: Since Azure does not support wildcard subject matches, you will need to add a federated credential for the azure-login environment as well as each environment that imports it.
- The Azure identity used for rotation must have the Application.ReadWrite.All Graph API permission, or the identity must be added as an Owner of the specific app registration whose secrets will be rotated.

How it works

Let’s assume your azure-login environment looks like this:

# my-org/logins/production
values:
 azure:
 login:
 fn::open::azure-login:
 clientId: <your-client-id>
 tenantId: <your-tenant-id>
 subscriptionId: <your-subscription-id>
 oidc: true

Create a new environment for your rotator. If you have the existing credentials, set them in the state object so the rotator will treat them as the current credentials.

# my-org/rotators/secret-rotator
values:
 appSecret:
 fn::rotate::azure-app-secret:
 inputs:
 login: ${environments.logins.production.azure.login}
 clientId: <target-app-client-id>
 lifetimeInDays: 180 # How long each new secret is valid (max 730 days)
 state:
 current:
 secretId: <secret-id>
 secretValue:
 fn::secret: <secret-value>

The lifetimeInDays field controls how long each generated secret remains valid before it expires. Azure allows a maximum of 730 days (two years), but shorter lifetimes are recommended for better security. Make sure to set a rotation schedule that runs before the lifetime expires so your credentials are always fresh.

Azure app registrations can have at most two client secrets at any given time, so the rotator maintains a current and previous secret. When a rotation occurs, the existing current secret becomes the previous secret, and a new secret is created to take its place as the new current. This ensures a smooth rollover with no downtime, since the previous secret remains valid until the next rotation.

Once this is set up, you’re ready to go! You never need to worry about your client secrets expiring, and you will always have the latest credentials in your ESC Environment.

Learn more

The fn::rotate::azure-app-secret rotator is available now in all Pulumi ESC environments. For more information, check out the fn::rotate::azure-app-secret documentation!

How We Eliminated Long-Lived CI Secrets Across 70+ Repos

Boris Schlosser — Tue, 31 Mar 2026 00:00:00 +0000

Supply chain attacks on CI/CD pipelines are accelerating. A growing pattern involves attackers compromising popular GitHub Actions through tag poisoning — rewriting trusted version tags to point to malicious code that harvests environment variables, cloud credentials, and API tokens from runner environments. The stolen credentials are then exfiltrated to attacker-controlled infrastructure, often before anyone notices.

For every engineering organization, the question is no longer if your CI pipeline will encounter a compromised dependency, but what is exposed when it does.

At Pulumi, we asked ourselves that question and decided the answer should be “nothing useful.” Here’s how we got there.

The problem with static CI secrets

Most organizations store long-lived cloud credentials, API tokens, and service account keys as GitHub repository or organization secrets. But this approach has several well-known problems:

Broad availability. Every workflow run on a repository can access every secret stored in that repo. A compromised action in any workflow can read them all.
No expiration. Secrets persist until someone manually rotates them. If exfiltrated, they give attackers persistent access for weeks or months.
No granular audit trail. GitHub tells you a secret was used, but not which workflow, which step, or what it was used for.
Secret sprawl. Across dozens or hundreds of repos, the same credentials are often duplicated, making rotation a coordinated, error-prone effort.

In a supply chain attack scenario, this is exactly what attackers count on: a single compromised action that can dump a trove of long-lived credentials.

Our approach: zero static secrets

We replaced every static GitHub Secret across our CI pipelines with short-lived, dynamically fetched credentials using Pulumi ESC and OpenID Connect (OIDC). The credential flow works in layers, each scoped and ephemeral:

GitHub generates a short-lived OIDC token scoped to the specific workflow run, repository, and branch. This token is cryptographically signed by GitHub’s OIDC provider.
The token is exchanged with Pulumi Cloud for a short-lived Pulumi access token. Pulumi Cloud validates the OIDC claims (organization, repository, branch) against a configured trust policy before issuing the token.
The Pulumi access token opens an ESC environment to retrieve the credentials the workflow needs — cloud provider keys, API tokens, or other secrets.
Cloud credentials themselves are dynamic. ESC environments use OIDC login providers to fetch short-lived credentials directly from AWS, Azure, or GCP. No static keys or cloud credentials are stored anywhere.

The pulumi/esc-action GitHub Action handles this entire flow in a single workflow step.

sequenceDiagram
participant Runner as GitHub Actions Runner
participant GH as GitHub OIDC Provider
participant PC as Pulumi Cloud
participant ESC as Pulumi ESC
participant Cloud as Cloud Provider (AWS/Azure/GCP)
Runner->>GH: Request OIDC token (scoped to workflow run)
GH-->>Runner: Short-lived JWT
Runner->>PC: Exchange JWT for Pulumi access token
PC-->>Runner: Short-lived access token
Runner->>ESC: Open environment with access token
ESC->>Cloud: OIDC login (assume role / federated identity)
Cloud-->>ESC: Short-lived cloud credentials
ESC-->>Runner: Cloud credentials + secrets
Note over Runner,Cloud: Nothing is stored. Everything expires.

What the change looks like

Before this migration, our workflows referenced static secrets stored in GitHub:

env:
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

After the migration, an ESC environment handles credential fetching via OIDC. Here is what the environment definition looks like:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-role
 sessionName: esc-${context.pulumi.user.login}
 # Optional: scope down the session beyond what the role allows
 policyArns:
 - arn:aws:iam::123456789012:policy/ci-build-minimal
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}

The roleArn and optional policyArns make least-privilege straightforward: each login provider assumes a specific role, and policyArns can scope the session down further. You can use multiple login providers in one environment or separate environments per workflow to match permissions to each job’s needs.

The workflow itself becomes minimal — a single step that authenticates via OIDC and injects the credentials:

permissions:
 contents: read
 id-token: write  # Required for OIDC

steps:
 - name: Fetch secrets from ESC
 uses: pulumi/esc-action@v3
 with:
 environment: '<your-organization>/<your-esc-env>'

The static secrets.* references are gone entirely. Every credential is fetched at runtime through ESC.

Scale: 70+ repos, zero static secrets

We didn’t do this for one or two flagship repos; we rolled it out across every Pulumi provider repository: AWS, Azure, GCP, Kubernetes, and over 60 more. The migration was managed centrally through our ci-mgmt tooling, which generates consistent workflow configurations across all provider repos.

The pattern is the same everywhere:

Each repo has a corresponding ESC environment under a github-secrets/ project.
All workflow-level ${{ secrets.* }} references have been removed.
The pulumi/esc-action step with OIDC auth is the single entry point for all credentials.

When every repo follows the same pattern like this, security posture is much more easily verifiable and auditable.

Auditability and centralized control

Beyond eliminating static secrets, this migration gave us centralized visibility and control that GitHub Secrets cannot provide:

Audit logging. ESC records which credentials were accessed, when, and by which workflow. This is a meaningful improvement over GitHub’s binary “secret was used” signal.
Centralized access policies. Access rules are defined once in ESC rather than scattered across individual repository settings pages.
Single-point rotation. Because ESC environments can import other environments, shared credentials live in a common base that all 70+ repo environments are composed of. Update it once, and every repo picks up the change on its next run.
Dynamic credentials by default. For cloud providers like AWS, Azure, and GCP, ESC fetches credentials via OIDC at open time. There is nothing to rotate because nothing is stored.

What happens if a GitHub Action is compromised

With this architecture in place, here is what an attacker gets if a compromised GitHub Action runs in our CI:

No GitHub Secrets to dump. The repository settings page has no stored secrets for a malicious action to exfiltrate.
OIDC tokens are scoped and short-lived. The GitHub-issued JWT is valid only for the specific workflow run and expires within minutes.
Cloud credentials are ephemeral. Any AWS, Azure, or GCP credentials fetched through ESC are short-lived and scoped to the role assumed during that run.
No persistent access. There are no long-lived tokens to reuse hours or days later.

Compare this to the traditional model, where a single compromised action could exfiltrate AWS access keys that remain valid until someone manually rotates them — which could be weeks or months.

The goal is not to prevent every possible attack. It is to make the blast radius as small as possible when something goes wrong.

Get started

If you want to adopt the same pattern in your own CI pipelines:

Tutorial: Using ESC with GitHub Actions — Step-by-step setup guide.
Announcing the Pulumi ESC GitHub Action — Full feature overview and capabilities.
Configuring OIDC for ESC — Set up OIDC trust between ESC and your cloud providers.
Pulumi ESC documentation — Full documentation for environments, secrets, and configuration.

Your CI secrets do not have to be a liability. With OIDC and Pulumi ESC, they do not have to exist at all.

Lock Down Values in Pulumi ESC with fn::final

Pablo Terradillos — Tue, 17 Mar 2026 11:00:00 -0700

Pulumi ESC (Environments, Secrets, and Configuration) allows you to compose environments by importing configuration and secrets from other environments, but this also means a child environment can silently override a value set by a parent. When that value is a security policy or a compliance setting, an accidental override can cause real problems. With the new fn::final built-in function, you can mark values as final, preventing child environments from overriding them. If a child environment tries to override a final value, ESC raises a warning and preserves the original value.

How it works

Let’s say you have a parent environment that sets the AWS region for all deployments. You can use fn::final to ensure no child environment can change it:

# project/parent-env
values:
 aws-region:
 fn::final: us-east-1

If a child environment tries to override the final value, ESC raises a cannot override final value warning.

# project/child-env
imports:
 - project/parent-env
values:
 aws-region: eu-west-1 # raises a warning

This evaluates to:

{
 "aws-region": "us-east-1"
}

In this scenario, the ESC environment is still valid, but the final value remains unchanged.

When to use fn::final

Use fn::final for:

Security-sensitive values that shouldn’t be changed
Compliance or policy settings enforced by a platform team
Shared base environments where certain values must remain consistent

Getting started

The fn::final function is available now in all Pulumi ESC environments. For more information, check out the fn::final documentation!

How We Load Data into Snowflake in Seconds with Pulumi

Pablo Seibelt — Mon, 23 Feb 2026 00:00:00 +0000

When you manage dozens of data-loading pipelines, copying and pasting IaC configurations between them is a recipe for mishap. IAM policies can drift, naming conventions diverge, and every new source is a new opportunity to make a mistake — not to mention compound the problem of duplication. In this post, we’ll show you how you can identify and encapsulate common patterns into composable components and walk through the production lessons we’ve learned running 25+ pipelines for over three years.

What we’ll cover

If you’re loading data into Snowflake and want reusable, composable infrastructure, this post is for you. Here’s what we’ll cover:

Handling and validating GitHub webhooks with AWS Lambda
Streaming webhook payloads directly into Snowflake with Amazon Data Firehose
Wiring it all up with a reusable Pulumi ComponentResource

The companion template also includes S3 auto-ingest and batch loading patterns, which we’ll cover in upcoming posts. We also use Pulumi ESC to handle authentication to both AWS and Snowflake using OpenID Connect.

Our own Josh Kodroff wrote an excellent introduction to Snowpipe with Pulumi. This post builds on his work using the newest Snowflake and AWS provider APIs and the direct Firehose-to-Snowflake destination, which wasn’t available when Josh wrote his post. Some resource names and grant patterns will also differ if you’re comparing the two.

Architecture overview

The following diagram shows the architecture in more detail:

GitHub sends webhook events to a Lambda Function URL.
Lambda validates the HMAC (Hash-based Message Authentication Code) signature and forwards the payload to Amazon Data Firehose.
Firehose streams records directly into Snowflake via the Snowpipe Streaming API. Data appears in Snowflake within seconds.
S3 is used only as a backup destination for failed records.

The direct Firehose-to-Snowflake destination is an AWS-native feature that works with any Snowflake account.

Project setup

Start a new Pulumi Python project and choose uv for dependency management when prompted:

mkdir snowpipe-data-loading && cd snowpipe-data-loading
pulumi new python

Notice Pulumi.yaml shows uv as your selected toolchain:

name: snowpipe-data-loading
runtime:
 name: python
 options:
 toolchain: uv

Add the provider dependencies for AWS, Snowflake, GitHub, and the Random and TLS providers:

uv add pulumi-aws pulumi-snowflake pulumi-github pulumi-random pulumi-tls

That’s it. uv creates the virtual environment and lockfile automatically, and Pulumi uses uv run under the hood to execute your program.

All examples in this post are in Python, but Pulumi supports multiple languages. You can implement the same components in TypeScript, Go, .NET, Java, or YAML.

Managing credentials with Pulumi ESC

This project needs credentials for AWS, Snowflake, and GitHub. Rather than managing long-lived secrets locally, we can use Pulumi ESC to obtain dynamic, short-lived OIDC credentials for AWS and Snowflake at runtime. When you run pulumi up, ESC exchanges a Pulumi-issued OIDC token for temporary credentials from each provider and injects them into your stack config automatically. If you prefer not to use ESC, you can set credentials directly with pulumi config set --secret.

Here is a single ESC environment that handles all three providers:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-oidc
 sessionName: pulumi-snowpipe-demo
 snowflake:
 login:
 fn::open::snowflake-login:
 oidc:
 account: <your-snowflake-account>
 user: ESC_SERVICE_USER
 organizationName: <your-org-name>
 accountName: <your-account-name>
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.login.sessionToken}
 SNOWFLAKE_USER: ${snowflake.login.user}
 SNOWFLAKE_TOKEN: ${snowflake.login.token}
 pulumiConfig:
 aws:region: us-west-2
 snowflake:organizationName: ${snowflake.organizationName}
 snowflake:accountName: ${snowflake.accountName}
 snowflake:authenticator: OAUTH
 snowflake:role: PULUMI_DEPLOYER
 github:token:
 fn::secret: <your-github-pat>
 github:owner: <your-github-org-or-user>

Then reference the environment from your stack config:

# Pulumi.<stack>.yaml
environment:
 - <project>/my-snowpipe-env
config:
 snowpipe-data-loading:database: LANDING_ZONE_WEBHOOKS
 snowpipe-data-loading:environment: dev
 snowpipe-data-loading:webhook-repo: <your-test-repo-name>
 snowflake:previewFeaturesEnabled:
 - snowflake_table_resource

Depending on your preferences, you can split credentials into separate per-provider environments and compose them with imports and reuse across stacks.

To set up OIDC trust for each provider, see the AWS OIDC guide and the Snowflake OIDC login guide. For GitHub authentication options (fine-grained PATs, classic PATs, or GitHub Apps), see the pulumi-github provider docs.

Shared infrastructure

The direct streaming pipeline needs an S3 bucket for backup/errors, a Snowflake database, and a schema.

Building the direct ingestion ComponentResource

Amazon Data Firehose supports Snowflake as a native destination via the Snowpipe Streaming API. Firehose streams records directly into Snowflake.

The Lambda handler

The Lambda function is the entry point for GitHub webhooks. It validates the HMAC-SHA256 signature, wraps the payload in an envelope with the event type, and forwards it to Firehose. Create this as lambda/webhook_handler.py:

import hashlib
import hmac
import json
import os

import boto3

firehose = boto3.client("firehose")

STREAM_NAME = os.environ["FIREHOSE_STREAM_NAME"]
WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"]


def handler(event, context):
 body = event.get("body", "")
 signature = (event.get("headers") or {}).get("x-hub-signature-256", "")

 # Validate HMAC-SHA256 signature
 expected = "sha256=" + hmac.new(
 WEBHOOK_SECRET.encode(), body.encode(), hashlib.sha256
 ).hexdigest()

 if not hmac.compare_digest(expected, signature):
 return {"statusCode": 401, "body": "Invalid signature"}

 github_event = (event.get("headers") or {}).get("x-github-event", "unknown")

 # Wrap in envelope - newline-delimited for S3 backup where Firehose concatenates records
 record = json.dumps({
 "github_event": github_event,
 "payload": json.loads(body),
 }) + "\n"

 firehose.put_record(
 DeliveryStreamName=STREAM_NAME,
 Record={"Data": record.encode()},
 )

 return {"statusCode": 200, "body": "OK"}

The envelope format {"github_event": "<type>", "payload": {...}}\n is important. The github_event field (e.g., push, pull_request, star) comes from the x-github-event header and lets downstream queries filter by event type. The trailing newline delimits records in the S3 backup destination, where Firehose concatenates them into files.

Another interesting part of this snippet is that instead of manually creating a secret which would have to be copy-pasted into both your webhook configuration and your Lambda environment, we are using random.RandomPassword to generate and store it securely it in Pulumi state.

The secret is automatically wired to both the Lambda env var and the GitHub webhook config, and it rotates cleanly if you ever need to replace it.

Why direct Firehose to Snowflake?

Data like this is normally written to and loaded from Amazon S3. But with an S3 intermediate path, you must wait for Firehose to buffer records (60 seconds), then for Snowpipe to detect the new file and load it. Total latency: about two minutes. With the direct Snowflake destination, Firehose uses the Snowpipe Streaming API to insert records as soon as they arrive, in seconds.

The direct path also removes the need for S3 event notifications, SQS queues, external stages, and pipe resources. S3 is still used, but only as a backup destination for failed records.

The DirectSnowflakeIngestion component

The component creates everything needed for the direct path: a TLS key pair for Snowflake authentication, a Snowflake service user with least-privilege grants, the landing table, a Firehose delivery stream with destination="snowflake", and a Lambda function with a public URL. Create an empty components/__init__.py so that Python treats the directory as a package.

The full component lives in components/direct_snowflake_ingestion.py:

import json
from dataclasses import dataclass

import pulumi
import pulumi_aws as aws
import pulumi_snowflake as snowflake
import pulumi_tls as tls

# In the example repository, you will find this class imported from a common library file instead
@dataclass
class ColumnDef:
 """Column definition for a Snowflake table."""

 name: str
 type: str
 nullable: bool = True

def strip_pem_headers(pem: str) -> str:
 """Remove PEM header/footer lines, returning only the base64 content."""
 lines = pem.strip().split("\n")
 return "".join(lines[1:-1])


@dataclass
class DirectSnowflakeIngestionArgs:
 bucket_arn: pulumi.Input[str]
 bucket_name: pulumi.Input[str]
 database: pulumi.Input[str]
 schema_name: pulumi.Input[str]
 table_name: str
 table_columns: list[ColumnDef]
 snowflake_account_url: pulumi.Input[str]
 snowflake_role_name: str
 lambda_code: pulumi.Archive
 lambda_handler: str
 lambda_environment: dict[str, pulumi.Input[str]]
 table_comment: str = ""
 s3_prefix: str = "direct-webhooks"
 s3_backup_mode: str = "FailedDataOnly"
 buffering_interval: int = 0
 buffering_size: int = 1
 retry_duration: int = 60
 data_loading_option: str = "VARIANT_CONTENT_AND_METADATA_MAPPING"
 content_column_name: str = "CONTENT"
 metadata_column_name: str = "METADATA"

class DirectSnowflakeIngestion(pulumi.ComponentResource):
 function_url: pulumi.Output[str]
 firehose_stream_name: pulumi.Output[str]
 snowflake_user_name: pulumi.Output[str]

 def __init__(
 self,
 name: str,
 args: DirectSnowflakeIngestionArgs,
 opts: pulumi.ResourceOptions | None = None,
 ):
 super().__init__(
 "snowpipe:direct:DirectSnowflakeIngestion", name, {}, opts
 )

 # --- TLS key pair for Snowflake auth ---
 key_pair = tls.PrivateKey(
 f"{name}-keypair",
 algorithm="RSA",
 rsa_bits=2048,
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Snowflake role, user, and grants ---
 sf_role = snowflake.AccountRole(
 f"{name}-sf-role",
 name=args.snowflake_role_name,
 opts=pulumi.ResourceOptions(parent=self),
 )

 user_name = f"FIREHOSE_{name.upper().replace('-', '_')}_USER"
 sf_user = snowflake.ServiceUser(
 f"{name}-sf-user",
 name=user_name,
 login_name=user_name,
 default_role=sf_role.name,
 rsa_public_key=key_pair.public_key_pem.apply(strip_pem_headers),
 opts=pulumi.ResourceOptions(parent=self),
 )

 # Landing table
 table = snowflake.Table(
 f"{name}-table",
 name=args.table_name,
 database=args.database,
 schema=args.schema_name,
 comment=args.table_comment,
 columns=[
 snowflake.TableColumnArgs(
 name=col.name, type=col.type, nullable=col.nullable,
 )
 for col in args.table_columns
 ],
 opts=pulumi.ResourceOptions(parent=self),
 )

 # Grants: DB USAGE, schema USAGE, table INSERT+SELECT
 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-db-usage",
 account_role_name=sf_role.name,
 privileges=["USAGE"],
 on_account_object=snowflake.GrantPrivilegesToAccountRoleOnAccountObjectArgs(
 object_type="DATABASE",
 object_name=args.database,
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-schema-usage",
 account_role_name=sf_role.name,
 privileges=["USAGE"],
 on_schema=snowflake.GrantPrivilegesToAccountRoleOnSchemaArgs(
 schema_name=pulumi.Output.all(
 args.database, args.schema_name
 ).apply(lambda parts: f'"{parts[0]}"."{parts[1]}"'),
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 table_name = args.table_name
 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-table",
 account_role_name=sf_role.name,
 privileges=["INSERT", "SELECT"],
 on_schema_object=snowflake.GrantPrivilegesToAccountRoleOnSchemaObjectArgs(
 object_type="TABLE",
 object_name=pulumi.Output.all(
 args.database, args.schema_name
 ).apply(
 lambda parts: f'"{parts[0]}"."{parts[1]}"."{table_name}"'
 ),
 ),
 opts=pulumi.ResourceOptions(parent=self, depends_on=[table]),
 )

 # --- Firehose IAM role (S3 backup write) ---
 firehose_role = aws.iam.Role(
 f"{name}-firehose-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": "sts:AssumeRole",
 "Principal": {"Service": "firehose.amazonaws.com"},
 }],
 }),
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicy(
 f"{name}-firehose-s3-policy",
 role=firehose_role.id,
 policy=args.bucket_arn.apply(
 lambda arn: json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": [
 "s3:AbortMultipartUpload",
 "s3:GetBucketLocation",
 "s3:GetObject",
 "s3:ListBucket",
 "s3:ListBucketMultipartUploads",
 "s3:PutObject",
 ],
 "Resource": [arn, f"{arn}/*"],
 }],
 })
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Firehose delivery stream (Snowflake destination) ---
 stream = aws.kinesis.FirehoseDeliveryStream(
 f"{name}-firehose",
 destination="snowflake",
 snowflake_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationArgs(
 account_url=args.snowflake_account_url,
 database=args.database,
 schema=args.schema_name,
 table=args.table_name,
 role_arn=firehose_role.arn,
 user=sf_user.name,
 private_key=key_pair.private_key_pem_pkcs8.apply(
 strip_pem_headers
 ),
 data_loading_option=args.data_loading_option,
 content_column_name=args.content_column_name,
 metadata_column_name=args.metadata_column_name,
 s3_backup_mode=args.s3_backup_mode,
 buffering_size=args.buffering_size,
 buffering_interval=args.buffering_interval,
 retry_duration=args.retry_duration,
 snowflake_role_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationSnowflakeRoleConfigurationArgs(
 enabled=True,
 snowflake_role=args.snowflake_role_name,
 ),
 s3_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationS3ConfigurationArgs(
 bucket_arn=args.bucket_arn,
 role_arn=firehose_role.arn,
 prefix=f"{args.s3_prefix}/backup/",
 error_output_prefix=f"{args.s3_prefix}/errors/",
 ),
 ),
 opts=pulumi.ResourceOptions(parent=self, depends_on=[table]),
 )

 # --- Lambda function + Function URL ---
 lambda_role = aws.iam.Role(
 f"{name}-lambda-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": "sts:AssumeRole",
 "Principal": {"Service": "lambda.amazonaws.com"},
 }],
 }),
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicyAttachment(
 f"{name}-lambda-basic-execution",
 role=lambda_role.name,
 policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicy(
 f"{name}-lambda-firehose-policy",
 role=lambda_role.id,
 policy=stream.arn.apply(
 lambda arn: json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": ["firehose:PutRecord"],
 "Resource": [arn],
 }],
 })
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 env_vars = {
 **args.lambda_environment,
 "FIREHOSE_STREAM_NAME": stream.name,
 }

 fn = aws.lambda_.Function(
 f"{name}-handler",
 runtime="python3.11",
 handler=args.lambda_handler,
 role=lambda_role.arn,
 timeout=30,
 code=args.lambda_code,
 environment=aws.lambda_.FunctionEnvironmentArgs(
 variables=env_vars,
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 fn_url = aws.lambda_.FunctionUrl(
 f"{name}-function-url",
 function_name=fn.name,
 authorization_type="NONE",
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.lambda_.Permission(
 f"{name}-function-url-permission",
 action="lambda:InvokeFunctionUrl",
 function=fn.name,
 principal="*",
 function_url_auth_type="NONE",
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Outputs ---
 self.function_url = fn_url.function_url
 self.firehose_stream_name = stream.name
 self.snowflake_user_name = sf_user.name

 self.register_outputs({
 "function_url": self.function_url,
 "firehose_stream_name": self.firehose_stream_name,
 "snowflake_user_name": self.snowflake_user_name,
 })

A few things to note:

TLS key pair for authentication. The component generates an RSA key pair using the pulumi-tls provider. The public key is assigned to the Snowflake service user; the private key (PKCS#8 format, base64-encoded) is passed to Firehose. No passwords or OAuth tokens are stored.
ServiceUser instead of User. Snowflake service users can’t log in interactively. They authenticate only via key pair, which is exactly what Firehose needs.
destination="snowflake" on Firehose. This tells Firehose to use the Snowpipe Streaming API rather than writing to S3. The s3_configuration block is still required, but only for backup/error records.
Immediate flushing. buffering_interval=0 and buffering_size=1 ensure records are sent to Snowflake as soon as they arrive, minimizing latency. Tune according to your needs.

Amazon Data Firehose does not connect from fixed IP addresses, so you cannot use Snowflake network policies to restrict access by IP. If your Snowflake account uses network policies, you have three options: use AWS PrivateLink (requires Snowflake Business Critical edition), allow public internet access for the Firehose service user, or switch to S3 auto-ingest via Snowpipe which does not require direct network access to Snowflake from Firehose.

Wiring it together

With the component defined, __main__.py wires the direct ingestion pipeline:

import pulumi
import pulumi_aws as aws
import pulumi_github as github
import pulumi_random as random
import pulumi_snowflake as snowflake
from components.direct_snowflake_ingestion import (
 DirectSnowflakeIngestion, DirectSnowflakeIngestionArgs,
)
from components.snowpipe_pipeline import ColumnDef

config = pulumi.Config()
database_name = config.get("database") or "LANDING_ZONE_WEBHOOKS"
environment = config.get("environment") or "dev"

# --- Shared infrastructure ---

# S3 bucket for backup/errors
bucket = aws.s3.Bucket(
 "data-landing-bucket",
)

# Snowflake database and schema
database = snowflake.Database("demo-database", name=database_name)
schema = snowflake.Schema(
 "demo-schema",
 name="GITHUB",
 database=database.name,
)

# --- Direct ingestion pipeline ---

webhook_repo = config.require("webhook-repo")

# Snowflake account URL for Firehose configuration
snowflake_config = pulumi.Config("snowflake")
snowflake_account_url = (
 f"https://{snowflake_config.require('organizationName')}"
 f"-{snowflake_config.require('accountName')}"
 f".snowflakecomputing.com"
)

# Landing table columns: CONTENT (webhook JSON) + METADATA (Firehose metadata)
DIRECT_COLUMNS = [
 ColumnDef(name="CONTENT", type="VARIANT", nullable=True),
 ColumnDef(name="METADATA", type="VARIANT", nullable=True),
]

# Step 1: Generate webhook secret for HMAC validation
direct_webhook_secret = random.RandomPassword(
 "github-direct-webhook-secret", length=32, special=False
)

# Step 2: Direct ingestion pipeline - Lambda validates, Firehose streams to Snowflake
direct = DirectSnowflakeIngestion(
 "github-webhooks-direct",
 DirectSnowflakeIngestionArgs(
 bucket_arn=bucket.arn,
 bucket_name=bucket.bucket,
 database=database.name,
 schema_name=schema.name,
 table_name="REPOSITORY_EVENTS_DIRECT",
 table_columns=DIRECT_COLUMNS,
 table_comment="GitHub webhook events loaded via direct Firehose to Snowflake",
 snowflake_account_url=snowflake_account_url,
 snowflake_role_name="FIREHOSE_DIRECT_LOADER",
 lambda_code=pulumi.AssetArchive({
 "webhook_handler.py": pulumi.FileAsset("lambda/webhook_handler.py"),
 }),
 lambda_handler="webhook_handler.handler",
 lambda_environment={"WEBHOOK_SECRET": direct_webhook_secret.result},
 ),
)

# Step 3: GitHub webhook - sends events to the Lambda Function URL
github.RepositoryWebhook(
 "github-direct-webhook",
 repository=webhook_repo,
 configuration=github.RepositoryWebhookConfigurationArgs(
 url=direct.function_url,
 content_type="json",
 secret=direct_webhook_secret.result,
 ),
 events=["push", "pull_request", "issues", "star"],
)

# Exports
pulumi.export("webhook_url", direct.function_url)
pulumi.export("firehose_stream", direct.firehose_stream_name)

That’s the entire pipeline. One component, one GitHub webhook, one secret. The DirectSnowflakeIngestion component handles the TLS key pair, Snowflake service user, landing table, Firehose stream, and Lambda function internally, and now you can reuse this component for as many pipelines as you need.

The full code for this example is available on GitHub:

github.com/pulumi-demos/examples/tree/main/python/aws-snowflake-data-loading-real-time

Testing the pipeline

Deploy the stack:

pulumi up

The entire stack deploys in about two minutes. Immediately after deployment, you’ll start seeing GitHub events flowing into Snowflake.

Before querying, grant the DATA_READER role to your Snowflake user:

GRANT ROLE DATA_READER TO USER <your-user>;

In production, you can manage this grant through Pulumi, manually, or automatically via SCIM provisioning from your identity provider.

No need to craft test payloads. Just interact with the test repo. Star it, push a commit, or open an issue, then wait about 30 seconds and query Snowflake using the least-privilege reader role:

USE ROLE DATA_READER;

SELECT CONTENT:github_event::STRING AS event_type,
 CONTENT:payload:repository:full_name::STRING AS repo,
 METADATA:IngestionTime::TIMESTAMP AS ingested_at
FROM LANDING_ZONE_WEBHOOKS.GITHUB.REPOSITORY_EVENTS_DIRECT
ORDER BY ingested_at DESC;

You should see rows with event types like star, push, or issues, real GitHub events flowing through the entire pipeline. The METADATA column includes Firehose metadata like IngestionTime, which you can use to track end-to-end latency.

Other loading patterns

Direct streaming is the fastest path, but two other patterns are available in the companion template for different requirements:

S3 auto-ingest via Snowpipe. Firehose buffers to S3, and Snowpipe auto-ingests new files. Latency is about two minutes. Best when you need S3 as the system of record or can’t use direct Snowpipe Streaming.
Batch loading. Your orchestrator (Airflow, Prefect, cron, etc.) runs COPY INTO on a schedule. Best for full control over timing and deduplication.

We’ll walk through both patterns in detail in upcoming posts.

Stamp out data pipelines with components

Encapsulate your Snowflake loading logic in a Pulumi component, then create new pipelines with a handful of configuration values instead of copied files.

Get started

Publishing as reusable components

Once your components are battle-tested, you can share them across teams and projects instead of copying files around.

The most straightforward approach: push your components to a Git repository with a PulumiPlugin.yaml file at the root:

runtime: python
name: snowpipe-components
version: 1.0.0

Consumers add the package to their project with pulumi package add:

pulumi package add github.com/your-org/pulumi-snowpipe@v1.0.0

Pulumi downloads the package and generates typed SDKs automatically. The consumer’s __main__.py imports your components as if they were local, but they’re versioned and pinned.

Pulumi Cloud Private Registry

For organization-level discoverability, publish to the Pulumi Cloud Private Registry with pulumi package publish:

pulumi package publish ./schema.json

This gives you auto-generated API docs, usage tracking across teams, and cross-language SDK generation. Your Python components become usable from TypeScript, Go, and C# without any extra work. Teams browse available components in the Pulumi Cloud console, see who’s using what, and get notified when new versions are published. As an additional benefit, Neo will be able to use these components and build new pipelines in minutes from a natural language request.

Conclusion

ComponentResource is the key abstraction that makes this architecture scale. Instead of copying and pasting resources for each new data source, you instantiate a component with a handful of configuration parameters.

The DirectSnowflakeIngestion component in this post delivers data from GitHub webhooks into Snowflake in seconds: Lambda validates the HMAC signature, Firehose streams directly to Snowflake via the Snowpipe Streaming API, and the TLS key pair is managed entirely within Pulumi. No S3 intermediate, no SQS queues.

The component accepts pluggable Lambda handlers, so swapping GitHub for Stripe webhooks or any other source is just a matter of providing different lambda_code and lambda_environment arguments. We’ve been running this pattern in production for over three years across dozens of pipelines without significant changes to the infrastructure code.

You’ll find the complete example in the GitHub repository.

Introducing the Terraform State Provider for Pulumi ESC

Claire Gaestel — Fri, 13 Feb 2026 00:00:00 +0000

Many organizations have years of infrastructure built and managed with Terraform. Outputs such as VPC IDs, subnet lists, database endpoints, and cluster names are the connective tissue between infrastructure layers. Getting those values into other tools and workflows often means manual copy-paste, wrapper scripts, or brittle glue code.

The terraform-state provider for Pulumi ESC helps bridge that gap. It reads outputs directly from your Terraform state files and makes them available as first-class values in your ESC environments — no scripts, no duplication, no drift. Any output marked as sensitive in your Terraform state is automatically treated as a secret in ESC. If you’ve used pulumi-stacks to read outputs from Pulumi stacks, this is the same idea for Terraform.

How it works

The terraform-state provider uses fn::open::terraform-state to read from a Terraform state file and surface its outputs as ESC values. Here’s an example that reads from an S3 backend, using the aws-login provider for credentials, and exports a KUBECONFIG for an EKS cluster managed by Terraform:

values:
 terraform:
 fn::open::terraform-state:
 backend:
 s3:
 login:
 fn::open::aws-login:
 oidc:
 roleArn: arn:aws:iam::123456789012:role/esc-oidc
 sessionName: pulumi-environments-session
 bucket: my-terraform-state-bucket
 key: path/to/terraform.tfstate
 region: us-west-2
 files:
 KUBECONFIG: ${terraform.outputs.kubeconfig}

Once the environment is opened, terraform.outputs contains every output from the Terraform state. In this example we take the kubeconfig output from a Terraform-managed EKS cluster and project it as a file, so any tool that reads KUBECONFIG - kubectl, helm, Pulumi - just works. You can also reference outputs in pulumiConfig to pass values like VPC IDs and subnet lists directly into Pulumi stacks.¹

Terraform Cloud support

If your state lives in Terraform Cloud (or any compatible remote backend), the provider supports that too:

values:
 terraform:
 fn::open::terraform-state:
 backend:
 remote:
 organization: my-terraform-org
 workspace: my-workspace
 token:
 fn::secret: tfc-token-value
 pulumiConfig:
 vpcId: ${terraform.outputs.vpc_id}
 subnetIds: ${terraform.outputs.subnet_ids}

You can point it at any Terraform Cloud-compatible backend by setting the optional hostname property.

Get started

Check out the full terraform-state provider documentation for the complete reference.

You can also consume Terraform outputs directly in a Pulumi program with the Pulumi Terraform provider. ↩︎

Schema Validation Comes to Pulumi ESC with fn::validate

Pablo Terradillos — Thu, 12 Feb 2026 11:00:00 -0300

Pulumi ESC environments can now validate configuration values against JSON Schema with the new fn::validate built-in function. Invalid configurations are caught immediately when you save, preventing misconfigurations from reaching your deployments.

Configuration errors are often discovered too late during deployment or, worse, in production. With fn::validate, you define validation rules directly in your environment, and ESC enforces them at save time. If a value doesn’t match its schema, the environment cannot be saved until the issue is resolved.

How it works

The fn::validate function takes a JSON Schema and a value. If the value conforms to the schema, it passes through unchanged. If not, ESC raises a validation error.

values:
 port:
 fn::validate:
 schema: { type: number, minimum: 1, maximum: 65535 }
 value: 8080

This validates that port is a number between 1 and 65535. The evaluated result is simply 8080.

Validating objects with required fields

For complex configurations, you can enforce structure and required fields:

values:
 database:
 fn::validate:
 schema:
 type: object
 properties:
 host: { type: string }
 port: { type: number }
 name: { type: string }
 required: [host, port, name]
 value:
 host: "db.example.com"
 port: 5432
 name: "myapp"

If any required field is missing or has the wrong type, the environment cannot be saved.

Reusing schemas across environments

Define schemas once and reference them across multiple environments. Using the environments built-in property keeps the schema out of your environment’s output:

Schema environment (my-project/schemas)

values:
 database-schema:
 type: object
 properties:
 host: { type: string }
 port: { type: number }
 required: [host, port]

Environment using the schema

values:
 database:
 fn::validate:
 schema: ${environments.my-project.schemas.database-schema}
 value:
 host: "prod-db.example.com"
 port: 5432

This pattern ensures consistent validation rules across teams and projects.

What happens when validation fails

When a value doesn’t conform to its schema, ESC returns a clear error message:

values:
 port:
 fn::validate:
 schema: { type: string }
 value: 8080

This raises: expected string, got number. The environment cannot be saved until you fix the value or update the schema.

When to use schema validation

Enable fn::validate for:

Values with specific type requirements (numbers, strings, arrays)
Objects that must have certain fields present
Numbers that must fall within a valid range
Configurations shared across multiple environments
Any value where catching errors early prevents downstream issues

Getting started

The fn::validate function is available now in all Pulumi ESC environments. Add schema validation to your existing environments or use it when creating new ones.

For more information, see the fn::validate documentation.

Deploy OpenClaw on AWS or Hetzner Securely with Pulumi and Tailscale

Engin Diri — Mon, 26 Jan 2026 00:00:00 +0000

Update (January 2026): The lobster has molted into its final form! From Clawdbot to Moltbot to OpenClaw. With 100k+ GitHub stars and 2M visitors in a week, the project finally has a name that’ll stick. The CLI command is now openclaw and the new handle is @openclaw. Same mission: AI that actually does things. Your assistant. Your machine. Your rules. See the official getting started guide for updated installation instructions.

Update (April 2026): Refreshed for OpenClaw 2026.4.27. Upstream now recommends Node 24, but the cloud-init script in this post still installs Node 22 — both work. If you’d like Node 24, change the nvm install 22 lines to nvm install 24.

The short version: Deploy OpenClaw to AWS or Hetzner with a Pulumi TypeScript program that provisions the VM, installs Docker, Node, and OpenClaw, then joins the instance to your Tailscale network so the gateway and browser ports stay private. One pulumi up to deploy, one pulumi destroy to tear down. Total cost: about $33/month on AWS or $7/month on Hetzner.

OpenClaw is everywhere right now. The open-source AI assistant gained 9,000 GitHub stars in a single day, received public praise from former Tesla AI head Andrej Karpathy, and has sparked a global run on Mac Minis as developers scramble to give this “lobster assistant” a home. Users are calling it “Jarvis living in a hard drive” and “Claude with hands”—the personal AI assistant that Siri promised but never delivered.

The Mac Mini craze is real: people are buying dedicated hardware just to run OpenClaw, with some enthusiasts purchasing 40 Mac Minis at once. Even Logan Kilpatrick from Google DeepMind couldn’t resist ordering one. But here’s the thing: you don’t actually need a Mac Mini. OpenClaw runs anywhere: on a VPS, in the cloud, or on that old laptop gathering dust.

With all this hype, I had to try it myself. But instead of clicking through the AWS console or running manual commands on a VPS, I wanted to do it right from the start: infrastructure as code with Pulumi. Why? Because when I inevitably want to tear it down, spin up a new instance, or deploy to a different region, I don’t want to remember which buttons I clicked three weeks ago. I want a single pulumi up command.

Dan got the assignment right:

In this post, I’ll show you how to deploy OpenClaw to AWS or Hetzner Cloud (if you want European data residency or just want to spend less). We’ll use Pulumi to define the infrastructure and Tailscale to keep your AI assistant off the public internet.

What is OpenClaw?

OpenClaw is an open-source AI assistant created by Peter Steinberger that runs on your own infrastructure. It connects to WhatsApp, Slack, Discord, Google Chat, Signal, and iMessage. It can control browsers, generate videos and images, clone your voice for voice notes, and run scheduled tasks via cron. There’s a skills system for extending functionality, and you can run it on pretty much anything: Mac Mini, Raspberry Pi, VPS, laptop, or gaming PC.

The difference from cloud-hosted AI? OpenClaw runs on your server, not Anthropic’s. It’s available 24/7 across all your devices, can schedule automated tasks, and keeps your entire conversation history locally. Check the official OpenClaw documentation for the full feature list.

Prerequisites

Before getting started, ensure you have:

Pulumi CLI installed and configured
A Pulumi Cloud account
AWS account (for AWS deployment)
Hetzner Cloud account (for European deployment)
Anthropic API key
Node.js 22+ installed (Node 24 recommended; the cloud-init script in this post installs Node 22 inside the VM)
Tailscale account with HTTPS enabled (one-time setup in admin console)

This guide uses Anthropic’s API, but OpenClaw works with other providers too. Check the providers documentation if you’d rather use OpenAI, Google Gemini, or a local model via Ollama.

Understanding OpenClaw architecture

OpenClaw uses a gateway-centric architecture where a single daemon acts as the control plane for all messaging, tool execution, and client connections:

Component	Port	Description
Gateway	18789	WebSocket server handling channels, nodes, sessions, and hooks
Browser control	18791	Headless Chrome instance for web automation
Docker sandbox	-	Isolated container environment for running tools safely

The Gateway connects to messaging platforms (WhatsApp, Slack, Discord, etc.), the CLI, the web UI, and mobile apps. The Browser component lets OpenClaw open web pages, fill forms, scrape data, and download files. Docker sandboxing runs bash commands in isolated containers so your bot can execute code without risking your host system.

Setting up ESC for secrets management

Deploying OpenClaw means handling sensitive credentials: API keys, auth tokens, cloud provider secrets. You don’t want these hardcoded or scattered across environment variables. Pulumi ESC (Environments, Secrets, and Configuration) stores them securely and passes them directly to your Pulumi program — the same pattern Pulumi uses to eliminate long-lived CI secrets across 70+ repos.

Create a new ESC environment:

pulumi env init <your-org>/openclaw-secrets

Add your secrets to the environment:

values:
 anthropicApiKey:
 fn::secret: "sk-ant-xxxxx"
 tailscaleAuthKey:
 fn::secret: "tskey-auth-xxxxx"
 tailnetDnsName: "tailxxxxx.ts.net"
 hcloudToken:
 fn::secret: "your-hetzner-api-token"
 pulumiConfig:
 anthropicApiKey: ${anthropicApiKey}
 tailscaleAuthKey: ${tailscaleAuthKey}
 tailnetDnsName: ${tailnetDnsName}
 hcloud:token: ${hcloudToken}

To find your tailnet DNS name, go to the Tailscale admin console, look under the DNS section, and find your tailnet name (e.g., tailxxxxx.ts.net). This is the domain suffix used for all machines in your Tailscale network.

Then create a Pulumi.dev.yaml file in your project to reference the environment:

environment:
 - <your-org>/openclaw-secrets

This approach keeps your secrets out of your codebase and passes them directly to OpenClaw during automated onboarding. To prevent stack-level overrides of these values, see locking down ESC values with fn::final.

Securing with Tailscale

By default, deploying OpenClaw exposes SSH (port 22), the gateway (port 18789), and browser control (port 18791) to the public internet. This is convenient for testing but not ideal for production use.

Tailscale creates a secure mesh VPN that lets you access your OpenClaw instance without exposing unnecessary ports publicly. When you provide a Tailscale auth key, the Pulumi program:

Removes gateway and browser ports from public access
Keeps SSH as fallback for debugging if Tailscale setup fails
Installs Tailscale on the instance during provisioning (after other dependencies)
Enables Tailscale SSH so you can SSH via Tailscale without managing keys
Joins your tailnet automatically using the auth key

The Pulumi program installs Docker, Node.js, and OpenClaw first, then configures Tailscale last. This ensures that even if the Tailscale auth key is invalid or expired, you can still SSH in via the public IP to troubleshoot.

To generate a Tailscale auth key:

Go to Tailscale Admin Console
Click “Generate auth key”
Enable “Reusable” if you plan to redeploy
Copy the key and add it to your ESC environment

Deploying to AWS

Let’s walk through the complete AWS deployment. Create a new Pulumi project:

mkdir openclaw-aws && cd openclaw-aws
pulumi new typescript

Install the required dependencies:

npm install @pulumi/aws @pulumi/tls

Do not use t3.micro instances for OpenClaw. The 1 GB memory is insufficient for installation. Use t3.medium (4 GB) or t3.large (8 GB) instead.

The Pulumi program

Running OpenClaw on AWS means setting up a VPC, subnets, security groups, an EC2 instance, SSH keys, and a cloud-init script that installs everything. That’s a lot of clicking in the AWS console. The Pulumi program below defines all of it in code.

Replace the contents of index.ts with the following:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as tls from "@pulumi/tls";

const config = new pulumi.Config();

const instanceType = config.get("instanceType") ?? "t3.medium";
const anthropicApiKey = config.requireSecret("anthropicApiKey");
const model = config.get("model") ?? "anthropic/claude-sonnet-4";
const enableSandbox = config.getBoolean("enableSandbox") ?? true;
const gatewayPort = config.getNumber("gatewayPort") ?? 18789;
const browserPort = config.getNumber("browserPort") ?? 18791;

const tailscaleAuthKey = config.requireSecret("tailscaleAuthKey");
const tailnetDnsName = config.require("tailnetDnsName");

// Generate a random token for gateway authentication
const gatewayToken = new tls.PrivateKey("openclaw-gateway-token", {
 algorithm: "ED25519",
}).publicKeyOpenssh.apply(key => {
 // Create a deterministic token from the public key (take first 48 hex chars)
 const hash = require("crypto").createHash("sha256").update(key).digest("hex");
 return hash.substring(0, 48);
});

const sshKey = new tls.PrivateKey("openclaw-ssh-key", {
 algorithm: "ED25519",
});

const vpc = new aws.ec2.Vpc("openclaw-vpc", {
 cidrBlock: "10.0.0.0/16",
 enableDnsHostnames: true,
 enableDnsSupport: true,
 tags: { Name: "openclaw-vpc" },
});

const gateway = new aws.ec2.InternetGateway("openclaw-igw", {
 vpcId: vpc.id,
 tags: { Name: "openclaw-igw" },
});

const subnet = new aws.ec2.Subnet("openclaw-subnet", {
 vpcId: vpc.id,
 cidrBlock: "10.0.1.0/24",
 mapPublicIpOnLaunch: true,
 tags: { Name: "openclaw-subnet" },
});

const routeTable = new aws.ec2.RouteTable("openclaw-rt", {
 vpcId: vpc.id,
 routes: [
 {
 cidrBlock: "0.0.0.0/0",
 gatewayId: gateway.id,
 },
 ],
 tags: { Name: "openclaw-rt" },
});

new aws.ec2.RouteTableAssociation("openclaw-rta", {
 subnetId: subnet.id,
 routeTableId: routeTable.id,
});

const securityGroup = new aws.ec2.SecurityGroup("openclaw-sg", {
 vpcId: vpc.id,
 description: "Security group for OpenClaw instance",
 ingress: [
 {
 description: "SSH access (fallback)",
 fromPort: 22,
 toPort: 22,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
 egress: [
 {
 fromPort: 0,
 toPort: 0,
 protocol: "-1",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
 tags: { Name: "openclaw-sg" },
});

const keyPair = new aws.ec2.KeyPair("openclaw-keypair", {
 publicKey: sshKey.publicKeyOpenssh,
});

const ami = aws.ec2.getAmiOutput({
 owners: ["099720109477"],
 mostRecent: true,
 filters: [
 { name: "name", values: ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"] },
 { name: "virtualization-type", values: ["hvm"] },
 ],
});

const userData = pulumi
 .all([tailscaleAuthKey, anthropicApiKey, gatewayToken])
 .apply(([tsAuthKey, apiKey, gwToken]) => {
 return `#!/bin/bash
set -e

export DEBIAN_FRONTEND=noninteractive

# System updates
apt-get update
apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker
usermod -aG docker ubuntu

# Install NVM and Node.js for ubuntu user
sudo -u ubuntu bash << 'UBUNTU_SCRIPT'
set -e
cd ~

# Install NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

# Load NVM
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

# Install Node.js 22
nvm install 22
nvm use 22
nvm alias default 22

# Install OpenClaw
npm install -g openclaw@latest

# Add NVM to bashrc if not already there
if ! grep -q 'NVM_DIR' ~/.bashrc; then
 echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc
 echo '[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"' >> ~/.bashrc
fi
UBUNTU_SCRIPT

# Set environment variables for ubuntu user
echo 'export ANTHROPIC_API_KEY="${apiKey}"' >> /home/ubuntu/.bashrc

# Install and configure Tailscale
echo "Installing Tailscale..."
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --authkey="${tsAuthKey}" --ssh || echo "WARNING: Tailscale setup failed. Run 'sudo tailscale up' manually."

# Enable systemd linger for ubuntu user (required for user services to run at boot)
loginctl enable-linger ubuntu

# Start user's systemd instance (required for user services during cloud-init)
systemctl start user@1000.service

# Run OpenClaw onboarding as ubuntu user (skip daemon install, do it separately)
echo "Running OpenClaw onboarding..."
sudo -H -u ubuntu ANTHROPIC_API_KEY="${apiKey}" GATEWAY_PORT="${gatewayPort}" bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw onboard --non-interactive --accept-risk \
 --mode local \
 --auth-choice apiKey \
 --gateway-port $GATEWAY_PORT \
 --gateway-bind loopback \
 --skip-daemon \
 --skip-skills || echo "WARNING: OpenClaw onboarding failed. Run openclaw onboard manually."
'

# Install daemon service with XDG_RUNTIME_DIR set
echo "Installing OpenClaw daemon..."
sudo -H -u ubuntu XDG_RUNTIME_DIR=/run/user/1000 bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw daemon install || echo "WARNING: Daemon install failed. Run openclaw daemon install manually."
'

# Configure gateway for Tailscale Serve (trustedProxies + skip device pairing + set token)
echo "Configuring gateway for Tailscale Serve..."
sudo -H -u ubuntu GATEWAY_TOKEN="${gwToken}" python3 << 'PYTHON_SCRIPT'
import json
import os
config_path = "/home/ubuntu/.openclaw/openclaw.json"
with open(config_path) as f:
 config = json.load(f)
config["gateway"]["trustedProxies"] = ["127.0.0.1"]
config["gateway"]["controlUi"] = {
 "enabled": True,
 "allowInsecureAuth": True
}
config["gateway"]["auth"] = {
 "mode": "token",
 "token": os.environ["GATEWAY_TOKEN"]
}
with open(config_path, "w") as f:
 json.dump(config, f, indent=2)
print("Configured gateway with trustedProxies, controlUi, and token")
PYTHON_SCRIPT

# Enable Tailscale HTTPS proxy (requires HTTPS to be enabled in Tailscale admin console)
echo "Enabling Tailscale HTTPS proxy..."
tailscale serve --bg ${gatewayPort} || echo "WARNING: tailscale serve failed. Enable HTTPS in your Tailscale admin console first."

echo "OpenClaw setup complete!"
`;
 });

const instance = new aws.ec2.Instance("openclaw-instance", {
 ami: ami.id,
 instanceType: instanceType,
 subnetId: subnet.id,
 vpcSecurityGroupIds: [securityGroup.id],
 keyName: keyPair.keyName,
 userData: userData,
 userDataReplaceOnChange: true,
 rootBlockDevice: {
 volumeSize: 30,
 volumeType: "gp3",
 },
 tags: { Name: "openclaw" },
});

export const publicIp = instance.publicIp;
export const publicDns = instance.publicDns;
export const privateKey = sshKey.privateKeyOpenssh;

// Construct the Tailscale MagicDNS hostname from the private IP
// AWS private IPs like 10.0.1.15 become hostnames like ip-10-0-1-15
const tailscaleHostname = instance.privateIp.apply(ip =>
 `ip-${ip.replace(/\./g, "-")}`
);

export const tailscaleUrlWithToken = pulumi.interpolate`https://${tailscaleHostname}.${tailnetDnsName}/?token=${gatewayToken}`;
export const gatewayTokenOutput = gatewayToken;

Deploying to Hetzner

Hetzner Cloud is a solid choice if you need European data residency or want to spend less money. Spoiler: it’s a lot less money.

Hetzner has similar concepts to AWS but different names. EC2 instances become Servers. Security groups become Firewalls. Same idea, different provider. The resource types come from @pulumi/hcloud.

Create a new project for Hetzner:

mkdir openclaw-hetzner && cd openclaw-hetzner
pulumi new typescript

Install the Hetzner provider:

npm install @pulumi/hcloud @pulumi/tls

The default server type cax21 is an ARM-based (Ampere) instance with 4 vCPUs and 8 GB RAM. ARM instances cost less for the same compute. If you need x86 architecture, use ccx13 or similar CCX series instead.

The Hetzner Pulumi program

Replace index.ts with the following:

import * as pulumi from "@pulumi/pulumi";
import * as hcloud from "@pulumi/hcloud";
import * as tls from "@pulumi/tls";

const config = new pulumi.Config();

const serverType = config.get("serverType") ?? "cax21";
const location = config.get("location") ?? "fsn1";
const anthropicApiKey = config.requireSecret("anthropicApiKey");
const model = config.get("model") ?? "anthropic/claude-sonnet-4";
const enableSandbox = config.getBoolean("enableSandbox") ?? true;
const gatewayPort = config.getNumber("gatewayPort") ?? 18789;
const browserPort = config.getNumber("browserPort") ?? 18791;

const tailscaleAuthKey = config.requireSecret("tailscaleAuthKey");
const tailnetDnsName = config.require("tailnetDnsName");

// Generate a random token for gateway authentication
const gatewayToken = new tls.PrivateKey("openclaw-gateway-token", {
 algorithm: "ED25519",
}).publicKeyOpenssh.apply(key => {
 const hash = require("crypto").createHash("sha256").update(key).digest("hex");
 return hash.substring(0, 48);
});

const sshKey = new tls.PrivateKey("openclaw-ssh-key", {
 algorithm: "ED25519",
});

const hcloudSshKey = new hcloud.SshKey("openclaw-sshkey", {
 publicKey: sshKey.publicKeyOpenssh,
});

const firewallRules: hcloud.types.input.FirewallRule[] = [
 {
 direction: "out",
 protocol: "tcp",
 port: "any",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound TCP",
 },
 {
 direction: "out",
 protocol: "udp",
 port: "any",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound UDP",
 },
 {
 direction: "out",
 protocol: "icmp",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound ICMP",
 },
 {
 direction: "in",
 protocol: "tcp",
 port: "22",
 sourceIps: ["0.0.0.0/0", "::/0"],
 description: "SSH access (fallback)",
 },
];

const firewall = new hcloud.Firewall("openclaw-firewall", {
 rules: firewallRules,
});

const userData = pulumi
 .all([tailscaleAuthKey, anthropicApiKey, gatewayToken])
 .apply(([tsAuthKey, apiKey, gwToken]) => {
 return `#!/bin/bash
set -e

export DEBIAN_FRONTEND=noninteractive

# System updates
apt-get update
apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker

# Create ubuntu user (Hetzner uses root by default)
useradd -m -s /bin/bash -G docker ubuntu || true

# Install NVM and Node.js for ubuntu user
sudo -u ubuntu bash << 'UBUNTU_SCRIPT'
set -e
cd ~

# Install NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

# Load NVM
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

# Install Node.js 22
nvm install 22
nvm use 22
nvm alias default 22

# Install OpenClaw
npm install -g openclaw@latest

# Add NVM to bashrc if not already there
if ! grep -q 'NVM_DIR' ~/.bashrc; then
 echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc
 echo '[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"' >> ~/.bashrc
fi
UBUNTU_SCRIPT

# Set environment variables for ubuntu user
echo 'export ANTHROPIC_API_KEY="${apiKey}"' >> /home/ubuntu/.bashrc

# Install and configure Tailscale
echo "Installing Tailscale..."
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --authkey="${tsAuthKey}" --ssh || echo "WARNING: Tailscale setup failed. Run 'sudo tailscale up' manually."

# Enable systemd linger for ubuntu user (required for user services to run at boot)
loginctl enable-linger ubuntu

# Start user's systemd instance (required for user services during cloud-init)
systemctl start user@1000.service

# Run OpenClaw onboarding as ubuntu user (skip daemon install, do it separately)
echo "Running OpenClaw onboarding..."
sudo -H -u ubuntu ANTHROPIC_API_KEY="${apiKey}" GATEWAY_PORT="${gatewayPort}" bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw onboard --non-interactive --accept-risk \
 --mode local \
 --auth-choice apiKey \
 --gateway-port $GATEWAY_PORT \
 --gateway-bind loopback \
 --skip-daemon \
 --skip-skills || echo "WARNING: OpenClaw onboarding failed. Run openclaw onboard manually."
'

# Install daemon service with XDG_RUNTIME_DIR set
echo "Installing OpenClaw daemon..."
sudo -H -u ubuntu XDG_RUNTIME_DIR=/run/user/1000 bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw daemon install || echo "WARNING: Daemon install failed. Run openclaw daemon install manually."
'

# Configure gateway for Tailscale Serve (trustedProxies + skip device pairing + set token)
echo "Configuring gateway for Tailscale Serve..."
sudo -H -u ubuntu GATEWAY_TOKEN="${gwToken}" python3 << 'PYTHON_SCRIPT'
import json
import os
config_path = "/home/ubuntu/.openclaw/openclaw.json"
with open(config_path) as f:
 config = json.load(f)
config["gateway"]["trustedProxies"] = ["127.0.0.1"]
config["gateway"]["controlUi"] = {
 "enabled": True,
 "allowInsecureAuth": True
}
config["gateway"]["auth"] = {
 "mode": "token",
 "token": os.environ["GATEWAY_TOKEN"]
}
with open(config_path, "w") as f:
 json.dump(config, f, indent=2)
print("Configured gateway with trustedProxies, controlUi, and token")
PYTHON_SCRIPT

# Enable Tailscale HTTPS proxy (requires HTTPS to be enabled in Tailscale admin console)
echo "Enabling Tailscale HTTPS proxy..."
tailscale serve --bg ${gatewayPort} || echo "WARNING: tailscale serve failed. Enable HTTPS in your Tailscale admin console first."

echo "OpenClaw setup complete!"
`;
 });

const server = new hcloud.Server("openclaw-server", {
 serverType: serverType,
 location: location,
 image: "ubuntu-24.04",
 sshKeys: [hcloudSshKey.id],
 firewallIds: [firewall.id.apply(id => Number(id))],
 userData: userData,
 labels: {
 purpose: "openclaw",
 },
});

export const ipv4Address = server.ipv4Address;
export const privateKey = sshKey.privateKeyOpenssh;

// Construct the Tailscale MagicDNS hostname from the server name
// Hetzner servers use their name as the hostname
const tailscaleHostname = server.name;

export const tailscaleUrlWithToken = pulumi.interpolate`https://${tailscaleHostname}.${tailnetDnsName}/?token=${gatewayToken}`;
export const gatewayTokenOutput = gatewayToken;

You can find both programs in the Pulumi examples repo under openclaw/:

github.com/pulumi/examples

Cost comparison

Before deploying, let’s compare the costs between AWS and Hetzner for running OpenClaw 24/7:

	AWS (t3.medium)	Hetzner (cax21)
vCPUs	2	4
Memory	4 GB	8 GB
Storage	30 GB gp3 (+$2.40/mo)	80 GB NVMe (included)
Traffic	Pay per GB	20 TB included
Architecture	x86 (Intel/AMD)	ARM (Ampere)
Hourly price	$0.0416	€0.0104 (~$0.011)
Monthly price	~$33 (with storage)	€6.49 (~$7)
Annual cost	~$396	~$84

Hetzner gives you double the vCPUs, double the RAM, at less than a quarter of the price. The trade-off? ARM architecture instead of x86. But OpenClaw doesn’t care - it’s just Node.js and Docker.

Prices are for on-demand instances as of January 2026. AWS prices are for us-east-1; Hetzner prices exclude VAT. Both include standard networking and storage. Check AWS EC2 pricing and Hetzner Cloud pricing for current rates.

Running the deployment

With your ESC environment configured in Pulumi.dev.yaml, deploy with:

pulumi up

After deployment completes, you’ll see outputs similar to:

Outputs:
 gatewayTokenOutput : "786c099cc8f8bf20dbebf40b8b51b75cf5cdab25..."
 privateKey : [secret]
 publicDns : "ec2-x-x-x-x.compute-1.amazonaws.com"
 publicIp : "x.x.x.x"
 tailscaleUrlWithToken: "https://ip-10-0-1-x.tailxxxxx.ts.net/?token=786c099..."

The tailscaleUrlWithToken output provides the complete URL with authentication token. Copy and paste it into your browser to access the OpenClaw web UI.

Output names vary slightly between providers: AWS uses publicIp and publicDns, while Hetzner uses ipv4Address. The Tailscale hostname is derived from the instance’s private IP (AWS) or server name (Hetzner).

Automated onboarding

The Pulumi program runs OpenClaw’s non-interactive onboarding during instance provisioning. It uses your Anthropic API key from ESC, binds the gateway to loopback with Tailscale Serve as the HTTPS proxy, generates a secure gateway token (exported in Pulumi outputs), installs the daemon as a systemd user service, and configures trustedProxies and controlUi.allowInsecureAuth to skip device pairing when accessed via Tailscale.

The cloud-init script runs openclaw onboard --non-interactive with all necessary flags, then configures the gateway for secure Tailscale access. Your instance is ready as soon as provisioning finishes.

Access the web UI

Please wait 3-5 minutes after pulumi up completes. The cloud-init script runs after the instance launches and needs time to install Docker, Node.js, OpenClaw, and Tailscale, then run the onboarding process and start the daemon. Periodically refresh the page until it loads. If the page still doesn’t load after 5 minutes, see Verify the deployment to troubleshoot.

The easiest way to access the OpenClaw web UI is to use the tailscaleUrlWithToken output from Pulumi:

# Get the full URL with token
pulumi stack output tailscaleUrlWithToken

Copy and paste this URL into your browser. The URL includes both the Tailscale MagicDNS hostname and the authentication token, so you can access the web UI directly.

Finding your tailnet DNS name: Go to the Tailscale admin console and look under the DNS section for your tailnet name (e.g., tailxxxxx.ts.net). You can also find your machines and their MagicDNS hostnames in the Machines tab.

Token-based authentication provides an additional layer of security on top of Tailscale’s network-level authentication. Only devices on your tailnet can reach the URL, and the token prevents unauthorized access if someone gains access to your tailnet.

From the web UI, you can connect messaging channels (WhatsApp, Discord, Slack), configure skills and integrations, and manage settings.

Verify the deployment (optional)

If you encounter issues accessing the web UI, you can SSH into your instance to troubleshoot:

# Check your Tailscale admin console for the new machine
ssh ubuntu@<tailscale-ip>

# Check OpenClaw gateway status
systemctl --user status openclaw-gateway

Test your assistant

Open the gateway dashboard using the tailscaleUrlWithToken output and use the built-in chat to test your assistant:

Your personal AI assistant is now running 24/7 on your own infrastructure, accessible securely through Tailscale.

Redeploy OpenClaw with one command

With your infrastructure defined in Pulumi, you can move OpenClaw to a new region, rebuild it, or tear it down with pulumi destroy, without remembering which console buttons you clicked.

Get started

Security considerations

When self-hosting an AI assistant, security matters. OpenClaw’s rapid adoption meant thousands of instances spun up in days, and not everyone locked them down. The community noticed:

The tweet isn’t exaggerating. A quick Shodan search shows exposed gateways on port 18789 with shell access, browser automation, and API keys up for grabs:

Don’t let your instance be one of them.

Concern	Without Tailscale	With Tailscale
SSH access	Public (port 22 open)	Public fallback + Tailscale SSH
Gateway access	Public (port 18789 open)	Private (Tailscale only)
Browser control	Public (port 18791 open)	Private (Tailscale only)
API keys in transit	Exposed if gateway accessed over HTTP	Protected by Tailscale encryption
Attack surface	3 open ports	1 open port (SSH fallback)

SSH remains accessible as a fallback even with Tailscale enabled. This allows you to troubleshoot if Tailscale fails to connect. Once you’ve confirmed Tailscale is working, you can manually remove the SSH ingress rule from your security group for maximum security.

My recommendations:

Always use Tailscale for production
Rotate your auth keys periodically
Use Pulumi ESC for secrets instead of hardcoding
Enable Tailscale SSH to avoid managing keys manually
Monitor your Tailscale admin console for unauthorized devices
Remove the SSH fallback after confirming Tailscale works if you want zero public ports

What’s next?

Now that OpenClaw is running, you can install skills (voice generation, video creation, browser automation), set up scheduled tasks with cron, invite colleagues to your tailnet for shared access, or connect additional channels like WhatsApp and Discord.

For a deeper case study of running an AI assistant on Pulumi-managed infrastructure, see how we built Platybot, or read our KubeCon EU 2026 recap on the year AI moved into production on Kubernetes.

Conclusion

Deploying OpenClaw with infrastructure as code means you can reproduce your setup anytime, version control it, and tear it down with a single command. Adding Tailscale keeps it private - no exposed ports, no hoping you configured your firewall correctly at 2am.

If you run into issues or have questions, drop by the Pulumi Community Slack or GitHub Discussions.

New to Pulumi? Get started here.

Introducing the new and improved ESC Editor

Sean Yeh — Thu, 15 Jan 2026 08:00:00 -0500

Pulumi ESC is Pulumi Cloud’s centralized solution for managing secrets and configuration across every vault and cloud provider you use. It helps teams secure their configuration while adopting modern best practices like short-lived credentials with OIDC and automated secret rotation.

Whether you’re configuring Pulumi programs, powering applications and services, or managing credentials for tools like the AWS CLI, ESC provides a single, consistent way to do it safely and at scale.

Behind the scenes, ESC integrates with multiple cloud providers and secret managers, supports composable environments, and offers rich built-in functions, from simple value transformations to encoding files as Base64.

With this level of power, usability matters more than ever. That’s why today we’re introducing the new and improved Pulumi ESC Web Editor, designed to make managing secrets and configuration easier, faster, and more intuitive.

Today, you can create and manage your Pulumi ESC configuration in multiple ways, such as using the CLI set and edit commands, or through our VS Code extension. For many users, however, their first experience with ESC happens in the Pulumi Cloud Console.

Based on feedback from users of both our YAML Document view and Table view in the Console, we’ve been working hard to create a new and improved unified editor experience that makes ESC even easier to work with. One of the most notable improvements is a brand new Inspect tab that lets you easily edit secrets and gain deeper insights into your configuration. With this new UI, you can now freely switch between writing YAML and using rich UI elements to manipulate your environment—and the editor keeps everything in sync, with clear, in-context information about what you’re doing and what’s possible at every step.

Let’s explore some of these use cases!

Adding and editing secrets

Adding secrets is now as simple as selecting Secret from the Add new menu.

The Inspect tab lets you view and edit your secret securely, automatically encrypting it as ciphertext in your environment definition. No more worrying about accidentally exposing sensitive values!

Using providers and built-in functions

ESC offers a large library of providers and built-in functions to use in your environment. The new editor makes discovering and using them effortless.

When you add a provider or function, the editor inserts it with example values to get you started quickly. The Inspect tab provides instant access to documentation, so you can more easily configure the integrations.

Exporting configurations

Consuming your configuration where you need it is now easier than ever. The Export menu in the Inspect sidebar lets you quickly expose values as Pulumi config for your stacks, or as environment variables in your shell.

Conclusion

The new Pulumi ESC Editor brings together the best of both worlds: the power of the YAML editor with the ease of UI controls. Try it out today in the Pulumi Cloud Console and let us know what you think!

Feature Flagging for Your Infrastructure

Elisabeth Lichtie — Sat, 06 Dec 2025 00:00:00 +0000

One of Pulumi’s foundational benefits is that it allows you to manage your infrastructure as software with rich programming languages, robust testing, and CI/CD patterns that you’d use with your application code. This post will cover applying another classic software development technique to your infrastructure: feature flagging. You can use feature flags to control change rollout, reduce the risk of new releases, and speed up the development of your infrastructure, the same way you do with your applications.

The examples in this post range from simply creating a flag and using it in a Lambda function to fully integrating with LaunchDarkly to build a comprehensive flagging system for your infrastructure.

Creating flags with Pulumi

Check out the code here: 01: Flag Application With LaunchDarkly.

When you define flags in your infrastructure as code, you can configure them alongside all the supporting infrastructure your code needs. You can also encourage flag creation as a best practice in Pulumi components or templates.

With Pulumi, you can use any Terraform provider to manage your feature flags, whether that’s LaunchDarkly, Statsig, Split, Flagsmith, or any other feature flagging tool with Terraform support. This lets you define flags, targeting rules, and the infrastructure they control in a single codebase. The example uses the LaunchDarkly provider to create flags and supporting infrastructure:

const flag = new launchdarkly.FeatureFlag("example-flag", {
 key: "exampleFlag",
 projectKey: "default",
 variationType: "boolean"
}, { provider });

The Lambda function evaluates the flag at runtime using the LaunchDarkly SDK key from environment variables, allowing you to toggle behavior without redeploying code.

Using flags to control infrastructure

Check out the code here: 02: Flaggable Infrastructure.

Before deploying this code, first deploy either 03 (for ESC Only) or 04 (for ESC and LaunchDarkly) so that the environment is created and available.

Once you have flags defined, you can use them to control infrastructure provisioning. Apply the same best practices you would for application feature flags: keep flags focused, remove them when they’re no longer needed, and document their purpose. In this example, a boolean flag controls whether internet-facing resources are created:

const config = new pulumi.Config();
const flags: Record<string, any> | undefined = config.getObject("flags");
const enableInternetAccess = flags ? flags["enableInternetAccess"] : false;

if (enableInternetAccess) {
 // Create Internet Gateway, public subnet, route table, and routes
}

To enable automatic infrastructure updates when flags change (as shown in examples 3 and 4), configure deployment settings in your stack. This requires:

The Pulumi GitHub App installed on your repository
Webhook configuration to trigger deployments when configuration values change

With these in place, changing a flag value automatically triggers a deployment.

Configuring values with ESC

Check out the code here: 03: ESC Auto-updating.

Ingesting config from ESC

Pulumi ESC provides a centralized way to manage configuration and secrets. For teams that don’t need the full feature set of a dedicated flagging service, you can store flag values directly in an ESC environment. Your infrastructure stack imports this environment and reads flag values at deployment time:

const pulumiProv = new pulumiservice.Environment("flaggingEnv", {
 name: "config",
 organization: pulumi.getOrganization(),
 yaml: new pulumi.asset.StringAsset(`values:
 pulumiConfig:
 flags:
 enableInternetAccess: 'true'`),
});

This approach keeps all configuration in one place, eliminates dependencies on external services, and works seamlessly with Pulumi’s existing stack configuration system. You can update flag values through the Pulumi CLI, API, or Cloud console.

Automatically triggering deployments with webhooks

There is some risk involved with automatically triggering updates. You can leave out the webhook and deploy manually if that’s safer for your team.

Set up a Pulumi Cloud webhook to automatically deploy infrastructure when ESC environment values change. The example creates an AWS Lambda function behind API Gateway that receives webhook events and triggers deployments via the Pulumi Deployments API:

const webhook = new pulumiservice.Webhook("escEnvironmentWebhook", {
 organizationName: pulumi.getOrganization(),
 displayName: "ESC Environment Change Webhook",
 payloadUrl: api.apiEndpoint.apply((endpoint) => `${endpoint}/webhook`),
 active: true,
 filters: ["environment_revision_created"],
});

When you update a flag value in ESC, the webhook fires, the Lambda identifies which stacks use that environment, and triggers their redeployment. This creates a continuous delivery pipeline for infrastructure configuration changes.

Flag your infrastructure with Pulumi

Define feature flags alongside the resources they control, store values in Pulumi ESC or your existing flagging service, and roll out infrastructure changes with less risk.

Get started

Ingesting flag values from LaunchDarkly

Check out the code here: 04: LaunchDarkly Auto-updating.

Ingesting flags from LaunchDarkly using ESC Connect

If your team already uses LaunchDarkly for application feature flags, you can integrate those same flags with your infrastructure using ESC Connect. This is also a good option if you need enterprise-grade features like advanced targeting rules or percentage rollouts.

This approach unifies application and infrastructure flag management. The ESC environment uses fn::open::external to query a Lambda function that fetches current flag values from LaunchDarkly:

values:
 pulumiConfig:
 flags: ${customSecrets.response}
 customSecrets:
 fn::open::external:
 url: ${getterUrl}
 request:
 secretName: ANY_SECRET
 secret: true

Your infrastructure code remains unchanged: it still imports the ESC environment and reads flag values the same way as in example 3. The difference is that flag values now come from LaunchDarkly, giving you access to advanced features like targeting rules, percentage rollouts, and flag scheduling.

Automatically triggering deployments with webhooks

There is some risk involved with automatically triggering updates. You can leave out the webhook and deploy manually if that’s safer for your team.

LaunchDarkly’s webhook system can trigger infrastructure updates whenever a flag changes. Here’s how the automation flow works:

graph LR
A[Update Flag in LaunchDarkly] --> B[LaunchDarkly Webhook]
B --> C[API Gateway]
C --> D[Lambda Function]
D --> E[Pulumi Deployments API]
E --> F[Infrastructure Updates]

When the infrastructure deploys, it retrieves the current flag values through ESC:

graph LR
A[Infrastructure Stack] --> B[ESC Environment]
B --> C[fn::open::external]
C --> D[Flag Getter Lambda]
D --> E[LaunchDarkly SDK]
E --> |Flag Value| D
D --> |Flag Value| C
C --> |Flag Value| B
B --> |Flag Value| A

The Lambda function is configured as a webhook endpoint in LaunchDarkly:

const launchDarklyWebhook = new LaunchDarklyWebhook("myWebhook", {
 pulumiProv: provCreation.pulumiEnv,
 flagKey: launchdarklyFlag.key,
 pulumiTokenSecret,
 lambdaRole,
}, { provider });

This creates a full automation loop: change a flag in LaunchDarkly’s UI, and your infrastructure automatically updates to reflect the new value. This pattern works with any feature flagging service that supports webhooks.

ESC Only vs ESC with LaunchDarkly

Both approaches enable automated infrastructure updates based on flag changes, but they suit different use cases:

Aspect	ESC Only (Example 3)	ESC with LaunchDarkly (Example 4)
Complexity	Simpler setup with fewer moving parts	More components but unified with application flags
Dependencies	No external services required	Requires LaunchDarkly subscription
Flag Management	Edit via Pulumi CLI or Cloud console	Edit via LaunchDarkly UI with full feature set
Advanced Features	Basic flag storage, versioning, and approvals	Targeting rules, percentage rollouts, experiments, scheduling
Best For	Teams wanting simple config management	Teams already using LaunchDarkly or needing sophisticated flag controls
Key Benefit	Centralized Pulumi-native configuration	Unified management of application and infrastructure flags

Conclusion

Infrastructure deserves the same development practices as application code. Feature flags let you control rollouts, reduce deployment risk, and coordinate changes across your entire system, not just in your applications, but in the infrastructure that supports them.

Pulumi makes infrastructure feature flagging practical. Whether you choose the simplicity of ESC-only flag management or integrate with an enterprise service like LaunchDarkly, you can build infrastructure that responds dynamically to configuration changes while maintaining the same development practices you use for application code.

Introducing ESC Connect: Integrate Any Secret Source with Pulumi ESC

Claire Gaestel — Mon, 01 Dec 2025 00:00:00 +0000

We’re excited to announce ESC Connect — a new capability that lets you integrate any secret source with Pulumi ESC by building simple HTTPS adapter services. If you’ve ever needed to pull secrets from a proprietary system, a legacy tool, or a third-party service that doesn’t have native ESC support, you no longer have to wait for us to build a provider. You can build your own adapter in an afternoon and start using it immediately.

Pulumi ESC has native integrations with popular secret management systems like AWS Secrets Manager, Azure KeyVault, HashiCorp Vault, 1Password, and others. But in real-world infrastructure, you often need to work with systems that fall outside this list. Maybe you built a custom secret management system years ago and it’s still running in production. Maybe you’re using a niche third-party service. Maybe your secrets are locked behind a firewall in a legacy system that predates modern APIs.

ESC Connect changes this by letting you build simple HTTPS adapter services using the external provider. Your adapter handles requests from ESC, fetches secrets from your custom source, and returns them. ESC handles authentication with signed JWT tokens, so you get fine-grained control over access without building a complete security infrastructure.

Building an adapter

Here’s an ESC environment configuration that uses ESC Connect:

values:
 customSecrets:
 fn::open::external:
 url: https://my-adapter.example.com/fetch-secrets
 request:
 secretName: DATABASE_PASSWORD

When you open this environment, ESC makes an authenticated POST request to your adapter. Your adapter validates the JWT token, fetches the secret from your source, and returns it:

const handler = async (event) => {
 // 1. Validate JWT from Authorization header
 const claims = await validateJWT(event.headers.Authorization);

 // 2. Verify audience and body hash for security
 verifyAudience(claims.aud, event.url);
 verifyBodyHash(event.body, claims.body_hash);

 // 3. Fetch secret from your source
 const request = JSON.parse(event.body);
 const secret = await fetchFromYourSource(request.secretName);

 // 4. Return the secret
 return { statusCode: 200, body: JSON.stringify(secret) };
};

The example reference implementation includes an ESCRequestValidator class that handles JWT verification and request integrity checking for you. See the documentation for detailed security requirements and examples in other languages.

Automated rotation

ESC Connect also supports automated secret rotation through fn::rotate::external. Your rotation adapter receives the current credential state, generates new credentials, updates your target system, and returns the new state. ESC handles scheduling and maintains both current and previous credentials during rotation transitions for zero-downtime rotation.

values:
 rotatedCredentials:
 fn::rotate::external:
 inputs:
 url: https://my-adapter.example.com/rotate
 request:
 service: database
 environment: production

Learn more about secret rotation in Pulumi ESC and the external rotator implementation patterns.

Try it out

ESC Connect is available now in Pulumi ESC. We’ve created a deployable reference adapter implementation on AWS Lambda that demonstrates secure request validation:

Check out the documentation for the external provider and external rotator to learn more about building production adapters.

To learn more about Pulumi ESC, explore the ESC documentation or get started for free. If you build an adapter for a system that others might find useful, share it in the Pulumi Community Slack — we’d love to see what you build.

Deletion Protection for Pulumi ESC Environments

Fausto Núñez Alberro — Mon, 03 Nov 2025 14:00:00 -0700

Pulumi ESC environments can now be protected from accidental deletion with a new deletion protection setting.

Environments often contain configuration that supports production workloads or is shared across multiple stacks. Deleting these environments by mistake can disrupt services and require time-consuming recovery. Deletion protection provides a safeguard against these scenarios.

How it works

When deletion protection is enabled for an environment, any attempt to delete it fails until protection is explicitly disabled. This applies to deletions from both the Pulumi Cloud console and the Pulumi CLI.

Protected environments display a shield icon in the environment list and on stack pages where the environment is imported. The icon links directly to the deletion protection settings.

Managing deletion protection

In the Pulumi Cloud console, navigate to your environment’s settings to enable or disable deletion protection with a toggle.

From the CLI, use the new pulumi env settings commands:

# Enable protection
pulumi env settings set myorg/myproject/prod deletion-protected true

# View current setting
pulumi env settings get myorg/myproject/prod deletion-protected

# Disable protection
pulumi env settings set myorg/myproject/prod deletion-protected false

Attempting to delete a protected environment returns a clear error message with instructions.

When to use deletion protection

Enable deletion protection for:

Production environments
Environments imported by multiple stacks
Environments shared across teams
Any environment containing configuration that should persist

Permissions

Only environment admins can modify deletion protection settings. This ensures that protection cannot be removed without appropriate authorization.

Getting started

Deletion protection is available now for all Pulumi ESC environments. Visit your environment settings or use the Pulumi CLI to enable it.

For more information, see the deletion protection documentation.

New Pulumi ESC Onboarding and OIDC Provider Experience

Pablo Terradillos — Fri, 17 Oct 2025 16:00:00 -0300

Pulumi ESC (Environments, Secrets, and Configuration) provides centralized secrets management and configuration orchestration across your infrastructure and applications. Today, we’re excited to introduce a redesigned onboarding experience and a simpler, automated way to set up Pulumi ESC as an OpenID Connect (OIDC) provider.

A new and refreshed onboarding experience

Pulumi ESC helps you manage configuration for Pulumi programs, handle secret rotation, and centralize secrets stored across multiple providers.

We’ve now started simplifying these workflows from the moment you create a new Environment. The redesigned onboarding experience helps you get the most out of ESC.

Using Pulumi ESC as an OIDC Provider

Among its many uses, Pulumi ESC can act as an OpenID Connect (OIDC) provider, issuing short-lived, signed tokens that external systems can exchange for temporary cloud credentials. This eliminates hard-coded credentials and improves your security posture.

Other Pulumi offerings, such as Pulumi Insights, use this mechanism to securely scan resources across your organization.

A simpler way to set up OIDC for your cloud providers

Setting up an OIDC provider manually can involve multiple steps and be error-prone. To simplify this, Pulumi ESC now offers automated setup for AWS, Azure, and Google Cloud directly from the onboarding flow.

For each login provider, we offer different setup options to suit your needs. You can log in with your cloud credentials and let Pulumi Cloud handle the rest, or you can create your resources from your own machine using the Pulumi CLI.

Previously, you had to go to the AWS Console to create an identity provider, create and configure an IAM Role, and set up the trust policy before finally creating your ESC Environment.

Now, you can just use SSO, giving Pulumi Cloud permissions to create the necessary resources automatically.

After we have finished setting up the trust relationship with AWS, we’ll set up a new ESC environment for you under the specified project.

Once setup is complete, your new environment is ready to issue short-lived credentials.

pulumi env run aws-login/tehsis-env -- aws s3 ls

You can now access your cloud resources securely, without worrying about managing long-lived credentials.

What’s next?

This is just the beginning of our work to simplify Pulumi ESC setup and onboarding. We’re continuing to refine how teams configure, secure, and manage environments to make getting started with ESC as smooth as possible.

Try Pulumi ESC today and see how it can streamline your secrets and configuration management!

Pulumi ESC: Open Approvals

Pablo Terradillos — Mon, 13 Oct 2025 16:00:00 -0300

Many teams live with the fear that a production environment might be accidentally opened, exposing credentials or sensitive systems before anyone even notices.

We’re excited to announce a new feature for Pulumi ESC: Open approvals. A governance capability that lets organizations require review and sign-off before an environment is opened (i.e. activated or exposed)

Expanding on our mission to enforce compliance and security without slowing teams down

You may recall that earlier this year we introduced Approvals for updates, which allows teams to require review and sign-off before applying modifications to environment configurations. We are now extending Pulumi ESC auditing and governance capabilities to enable Just in time (JIT) access control.

Many organizations require more robust guardrails around when an environment becomes active or accessible. Open Approvals introduces a gate before environment activation, enforcing that every environment open is intentional, reviewed, and governed.

Open Approvals introduces a gate before environment activation, enforcing that every environment open is intentional, reviewed, and governed.

How it works

When configuring Approvals gates for your environments under Settings → Approval Rulesets, you can now gate Open actions in addition to Updates. From this very same UI, you can define the approvals requirements:

Number of required reviewers
Specific teams or individuals allowed to approve
Whether self-approval is permitted

Once a ruleset is defined, all open operations in the corresponding environment will require to be approved before proceeding.

To create an open request, users need to specify the approval duration, how long they need access after opening the environment, and a description explaining why access is needed.

Approvals can be later review and approved from the Approvals tab.

A request access can also be created from the CLI via the newly introduced pulumi env open-request command.

$ pulumi env open-request org/project/env --access-duration-seconds=2h --grant-expiration-seconds=1h

Closing notes

Open approvals expands on Pulumi ESC support for compliance and governance capability while we stay true to our goal: empower teams to move fast while staying secure and compliant: no matter where or how changes happen.

We’re excited about what’s ahead and look forward to building it together with your feedback.

You can learn more about Approvals in Pulumi ESC Docs.

Introducing Approvals in Pulumi ESC

Pablo Terradillos — Mon, 04 Aug 2025 09:00:00 -0300

Did you know that 80% of unplanned outages aren’t caused by hardware failures or cyberattacks, but by the very changes we make to improve our systems?

Pulumi ESC already enables safer change management with our innovative versioning capability which allows users to track and roll back environment revisions.

Building on this foundation, we’re excited to announce the release of Approvals in Pulumi ESC—a new feature that enables organizations to bring governance and oversight directly into their environment configuration workflows.

With Approvals, teams can require explicit review and sign-off before applying changes to ESC-managed environments, bringing the same rigor to configuration as they already have with infrastructure-as-code and application development.

Enforce Change Management Without Slowing Teams Down

Pulumi ESC helps teams manage environment configurations—such as secrets and application settings—across services and environments, from development through production.

In fast-moving teams, managing these settings safely and consistently is critical. But as the number of contributors grows, so does the risk of accidental or unreviewed changes making it into critical environments.

That’s where Approvals come in.

With Approvals, any proposed change to an ESC environment—whether through the Pulumi Console or CLI—must go through a structured review process before it’s applied. Similar to submitting a Pull Request, contributors can propose changes that are reviewed and approved by designated team members.

This gives you a native, consistent workflow for gating configuration updates without needing external tools or manual oversight.

Why use Approvals?

Teams can use Pulumi ESC Approvals to apply the same rigor of code review workflows to environment configurations and secrets management. Approvals help organizations enforce governance policies, meet compliance requirements, and reduce the risk of misconfigurations by requiring explicit review and sign‑off before applying changes. Each request creates a clear, auditable record of who approved what, when, and why, which is especially valuable for regulated industries or teams with strict change‑management processes.

By introducing a controlled review process, Approvals let developers propose updates while ensuring only authorized reviewers can approve and apply them. This balances collaboration with security and accountability, and works seamlessly in both the Pulumi Cloud console and the Pulumi CLI.

How It Works

To get started, navigate to your environment’s Settings → Approval Rulesets, where you can define approval requirements such as:

Number of required reviewers
Specific teams or individuals allowed to approve
Whether self-approval is permitted
Whether changes require reapproval if modified

Once a ruleset is enabled, any environment update must go through a change request workflow. Instead of directly saving changes, contributors create a draft, which then must be reviewed and approved before being applied.

Changes pending approval are clearly visible to your team, and reviewers can inspect the diff, leave feedback, or revise the request before approving.

Approvals are also fully supported in the Pulumi CLI, using the --draft flag:

$ pulumi env set --draft org/project/env FEATURE_X_ENABLED true

In addition to the CLI, Approvals can be used directly from the ESC SDK—making it easy to integrate change‑management workflows into your own applications or automation scripts.

They are also available through the Pulumi Visual Studio Code Extension, allowing developers to review, approve, and apply configuration changes without leaving their editor.

Finally, the Pulumi Service Provider supports managing approval rules as code. For example:

import * as service from "@pulumi/pulumiservice";
import * as pulumi from "@pulumi/pulumi";

// ...

var approvalRule = new service.ApprovalRule("rule-test", {
 name: "My rule!",
 enabled: true,
 targetActionTypes: ["update"],
 environmentIdentifier: {
 organization: environment.organization,
 project: environment.project,
 name: environment.name,
 },
 approvalRuleConfig: {
 numApprovalsRequired: 3,
 allowSelfApproval: true,
 requireReapprovalOnChange: true,
 eligibleApprovers: [
 {
 rbacPermission: "environment:write",
 },
 {
 user: "pulumi-bot",
 },
 {
 user: "IaroslavTitov",
 },
 ]
 }
})

You can find more examples for different languages at the Pulumi Service Provider Github repository.

Built for Collaboration and Compliance

Whether you’re enforcing separation of duties, complying with industry standards or regulations, or simply want better visibility into changes, Approvals helps bring process to how configuration flows through your systems.

It also integrates seamlessly into existing workflows—no need to reinvent how teams work. Just add governance where it matters most.

What’s Next

Approvals in ESC is just the beginning. We’re exploring how approval workflows can be extended to other areas of the Pulumi ecosystem to provide a consistent, governed experience across the entire software delivery lifecycle.

Our goal is to empower teams to move fast while staying secure and compliant—no matter where or how changes happen. We’re excited about what’s ahead and look forward to building it together with your feedback.

Happy building!

Secrets Management Tools: The Complete 2025 Guide

Asaf Ashirov — Thu, 24 Jul 2025 00:00:00 +0000

Every modern application depends on secrets to function: database passwords, API keys, certificates, and configuration values that enable secure communication between services. But here’s the challenge: as your infrastructure grows, managing these secrets becomes exponentially more complex.

The numbers tell a stark story. According to the Akeyless State of Secrets Management Report, 96% of organizations struggle with secrets sprawl such as credentials scattered across code repositories, configuration files, and deployment scripts. The consequences are severe: Verizon’s 2025 Data Breach Investigations Report found that 88% of data breaches involved compromised credentials, with IBM’s research showing the average breach now costs organizations $4.88 million.

The path forward requires more than just storing secrets securely, it demands intelligent orchestration across your entire infrastructure. But with dozens of solutions available, from traditional enterprise vaults to modern orchestration platforms, finding the right approach can feel overwhelming.

This guide cuts through the complexity. We’ll examine the leading secrets management tools across multiple categories, helping you understand not just what they do, but when and why to use them. Whether you’re architecting your startup’s first production environment or modernizing enterprise legacy systems, you’ll find actionable insights to make the right choice for your specific needs.

Secrets Management Tools Overview

The Evolution Beyond Simple Secret Storage

What started as simple password vaults has evolved into something far more sophisticated. Traditional secret storage addressed only part of the challenge: keeping credentials secure. But modern infrastructure demands orchestration across complex, distributed environments where applications need dozens of different secrets available across multiple cloud providers and deployment stages.

The fundamental shift happened when teams realized that managing secrets effectively requires treating them as infrastructure components, not just security artifacts. Your applications don’t just need database passwords stored securely—they need those credentials dynamically generated, automatically rotated, and consistently available across development, staging, and production environments without manual intervention.

Today’s leading platforms function as intelligent orchestrators rather than simple storage systems. They aggregate secrets from multiple existing sources, provide configuration-as-code capabilities that enable version control and reproducible deployments, and generate dynamic credentials with automatic expiration. This orchestration approach means you can work with your existing secret stores while gaining centralized management and modern capabilities like hierarchical organization that reduces duplication across environments.

The best solutions recognize that secrets management is fundamentally an infrastructure orchestration problem. They provide unified access patterns that work seamlessly across multi-cloud environments, comprehensive audit logging for compliance requirements, and fine-grained access controls that adapt to your organizational structure all while maintaining compatibility with the tools and workflows your teams already use.

Most Popular Secrets Management Tools

Secrets Orchestration Platforms

This represents the newest evolution in secrets management: platforms that connect and coordinate multiple secret sources rather than forcing you to abandon your existing infrastructure. Instead of requiring wholesale migration to yet another secret store, these intelligent brokers provide unified interfaces that work with your current tools while adding modern capabilities like dynamic credential generation and configuration-as-code workflows.

Pulumi ESC (Environments, Secrets, and Configuration)

Pulumi ESC takes a fundamentally different approach to secrets management. Rather than forcing you to migrate from your existing secret stores, it orchestrates them connecting HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, and over 20 other providers through a unified interface. This means you can modernize your secrets workflow without abandoning your current infrastructure investments.

The platform’s strength lies in its configuration-as-code approach. You define hierarchical YAML environments that cascade from base configurations through development, staging, and production. This eliminates the manual configuration copying that leads to secrets sprawl: define common settings once, inherit them everywhere they’re needed, while maintaining appropriate security boundaries between environments.

Enterprise-Grade Security and Compliance: ESC provides comprehensive audit logging with detailed access trails, including user identity, timestamp, access method, and resource details. The platform’s audit capabilities support compliance requirements for frameworks like SOC 2, GDPR, and industry-specific regulations by providing immutable audit logs and real-time monitoring of secret access patterns.

Performance and Scalability: ESC delivers enterprise-scale performance with low-latency secret retrieval, high availability across multiple regions, and the ability to handle thousands of concurrent requests. The architecture is designed to scale automatically with organizational growth while maintaining consistent sub-second response times for secret operations.

Dynamic credential generation sets Pulumi ESC apart from traditional storage-focused tools. Instead of storing long-lived AWS access keys or Azure service principal secrets, it automatically provisions short-lived OIDC tokens on-demand. This dynamic approach eliminates entire categories of security risk by ensuring credentials expire automatically and can’t be misused over extended periods.

Advanced Key Management: ESC provides advanced key management capabilities with support for industry-standard encryption algorithms, automated key rotation, and secure key derivation. The platform offers envelope encryption and fine-grained access controls that ensure encryption keys are managed securely and in accordance with regulatory requirements.

The platform maintains a zero vendor lock-in philosophy through its open-source engine and comprehensive multi-language SDK support for TypeScript, Python, Go, and .NET. Teams can integrate ESC into existing workflows through familiar CLI tooling while maintaining full visibility into orchestration logic through transparent, auditable code.

Cost-Effective Enterprise Solution: Pulumi ESC offers significant cost advantages through its unified platform approach, reducing the need for multiple point solutions and simplifying operational overhead. ESC’s integration with existing infrastructure workflows eliminates the need for separate tools and reduces training costs, while its automated rotation and compliance features reduce manual administrative tasks and associated labor costs. For organizations with strict data residency requirements, Pulumi Service offers self-hosting options that provide additional cost predictability and security control.

Pulumi ESC offers a generous free tier with unlimited environments, making it accessible for teams to experiment with configuration-as-code principles. Usage-based pricing for advanced features scales naturally with organizational growth, making it particularly attractive for multi-cloud environments and teams prioritizing systematic approaches to eliminating secrets sprawl.

Doppler

Doppler focuses on developer experience with an intuitive interface and extensive integrations for modern development workflows. The platform provides branch-based environment management and real-time secret synchronization across connected services, with plans starting at $3 per user per month.

Infisical

Infisical combines open-source transparency with modern secrets management, offering both self-hosted and cloud options. The platform includes built-in secrets scanning capabilities and comprehensive SDK support, with cloud service starting at $8 per user per month.

Enterprise Secrets Vaults

Traditional enterprise vaults focus on maximum security and compliance, offering comprehensive secret storage with advanced features designed for large organizations with sophisticated security requirements.

HashiCorp Vault

HashiCorp Vault remains the established standard for enterprise secrets management, offering unmatched flexibility and a comprehensive feature set for complex environments. The platform provides dynamic secrets generation with automatic expiration for over 50 different systems, enabling organizations to move away from static, long-lived credentials toward more secure, just-in-time access patterns.

Vault’s architecture supports over 100 authentication methods and secret engines, providing flexibility to integrate with virtually any existing infrastructure or identity system. Multi-region clustering with strong consistency guarantees ensures high availability and data integrity across geographically distributed deployments. Extensive audit logging and compliance reporting capabilities help organizations meet regulatory requirements while maintaining operational visibility.

The platform’s cloud-agnostic deployment options support both Kubernetes and traditional VM-based infrastructure, making it suitable for organizations with diverse technical environments. Vault’s open-source core provides transparency and community-driven development, while commercial enterprise features starting at $2 per node per hour add advanced management, replication, and support capabilities.

HashiCorp Vault excels in organizations requiring maximum flexibility and customization, complex multi-cloud environments, and teams with strong DevOps expertise capable of managing sophisticated infrastructure platforms.

CyberArk Conjur

CyberArk Conjur focuses on enterprise security with advanced compliance features and integration with broader privileged access management systems. The platform provides enterprise-grade role-based access control with comprehensive audit capabilities, enabling organizations to implement fine-grained security policies across their entire infrastructure.

Modern versions of Conjur incorporate AI-powered anomaly detection and threat analysis, helping security teams identify unusual access patterns that may indicate compromise or policy violations. Automated compliance reporting supports SOC 2, PCI-DSS, and other regulatory frameworks, reducing the operational overhead of maintaining compliance documentation.

Conjur’s integration with CyberArk’s broader privileged access management suite provides a comprehensive approach to credential security that extends beyond simple secret storage to include session monitoring, just-in-time access provisioning, and threat detection. The platform’s advanced policy engine enables fine-grained access controls that can adapt to changing organizational and regulatory requirements.

With enterprise licensing based on scale and deployment requirements, CyberArk Conjur targets regulated industries, enterprises with strict compliance requirements, and organizations prioritizing maximum security over operational simplicity.

Akeyless

Akeyless offers a modern, cloud-native approach to enterprise secrets management without the operational complexity of traditional self-hosted solutions. The platform’s SaaS-first architecture eliminates infrastructure management overhead while providing enterprise-grade capabilities including just-in-time access and automated credential rotation.

The platform implements zero-knowledge encryption with client-side key management, ensuring that even Akeyless cannot access customer secrets. This approach addresses concerns about cloud-based secret storage while maintaining the operational benefits of a managed service. A comprehensive API and integration ecosystem enables connection with existing tools and workflows.

Akeyless provides a generous free tier supporting up to 5 clients, with usage-based pricing that scales naturally with organizational growth. This pricing model, combined with the elimination of operational overhead, makes Akeyless attractive for organizations seeking enterprise features without the complexity of self-managed infrastructure, particularly cloud-first teams prioritizing rapid deployment and scaling.

Cloud-Native Secrets Managers

Cloud-native solutions provide tight integration with specific cloud platforms, offering optimized experiences for organizations committed to particular cloud ecosystems. These platforms leverage native cloud services and identity systems to provide seamless integration with existing infrastructure.

AWS Secrets Manager

Amazon’s native secrets management service provides deep integration with the AWS ecosystem and automatic rotation capabilities specifically designed for AWS services. The platform offers native integration with over 50 AWS services including RDS, Lambda, ECS, and EKS, enabling seamless credential management across the entire AWS service portfolio.

Automatic rotation capabilities for supported databases and AWS services eliminate the operational overhead of manual credential management while improving security through regular rotation cycles. Cross-region replication with automatic failover ensures high availability for critical secrets, while fine-grained IAM permissions with resource-based policies provide precise access control aligned with AWS security best practices.

VPC endpoint support enables private network access without internet routing, addressing security requirements for air-gapped or highly secure environments. The service’s pricing model of $0.40 per secret per month plus $0.05 per 10,000 API calls provides predictable costs that scale with usage.

AWS Secrets Manager excels for AWS-native organizations, applications requiring automatic credential rotation, and teams leveraging extensive AWS service portfolios where tight integration provides operational efficiency and security benefits.

Azure Key Vault

Microsoft’s comprehensive platform manages secrets, keys, and certificates within the Azure ecosystem while providing strong compliance features particularly valued by government and enterprise customers. The platform integrates seamlessly with Azure services and Active Directory authentication, leveraging existing identity infrastructure to provide consistent access control across the Azure ecosystem.

FIPS 140-2 Level 2 validated Hardware Security Modules provide the highest levels of cryptographic security, making Azure Key Vault suitable for government contractors and organizations with stringent security requirements. Managed HSM support extends these capabilities for scenarios requiring maximum security control and audit capabilities.

Certificate lifecycle management with automatic renewal reduces operational overhead while ensuring consistent security posture across web applications and services. Virtual network integration and private endpoint support enable secure access patterns that align with enterprise network security architectures.

Azure Key Vault’s standard tier pricing of $0.03 per 10,000 operations provides cost-effective secret management for Azure-centric organizations, government contractors requiring FIPS compliance, and enterprises with Microsoft-heavy infrastructure environments.

Google Secret Manager

Google Cloud’s scalable secrets management service is optimized for high-volume, global deployments with strong integration across GCP services. The platform provides automatic global replication across all GCP regions, ensuring low-latency access to secrets regardless of geographic distribution.

IAM integration supports fine-grained, condition-based permissions that can adapt to complex organizational structures and security requirements. Secret versioning with automatic rollback capabilities provides operational safety for configuration changes, while integration with Cloud Build, GKE, Cloud Run, and App Engine enables seamless secret management across Google Cloud’s entire service portfolio.

The platform’s high-performance API with global edge caching minimizes latency for high-frequency secret retrieval operations, making it suitable for applications with demanding performance requirements. Pricing of $0.06 per secret version per month plus $0.03 per 10,000 API calls provides cost-effective scaling for high-volume usage patterns.

Google Secret Manager serves Google Cloud-native organizations, applications requiring massive scale, and global deployments where low latency and high availability are critical operational requirements.

Developer-Focused Tools

These tools prioritize user experience and ease of adoption, often serving as excellent entry points for teams beginning their secrets management journey. They emphasize intuitive interfaces and efficient workflows that reduce barriers to adoption.

1Password Secrets Automation

1Password has successfully expanded beyond personal password management to offer comprehensive business secrets management while maintaining their renowned user experience. The platform implements zero-knowledge architecture with client-side encryption, ensuring that even 1Password cannot access customer secrets while providing the usability that made their consumer products successful.

The intuitive interface remains accessible to both technical and non-technical users, addressing the common challenge of secrets management tools that require specialized expertise. Comprehensive SDK and CLI support enables developer workflows while maintaining the simplicity that drives organization-wide adoption. Integration with popular CI/CD platforms and development tools provides the automation capabilities modern development teams require.

Advanced sharing capabilities include granular, time-limited permissions that enable secure collaboration while maintaining audit trails and access control. This combination of security and usability makes 1Password particularly effective in mixed environments where both technical and business users need access to secrets.

At $8 per user per month with volume discounts available, 1Password Secrets Automation targets teams prioritizing usability over advanced features, mixed technical/non-technical environments, and organizations already invested in the 1Password ecosystem.

Bitwarden Secrets Manager

Bitwarden leverages their password management expertise to provide cost-effective secrets management specifically designed for development teams. The platform includes machine account support for automated CI/CD workflows, addressing the common requirement for non-human access to secrets in modern development processes.

CLI and SDK integration covers major programming languages while maintaining the straightforward approach that characterizes Bitwarden’s products. Comprehensive event logging and audit trails provide the visibility required for security and compliance, while two-person integrity controls add an extra layer of protection for sensitive operations.

The platform’s competitive per-user pricing of $3 per month with transparent costs makes it particularly attractive for small to medium teams, organizations seeking simple and cost-effective solutions, and existing Bitwarden users looking to extend their investment into development workflows.

Application Security & Scanning

These tools focus on preventing secrets leakage by detecting credentials in code repositories, CI/CD pipelines, and other development artifacts. Rather than managing secrets after they’re created, these platforms prevent security incidents by identifying and remediating credential exposure.

GitGuardian

GitGuardian provides comprehensive secrets detection and remediation across the entire software development lifecycle with high accuracy and low false positives. The platform offers real-time scanning of commits, pull requests, and issues across all major Git platforms, ensuring that credential exposure is identified and addressed before code reaches production environments.

Advanced machine learning algorithms detect over 350 secret types while minimizing false positives that can lead to alert fatigue. Developer-friendly remediation workflows provide guided resolution steps that help development teams address issues quickly without disrupting their normal workflows. Historical repository scanning capabilities enable comprehensive security assessment of existing codebases.

Integration with GitHub, GitLab, Bitbucket, and Azure DevOps ensures coverage across diverse development environments. The platform’s free tier supports small teams, while enterprise plans starting at $18 per developer per month provide advanced features and support for larger organizations.

GitGuardian excels in organizations with large codebases, teams implementing DevSecOps practices, and compliance-focused environments where preventing credential exposure is critical to maintaining security posture.

TruffleHog

TruffleHog offers powerful open-source secrets detection with both community and enterprise versions for comprehensive credential scanning. The platform uses high-performance scanning algorithms that combine advanced entropy analysis with pattern matching to identify credentials across diverse file types and encoding formats.

Support for over 700 credential types includes custom patterns that can adapt to organization-specific secret formats. CI/CD integration with GitHub Actions, GitLab CI, and Jenkins enables automated scanning as part of existing development workflows. Historical git repository scanning provides commit-level analysis that can identify when and how credentials entered the codebase.

The active open-source community ensures regular updates and improvements while providing transparency into detection algorithms. Enterprise features and support options are available for organizations requiring commercial backing and advanced capabilities.

TruffleHog serves open-source advocates, security-conscious development teams, and organizations with strong technical security expertise capable of implementing and maintaining open-source security tools.

Specialized & Integration Tools

These tools address specific use cases or provide bridge functionality between secrets managers and deployment platforms, filling gaps in comprehensive secrets management architectures.

External Secrets Operator

The External Secrets Operator provides Kubernetes-native integration with external secrets management systems, enabling GitOps workflows with secure secret handling. The platform uses Kubernetes Custom Resource Definitions to provide native resource management that integrates seamlessly with existing Kubernetes operational patterns.

Multi-provider support includes Vault, AWS, Azure, GCP, and numerous other secret sources, enabling organizations to maintain consistent Kubernetes secret management regardless of their underlying secret storage choices. Automatic secret synchronization with configurable refresh intervals ensures that Kubernetes secrets remain current with external sources while minimizing API load.

Template-based secret transformation and formatting capabilities enable adaptation of external secret formats to Kubernetes requirements. As an active CNCF community project with growing enterprise adoption, the External Secrets Operator benefits from diverse contributions and broad compatibility testing.

This free and open-source tool excels in Kubernetes-heavy environments, teams adopting GitOps practices, and organizations requiring cloud-agnostic secret synchronization across diverse infrastructure platforms.

Berglas

Google’s Berglas provides optimized secrets injection for serverless and container environments, particularly Google Cloud Run and similar platforms. The platform’s serverless-optimized secret injection minimizes cold start impact, addressing one of the key performance challenges in serverless architectures.

Deep integration with Google Cloud services and IAM leverages existing cloud infrastructure to provide secure, efficient secret access. Container-native deployment patterns use init container approaches that work seamlessly with existing containerized application architectures. Minimal runtime overhead and memory footprint ensure that secret management doesn’t impact application performance.

As an open-source project backed by Google’s engineering team, Berglas benefits from direct integration with Google Cloud services and ongoing development aligned with Google Cloud platform evolution. The free and open-source nature makes it accessible for teams of all sizes.

Berglas serves Google Cloud serverless deployments, container-native applications, and teams prioritizing minimal overhead in performance-sensitive environments.

Confidant

Lyft’s open-source Confidant offers a production-proven approach to secrets management with focus on developer experience and AWS optimization. Battle-tested at Lyft’s scale with millions of requests daily, Confidant provides real-world validation of its architecture and performance characteristics.

AWS-optimized architecture leverages KMS and DynamoDB to provide secure, scalable secret storage using proven AWS services. Developer-centric design emphasizes simple API and CLI interfaces that reduce friction in day-to-day development workflows. The blind credentials feature provides enhanced security by ensuring that even system administrators cannot access certain types of sensitive information.

IAM integration with fine-grained permissions leverages AWS’s native access control systems to provide consistent security policies across the entire infrastructure. As a free and open-source solution, Confidant enables organizations to implement enterprise-grade secret management without licensing costs.

Confidant excels in AWS-heavy environments, teams with strong technical capabilities, and organizations seeking proven open-source solutions with demonstrated scale and reliability.

Chamber

Segment’s Chamber provides elegant secrets management using AWS Parameter Store, optimizing for simplicity and cost-effectiveness. The platform leverages existing AWS infrastructure to minimize complexity while providing essential secret management capabilities.

Cost-effective operations use Parameter Store’s generous free tier to provide secret storage without additional infrastructure or licensing costs. Simple deployment requires minimal operational overhead, making it suitable for teams with limited DevOps resources. Version control integration supports infrastructure as code practices by enabling secret management through familiar development workflows.

Namespace organization with hierarchical parameter structure provides logical organization that scales with application complexity while maintaining simplicity of operation. As a free and open-source tool with only AWS Parameter Store costs, Chamber provides extremely cost-effective secret management.

Chamber serves AWS-native deployments seeking simplicity, teams with cost constraints, and organizations preferring minimal tooling complexity over advanced feature sets.

Top Features to Look for in Secrets Management Tools

Secrets Orchestration and Multi-Source Integration

The most advanced secrets management platforms function as intelligent orchestrators, connecting multiple secret sources and providing unified access patterns. This capability is crucial for organizations with heterogeneous environments where secrets are distributed across multiple systems. Rather than forcing migration to a single secret store, orchestration platforms work with existing infrastructure investments while providing centralized management and consistent access patterns.

Effective orchestration platforms can aggregate secrets from existing enterprise vaults like HashiCorp Vault and CyberArk, cloud-native services including AWS Secrets Manager, Azure Key Vault, and Google Secret Manager, legacy systems with database configuration and file-based secrets, and third-party services like 1Password, Bitwarden, and specialized tools. Pulumi ESC exemplifies this orchestration approach, providing a unified interface across over 20 secret sources while maintaining compatibility with existing infrastructure investments.

This orchestration capability reduces migration risk and preserves existing investments while enabling gradual adoption of modern secret management practices. Organizations can maintain their current secret stores while gaining the benefits of centralized management, consistent access patterns, and advanced features like dynamic credential generation.

Configuration as Code and Environment Management

Modern secrets management extends beyond simple credential storage to comprehensive configuration management. The best platforms support configuration-as-code principles that bring software engineering discipline to environment management and secret distribution.

Version control integration with full change tracking and rollback capabilities ensures that configuration changes undergo the same scrutiny as application code. Hierarchical organization reduces duplication through inheritance and composition, enabling teams to define common configurations once and apply them consistently across multiple environments. Reproducible deployments ensure consistency across environments while code review processes apply software engineering practices to configuration changes. Automated testing capabilities validate configuration changes before deployment, preventing configuration errors that could lead to security vulnerabilities or application failures.

This approach transforms secrets from scattered, manual processes into systematic, auditable infrastructure that follows established software engineering best practices. Teams can apply familiar development workflows to secret management, improving both security and operational efficiency.

Dynamic Credential Generation

Static secrets pose inherent security risks due to their long-lived nature and potential for unauthorized sharing. Dynamic credential generation addresses these challenges by creating credentials on-demand with automatic expiration, significantly reducing the attack surface associated with credential compromise.

Effective dynamic credential systems provide short-lived tokens with automatic expiration and renewal, eliminating the need for manual rotation while ensuring credentials cannot be misused over extended periods. Just-in-time access generates credentials only when needed, reducing the window of opportunity for credential misuse. OIDC integration leverages identity providers for cloud access, enabling seamless authentication without storing long-lived cloud credentials.

Database credentials with automatic rotation for major database systems ensure that application access remains current while removing the operational burden of manual credential management. API keys with configurable lifespans and scope limitations provide fine-grained control over third-party service access. This dynamic approach significantly improves security posture while reducing operational overhead.

Multi-Platform Integration and SDK Support

Your secrets management platform should integrate seamlessly with your existing development and deployment infrastructure, providing native support for the tools and workflows your teams already use. Effective integration reduces friction in adoption while ensuring that secret management becomes a natural part of existing processes rather than an additional burden.

Development integration should include CI/CD platforms like GitHub Actions, GitLab CI, Jenkins, and Azure DevOps, enabling automated secret management as part of deployment workflows. Container orchestration platforms including Kubernetes, Docker Swarm, and Nomad should provide native secret injection capabilities. Infrastructure as Code tools like Terraform, Pulumi, and CloudFormation should support secret management through familiar provisioning workflows. Local development integration should include environment synchronization and IDE integration to provide consistent experiences across development and production environments.

Runtime integration capabilities should extend to application frameworks like Spring Boot, Django, and Express.js, enabling developers to access secrets through familiar programming patterns. Cloud platforms including AWS Lambda, Azure Functions, and Google Cloud Run should provide optimized secret injection with minimal performance impact. Monitoring tools, service meshes, and other infrastructure components should integrate seamlessly to provide comprehensive observability and management capabilities.

Comprehensive Audit and Compliance Capabilities

Security and compliance requirements demand detailed visibility into secret access and management activities. Effective audit capabilities provide the transparency required for security monitoring, compliance reporting, and incident investigation while enabling organizations to demonstrate adherence to regulatory requirements.

Complete audit trails with detailed attribution, timestamps, and context enable security teams to understand exactly who accessed which secrets when and from where. Change tracking for all secret modifications, rotations, and access grants provides visibility into the complete lifecycle of credential management. Compliance reporting supporting SOC 2, ISO 27001, PCI-DSS, and other regulatory frameworks automates the documentation required for compliance audits.

Anomaly detection capabilities identify unusual access patterns that may indicate compromise or policy violations, enabling proactive security response. Integration with SIEM systems provides centralized security monitoring and alerting, ensuring that secret management activities are included in overall security operations. These capabilities transform secret management from a potential compliance liability into a security asset that enhances overall organizational security posture.

Performance and Scalability

Consider both current requirements and future growth when evaluating secrets management platforms. API performance with low latency for high-frequency secret retrieval ensures that secret management doesn’t become a bottleneck in application performance. Geographic distribution supporting global deployments with regional access provides consistent performance regardless of deployment location.

High availability with automatic failover and disaster recovery ensures that secret management doesn’t become a single point of failure in critical applications. Horizontal scaling capabilities handle increased load without performance degradation as organizations grow. Intelligent caching strategies optimize performance while maintaining security, reducing API load while ensuring secrets remain current.

Effective performance and scalability ensure that secret management enhances rather than hinders application performance, enabling organizations to scale their security practices alongside their infrastructure growth.

Try secrets orchestration

Pulumi ESC connects your existing secret stores behind a single interface, adding configuration as code and dynamic, short-lived credentials.

Get started

Key Considerations for Choosing Secrets Management Tools

Orchestration vs. Replacement Strategy

The fundamental choice between orchestration and replacement approaches significantly impacts both implementation complexity and long-term flexibility. Orchestration platforms like Pulumi ESC work with existing infrastructure, providing unified interfaces without requiring complete migration. This approach offers reduced migration risk with gradual adoption, preserved investments in existing secret stores and processes, flexibility to use best-of-breed solutions for different use cases, and simplified operations through centralized management of distributed secrets.

Replacement approaches require migration of all secrets to a single platform. While potentially simpler architecturally, this approach involves higher migration costs and extended transition periods, vendor lock-in that makes future changes more difficult, all-or-nothing adoption requiring complete workflow changes, and potential loss of functionality from specialized existing systems.

Organizations with heterogeneous environments, multiple cloud providers, or significant existing investments in secret management infrastructure typically benefit from orchestration approaches that preserve flexibility while providing centralized management capabilities.

Cloud Strategy Alignment

Your choice of secrets management platform should align with your overall cloud strategy and deployment patterns. Organizations with multi-cloud or hybrid strategies benefit from cloud-agnostic solutions that provide consistent experiences across platforms. Pulumi ESC, HashiCorp Vault, and other orchestration platforms excel in these environments by providing unified interfaces regardless of underlying infrastructure.

Organizations committed to specific cloud platforms can leverage native services like AWS Secrets Manager, Azure Key Vault, and Google Secret Manager for optimal integration and cost efficiency within their chosen ecosystem. These platforms provide deep integration with cloud-native services while optimizing costs through native billing and resource management.

Migration flexibility becomes important as organizational requirements evolve. Orchestration platforms provide natural migration paths between different underlying secret stores as requirements change, while cloud-native solutions may require more significant changes if cloud strategy evolves. Consider solutions that maintain flexibility for future changes while meeting current operational requirements.

Team Structure and Expertise

The technical complexity and operational requirements of different secrets management platforms vary significantly, making team structure and expertise critical factors in platform selection. High-complexity tools like HashiCorp Vault and CyberArk Conjur suit organizations with dedicated security or platform teams capable of managing sophisticated infrastructure. These platforms offer maximum flexibility and features but require significant expertise to implement and operate effectively.

Medium-complexity tools like Pulumi ESC and Doppler balance advanced features with operational simplicity, making them suitable for organizations with strong technical teams that prefer managed services or simplified operations. These platforms provide enterprise-grade capabilities without the operational overhead of self-managed infrastructure.

Low-complexity tools like 1Password and Bitwarden prioritize ease of use over advanced capabilities, making them suitable for organizations with limited technical resources or mixed teams where non-technical users require access to secrets. These platforms excel in environments where adoption and usability take priority over advanced features.

Organizational structure also influences platform choice. Centralized teams can manage sophisticated platforms and enforce consistent policies across the organization, while distributed teams benefit from self-service capabilities and intuitive interfaces that enable independent operation. Mixed environments require tools accessible to both technical and non-technical users, emphasizing usability and clear documentation.

Security and Compliance Requirements

Regulatory and security requirements significantly influence platform selection, with different industries and use cases demanding specific capabilities and certifications. Government and defense organizations may require FIPS 140-2 Level 2 validation provided by platforms like Azure Key Vault and CyberArk, while financial services often need comprehensive audit trails and segregation of duties capabilities.

Healthcare organizations require HIPAA-compliant secret handling and access controls, while European operations must consider GDPR implications for secret storage and processing. Understanding these requirements early in the selection process helps narrow platform choices and avoid costly compliance gaps.

Security model preferences also vary significantly between organizations. Zero-knowledge encryption models where service providers cannot access secrets appeal to security-conscious organizations, while Hardware Security Module support provides the highest levels of cryptographic security for sensitive applications. Network isolation capabilities become important for air-gapped or highly secure environments, while multi-factor authentication integration with existing identity systems ensures consistent security policies.

Cost Optimization

Understanding the total cost of ownership for different secrets management approaches requires consideration of both direct and indirect costs. Per-user pricing models suit teams with predictable user counts but can become expensive as organizations scale. Usage-based pricing scales naturally with actual consumption but requires careful monitoring to avoid unexpected costs. Per-secret pricing works well for applications with many users but few secrets, while enterprise licensing may provide cost-effective solutions for large deployments.

Total cost of ownership extends beyond licensing to include operational overhead for self-hosted versus managed solutions, training and certification requirements for specialized platforms, integration costs including development time and ongoing maintenance, and migration expenses from existing solutions. Managed services may appear more expensive initially but often provide better total cost of ownership when operational overhead is considered.

Consider both current costs and future scaling requirements when evaluating pricing models. Platforms that provide cost-effective entry points while supporting growth can provide better long-term value than solutions optimized only for current requirements.

The Future of Secrets Management: Orchestration and Configuration as Code

The secrets management landscape is undergoing a fundamental shift from simple storage systems to sophisticated orchestration platforms. This evolution reflects the reality of modern infrastructure—applications span multiple clouds, integrate with dozens of services, and require dynamic adaptation to changing requirements.

From Storage to Orchestration

Traditional secret management models rely on centralized vaults where all secrets are stored in a single system, requiring applications to understand specific retrieval mechanisms and access patterns. This approach often creates vendor lock-in and forces organizations to migrate from existing secret stores, disrupting established workflows and creating migration risk.

The emerging orchestration model uses intelligent brokers that aggregate secrets from multiple sources—existing vaults, cloud services, legacy systems—presenting them through unified, consistent interfaces. This orchestration approach provides investment protection by working with existing secret stores, gradual migration without disruptive all-or-nothing transitions, best-of-breed integration leveraging specialized solutions where appropriate, and operational simplification through centralized management of distributed secrets.

Orchestration platforms enable organizations to maintain their current infrastructure investments while gaining the benefits of centralized management, consistent access patterns, and modern capabilities like dynamic credential generation. This approach reduces both risk and cost while providing a clear path toward modern secret management practices.

Configuration as Code Revolution

The convergence of secrets and configuration management represents the next major evolution in infrastructure management. Modern applications require more than isolated credentials—they need comprehensive environment configuration including secrets, feature flags, service endpoints, and deployment parameters. Managing these elements separately creates complexity and increases the risk of configuration drift between environments.

Configuration-as-code principles bring software engineering discipline to environment management by providing version control with complete change history and rollback capabilities, code review processes that ensure changes undergo appropriate scrutiny, automated testing that validates configuration changes before deployment, and reproducible environments that eliminate configuration drift between stages.

Pulumi ESC pioneered this approach, demonstrating how configuration-as-code can systematically address secrets sprawl while improving security, reliability, and developer productivity. This convergence transforms environment management from manual, error-prone processes into systematic, auditable infrastructure that follows established software engineering best practices.

Dynamic Credentials and Zero-Trust Architecture

The shift toward dynamic, short-lived credentials aligns with zero-trust security principles that assume compromise and limit blast radius through minimal access grants. Rather than managing long-lived secrets that require careful rotation and access control, modern platforms generate credentials on-demand with automatic expiration.

This approach provides reduced attack surface through minimal credential lifetime, simplified rotation with automatic credential lifecycle management, enhanced audit trails with precise attribution and timing, and improved compliance through automated policy enforcement. Dynamic credentials eliminate many of the operational challenges associated with traditional secret management while significantly improving security posture.

Organizations adopting zero-trust architectures benefit from secret management platforms that support dynamic credential generation, fine-grained access controls, and comprehensive audit capabilities. These platforms become integral components of zero-trust implementations rather than separate security tools.

AI and Machine Learning Integration

Advanced platforms are incorporating AI and machine learning capabilities to enhance security and operational efficiency. Anomaly detection algorithms identify unusual access patterns that may indicate compromise or policy violations, enabling proactive security response before incidents escalate. Intelligent rotation systems optimize credential lifecycles based on usage patterns and risk assessment, balancing security requirements with operational efficiency.

Automated compliance capabilities ensure policies are consistently applied across all environments while predictive security features identify potential vulnerabilities before they’re exploited. These AI-powered capabilities transform secret management from reactive operations into proactive security systems that enhance overall organizational security posture.

Choosing the Right Path Forward

The secrets management landscape has fundamentally shifted from simple storage to intelligent orchestration. Your choice today will determine not just your current security posture, but your ability to scale and adapt as your infrastructure grows more complex.

The decision framework is clearer than you might expect. If you’re managing multi-cloud environments or have significant existing secret store investments, orchestration platforms like Pulumi ESC provide the most strategic value. They let you modernize your approach without disrupting current workflows, while adding configuration-as-code capabilities and hierarchical environment management that systematically address secrets sprawl.

Organizations committed to maximum flexibility and customization will find HashiCorp Vault’s extensibility unmatched, though it requires substantial technical expertise to operate effectively. Cloud-native teams can leverage AWS Secrets Manager, Azure Key Vault, or Google Secret Manager for seamless integration within their chosen ecosystem, optimizing both cost and operational complexity.

Teams prioritizing rapid adoption and developer experience should consider Doppler, Infisical, or 1Password Secrets Automation. These platforms excel at driving organization-wide adoption through intuitive interfaces while providing essential capabilities without overwhelming complexity.

Here’s the critical insight: secrets management isn’t just a security decision—it directly impacts your development velocity and operational efficiency. Platforms that embrace orchestration, configuration-as-code, and dynamic credentials enable faster development cycles while reducing the operational overhead that comes from managing scattered, manual processes.

The cost of maintaining ad-hoc secrets management extends far beyond security incidents. Developer productivity losses from hunting down credentials, operational complexity from manual rotation processes, and the constant risk of configuration drift between environments create technical debt that compounds over time. The investment in systematic solutions pays dividends immediately through reduced operational overhead and improved security posture.

The industry’s evolution toward zero-trust architectures and configuration-as-code practices isn’t slowing down. Organizations that adopt systematic approaches today position themselves to handle tomorrow’s infrastructure complexity with confidence, while those that delay face increasingly difficult migration challenges as their systems grow more distributed and interdependent.

Ready to revolutionize your secrets and configuration management? Explore Pulumi ESC and discover how secrets orchestration can transform your development workflows while eliminating sprawl across your entire infrastructure.

Announcing Doppler Providers for Pulumi ESC: Dynamic Login and Dynamic Secrets

Robert Harris — Thu, 26 Jun 2025 00:00:00 +0000

We are excited to announce support for Doppler within Pulumi ESC! Pulumi ESC centralizes secrets and configuration management, providing a unified source of truth across your environments. With the addition of Doppler, a popular secrets management platform, ESC further extends its ecosystem, enabling seamless and secure access to secrets stored across diverse systems.

This release introduces two distinct dynamic providers for Doppler, each designed to improve security and streamline your workflows:

doppler-login (Dynamic Login): This provider securely generates short-lived OIDC access tokens for authenticating to Doppler. Static, long-lived credentials are a significant security risk. The doppler-login provider directly addresses this by generating temporary, just-in-time credentials using OIDC. Use this provider when you need temporary credentials to interact directly with Doppler, for instance, using the Doppler CLI or SDKs in local development or CI/CD pipelines, without storing long-lived static tokens. ESC manages the OIDC flow, providing a fresh token when needed.
doppler-secrets (Dynamic Secrets): This provider dynamically fetches secrets stored within your Doppler configs and makes them available within the Pulumi ESC environment. Use this provider when you need specific secrets from Doppler to configure your applications or infrastructure managed via ESC. This centralizes secret consumption, allowing you to access Doppler secrets using the same consistent ESC patterns used for AWS, Azure, GCP, Infisical, Vault, 1Password, and more.

Pulumi ESC acts as a robust secrets broker provider consistent API interface for all your tools, applications and workflows. It securely handles both the generation of temporary authentication credentials (like with doppler-login) and the fetching of application secrets (like with doppler-secrets) from various providers such as Doppler, cloud platforms (AWS, Azure, GCP), and other secret managers (Infisical, Vault, 1Password). Once centralized in ESC, these secrets and configurations are consistently available for you to consume via ESC’s many developer friendly methods including the ESC SDK, Pulumi CLI, Kubernetes External Secrets Operator, CSI Driver, or sync them to various platforms where they are needed such as GitHub Secrets, AWS Secrets Manager, and more!

Getting Started: Using the Doppler Providers

Let’s walk through how to configure and use each provider.

Prerequisites

Your Doppler workplace must be on the Team or Enterprise plan in order to use OIDC (via service account identities).

Configure Doppler for OpenID Connect(OIDC) before you try out the providers. Follow the steps in Doppler OIDC documentation. This involves creating a Service Account Identity in Doppler, with OIDC authentication pointing to https://api.pulumi.com/oidc, and configuring the audience and subject claims. Note down the Identity ID.

Create a Pulumi ESC environment (e.g., pulumi-org/doppler-auth/oidc-login) with the following environment definition and update the identityId.

# Environment: pulumi-org/doppler-auth/oidc-login
values:
 doppler:
 # Configure the Dynamic Login provider using OIDC
 login:
 fn::open::doppler-login:
 oidc:
 identityId: <your-identity-id> # Replace with your Doppler Identity ID

 # Expose the token as an environment variable for easy consumption
 environmentVariables:
 DOPPLER_TOKEN: ${doppler.login.accessToken}

Save the environment.
Validate the environment by clicking on Open in the Pulumi Cloud console, or running pulumi env open pulumi-org/doppler-auth/oidc-login in your CLI. The output will include the doppler.login.accessToken.

Usage Example: Run Doppler CLI commands dynamically:

pulumi env run pulumi-org/doppler-auth/oidc-login -- doppler secrets download --no-file --format=json --project=<your-project-id> --config=<your-config-id>
# The DOPPLER_TOKEN env var is automatically injected

How to Use the doppler-secrets Provider (Dynamically Fetching Secrets)

Use this provider to pull secrets from Doppler into your ESC environment for consumption by your applications, CI/CD systems, Pulumi IaC, Terraform and more!

Create an ESC environment where you need the secrets (e.g., pulumi-org/my-app/dev).
Import the dynamic login environment (if using OIDC for authentication, which is recommended). This makes the temporary Doppler token available.
Configure the doppler-secrets provider, referencing the imported login details. See example below.

Specify the secrets to fetch using the get block. Replace placeholders.

# Environment: pulumi-org/my-app/dev
imports:
 # Import the environment performing Dynamic Login (recommended)
 - pulumi-org/doppler-auth/oidc-login # Use the path to your login environment

values:
 # Define a structure to hold secrets fetched from Doppler
 dopplerSecrets:
 fn::open::doppler-secrets:
 # Authenticate using the token from the imported Dynamic Login environment
 login: ${doppler.login} # Pass the login object from the import
 # Specify the Doppler project and config to retrieve secrets from
 project: example-project
 config: dev
 # Specify secrets to retrieve from Doppler
 get:
 # Define names for the secrets as they will appear in ESC's output under 'dopplerSecrets'
 apiKey: # This is the name within ESC
 name: API_KEY # The name of the secret in Doppler
 appSecret: # Pull another secret into ESC
 name: APP_SECRET

 # Optionally, map fetched secrets to environment variables for application consumption
 environmentVariables:
 API_KEY: ${dopplerSecrets.apiKey}
 APP_SECRET: ${dopplerSecrets.appSecret}

Save the environment.
Validate the environment by clicking on Open in the Pulumi Cloud console, or running pulumi env open pulumi-org/my-app/dev in your CLI. The output will show the imported doppler.login, the fetched secrets under dopplerSecrets, and the mapped environmentVariables.

Usage Example: Run an application that needs these secrets:

pulumi env run pulumi-org/my-app/dev -- node app.js
# The API_KEY and APP_SECRET env vars are automatically injected

Conclusion

The new doppler-login and doppler-secrets providers for Pulumi ESC offer powerful and secure ways to interact with Doppler. Use doppler-login for dynamic, short-lived OIDC authentication when interacting directly with Doppler APIs or CLIs. Use doppler-secrets to securely fetch secrets from Doppler into your centralized ESC environment. Together, they enhance your security posture and simplify configuration management.

We encourage you to explore these new capabilities. Dive into the doppler-login, doppler-secrets for detailed configuration options, check out the broader Pulumi ESC Documentation, and share your feedback or ask questions in the Pulumi Community Slack.

Bring Your Own Keys With Pulumi ESC

Pablo Terradillos — Wed, 18 Jun 2025 13:25:26 -0300

Today we’re excited to launch support for Customer-Managed Keys (CMKs) in Pulumi ESC. This feature gives your organization full control over how your secrets and state are encrypted — empowering you to meet the most demanding compliance requirements like HIPAA, GDPR, and FedRAMP, all while maintaining the ease-of-use that Pulumi is known for.

Why Customer-Managed Keys?

Pulumi encrypts secrets and other sensitive information stored in ESC using data keys, which are encrypted with Pulumi-Managed Keys using strong security standards. However, customers might need to use their own keys to gain full control over their lifecycle and revocation, maintain a centralized audit trail, and enforce custom rotation policies. This is especially important to meet compliance requirements or specific regulations (e.g., HIPAA, GDPR, FedRAMP, etc.). Customer-Managed Keys give customers that control while keeping these aspects transparent for the consumers of those secrets.

How It Works

Customer-Managed Keys integrate seamlessly with your Pulumi Cloud Organization. Once enabled, all existing data keys used to encrypt your ESC secrets, previously encrypted with the Pulumi-managed key, will be automatically re-encrypted with the new CMK.

Setting it up takes just a few minutes:

Create a KMS key in your AWS account.
Grant access to the KMS key using OIDC in your AWS account.
Configure your Pulumi Cloud organization to use your new KMS key.

Available Today

Customer-Managed Keys with support for AWS KMS are available today for all organizations on the Pulumi Enterprise or Business Critical plans. You can set it up by following the instructions in the Pulumi Cloud documentation.

We expect to add support for more cloud providers in the future. Please vote on the following GitHub issues for your favorite to support prioritization:

We’re excited to see how Customer-Managed Keys help you and your organization meet your compliance goals with Pulumi Cloud. Your feedback is essential as we continue to evolve this feature:

Connect with us in the Pulumi Community Slack
Open an issue on GitHub
Read the full documentation on Customer-Managed Keys

Try this today, Get Started with Pulumi for free

Announcing Snowflake Dynamic and Rotated Credentials with Pulumi ESC

Claire Gaestel — Thu, 01 May 2025 00:00:00 +0000

Snowflake is the data cloud powerhouse for countless businesses, critical for everything from customer dashboards to billing pipelines. The stakes are immense: this data must be strictly secured and always available. But managing this with static credentials or manual key rotation creates persistent security vulnerabilities and introduces operational instability, risking disruptions during clumsy updates. Pulumi ESC eliminates this dilemma with two purpose-built Snowflake integrations:

snowflake-login: Provides dynamic, short-lived OIDC tokens for temporary authentication to Snowflake.
snowflake-user: Automates the rotation of RSA keypair secrets for Snowflake users, essential for secure key-pair authentication.

Pulumi ESC allows you to securely manage and consume these Snowflake credentials in your applications, development workflows through ESC SDK, Pulumi CLI, Kubernetes integrations, and more!

Use the snowflake-login provider when you need temporary, just-in-time credentials to interact with Snowflake, such as from local development environments, CI/CD pipelines, or scripts. It leverages OIDC to mint short-lived access tokens, eliminating the need to store static tokens.

This approach significantly reduces the attack surface by removing long-lived credentials required for authentication.

Setup:

Configure OIDC in Snowflake: Create a Security Integration in Snowflake to trust the Pulumi OIDC provider (https://api.pulumi.com/oidc), map the appropriate user claims and create a Snowflake user. Check out the docs] for more details
Configure ESC Environment: Define the snowflake-login provider in your ESC environment, specifying the Snowflake account and the user configured for OIDC.

# my-org/logins/snowflake
values:
 snowflake:
 login:
 fn::open::snowflake-login:
 oidc:
 account: myorganization-account
 user: ESC_LOGIN_USER
 role: ESC_ROLE  # Optional

When this environment is opened, ESC securely handles the OIDC flow and makes the temporary token available under snowflake.login.token.

snowflake-user: Automated Rotation for RSA Key Secrets

For applications or services using Snowflake’s key-pair authentication, maintaining the security of those RSA keys is crucial. The snowflake-user rotator automates the lifecycle management of these keys as rotated secrets within ESC. This rotater uses the same two-secret rotation strategy we use in rest of ESC’s rotated secrets providers. This eliminates manual rotation toil and ensures keys are regularly refreshed according to policy, enhancing the security of the user secret itself. Check out the rotated secrets blog post to learn more about its benefits.

Setup:

Prepare Snowflake:
- Create the target user (e.g., MY_APP_SNOWFLAKE_USER) whose keys need rotation.
- Set up a dedicated rotation role with OWNERSHIP permission over the users you wish to rotate, so it can modify their keys.
- Create a service user using the rotation role, and setup OIDC to allow ESC to assume the role for rotation.
Rotation Environment: Define the rotation, importing the managing credentials and specifying the target user.

# Environment: my-org/rotators/snowflake-app-key
values:
 rotatedKey:
 fn::rotate::snowflake-user:
 inputs:
 login: ${environments.logins.snowflake.snowflake.login} #reference credentials created using `snowflake-login`
 targetUser: ESC_ROTATION_DEMO_USER # User whose keys rotate

After setup, use the triple-dot menu -> “Rotate Secrets” in the Pulumi Cloud UI and ensure rotation happens. Once you rotate a couple of time and ensured everything works well, navigate to the “Secret Rotation” tab for your Rotation environment in the Pulumi Cloud console and define the rotation schedule.

Feature	`snowflake-login` (Dynamic OIDC Login)	`snowflake-user` (Rotated Secrets)
Credential Type	Short-lived OIDC access token	Long-lived RSA Private Key secret (managed by ESC) + Public Keys (in Snowflake)
Primary Use Case	Temporary authentication to Snowflake (CLI, scripts, dev, CI/CD).	Key-pair authentication for long-lived applications/services connecting to Snowflake.
Ideal Scenario	Short-lived tasks; frequent, temporary access needs.	Applications needing persistent connections; keys required at deploy time; apps sensitive to auth blips.
Lifecycle	Generated on-demand, expires quickly (minutes/hours).	Rotated automatically on a schedule (e.g., days/months).
Connection Impact	Requires re-authentication or new token fetch when token expires.	Rotation is typically seamless; two-key strategy avoids connection drops if apps update keys promptly.
Benefit Focus	Eliminates static authentication tokens/credentials.	Automates lifecycle management of user key-pair secrets, ensures key freshness.

Conclusion

Pulumi ESC’s new Snowflake integrations provide robust solutions for modern credential management challenges. Use snowflake-login for secure, temporary OIDC-based access and snowflake-user to automate the rotation of critical RSA key secrets. By adopting these capabilities, you can significantly enhance your Snowflake security posture, reduce operational overhead, and simplify compliance.

Explore the detailed documentation for snowflake-login and snowflake-user, and see how Pulumi ESC can centralize and secure your configuration and secrets across all your environments. Join the conversation in the Pulumi Community Slack or our GitHub repository.

Introducing Automated Database Credential Rotation for PostgreSQL and MySQL in Pulumi ESC

Sean Yeh — Wed, 30 Apr 2025 00:00:00 +0000

Securing access to critical data stores is paramount in today’s cloud-native world. Yet, managing database credentials often involves static, long-lived passwords – a significant security blind spot. These static secrets, frequently embedded in application configurations or accessible to multiple team members, represent a prime target for attackers. Manually rotating these credentials is a cumbersome, error-prone task that’s often neglected, leaving databases vulnerable for extended periods. Building on our commitment to robust secrets management, we are excited to launch Automated Database Credential Rotation for PostgreSQL and MySQL in Pulumi ESC!

The Persistent Challenge of Static Database Secrets

Relying on static database credentials introduces substantial risks, which automated rotation helps mitigate:

Security Vulnerabilities & Exposure: Static credentials, if compromised through leaks, phishing, or unauthorized access, provide long-term access to attackers. Automated rotation significantly shrinks this window of opportunity.
Operational & Compliance Burdens: Manually rotating database credentials is complex, error-prone, and requires careful coordination to avoid downtime. These manual rotations are hard to audit and demonstrate compliance with regulations like SOC 2, GDPR, or HIPAA (which often mandate rotation).
Network Complexity & The DIY Burden: Databases in private networks (like AWS VPCs) require secure access for rotation. We provide an open-source Lambda connector for AWS VPCs. While homegrown solutions can bridge the network gap, building and maintaining the actual rotation logic (state, error handling, two-secret strategy) is complex. Crucially, DIY approaches typically lack integrated revision history, centralized auditing, and unified management provided by platforms like Pulumi ESC, increasing operational overhead.
Tooling, Ecosystem, and Cloud-Native Connectivity: Despite cloud IAM authentication options, many standard tools (GUIs, BI, ETL), legacy apps, and even certain cloud-native configurations (multi-cloud, specific Kubernetes setups) still rely on username/password. Direct IAM integration isn’t always feasible or practical, making automated rotation of traditional credentials essential for broad compatibility and security.

Pulumi ESC addresses these challenges by automating the secrets rotation process, including connectivity options for databases in private AWS networks using our Lambda connector, and allows you to consume them in your applications through ESC’s various developer-friendly methods including ESC SDK, Pulumi CLI, Kubernetes External Secrets Operator, CSI Driver, etc.

Introducing ESC Rotated Secrets for Databases

Pulumi ESC’s Rotated Secrets capability now extends to your critical databases, bringing automation and enhanced security to credential lifecycle management. We are adding support for the following databases and connectivity options:

Database	Publicly Accessible	AWS VPCs	Azure VNETs	GCloud VPCs
PostgreSQL	✅	✅ (via Lambda)	Planned (upvote issue)	Planned (upvote issue)
MySQL	✅	✅ (via Lambda)	Planned (upvote issue)	Planned (upvote issue)

Key benefits include:

Automated Rotation Schedules: Effortlessly define rotation frequencies (daily, weekly, monthly) that align with your security policies, eliminating manual toil and ensuring consistency.
On-Demand Rotation: Instantly rotate credentials in response to security events, personnel changes, or policy updates with a single click or command.
Seamless ESC Environment Integration: Manage rotated database credentials alongside all your other application configurations within ESC Environments, leveraging composition and environment inheritance.
Two-Secret Strategy: ESC maintains both the current and previous valid credentials simultaneously. This prevents application downtime during rotation, as instances can gracefully transition to the new credential.
Auditing and Tracking: Maintain a complete audit trail of credential rotations, including who initiated the rotation and when, simplifying compliance and governance.
Admin and Creator Control: Securely configure rotation using privileged database “managing user” credentials stored within ESC, keeping them separate from the rotated credentials consumed by applications.
Configure Webhooks for Automation: Trigger notifications, CI/CD pipelines (e.g., Pulumi Deployments to update application stacks), or custom workflows upon successful rotation, keeping dependent systems synchronized.

The following setup guide provides a detailed walkthrough for rotating PostgreSQL credentials for an AWS RDS instance hosted within a private VPC, which requires the AWS Lambda-based connector. For publicly accessible databases (PostgreSQL or MySQL), you can use the simpler Direct Connect method documented on the respective integration pages. Please refer to the MySQL documentation page for specific instructions on MySQL rotation (both Direct Connect and Lambda Connector).

How to Set Up and Use Rotation for AWS RDS PostgreSQL running in VPC

Setting up rotation for AWS RDS or Aurora PostgreSQL databases running inside an AWS VPC involves creating the required users within your database, deploying the AWS Lambda connector infrastructure via a Pulumi template, configuring ESC environments, and scheduling the rotation.

Understanding the AWS Lambda Connector: The open-source ESC Lambda connector acts as a secure proxy for database credential rotation within your VPC. It enables Pulumi ESC to safely rotate database credentials without needing direct network access from ESC to your database and without exposing your database to external networks. This connector is currently required to rotate credentials for PostgreSQL or MySQL databases hosted within an AWS VPC. Note: The AWS Lambda Connector is only available to Enterprise and Business Critical customers.

Step 1: Prepare Your PostgreSQL Database Users

You’ll need two types of users configured, resulting in three specific database user accounts:

Managing User: A privileged user that ESC will use (via the Lambda connector) to perform the password rotations. This user needs permissions to change the passwords of the application users.
Application User 1 and Application User 2: The two application users whose passwords ESC will manage. ESC rotates between these two users, ensuring two users are always active as applications gradually phase out the previous user credentials.

Connect to the database as a superuser and run the following SQL commands:

Create the application users whose passwords you want ESC to rotate automatically:

```sql
CREATE USER user1 WITH PASSWORD 'initial_password';
GRANT SELECT, INSERT, UPDATE ON yourDatabase TO user1;
CREATE USER user2 WITH PASSWORD 'initial_password';
GRANT SELECT, INSERT, UPDATE ON yourDatabase TO user2;
```

Create the designated “managing user” account with privileges to modify passwords for the application users:

```sql
CREATE USER managing_user WITH PASSWORD 'manager_password';
ALTER USER managing_user WITH CREATEROLE;
GRANT user1 TO managing_user WITH ADMIN OPTION;
GRANT user2 TO managing_user WITH ADMIN OPTION;
```

Step 2: Bootstrap Rotation Infrastructure (Lambda Connector) and Create ESC Environments

To rotate credentials for PostgreSQL databases within AWS VPCs securely, you need to configure an AWS Lambda function (the connector) within your VPC and create the corresponding ESC environments.

If your PostgreSQL instance is running on AWS RDS, we recommend using our Pulumi Template to simplify the process of creating the Lambda function, associated AWS IAM Roles, Security Groups, and the basic ESC environment structure. You can start it from the Pulumi console or by running the following command in your terminal:

Run pulumi new esc-connector-lambda-typescript in your terminal.
This command initializes a new Pulumi project designed to provision the necessary AWS Lambda connector resources and create the ESC environments.
Follow the prompts. You will first provide a Pulumi project name (e.g., my-db-rotation-infra). Then, you will be prompted for configuration values like:
- Your target AWS Region (aws:region)
- The RDS Database Identifier (rdsDbIdentifier)
- The specific name for the ESC rotation environment (e.g., postgresrotation/Rotator). This environment will contain the two application users’ credentials and configuration for rotation.
- The specific name for the ESC managing credentials environment (e.g., postgresrotation/ManagingCreds).
Run pulumi up to deploy the AWS resources and create the ESC environments using the names you provided.

Step 3: Configure ESC Environments and Test

The Pulumi template creates two ESC environments with most of the configuration pre-filled. You need to update them with your specific PostgreSQL user credentials and database name.

Managing Credentials Environment (e.g., postgresrotation/ManagingCreds): Navigate to this environment in the Pulumi Cloud console. Update the managing user’s username and password. The awsLogin section, which provides credentials for the Lambda connector to assume, should be pre-filled by the template.

# postgresrotation/ManagingCreds
values:
 managingUser:
 username: managing_user # Replace with your managing user's name
 password:
 # Securely replace the placeholder with your managing user's password using fn::secret
 fn::secret:
 ciphertext: ZXNjeA... # Replace with your encrypted password
 awsLogin:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::616138583583:role/PulumiEscSecretRotatorLambda-InvocationRole-7e4646f # Example Role ARN populated by template
 sessionName: pulumi-esc-secret-rotator-${esc.context.environment} # Example Session Name populated by template

Rotation Environment (e.g., <project-name>/Rotator): Navigate to this environment. Most fields related to the Lambda connector (Lambda ARN, host, port, references) are pre-filled by the template outputs. You must update the database name and the application usernames mapping. Note the connector.awsLambda block indicating the use of the Lambda connector.

# postgresrotation/Rotator
values:
 dbRotator:
 fn::rotate::postgres:
 inputs:
 database:
 # Configure the AWS Lambda connector for private VPC access
 connector:
 awsLambda:
 login: ${environments.postgresrotation.ManagingCreds.awsLogin}
 lambdaArn: arn:aws:lambda:us-west-2:616138583583:function:PulumiEscSecretRotatorLambda-Function-d9932fe # Example Lambda ARN populated by template
 # Database connection details (used by the Lambda connector)
 database: mydb # Replace with your DB name
 host: tf-20250428212643894700000001.cluster-chuqccm8uxqx.us-west-2.rds.amazonaws.com # Example Host populated by template
 port: 5432 # Example Port populated by template
 managingUser: ${environments.postgresrotation.ManagingCreds.managingUser}
 # List of application users whose passwords will be rotated
 rotateUsers:
 username1: user1 # Replace DB username if different from logical name
 username2: user2 # Replace DB username if different from logical name

 # state section is automatically managed by ESC after first rotation
 # Do not provide state unless migrating existing, known credentials.
 # state:
 # current:
 # password:
 # fn::secret:
 # ciphertext: ZXNjeAAAA...
 # username: user1
 # previous:
 # password:
 # fn::secret:
 # ciphertext: ZXNje21AA...
 # username: user2

Test with Manual Rotation:
- Use the triple-dot menu -> “Rotate Secrets” in the Pulumi Cloud UI for the Rotator environment.
- Alternatively, use the Pulumi CLI: pulumi env rotate <your-org>/<project-name>/Rotator> (e.g., pulumi env rotate myorg/postgresrotation/Rotator). Remember to replace <your-org> and <project-name>.
- Perform a rotation. If you didn’t provide the state block, this first rotation generates a new password for the user designated as current (e.g., user1 associated with username1) and updates it in the database via the Lambda connector. The dbRotator value in ESC will now show this user/password under current, and the state block will be populated.
- Perform a second rotation. Observe how the previous current user (user1/username1) and its password become the previous entry in the ESC value. The system now rotates the password for the other user (user2/username2), makes it the new current, and updates the database accordingly. Subsequent rotations cycle between user1 and user2.
- Check the Environment Revision History in the Pulumi Cloud console to track the changes to the current and previous state fields after each rotation.

Step 4: Schedule Automated Rotations

Once you are satisfied that manual rotations via the Lambda connector are working correctly, navigate to the “Secret Rotation” tab for your Rotator environment (e.g., postgresrotation/Rotator) in the Pulumi Cloud console.
Define your desired automated rotation schedule (e.g., every 30 days). ESC will automatically trigger rotations according to this schedule, ensuring your credentials stay fresh without manual intervention.

What’s Next?

This launch significantly enhances database security posture through automation for PostgreSQL and MySQL, but we’re just getting started. We plan to expand our rotated secrets capabilities:

More Databases: Support for MongoDB, Microsoft SQL Server, Oracle, and others.
Databases in Private Networks on Other Clouds: Rotation solutions for databases hosted in private networks within Azure, Google Cloud, and more.

Have a specific database or integration in mind? Upvote existing issues or open a new request in our GitHub repository!

Conclusion

Pulumi ESC’s Automated Database Credential Rotation drastically simplifies a critical security task. By automating the rotation of PostgreSQL and MySQL credentials, ESC reduces operational overhead, minimizes the risk associated with static secrets, and helps you meet compliance requirements more easily. Whether your databases are publicly accessible (using Direct Connect) or in private AWS networks (using the Lambda connector), ESC provides a unified way to manage their lifecycle securely.

We encourage you to explore the Pulumi ESC documentation and try out rotated database secrets for PostgreSQL and MySQL. Your feedback is invaluable – please share your experiences and suggestions on our GitHub repository. Secure your databases with automated rotation and build with confidence!

Announcing Infisical Providers for Pulumi ESC: Dynamic Login and Dynamic Secrets

Boris Schlosser — Mon, 28 Apr 2025 00:00:00 +0000

We are thrilled to announce enhanced integration support for Infisical within Pulumi ESC! Pulumi ESC centralizes secrets and configuration management, providing a unified source of truth across your environments. With the addition of Infisical, a popular open-source secrets management platform, ESC further extends its ecosystem, enabling seamless and secure access to secrets stored across diverse systems.

This release introduces two distinct dynamic providers for Infisical, each designed to improve security and streamline your workflows:

infisical-login (Dynamic Login): This provider securely generates short-lived OIDC access tokens for authenticating to Infisical. Static, long-lived credentials are a significant security risk. The infisical-login provider directly addresses this by generating temporary, just-in-time credentials using OIDC. Use this provider when you need temporary credentials to interact directly with Infisical, for instance, using the Infisical CLI or SDKs in local development or CI/CD pipelines, without storing long-lived static tokens. ESC manages the OIDC flow, providing a fresh token when needed.
infisical-secrets (Dynamic Secrets): This provider dynamically fetches secrets stored within your Infisical projects and makes them available within the Pulumi ESC environment. Use this provider when you need specific secrets from Infisical to configure your applications or infrastructure managed via ESC. This centralizes secret consumption, allowing you to access Infisical secrets using the same consistent ESC patterns used for AWS, Azure, GCP, Vault, 1Password, and more.

Pulumi ESC acts as a robust secrets broker provider consistent API interface for all your tools, applications and workflows. It securely handles both the generation of temporary authentication credentials (like with infisical-login) and the fetching of application secrets (like with infisical-secrets) from various providers such as Infisical, cloud platforms (AWS, Azure, GCP), and other secret managers (Vault, 1Password). Once centralized in ESC, these secrets and configurations are consistently available for you to consume via ESC’s many developer friendly methods including the ESC SDK, Pulumi CLI, Kubernetes External Secrets Operator, CSI Driver, or sync them to various platforms where they are needed such as GitHub Secrets, AWS Secrets Manager, and more!

Getting Started: Using the Infisical Providers

Let’s walk through how to configure and use each provider.

Prerequisites

Configure Infisical for OpenID Connect(OIDC) before you try out the providers. Follow the steps in Infisical OIDC documentation. This involves creating an Identity in Infisical, adding OIDC authentication pointing to https://api.pulumi.com/oidc, and configuring the audience and subject claims. Note down the Identity ID.

Create a Pulumi ESC environment (e.g., pulumi-org/infisical-auth/oidc-login) with the following environment definition and update the identityId.

# Environment: pulumi-org/infisical-auth/oidc-login
values:
 infisical:
 # Configure the Dynamic Login provider using OIDC
 login:
 fn::open::infisical-login:
 oidc:
 identityId: <your-identity-id> # Replace with your Infisical Identity ID

 # Expose the token as an environment variable for easy consumption
 environmentVariables:
 INFISICAL_TOKEN: ${infisical.login.accessToken}

Save the environment.
Validate the environment by clicking on Open in the Pulumi Cloud console, or running pulumi env open pulumi-org/infisical-auth/oidc-login in your CLI. The output will include the infisical.login.accessToken.

Usage Example: Run Infisical CLI commands dynamically:

pulumi env run pulumi-org/infisical-auth/oidc-login -- infisical secrets get API_KEY --projectId=<your-project-id>
# The INFISICAL_TOKEN env var is automatically injected

How to Use the infisical-secrets Provider (Dynamically Fetching Secrets)

Use this provider to pull secrets from Infisical into your ESC environment for consumption by your applications, CI/CD systems, Pulumi IaC, Terraform and more!

Create an ESC environment where you need the secrets (e.g., pulumi-org/my-app/dev).
Import the dynamic login environment (if using OIDC for authentication, which is recommended). This makes the temporary Infisical token available.
Configure the infisical-secrets provider, referencing the imported login details. See example below.

Specify the secrets to fetch using the get block. Replace placeholders.

# Environment: pulumi-org/my-app/dev
imports:
 # Import the environment performing Dynamic Login (recommended)
 - pulumi-org/infisical-auth/oidc-login # Use the path to your login environment

values:
 # Define a structure to hold secrets fetched from Infisical
 infisicalSecrets:
 fn::open::infisical-secrets:
 # Authenticate using the token from the imported Dynamic Login environment
 login: ${infisical.login} # Pass the login object from the import
 # Specify secrets to retrieve from Infisical
 get:
 # Define names for the secrets as they will appear in ESC's output under 'infisicalSecrets'
 apiKey: # This is the name within ESC
 projectId: <your-infisical-project-id>
 environment: prod # The Infisical environment slug (e.g., 'prod', 'dev')
 secretKey: api-key # The key of the secret in Infisical
 appSecret: # Pull another secret `app-secret` into ESC
 projectId: <your-infisical-project-id>
 environment: dev
 secretKey: app-secret

 # Optionally, map fetched secrets to environment variables for application consumption
 environmentVariables:
 API_KEY: ${infisicalSecrets.apiKey}
 APP_SECRET: ${infisicalSecrets.appSecret}

Save the environment.
Validate the environment by clicking on Open in the Pulumi Cloud console, or running pulumi env open pulumi-org/my-app/dev in your CLI. The output will show the imported infisical.login, the fetched secrets under infisicalSecrets, and the mapped environmentVariables.

Usage Example: Run an application that needs these secrets:

pulumi env run pulumi-org/my-app/dev -- node app.js
# The API_KEY and APP_SECRET env vars are automatically injected

Conclusion

The new infisical-login and infisical-secrets providers for Pulumi ESC offer powerful and secure ways to interact with Infisical. Use infisical-login for dynamic, short-lived OIDC authentication when interacting directly with Infisical APIs or CLIs. Use infisical-secrets to securely fetch secrets from Infisical into your centralized ESC environment. Together, they enhance your security posture and simplify configuration management.

We encourage you to explore these new capabilities. Dive into the infisical-login, infisical-secrets for detailed configuration options, check out the broader Pulumi ESC Documentation, and share your feedback or ask questions in the Pulumi Community Slack.

Why Every Cloud Engineer Needs Pulumi ESC for Secrets Management

Sara Huddleston — Mon, 17 Mar 2025 07:28:56 +0000

Managing secrets is one of the most critical responsibilities in cloud engineering. Secrets like API keys, database credentials, and encryption tokens are the backbone of secure and seamless cloud operations. Yet they are so often an afterthought. They get replicated across cloud-specific secrets managers and stuffed in GitHub secrets, compromising security for the sake of simplicity. ¿Por que no los dos? Why can’t secrets management be secure and simple?

Enter Pulumi ESC (Environments, Secrets, and Configuration)—a breakthrough in taming secrets sprawl and streamlining configuration management across infrastructure. Let’s explore why Pulumi ESC is a necessity for cloud engineers, helping make secrets management secure while keeping it simple.

The Challenge of Secrets Management in Modern Cloud Environments

Secrets management has changed significantly over the past decade. Gone are the days when secrets could be hardcoded, manually maintained in static configuration files, or left floating around in plaintext on dev machines. Nowadays, cloud environments demand a higher level of sophistication:

Distributed Systems: Even in multi-cloud and hybrid setups, secrets must be accessible across varied platforms without exposing vulnerabilities.
Dynamic Infrastructure: Kubernetes, serverless architectures, and ephemeral environments must have secrets that adapt dynamically.
Security Risks: Hardcoded or poorly managed secrets can lead to catastrophic breaches, costing companies millions in data recovery and compliance penalties.
Operational Burdens: Manual secrets management is error-prone, inefficient, unscalable, and needs to align with DevOps and DevSecOps best practices.

Pulumi ESC addresses these issues head-on, redefining how cloud engineers manage secrets efficiently, securely, and at scale.

Jk Jensen, Team Lead at Mysten Labs, said:

“Pulumi ESC has been a lifesaver for us. It’s nice to throw everything behind an ESC environment and eliminate one-off granting/IAM permissions and other issues related to static credentials. It gives us peace of mind knowing that we can grant permissions quickly and revoke easily limiting blast radius for any access.”

What is Pulumi ESC?

Pulumi ESC simplifies how organizations manage secrets and configurations. It is designed to secure sensitive configurations across modern cloud environments and supports seamless integration, enabling engineers to:

Access, share, and manage secrets, passwords, API keys, and configuration settings like network and deployment options.
Synchronize secrets and configuration from any secrets manager to any app, tool, or CI/CD platform.
Access secrets securely through CLI, API, Kubernetes operator, the Pulumi Cloud UI, and in-code with Typescript/Javascript, Python, and Go SDKs.
Connect to cloud providers and secrets stores via OIDC to generate dynamic, short-lived, auto-expiring credentials.
Set role-based access controls (RBAC), making securing secrets and configurations easy by assigning permissions to users based on their roles.

Whether integrated with Pulumi’s Infrastructure as Code (IaC) platform or used as a standalone service, Pulumi ESC enables cloud engineers to streamline secrets management with centralized control.

Key Features of Pulumi ESC

1. Seamless Integration with External Platforms

Pulumi ESC integrates with popular secrets providers, including AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, 1Password, and HashiCorp Vault, making it adaptable for multi-cloud and hybrid cloud architectures.

2. Dynamic Secrets Synchronization with ESO

Partnering with the External Secrets Operator, Pulumi ESC synchronizes secrets securely into Kubernetes clusters. This eliminates hardcoding secrets into manifests or relying on unsecured manual processes.

3. Safely Roll Back to a Previous Version

Pulumi ESC Versioning gives you unprecedented control over your secrets and configuration. Every change is captured in an immutable revision history, allowing you to audit modifications, compare versions, and safely roll back.

4. Secure by Design

Pulumi ESC follows a “secure by default” model, employing encryption, access control, and detailed audit trails. Engineers can meet compliance regulations effortlessly while gaining full visibility into secret access patterns.

5. Automated Rotation and Expiry

Pulumi ESC minimizes security risks by automating the rotation of secrets. This feature aligns secrets management with CI/CD processes for cloud engineers focused on DevOps, ensuring credentials remain valid only when needed.

6. Configuration-as-Code, Automation, and Integration Everywhere

Pulumi ESC embraces an “as-code” approach, enabling configuration and secrets management using TypeScript, JavaScript, Go, Python, or YAML. The ’esc’ CLI and API support automation in CI/CD environments, reducing credential duplication and ensuring a single source of truth.

7. Dev Tools Integrations

Pulumi ESC’s metadata and support for popular configuration formats enable seamless integration with tools like Direnv, Docker, and GitHub, allowing easy management of environment variables, secrets, and configurations.

8. Infrastructure Tools Integrations

Pulumi ESC extends its capabilities beyond Pulumi IaC by integrating with other infrastructure tools such as Cloudflare, Terraform, and OpenTofu. These integrations enable seamless provisioning of cloud credentials and input variables directly from ESC environments.

Why Cloud Engineers Need Pulumi ESC

Enhanced Security and Compliance

Cloud engineers often have to balance security and operations. Pulumi ESC centralizes secrets in a secure vault, ensuring encrypted storage, access control, and audit visibility. It also complies with industry standards like SOC 2, PCI-DSS, and MTCS.

Streamlined Multi-Cloud Management

Managing secrets across AWS, Azure, and GCP can quickly become chaotic. Pulumi ESC’s cross-platform integration simplifies this process, ensuring engineers can manage and sync secrets from a single interface.

Simplified Kubernetes Secrets

Kubernetes’ default secrets offer limited security and scalability. Pulumi ESC with ESO and Secrets Store CSI Driver overcomes these limitations by securely synchronizing and managing secrets within Kubernetes clusters. This prevents unauthorized access and provides automated updates.

Zero Downtime Through Automation

Manual secrets management often leads to errors such as expired credentials or outdated tokens. Pulumi ESC automates the entire lifecycle of secrets—creation, rotation, replication, and expiry—guaranteeing uninterrupted services.

Developer-Friendly Workflows

Tools should make engineers’ lives easier, not harder. Pulumi ESC’s CLI, SDKs, and API provide intuitive ways to integrate into existing workflows. For cloud engineers leveraging Infrastructure as Code with Pulumi, managing secrets alongside the stack becomes effortless.

Thomas Meckel, Senior Cloud Architect at Ophios GmbH, explains:

“Pulumi’s built-in encryption of secrets sets it apart as a leader in secure infrastructure management. Unlike other tools that leave sensitive data exposed or require complex configurations, Pulumi ensures secrets are encrypted by default, leveraging flexible options like Azure Key Vault or AWS KMS. This eliminates the risk of plaintext exposure and simplifies secure deployments. For any organization serious about cloud security, Pulumi’s approach to secrets management is a critical differentiator, combining ease of use with robust protection against breaches.”

Kubernetes Secrets Management

For secrets management in Kubernetes environments, Pulumi ESC becomes even more powerful when paired with External Secrets Operator (ESO) or the Secrets Store CSI Driver. Here’s how it works:

Using Pulumi ESC with External Secrets Operator

External Secrets Operator is an open-source Kubernetes operator that syncs secrets from external providers (like Pulumi ESC) into Kubernetes as native secrets.

Centralized Storage: Store secrets securely in Pulumi ESC and synchronize them with Kubernetes across multiple clusters.
Dynamic Updates: Whenever a secret is updated in ESC, ESO automatically replicates the changes into Kubernetes, eliminating manual intervention.
Granular Management: Define which secrets are synchronized and to which namespaces, ensuring tight access controls and minimizing risk.
Automated Secret Rotation: By leveraging ESC’s rotation capabilities, ESO ensures Kubernetes receives refreshed credentials without requiring downtime.

Learn how to manage Kubernetes secrets with Pulumi ESC and External Secrets Operator: The Perfect Solution for Today’s Cloud-Native Secret Management.

Using Pulumi ESC with Secrets Store CSI Driver

The Secret Store CSI Driver mounts secrets, certificates, and keys from external stores into Kubernetes pods as volumes.

Automated Secrets Injection: Secrets Store CSI Driver enables Pulumi ESC to automatically inject secrets into Kubernetes pods as mounted volumes or environment variables. This reduces the manual overhead of managing secrets directly in Kubernetes.
Secure and Dynamic Secret Access: By leveraging Pulumi ESC, Kubernetes applications can securely fetch secrets from external providers, ensuring dynamic access without exposing credentials in pod specifications.
Streamlined Operations: This integration simplifies the process of synchronizing secrets from Pulumi ESC with Kubernetes-native constructs. The automation reduces errors and boosts efficiency.

Learn how to Master Kubernetes Secrets with Pulumi ESC + Secrets Store CSI Driver.

The Future of Secrets Management with Pulumi ESC

Pulumi ESC is more than just a secrets management tool — it’s the foundation of secure, scalable, and agile cloud operations. For cloud engineers grappling with multi-cloud complexity, Kubernetes deployments, and compliance concerns, it provides a streamlined, automated approach that enhances efficiency and security.

With rising cyber threats and stricter data regulations, adopting tools like Pulumi ESC is no longer optional. It’s a competitive advantage for cloud engineers and organizations aiming to lead in today’s cloud-native world.

Next Steps to Secure Your Secrets

Secrets management doesn’t have to be a headache! With Pulumi ESC, you can safeguard your infrastructure, ensure compliance, and eliminate manual errors—all while enhancing security and agility.

Get started with Pulumi ESC today and experience the future of secrets management. Take control of your cloud environment—securely and effortlessly.

Announcing the Pulumi ESC GitHub Action

Komal Ali — Wed, 05 Mar 2025 00:00:00 +0000

We’re excited to share our latest addition to the Pulumi Ecosystem: the Pulumi ESC GitHub Action. This Action lets you inject secrets and configuration securely into your GitHub Actions workflows as they are needed, rather than storing them as static, long-lived secrets.

Why we built it

Secret management continues to be a challenge for organizations of all sizes. Storing long-lived credentials in multiple places (e.g., GitHub, third-party CI systems, and internal servers) often leads to “secret sprawl” and potential security gaps. With Pulumi ESC, you can migrate to dynamic credentials, and ensure secrets don’t persist beyond their necessary lifetime. If your process requires longer-lived static credentials, with ESC you can manage credentials in a central place and rotate them automatically.

By packaging ESC functionality into a GitHub Action, we’re enabling teams to inject secrets and run ESC commands directly within their GitHub workflows. This allows for a streamlined approach to environment management and secrets rotation—without leaving GitHub.

Key benefits

Secure Secrets Injection - Inject secrets into your GitHub Actions workflows as they are needed, rather than storing them as static, long-lived secrets. This dynamic approach reduces the risk of secrets leaking in logs or pulled from unauthorized sources.
Automatic Secret Rotation - Pulumi ESC handles rotation for you behind the scenes, ensuring that any secrets or credentials you inject into GitHub Actions are valid at the time of use, but rotated on a regular schedule to avoid the risks that come with long-lived static secrets.
Streamlined CI/CD Pipeline - In addition to injecting environment variables, you can run any Pulumi ESC command directly in your GitHub Actions workflows. Automate the creation, update, or teardown of ESC Environments as part of your CI/CD process.
Seamless Integration with Pulumi Cloud - Combine the Pulumi ESC GitHub Action with pulumi/auth-actions to authenticate securely with Pulumi Cloud. No need for long-lived tokens — simply leverage short-lived tokens provided at runtime.

How it works

At a high level, the Pulumi ESC GitHub Action can do three things:

Install the Pulumi CLI - By default, the action ensures the Pulumi CLI is installed in your GitHub Actions environment. If you provide a specific version, that version is installed.
Inject all environment variables from an ESC environment - If you specify an environment (for example, tinyco/someProject/myEnv), the action will inject all environment variables and projected files associated with that environment into the GitHub Actions context.
Inject specific environment variables from an ESC environment - If you only need certain keys, you can specify them in the action’s inputs to pick and choose what gets injected, giving you fine-grained control.

Example workflow

Below is a minimal workflow that showcases injecting an entire environment’s variables into your GitHub Actions job:

on:
 - pull_request

permissions:
 id-token: write
 contents: read

jobs:
 test-env-injection:
 runs-on: ubuntu-latest
 steps:
 - name: Check out repository
 uses: actions/checkout@v4
 - name: Authenticate with Pulumi Cloud
 uses: pulumi/auth-actions@v1
 with:
 organization: pulumi
 requested-token-type: urn:pulumi:token-type:access_token:organization
 - name: Install and inject ESC environment variables
 uses: pulumi/esc-action@v3
 with:
 environment: 'tinyco/someProject/myEnv@stable'
 - name: Verify environment variables
 run: |
 echo "FOO=$FOO"
 echo "SOME_IMPORTANT_KEY=$SOME_IMPORTANT_KEY"
 echo "TEST_ENV=$TEST_ENV"

The GitHub Action workflow above does the following:

Authenticate with Pulumi Cloud using pulumi/auth-actions. Instead of storing a permanent token, we use a short-lived token to authenticate with Pulumi Cloud.
Inject the environment variables from tinyco/someProject/myEnv into the current GitHub Actions environment, making them accessible to subsequent steps in the workflow.

What’s Next?

Explore More ESC Features: Dive deeper into ESC’s environment management, secret rotation, and dynamic credentials.
Contribute to the GitHub Action: The Pulumi ESC GitHub Action is open source. Feel free to open issues, submit PRs, or suggest enhancements.
Join the Pulumi Community: Connect with other Pulumi users and share best practices in our Community Slack.

We’re thrilled to bring you the Pulumi ESC GitHub Action, allowing you to automate your infrastructure and streamline secrets management from within GitHub. This integration makes it simpler than ever to adopt best-in-class secrets management and environment configuration practices, all while speeding up your CI/CD pipelines.

Have questions or feedback? Feel free to reach out to us in our Community Slack or submit an issue on GitHub. We can’t wait to see what you build!

Introducing Rotated Secrets in Pulumi ESC

Claire Gaestel — Wed, 19 Feb 2025 00:00:00 -0400

Managing secrets effectively is no longer a “nice-to-have”—it’s a must-have for any organization building and scaling applications in the cloud. Static, long-lived credentials like database passwords, API keys, and IAM user credentials are a major security vulnerability. They’re often overexposed, residing in source code, configuration files, or other easily accessible locations. Manual rotation processes are tedious, error-prone, and infrequent, leaving a wide window of opportunity for potential breaches. Today, we’re thrilled to announce a powerful new capability in Pulumi ESC that directly addresses this challenge: Rotated Secrets.

The Challenge with static secrets: A ticking time bomb

Traditional secrets management often relies on static, long-lived credentials. This approach creates several critical problems:

Increased attack surface: The longer a secret remains unchanged, the greater the chance it will be compromised through accidental exposure, phishing attacks, or insider threats.
Manual, error-prone rotation: Manual rotation processes are time-consuming and require significant coordination across teams. A single mistake can lead to application downtime or even data breaches.
Inconsistent practices: Ad-hoc scripts and manual procedures often lead to inconsistent rotation policies, making it difficult to enforce security best practices across the organization.
Compliance headaches: Many compliance regulations (like SOC 2, GDPR, and HIPAA) require regular secret rotation. Manual processes make it difficult to demonstrate compliance and can lead to costly penalties.

These challenges are amplified in complex, multi-cloud, hybrid, or on-premises environments where secrets are distributed across numerous systems. While Pulumi ESC’s Dynamic Secrets addresses all of these challenges and makes secret management much easier, static secrets remain a reality for many organizations. Legacy applications, compliance mandates, or integration limitations often prevent a full switch to dynamic credentials. In such cases, Rotated Secrets provide a second-best option—automating credential rotation to minimize risk while integrating seamlessly with existing workflows.

Introducing ESC Rotated Secrets

Pulumi ESC’s new Rotated Secrets feature provides an elegant solution to the challenges of secret lifecycle management. It enables automated, seamless rotation of credentials, ensuring that your secrets are always up-to-date and secure. Here’s how it solves the problems outlined above and more:

Automated rotation schedules: Define flexible rotation schedules tailored to your security and operational requirements. This eliminates manual effort and ensures consistent, frequent rotations.
On-demand rotation: Need to rotate a secret immediately due to a suspected breach or policy change? ESC allows you to trigger rotations on demand, giving you immediate control.
Seamless integration with ESC Environments: Rotated secrets are seamlessly integrated into your existing ESC environment definitions. This allows you to leverage the power of ESC’s composability, allowing you to manage rotated secrets alongside your other configuration values.
Two-secret strategy: ESC Rotated secrets uses a two-secret strategy in which two secrets are active and valid at any point in time. This is especially useful when multiple instances of an application share a credential but not all instances pull in the latest credential at the same time, allowing you to rotate a secret without worrying about some instances being unavailable due to invalid credentials.
Auditing and tracking: Gain complete visibility into credential usage, including a full history of generated credentials, active credentials, and the principals accessing them. This enhances governance and simplifies compliance.
Admin and creator control: Admins and creators/writers of rotated secrets can configure rotations using privileged user credentials and keep those credentials private while ensuring the consumers can still consume the resulting rotated secrets. This provides a clear separation of concern for administration and individual usage.
Configure webhooks for automation: Notify teams, trigger custom workflows, or trigger a Pulumi Deployment to update your IaC stacks whenever rotations occur, ensuring dependent teams and systems are updated on a timely basis

With today’s launch, we support AWS IAM user credential rotation with many more databases and cloud integrations on the way. Upvote on our GitHub issues to get support for new integrations.

How to Use ESC Rotated Secrets

Below are the steps to configure AWS IAM user rotated secrets using best practices. You can also configure this via Pulumi IaC. Check out the example program.

1. Provision an AWS IAM User on your AWS console

Create an AWS IAM user whose access keys you want to rotate.

2. Create a new environment with managing user credentials

In the example environment credentials/aws-creds below, we use the ESC’s aws-login provider using OIDC. This credential was configured with the privileged access to be able to rotate the IAM user credentials.

The minimal permissions the managing credential requires are IAM List, Create, and DeleteAccessKeys over the IAM user you created.

# credentials/aws-creds

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::12345689:role/test-managing-role
 sessionName: pulumi-environments-session
 subjectAttributes:
 - currentEnvironment.name

3. Create a new environment with the IAM user rotation configuration

Define the rotation using rotated secrets aws-iam provider. Below is an example definition. Notice how the managing user credentials created in step 2 is referenced in the rotation provider input. This credential will only be used during rotation and never be exposed to the consumers of the rotated IAM user credential.

Providing current and previous credentials is optional. If you already have IAM user access keys that are actively used, you can specify them in the state field as current and previous. If not, you can leave the state empty.

values:
 rotated-creds:
 fn::rotate::aws-iam:
 inputs:
 region: us-west-1
 login: ${environments.credentials.aws-creds.aws.login}
 userArn: arn:aws:iam::{accountId}:user/{username}
 # state section below is optional.
 state:
 current:
 accessKeyId: AKIA...
 secretAccessKey:
 fn::secret: ASIA....
 previous:
 accessKeyId: AKIA...
 secretAccessKey:
 fn::secret: ASIA....

4. Test the setup with a manual rotation

In the Environment Editor screen, open the triple-dot menu and select Rotate Secrets. Alternatively, you can perform the rotation from the ‘secrets rotation’ tab or use the rotate CLI command. Every rotation creates a new revision.

If you did not provide current or precious credentials in step 3, rotation will generate new credentials automatically.

5. Monitor rotations

Perform multiple rotations to observe how access keys move from current → previous → deleted. Use the Environment Revision History to track changes.

6. Schedule rotations

Now that you’ve correctly configured your rotated secrets, set up an automated rotation schedule from the Secrets Rotation tab.

Rotated Secrets vs. Dynamic Credentials: Choosing the right tool

Pulumi ESC offers both Dynamic Secrets and Rotated Secrets. While both enhance security, they serve different purposes:

Dynamic secrets: These are short-lived, ephemeral credentials with a Time-To-Live (TTL) typically ranging from 1 to 8 hours. A new set of credentials is generated every time the environment is opened. This is ideal for minimizing the impact of compromised credentials, as they quickly become invalid.
Rotated secrets: These are longer-lived credentials, potentially lasting for days, weeks, or even months. The same latest credential is returned every time the environment is opened until the next scheduled rotation occurs. This provides stability and predictability for applications that need consistent credentials over a longer period.

Here are scenarios where rotated secrets are the preferred choice:

Legacy applications: Many legacy applications were built before modern secret management practices existed and cannot easily be rewritten to support dynamic secrets. For these applications, Rotated Secrets offer a practical way to enhance security without requiring a full application redesign or re-architecture.
Applications that can only update credentials during restarts and redeployments: Some applications require credentials to be available at startup and cannot refresh them dynamically. Rotated Secrets works well for this scenario, providing rotations and ensuring applications always have a valid credential when restarted.
Applications requiring persistent connections: High-performance applications or applications that have a long-lived database connections cannot afford to fetch fresh credentials every few hours. Rotated Secrets ensure such applications continue to function as needed while boosting security.
Third-Party API keys (Support coming soon): Some external services cannot provision dynamic short-term credentials. Rotated Secrets provide a way to comply with periodic key rotation policies without requiring real-time credential generation.
Databases in private networks (Support coming soon): Many Enterprise databases are not connected to the internet, thereby requiring a proxy or an agent running in their internal network to perform the rotation. Rotated Secrets offer stability in such environments by reducing the number of external calls, even when routed through a proxy.

What’s next?

This is just the beginning! We’re committed to expanding the capabilities of Rotated Secrets. Here’s a glimpse of what’s on the roadmap:

Database Secrets Rotation: Automated secrets rotation support for a wide range of databases including PostgreSQL, Snowflake, MySQL, and more!
Secret rotations for databases running in private networks
3rd party API key rotation (example: Twilio)
Other cloud provider support: Azure, GCP

Upvote integrations here. Want an integration not mentioned? Click here to raise a request.

Conclusion

Pulumi ESC’s Rotated Secrets feature represents a significant advancement in modern secrets management. By automating a traditionally manual and error-prone process, ESC reduces operational risks, strengthens your security posture, and helps you achieve compliance – all without slowing down development.

We encourage you to explore the docs and try out Rotated Secrets. Share any feedback or open new issues in our GitHub repository. We’re excited to see how you use this powerful new feature to build more secure and resilient applications!

Master Kubernetes Secrets with Pulumi ESC + Secrets Store CSI Driver

Engin Diri — Wed, 18 Dec 2024 09:58:01 +0100

Welcome to the second blog post of the Pulumi ESC and Kubernetes secrets series. If you haven’t had the chance to read the first blog post, go ahead and read it here.

In the previous blog post, we have learned how to manage secrets with Pulumi ESC and the External Secrets Operator. While the External Secrets Operator is a great tool to manage secrets in a cloud-native way, it still creates Kubernetes secrets in the cluster. Depending on your security requirements, you might want to avoid the use of Kubernetes secrets in your cluster at all. This is the point where you hit the limits of the External Secrets Operator.

But don’t worry, we also have a solution for this problem. In this blog post, we will introduce the Secrets Store CSI Driver.

TL;DR?

Jump straight to the comparison of the External Secrets Operator and the Secrets Store CSI Driver here.

Refresher: Why Should You Avoid Using Kubernetes Secrets?

Before we dive into the Secrets Store CSI Driver, let’s quickly recap why you might want to avoid using native Kubernetes secrets. I wrote in depth about this topic in the previous blog post, but here is a quick summary:

By default, etcd stores them in a base64-encoded format, which is not secure.
Developers create them either manually by using kubectl commands or by using a manifest file, which makes them hard to manage at scale.
Hard to manage and synchronise secrets across different environments and clusters.
There is no default way to rotate secrets automatically.

Point 1 is the most critical one, as there is even a warning in the Kubernetes documentation:

Kubernetes Secrets. Credit: kubernetes.io/docs/concepts/configuration/secret/

What is the Secrets Store CSI Driver?

The first thing you may notice is that in the name of the Secret Store CSI Driver, there is a reference to the Container Storage Interface (CSI) standard.

What is CSI and why it is used in the Secret Store CSI Driver*?*

How to write a Container Storage Interface (CSI) plugin. Credit: arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/

Container Storage Interface (CSI)

The Container Storage Interface (CSI) is a standard to unify the interface between container orchestrators (like Kubernetes) and different storage vendors (like NetApp, Ceph, etc.). This helps to guarantee that implementing a CSI for a storage vendor is going to work with all orchestrators that support CSI.

Before CSI, we needed to create volume plugins for every orchestrator and storage vendor. This meant that developers coupled volume plugin development with the Kubernetes version and made it dependent on it. Bugs in volume plugins could break the Kubernetes components, instead of the volume plugin. Besides, volume plugins had full privileges on the Kubernetes components like the kubelet.

Combining CSI, Kubernetes and Secrets = Secret Store CSI Driver

The Secret Store CSI Driver is a CSI driver that allows you to mount many secrets, certificates, and keys from external secret stores into Kubernetes pods as volumes. After attaching the volume, the system mounts the secrets into the container file system.

The benefits of using the Secret Store CSI Driver are that you manage the lifecycle of the secrets outside of Kubernetes while still providing a Kubernetes-native experience of using the secrets in your pods.

The Architecture of the Secret Store CSI Driver

The Secret Store CSI Driver is a daemonset that lets all kublets communicate with each other and uses gRPC to talk to a provider. A SecretProviderClass custom resource specifies the definition of the external Secret Store. Then the system mounts a volume in the pod as tmpfs, and it injects the secret into the volume. When you delete the pod, the system cleans up the volume and removes the secret from the tmpfs volume.

How to Use the Secrets Store CSI Driver with Pulumi ESC

Before we start, make sure you have the following prerequisites:

A Pulumi Cloud account. If you don’t have one, you can create one for free.
A Kubernetes cluster (I will be using a local KinD cluster, but you can use any Kubernetes cluster)
Pulumi CLI installed. You can use the pulumi env commands to manage secrets and configurations.
kubectl CLI installed for some debugging

Step 1: Deploy the Secrets Store CSI Driver and Pulumi ESC CSI Provider

We will use Pulumi with the pulumi-kubernetes provider to deploy the Secrets Store CSI Driver and the Pulumi ESC CSI Provider to our Kubernetes cluster.

And here we also use the integration of Pulumi ESC with Pulumi IaC to supply the Pulumi access token in a secure way.

First, we need to create a new Pulumi project:

# Choose your favorite Pulumi supported language
pulumi new kubernetes-<your-programming-language> --name

Before we dig into the code, let’s head to the Pulumi Cloud Console and create a new Pulumi ESC project with the name esc-secrets-store-csi-driver-demo.

And create the environment dev:

In the editor add the following yaml into the Environment definition:

values:
 pulumiConfig:
 pulumi-pat:
 fn::secret: <your-pulumi-pat>

Don’t know how to create a PAT? Check out the official Pulumi documentation.

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/esc-secrets-store-csi-driver-demo/dev

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/esc-secrets-store-csi-driver-demo/dev

Now, we need to link the Pulumi ESC project to the Pulumi IaC project. To do this, we need to add the following to your Pulumi.dev.yaml:

environment:
- esc-secrets-store-csi-driver-demo/dev

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

const secretsStoreCSIDriver = new k8s.helm.v4.Chart("secrets-store-csi-driver", {
 chart: "secrets-store-csi-driver",
 namespace: "kube-system",
 repositoryOpts: {
 repo: "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 },
 values: {
 nodeSelector: {
 "kubernetes.io/os": "linux",
 },
 },
});

const secretsStoreCSIPulumiESCProvider = new k8s.helm.v4.Chart(
 "secrets-store-csi-pulumi-esc-provider",
 {
 chart: "oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 namespace: "kube-system",
 values: {
 nodeSelector: {
 "kubernetes.io/os": "linux",
 },
 },
 },
 { dependsOn: secretsStoreCSIDriver },
);

const config = new pulumi.Config();

const mySecret = new k8s.core.v1.Secret(
 "my-secret",
 {
 metadata: {
 namespace: "default",
 name: "pulumi-access-token",
 },
 stringData: {
 "pulumi-access-token": config.require("pulumi-pat"),
 },
 type: "Opaque",
 },
 { dependsOn: secretsStoreCSIPulumiESCProvider },
);

import pulumi
import pulumi_kubernetes as k8s

secrets_store_csi_driver = k8s.helm.v4.Chart(
 "secrets-store-csi-driver",
 k8s.helm.v4.ChartArgs(
 chart="secrets-store-csi-driver",
 namespace="kube-system",
 repository_opts=k8s.helm.v4.RepositoryOptsArgs(
 repo="https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 ),
 values={
 "nodeSelector": {
 "kubernetes.io/os": "linux",
 },
 },
 ),
)

secrets_store_csi_pulumi_esc_provider = k8s.helm.v4.Chart(
 "secrets-store-csi-pulumi-esc-provider",
 k8s.helm.v4.ChartArgs(
 chart="oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 namespace="kube-system",
 values={
 "nodeSelector": {
 "kubernetes.io/os": "linux",
 },
 },
 ),
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_driver]),
)

config = pulumi.Config()

my_secret = k8s.core.v1.Secret(
 "my-secret",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 namespace="default", name="pulumi-access-token"
 ),
 string_data={
 "pulumi-access-token": config.require("pulumi-pat"),
 },
 type="Opaque",
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_pulumi_esc_provider]),
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 appv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 chartv4 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v4"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 secretsStoreCsiDriver, err := chartv4.NewChart(ctx, "secrets-store-csi-driver", &chartv4.ChartArgs{
 Chart: pulumi.String("secrets-store-csi-driver"),
 Namespace: pulumi.String("kube-system"),
 RepositoryOpts: chartv4.RepositoryOptsArgs{
 Repo: pulumi.String("https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"),
 },
 Values: pulumi.Map{
 "nodeSelector": pulumi.Map{
 "kubernetes.io/os": pulumi.String("linux"),
 },
 },
 })
 if err != nil {
 return err
 }

 secretsStoreCsiPulumiEscProvider, err := chartv4.NewChart(ctx, "secrets-store-csi-pulumi-esc-provider", &chartv4.ChartArgs{
 Chart: pulumi.String("oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider"),
 Namespace: pulumi.String("kube-system"),
 Values: pulumi.Map{
 "nodeSelector": pulumi.Map{
 "kubernetes.io/os": pulumi.String("linux"),
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiDriver}))
 if err != nil {
 return err
 }

 pulumiPAT := config.Get(ctx, "pulumi-pat")

 mySecret, err := corev1.NewSecret(ctx, "my-secret", &corev1.SecretArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: pulumi.String("default"),
 Name: pulumi.String("pulumi-access-token"),
 },
 StringData: pulumi.StringMap{
 "pulumi-access-token": pulumi.String(pulumiPAT),
 },
 Type: pulumi.String("Opaque"),
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiPulumiEscProvider}))
 if err != nil {
 return err
 }


 return nil
 })
}

using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Helm;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using Pulumi.Kubernetes.ApiExtensions;
using System.Collections.Generic;



return await Deployment.RunAsync(() =>
{
 var secretsStoreCsiDriver = new Release("secrets-store-csi-driver", new()
 {
 Chart = "secrets-store-csi-driver",
 Namespace = "kube-system",
 RepositoryOpts = new Pulumi.Kubernetes.Types.Inputs.Helm.V3.RepositoryOptsArgs
 {
 Repo = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 },
 Values = new Dictionary<string, object>
 {
 { "nodeSelector", new Dictionary<string, object>
 {
 { "kubernetes.io/os", "linux" },
 } },
 },
 });

 var secretsStoreCsiPulumiEscProvider = new Release("secrets-store-csi-pulumi-esc-provider", new()
 {
 Chart = "oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 Namespace = "kube-system",
 Values = new Dictionary<string, object>
 {
 { "nodeSelector", new Dictionary<string, object>
 {
 { "kubernetes.io/os", "linux" },
 } },
 },
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiDriver } });

 var config = new Config();
 var pulumiPAT = config.Require("pulumi-pat");

 var mySecret = new Secret("my-secret", new SecretArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Namespace = "default",
 Name = "pulumi-access-token",
 },
 StringData =
 {
 { "pulumi-access-token", pulumiPAT }
 },
 Type = "Opaque",
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiPulumiEscProvider } });

});


class SecretProviderClassArgs : CustomResourceArgs
{
 public SecretProviderClassArgs(): base("secrets-store.csi.x-k8s.io/v1", "SecretProviderClass")
 {
 }

 [Input("spec")]
 public Input<SecretProviderClassSpecArgs>? Spec { get; set; }
}


class SecretProviderClassSpecArgs : ResourceArgs
{
 [Input("provider")]
 public Input<string>? Provider { get; set; }

 [Input("parameters")]
 public Input<InputMap<object>>? Parameters { get; set; }
}

class SecretProviderParametersArgs : ResourceArgs
{
 [Input("apiUrl")]
 public Input<string>? ApiUrl { get; set; }

 [Input("organization")]
 public Input<string>? Organization { get; set; }

 [Input("project")]
 public Input<string>? Project { get; set; }

 [Input("environment")]
 public Input<string>? Environment { get; set; }

 [Input("authSecretName")]
 public Input<string>? AuthSecretName { get; set; }

 [Input("authSecretNamespace")]
 public Input<string>? AuthSecretNamespace { get; set; }

 [Input("secrets")]
 public Input<string>? Secrets { get; set; }
}

Deploy the stack by running:

pulumi up

And you should see that the secret was created in the Kubernetes cluster and the ESO instance was deployed successfully.

kubectl get secret pulumi-access-token -o jsonpath='{.data.PULUMI_ACCESS_TOKEN}' | base64 -d

Step 2: Create a SecretProviderClass

Now, we will create a secret in the Pulumi ESC project and synchronize it into the Kubernetes cluster by creating an SecretProviderClass:

Create a new ESC environment called csi-secrets-store-app in the esc-secrets-store-csi-driver-demo project:

values:
 app:
 hello: world
 hello-secret:
 fn::secret: world

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/esc-secrets-store-csi-driver-demo/csi-secrets-store-app

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/esc-secrets-store-csi-driver-demo/csi-secrets-store-app

Either way, you should see following environment configuration in the Pulumi Cloud Console:

We can now create the SecretProviderClass in the Kubernetes cluster:

const secretProviderClass = new k8s.apiextensions.CustomResource(
 "example-provider-pulumi-esc",
 {
 apiVersion: "secrets-store.csi.x-k8s.io/v1",
 kind: "SecretProviderClass",
 metadata: {
 name: "example-provider-pulumi-esc",
 namespace: "default",
 },
 spec: {
 provider: "pulumi",
 parameters: {
 apiUrl: "https://api.pulumi.com/api/esc",
 organization: "dirien",
 project: "esc-secrets-store-csi-driver-demo",
 environment: "csi-secrets-store-app",
 authSecretName: mySecret.metadata.name,
 authSecretNamespace: mySecret.metadata.namespace,
 secrets: `- secretPath: "/"
 fileName: "hello"
 secretKey: "app.hello"
`,
 },
 },
 },
 { dependsOn: secretsStoreCSIPulumiESCProvider },
);

secret_provider_class = k8s.apiextensions.CustomResource(
 "example-provider-pulumi-esc",
 api_version="secrets-store.csi.x-k8s.io/v1",
 kind="SecretProviderClass",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="example-provider-pulumi-esc", namespace="default"
 ),
 spec={
 "provider": "pulumi",
 "parameters": {
 "apiUrl": "https://api.pulumi.com/api/esc",
 "organization": "dirien",
 "project": "esc-secrets-store-csi-driver-demo",
 "environment": "csi-secrets-store-app",
 "authSecretName": my_secret.metadata["name"],
 "authSecretNamespace": my_secret.metadata["namespace"],
 "secrets": '- secretPath: "/"\n fileName: "hello"\n secretKey: "app.hello"\n',
 },
 },
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_pulumi_esc_provider]),
)

 secretProviderClass, err := apiextensions.NewCustomResource(ctx, "example-provider-pulumi-esc", &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("secrets-store.csi.x-k8s.io/v1"),
 Kind: pulumi.String("SecretProviderClass"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-provider-pulumi-esc"),
 Namespace: pulumi.String("default"),
 },
 OtherFields: k8s.UntypedArgs{
 "provider": pulumi.String("pulumi"),
 "parameters": pulumi.Map{
 "apiUrl": pulumi.String("https://api.pulumi.com/api/esc"),
 "organization": pulumi.String("dirien"),
 "project": pulumi.String("esc-secrets-store-csi-driver-demo"),
 "environment": pulumi.String("csi-secrets-store-app"),
 "authSecretName": mySecret.Metadata.Name().Elem(),
 "authSecretNamespace": mySecret.Metadata.Namespace().Elem(),
 "secrets": pulumi.String(`- secretPath: "/"
 fileName: "hello"
 secretKey: "app.hello"
`),
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiPulumiEscProvider}))
 if err != nil {
 return err
 }

 var secretProviderClass = new Pulumi.Kubernetes.ApiExtensions.CustomResource("example-provider-pulumi-esc", new SecretProviderClassArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "example-provider-pulumi-esc",
 Namespace = "default",
 },
 Spec = new SecretProviderClassSpecArgs
 {
 Provider = "pulumi",
 Parameters = new InputMap<object>
 {
 { "apiUrl", "https://api.pulumi.com/api/esc" },
 { "organization", "dirien" },
 { "project", "esc-secrets-store-csi-driver-demo" },
 { "environment", "csi-secrets-store-app" },
 { "authSecretName", mySecret.Metadata.Apply(metadata => metadata.Name) },
 { "authSecretNamespace", mySecret.Metadata.Apply(metadata => metadata.Namespace) },
 { "secrets", "- secretPath: \"/\"\n fileName: \"hello\"\n secretKey: \"app.hello\"\n" }
 },
 },
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiPulumiEscProvider } });

We can check that the secret was successfully synchronized by running:

kubectl get secretproviderclasses example-provider-pulumi-esc

You might also like:

Step 3: Deploy an Application and Mount the Secret

Now, we can deploy an application that references the secret from the Kubernetes cluster. I am going to use busybox that reads mounted file in.

const deployment = new k8s.apps.v1.Deployment("example-provider-pulumi-esc", {
 metadata: {
 name: "example-provider-pulumi-esc",
 namespace: "default",
 labels: {
 app: "example-provider-pulumi-esc",
 },
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: {
 app: "example-provider-pulumi-esc",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "example-provider-pulumi-esc",
 },
 },
 spec: {
 containers: [
 {
 name: "client",
 image: "busybox:latest",
 command: ["sh", "-c"],
 args: [
 `set -eux
 ls /run/secrets
 find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo "$(cat "{}")"'
 tail -f /dev/null`,
 ],
 volumeMounts: [
 {
 name: "data",
 mountPath: "/run/secrets",
 },
 ],
 },
 ],
 volumes: [
 {
 name: "data",
 csi: {
 driver: "secrets-store.csi.k8s.io",
 readOnly: true,
 volumeAttributes: {
 secretProviderClass: secretProviderClass.metadata.name,
 },
 },
 },
 ],
 },
 },
 },
});

export const deploymentName = deployment.metadata.name;

deployment = k8s.apps.v1.Deployment(
 "example-provider-pulumi-esc",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 namespace="default",
 name="example-provider-pulumi-esc",
 labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 spec=k8s.apps.v1.DeploymentSpecArgs(
 replicas=1,
 selector=k8s.meta.v1.LabelSelectorArgs(
 match_labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 template=k8s.core.v1.PodTemplateSpecArgs(
 metadata=k8s.meta.v1.ObjectMetaArgs(
 labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 spec=k8s.core.v1.PodSpecArgs(
 containers=[
 k8s.core.v1.ContainerArgs(
 name="client",
 image="busybox:latest",
 command=["sh", "-c"],
 args=[
 "set -eux\nls /run/secrets\nfind /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo \"$(cat \"{}\")\"'\ntail -f /dev/null",
 ],
 volume_mounts=[
 k8s.core.v1.VolumeMountArgs(
 name="data",
 mount_path="/run/secrets",
 ),
 ],
 ),
 ],
 volumes=[
 k8s.core.v1.VolumeArgs(
 name="data",
 csi=k8s.core.v1.CSIVolumeSourceArgs(
 driver="secrets-store.csi.k8s.io",
 read_only=True,
 volume_attributes={
 "secretProviderClass": secret_provider_class.metadata[
 "name"
 ],
 },
 ),
 ),
 ],
 ),
 ),
 ),
)

pulumi.export("deploymentName", deployment.metadata["name"])

 deployment, err := appv1.NewDeployment(ctx, "example-provider-pulumi-esc", &appv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-provider-pulumi-esc"),
 Namespace: pulumi.String("default"),
 Labels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Spec: &appv1.DeploymentSpecArgs{
 Replicas: pulumi.Int(1),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("client"),
 Image: pulumi.String("busybox:latest"),
 Command: pulumi.StringArray{
 pulumi.String("sh"),
 pulumi.String("-c"),
 },
 Args: pulumi.StringArray{
 pulumi.String(`set -eux
 ls /run/secrets
 find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo "$(cat "{}")"'
 tail -f /dev/null`),
 },
 VolumeMounts: &corev1.VolumeMountArray{
 &corev1.VolumeMountArgs{
 Name: pulumi.String("data"),
 MountPath: pulumi.String("/run/secrets"),
 },
 },
 },
 },
 Volumes: &corev1.VolumeArray{
 &corev1.VolumeArgs{
 Name: pulumi.String("data"),
 Csi: corev1.CSIVolumeSourceArgs{
 Driver: pulumi.String("secrets-store.csi.k8s.io"),
 ReadOnly: pulumi.Bool(true),
 VolumeAttributes: pulumi.StringMap{
 "secretProviderClass": secretProviderClass.Metadata.Name().Elem(),
 },
 },
 },
 },
 },
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretProviderClass}))
 if err != nil {
 return err
 }

 var deployment = new Pulumi.Kubernetes.Apps.V1.Deployment("example-provider-pulumi-esc", new Pulumi.Kubernetes.Types.Inputs.Apps.V1.DeploymentArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "example-provider-pulumi-esc",
 Namespace = "default",
 Labels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Spec = new Pulumi.Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Replicas = 1,
 Selector = new Pulumi.Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Template = new Pulumi.Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Spec = new Pulumi.Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 Containers =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "client",
 Image = "busybox:latest",
 Command =
 {
 "sh",
 "-c",
 },
 Args =
 {
 "set -eux\nls /run/secrets\nfind /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo \"$(cat \"{}\")\"'\ntail -f /dev/null",
 },
 VolumeMounts =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.VolumeMountArgs
 {
 Name = "data",
 MountPath = "/run/secrets",
 },
 },
 },
 },
 Volumes =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.VolumeArgs
 {
 Name = "data",
 Csi = new Pulumi.Kubernetes.Types.Inputs.Core.V1.CSIVolumeSourceArgs
 {
 Driver = "secrets-store.csi.k8s.io",
 ReadOnly = true,
 VolumeAttributes =
 {
 { "secretProviderClass", secretProviderClass.Metadata.Apply(metadata => metadata.Name) },
 },
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions { DependsOn = { secretProviderClass } });

After deploying the stack, you can get the logs of the busybox pod to see that the secret was successfully mounted:

NAME=$(kubectl get pods -o name | grep example-provider-pulumi-esc | cut -d'/' -f2)
kubectl logs $NAME

You should see the following output:

+ ls /run/secrets
hello
+ find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*'
+ xargs -t -I '{}' sh -c 'echo "$(cat "{}")"'
sh -c echo "$(cat "/run/secrets/hello")"
world
+ tail -f /dev/null

Or you can exec into the pod and check the content of the mounted secret:

kubectl exec -it $NAME -- cat /run/secrets/hello

You should see the following output:

world

Step 4: Clean Up

After you are done with the demo, you can clean up the resources by running:

pulumi destroy

Conclusion

The Secret Store CSI Driver is a great option for managing secrets in a cloud-native way when the use of Kubernetes secrets is not an option due to enhanced security requirements.

Below is a quick comparison of the External Secrets Operator and the Secrets Store CSI Driver to help you decide which one is the best fit for your use case. As always, the best way to find out is to try both solutions and see which one fits your requirements best.

The good part is, whatever you choose, you can use Pulumi ESC to manage your secrets and avoid any secret sprawl in your organization as both solutions are supported by Pulumi ESC.

Comparison of the External Secrets Operator and the Secrets Store CSI Driver

Feature/Aspect	External Secrets Operator	CSI Secrets Driver
Primary Use Case	Synchronizing external secrets into Kubernetes as native secrets.	Mounting secrets directly to pods as files or environment variables.
Integration with External Secret Stores	Supports multiple external secret stores like AWS Secrets Manager, Google Secret Manager, Vault, Azure Key Vault, etc.	Primarily supports integrations defined by the CSI specification (e.g., Vault, AWS, Azure Key Vault, Pulumi ESC).
Mechanism for Secrets Delivery	Secrets are pulled and stored as Kubernetes `Secret` resources.	Secrets are mounted directly into pods as files or injected as environment variables.
Kubernetes Resource Requirements	Requires a custom Kubernetes resource (`ExternalSecret`) and the External Secrets Operator.	Relies on the Kubernetes CSI (Container Storage Interface) standard and requires a specific CSI driver.
Secret Rotation	Supports automatic rotation by polling the external secret store at defined intervals.	Supports rotation, but the application needs to handle reloading mounted secrets.
Security Considerations	Secrets are stored temporarily in Kubernetes as native secrets, which may pose a risk if not properly managed.	Secrets are mounted directly and not stored as native Kubernetes secrets, reducing exposure but relying on file permissions.
Customization and Features	Highly customizable with fine-grained mappings and transformation options.	Limited customization, focused on mounting secrets as-is.
Application Compatibility	Applications access secrets as Kubernetes secrets (e.g., via env variables or volumes).	Applications access secrets directly via mounted files or injected env variables.
Performance Considerations	Potential slight delay due to syncing secrets to Kubernetes secrets.	Faster access as secrets are directly mounted.
Community and Ecosystem	Supported by a growing community; integrates well with diverse secret management solutions.	Part of the Kubernetes ecosystem; supports CSI-compliant secret storage solutions.
Best Use Cases	Ideal for scenarios where secrets need to be accessed as Kubernetes native secrets.	Best for applications requiring direct access to secrets without intermediate storage in Kubernetes secrets.

You can find detailed information about all current available Pulumi ESC Kubernetes integrations in the Pulumi ESC documentation.

Let us know what you think about the Secret Store CSI Driver and how you are managing secrets in your Kubernetes clusters. We would love to hear your feedback and experiences.

Try Pulumi for Free

Announcing AWS Systems Manager - Parameter Store Support for Pulumi ESC

Sean Yeh — Mon, 16 Dec 2024 12:00:00 -0600

We are super excited to announce integration support for AWS Systems Manager - Parameter Store within Pulumi Environments, Secrets, and Configuration (ESC). Parameter Store is a popular managed service by AWS for storing and managing secrets and other configuration, and its integration with ESC has been highly requested among the community.

With the AWS Parameter Store ESC Provider, development teams can now import their AWS Parameter Store parameters as environment variables or as Pulumi Config in their workflows, all in a secure and convenient way.

As we continue to expand our roster of integrations, it’s easier now than ever to manage and use secrets and configuration all in one place by using ESC. If you or your team are users of AWS Parameter Store, say goodbye to managing multiple configuration sources and give this provider a try!

How does it work?

Add parameters to AWS Parameter Store.
Use aws-login to authenticate with AWS.
Define a path name, and enter the parameter name you want to retrieve. This will be imported into the path name that you have defined.

AWS Parameter Store provider syntax:

values:
 params:
 fn::open::aws-parameter-store:
 region: us-west-1
 login: ${aws.login}
 get:
 myKey:
 name: /myNamespace/myKey
 myKeyByVersion:
 name: /myNamespace/myKey:1
 myKeyByVersionLabel:
 name: /myNamespace/myKey:stable
 secureKey:
 name: /myNamespace/secureKey
 decrypt: true

Demo

Now that we know how to use the Pulumi ESC Parameter Store provider, let’s use it to pull some parameters!

Setup OpenID Connect for AWS. We will use OIDC to authenticate with AWS.
Store some parameters in Parameter Store. In the demo, we will store a String /test/api-endpoint and a SecureString /test/api-key
Create a new Pulumi ESC Environment with the following definition. Make sure to replace the OIDC roleArn with your own.

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789:role/esc-oidc
 sessionName: pulumi-environments-session
 params:
 fn::open::aws-parameter-store:
 region: us-west-1
 login: ${aws.login}
 get:
 api-endpoint:
 name: /test/api-endpoint
 api-key:
 name: /test/api-key
 decrypt: true
 environmentVariables:
 API_ENDPOINT: ${params.api-endpoint}
 API_KEY: ${params.api-key}

Open the environment to ensure we are able to successfully pull the credentials from AWS Parameter Store
Run pulumi env open <project/environment> --format shell to view the exported environment variables from this ESC environment.
Use pulumi env run to run any command with these environment variables. For example, we can now run pulumi env run <project/environment> -- <your command>

pulumi env run passes the configuration stored under the environmentVariables section for use with the provided command, without ever storing them locally on your machine.

Conclusion

We are excited to expand our roster of integrations with the new AWS Parameter Store ESC Provider, providing developers more flexibility and choices. We look forward to seeing what you will create next using Pulumi ESC!

Visit the AWS Parameter Store Provider Page for more detailed documentation on this provider.
Visit our Pulumi ESC docs to learn more about Pulumi ESC and its supported integrations

As always, we deeply value your insights. Your feedback is instrumental in helping us refine and enhance our solutions to better align with your needs.

Pulumi ESC: Discovering Environment Imports

Sean Yeh — Fri, 13 Dec 2024 00:00:00 -0700

Managing secrets and configuration across multiple environments and stacks can easily become complex, leading to duplicated values, inconsistencies, and security risks. Pulumi ESC solves this with composable environments via imports, allowing you to define configuration once and reuse it organization-wide. Now, with the new capabilities to discover environment imports, you gain unprecedented visibility and control, simplifying the management of even the most complex infrastructure and applications.

Understanding Pulumi ESC Imports

Pulumi ESC lets you define secrets and configuration as code collections called environments. These environments can import other environments, creating a hierarchical structure that promotes reusability, and reduces duplication and sprawl. You can centralize secrets and configuration within ESC and use them across Pulumi IaC Stacks, Pulumi Insights Accounts, and various other platforms through our integrations.

By using version tags, imports can target specific versions of an environment. This locks down the configuration used by dependents, ensuring stability and preventing unintended changes from rolling out to your critical environments.

What’s New?

We’ve added powerful discoverability features to give you clear insight into your environment dependencies:

New “Imported By” tab: Easily understand downstream dependencies by clicking the “Imported By” tab for each environment. This tab shows all environments and stacks that import a specific environment, including the specific tagged version used. Filter and sort by imported versions and downstream resources to narrow in on the resources that will be affected by a change you are trying to make.
Warnings for deletions and tag movements: Pulumi ESC now warns you about the potential impact of deleting an environment or modifying version tags before you make the change, helping you avoid disruptions.
Enhanced environment page: The Environments page now clearly indicates if an environment is used by other environments, stacks, or insights accounts, providing valuable relationship context at a glance.
Pulumi IaC stack visibility: Environment import information is also available on the Stacks page and within individual stack overview pages.

Conclusion

Pulumi ESC’s import feature is a paradigm shift in how you manage your secrets and configuration. The new suite of import discoverability features gives you the confidence to manage complex deployments with ease. Explore the new features and share your feedback through our Community Slack or GitHub Repo.

Secret Rotation with Pulumi ESC

Claire Gaestel — Tue, 10 Dec 2024 15:43:02 -0800

Pulumi ESC now natively supports secrets rotation that makes secrets lifecycle management much easier. Check out the launch blogpost and docs.

Managing secrets in modern cloud applications can be challenging, particularly when it comes to rotation policies. While dynamic secrets (like AWS IAM temporary credentials) handle this automatically, many systems still rely on static secrets that require periodic rotation.

Static secrets, like database passwords or API keys, should be rotated regularly to maintain security, and services depending on these secrets need time to transition to new credentials to avoid downtime. This makes rotating credentials error-prone, and often forgotten.

In this post, we’ll explore an approach for automating static secret rotation using Pulumi ESC combined with Pulumi IaC.

A pattern for extending ESC

We can take advantage of ESC’s integration with Pulumi Deployments to create an ergonomic way of managing rotation schedules ourselves.

We’ll start by defining a custom declarative configuration format for managing a rotation schedule:

# we'll create our own "extended" fn config for ESC
xfn::pulumi-scheduled-update:
 # stack reference to a pulumi program that manages a rotated secret
 stack: "example-rotator-stack-reference/dev"
 # schedule that will drive an automatic scheduled deployment
 scheduleCron: "0 0 * * 0"
 # trigger a manual unscheduled rotation whenever this value changes
 trigger: "break-glass"

Using webhooks, we’ll monitor when the environment is updated, then search for these configuration blocks, and use them to drive scheduled deployments of a stack that rotates credentials. Using ESC’s composition primitives, we can combine this rotation schedule config with a fn::open::pulumi-stacks provider to read the current secret value from the rotator stack to provide to our application.

Here is how the solution fits together: Let’s break down each component and see how they work together to solve our rotation challenges.

The Rotator: Managing Credential Lifecycles

At the heart of our solution is a generic Rotator component that manages credential pairs. It’s designed to handle any type of static secret while ensuring zero-downtime rotations. The way it does this is by maintaining two versions of each secret: “current” and “previous”. Each time the stack is deployed, “current” is replaced with a newly provisioned credential, the old value is demoted to “previous”, and the old previous credential is decommissioned. Crucially, the demoted secret remains valid, which allows consuming services enough time to switch over to the new secret.

const creds = new Rotator("rotating-creds", {
 // rotate credentials on every deployment.
 trigger: Date().toString(),
 // create a set of equivalent credentials that will be rotated between.
 // a particular credential should be replaced whenever `trigger` is updated.
 construct: (name, trigger) => {
 // in this example we're just creating passwords,
 // but these could be mysql users or anything else.
 return new random.RandomPassword(name, {
 length: 10,
 keepers: {trigger}
 })
 },
})

// export when the last rotation happened
export const lastUpdate = creds.lastUpdate.apply(date => date.toISOString());
// export the currently active credential, which will be imported by the downstream ESC environment.
export const current = creds.current.result;

The Scheduler: Orchestrating Rotations

The scheduler component acts as an orchestrator, watching an ESC environment for updates and managing deployment schedules of the rotator stacks. Whenever a change to the environment is saved, a webhook invokes the scheduler, which parses the environment definition to find xfn::pulumi-scheduled-update configuration blocks. Based on these schedule configurations, it creates scheduled deployments for the referenced rotator stacks automatically.

Bringing It Together: Environment Configuration

Once wired together, the magic happens in the ESC environment configuration, where we compose a rotation schedule with dynamic credential retrieval, creating a rotated credential that is automatically kept up to date:

values:
 db-credentials:
 stack: database-credential-rotator/dev
 schedule:
 xfn::pulumi-scheduled-update:
 stack: ${db-credentials.stack}
 scheduleCron: "0 0 * * 0" # weekly rotation
 trigger: "break-glass" # manual trigger
 secrets:
 fn::open::pulumi-stacks:
 stacks:
 credentials:
 stack: ${db-credentials.stack}

 api-keys:
 stack: api-key-rotator/dev
 schedule:
 xfn::pulumi-scheduled-update:
 stack: ${api-keys.stack}
 scheduleCron: "0 0 1 * *" # monthly rotation
 secrets:
 fn::open::pulumi-stacks:
 stacks:
 keys:
 stack: ${api-keys.stack}

 environmentVariables:
 DB_CONNECTION_STRING: ${db-credentials.secrets.credentials.current}
 API_KEY: ${api-keys.secrets.keys.current}

This configuration demonstrates how our custom extension is able to seamlessly integrate with ESC to manage multiple rotating secrets with different schedules declaratively, co-located with dynamic retrieval of the latest credentials from the rotator stack outputs. Applications consuming this environment will automatically receive the latest credentials without any additional configuration.

Trying it out

Open the example environment and take note of the current secrets:

❯ pulumi env open esc-rotation-demo/test db-credentials.secrets
{
"credentials": {
"current": "OBA=wxS:VT",
"lastUpdate": "2024-11-26T17:49:03.000Z",
"previous": "MUgPXkJ+kE"
}
}

Now lets force a rotation by changing the manual trigger:

❯ pulumi env set esc-rotation-demo/test \
db-credentials.schedule.xfn::pulumi-scheduled-update.trigger \
break-glass2

After a few minutes we can see that the secret has rotated successfully! We can also observe that the previous secret remains valid, giving currently deployed consumers time to update to the new credentials.

$ pulumi env open esc-rotation-demo/test db-credentials.secrets
{
"credentials": {
"current": "X2WSmF3+29",
"lastUpdate": "2024-11-26T17:57:05.000Z",
"previous": "OBA=wxS:VT" <---- old "current" has been demoted
}
}

Conclusion

Secret rotation doesn’t have to be a manual, error-prone process. By leveraging Pulumi ESC and this component architecture, we can automate and streamline the rotation of static secrets while maintaining system stability and security. The solution is flexible enough to handle various types of secrets while being simple to configure and maintain. This approach offers several advantages:

Automated: Set-and-forget secret rotation
Zero-downtime: Smooth transitions between credentials
Flexible: Works with any type of static secret
Declarative: Configuration through ESC environments
Auditable: Clear tracking of rotation history
Scalable: Easy to manage across multiple environments

The complete example is available for your perusal here.

This pattern of extending ESC through configuration-driven components is a powerful technique that can be applied to other use cases. We’re excited to see what else you come up with 🙂

How Secrets Sprawl is Slowing You Down—And What to Do About It.

Engin Diri — Mon, 28 Oct 2024 12:40:10 +0200

Only a few things are certain in the lives of developers and DevOps engineers: taxes, yearly performance reviews, and secret sprawl. While the first two are inevitable, the last one is something that can be managed.

As we keep adding new cloud resources and releasing new applications, the number of secrets we need to manage keeps growing: passwords, API keys, certificates, and more. And as if this isn’t enough, we need to manage secrets across different systems and environments with different teams that need to access them; we end up with duplicates.

Let’s have a look at a typical environment and see what teams use: Everything container-related may use Docker secrets or Kubernetes secrets. There is also a high chance that they might use cloud-specific secrets managers like Azure Key Vault, AWS Secrets Manager, or Google Secret Manager. In addition to these, there is a high chance that they might use HashiCorp Vault or some enterprise-grade secrets manager like CyberArk.

There are still a lot of organizations that store their secrets (like the API key for a cloud service) in environment variables to configure their applications on virtual machines, containers, or serverless functions. And as if that weren’t risky enough, some developers still hardcode sensitive information in their application code. Or use plain unencrypted text files such as .env files and share them inside the team. To be fair, this is not something that is done on purpose, and most teams realize the risks of these practices. But the problem is that they have not found a better way to manage their secrets that won’t slow them down and impact their productivity.

As the IT infrastructure grows, it becomes increasingly challenging for organizations to secure, organize, and distribute their secrets. This is where secrets sprawl comes into play. Secrets sprawl is the uncontrolled and unmanaged distribution of secrets across different systems and environments.

Modern architectures are adding to the problem of secret sprawl as modern applications are more distributed and consist of smaller building blocks with a growing number of different technologies.

While teams are ensuring that each of their applications has everything it needs to run, it’s also easy for them to lose track of the secrets they are using. Everything can be fine until they need to rotate a secret or revoke a compromised one. Then they need to go through all the systems and environments to find and update the secret. Tracking down all the places where a secret is used can be a time-consuming and error-prone process with a high risk of missing some places.

So what are some of the pain points of secret sprawl, and how can we overcome them to regain our productivity?

Pain Points of Secrets Sprawl

With secrets sprawled all across our teams (development and operations) and infrastructure, we face several points:

It is challenging to keep track of all the secrets and where they are used. When we need, for example, to find a specific secret, we need to go through different secret managers, code, and, in the worst case, the local files of one of the developers.

Do we need to rotate a secret? Now we need to go through all the systems and environments to find and update every duplicated secret.
Security risk is another pain point to mention. When a secret is spread throughout the organization, it’s hard to manage access control and enforce specific access policies.

Security best practices also recommend that secrets should be rotated regularly. But with secrets sprawled all over the place, it makes it a lot of work to rotate them. Time we could spend on more important tasks like developing new features.
Secrets sprawl makes it extremely challenging to distribute secrets. Having a secret, for example, in a cloud provider’s secret manager is cool and great, but what if we need to use it in a Kubernetes cluster? What is the process for that?

Also, having secrets in multiple places means there are different owners for them. These secrets are now some kind of gatekeeping by the team that owns them. This makes it hard for other teams to access them when they need them. And even if they get access, they will then be mostly shared in plain text as the other team does not have access to the secret manager itself.

As we can see, secret sprawl is a real problem that can slow down our teams and impact our productivity. But it also affects how fast an organization can scale its development and operations teams.

With an “organic” way of managing secrets and no real transparency, it’s hard to keep up with the pace of the business, and secret sprawl becomes a major bottleneck. Additionally, that growth means a whole bunch of new secrets that need to be managed.

Rinse and repeat.

But behold, there is a solution to this problem. And it’s called centralized secrets management.

Centralized Secrets Management against Secrets Sprawl

The cure for secrets sprawl is centralized secrets management. When you centralize your secrets in one place, you know where they are and who has access to them.

This makes it easier to manage access control and enforce specific access policies. It also makes it easier to rotate secrets as you only need to do it in one place.

Have a look at an example scaling of an organization or team:

When the company is small, the team decides to use Kubernetes to deploy their applications, and they store their secrets in Kubernetes secrets. This first step is already a centralization of secrets as it tightens access to the secrets and makes it easier to audit them. Maintaining an encrypted secrets store and centralizing the secrets in one place is a good start and ensures the organization is following a consistent threat protection policy.

As the company grows, one DevOps engineer is responsible for multiple Kubernetes clusters. In these cases, individual users can’t enforce their own application-wide policies, meaning secrets are better managed and controlled.

But what if the company grows even more and the team decides to use multiple cloud providers? There are many reasons for this, like costs, different service offerings, and better disaster recovery when using multiple cloud providers.

Sticking to the built-in secrets managers of the cloud providers is not a good idea as it will lead to secrets sprawl.

You might also like:

Why Pulumi ESC is the right choice for centralized multi-cloud secrets management

Pulumi ESC is the right choice for centralized multi-cloud capable secrets management as it reduces secrets sprawl by centralizing secrets storage with a lot of integrations for a wide range of platforms.

Pulumi ESC delivers a seamless integration so you can still use your existing secrets managers like Azure Key Vault, AWS Secrets Manager, Google Secret Manager, or HashiCorp Vault. Secrets are automatically synced to Pulumi ESC, saving developers a lot of time so they can focus on their main tasks like developing and building new products and features.

Pulumi ESC is a completely managed service that again saves time and resources for the organization as they don’t need to worry about the infrastructure, maintenance, and support of their self-hosted secrets manager.

Wrapping Up

Every organization faces the risk of secrets sprawl as they grow and add more cloud resources and applications, which they may not even be aware of.

Multi-cloud scenarios make it even more challenging to manage secrets across different systems and environments.

As teams have to juggle between delivering and building new products and features, managing secrets in a secure and efficient way becomes a major bottleneck on productivity and a major security risk. A ticking time bomb that can explode at any time.

Secrets managers can help you reduce secrets sprawl by centralizing secrets storage and providing a seamless way to use, change, and distribute secrets across different systems and environments. Independent of the size of your organization, a startup or a large enterprise, it’s never too late to start managing your secrets in a secure and efficient way.

Pulumi ESC is on the way to becoming the best choice to solve the problem of secrets sprawl. Making both developers and security teams happy by providing a secure and efficient way to manage secrets across different systems and environments.

Start using Pulumi ESC and ESO today by creating an account on the Pulumi Cloud Console and begin managing your secrets in a secure and efficient way.

Create an Account

Pulumi ESC and External Secrets Operator: The Perfect Solution for Today's Cloud-Native Secret Management

Engin Diri — Fri, 04 Oct 2024 00:00:00 +0000

Managing secrets in a cloud-native environment can be challenging, but it is crucial for ensuring the security and integrity of any application or infrastructure. We encounter a lot of different types of secrets, from API keys, database passwords, and certificates to tokens and passwords. These secrets need to be stored securely and accessed by different services in a secure way without exposing any sensitive information to unauthorized users.

Here is where Pulumi ESC and External Secrets Operator come into play by providing a secure and efficient solution for cloud-native secret management.

Pulumi ESC (Environments, Secrets, and Configuration)

Pulumi ESC is our new secrets management & orchestration service for environments, secrets, and configurations. It can be used as part of your Pulumi IaC (Infrastructure as Code) projects using pulumiConfig to expose the stored values to your Pulumi stacks.

But it can also be used as a standalone service to manage secrets and configurations. Pulumi ESC provides support for applications through its dedicated SDKs built for different programming languages. It can also be managed from the command line with the pulumi env commands in the Pulumi CLI, which can inject secrets and configuration as a context for your CLI tools (e.g. terraform).

For more information on Pulumi ESC, you can check out the official documentation.

External Secrets Operator (ESO)

The Open Source project External Secrets Operator (ESO) is a Kubernetes operator that helps manage secrets in a highly secure and scalable way. ESO synchronizes secrets stored in external secret management systems like Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and more into Kubernetes secrets. This approach allows us to control secret sprawl and centralize the management of secrets in a single place while providing a secure and controlled way to access them.

Why Kubernetes secrets are not enough?

Kubernetes secrets are a great way to store sensitive information in a Kubernetes cluster. Kubernetes secrets are created and stored as a Kubernetes resource object and then used by pods or other resources inside the cluster.

Here is an example of a Kubernetes secret manifest file:

apiVersion: v1
kind: Secret
metadata:
 name: mysecret
type: Opaque
data:
 username: YWRtaW4= # admin
 password: YWRtaW4= # admin

or using kubectl:

kubectl create secret generic mysecret --from-literal=username=admin --from-literal=password=admin

Kubernetes secrets have several limitations. Here are the top ones:

They are stored by default in etcd in base64 encoded format, which is not secure.
They are created either manually by using kubectl commands or by using a manifest file, making them hard to manage at scale.
Hard to manage and synchronize secrets across different environments and clusters.
There is no default way to rotate secrets automatically.

ESO shines by tackling all the challenges listed and providing a way to manage secrets in a more secure and efficient manner. Benefits include:

Mitigate security risks as secrets are stored in a secure external secret management system like Pulumi ESC, which provides more robust security features like encryption, access control, and auditing.
Eliminate manual management of secrets by synchronizing them from external secret management systems into the Kubernetes cluster, reducing the need for manual intervention and minimizing the risk of human error.
ESO provides a unified interface, making it easy to manage secrets across clusters and multi-cloud environments.
ESO integrates with external secret management systems which ensure that access to secrets is tracked and an audit trail is maintained. This is crucial for security compliance and adhering to regulatory standards.
Automate secrets rotation when the external secret management system has support it.

External Secrets Operator architecture

The central part of the ESO architecture is the extension of the Kubernetes API with a new Custom Resource Definition (CRD), which defines where the secrets are stored and how they are synchronized into Kubernetes secrets. The ESO controller then retrieves the secrets from the external secret management system and creates or updates the Kubernetes secrets. Every time a secret is updated in the external secret management system, the ESO controller will synchronize the secret in the Kubernetes cluster and update the secret accordingly.

Core resources of ESO

(Cluster)SecretStore: Defines the external secret management system where the secrets are stored. It contains the necessary information to connect to the external secret management system, such as the endpoint, authentication credentials, and other configuration options. The SecretStore is namespaced while the ClusterSecretStore is cluster wide, which allows you cross-namespaced access to the secrets.
(Cluster)ExternalSecret: Defines the secret that needs to be synchronized into the Kubernetes cluster. It contains the reference to the SecretStore and the secret key and how the data should be transformed and saved as a Kubernetes secret.
PushSecret: With the PushSecret resource, you can push secrets from the Kubernetes cluster to the external secret management system. The Pulumi ESC provider for ESO supports this feature, which allows you to push secrets from the cluster into Pulumi ESC. This feature is useful when you create secrets through other means and want to synchronize them into Pulumi ESC to manage them centrally.

How to Use Pulumi ESC and ESO Together

To demonstrate how to use Pulumi ESC and ESO together, we will store some secrets and configuration values in Pulumi ESC, then synchronize them into a Kubernetes cluster using ESO.

To follow along, you’ll need the following prerequisites:

A Kubernetes cluster (I will be using a local KinD cluster, but you can use any Kubernetes cluster)
Pulumi CLI installed. You can use the pulumi env commands to manage secrets and configurations.
kubectl CLI installed for some debugging

Step 1: Install ESO and Link to Pulumi ESC

We will use Pulumi with the pulumi-kubernetes provider to deploy ESO into the Kubernetes cluster. The power of Pulumi ESC integration comes in two ways: First, to supply the Pulumi access token to ESO via Pulumi IaC, allowing the new ESO instance to access Pulumi ESC later. Second, to provide configuration and secrets from Pulumi ESC, via ESO, to the PodInfo application within the Kubernetes cluster.

pulumi new kubernetes-typescript --name eso-pulumi-esc

You should see the following output:

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

Project description (A minimal Kubernetes TypeScript Pulumi program):
Created project 'eso-pulumi-esc'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
Stack name (dev):
Created stack 'dev'

The package manager to use for installing dependencies npm
Installing dependencies...


added 310 packages, and audited 311 packages in 12s

44 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

Head to Pulumi Cloud Console and create a new Pulumi ESC project with the name eso-esc-demo:

And the environment eso-esc-dev:

In the editor add the following yaml into the Environment definition:

values:
 pulumiConfig:
 pulumi-pat:
 fn::secret: <your-pulumi-pat>

Don’t know how to create a PAT? Check out the official Pulumi documentation.

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/eso-esc-demo/eso-esc-dev

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-esc-demo/eso-esc-dev

Now, we need to link the Pulumi ESC project to the Pulumi IaC project. To do this, we need to add the following to your Pulumi.dev.yaml:

environment:
- eso-esc-demo/eso-esc-dev

Now that all the wiring is done, we can deploy the ESO instance into the Kubernetes cluster. Open the index.ts file and add the following code:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();

// Create a Kubernetes namespace
const ns = new k8s.core.v1.Namespace("external-secrets", {
 metadata: {
 name: "external-secrets",
 },
});

// Deploy a Helm release into the namespace
const externalSecrets = new k8s.helm.v3.Release("external-secrets", {
 chart: "external-secrets",
 version: "0.10.4", // Specify the version of the chart
 namespace: ns.metadata.name,
 repositoryOpts: {
 repo: "https://charts.external-secrets.io",
 },
});

// Deploy a secret into the namespace
const mySecret = new k8s.core.v1.Secret("my-secret", {
 metadata: {
 namespace: ns.metadata.name,
 name: "pulumi-access-token",
 },
 stringData: {
 "PULUMI_ACCESS_TOKEN": config.require("pulumi-pat"),
 },
 type: "Opaque",
});

// Export the namespace name
export const namespaceName = ns.metadata.name;

// Export the Helm release name
export const helmReleaseName = externalSecrets.name;

Deploy the stack by running:

pulumi up

And you should see that the secret was created in the Kubernetes cluster and the ESO instance was deployed successfully.

kubectl get secret -n external-secrets pulumi-access-token -o jsonpath='{.data.PULUMI_ACCESS_TOKEN}' | base64 -d

Step 2: Create an External Secret

Now, we will create a secret in the Pulumi ESC project and synchronize it into the Kubernetes cluster by creating an ESO reference to the external secret.

Create a new ESC environment called esp-esc-app in the eso-esc-demo ESC project.

Add the following YAML into the Environment definition:

values:
 app:
 hello: world
 hello-secret:
 fn::secret: world

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/eso-esc-demo/esp-esc-app

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-esc-demo/esp-esc-app

Either way, you should see following environment configuration in the Pulumi Cloud Console:

We can now create ClusterExternalSecret in the index.ts and reference the ESC secret.

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// cut for brevity

const crd = new k8s.apiextensions.CustomResource("cluster-secret-store", {
 apiVersion: "external-secrets.io/v1beta1",
 kind: "ClusterSecretStore",
 metadata: {
 name: "secret-store",
 },
 spec: {
 provider: {
 pulumi: {
 organization: pulumi.getOrganization(),
 project: "eso-esc-demo",
 environment: "esp-esc-app",
 accessToken: {
 secretRef: {
 name: mySecret.metadata.name,
 key: "PULUMI_ACCESS_TOKEN",
 namespace: mySecret.metadata.namespace,
 },
 },
 },
 },
 },
});

We can check that the secret was successfully synchronized by running:

kubectl get clustersecretstore secret-store -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

Step 3: Deploy an Application and Reference the Secret

Now, we can deploy an application that references the secret from the Kubernetes cluster. I am going to deploy the app called PodInfo, which is an application that exposes the pod information as an API. Add the following code to the index.ts:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// cut for brevity
const podInfo = new k8s.helm.v3.Release("podinfo", {
 chart: "podinfo",
 version: "6.7.0",
 namespace: "podinfo",
 createNamespace: true,
 repositoryOpts: {
 repo: "https://stefanprodan.github.io/podinfo",
 },
 values: {
 extraEnvs: [
 {
 name: "FROM_ESC_VIA_ESO",
 valueFrom: {
 secretKeyRef: {
 name: "esc-secret-store",
 key: "hello-secret",
 }
 }
 },
 {
 name: "FROM_ESC_VIA_ESO_2",
 valueFrom: {
 secretKeyRef: {
 name: "esc-secret-store",
 key: "hello",
 }
 }
 }
 ]
 },
});

const externalSecretPodInfo = new k8s.apiextensions.CustomResource("external-secret-podinfo", {
 apiVersion: "external-secrets.io/v1beta1",
 kind: "ExternalSecret",
 metadata: {
 name: "esc-secret-store",
 namespace: podInfo.namespace,
 },
 spec: {
 dataFrom: [
 {
 extract: {
 conversionStrategy: "Default",
 key: "app",
 }
 }
 ],
 refreshInterval: "1h",
 secretStoreRef: {
 kind: clusterSecretStore.kind,
 name: clusterSecretStore.metadata.name,
 }
 },
});

This will deploy the PodInfo application and create an ExternalSecret resource that references the secret from the Pulumi ESC project. We mount the secret into the pod as an environment variable by using the extraEnvs field in the Helm chart.

In a real-world application, you might use alternate ways to reference the secret, like mounting it as a volume or using fromEnv field in the Helm chart. It depends on your application and how you want to consume the secret.

After deploying the stack, you should see that the PodInfo application is running. To access the application, we can port-forward the service by running:

kubectl port-forward svc/podinfo 9898:9898 -n podinfo

Our PodInfo demo application provides a large selection of different endpoints to interact with. We are interested in the /env endpoint to check if the secret was successfully mounted into the pod. Let’s try that:

curl -s localhost:9898/env | grep FROM
 "FROM_ESC_VIA_ESO=world",
 "FROM_ESC_VIA_ESO_2=world",

And here we see secret values from the ESC project, proxied by ESO, and passed to the PodInfo application as environment variables. That means the secret was successfully mounted into the pod and the application is running as expected.

Step 4: Clean Up

After you are done with the demo, you can clean up the resources by running:

pulumi destroy

Conclusion

The combination of Pulumi ESC and External Secrets Operator provides a secure and efficient way to manage secrets in a cloud-native environment, improving security, efficiency, and compliance when consuming secrets in your Kubernetes cluster. By using Pulumi ESC, you can manage secrets and configurations in a centralized way, and with ESO, you now have a streamlined process to synchronize secrets into your Kubernetes cluster.

Start using Pulumi ESC and ESO today by creating an account on the Pulumi Cloud Console and begin managing your secrets in a secure and efficient way.

Create an Account

Pulumi - Automate, Secure, and Manage Everything You Run in the Cloud

Joe Duffy — Wed, 18 Sep 2024 09:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

At our fourth annual PulumiUP conference, we unveiled a new vision for Pulumi, going beyond infrastructure as code to a comprehensive product suite. Pulumi’s platform now consists of three core products: Pulumi IaC is open source infrastructure as code in any programming language; Pulumi ESC is centralized secrets management that scales; and Pulumi Insights delivers instant visibility, intelligence, and control over all of your infrastructure. Each product can be adopted independently but is better together. In this post, we will share more about how Pulumi is your trusted partner to automate, secure, and manage everything you run in the cloud.

Pulumi - One Unified Platform For All Your Infrastructure Needs

The Pulumi platform solves a broad set of problems that engineering leaders face in the modern cloud era. This spans cloud automation, security, and management, which aligns to our three products: Pulumi IaC, Pulumi ESC, and Pulumi Insights. Our unified approach ensures developers, infrastructure experts, and security teams can collaborate closely and ship faster with high confidence.

In addition to our three products, Pulumi’s platform incorporates three core capabilities that span the entire product suite. Pulumi CrossGuard is our policy as code engine which ensures security, compliance, and best practices; Pulumi Deployments is our deployment workflow technology that lets us orchestrate infrastructure-oriented tasks; and Pulumi Copilot is our generative AI technology that deeply understands the cloud and infrastructure management and operations.

You can see this visually depicted here:

We’ve now worked with 3,000 customers to help with their cloud journey and have listened to and solved the problems we regularly hear about. That includes Nvidia who is at the forefront of the AI revolution as well as BMW who is reimagining their entire engineering mindset in a cloud-first world, both leveraging Pulumi. Unity Games now ships 5x faster from code to cloud, and Tivity Health saved 71% of their cloud spend, automating everything and saving $6.5M annually.

Now let’s dive into the three products and what we announced at PulumiUP:

Pulumi IaC - Infrastructure as Code in Any Programming Language

Pulumi IaC is our flagship open source technology, and it powers virtually everything we do, even the new products. We have seen accelerated adoption of Pulumi IaC. We’ve had over 100M downloads, are about to crack 21K GitHub stars (give us a star and bump us over!), and have seen 167% the contributions of Terraform this past month and 300% of OpenTofu. Not only is Pulumi the most powerful IaC technology, it is becoming more popular and vibrant too.

We are always improving Pulumi IaC and today is no different. We launched a number of new and exciting IaC features. That includes using any Terraform provider, beyond the nearly 200 providers that Pulumi already supports. We also shipped a new Pulumi VS Code Extension that enables rich debugging and easier editing of your IaC programs. Pulumi Copilot can now diagnose and fix your IaC errors. And finally, we’ve improved our Kubernetes support with powerful controls over await logic for more robust cloud native deployments. And we’re not slowing down anytime soon.

Pulumi is open source, of course, but works best with Pulumi Cloud, the easiest, most secure, and most reliable way to adopt Pulumi IaC. Starburst saved 100 days per year by moving from DIY state management to Pulumi Cloud. By using Pulumi Cloud, you also gain access to the other products in this post. Try IaC now and choose the option that suits you best.

Pulumi ESC - Centralized Secrets Management and Orchestration that Scales

Pulumi ESC is a new approach to secrets and configuration management that is scalable, secure automatically with easy dynamic credentials, and tames sprawl with engineering best practices. ESC is not just a feature of our IaC product — it can be adopted entirely independently of it for application and infrastructure secrets. We initially launched ESC in preview nearly a year ago, and since then have had hundreds of customers and thousands of users try it out – many at production scale – who have given us amazing feedback. We are excited to announce that ESC is generally available.

Being generally available is a big deal to us, and means it is ready for robust and secure enterprise-grade workloads. New capabilities available at GA include a new projects system which lets you group your environments into familiar hierarchies. We have added versioning, tagging, and pinning, so that you can institute more robust change management practices. ESC can manage encrypted secrets directly, but we have always supported pulling secrets from other secret stores, like AWS Secrets Manager, Azure KeyVault, Google Secret Manager, and HashiCorp Vault. We have now added 1Password to the list. We now also support secrets sync, which lets you still edit your secrets straight from Pulumi ESC, but have them automatically mirrored back to the backing secrets store. Flexible webhooks now support custom workflows triggered by environment changes. The new Pulumi VS Code Extension helps you edit your environments in code. We’ve launched application SDKs for Node.js, Python, and Go, as well as a Kubernetes Operator to inject secrets into Kubernetes workloads.

We aim for you to “fall into the pit of success” with security, so that it is built-in and automatic, and ESC is a major enabler of that. Try out ESC now, or read more in the launch post.

Pulumi Insights - Visibility, Intelligence, and Controls Over All Infrastructure

Pulumi Insights already delivers visibility, intelligence, and controls over your infrastructure resources. We began with Insights a year and a half ago as a collection of IaC features. We added resource search with structured and natural language search syntax, to help you find your IaC infrastructure no matter where it is running. We added the ability to export your infrastructure data and slice and dice it from data tools like Snowflake. We also added new dashboards to help visualize your infrastructure at scale.

The big news with Pulumi Insights 2.0 is that we’re bringing these capabilities to all of your cloud infrastructure, no matter where it is, and no matter how it was created. You point Pulumi at your cloud accounts for any of nearly 200 supported clouds, and you get an instant asset inventory of everything you have. This is true even for resources created by Terraform, CloudFormation, Azure Resource Manager, cloud CLIs and SDKs, or even by clicking in your cloud console. An improved resource view gives you pivot-table-like capabilities so you can understand everything you have, and new graph visualizations with inferred dependencies help you to make sense of how things relate. Search and Pulumi Copilot work over discovered cloud resources, letting you explore and ask questions interactively. Pulumi Insights leverages Pulumi CrossGuard, our policy as code technology, and can tap into any existing IaC policies to find issues pertaining to security, compliance, cost, and best practices. Pulumi’s compliance-ready policy packs just work. Insights can also leverage CrossGuard’s ability to automate remediations so that Insights not only tells you about your issues, but helps you fix them with the click of a button. Finally, Insights is better together with Pulumi IaC, and enables you to migrate any resources in the cloud to Pulumi IaC – improving the robustness and built-in security of your operations – with a fully interactive import experience.

This marks the start of an ambitious long-term vision for Pulumi Insights. Some of what was announced is actively shipping while other pieces will ship in the coming weeks. Learn more in the launch blog and, get started with Pulumi Insights 2.0.

Thank You

I am incredibly thankful to our community, 3,000 customers, and the partners and speakers who made PulumiUP 2024 a reality. Your feedback and continued belief in Pulumi has fueled the innovations we’re shipping today and into the future.

All of the talks are streaming live here and will be up on YouTube afterwards.

We can’t wait to hear what you think about the new vision for Pulumi’s platform as well as the new IaC, ESC, and Insights capabilities announced at PulumiUP 2024. As always, you can try Pulumi for free here.

Happy cloud spelunking.

-Joe

Announcing the General Availability of Pulumi ESC - Welcome to a New Era of Secrets Management

Arun Loganathan — Wed, 18 Sep 2024 08:59:00 +0000

At Pulumi, we’ve been at the forefront of revolutionizing infrastructure management and DevOps processes. We’ve built the industry’s most comprehensive secrets management solution for IaC, securing production cloud credentials and secrets for countless organizations. Now, we’re taking that expertise and applying it to the broader challenges of managing secrets at scale with an open ecosystem approach.

Pulumi Environments, Secrets, and Configuration (Pulumi ESC), launched in preview in October 2023, is now generally available.

This next-generation secrets management and orchestration service is built with a robust software engineering approach, designed to bring security and simplicity to even the most complex secrets management needs.

Challenges in Secrets Management

In today’s cloud-native world, secrets are the keys to the cloud estate, but managing them effectively is a growing challenge. Modern applications rely on a complex web of secrets, but traditional approaches to secrets management are struggling to keep pace. This complexity creates major challenges for security and engineering teams:

Secrets Sprawl: Secrets scattered across numerous systems and platforms make it nearly impossible to track their usage, enforce consistent security policies, and prevent unauthorized access, significantly increasing the likelihood of a breach.
Long-Lived Credentials: Storing long-lived credentials, especially locally, creates a significant security liability. If these credentials are compromised, they can grant attackers prolonged access to sensitive systems.
The Trade-off Between Speed and Security: Many security measures designed to safeguard secrets can inadvertently impede development velocity. Complex storage and retrieval processes create unnecessary friction, leading to frustration and delays.
Insecure .env File Sharing: Developers often copy/paste secrets across many different files, including formats like .env files stored locally on their machine or shared through insecure channels (like Slack), creating blind spots for security teams and increasing the risk of data leaks.

Introducing Pulumi ESC: A Modern Approach to Environments, Secrets, and Configuration

Pulumi ESC is the modern secret management platform designed to streamline and secure how you manage environments, secrets, and configuration across your entire organization. Built with flexibility and security at its core, Pulumi ESC empowers you to:

Centralize and Simplify Secrets Management:

Unify Your Secrets: Say goodbye to secrets scattered across systems. Pulumi ESC lets you pull and synchronize secrets from any source—HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, 1Password, and more—and consume them securely from anywhere.
Streamline Access and Control: Access secrets seamlessly via CLI, API, Kubernetes operator, the intuitive Pulumi Cloud UI, or directly within your code using our TypeScript, Python, and Go SDKs. Robust RBAC ensures the right people have the right access at all times.
Eliminate .env File Risks: Stop sharing secrets through insecure channels. Pulumi ESC provides a secure and centralized way to manage and access secrets, eliminating the need for risky .env files.

Boost Developer Velocity and Security:

Empower Developers, Not Slow Them Down: Pulumi ESC is designed for developer convenience without compromising security. Access secrets programmatically, integrate with your existing workflows, and leverage flexible authoring options: Pulumi Cloud Editor, VS Code Plugin, SDKs, the Pulumi Service Provider, and our REST API - the choice is yours!
Gain Complete Visibility and Control: Track all secret access and changes with detailed audit logs. Understand the downstream impact of changes and ensure your secrets are used securely and consistently across your organization.
Secrets as Code: Pulumi ESC supports flexible composition of environments by allowing imports between one another. Construct hierarchical environments that avoid the need to copy/paste secrets and configurations, ensuring that changes are consistently propagated across all related environments.

Built for the Modern Cloud, Open by Design:

Use It Your Way: Pulumi ESC integrates seamlessly with other Pulumi products and features such as Pulumi IaC and Pulumi Deployments for a unified infrastructure management experience. But its flexibility extends to any platform or toolset, including HashiCorp Terraform.
Embrace the Open Ecosystem: We believe in freedom of choice. That’s why Pulumi ESC is built on an open-source foundation (Apache 2.0 license) and designed to retrieve and synchronize secrets with various platforms seamlessly. We empower our customers to choose the best tools for their needs and supercharge their workflows with Pulumi ESC.

Customer Stories

Since launch, we’ve seen an exponential increase in our customers using Pulumi ESC. Our users have reduced the duplication of their configuration and secrets by over 90% and made Pulumi ESC a core part of running and managing their mission-critical infrastructure and applications.

Hear from our customers directly what they had to say about Pulumi ESC:

“With Pulumi ESC, our developers get dynamic AWS and Azure credentials on-demand, that removes the need for long-lived tokens and enhances security. Onboarding new developers is quick and secure, with no more manually filling in .env templates, since a common ESC dev environment with RBAC is all we need. The integration with the Pulumi SDKs is a huge productivity boost, and allows us to integrate secrets seamlessly across all of our development workflows." states Liam White, Platform Lead at Tetrate.”
“Pulumi ESC has been a lifesaver for us”; said JK Jensen, Team Lead at Mysten Labs, “it’s nice to throw everything behind an ESC environment and eliminate one-off granting IAM permissions and other issues related to static credentials. It gives us peace of mind knowing that we can grant permissions quickly and revoke easily, limiting the blast radius for any access.”

“With Pulumi ESC, our developers get dynamic AWS/GCP credentials on-demand; that removes the need for long-lived tokens and enhances security. ESC allows for Pulumi programs to share secure credentials and access secrets in their given cloud environments. ESC is a great configuration ‘/and secrets management sharing tool to allow for a DRY (Don’t Repeat Yourself) approach to development,” says Richard Genthner, Lead DevOps Engineer at Boost Insurance.

Watch the snippet from our PulumiUP Keynote that walks you through how to achieve all these benefits our customers have experienced. Sign-up here to watch the full keynote.

Beyond Preview: The Evolution of Pulumi ESC

Since the launch of Pulumi ESC in preview, over the last 11 months, we have launched numerous features and enhancements driven by our customers. Some of the major ones include:

SDKs for TypeScript, Python, and Go to leverage Pulumi ESC within all your applications
Versioning that maintains full history of the changes you have made to your secrets and configuration and rollback to specific versions when needed
Version Tags allow you to assign tags and import specific versions by tags within your applications such that moving tags across versions changes the configuration but requires zero changes in your application code
Syncing your secrets to several external sources including GitHub and AWS SecretsManager
Kubernetes secrets operator to use Pulumi ESC within your Kubernetes applications to retrieve secrets and configuration using external secrets operator during runtime
VS Code Extension to author and view your environments directly from where you code
Webhooks to trigger your automated CI/CD workflows, restart applications with the latest secrets and configuration, notify your teams
Pulumi OIDC Trust relationship to get Pulumi short-term credentials using your existing Identity provider
Projects and Environment Tags to help you organize your Pulumi ESC environments and allow them to scale as your organization evolves
1Password integration to pull secrets stored in 1Password during runtime
In-built Document Editor Enhancements to ease authoring with Pulumi Cloud
Table Editor to support users who prefer point-and-click UI-driven method to secrets management
Pulumi Stack reference provider offering tighter integration with Pulumi IaC
Files support that allows getting Kubernetes credentials dynamically
Audit logs to monitor your environments

Pricing

Starting October 1st, 2024, Pulumi ESC will be billed per secret. We have heard from customers that this is a familiar pricing dimension that aligns to the value customers get out of ESC. To help you get started, we have included a generous free tier of secrets and API calls.

	Free	Team	Enterprise	Business Critical
Price per secret/month	25 free	$0.50	$0.75 / Custom	Custom
Price per plaintext config	Free	Free	Free	Free
Price per 10K API calls	10K free	$0.10	$0.10	$0.10
Commitment discounts			✔	✔

Check out our pricing page for more information about the various packages available. Contact us if you are interested in a demo, have any questions, or would like to discuss volume discounts.

Conclusion

Pulumi ESC represents a significant advancement in secrets and config management. By addressing the challenges faced by organizations today, such as secrets sprawl and the trade-off between speed and security, Pulumi ESC offers a modern, flexible solution. Its unique features, including support for dynamic, short-lived credentials and seamless integration with major cloud providers, GitHub, etc. make it an invaluable asset for teams looking to maintain agility without compromising on security. Pulumi ESC is set to redefine the standards of secrets management, enabling organizations to achieve their security and operational goals more effectively.

We can’t wait to see how you leverage Pulumi ESC. Sign-up for Pulumi Cloud to start using Pulumi ESC and check out our getting started guide to learn more about Pulumi ESC.

Pushing Pulumi ESC Secrets into External Platforms

Komal Ali — Tue, 17 Sep 2024 00:00:00 -0300

This content now lives in the Pulumi docs. See Sync secrets to external platforms for the maintained version.

Managing secrets across multi-cloud infrastructures has long been a challenge for developers and operations teams. This article explores Pulumi IaC-based strategy to centrally define secrets and configuration in Pulumi ESC and automatically sync these values across the external platforms where they will be utilized, effectively reducing secret sprawl and manual overhead.

The Secret Management Conundrum

Traditional secret management often leads to a phenomenon known as secret sprawl. This occurs when secrets are duplicated across multiple systems, including various repositories, cloud providers, and CI/CD pipelines. The result is a tangled web of inconsistencies that makes tracking and managing secrets a Herculean task.

Updating secrets manually across all these systems is not only tedious but also error-prone. Each time a secret changes, it needs to be updated in every repository, cloud platform, and CI/CD tool where it’s used. This process is time-consuming and frustrating, often leading to mistakes that can compromise security.

Moreover, the lack of a centralized system makes it difficult to track where secrets are stored and who has access to them. This complicates security audits and compliance efforts. When a secret is compromised, the process of revoking and rotating it across multiple systems can be slow and disruptive, potentially leading to application downtime.

ESC Sync with Infrastructure as Code

Using Pulumi building blocks like ESC and IaC, we can create a pattern where secrets and configuration can be centrally defined in ESC, and then automatically synced across the external platforms where they will be utilized.

Consider the following ESC environment, which imports an environment “my-project/my-imported-env” and defines a set of secrets and configuration values to be synced:

imports:
 - my-project/my-imported-env@stable
values:
 sync:
 awsSecretsManager:
 value:
 myConfigKey: ${my-imported-env-foo}
 myNestedKey:
 haha: ${my-imported-env-bar}
 mySecret: ${my-imported-env-password}
 name: name-in-secrets-manager

The sync block defines the secrets and configuration values to be synced. In this example, we are syncing values to AWS Secrets Manager. The value field contains the actual values to be synced, while the name field specifies the name of the secret in AWS Secrets Manager.

To automate the synchronization process, we define a Pulumi program that creates a Pulumi ESC environment, Pulumi IaC stack, and Pulumi deployment settings for the target stack. We then define a set of pre-run commands that extract the values from the environment and set them as configuration values in the target stack. The syncCronSchedule variable specifies how often the synchronization process should run.

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();
const orgName = pulumi.getOrganization();

// projectName is the name of the project where the target stack is located
const projectName = config.require("projectName");

// stackName is the name of the target stack
const stackName = config.get("stackName") || "dev";

// repository is the GitHub repository where the target stack is located (for deployment settings)
const repository = config.require("repository");

// how often you want to sync. Default is hourly
const syncCronSchedule = config.get("syncCronSchedule") || "0 * * * *"

// envPath is the path to the environment file that contains the secrets or configuration to be synced
const envPath = config.get("envPath") || "syncEnv.yaml";

const env = new service.Environment("env", {
 organization: orgName,
 project: projectName,
 name: stackName,
 yaml: new pulumi.asset.FileAsset(envPath)
});

const stack = new service.Stack("esc-sync-aws-secretsmanager", {
 organizationName: orgName,
 projectName,
 stackName,
})

const fullyQualifiedStackName = pulumi.interpolate`${orgName}/${projectName}/${stackName}`;
const fullyQualifiedEnvName = pulumi.interpolate`${orgName}/${projectName}/${env.name}`;

const settings = new service.DeploymentSettings("deployment_settings", {
 organization: orgName,
 project: stack.projectName,
 stack: stack.stackName,
 github: {
 repository,
 },
 sourceContext: {
 git: {
 branch: "main",
 repoDir: "sync/target",
 }
 },
 operationContext: {
 preRunCommands: [
 'pulumi login',
 pulumi.interpolate`pulumi config env add ${projectName}/${env.name} -s ${fullyQualifiedStackName} --yes`,
 pulumi.interpolate`pulumi env open ${fullyQualifiedEnvName} sync.awsSecretsManager.value > sync.json`,
 pulumi.interpolate`pulumi config set -s ${fullyQualifiedStackName} secretName $(pulumi env open ${fullyQualifiedEnvName} sync.awsSecretsManager.name)`,
 ]
 }
});

const schedule = new service.DeploymentSchedule("update_schedule", {
 organization: orgName,
 project: settings.project,
 stack: settings.stack,
 scheduleCron: syncCronSchedule,
 pulumiOperation: "update",
})

Finally, we define a Pulumi program that creates the AWS Secrets Manager secret and secret version based on the values extracted from the environment file. The sync.json file contains the values to be synced, which are read and set as the secret string in the secret version. This Pulumi program is executed on a schedule to ensure that the secrets and configuration values are up-to-date between Pulumi ESC and AWS Secrets Manager.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from "fs";

const config = new pulumi.Config();
const name = config.require("secretName");

// Read a json file from the local filesystem using node.js fs module
const json = fs.readFileSync("sync.json", "utf8");

const secret = new aws.secretsmanager.Secret(name, {
 recoveryWindowInDays: 0,
})

const secretVersion = new aws.secretsmanager.SecretVersion(`${name}-version`, {
 secretId: secret.id,
 secretString: json,
})

// Export the name of the secret
export const secretName = secret.name;

This pattern allows you to define secrets and configuration values in a single location and automatically sync them across external platforms. The same pattern can be extended to create secrets in other platforms like Azure Key Vault, GCP Secrets Manager, GitHub Secrets, HashiCorp Vault and countless others.

Webhooks: An Event-Driven Alternative Approach

In addition to the scheduled synchronization approach, Pulumi ESC also supports webhooks. Webhooks allow you to respond to events in real-time, such as when a secret is updated in ESC, and take immediate action to sync the new value to an external platform.

In the example below, instead of a scheduled deployment, we define a webhook that triggers when the Pulumi ESC environment is updated. This ensures that the secrets and configuration values are always in sync between Pulumi ESC and the external platform.

const webhook = new service.Webhook("webhook", {
 organizationName: orgName,
 projectName: settings.project,
 stackName: settings.stack,
 displayName: "Sync to AWS Secrets Manager",
 environmentName: pulumi.interpolate`${env.project}/${env.name}`,
 filters: [WebhookFilters.ImportedEnvironmentChanged],
 format: WebhookFormat.PulumiDeployments,
 payloadUrl: pulumi.interpolate`${stack.projectName}/${stack.stackName}`,
 active: true,
})

When the webhook is triggered (i.e. the “my-project/my-imported-env” environment is updated), a deployment is automatically triggered, and the secrets and configuration values are synced to AWS Secrets Manager.

Embracing a New Era of Secret Management

Using Pulumi IaC to synchronize secrets and configuration across platform boundaries represents a giant step forward in secret management. By centralizing and automating the handling of sensitive data, it addresses the key challenges of traditional approaches. Secret sprawl becomes a thing of the past, and the risks associated with manual updates are significantly reduced.

Whether you’re working with AWS Secrets Manager, Azure Key Vault, GCP Secrets Manager, GitHub Secrets or other platforms, Pulumi ESC provides a streamlined solution that ensures your secrets are always secure and up-to-date. It offers the convenience and security of a single source of truth for all your secrets, allowing you to focus on building and improving your applications rather than wrestling with secret management.

We encourage you to explore Pulumi’s documentation and dive into the Pulumi ESC Sync Examples repository to see how you can implement this powerful tool in your own projects. Join our vibrant community on Slack to discuss your experiences, ask questions, and share insights. Together, we can make secret management simpler, more secure, and more efficient.

Introducing Webhooks for Pulumi ESC

Iaroslav Titov — Fri, 13 Sep 2024 00:00:00 -0300

Managing secrets and configurations across multiple environments and teams can be a complex juggling act for development teams. Pulumi ESC, our developer-friendly secrets management product, simplifies this process and accelerates your development cycle. Today, we’re excited to make this process even easier with the launch of Webhooks for Pulumi ESC, a powerful new feature that allows you to send a custom trigger to any URL endpoint, enabling you to automate workflows and keep your infrastructure and applications up to date. Pulumi ESC Webhooks also integrates seamlessly with popular communication platforms like Slack and Microsoft Teams.

With Pulumi ESC Webhooks, you can now receive real-time notifications and trigger automated actions based on specific events within your Pulumi ESC Environments. For example, you can set up a Webhook to notify your team’s Slack channel or automatically trigger a deployment pipeline via a POST request when a new environment revision is created due to changes in secrets and configuration. Webhooks can be triggered by changes to version tags and environment tags, and can be configured per environment or at the organization level for all environments.

Take your automation further by creating Webhooks that trigger Pulumi Deployments, automatically updating your infrastructure in response to configuration changes in Pulumi ESC—without writing custom code and all within the Pulumi Cloud Console.

Hierarchical configurations where one environment imports others are fundamental to Pulumi ESC. If an environment is importing another environment, any changes made to the imported environment trigger a Webhook configured within the current environment. The Webhook will notify you that the environment has changed, and you can call back to open the environment using SDK, REST API, Pulumi Service Provider, and Automation API, and get the latest secrets and configuration.

Getting Started with Pulumi ESC Webhooks

Setting up Webhooks is straightforward. You can use the Pulumi Cloud Console, the Pulumi Service Provider, or REST API.

Using Pulumi Cloud Console

Navigate to the “Webhooks” tab under settings for organization-wide Webhooks or within an individual environment’s top navigation bar.
Select your desired integration (Slack, Microsoft Teams, Pulumi Deployments, or generic JSON Webhooks).
Configure the Webhook details, including the events you want to trigger notifications on.

Using Pulumi Service Provider

Here is a sample code in TypeScript:

const environmentWebhook = new service.Webhook("env-webhook", {
 active: true,
 displayName: "env-webhook",
 organizationName: "my-org",
 projectName: environment.project,
 environmentName: environment.name,
 payloadUrl: "https://example.com",
 filters: [WebhookFilters.EnvironmentRevisionCreated, WebhookFilters.ImportedEnvironmentChanged],
})

Benefits of Pulumi ESC Webhooks

Pulumi ESC Webhooks offer a range of benefits designed to enhance visibility, streamline workflows, and automate key processes related to your secrets and configuration management:

Effortless Automation and Streamlined Workflows: Automatically trigger actions based on ESC events such as application restarts, infrastructure deployments, or configuration updates. Easily integrate Pulumi ESC Webhooks into your CI/CD pipelines to automate builds, tests, and deployments whenever environment updates occur.
Seamless Integration with Pulumi Deployments: Whether you need to update your infrastructure, preview changes, or create temporary infrastructure with TTL or review stacks, Pulumi ESC Webhooks handles all these scenarios, making deployments faster and easier.
Real-time Awareness and Collaboration: Receive instant notifications about critical changes in your Pulumi ESC environments, fostering better communication and collaboration between developers, operations teams, and security personnel. This real-time awareness enables faster incident response times, minimizing downtime and security risks.

Conclusion

Webhooks for Pulumi ESC equip your team with tools to improve visibility, streamline workflows, and automate key processes related to your secrets and configuration management. By integrating seamlessly with your existing communication and development tools, Pulumi ESC Webhooks empower you to automate and streamline your infrastructure management, giving you greater control and agility.

Dive into our documentation and explore the possibilities of Pulumi ESC Webhooks! As always, share any feedback or open new issues in our GitHub repository.

Introducing Pulumi ESC Projects and Environment Tags

Derek Schaller — Thu, 12 Sep 2024 13:00:00 -0400

We’re thrilled to unveil two new features that will empower you to organize your collection of Pulumi ESC Environments: Projects and Environment Tags. Projects offer a structured way to group related environments and Environment Tags allow you to add contextual information to each environment. Together, they offer a powerful way for you to manage, navigate, and collaborate on your secrets and configurations.

Overview of Pulumi ESC

Pulumi ESC is a developer-first platform designed to simplify the management of secrets and configurations into collections called environments. As a fully managed solution, it offers dynamic cloud provider credential resolution, a rich set of providers to retrieve secrets from external platforms, and the ability to use the secrets and configurations you have defined across any surface, including your applications and infrastructure via Multi-language SDKs, REST APIs, CLI, Pulumi-Service Provider, and Automation API. Like all Pulumi functionality, Pulumi ESC focuses on enhancing both agility and security in modern cloud development.

As developers have embraced Pulumi ESC to manage their collections of secrets and configurations across many teams and projects, the demand for more sophisticated organizational tools has increased. To address this growing need for more organized and efficient management of cloud environments, we are introducing Projects and Environment Tags in Pulumi ESC.

Introducing Projects

Projects offer a structured way to organize environments into logical groupings. Each Project has a distinct name, providing context for the environments within it, and can contain multiple environments with variations, each having a specific purpose. Additionally, Projects help large organizations simplify environment management by mitigating possible environment name conflicts and ensuring changes are made in isolation to the correct environments.

For example, your organization could have a project named “payments-app” containing dev, staging, and prod environments within it. Your organization could also have a project called “dev-credentials” which contains different environments used to provide temporary runtime access to credentials for developers while working.

We have two additional features as part of this launch:

Grouping and Search: You can group environments by either Project name or by Environment Tags, as well as search by them, making it simple to locate and manage your environments.
Environment Clone: Using the built-in cloning functionality, you can securely copy environments with sensitive information, preserving either the entire version history or just the latest environment state, depending on your needs.

To use Projects and Environment Tags, please update your Pulumi CLI and SDK to the latest.

Environments will now need to be referenced by specifying the project as part of the environment identifier. Any existing environments within your organization will be available in the default project. Projects are fully backward compatible - all your existing programs, imports and CLI commands will work without requiring any changes at the moment. Note that only environments within the default project will be backwards compatibly and resolve without a project prefix.

Users will be able to create new environments in the default project until Nov 30, 2024. Starting Dec 1, 2024, users will not be able to create new environments in the default project.

Starting Apr 1, 2025, users will not be able to edit their environments within the ‘default’ Project. We recommend customers migrate their existing environments to user-created projects at the earliest to take advantage of new Pulumi ESC capabilities.

Moving environments out of the default project is as easy as cloning the environment into a new project and updating any references to it. An example clone command that preserves all environment history, tags, and team access looks as follows

 $ pulumi env clone default/dev your-project/dev --preserve-history --preserve-env-tags --preserve-rev-tags --preserve-access

We offer numerous ways for you to get started with Projects via the Pulumi Web Console, SDK, REST API, CLI and Pulumi-Service Provider - choose what best fits your workflow. Here is one example using the Pulumi CLI to create and list environments within a project:

 $ pulumi env init cloud-pe/demo
 Environment created: dschaller/cloud-pe/demo

 $ pulumi env ls -p cloud-pe
 cloud-pe/demo
 pulumi/cloud-pe/demo

Breaking Changes

With the introduction of Projects, all existing environments part of the default project will continue to work without requiring any changes. However, when moving existing environments out of the default project you may notice some differences that we have outlined below.

If your environment was interpolating the context.currentEnvironment.name or context.rootEnvironment.name from Contextual information, this value will now include the project for any environments outside of the default project.
If you were using any of the secrets providers with OIDC, you will need to update both the subject and audience if a trust relationship has been set up. The subject will be either pulumi:environments:org:<organization name>:env:<project name>/<environment name> if no subjectAttributes are specified or if the currentEnvironment.name subject attribute is specified it will now resolve to currentEnvironment.name:<project name>/<environment name>. The audience will be the provider’s platform name (aws, azure, or gcp) followed by the org name of the environment (e.g. aws:pulumi).

Introducing Environment Tags

Using Environment Tags, teams can now assign any number of custom tags to each environment within Pulumi ESC. These tags serve as contextual identifiers that can streamline workflows by enabling you to group and search across environments based on specific criteria such as teams, project stages, deployment environments, compliance requirements, or geographic locations. This multi-dimensional approach provides unparalleled flexibility and control, ensuring your Pulumi ESC environments can scale seamlessly with your organizations evolving needs.

For example, suppose your organization has multiple environments across different geographic regions. You can use Environment Tags like ‘region: us-east’ or ‘region: eu-central-1’ to quickly filter environments by region. Similarly, for security audits, you might tag environments with ‘Compliance: SOC2’ or ‘Compliance: GDPR’ to easily find relevant environments and ensure they are compliant.

You can use the Pulumi Web Console and the CLI to create and manage Environment Tags. Here is an example of using the CLI to add and list Environment Tags.

$ pulumi env tag cloud-pe/demo region us-east-1
Name: region
Value: us-east-1
Last updated at 2024-09-10 11:03:05.708 -0700 PDT by Derek <dschaller>

$ pulumi env tag ls cloud-pe/demo
Name: region
Value: us-east-1
Last updated at 2024-09-10 11:03:05.708 -0700 PDT by Derek <dschaller>

Conclusion

Pulumi ESC Projects and Environment Tags are powerful additions that bring order and clarity to your secrets and configuration to even the most complex infrastructure landscapes. By enabling you to organize, group, and search for your environments with ease, we’re empowering you to scale your infrastructure with high confidence and flexibility.

We’re excited to see how you leverage these new capabilities. Check out the docs to learn more about Pulumi ESC Projects and Environment Tags and start organizing your environments today!

As always, please share your feedback on how we can further improve Pulumi ESC to suit your needs.

Pulumi ESC Table Editor Now Supports Dynamic Credential and Secret Integrations

Kimberley Mackenzie — Tue, 10 Sep 2024 00:00:00 -0700

We are pleased to announce a major update to the Pulumi ESC’s Table Editor: full support for provider configurations is now live! When we first introduced the ESC Table view, our goal was to offer a user-friendly interface for managing complex configuration and secrets for your applications and infrastructure. This has empowered teams to focus on what they do best—building and deploying software.

In our initial launch, the Table Editor allowed you to perform CRUD operations on your secrets and plaintext values, decrypt secrets, import environments, and manage variables—all within a visual interface. Today, we’re taking the next step by enabling full read-write support for provider configurations within the Table Editor. This new capability allows you to add, edit, and manage your provider configurations directly from the table view, offering a more integrated and seamless experience.

Primer on ESC Provider Configuration

Pulumi ESC’s provider ecosystem is one of its most powerful features. It supports two categories of providers:

Dynamic Cloud Provider Credentials: Dynamically acquire credentials from cloud providers like AWS, Azure, GCP, and Vault at runtime, eliminating the need for static, long-lived credentials. This enhances security by removing the reliance on dotenv files, locally stored secrets, or long-lived credentials in CI/CD environments, strengthening your workflows’ overall security.
Dynamic Secret Retrieval: ESC retrieves secrets from external secrets managers, such as AWS Secrets Manager, 1Password, and HashiCorp Vault, securely pulling them directly into your environments. Whether you are migrating or utilizing Pulumi ESC’s rich toolset, you can retrieve sensitive information at runtime and apply it seamlessly to your applications.

Enhancing the Table Editor: Provider Configuration Support

With the addition of provider configuration support in the Table view, incorporating Pulumi ESC provider capabilities into your workflow is now more straightforward. A clean, user-friendly form provides step-by-step guidance, promoting best practices like OIDC adoption and the secure handling of sensitive data. This streamlined approach reduces the need for external documentation, making the entire process more intuitive and efficient.

Conclusion

The new provider configuration support in the ESC Table view marks another step in our efforts to simplify and enhance the developer experience. We’re eager to see how you’ll leverage this new capability to streamline your workflows and strengthen your infrastructure’s security. As always, your feedback is invaluable to our development process, and we’re committed to continually improving the Table Editor and other Pulumi tools based on your needs. Please share your feedback with us here.

Check out our documentation to learn more about Pulumi ESC.

Secrets and Configuration for Software Engineers: Pulumi ESC Versioning, SDKs and More

Luke Hoban — Wed, 05 Jun 2024 04:00:00 -0700

Last October we launched Pulumi ESC, a new secrets and configuration management offering from Pulumi that manages and tames secrets and configuration complexity across all of your cloud applications and infrastructure. We’ve seen incredible adoption of ESC over the last 6 months, with thousands of developers now managing their secrets and configuration via ESC.

Today, we are excited to announce the most significant collection of Pulumi ESC enhancements since the initial launch, with three major new features that bring best-in-class software engineering capabilities to bear on your Secrets and Configuration management needs.

Versioning: Pulumi ESC now supports versioning of environments, allowing you to see and audit every change to the secrets and configuration for an environment, pin references to an environment to a specific version or version tag, and safely roll back an environment to a previous version.
SDKs: Pulumi ESC now has SDKs available for Python, TypeScript/JavaScript and Go, enabling ESC to be used directly within applications, tools and services to retrieve and manage secrets and configuration values at runtime.
Environments as Code with IaC: New support for defining and managing Pulumi ESC environments, secrets, and configuration from within Pulumi IaC programs allows source-controlled environment specification and managing secrets and configuration lifecycles via code.

All three of these new features double down on Pulumi ESC’s unique approach to secrets and configuration management, which combines a hierarchical and composable approach to defining environments, integration with a wide variety of existing secrets stores and a wide variety of secrets consumers, and a rich configuration-as-code approach.

Together, these provide the richest experience in the market today for developers to apply software engineering best practices to their secrets and configuration management, enabling the management of secrets and configuration complexity at scale across complex collections of environments and applications.

What is Pulumi ESC?

Pulumi ESC is a new category of configuration-as-code product, motivated by our experience working with hundreds of Pulumi IaC customers to address their needs in managing secrets and configuration at scale within their Pulumi infrastructure and across other cloud applications and infrastructure projects.

Pulumi ESC enables teams to aggregate secrets and configuration from many sources, manage hierarchical collections of configuration and secrets (“environments”), and consume those configuration and secrets from a variety of different application and infrastructure services. Pulumi ESC works hand-in-hand with Pulumi IaC to simplify configuration management, but also works independently from Pulumi IaC, as a solution for managing environments, secrets and configuration for any application or infrastructure project.

ESC Versioning

Safely managing the rollout of secrets and configuration changes to applications and infrastructure is one of the fundamental and critically important jobs that a software engineering team does. And yet, state-of-the-art secrets and configuration versioning still typically involve inflexible and bespoke systems for modifying configuration and/or copy/pasting across different environments. Pulumi ESC offers the ability to compose environments to avoid copy/paste, and also now supports rich versioning features to enable full auditability and safe rollout of configuration changes across environments.

There are two core features to Pulumi ESC versioning:

Visibility into the history of every change made to an environment.
The ability to pin any environment reference to a specific revision, or specific tagged version.

These offer the same sort of rich versioning typically seen in other software engineering ecosystems, like Git, Docker, or NPM - but now applied to your secrets and configuration management as well.

You can see the history of changes to an environment:

And then see the diff between any two versions of an environment:

Revisions can also be tagged to provide semantically meaningful names to specific revisions. These tags can be moved later on to point to new versions when you are ready to roll them out to consumers. The latest tag automatically always points to the latest revision.

Most importantly though, you can also then refer to a specific version of an environment anywhere you reference one.

When you run a command with pulumi env run you can specify providers@v2 to pick the v2 tag instead of the default latest:

$ pulumi env run acmecorp/providers@v2 -- aws sts get-caller-identity
{
 "UserId": "AROAY65FYVYP33NYY5FOU:environments-session",
 "Account": "358361613858",
 "Arn": "arn:aws:sts::358361613858:assumed-role/environments-oidc/environments-session"
}

When you import another environment you can import a pinned version. This enables changes to be made to the target environment without automatically being picked up by the consumer. This allows phased rollout, with careful testing of new major versions before they are pushed out to production environments.

imports:
 - ai-staging-aws@v3
values:
 pulumiConfig:
 clusterPetName: gold-kiwi
 clusterTags:
 cluster: ${pulumiConfig.clusterPetName}
 aws:defaultTags:
 tags: ${pulumiConfig.clusterTags}

Read more about Pulumi ESC versioning in the launch blog post.

ESC SDKs for Python, TypeScript/JavaScript and Go

Pulumi ESC was designed from the beginning to support consuming secrets and configuration from any application or infrastructure project. While we have seen lots of early usage combining ESC with Pulumi IaC, the two are not fundamentally tied together at all, and ESC is designed to serve the needs of any complex configuration and secrets management workload.

One of the most important places where secrets and configuration are consumed is in application and service code at runtime. This can take many forms - a few illustrative examples include:

An application that pulls configuration data at startup time
An operational tool that pulls in temporary credentials to perform operations within a cloud account
A control plane service that pushes configuration and secrets changes into other systems to keep configuration up to date and secrets actively rotated

In all cases, it’s important to be able to embed ESC within your application code. At launch, ESC supported a public REST API, but now this has been extended with convenient and easy to use SDKs in the most popular languages - Python, TypeScript/JavaScript and Go.

To use ESC from within your applications, just import the SDK and add a few lines of code:

import pulumi_esc_sdk as esc
import os

configuration = esc.Configuration(access_token=os.environ['PULUMI_ACCESS_TOKEN'])
client = esc.EscClient(configuration)

env, vals, txt = client.open_and_read_environment("acmecorp", "dev-stacks")

print(vals['aws']['secrets']['stripeApiKey'])

Running this gets the Stripe API Key for this environment:

$ python3 __main__.py
rk_test_3c88Pc8ZfdBredactedrEhDU11b42wI5KZ

Lots more details on the features and capabilities of the new Pulumi ESC SDKs in their launch blog post.

Environments as Code with IaC

Pulumi ESC helps to manage configuration and secrets complexity at scale via composition and reuse. But many users still want to go further and version control the definitions of their environments, secrets and configuration.

To enable this, along with many other powerful code-based environment management scenarios, we’ve added support for defining and managing Pulumi ESC Environments via Pulumi IaC itself. New Environment and EnvironmentVersionTag resources are available now in the Pulumi Cloud Provider in the Pulumi registry.

Managing the desired state of environments via infrastructure-as-code programs brings the most robust tools for managing version-controlled desired state definition possible to the realm of secrets and configuration management. And leveraging Pulumi’s best-in-class IaC capabilities and programming model offers the deepest possible software engineering capabilities for defining and managing configuration and secrets as code across environments.

Environment contents can be specified either inline, or in external files, and changes deployed into your Pulumi Cloud account via a simple pulumi up.

import * as service from "@pulumi/pulumiservice";
import * as pulumi from "@pulumi/pulumi";

var environment = new service.Environment("testing-environment", {
 organization: "service-provider-test-org",
 name: "testing-environment-ts",
 yaml: new pulumi.asset.StringAsset(`\
values:
 myKey1: "myValue1"
 myNestedKey:
 myKey2: "myValue2"
 myNumber: 1`,
 ),
});

For more on managing Pulumi ESC Environments using Pulumi IaC, check out the launch blog post.

Conclusion

Pulumi ESC is a secrets and configuration management offering that provides best-in-class software engineering features for managing secrets and configuration complexity at scale. Today, we are extending this even further with:

Rich versioning support for auditability and controlled configuration change rollout.
SDKs to easily embed ESC at runtime within applications, tools and services.
Support for source controlling environment specifications using Environment resources in Pulumi IaC programs.

Pulumi ESC has already been adopted by thousands of engineers to manage secrets and configuration complexity. Check it out and get started with Pulumi ESC today!

You can learn more about Pulumi ESC at:

Introducing Pulumi ESC Support in Automation API and the Pulumi Service Provider

Arun Loganathan — Wed, 05 Jun 2024 00:00:00 -0500

We’re excited to announce two powerful new capabilities for Pulumi Environments Secrets and Configurations (ESC) that supercharge how you manage and control your infrastructure and application secrets and configurations:

Pulumi Service Provider Support for ESC Environments: Manage your environments as code right alongside your infrastructure including fine-grained access control using Pulumi Service Provider.
Automation API Enhancements for ESC: Seamlessly integrate ESC Environments into your Automation API-based Pulumi projects.

These updates build upon Pulumi ESC’s commitment to simplify and streamline secrets and configuration management providing you with greater control, security, and automation capabilities.

Manage ESC Environments as Code with the Pulumi Service Provider

You can now manage your Pulumi ESC Environments directly within the Pulumi Service using the powerful Pulumi Service Provider. This means you can define environments, add version tags, and even control access using familiar Infrastructure as Code (IaC) practices ensuring consistency and repeatability across your deployments.

Here’s a glimpse of what you can achieve:

Programmatically create and manage ESC Environments: Define environments alongside your infrastructure code for a unified workflow.
Add version tags to environments: Add tags when you are creating the environment.
Enforce granular team permissions: Control who can read, open, or modify your environments ensuring secure collaboration and preventing unauthorized access.

Bringing Pulumi ESC Environments support into the Pulumi Service Provider empowers you to manage your entire infrastructure and application landscape through a unified approach.

Pulumi Service Provider ESC capabilities are available for TypeScript/JavaScript, Go, Python, C#, Java, and YAML. See our Environment, Environment Version Tag, and Environment Permissions Pulumi Service Provider docs. Here are examples of creating environments in TypeScript/JavaScript, Go, and Python:

import * as service from "@pulumi/pulumiservice";
import * as pulumi from "@pulumi/pulumi";

var environment = new service.Environment("testing-environment", {
 organization: "service-provider-test-org",
 name: "testing-environment-ts",
 yaml: new pulumi.asset.StringAsset(
 `values:
 myKey1: "myValue1"
 myNestedKey:
 myKey2: "myValue2"
 myNumber: 1`
 )
})

import pulumi
from pulumi_pulumiservice import Environment

environment = Environment(
 "testing-environment",
 organization="service-provider-test-org",
 name="testing-environment-py",
 yaml=pulumi.StringAsset("""
 values:
 myKey1: "myValue1"
 myNestedKey:
 myKey2: "myValue2"
 myNumber: 1
 """)
)

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := pulumiservice.NewEnvironment(ctx, "testing-environment", &pulumiservice.EnvironmentArgs{
 Name: pulumi.String("testing-environment-go"),
 Organization: pulumi.String("service-provider-test-org"),
 Yaml: pulumi.NewStringAsset(`
 values:
 myKey1: "myValue1"
 myNestedKey:
 myKey2: "myValue2"
 myNumber: 1`),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

Streamline Automated Workflows with Automation API Enhancements

We’re expanding the powerful Pulumi Automation API to include new methods for interacting with Pulumi ESC Environments programmatically. This enables you to seamlessly integrate environment management into your automated workflows and build sophisticated custom tooling.

The new Automation API methods for ESC include:

addEnvironments(...): Append environments in order to your Pulumi stack’s import list.
listEnvironments(): Retrieve a list of environments currently imported into your stack.
removeEnvironment(environment): Remove a specific environment from your stack’s import list.

These additions provide the building blocks for advanced automation scenarios such as dynamically configuring applications based on environments, managing environment dependencies, and integrating ESC into CI/CD pipelines.

Pulumi ESC Automation API capabilities are available for TypeScript/JavaScript, Go, and Python. Here are some examples:

import * as pulumi from "@pulumi/pulumi";
import * as automation from "@pulumi/pulumi/automation";

async function manageEnvironments() {
 const projectName = "myProject";
 const stackName = "dev";

 // Set up your stack options. Here we're using an empty program
 // since we're only manipulating environments, not deploying resources.
 const stackArgs: automation.InlineProgramArgs = {
 stackName,
 projectName,
 program: async () => { },
 };

 // Create or select an existing stack.
 const stack = await automation.LocalWorkspace.createOrSelectStack(stackArgs);

 // Add environments to your stack's configuration.
 // This is like adding them to the `imports` section of an ESC environment.
 await stack.addEnvironments("env1", "env2");
 // The stack's configuration now includes `env1` and `env2`.

 // List the environments currently configured for the stack.
 const environments = await stack.listEnvironments();
 console.log("Current environments:", environments); // Output: ["env1", "env2"]

 // Remove an environment from the stack's configuration.
 await stack.removeEnvironment("env1");
 // The stack's configuration now only includes `env2`.

 // List environments again to confirm the removal.
 const updatedEnvironments = await stack.listEnvironments();
 console.log("Updated environments:", updatedEnvironments); // Output: ["env2"]
}

// Call the async function to execute.
manageEnvironments().catch(console.error);

import pulumi
from pulumi.automation import LocalWorkspace, ProjectSettings, Stack

def manage_environments():
 stack_name = "dev"
 project_name = "myProject"

 # Define the project settings.
 project_settings = ProjectSettings(
 name=project_name,
 runtime="python",
 )

 # Create a LocalWorkspace with the project settings.
 workspace = LocalWorkspace(
 work_dir=".", # Use the current directory as the working directory
 project_settings=project_settings,
 )

 # Create or select an existing stack.
 stack = Stack.create(stack_name=stack_name, workspace=workspace)

 # Add environments to the stack's configuration.
 stack.add_environments("env1", "env2")

 # List the environments associated with the stack.
 environments = stack.list_environments()
 print("Current environments:", environments) # Output: ['env1', 'env2']

 # Remove an environment from the stack's configuration.
 stack.remove_environment("env1")

 # List the environments again to verify the removal.
 updated_environments = stack.list_environments()
 print("Updated environments:", updated_environments) # Output: ['env2']

# Run the function to manage environments.
manage_environments()

package main

import (
 "context"
 "fmt"
 "log"
 "github.com/pulumi/pulumi/sdk/v3/go/auto"
 "github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
 "github.com/pulumi/pulumi/sdk/v3/go/common/workspace"
)

func main() {
 ctx := context.Background()
 projectName := "myProject"
 stackName := "dev"

 // Define the project configuration.
 project := workspace.Project{
 Name: tokens.PackageName(projectName),
 Runtime: workspace.NewProjectRuntimeInfo("go", nil),
 }

 // Create a workspace with the project configuration.
 // Here we're using an empty program since we're only manipulating environments, not deploying resources.
 ws, err := auto.NewLocalWorkspace(ctx, auto.Project(project), auto.Program(nil))
 if err != nil {
 log.Fatalf("Failed to create workspace: %v", err)
 }

 // Create or select an existing stack.
 stack, err := auto.UpsertStack(ctx, stackName, ws)
 if err != nil {
 log.Fatalf("Failed to create or select stack: %v", err)
 }

 // Add environments to the stack's configuration.
 // This is like adding them to the `imports` section of an ESC environment.
 err = stack.AddEnvironments(ctx, "env1", "env2")
 if err != nil {
 log.Fatalf("Failed to add environments: %v", err)
 }
 // The stack's configuration now includes `env1` and `env2`.

 // List the environments associated with the stack.
 envs, err := stack.ListEnvironments(ctx)
 if err != nil {
 log.Fatalf("Failed to list environments: %v", err)
 }
 fmt.Println("Current environments:", envs) // Output: map[env1:{env1} env2:{env2}]

 // Remove an environment from the stack's configuration.
 err = stack.RemoveEnvironment(ctx, "env1")
 if err != nil {
 log.Fatalf("Failed to remove environment: %v", err)
 }
 // The stack's configuration now only includes `env2`.

 // List the environments again to confirm the removal.
 envs, err = stack.ListEnvironments(ctx)
 if err != nil {
 log.Fatalf("Failed to list environments: %v", err)
 }
 fmt.Println("Updated environments:", envs) // Output: map[env2:{env2}]
}

What’s Coming in the Future

We have exciting plans to further enhance these capabilities. Here’s a sneak peek at what’s on the horizon:

Pulumi service Provider: Capabilities to Open, Read, and List any environment versions and version tags
Automation API: Capabilities to author new environments, and assign version tags

Need more features or want to share feedback? We encourage you to open new issues on our GitHub repository or upvote existing ones.

Get Started Today!

These powerful new features are available now! We’re excited to see how you leverage these updates to simplify your secrets and configuration management, enhance your security posture, and unlock new levels of automation.

To learn more and start leveraging the enhanced capabilities of Pulumi ESC check out the following resources:

Pulumi Service Provider for ESC Environment, Environment Version Tag, and Environment Permissions
Pulumi ESC Automation API in TypeScript/JavaScript, Go, and Python

Introducing Pulumi ESC SDK: Streamline Your Application Secrets Management

Arun Loganathan — Wed, 05 Jun 2024 00:00:00 -0400

Managing secrets and application configurations effectively is crucial for building secure and maintainable software. However, developers often face challenges such as hardcoded credentials, configuration inconsistencies, and security risks. Pulumi Environments Secrets and Configuration (ESC) simplifies the management of sensitive data and configuration across your entire application lifecycle. Today, we’re thrilled to introduce the official Pulumi ESC SDK in TypeScript/JavaScript, Go, and Python, making it even easier to integrate ESC directly into your applications.

Primer on Pulumi ESC

Pulumi ESC (Environments, Secrets, and Configuration) provides a developer-first solution to simplify how you manage sensitive data and configuration across your entire application lifecycle. It’s a fully managed solution allowing teams to generate dynamic cloud provider credentials, aggregate secrets and configuration from multiple sources, and manage them through composable collections called “environments.” These environments can be consumed from anywhere, making Pulumi ESC ideal for any application and development workflow. Additionally, while Pulumi ESC works independently to eliminate duplication and reduce drift and sprawl of secrets and configuration for all your applications, it also integrates smoothly with Pulumi Infrastructure as Code (IaC) to enhance these capabilities within the Pulumi ecosystem.

Introducing the Pulumi ESC SDK

We’re excited to unveil the official Pulumi ESC SDK, making it even easier to harness the power of ESC directly within your applications using your favorite programming languages. The SDK provides a simple programmatic interface to all of Pulumi ESC’s robust features, allowing you to:

Manage the Entire Lifecycle of Your Environments: Create new environments, list existing ones, and easily update or delete them as your needs evolve. You can even add version tags to your environments, making it simple to track changes and roll back to previous states if needed.
Seamlessly Integrate Secrets and Configurations: Securely access and utilize secrets and configurations within your applications, ensuring consistency of configuration across environments. The SDK provides a streamlined way to fetch the information you need, whether it’s dynamic cloud credentials, database connection strings, feature flags, or any other sensitive data.

The Pulumi ESC SDK streamlines secret and configuration management, promoting secure handling best practices. It handles the secure storage and retrieval of your sensitive data at runtime, eliminating the need to store credentials long-term and minimizing the risk of accidental exposure. The SDK is a natural extension of your development workflow, offering familiar objects and methods, smooth integration with your IDE, and the benefits of type safety such as compile-time checks and helpful code suggestions that reduce errors and accelerate development.

How to Get Started

Here are a few examples of how to use the ESC SDK for various languages:

import * as esc from "@pulumi/esc-sdk";

async function main() {
 const PULUMI_ACCESS_TOKEN = process.env.PULUMI_ACCESS_TOKEN!;
 const orgName = process.env.PULUMI_ORG!;
 const config = new esc.Configuration({ accessToken: PULUMI_ACCESS_TOKEN });
 const client = new esc.EscApi(config);

 const envName = "sdk-typescript-example";

 // Create a new environment
 await client.createEnvironment(orgName, envName);

 const envDef: esc.EnvironmentDefinition = {
 values: {
 my_secret: {
 "fn::secret": "shh! don't tell anyone",
 },
 },
 };

 // Update the environment with the new definition
 await client.updateEnvironment(orgName, envName, envDef);

 // Open and read the environment
 const openEnv = await client.openAndReadEnvironment(orgName, envName);

 if (!openEnv) {
 console.error("Failed to open and read the environment");
 return;
 }

 // Acces the value of the secret
 const secretValue = openEnv.values?.my_secret;
 console.log(`Secret value: ${secretValue}\n`);

 // List all the environments in the organization
 const orgEnvs = await client.listEnvironments(orgName);
 if (!orgEnvs || !orgEnvs.environments) {
 console.log("No environments found");
 return;
 }

 for (const env of orgEnvs.environments) {
 console.log(`Environment: ${env.name}`);
 }
}

(async ()=>{
 await main();
})();

import pulumi_esc_sdk as esc
import os

accessToken = os.getenv("PULUMI_ACCESS_TOKEN")

orgName = os.getenv("PULUMI_ORG")

configuration = esc.Configuration(access_token=accessToken)
client = esc.EscClient(configuration)

envName = "sdk-python-example"

# Create environment
client.create_environment(orgName, envName)

# create a new EnvironmentDefinition with "my_secret" as a secret in values additional_properties
envDef = esc.EnvironmentDefinition(
 imports=[],
 values=esc.EnvironmentDefinitionValues(
 additional_properties={
 "my_secret": {
 "fn::secret": "shh! don't tell anyone"
 }
 }
 )
)

# Update environment
client.update_environment(orgName, envName, envDef)

# Open and read the environment

env, values, yaml = client.open_and_read_environment(orgName, envName)

secret = values["my_secret"]
print(f'Secret: {secret}\n')

# List environments

environments = client.list_environments(orgName)

for env in environments.environments:
 print(f'Environment: {env.name}')

package main

import (
 "log"
 "os"

 esc "github.com/pulumi/esc-sdk/sdk/go"
)

func main() {
 accessToken := os.Getenv("PULUMI_ACCESS_TOKEN")
 orgName := os.Getenv("PULUMI_ORG")
 configuration := esc.NewConfiguration()
 escClient := esc.NewClient(configuration)
 authCtx := esc.NewAuthContext(accessToken)

 envName := "sdk-go-example"
 // Create Environment
 err := escClient.CreateEnvironment(authCtx, orgName, envName)
 if err != nil {
 log.Fatalf("Failed to create environment: %v", err)
 }

 // Update Environment with Secrets
 updatePayload := &esc.EnvironmentDefinition{
 Values: &esc.EnvironmentDefinitionValues{
 AdditionalProperties: map[string]interface{}{
 "my_secret": map[string]string{
 "fn::secret": "shh! don't tell anyone",
 },
 },
 },
 }

 _, err = escClient.UpdateEnvironment(authCtx, orgName, envName, updatePayload)
 if err != nil {
 log.Fatalf("Failed to update environment: %v", err)
 }

 // Open and View Secrets
 _, values, err := escClient.OpenAndReadEnvironment(authCtx, orgName, envName)
 if err != nil {
 log.Fatalf("Failed to open environment: %v", err)
 }

 mySecret, ok := values["my_secret"]
 if !ok {
 log.Fatalf("Secret 'my_secret' not found in environment %s", envName)
 }

 log.Printf("my_secret: %v\n", mySecret)

 orgEnvs, err := escClient.ListEnvironments(authCtx, orgName, nil)
 if err != nil {
 log.Fatalf("Failed to list environments: %v", err)
 }

 for _, orgEnv := range orgEnvs.Environments {
 log.Printf("Environment: %v\n", orgEnv.Name)
 }

}

Example Scenarios: Pulumi ESC SDK in Action

Here are some real-world scenarios showcasing how you can leverage the Pulumi ESC SDK:

Database Credentials: Applications can use the ESC SDK to fetch database credentials with a specified TTL, ensuring they always use valid credentials, even as the credentials are rotated in the background. This eliminates service interruptions due to expired credentials.
Secure Serverless Deployments: A Google Cloud Function requiring access to an AWS S3 bucket can leverage the ESC SDK to obtain temporary AWS credentials during runtime. This enables secure cross-cloud interactions without embedding long-lived AWS credentials directly within the function’s code.
Microservices Configuration: With ESC, development teams can manage diverse configurations for their microservices, including API endpoints, retry policies, and logging levels, specific to each environment (development, staging, production). The ESC SDK allows each microservice to retrieve the correct configuration values during runtime, simplifying environment-based customizations.

Get Started Today!

The Pulumi ESC SDK is available now! To learn more and start simplifying your application secrets workflow, check out the Pulumi ESC SDK Documentation. We’re confident that this powerful new tool will streamline your development process, enhance your application security, and empower you to build innovative solutions with confidence. Join us on this journey towards a more secure and manageable future for application secrets and configurations!

Unveiling Pulumi ESC Versioning: Manage Secrets and Deployments with Confidence

Arun Loganathan — Wed, 05 Jun 2024 00:00:00 -0300

Since the launch of Pulumi Environments, Secrets and Configuration (ESC), our developer-first configuration and secrets management platform, we’ve seen exponential growth in usage. Customers have used it to simplify their secrets and configuration management by organizing them into composable collections called ’environments’. Today, we are thrilled to introduce a comprehensive suite of versioning features, giving you unprecedented control and confidence in managing your environments.

Uncover the Key Features of Pulumi ESC Versioning

Pulumi ESC Versioning introduces a range of powerful features:

Immutable Revision History: Every time you save an environment, a new, immutable revision is created, preserving a complete and tamper-proof history of your changes.
Compare Revisions Side-by-Side: Gain a clear understanding of changes between revisions with our detailed side-by-side comparison view. This helps identify differences, troubleshoot issues, and ensure smooth transitions between versions.
Tag Revisions: Assign meaningful names to specific revisions, enabling you to easily manage and deploy different versions of your environments. You can use tags like production, v1.2.1, stable, pre-release, and in-development to clearly communicate the purpose or stage of each version. Each environment has a built-in latest tag that always points to the environment’s most recent revision.
Import Specific Versions: Precisely control which version of an environment is imported, either by specifying a tag (e.g., environmentA@prod) or using a specific revision number (e.g., environmentB@2). This method also applies when importing environments via Pulumi Stack Config or via other environments, to effectively ‘pin’ the desired version.
Rollback to a Specific Version: Quickly revert your environment to a previous state by rolling back to a specific version or revision. This provides a safety net for deployments and makes it easy to undo unwanted changes.
Retract a Revision: Quickly remove a revision from your history – crucial for situations like accidentally exposing a secret in plaintext.

Level Up Your Workflow

Pulumi ESC Versioning delivers significant benefits that can transform your development process:

Disaster Prevention: By pinning a version during import, you can test new configurations in isolation before applying them to critical production systems. This minimizes downtime, reduces risk, and ensures business continuity, boosting confidence in deployments and supporting agile practices.
Improved Auditing and Collaboration: Easily track the complete history of your environments, including who made changes and when. This granular audit trail enhances security, ensures compliance, and promotes seamless collaboration within your team.
Streamlined Development Workflows: Similar to Docker tags, Pulumi ESC version tags provide a familiar and intuitive way to categorize and utilize specific revisions to manage different stages of your environments.

Tags and specific version imports are available on the Enterprise and Business Critical editions of Pulumi Cloud. The Team Edition offers access to the latest 5 revisions.

How to Get Started

Ready to start using versioning in Pulumi ESC? Here’s how you can get started using the CLI or the Console:

Using the CLI

Make sure you have the latest Pulumi CLI installed before you perform these steps. In your terminal,

Run pulumi env edit <environment-name> to edit an environment
Run pulumi env version history <environment-name> to view all revision history
Run pulumi env diff <environment-name>[@<version>] [[<environment-name>]@<version>] to compare changes between versions
Run pulumi env version tag <environment-name>@<tag> to assign a tag to the latest revision
Create a new environment using pulumi env init <new-environment-name> and import the previously edited environment using this syntax:
```
imports:
- <environment-name>@<tag>
```
Run pulumi env open <new-environment-name> to open and confirm that you have imported the right environment version

Check out the Pulumi CLI documentation for more details on available options and commands.

Using the Console

In your Pulumi Cloud Console,

Open an environment, make a change, and click Save
Go to the Versions Tab and select Revision History view to see the history of changes made to the environment
Click on a revision to see its details or view the differences between revisions using compare revisions
Add a tag to a revision by clicking on the Actions menu
Go to the Tags view and copy the reference to the tag
Create a new environment and import the tag

Real-world Scenarios

Here are a few examples of how you might use these features:

Global Infrastructure Update: You need to deploy a new configuration to a global service spanning multiple regions. Tag the updated configuration as next and deploy it to one region for testing. After verifying stability, you can confidently roll it out to other regions. If issues arise, quickly revert to the previous stable version, ensuring resilient and minimally disruptive deployments.
Platform Team Releases: If you’re a Platform team responsible for generating configurations, by tagging releases with labels like v1.0-stable and v1.2-development, you can clearly communicate which configurations are production-ready, keeping other teams informed about the ongoing development, and fostering seamless collaboration.
Code Review and Debugging: You’re working in a development environment where the latest revision includes multiple recent changes, and you notice that the application is experiencing performance issues. By using the Compare Revisions Side-by-Side feature or pulumi env diff, you can quickly identify the specific changes that might be causing the problem, enabling faster troubleshooting and resolution.

These examples are just a glimpse of how Pulumi ESC’s versioning capabilities can streamline your development and deployment process. The possibilities are vast, and you can tailor your tagging strategy to fit your specific needs and operational workflows.

Conclusion

The versioning capabilities in Pulumi ESC mark a significant step towards a more developer-friendly and efficient platform for configuration and secrets management. These features provide a robust foundation for secure, reliable, and auditable deployments, whether you’re rolling out infrastructure updates or ensuring compliance with industry regulations.

We’re excited to see how you leverage these powerful capabilities. Your feedback is crucial in helping us shape the product to better serve your needs. We encourage you to open new issues on our GitHub repository, upvote existing ones, or join the conversation on our community Slack channel.

Check out the Pulumi CLI and Versioning docs.

Announcing 1Password Support for Pulumi ESC in Public Preview

Tejitha Raju — Wed, 27 Mar 2024 12:00:00 -0600

Today, we are thrilled to add integration support for 1Password in preview within Pulumi Environments, Secrets, and Configuration (ESC). Since its launch, numerous organizations have leveraged Pulumi ESC to manage secrets and simplify configurations using composable environments. Leveraging Pulumi ESC’s comprehensive set of providers, users have obtained dynamic cloud provider credentials and retrieved secrets from other secrets management platforms, including HashiCorp Vault.

The addition of 1Password, known for its developer-centric approach to secrets management, to our roster of integrations has been a top request among our community. As users of 1Password ourselves, this collaboration represents more than just a feature release; it embodies our commitment to enriching the developer experience and fostering secure, efficient development workflows.

Pulumi ESC’s addition of the 1Password provider empowers development teams to seamlessly incorporate secrets and configurations stored within 1Password into their workflows. By using the retrieved values as environment variables or within Pulumi Config at runtime, teams can bypass the manual and error-prone process of copying and pasting secrets, significantly enhancing security and efficiency.

Join our upcoming workshop, “Managing team secrets with 1Password & Pulumi ESC”, to explore Pulumi ESC and the 1Password integration in detail.

How does it work?

Create a 1Password service account with access to necessary vaults
Enter the service token into the 1Password provider configuration within your Pulumi ESC Environment (See syntax below)
Define a path-name, and enter the vault name, item name and field name that you want to import into the path name you defined

1Password provider syntax:

values:
 1password:
 secrets:
 fn::open::1password-secrets:
 login:
 serviceAccountToken:
 fn::secret: <input your service token here>
 get:
 <path-name>:
 ref: op://<vault-name>/<item-name>/[section-name/]field-name

Demo

Now that we know how to use the Pulumi ESC 1Password provider, let’s use it to pull AWS CLI credentials stored in 1Password and use them within Pulumi ESC.

Create AWS CLI credentials through AWS IAM and store it in 1Password Vault. In the demo, we’ve stored the AWS credentials in Engineering vault with aws-cli-creds item name with two fields access-key and secret-access-key
Create a new Pulumi ESC Environment with the following definition. Ensure to replace the serviceAccountToken with your 1Password token

values:
 1password:
 secrets:
 fn::open::1password-secrets:
 login:
 serviceAccountToken:
 fn::secret: ops_eyJzaWduSW5B..[Redacted]
 get:
 aws-access-key:
 ref: op://Engineering/aws-cli-creds/access-key
 aws-secret-access-key:
 ref: op://Engineering/aws-cli-creds/secret-access-key
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${1password.secrets.aws-access-key}
 AWS_SECRET_ACCESS_KEY: ${1password.secrets.aws-secret-access-key}

Open the environment to ensure we are able to successfully pull the credentials from 1Password
Use the Pulumi ESC run command to run any AWS CLI commands

pulumi env run passes the configuration stored under the environmentVariables section into a temporary environment’s env variables. The secure credentials are never stored locally on your machine.

Unlocking New Possibilities

With 1Password integration available within Pulumi ESC, we unlock a range of opportunities for developers to enhance workflow efficiency and security:

Broad Cloud Provider Support: What we demonstrated with AWS applies equally to other cloud providers.
Database Secrets Management: Securely manage and inject database credentials for PostgreSQL, MySQL, etc., into your infrastructure as code (IaC) projects.
CI/CD Integration: Utilize Pulumi ESC and 1Password within CI/CD pipelines, such as GitHub Actions, to ensure secure and efficient workflows.

These examples merely scratch the surface of what’s possible. The combination of 1Password and Pulumi ESC opens up a wealth of scenarios to streamline operations and secure your infrastructure.

Conclusion

Pulumi ESC’s 1Password integration reiterates our commitment to providing choice and flexibility for developers. The 1Password provider brings the power of 1Password within ESC in a highly secure manner. We are excited to continue to work with our partners at 1Password and evolve our integration.

Your journey towards streamlined, secure cloud infrastructure management begins here. We can’t wait to see what you build with Pulumi ESC.

Visit the 1Password CI/CD Integrations page for links to the Pulumi ESC 1Password provider docs
Visit our Pulumi ESC docs to learn more about Pulumi ESC and its supported integrations

As always, we deeply value your insights. Your feedback is instrumental in helping us refine and enhance our solutions to better align with your needs.

Introducing the Table Editor for Pulumi ESC

Arun Loganathan — Tue, 26 Mar 2024 00:00:00 -0700

We’re excited to announce the launch of the Pulumi Environments, Secrets, and Configurations (ESC) Table Editor. At Pulumi, we’re committed to providing flexible solutions that cater to diverse development needs and practices. This latest addition underscores our dedication to this principle by offering a user-friendly interface for managing complex configurations and secrets.

Pulumi ESC simplifies the management of secrets and configurations by reducing duplication and drift, empowering development teams to focus on building and deploying. The existing YAML-based code editor offers a native way for developers to author and manage configurations.

The Table Editor we are launching today introduces an alternate visual interface for handling configuration tasks, enabling the direct input of plaintext and secret values. Key capabilities of the Table Editor include:

CRUD Operations: Perform Create, Read, Update, and Delete (CRUD) operations on your secrets and plaintext values
Decrypting Secrets: Safely decrypt secrets one at a time, directly within the editor
Environment Imports: Import environments and define the order of imports to suit your project’s needs
Variables Projections: Make variables available as Pulumi Config, environment variables, and files, streamlining your deployment and runtime configuration.
Interpolations and References: Add references to existing path-values, or add interpolations to dynamically generate values

In the current release of the Table view, provider configurations and functions are set to read-only.

What’s Next?

Looking to the future, we are already working on the next set of enhancements for the Table view. Soon, you will have the capability to add and edit provider configurations directly within the table view.

Pulumi ESC’s diverse array of providers not only allows teams to dynamically acquire cloud provider credentials at runtime, thus eliminating the need for static, long-lived credentials but also enables retrieving secrets from external secrets managers, such as AWS Secrets Manager and HashiCorp Vault, directly within ESC. This means users can seamlessly pull and utilize secrets within ESC, enhancing security and flexibility. The upcoming provider configuration support in the Table view will offer a guided experience, simplifying the process of leveraging Pulumi ESC’s comprehensive set of providers.

Conclusion

The launch of the Table view marks an important step in our mission to make infrastructure as code more accessible and manageable for developers.

Your feedback is the cornerstone of our development process. As we look ahead, we’re focusing on incorporating your insights to further enhance the Table view. Together, we can continue to refine and advance Pulumi ESC, making it an even more powerful tool for the developer community.

Check out our documentation to learn more about Pulumi ESC.

Introducing the Latest ESC Editor Enhancements for Improved Authoring

Pat Gavlin — Thu, 22 Feb 2024 00:00:00 -0700

With Pulumi ESC, our goal is to not only create a tool that simplifies the development process but also one that developers love. In pursuit of this goal, we’re excited to announce enhancements to the Pulumi ESC environment editor. These enhancements are focused on addressing common challenges encountered when authoring environments: syntax errors, type errors, frequent context switches to and from documentation, and more. Our aim is to make the process of authoring environments as straightforward as possible by removing common hurdles.

Highlights

Auto-complete: This feature provides suggestions for code snippets and resource properties, easing the need to reference documentation. It is engineered to accelerate and refine the coding process, enabling developers to concentrate on innovation and problem-solving.
Enhanced Diagnostics: These diagnostics go beyond basic syntax checks by providing in-depth, as-you-type insights into errors.
Hover Documentation: Hover over any piece of code to see a detailed tooltip with relevant documentation and examples. This instant access to information reduces the need to context switch, streamlining the development process and minimizing disruptions.

Tap ‘Ctrl+Space’ in the editor reveal the list of functions and properties.

Contextual Information to your environments

We have also added support for accessing contextual information from within an ESC environment such as the current user, organization, and environment name. You can use this information to enrich audit logs, refine OIDC claims for more granular access, and more! This information is available by accessing the context property, and contains the following information:

context.rootEnvironment.name: the name of the root environment being evaluated
context.currentEnvironment.name: the name of the current environment being evaluated
context.pulumi.user.login: the login of the user evaluating the environment
context.pulumi.organization.login: the name of the user’s organization

values:
 rootEnv: ${context.rootEnvironment.name}
 currentUser: ${context.pulumi.user.login}

For the current, complete list of context properties, see Built-in properties.

A Seamless Authoring Experience Awaits

The updated editor experience is designed to minimize friction, boost productivity, and transform cloud development into a more enjoyable and effective journey. We are eager for you to explore these new features and see firsthand how they enhance your development process.

Discover more about the capabilities of Pulumi ESC and how it can support your projects by checking out our documentation.

Secure your Kubernetes toolchain with Pulumi ESC and OIDC

Levi Blackstone — Tue, 21 Nov 2023 00:00:00 -0700

Keeping long-lived kubeconfig around on disk is insecure and error-prone. You need a secure workflow that removes tedium. With Pulumi and ESC, we provide an automated workflow that generates a kubeconfig on-the-fly for every command using short-term credentials issued via OIDC. This makes it easy for your team to connect to a given Kubernetes environment, and it works well with Kubernetes tools such as kubectl and the Pulumi Kubernetes provider. Let’s take a look.

In this post, we will create a Kubernetes cluster, deploy an application to the created cluster, and then use kubectl to check on the deployed application. Pulumi gives you great flexibility to factor your infrastructure into reusable parts, and we’ll do that here. With ESC, we’ll define an environment that provides access to an AWS account, and use that environment to create the cluster. Then we’ll define an environment that encapsulates the kubeconfig needed to connect to that cluster.

Create a Kubernetes Cluster

First, we create a Kubernetes cluster using the pulumi-eks component.

import * as pulumi from "@pulumi/pulumi";
import * as eks from "@pulumi/eks";

const eksCluster = new eks.Cluster("eks-cluster", {});
export const kubeconfig = eksCluster.kubeconfig;

import pulumi
import pulumi_eks as eks

eks_cluster = eks.Cluster("eks-cluster")
pulumi.export("kubeconfig", eks_cluster.kubeconfig)

import (
 "github.com/pulumi/pulumi-eks/sdk/go/eks"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 eksCluster, err := eks.NewCluster(ctx, "eks-cluster", nil)
 if err != nil {
 return err
 }
 ctx.Export("kubeconfig", eksCluster.Kubeconfig)
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Eks = Pulumi.Eks;

return await Deployment.RunAsync(() =>
{
 var eksCluster = new Eks.Cluster("eks-cluster");

 return new Dictionary<string, object?>
 {
 ["kubeconfig"] = eksCluster.Kubeconfig,
 };
});

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.eks.Cluster;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var eksCluster = new Cluster("eksCluster");

 ctx.export("kubeconfig", eksCluster.kubeconfig());
 }
}

name: eks-cluster
runtime: nodejs
description: EKS Kubernetes Cluster
resources:
 eks-cluster:
 type: eks:Cluster
outputs:
 kubeconfig: ${eks-cluster.kubeconfig}

This component includes an output value for the cluster’s kubeconfig that we can use to connect to the cluster. However, before we create the cluster, we need to configure AWS credentials so that Pulumi can deploy the requested changes. While it is possible to set this configuration directly on the stack, let’s see how it can be done with Pulumi ESC. First, we create the following ESC environment (the aws-sandbox environment):

values:
 aws:
 creds:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::0123456789:role/deploy-oidc
 sessionName: pulumi-environments-session
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.creds.keyId}
 AWS_SECRET_ACCESS_KEY: ${aws.creds.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.creds.sessionToken}
 pulumiConfig:
 aws:region: us-west-2

Notice that this environment uses the aws-login ESC provider to dynamically load short-lived credentials using OIDC. These credentials are then exposed as environment variables for consumers of this environment. We also set the AWS region in the pulumiConfig section to configure the region for pulumi to manage resources with the pulumi-aws provider.

To have Pulumi use this ESC environment when deploying the cluster, we set the following configuration for our stack:

environment:
 - aws-sandbox

The environment key accepts a list of ESC environments to import. When the stack is updated, pulumi automatically sets the configured environment variables and stack configuration based on the ESC environment. We run our update, and Pulumi creates our EKS cluster.

pulumi up --skip-preview
Updating (demo)
Type Name Status
+ pulumi:pulumi:Stack eks-cluster-demo created (577s)
+ └─ eks:index:Cluster eks-cluster created (561s)
+ ├─ eks:index:ServiceRole eks-cluster-eksRole created (2s)
+ │ ├─ aws:iam:Role eks-cluster-eksRole-role created (0.78s)
+ │ └─ aws:iam:RolePolicyAttachment eks-cluster-eksRole-4b490823 created (0.44s)
+ ├─ eks:index:ServiceRole eks-cluster-instanceRole created (3s)
+ │ ├─ aws:iam:Role eks-cluster-instanceRole-role created (0.85s)
+ │ ├─ aws:iam:RolePolicyAttachment eks-cluster-instanceRole-3eb088f2 created (0.46s)
+ │ ├─ aws:iam:RolePolicyAttachment eks-cluster-instanceRole-03516f97 created (0.59s)
+ │ └─ aws:iam:RolePolicyAttachment eks-cluster-instanceRole-e1b295bd created (0.75s)
+ ├─ eks:index:RandomSuffix eks-cluster-cfnStackName created (0.49s)
+ ├─ aws:ec2:SecurityGroup eks-cluster-eksClusterSecurityGroup created (1s)
+ ├─ aws:iam:InstanceProfile eks-cluster-instanceProfile created (0.48s)
+ ├─ aws:eks:Cluster eks-cluster-eksCluster created (441s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksClusterInternetEgressRule created (0.81s)
+ ├─ aws:ec2:SecurityGroup eks-cluster-nodeSecurityGroup created (1s)
+ ├─ pulumi:providers:kubernetes eks-cluster-eks-k8s created (0.14s)
+ ├─ eks:index:VpcCni eks-cluster-vpc-cni created (2s)
+ ├─ kubernetes:core/v1:ConfigMap eks-cluster-nodeAccess created (0.60s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksExtApiServerClusterIngressRule created (1s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksClusterIngressRule created (1s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksNodeInternetEgressRule created (1s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksNodeClusterIngressRule created (2s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksNodeIngressRule created (2s)
+ ├─ aws:ec2:LaunchConfiguration eks-cluster-nodeLaunchConfiguration created (1s)
+ ├─ aws:cloudformation:Stack eks-cluster-nodes created (108s)
+ └─ pulumi:providers:kubernetes eks-cluster-provider created (0.12s)
Outputs:
kubeconfig: {
apiVersion : "v1"
clusters : [
[0]: {
cluster: {
certificate-authority-data: "<redacted>"
server : "https://FB9978041F7B72B227B7E95F076822E4.gr7.us-west-2.eks.amazonaws.com"
}
name : "kubernetes"
}
]
contexts : [
[0]: {
context: {
cluster: "kubernetes"
user : "aws"
}
name : "aws"
}
]
current-context: "aws"
kind : "Config"
users : [
[0]: {
name: "aws"
user: {
exec: {
apiVersion: "client.authentication.k8s.io/v1beta1"
args : [
[0]: "eks"
[1]: "get-token"
[2]: "--cluster-name"
[3]: "eks-cluster-eksCluster-1268ff9"
]
command : "aws"
env : [
[0]: {
name : "KUBERNETES_EXEC_INFO"
value: (json) {
apiVersion: "client.authentication.k8s.io/v1beta1"
}
}
]
}
}
}
]
}
Resources:
+ 27 created
Duration: 9m38s

Connect to the cluster

Let’s create another Pulumi ESC environment called eks-sandbox to encapsulate the configuration needed to connect to the EKS cluster.

imports:
 - aws-sandbox
values:
 stacks:
 fn::open::pulumi-stacks:
 stacks:
 eks-cluster:
 stack: eks-cluster/demo
 kubeconfig: {'fn::toJSON': "${stacks.eks-cluster.kubeconfig}"}
 pulumiConfig:
 kubernetes:kubeconfig: ${kubeconfig}
 files:
 KUBECONFIG: ${kubeconfig}

This ESC environment uses another ESC provider to reference outputs from Pulumi stacks in your organization. In this case, we specify that we want to read outputs from the eks-cluster/demo stack, and then export the kubeconfig output from that stack as pulumiConfig and files. As we saw on the cluster environment, pulumiConfig sets configuration for Pulumi providers, so this allows us to run our stack and deploy the application to our cluster.

We import the aws-sandbox environment defined earlier, to provide the user with AWS credentials to authenticate to the EKS cluster.

Since the eks-sandbox environment defines a KUBECONFIG variable that kubectl understands, we can run kubectl in the context of this environment like this:

$ pulumi env run eks-sandbox -- kubectl cluster-info
Kubernetes control plane is running at https://2F3303BF9E03F8D61099AAB5ED6F31A4.gr7.us-west-2.eks.amazonaws.com
CoreDNS is running at https://2F3303BF9E03F8D61099AAB5ED6F31A4.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Deploy an application to the cluster

Now, we want to deploy an application to our new EKS cluster. While we could have included this in the same stack as the cluster, it is generally better to follow the separation of concerns principle, and manage applications separately from the underlying infrastructure. To achieve this goal, we create a new application stack with the following program:

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";

const nginxDeployment = new kubernetes.apps.v1.Deployment("nginx-deployment", {
 metadata: {
 name: "nginx",
 labels: {
 app: "nginx",
 },
 },
 spec: {
 replicas: 3,
 selector: {
 matchLabels: {
 app: "nginx",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "nginx",
 },
 },
 spec: {
 containers: [{
 name: "nginx",
 image: "nginx",
 ports: [{
 containerPort: 80,
 }],
 }],
 },
 },
 },
});
const nginxService = new kubernetes.core.v1.Service("nginx-service", {
 metadata: {
 name: "nginx",
 },
 spec: {
 type: "LoadBalancer",
 selector: {
 app: "nginx",
 },
 ports: [{
 protocol: "TCP",
 port: 80,
 }],
 },
});

import pulumi
import pulumi_kubernetes as kubernetes

nginx_deployment = kubernetes.apps.v1.Deployment("nginx-deployment",
 metadata=kubernetes.meta.v1.ObjectMetaArgs(
 name="nginx",
 labels={
 "app": "nginx",
 },
 ),
 spec=kubernetes.apps.v1.DeploymentSpecArgs(
 replicas=3,
 selector=kubernetes.meta.v1.LabelSelectorArgs(
 match_labels={
 "app": "nginx",
 },
 ),
 template=kubernetes.core.v1.PodTemplateSpecArgs(
 metadata=kubernetes.meta.v1.ObjectMetaArgs(
 labels={
 "app": "nginx",
 },
 ),
 spec=kubernetes.core.v1.PodSpecArgs(
 containers=[kubernetes.core.v1.ContainerArgs(
 name="nginx",
 image="nginx",
 ports=[kubernetes.core.v1.ContainerPortArgs(
 container_port=80,
 )],
 )],
 ),
 ),
 ))
nginx_service = kubernetes.core.v1.Service("nginx-service",
 metadata=kubernetes.meta.v1.ObjectMetaArgs(
 name="nginx",
 ),
 spec=kubernetes.core.v1.ServiceSpecArgs(
 type="LoadBalancer",
 selector={
 "app": "nginx",
 },
 ports=[kubernetes.core.v1.ServicePortArgs(
 protocol="TCP",
 port=80,
 )],
 ))

import (
 appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := appsv1.NewDeployment(ctx, "nginx-deployment", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx"),
 Labels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Spec: &appsv1.DeploymentSpecArgs{
 Replicas: pulumi.Int(3),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx"),
 Ports: corev1.ContainerPortArray{
 &corev1.ContainerPortArgs{
 ContainerPort: pulumi.Int(80),
 },
 },
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }
 _, err = corev1.NewService(ctx, "nginx-service", &corev1.ServiceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx"),
 },
 Spec: &corev1.ServiceSpecArgs{
 Type: pulumi.String("LoadBalancer"),
 Selector: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 Ports: corev1.ServicePortArray{
 &corev1.ServicePortArgs{
 Protocol: pulumi.String("TCP"),
 Port: pulumi.Int(80),
 },
 },
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 var nginxDeployment = new Kubernetes.Apps.V1.Deployment("nginx-deployment", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "nginx",
 Labels =
 {
 { "app", "nginx" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Replicas = 3,
 Selector = new Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "nginx" },
 },
 },
 Template = new Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Labels =
 {
 { "app", "nginx" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 Containers = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "nginx",
 Image = "nginx",
 Ports = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerPortArgs
 {
 ContainerPortValue = 80,
 },
 },
 },
 },
 },
 },
 },
 });

 var nginxService = new Kubernetes.Core.V1.Service("nginx-service", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "nginx",
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.ServiceSpecArgs
 {
 Type = "LoadBalancer",
 Selector =
 {
 { "app", "nginx" },
 },
 Ports = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ServicePortArgs
 {
 Protocol = "TCP",
 Port = 80,
 },
 },
 },
 });

});

import java.security.Provider.Service;
import java.util.Map;

import javax.naming.Context;

import com.pulumi.Pulumi;
import com.pulumi.kubernetes.apps_v1.Deployment;
import com.pulumi.kubernetes.apps_v1.DeploymentArgs;
import com.pulumi.kubernetes.apps_v1.inputs.DeploymentSpecArgs;
import com.pulumi.kubernetes.core_v1.ServiceArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodTemplateSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.ServiceSpecArgs;
import com.pulumi.kubernetes.meta_v1.inputs.LabelSelectorArgs;
import com.pulumi.kubernetes.meta_v1.inputs.ObjectMetaArgs;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var nginxDeployment = new Deployment("nginxDeployment", DeploymentArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx")
 .labels(Map.of("app", "nginx"))
 .build())
 .spec(DeploymentSpecArgs.builder()
 .replicas(3)
 .selector(LabelSelectorArgs.builder()
 .matchLabels(Map.of("app", "nginx"))
 .build())
 .template(PodTemplateSpecArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .labels(Map.of("app", "nginx"))
 .build())
 .spec(PodSpecArgs.builder()
 .containers(ContainerArgs.builder()
 .name("nginx")
 .image("nginx")
 .ports(ContainerPortArgs.builder()
 .containerPort(80)
 .build())
 .build())
 .build())
 .build())
 .build())
 .build());

 var nginxService = new Service("nginxService", ServiceArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx")
 .build())
 .spec(ServiceSpecArgs.builder()
 .type("LoadBalancer")
 .selector(Map.of("app", "nginx"))
 .ports(ServicePortArgs.builder()
 .protocol("TCP")
 .port(80)
 .build())
 .build())
 .build());

 }
}

name: yaml-example
runtime: yaml
description: Deployment + Service
resources:
 nginx-deployment:
 type: kubernetes:apps/v1:Deployment
 properties:
 metadata:
 name: nginx
 labels:
 app: nginx
 spec:
 replicas: 3
 selector:
 matchLabels:
 app: nginx
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: nginx
 image: "nginx"
 ports:
 - containerPort: 80
 nginx-service:
 type: kubernetes:core/v1:Service
 properties:
 metadata:
 name: nginx
 spec:
 type: LoadBalancer
 selector:
 app: nginx
 ports:
 - protocol: TCP
 port: 80

We need to set some configuration before we deploy this stack. In this case, we need a kubeconfig to allow the pulumi-kubernetes provider to connect to our EKS cluster.

We import the eks-sandbox environment with the following stack configuration:

environment:
 - eks-sandbox

Now we update our stack, and see that our application gets deployed to the cluster, without needing to manually configure any credentials – ESC automatically injects them for us!

pulumi up --skip-preview
Updating (demo)
Type Name Status
+ pulumi:pulumi:Stack yaml-example-demo created (0.31s)
+ ├─ kubernetes:core/v1:Service nginx-service created (10s)
+ └─ kubernetes:apps/v1:Deployment nginx-deployment created (2s)
Resources:
+ 3 created
Duration: 13s

Now that the application is deployed, let’s take a look using kubectl:

$ pulumi env run eks-sandbox -- kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/3 3 3 103s

ESC environments can be used with the pulumi CLI using the pulumi env command.

Export Files

As we see in the ESC environment definition named eks-sandbox, there’s a files section (under values) for defining values that are automatically exported as temporary files when you run the environment. You can use this feature to generate configuration files to support your preferred tools. Pulumi exports each key-value pair as a temporary file containing the actual value, and an associated environment variable pointing to it.

Earlier we exported an environment variable named KUBECONFIG with a dynamic value based on a stack output. Here’s a similar example with the configuration declared statically:

values:
 files:
 KUBECONFIG: |
 apiVersion: v1
 kind: Config
 clusters:
 - cluster:
 server: https://127.0.0.1:6443
 name: docker-desktop

Conclusion

Pulumi ESC makes it easier than ever to tame infrastructure complexity. For this demo, we created two reusable ESC environments to configure AWS and Kubernetes credentials, created an EKS cluster, deployed a load-balanced application to the cluster, and configured our local toolchain to connect to the cluster. Since we factored the shared configuration into ESC environments, it is easy to extend this example to deploy additional AWS or Kubernetes resources. Pulumi ESC supports dynamic credentials using OIDC across AWS, Azure, and Google Cloud, with additional providers on the roadmap. Check out the following links to learn more about Pulumi ESC today!