How Pulumi ESC works

This article details how Pulumi ESC works, including its architecture, supported integrations, and core functionalities. To understand some of the core ESC concepts, like environments, dynamic secrets, and configuration-as-code, please read through the ESC Concepts doc.

Architecture

Pulumi ESC is available as a hosted service provided as part of Pulumi Cloud or can be self-hosted on your own infrastructure. ESC stores your secrets and configuration and proxies access to other secret stores through provider plugins.

Using configuration-as-code, these secrets and configuration values are composed into environments which are defined in YAML. ESC then makes these environments available to targets via common mechanisms like environment variables, configuration files, and directly via the ESC API and multi-language SDKs.

Secrets and configuration sources

By default, Pulumi ESC stores your configuration and secrets in Pulumi Cloud. However, ESC also integrates with a variety of third-party sources through an extensible provider plugin model. This allows teams to use their preferred providers without needing to manually copy or paste secrets across environments. The secrets will be dynamically fetched from the third-party API and integrated into your ESC environments.

By aggregating secrets from these providers, Pulumi ESC provides a unified interface for retrieving, using, and managing secrets, regardless of their source.

Supported login and secret providers

Pulumi ESC integrates with many popular cloud login providers and secrets managers, including:

• AWS OIDC and AWS Secrets Manager
• Azure OIDC and Azure KeyVault
• GCP OIDC and GCP Secrets Manager
• HashiCorp Vault OIDC and Vault Secrets
• 1Password, Kubernetes, among others.

Teams can setup OpenID Connect integration in their cloud providers to allow ESC environments to pull short-lived credentials via OIDC for secure, time-limited access to secrets. These credentials can then be used in both Pulumi IaC workflows and external CLIs like aws, kubectl, etc.

The ESC data model

Configuration and Secrets as Code

Pulumi ESC uses a configuration-as-code approach. ESC environments are defined in YAML, which contain named configuration values, encrypted secrets, and provider references.

A simple static environment can be thought of as a collection of key/value pairs. They can also contain interpolated values and complex structured configuration. Static secrets are also defined in YAML and are encrypted before they are stored to ensure security.

Example: Defining configuration values, structured data, and encrypted secrets in YAML.

values:
  name: world
  salutation: hello
  greeting: ${salutation}, ${name}

  structured_data:
    active: true
    nums:
      - 1
      - 2

  myPassword:
    fn::secret:
      ciphertext: ZXNjeAA....

Dynamic and Projected Values

Dynamic secrets are defined in YAML as provider references. The interpolation mechanism can also be used to project these ESC values as environment variables or files.

Example: Defining dynamic secrets and projecting environment variables.

values:
  aws:
    login:
      fn::open::aws-login:
        oidc:
          roleArn: arn:aws:iam::01234567891011:role/some-role
          sessionName: some-session
          duration: 1h

  environmentVariables:
    AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
    AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
    AWS_SESSION_TOKEN: ${aws.login.sessionToken}
    MY_ENV_VAR: "true"

Tags and Versions

Every change to an ESC environment definition is recorded as a unique numeric version. Environments can be tagged at specific versions to allow for multiple versions of the same environment to be in use at the same time (for example, production vs staging, or for A/B testing). This also facilitates easy rollbacks of both secrets and credentials in a single action.

Composing environments

ESC environments can be composed from other environments allowing for modularity and inheritance – concepts usually only found in code:

• Inheritance: One environment can inherit values from another environment.
• Overrides: Values in child environments can override inherited values.
• Nesting: Environments can be arbitrarily nested for maximum flexibility, allowing complex hierarchies to be built out.

Managing access: RBAC and auditing

RBAC Implementation

Pulumi ESC handles access control with robust security measures, leveraging RBAC (Role-Based Access Control) to manage permissions from within Pulumi Cloud. ESC environment permissions are set at organizational, team, and individual levels, allowing fine-grained control over who can update, preview, or retrieve environments and secrets. This ensures only authorized team members can interact with sensitive configuration or secrets..

Auditing and Versioning

Every action, including opening an environment or making changes, is recorded in an audit log. This log tracks what changes were made, by whom, and when, providing a full history of interactions with secrets and configurations. Environments are also versioned, allowing teams to roll back changes if needed.

Integration with the Pulumi ecosystem and beyond

Pulumi IaC integration

Pulumi ESC integrates tightly with Pulumi Infrastructure as Code (IaC), allowing environments to be accessed during the deployment process using commands like pulumi up. Adding an ESC environment to a Pulumi program requires only a single YAML value indicating which environments your Pulumi stack needs access to.

Example: Accessing an ESC environment from a Pulumi program.

# Pulumi.dev.yaml
environment:
  - myapp-dev

Every function of the ESC CLI is also available as a subcommand in the Pulumi IaC CLI.

Standalone CLI and API

The Pulumi ESC CLI and ESC Automation API allow teams to interact with environments outside of Pulumi IaC. This means secrets and configurations can be accessed from external applications, infrastructure providers, or automation systems.

First-Class developer tools integration

Pulumi ESC also integrates with popular developer tools like GitHub Actions, DirEnv, and Docker, enabling teams to pull configurations into their CI/CD pipelines and local development workflows.

Multi-language SDKs

Pulumi ESC can also be used directly in your TypeScript/JavaScript, Go, or Python code using one of our language-specific SDKs. From the comfort of your favorite IDE, you can create and manage environments, retrieve dynamic secrets and configuration, and automate tagging and versioning directly from your CI/CD workflows.