Pulumi ESC Environments

Pulumi ESC (Environments, Secrets, and Configuration) lets you define collections of configuration settings and secrets called environments and use them in any application or service. Environments are YAML documents composed of static key-value pairs, programmatic expressions, dynamically retrieved values from supported providers including all major clouds through OpenID Connect (OIDC), and other Pulumi ESC environments.

Environments are accessible with the standalone esc CLI, the pulumi CLI, the Pulumi SDK, and the Pulumi Cloud console and REST API, and you can have as many environments as you need. Pulumi ESC is a service of Pulumi Cloud and is currently in public preview.

Creating a new environment

To create a new, empty environment, use esc env init [<org-name>/]<environment-name>, where <org-name> is optional and defaults to your Pulumi Cloud username.

Environment names must be unique within an organization and may only contain alphanumeric characters, hyphens, underscores, and periods.

$ esc env init myorg/test
Environment created.

Listing environments

To list the environments you have access to, use esc env ls:

$ esc env ls
myorg/test

You can filter this list to a particular organization by passing its name:

$ esc env ls --organization myorg
myorg/test

Getting and setting environment values

Setting a value

To set a new value or update an existing value, use esc env set <key> <value>:

$ esc env set myorg/test foo bar

Getting a value

To retrieve a single value and its definition, use esc env get <environment-name> <key>:

$ esc env get myorg/test foo

   Value

    "bar"

   Definition

    bar


   Defined at

  • test:2:10

Getting all values

To retrieve all values in an environment, run esc env get <environment-name>:

$ esc env get myorg/test

   Value

    {
      "foo": "bar"
    }

   Definition

    values:
      foo: bar

You can also get values in other formats, such as json, dotenv, or shell:

$ esc env get myorg/test --value json
{
  "foo": "bar"
}

$ esc env get myorg/test foo --value json
"bar"

Note that the dotenv and shell options return values only when an environment defines one or more environment variables. See Projecting environment variables for details.

Structured configuration

Structured data like maps and arrays can also be set with esc env set <environment-name> <key> <value> using object-property (.) or array ([]) syntax:

$ esc env set myorg/test 'data.active' true
$ esc env set myorg/test 'data.nums[0]' 1
$ esc env set myorg/test 'data.nums[1]' 2
$ esc env get myorg/test

  Value

  {
    "data": {
      "active": true,
      "nums": [
        1,
        2
      ]
    },
    "foo": "bar"
  }

  Definition

  values:
    foo: bar
    data:
      active: true
      nums:
        - 1
        - 2

Boolean and numeric values are implicitly converted and persisted in their respective types.

You can also fetch individual values at any level in the hierarchy. For example, to fetch the second item from the nested array in the example above:

$ esc env get myorg/test 'data.nums[1]'

   Value

    2

   Definition

    2


   Defined at

  • test:7:15

Interpolating values

Values don’t have to be static. You can also use string interpolation to transform values or compose new values from other values. For example:

$ esc env set myorg/test salutation Hello
$ esc env set myorg/test name World
$ esc env set myorg/test greeting '${salutation}, ${name}'
$ esc env get myorg/test greeting

   Value

    "Hello, World"

   Definition

    ${salutation}, ${name}


   Defined at

  • test:10:15

Pulumi Contextual information

Contextual information is available and can be used to interpolate on any environment value.

values:
  personalNamespace: ${context.rootEnvironment.name}/${context.pulumi.user.login}

It can be accessed through the context attribute with the following options:

• context.rootEnvironment.name: the name of the root environment being evaluated
• context.currentEnvironment.name: the name of the current environment being evaluated
• context.pulumi.user.login: the user login identifier
• context.pulumi.organization.login: the organization login identifier

Editing environments

Environments may be edited in a number of ways.

With the Pulumi ESC CLI

To edit an environment using your shell’s default text editor (as defined by the $EDITOR environment variable), use esc env edit <environment-name>:

$ esc env edit myorg/test

When $EDITOR is unset, esc uses vi as a fallback:

values:
    foo: bar
    data:
        active: true
        nums:
            - 1
            - 2
    salutation: Hello
    name: World
    greeting: ${salutation}, ${name}
---
# Please edit the environment definition above.
# The object below is the current result of
# evaluating the environment and will not be
# saved. An empty definition aborts the edit.

{
  "data": {
    "active": true,
    "nums": [
      1,
      2
    ]
  },
  "foo": "bar",
  "greeting": "Hello, World",
  "name": "World",
  "salutation": "Hello"
}
~
~
~
"/tmp/2198639483.yaml" 29L, 528B

You can also specify the editor you want to use by passing its name with the --editor flag:

$ esc env edit --editor="code" myorg/test

Doing so overrides any value set in $EDITOR. See esc env edit --help for more options.

In the Pulumi Cloud console

To edit an environment in the Pulumi Cloud console, select your organization, choose Environments in the left-hand menu, and select the environment you wish to edit:

Click Save to apply your changes immediately or Delete to remove the environment.

With the Pulumi Cloud REST API

You can also use the Pulumi Cloud REST API to perform standard CRUD operations on your environments. See the Environments section of the REST API docs for details.

Using secrets providers and OIDC

In addition to static and interpolated values, environments can incorporate dynamically retrieved settings and secrets from many supported providers, including cloud providers via OpenID Connect (OIDC).

The following example combines the aws-login and aws-secrets providers to obtain short-lived credentials from AWS to pull two secrets from AWS Secrets Manager (api-key and app-secret) into an environment:

values:
  aws:
    login:
      fn::open::aws-login:
        oidc:
          roleArn: arn:aws:iam::01234567891011:role/some-role
          sessionName: some-session

    secrets:
      fn::open::aws-secrets:
        region: us-west-1
        login: ${aws.login}
        get:
          api-key:
            secretId: api-key
          app-secret:
            secretId: app-secret

Opening an environment

As mentioned, esc env get doesn’t resolve dynamic or secret values — for example, those managed with third-party services like AWS Secrets Manager. Instead, to gain access to these values, you must open the environment.

To do so, use esc env open <environment-name>:

$ esc env open myorg/test
{
  "data": {
    "active": true,
    "nums": [
      1,
      2
    ]
  },
  "foo": "bar",
  "greeting": "Hello, World",
  "name": "World",
  "salutation": "Hello",
  "aws": {
    "login": {
      "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
      "secretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE",
      "sessionToken": "eyJpc3MiOiJPbmxpbmUgSldUIEJ1aWxkZXIiLCJpYXQiOEXAMPLE"
    },
    "secrets": {
      "api-key": "{\"keyName\":\"keyValue\"}",
      "app-secret": "{\"secretName\":\"secretValue\"}"
    }
  }
}

You can also use open to resolve the value of an individual key:

$ esc env open myorg/test 'data.active'
true

$ esc env open myorg/test 'data.nums'
[
  1,
  2
]

$ esc env open myorg/test 'aws.secrets.app-secret' | jq -r 'fromjson | . .secretName'
secretValue

Projecting environment variables

Pulumi ESC can automatically project the settings of a given environment as a set of environment variables. This projection does not happen by default, however; instead, you must define which settings to project, as well as how to name and format them.

Environment variables are defined under the optional environmentVariables key, which can accept either static or interpolated values based on settings defined within the environment or imported from other environments:

values:
  aws:
    login:
      fn::open::aws-login:
        oidc:
          roleArn: arn:aws:iam::01234567891011:role/some-role
          sessionName: some-session

  environmentVariables:
    AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
    AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
    AWS_SESSION_TOKEN: ${aws.login.sessionToken}
    MY_ENV_VAR: "true"

To render these environmentVariables for use in the shell, use esc env open, passing either --format shell or --format dotenv:

$ esc env open myorg/test --format shell
export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE"
export AWS_SESSION_TOKEN="eyJpc3MiOiJPbmxpbmUgSldUIEJ1aWxkZXIiLCJpYXQiOEXAMPLE"
export MY_ENV_VAR="true"

$ esc env open myorg/test --format dotenv
AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE"
AWS_SESSION_TOKEN="eyJpc3MiOiJPbmxpbmUgSldUIEJ1aWxkZXIiLCJpYXQiOEXAMPLE"
MY_ENV_VAR="true"

Export them into the current shell using eval or similar:

$ eval $(esc env open myorg/test --format shell)
$ echo $MY_ENV_VAR
true

Running commands with environment variables

You can also run CLI commands directly, using environment variables obtained with Pulumi ESC — without having to export those variables into the shell first.

To do this, use esc run <environment-name> <command>:

$ esc run myorg/test aws s3 ls
2023-10-10 16:09:19 my-s3-bucket

If you need to pass one or more flags to the command, prefix the command with --:

$ esc run myorg/test -- aws s3 ls s3://my-s3-bucket --recursive --summarize
...
Total Objects: 5087
   Total Size: 2419123156

For additional options and details, see esc run --help.

Importing other environments

Environments can also be composed from other environments.

Different applications are often configured in similar ways and with common values — for example, an e-commerce site and order-management system both configured to use the same cloud account, database-connection string, and third-party API key. Managing the duplication of these values across multiple configuration files, however, can be difficult, especially when one of those values changes — e.g., when an API key is regenerated.

To address these challenges, Pulumi ESC allows you to identify common or closely related configuration settings and define them only once, as individual environments, and then import those environments into other, more specialized environments as needed. Imports also allow you to expose certain environments without having to distribute any concrete values and to delegate responsibility for particular environments to other teams in your organization. Environments can import both static and dynamic values, including secrets, from any number of other environments.

In the following example, two environments, aws-dev and stripe-dev, are used to compose a third environment, myapp-dev:

# myorg/aws-dev
values:
  aws:
    region: us-west-2
    login:
      fn::open::aws-login:
        static:
          accessKeyId:
            fn::secret: AKIAIOSFODNN7EXAMPLE
          secretAccessKey:
            fn::secret: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLE

# myorg/stripe-dev
values:
  stripe:
    apiURL: https://api.stripe.com
    apiKey:
      fn::secret: sk_XemWAl12i4x3hZhp4vBKDEXAMPLE

The application-specific myapp-dev environment then imports these two environments and use their settings to compose new values:

# myorg/myapp-dev
imports:
  - aws-dev
  - stripe-dev

values:
  greeting: Hello from the dev environment!

  environmentVariables:
    AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
    AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
    STRIPE_API_KEY: ${stripe.apiKey}
    STRIPE_API_URL: ${stripe.apiURL}
    GREETING: ${greeting}

Finally, esc run renders myapp-dev’s environment variables for use on the command line:

$ esc run myorg/myapp-dev -- bash -c 'echo $GREETING'
Hello from the dev environment!

$ esc run myorg/myapp-dev -- bash -c 'echo $STRIPE_API_URL'
https://api.stripe.com

$ esc run myorg/myapp-dev -- bash -c 'echo $STRIPE_API_KEY'
[secret]

$ esc run myorg/myapp-dev -- bash -c 'echo $AWS_SECRET_ACCESS_KEY'
[secret]

$ echo "'$GREETING'"
''

Notice in the example that the environmentVariables were exposed to the bash command, but not to the surrounding shell, and that the values marked as secrets with fn::secret were protected from exposure.

Using environments with Pulumi IaC

With support for Pulumi ESC built into the Pulumi CLI, you can expose an environment’s settings and secrets to any or all of your Pulumi stacks, bypassing the need to define and maintain individual configuration settings or secrets “locally” in Pulumi config files. The optional pulumiConfig key enables this.

The following example updates the myorg/myapp-dev environment by adding a pulumiConfig block. This block specifies the Pulumi configuration settings to expose to the Pulumi stack at runtime:

# myorg/myapp-dev
imports:
  - aws-dev
  - stripe-dev

values:
  greeting: Hello from the dev environment!

  environmentVariables:
    AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
    AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
    STRIPE_API_KEY: ${stripe.apiKey}
    STRIPE_API_URL: ${stripe.apiURL}
    GREETING: ${greeting}

  # Add a `pulumiConfig` block to expose these settings to your Pulumi stacks.
  pulumiConfig:
    aws:region: ${aws.region}
    stripeApiKey: ${stripe.apiKey}
    stripeApiURL: ${stripe.apiURL}
    greeting: ${greeting}

Any stack belonging to the myorg organization can inherit these settings by adding the optional environment block to its stack-configuration file:

# Pulumi.dev.yaml
environment:
  - myapp-dev

Values are accessible using the standard configuration API:

// index.ts
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Import the values using the standard Pulumi configuration API.
const config = new pulumi.Config();
const greeting = config.require("greeting");
const stripeApiKey = config.requireSecret("stripeApiKey");
const stripeApiURL= config.requireSecret("stripeApiURL");

const callbackFunction = new aws.lambda.CallbackFunction("callback", {
    callback: async () => ({
        statusCode: 200,
        body: JSON.stringify({
            greeting,

            // Use them in your program as would any config value.
            stripeApiURL: process.env.STRIPE_API_URL,
         }),
    }),.
    environment: {
        variables: {
            STRIPE_API_URL: stripeApiURL,
        },
    },
});

const functionUrl = new aws.lambda.FunctionUrl("url", {
    functionName: callbackFunction.name,
    authorizationType: "NONE",
});

export const url = functionUrl.functionUrl;

Stacks may only read from environments that belong to the same Pulumi organization.

Precedence rules

When multiple environment sources are combined and settings overlap, values are applied successively in the order in which they’re imported and defined.

For example, in the following scenario, three environments define a key foo, each with a different value. The third environment, environment-c, imports environment-a and environment-b (importantly, in that order):

# environment-a
values:
  foo: bar

# environment-b
imports:
  - environment-a
values:
  foo: baz
  pulumiConfig:
    foo: ${foo}

# environment-c
imports:
  - environment-a
  - environment-b
values:
  foo: qux
  pulumiConfig:
    foo: ${foo}

Notice how the value of foo is overwritten with each successive environment:

$ esc env open environment-a foo
"bar"

$ esc env open environment-b foo
"baz"

$ esc env open environment-c foo
"qux"

$ pulumi preview
Diagnostics:
  pulumi:pulumi:Stack (dev):
    { foo: 'qux' }

Also notice that when the local definition of foo is removed from environment-c and its imports are reordered, the value of foo changes to reflect the value inherited from environment-a — i.e., the last-imported one:

# environment-c
imports:
  - environment-b
  - environment-a
values:
  pulumiConfig:
    foo: ${foo}

$ esc env open environment-c foo
"bar"

$ pulumi preview
Diagnostics:
  pulumi:pulumi:Stack (dev):
    { foo: 'bar' }

To unset a value inherited from another environment, overwrite it with null.

Setting up access to environments

Organization-wide permissions

Go to the Access Management page under Settings to set Organization-wide environment permissions. Members of the organization will receive these permissions. By default, the environment permissions for the organization is set to write. There are four options:

• none: Members have access to none of the environments
• read: Members can view only plaintext key values (i.e., the definition of the environment). They won’t be able to see the secret values in plaintext, run any provider configurations to retrieve credentials or run any functions. They cannot perform any Pulumi IaC operations such as refresh, up, destroy on stacks that imports the environment
• open: Members with ‘open’ permissions can decrypt secrets and see them in plaintext. Additionally, they can get dynamic credentials using provider configurations and evaluate functions defined in the environment. They can perform any Pulumi IaC operation on stacks that import an environment as long as they have ‘write’ access to the stack and ‘open’ access to the environment
• write: Members will have permissions to open and update any environment

Team permissions

You can grant environment-wise permissions to members of a Team. There are four roles:

• Environment reader: Team members will have read permissions
• Environment opener: Team members will have open permissions
• Environment editor: Team members will have write permissions
• Environment admin: Team members will have write and delete permissions

Deleting an environment

To remove an environment, use esc env rm [<org-name>/]<environment-name>:

$ esc env rm myorg/test
This will permanently remove the "myorg/test" environment!
Please confirm that this is what you'd like to do by typing `myorg/test`: myorg/test
Environment "myorg/test" has been removed!

Environments cannot be renamed.