Pulumi ESC: Kubernetes cluster access

Overview

Pulumi ESC integrates with Kubernetes to help developers and operators connect to Kubernetes clusters using centrally-managed configuration and credentials. This works with Kubernetes CLI tools such as kubectl and helm, and with Pulumi programs that use the Kubernetes provider for Pulumi.

Prerequisites

To complete the steps in this guide, you will need to install the following prerequisites:

• the Pulumi CLI
• a Kubernetes cluster
• kubectl

Access Kubernetes clusters from command line tools

ESC integrates with Kubernetes client tools by setting the KUBECONFIG environment variable to point to a generated Kubernetes configuration file.

The kubeconfig content is determined by the ESC environment definition, and may include information from a variety of sources, including Pulumi IaC stack outputs.

Generate a kubeconfig file

The following environment defines a kubeconfig file to connect to a local cluster:

values:
  files:
    KUBECONFIG: |
      apiVersion: v1
      kind: Config
      clusters:
      - cluster:
          server: https://127.0.0.1:6443
        name: docker-desktop

To try it, create an environment with pulumi env init, then open it with pulumi env edit and paste in the definition. You can also create and edit environments in the Pulumi Cloud console.

$ pulumi env init <your-project-name>/<your-environment-name>
$ pulumi env edit <your-project-name>/<your-environment-name>

The Kubernetes configuration is picked up automatically by the kubectl command:

$ pulumi env run <your-project-name>/<your-environment-name> -- kubectl get namespaces

This command opens the environment you created, renders the kubeconfig as a temporary file, sets the KUBECONFIG environment variable to refer to the file, and then runs a kubectl command.

Use stack outputs to generate a kubeconfig file

Instead of hardcoding a kubeconfig, you also have the option of using ESC providers to dynamically import cluster information from stack outputs of Pulumi IaC programs. The pulumi-stacks ESC provider allows you to read Pulumi IaC stack outputs (in this example, a kubeconfig for an EKS cluster) at runtime and map those values to environment variables, to temporary files, or directly to other Pulumi IaC programs as stack configuration values. Check out the Getting Started guide if you need help setting up a Kubernetes cluster.

The following environment defines a kubeconfig file based on a stack output:

values:
  stacks:
    fn::open::pulumi-stacks:
      stacks:
        eks-cluster:
          stack: eks-cluster/demo
  kubeconfig: {'fn::toJSON': "${stacks.eks-cluster.kubeconfig}"}
  files:
    KUBECONFIG: ${kubeconfig}

This definition loads a Pulumi stack named eks-cluster/demo, serializes the kubeconfig output to a JSON string, and then defines a file named KUBECONFIG with that string.

Use the kubeconfig in a Pulumi IaC program

To use the kubeconfig in a Pulumi program, use the pulumiConfig section of your environment file to directly configure the Pulumi Kubernetes provider.

The following environment configures the kubernetes:kubeconfig setting on the Kubernetes provider.

values:
  stacks:
    fn::open::pulumi-stacks:
      stacks:
        eks-cluster:
          stack: eks-cluster/demo
  kubeconfig: {'fn::toJSON': "${stacks.eks-cluster.kubeconfig}"}
  pulumiConfig:
    kubernetes:kubeconfig: ${kubeconfig}

In practice, your environment should define both a kubeconfig file and a provider configuration to reuse the same environment for both CLI access and Pulumi IaC.

Next, import your environment into your Pulumi project. Open your Pulumi.<your-stack-name>.yaml file and add an environment entry, replacing <your-project-name>/<your-environment-name> with the identifier of your own environment:

environment:
  - <your-project-name>/<your-environment-name>

Authenticate to a Kubernetes cluster

ESC enables you to connect to your Kubernetes cluster using credentials obtained from an ESC provider. For example, you can connect to an Amazon EKS cluster using AWS OIDC credentials returned by the aws-login provider.

Kubernetes tools typically execute an external command to obtain user credentials (see documentation). These credential plugin commands rely on environment variables and/or configuration files, and we’ll use Pulumi ESC to set up the necessary values.

Configure AWS credentials

Taking an EKS cluster as an example, your environment can produce temporary AWS credentials using the aws-login provider. The choice of AWS and EKS is only for the purposes of this example. Pulumi ESC has similar providers to get temporary credentials for Microsoft Azure and Google Cloud.

Save the following definition as its own environment, for example demos/aws-access, so that other environments can import it (see Import another environment).

values:
  aws:
    creds:
      fn::open::aws-login:
        oidc:
          duration: 1h
          roleArn: arn:aws:iam::0123456789:role/cluster-admin
          sessionName: <your-project-name>/<your-environment-name>
  environmentVariables:
    AWS_ACCESS_KEY_ID: ${aws.creds.accessKeyId}
    AWS_SECRET_ACCESS_KEY: ${aws.creds.secretAccessKey}
    AWS_SESSION_TOKEN: ${aws.creds.sessionToken}

This definition sets the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables that are understood by the aws-iam-authenticator exec plugin, as described here. This assumes that the kubeconfig is configured appropriately to use the authenticator.

Import another environment

A good practice is to compose an environment by importing other environments. Consider encapsulating your cloud credentials in a separate environment file, and reference the environment as necessary.

The following environment imports the demos/aws-access environment from Configure AWS credentials to incorporate credentials, and then prepares a kubeconfig for kubectl and for the Pulumi Kubernetes provider.

imports:
  - demos/aws-access
values:
  stacks:
    fn::open::pulumi-stacks:
      stacks:
        eks-cluster:
          stack: eks-cluster/demo
  kubeconfig: {'fn::toJSON': "${stacks.eks-cluster.kubeconfig}"}
  pulumiConfig:
    kubernetes:kubeconfig: ${kubeconfig}
  files:
    KUBECONFIG: ${kubeconfig}

When this environment is opened, expect to see a combination of environment variables and files.