Pulumi ESC: Kubernetes Cluster Access

Overview

Pulumi ESC integrates with Kubernetes to help developers and operators connect to Kubernetes clusters using centrally-managed configuration and credentials. This works with Kubernetes CLI tools such as kubectl and helm, and with Pulumi programs that use the Kubernetes provider for Pulumi.

Prerequisites

To complete the steps in this tutorial, you will need to install the following prerequisites:

• the Pulumi ESC CLI
• a Kubernetes cluster
• kubectl

Access Kubernetes Clusters from Command Line Tools

ESC integrates with Kubernetes client tools by setting the KUBECONFIG environment variable to point to a generated Kubernetes configuration file.

The kubeconfig content is determined by the ESC environment definition, and may include information from a variety of sources, including Pulumi IaC stack outputs.

Generate a kubeconfig file

The following environment defines a kubeconfig file to connect to a local cluster:

values:
  files:
    KUBECONFIG: |
      apiVersion: v1
      kind: Config
      clusters:
      - cluster:
          server: https://127.0.0.1:6443
        name: docker-desktop      

The Kubernetes configuration is picked up automatically by the kubectl command:

$ esc run <your-project-name>/<your-environment-name> -- kubectl get namespaces

This command opens the environment you just created, renders the kubeconfig as a temporary file, sets the KUBECONFIG environment variable to refer to the file, and then runs a kubectl command.

Use stack outputs to generate a kubeconfig file

Instead of hardcoding a kubeconfig, you also have the option of using ESC providers to dynamically import cluster information from stack outputs of Pulumi IaC programs. The pulumi-stacks ESC provider allows you to read Pulumi IaC stack outputs (in this example, a kubeconfig for an EKS cluster) at runtime and map those values to environment variables, to temporary files, or directly to other Pulumi IaC programs as stack configuration values. Check out the Getting Started guide if you need help setting up a Kubernetes cluster.

The following environment defines a kubeconfig file based on a stack output:

values:
  stacks:
    fn::open::pulumi-stacks:
      stacks:
        eks-cluster:
          stack: eks-cluster/demo
  kubeconfig: {'fn::toJSON': "${stacks.eks-cluster.kubeconfig}"}
  files:
    KUBECONFIG: ${kubeconfig}

This definition loads a Pulumi stack named eks-cluster/demo, parses the kubeconfig output as a JSON object, and then defines a file named KUBECONFIG with the object.

Use the kubeconfig in a Pulumi IaC program

To use the kubeconfig in a Pulumi program, use the pulumiConfig section of your environment file to directly configure the Pulumi Kubernetes provider.

The following environment configures the kubernetes:kubeconfig setting on the Kubernetes provider.

values:
  stacks:
    fn::open::pulumi-stacks:
      stacks:
        eks-cluster:
          stack: eks-cluster/demo
  kubeconfig: {'fn::toJSON': "${stacks.eks-cluster.kubeconfig}"}
  pulumiConfig:
    kubernetes:kubeconfig: ${kubeconfig}

In practice, your environment should define both a kubeconfig file and a provider configuration in order to reuse the same environment for both CLI access and Pulumi IaC.

Next, you will need to import your environment file into your Pulumi project. To do this, open your Pulumi.<your-stack-name>.yaml file and update it to import your environment as shown below, making sure to replace the value of <your-project-name>/<your-environment-name> with the identifier of your own environment:

environment:
  - <your-project-name>/<your-environment-name>

Authenticate to a Kubernetes Cluster

ESC enables you to connect to your Kubernetes cluster using credentials obtained from an ESC provider. For example, to connect to an AWS EKS cluster using AWS OIDC credentials returned by the aws-login provider.

Kubernetes tools typically execute an external command to obtain user credentials (see documentation). These credential plugin commands rely on environment variables and/or configuration files, and we’ll use Pulumi ESC to set up the necessary values.

Configure AWS credentials

Taking an EKS cluster as an example, your environment can produce temporary AWS credentials using the aws-login provider. The choice of AWS and EKS is only for the purposes of this example. Pulumi ESC has similar plugins to obtain temporary credentials for Microsoft Azure and Google Cloud.

values:
  aws:
    creds:
      fn::open::aws-login:
        oidc:
          duration: 1h
          roleArn: arn:aws:iam::0123456789:role/cluster-admin
          sessionName: <your-project-name>/<your-environment-name>
  environmentVariables:
    AWS_ACCESS_KEY_ID: ${aws.creds.keyId}
    AWS_SECRET_ACCESS_KEY: ${aws.creds.secretAccessKey}
    AWS_SESSION_TOKEN: ${aws.creds.sessionToken}

This definition sets the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables that are understood by the aws-iam-authenticator exec plugin, as described here. This assumes that the kubeconfig is configured appropriately to use the authenticator.

Import another environment

A good practice is to compose an environment by importing other environment(s). Consider encapsulating your cloud credentials in a separate environment file, and reference the environment as necessary.

The following environment imports an environment named aws-access within the demos project to incorporate credentials, and then prepares a kubeconfig for kubectl and for the Pulumi Kubernetes provider.

imports:
  - demos/aws-access
values:
  stacks:
    fn::open::pulumi-stacks:
      stacks:
        eks-cluster:
          stack: eks-cluster/demo
  kubeconfig: {'fn::toJSON': "${stacks.eks-cluster.kubeconfig}"}
  pulumiConfig:
    kubernetes:kubeconfig: ${kubeconfig}
  files:
    KUBECONFIG: ${kubeconfig}

When this environment is opened, expect to see a combination of environment variables and files.