Skip to main content

Integrate with External Secrets Operator (ESO)

15 min

External Secrets Operator is a Kubernetes operator that integrates external secret management systems with Kubernetes. By using External Secrets Operator, you have several advantages over Kubernetes native secrets:

  • Sensitive data is stored and managed by an external service outside the Kubernetes cluster and this leads to better security and compliance.
  • External Secrets Operator supports multiple different sources, so you can use the same operator to manage secrets and configuration from different sources.
  • Take advantage of the advanced features of the secret provider, such as encryption of data at rest and scenarios like secret rotation.

Since version 0.10.0 External Secrets Operator supports Pulumi ESC as a secret provider.

What you'll learn
  • How to deploy the External Secrets Operator using Helm or Pulumi
  • How to manage secrets stored in Pulumi ESC using External Secrets Operator
  • How to push secrets from Kubernetes to Pulumi ESC
Prerequisites

Deploy External Secrets Operator#

Deploy using Helm#

Install from Helm Chart Repository#

Terminal window
# add ESO Operator Chart repo into helm
helm repo add external-secrets https://charts.external-secrets.io
helm repo update


# installs latest ESO Operator Chart, creates also namespace
helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --wait

Create secret containing Pulumi access token#

Terminal window
# create a local k8s secret to store(PAT), to allow accessing Pulumi Cloud Resources
kubectl create secret generic pulumi-access-token --from-literal=PULUMI_ACCESS_TOKEN=${PULUMI_ACCESS_TOKEN} \
    --namespace external-secrets

Create ClusterSecretStore#

You can create a SecretStore resource to notify the External Secrets Operator to use Pulumi ESC as a secret provider for a specific namespace.

However, if you want to expand the scope to an entire cluster you can use a ClusterSecretStore resource instead.

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: secret-store
  namespace: external-secrets
spec:
  provider:
    pulumi:
      organization: {PULUMI_ORG_NAME}
      project: {ESC_PROJECT_NAME}
      environment: {ESC_ENV_NAME}
      accessToken:
        secretRef:
          name: pulumi-access-token
          key: PULUMI_ACCESS_TOKEN
          namespace: external-secrets
EOF

Please replace ${PULUMI_ORG_NAME}, ${ESC_PROJECT_NAME}, ${ESC_ENV_NAME} with your Pulumi organization, project, and environment names.

For demo purposes, we assume that we already have an ESC environment my-org/my-project/my-env with a secret my-secret that we want to manage using the External Secrets Operator.

values:
  hello: world

Create ExternalSecret#

Now you can create an ExternalSecret resource that will tell External Secrets Operator to fetch the secret from Pulumi ESC. There is also an ClusterExternalSecret resource, which is a cluster scoped resource that can be used to manage ExternalSecret resources in specific namespaces.

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: secret
  namespace: external-secrets
spec:
  data:
  - secretKey: esc-secret
    remoteRef:
      key: hello
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: secret-store
EOF

There a many other options available for ExternalSecret resource to control the creation of the secret like rewriting keys, templating and more. You can find more information in the External Secrets Operator documentation.

Verify the secret#

With the following command, you can verify that the secret has been created in the cluster:

Terminal window
kubectl get secret secret --namespace external-secrets -o jsonpath='{.data.esc-secret}'  | base64 -d
# Output:
world

Cleanup#

To remove the External Secrets Operator and the created resources, you can use the following commands:

Terminal window
kubectl delete externalsecret secret
kubectl delete clustersecretstore secret-store
kubectl delete secret pulumi-access-token -n external-secrets
helm uninstall external-secrets -n external-secrets

Deploy using Pulumi#

Of course, you can also deploy the External Secrets Operator using Pulumi. The following examples shows how to deploy the External Secrets Operator on some of the Pulumi supported languages.

Select your Pulumi supported language#

Create a new Pulumi program in your preferred language and add the following code to deploy the External Secrets Operator.

Terminal window
pulumi new <your-preferred-language> --name external-secrets-operator

Add the following code to the index.ts, index.py, or main.go file:

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";


const config = new pulumi.Config();


const externalSecretsNamespace = new kubernetes.core.v1.Namespace("external-secrets-namespace", {
    metadata: {
        name: "external-secrets",
    }
});
const externalSecretsRelease = new kubernetes.helm.v3.Release("external-secrets-release", {
    chart: "external-secrets",
    version: "0.10.4",
    namespace: externalSecretsNamespace.metadata.apply(metadata => metadata.name),
    repositoryOpts: {
        repo: "https://charts.external-secrets.io",
    },
});
const patSecret = new kubernetes.core.v1.Secret("patSecret", {
    metadata: {
        namespace: externalSecretsNamespace.metadata.apply(metadata => metadata.name),
        name: "pulumi-access-token",
    },
    stringData: {
        PULUMI_ACCESS_TOKEN: config.require("pulumi-pat"),
    },
    type: "Opaque",
});


const clusterSecretStore = new kubernetes.apiextensions.CustomResource("cluster-secret-store", {
    apiVersion: "external-secrets.io/v1beta1",
    kind: "ClusterSecretStore",
    metadata: {
        name: "secret-store",
    },
    spec: {
        provider: {
            pulumi: {
                organization: pulumi.getOrganization(),
                project: "dirien",
                environment: "hello-world",
                accessToken: {
                    secretRef: {
                        name: patSecret.metadata.name,
                        key: "PULUMI_ACCESS_TOKEN",
                        namespace: patSecret.metadata.namespace,
                    },
                },
            },
        },
    },
}, { dependsOn: externalSecretsRelease });


const externalSecret = new kubernetes.apiextensions.CustomResource("external-secret", {
    apiVersion: "external-secrets.io/v1beta1",
    kind: "ExternalSecret",
    metadata: {
        name: "secret",
        namespace: "default",
    },
    spec: {
        data: [
            {
                secretKey: "esc-secret",
                remoteRef: {
                    key: "hello",
                }
            }
        ],
        refreshInterval: "1h",
        secretStoreRef: {
            kind: clusterSecretStore.kind,
            name: clusterSecretStore.metadata.name,
        }
    },
}, { dependsOn: externalSecretsRelease });

You can then deploy the Pulumi program using below command as you would normally do with any Pulumi program:

Terminal window
pulumi up

Cleanup#

To remove the External Secrets Operator and the created resources, you can use the following commands:

Terminal window
pulumi destroy

Push Secrets to Pulumi ESC#

The Pulumi ESC provider for External Secrets Operator supports also PushSecrets. This feature allows you to push secrets from a Kubernetes cluster to Pulumi ESC. This is useful, if you have for example other operators that create secrets in the cluster and you want automatically push them to Pulumi ESC for further management.

Here is an example of a PushSecret resource:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: push-secret-example
spec:
  refreshInterval: 1h
  selector:
    secret:
      name: <NAME_OF_THE_KUBERNETES_SECRET>
  secretStoreRefs:
  - kind: ClusterSecretStore
    name: secret-store
  data:
  - match:
      secretKey: <KEY_IN_KUBERNETES_SECRET>
      remoteRef:
        remoteKey: <PULUMI_PATH_SYNTAX>

Let’s see PushSecret in action:

This is how out demo Pulumi ESC environment looks like before we push the secret:

img.png

Only one configuration hello: world is present. Now, assume that another process (for example an application, operator, etc.) creates during runtime a Kubernetes secret.

This secret is called my-secret and contains the following data:

apiVersion: v1
data:
  world: aGVsbG8=
kind: Secret
metadata:
  name: test-cred
  namespace: default
type: Opaque

The world key contains the value hello. We can now create a PushSecret resource that will select the secret test-cred and push the world key to the Pulumi ESC environment.

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: pushsecret-example # Customisable
  namespace: default # Same of the SecretStores
spec:
  refreshInterval: 1h
  selector:
    secret:
      name: test-cred
  secretStoreRefs:
  - kind: ClusterSecretStore
    name: secret-store
  data:
  - match:
      secretKey: world
      remoteRef:
        remoteKey: world

This PushSecret resource will push the world key from the test-cred secret to the Pulumi ESC environment. After applying this resource, the Pulumi ESC environment will look like this:

img.png

This is a great way to manage secrets that are created by other processes in the cluster and ensure that they are stored in a secure and compliant way reducing the risk of secret sprawl.

Next steps#

This tutorial showed you how to deploy the External Secrets Operator and use it to manage secrets stored in Pulumi ESC. This gives you the ability to elevate Pulumi ESC as the single source of truth for your secrets even when you are not using Pulumi to manage your Kubernetes resources.

As we continue to improve the integration between Pulumi ESC and External Secrets Operator, we highly recommend you to check the External Secrets Operator Pulumi ESC for the latest features.

To dive deeper into using Pulumi ESC for advanced scenarios, check out the following resources:

  • Environment Composition: Learn more about to effectively compose multiple environments to manage configurations across your infrastructure. Explore the Pulumi documentation on environment imports.

  • Managing Secrets: Learn how to securely manage and adopt dynamic, short-lived secrets on demand using Pulumi ESC, ensuring sensitive information is protected across different environments. Read more in the Pulumi ESC documentation.

KubernetesTypeScriptPythonGoYAMLPulumi ESCSecrets

Related

Tutorial

Integrate Pulumi with Secret Store CSI Driver in Kubernetes

Learn how to integrate the Secret Store CSI Driver with Pulumi ESC to securely manage your secrets in Kubernetes.
15 min
Tutorial

Secure Kubernetes Application Secrets with Pulumi ESC

Securely manage secrets and configurations in a Kubernetes app with Pulumi ESC.
15 min

The infrastructure as code platform for any cloud.

Contact usGet started