Kubernetes External Secrets Management for Machine Learning Ops

Question

Pulumi · Accepted Answer

Managing external secrets in Kubernetes is essential for maintaining the security and integrity of your infrastructure, particularly in Machine Learning Operations (MLOps), where sensitive data such as API keys, database credentials, and model parameters often need to be securely stored and accessed.

Pulumi provides various ways to manage Kubernetes secrets, leveraging both native Kubernetes means and integrating with external secret management systems like HashiCorp Vault or cloud-provided secrets managers. For the sake of this demonstration, we’ll utilize the native Kubernetes secret management using a Pulumi program.

Below is a Pulumi Python program that demonstrates how you can create a Kubernetes Secret resource to manage your sensitive data. This secret can then be referenced in your Kubernetes deployments, ensuring that sensitive information like database credentials or API keys are never hard-coded into your version control system.

The `pulumi_kubernetes` package is used to interact with Kubernetes resources. We first import the necessary Pulumi Kubernetes package and create a new Secret resource. The `metadata` dictionary provides information about the Secret, such as its name and namespace. In the `data` field, we provide the actual secrets; however, each secret needs to be base64 encoded.

In a real-world scenario, you would retrieve the secret data from a secure location during runtime or keep it in an environment variable instead of hard-coding it in your program. Also, the use of `.apply()` method is shown in the program to translate `base64` encoded values into a readable format for exporting.

```python
import pulumi
import pulumi_kubernetes as kubernetes
import base64

# Configuration for the Secret
app_secrets = {
    "DATABASE_PASSWORD": "my-secure-database-password",
    "API_KEY": "my-secret-api-key"
}

# A helper function to encode our secrets to base64 before providing them to the Kubernetes Secret
def encode_secret(data):
    return {k: base64.b64encode(v.encode('utf-8')).decode('utf-8') for k, v in data.items()}

# Define the Kubernetes Secret resource
k8s_secret = kubernetes.core.v1.Secret(
    "ml-app-secret",
    metadata=kubernetes.meta.v1.ObjectMetaArgs(
        name="ml-app-secret",
        namespace="mlops"
    ),
    # The actual secrets are defined here. Values are base64 encoded.
    data=encode_secret(app_secrets)
)

# Export the Secret name to be used in other resources or outputted in the CLI
pulumi.export("secret_name", k8s_secret.metadata.apply(lambda metadata: metadata.name))
```

In the `pulumi.export` statement, we're exporting the name of the secret so that it can be easily referenced in the future, for example by a deployment that needs to consume the secret values.

To use the secret within a Kubernetes pod, you would reference the secret name in your pod's spec under `envFrom` or as individual environment variables under `env`.

Please note that you should never store plain text sensitive information in your version control system. Always retrieve this information securely during your deployment process rather than hard coding it into your Pulumi program.