Vault-Managed Credentials for Kubernetes-Based Training Jobs

Question

Pulumi · Accepted Answer

When you want to run training jobs on a Kubernetes cluster and manage the credentials for those jobs with HashiCorp Vault, you will need a way to allow the Kubernetes cluster to authenticate with Vault and retrieve the necessary secrets at runtime. For this purpose, we'll configure Vault to work with Kubernetes, create a Kubernetes job that references a secret for accessing an external service, and use Vault's Kubernetes auth method.

Here's how you can achieve this using Pulumi:

Setup Vault's Kubernetes Authentication: You need to configure Vault to accept authentication requests from Kubernetes pods. This involves setting up a Vault role that defines the access policies and Kubernetes service account names tied to this role.
Define a Kubernetes Job: With Pulumi, you create a Kubernetes Job resource that runs your training workload. This job will reference a Kubernetes secret which theoretically could contain credentials received from Vault.
Vault Integration: Normally, you would use an init container or sidecar in your Kubernetes job to authenticate with Vault and write the credentials to a shared volume. However, Pulumi's Kubernetes Job doesn't directly support interacting with Vault.

For this demonstration, we're going to create a Secret resource that could be updated to get its values from Vault using external logic. Typically, you'd use an operator or a custom script that uses the Vault SDK to authenticate using the Kubernetes service account token, read the desired secrets, and create/update the Kubernetes secret.

Here's a Pulumi program in Python that outlines these steps. In a real scenario, you would integrate with Vault programmatically to populate the Kubernetes secret dynamically.

import pulumi
import pulumi_kubernetes as kubernetes
import pulumi_vault as vault

# Assuming you already have Vault deployed and running 

# Step 1: Setup Vault's Kubernetes Authentication
# Configures a Vault auth method for Kubernetes by providing the necessary settings,
# such as the Kubernetes host, CA certificate, and so on.
auth_backend_config = vault.kubernetes.AuthBackendConfig("k8s-auth",
    kubernetes_host="<K8S_HOST>",
    kubernetes_ca_cert="<K8S_CA_CERT>",
    token_reviewer_jwt="<TOKEN_REVIEW_JWT>",
)

# Define a role in Vault that connects the Kubernetes service account to Vault policies.
# This role will determine which policies are applied when a Kubernetes service account authenticates.
vault_role = vault.kubernetes.Role("vault-role",
    bound_service_account_names=["my-service-account"],
    bound_service_account_namespaces=["default"],
    policies=["my-vault-policy"], # Replace with the name of your Vault policy
    ttl="24h",
)

# Step 2: Define a Kubernetes Job
# Creates a Kubernetes job with a simple workload, this is where your training job would run.
# For simplicity, we're using a dummy 'echo' command.
training_job = kubernetes.batch.v1.Job("training-job",
    metadata=kubernetes.meta.v1.ObjectMetaArgs(
        name="training-job",
    ),
    spec=kubernetes.batch.v1.JobSpecArgs(
        template=kubernetes.core.v1.PodTemplateSpecArgs(
            spec=kubernetes.core.v1.PodSpecArgs(
                containers=[kubernetes.core.v1.ContainerArgs(
                    name="training-container",
                    image="busybox",
                    command=["/bin/sh"],
                    args=["-c", "echo 'Starting Training Job'; sleep 30; echo 'Training Job Completed'"],
                )],
                restart_policy="Never",
            ),
        ),
    ),
)

# Step 3: Define a Secret in Kubernetes for storing credentials.
# In a real-world case, we would retrieve these credentials from Vault directly, but for this
# example, we just use static values.
training_job_secret = kubernetes.core.v1.Secret("training-job-secret",
    metadata=kubernetes.meta.v1.ObjectMetaArgs(
        name="training-job-secret",
    ),
    type="Opaque",
    string_data={
        # These fields would be dynamically populated with credentials from Vault.
        "api_key": "example_key",
        "api_secret": "example_secret",
    },
)

# Export the job name
pulumi.export("job_name", training_job.metadata.name)
# Export the secret name that would store the credentials
pulumi.export("secret_name", training_job_secret.metadata.name)

For the Vault integration to work properly in a production setup, you would implement a workflow wherein:

A Kubernetes service account token is used to authenticate with Vault.
The pulumi_vault package might provide direct Vault integration, but typically you'll be using an operator like Vault Secrets Operator or writing custom code to fetch and inject secrets into your Kubernetes jobs.

Remember, when working with secrets and sensitive information, always ensure that you're following best practices for secret management.