Automated Credential Rotation for AI Workloads using Kubernetes External Secrets

Question

Pulumi · Accepted Answer

Credential rotation is a security best practice that involves changing credentials like passwords at regular intervals to minimize the risk of unauthorized access. For workloads running on Kubernetes, automation around credential rotation can be crucial, especially for AI applications which may continuously access sensitive data.

To achieve this in Kubernetes, you could make use of a combination of cloud provider secrets management (like AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager), HashiCorp Vault for secrets storage and access control, along with the External Secrets Operator in Kubernetes to sync these secrets into the Kubernetes cluster as native Secret objects.

In this program, I'll demonstrate how to use Pulumi to set up a system that:
1. Creates a secret in AWS Secrets Manager.
2. Schedules automatic rotation for that secret using AWS's lambda-based rotation.
3. Deploys the Kubernetes External Secrets operator to your Kubernetes cluster (using the `kubernetes` package).
4. Creates a Kubernetes ExternalSecret that references the AWS secret, so when the secret is rotated in AWS Secrets Manager, the new value is automatically synchronized to the Kubernetes Secret.

Here's a Pulumi program written in Python to accomplish this:

```python
import pulumi
import pulumi_aws as aws
import pulumi_kubernetes as kubernetes

# Step 1: Create a secret in AWS Secrets Manager
secret = aws.secretsmanager.Secret("aiWorkloadSecret",
    description="This secret contains credentials for AI Workload",
)

# Step 2: Schedule automatic rotation for the secret using AWS's lambda rotation
# Note: In a real-world scenario, you would set up a Lambda function which implements the
#       rotation logic. Here we're just providing a dummy ARN for demonstration purposes.
lambda_function_arn = "arn:aws:lambda:region:account-id:function:FunctionName"
rotation = aws.secretsmanager.SecretRotation("aiWorkloadSecretRotation",
    secret_id=secret.id,
    rotation_lambda_arn=lambda_function_arn,
    rotation_rules={
        "automaticallyAfterDays": 30,
    },
)

# Step 3: Deploy the Kubernetes External Secrets Operator to your Kubernetes cluster
# Note: This assumes that you have a configured Kubernetes cluster available. In practice,
#       include the installation steps for the External Secrets Operator (e.g., using a Helm chart).
#       For simplicity, those steps are not shown here.

# Step 4: Create a Kubernetes ExternalSecret that references the AWS secret
external_secret = kubernetes.apiextensions.CustomResource("aiWorkloadExternalSecret",
    api_version="kubernetes-client.io/v1",
    kind="ExternalSecret",
    metadata={"name": "ai-workload-external-secret"},
    spec={
        "backendType": "secretsManager",
        "data": [{
            "key": secret.id,
            "name": "credentials",
        }],
        "template": {
            "type": "Opaque",
        },
    })

# Export the secret ARN and the ExternalSecret name for reference
pulumi.export("secret_arn", secret.arn)
pulumi.export("external_secret_name", external_secret.metadata["name"])
```

In this program:

- We first create an AWS Secrets Manager secret meant to hold the credentials for an AI workload.
- We then provision a Secrets Manager SecretRotation with a dummy Lambda function ARN and specify that the secret should be rotated every 30 days. Please note that you must implement a real Lambda function for the rotation logic to work.
- Afterwards, we assume that the External Secrets Operator is already installed on your Kubernetes cluster, which is why the deployment step for the operator isn't included. You would typically install this using a Helm chart or applying the YAML manifest directly.
- Finally, we create an `ExternalSecret` Custom Resource, pointing to our managed AWS secret. The External Secrets Operator will watch this resource and create a corresponding Kubernetes `Secret` with the credentials.

This setup will ensure that when the secret is rotated in AWS Secrets Manager, the new credentials will be pulled into your Kubernetes environment automatically, keeping your AI workloads secure without manual intervention.

Remember to replace the placeholder for `lambda_function_arn` with the ARN of a real Lambda function that you have prepared for secret rotation. The Lambda function's implementation will vary depending on the type of credential and the service for which the credential is used.

When it comes to deployment, make sure you have configured Pulumi with the appropriate cloud provider credentials and have access to the Kubernetes cluster where the External Secrets Operator will be deployed. This configuration step is typically done outside of the Pulumi program using the Pulumi CLI or the respective cloud provider's CLI.