Secrets Management for Machine Learning Pipelines with Vault

Question

Pulumi · Accepted Answer

Managing secrets in machine learning (ML) pipelines is crucial for protecting sensitive data such as API keys, database credentials, and other confidential information. HashiCorp's Vault is a widely adopted solution for secrets management due to its robust access control mechanisms, secret leasing/renewal, and numerous secret engines.

In a Pulumi context, you can use HashiCorp's Vault to supply secrets to your cloud infrastructure in a secure and auditable way. Pulumi integrates with Vault using the `pulumi_vault` provider, allowing you to manage Vault resources and retrieve secrets for your cloud resources during deployment.

Below, you'll find a Python program that demonstrates how you might use Pulumi with the Vault provider. This Pulumi program:

1. Sets up a Vault server (in a real-world scenario, you would have this already running).
2. Configures a secret within Vault for a hypothetical database password.
3. Generates a machine learning pipeline infrastructure, such as an AWS S3 bucket, where the secret from Vault is used as an environment variable for an AWS Lambda function that could be part of the ML pipeline.

```python
import pulumi
import pulumi_vault as vault
import pulumi_aws as aws

# Configuring a Vault 'secret' backend that will store the secrets
# In this example, we create a new secret backend, however, in a real-world use case,
# you would likely interact with an existing secret backend.
secret_backend = vault.Mount("secret-backend", path="secret", type="kv", description="Key/Value secret storage")

# Writing a secret to the key/value store in Vault using the `vault.GenericSecret` resource.
# This could be a database password or any other sensitive piece of data required by your application.
db_password = vault.GenericSecret("db-password",
    path="database/config",
    data_json=pulumi.Output.secret("{ \"password\": \"my-secure-password\" }").apply(lambda s: s),
    mount=secret_backend.path)

# Retrieve the secret to be used in our cloud infrastructure
# Using `.get_secret()` method to read a secret from Vault.
db_password_data = vault.get_generic_secret(secret_id=db_password.id)

# Create an AWS S3 Bucket for ML Data Storage
ml_data_bucket = aws.s3.Bucket("mlDataBucket")

# Create an AWS Lambda function, which would be part of your ML pipeline.
# The function uses the Vault secret, `db_password_data`, as an environment variable.
ml_pipeline_lambda = aws.lambda_.Function("mlPipelineLambda",
    role=my_lambda_role.arn,
    runtime="python3.8",
    handler="handler.main",
    code=pulumi.FileArchive("./lambda.zip"),
    environment=aws.lambda_.FunctionEnvironmentArgs(
        variables={
            # Assigning the secret value from Vault to the LAMBDA_DB_PASSWORD environment variable.
            "LAMBDA_DB_PASSWORD": db_password_data.data["password"]
        },
    ))

# Export relevant URLs and ARNs so they can be used with other parts of your infrastructure
# or by other Pulumi programs.
pulumi.export("ml_data_bucket_name", ml_data_bucket.id)
pulumi.export("ml_pipeline_lambda_arn", ml_pipeline_lambda.arn)
```

This program starts with setting up a Vault `secret` backend (which you'd typically have already running in a production environment). It writes a secret to Vault and retrieves it using Pulumi's Vault integration. In this scenario, an AWS Lambda function, which might be a component of a machine learning pipeline, retrieves a database password from Vault and uses it for its operations.

The secret is never exposed in the Pulumi state or console output, as we ensure to use Pulumi's secret handling (the `pulumi.Output.secret` wrapper and the `vault.get_generic_secret` data source) to keep the actual values encrypted.

Please adapt paths, resource names, runtime choices, and deployment packages (`./lambda.zip`) as necessary to match your specific environment and needs. In a typical workflow, the Vault infrastructure would be separate from your application stack, and you would only interact with Vault to retrieve or configure secrets.