Automated Secret Rotation for AI Workloads Using Databricks SecretScope

Question

Pulumi · Accepted Answer

To set up an automated secret rotation for AI workloads using Databricks SecretScope, you will typically want to coordinate between your cloud provider's secret management service (such as AWS Secrets Manager) and Databricks. The process involves creating a secret in the secret management service, setting up a SecretScope in Databricks, and then defining policies to handle the rotation of the secret automatically.

In AWS, the `aws.secretsmanager.Secret` resource represents a secret and `aws.secretsmanager.SecretRotation` sets up the rotation policy. In Databricks, `databricks.SecretScope` can be used to create a scope, and `databricks.Secret` resources to add secrets to that scope.

Here's a Pulumi Python program that demonstrates how to automate secret rotation for AI workloads using Databricks SecretScopes:

```python
import pulumi
import pulumi_aws as aws
import pulumi_databricks as databricks

# Create a secret in AWS Secrets Manager
secret = aws.secretsmanager.Secret("aiSecret",
    description="Secret for AI workload",
    # 'name' is optional; if not provided, AWS generates a name.
)

# Define the rotation function for the secret
# This requires an existing AWS Lambda function that handles the rotation logic
rotation_function = aws.lambda_.Function("rotationFunction")  # Placeholder for an existing Lambda function

# Set the rotation policy for the secret
secret_rotation = aws.secretsmanager.SecretRotation("aiSecretRotation",
    secret_id=secret.id,
    rotation_lambda_arn=rotation_function.arn,
    rotation_rules=aws.secretsmanager.SecretRotationRotationRulesArgs(
        automatically_after_days=30,
    ),
)

# Create a SecretScope in Databricks
databricks_secret_scope = databricks.SecretScope("aiSecretScope",
    initial_manage_principal="users",
    # 'name' is optional; if not provided, Databricks generates a name.
)

# Add the AWS secret to the Databricks SecretScope
databricks_secret = databricks.Secret("aiSecretInScope",
    string_value=secret.id.apply(lambda secret_id: f"{{secretsmanager:{secret_id}}}"),
    key="aiSecretKey",
    scope=databricks_secret_scope.name,
)

# Export the Databricks SecretScope name
pulumi.export("databricks_secret_scope_name", databricks_secret_scope.name)
```

Explanation:

- We create an AWS secret using the `aws.secretsmanager.Secret` resource. This secret will store sensitive data for your AI workloads.
- The secret rotation policy is defined using the `aws.secretsmanager.SecretRotation` resource. It references an existing AWS Lambda function through the `rotation_lambda_arn` attribute. This function contains the logic to rotate the secret safely.
- Then, we create a SecretScope in Databricks with the `databricks.SecretScope` resource. The scope functions as a namespace for related secrets.
- The secret from AWS Secrets Manager is added to the Databricks SecretScope with the `databricks.Secret` resource. It uses the ID of the AWS secret, and Databricks references the secret value using the AWS secret manager's syntax.
- We export the SecretScope's name using `pulumi.export` so that you can use it for reference in your applications or further automation tasks.

This program expects that you have configured Pulumi with the necessary cloud provider credentials and installed the required Pulumi providers for AWS and Databricks. The AWS Lambda function for rotating the secret would need to be created beforehand, as it contains specific logic for updating the secret value which depends on the use case.

Finally, ensure that your Databricks workspace is configured to allow integration with AWS Secrets Manager and has permissions to access the secrets.