1. Automated Secret Rotation for AI Workloads Using Databricks SecretScope


    To set up an automated secret rotation for AI workloads using Databricks SecretScope, you will typically want to coordinate between your cloud provider's secret management service (such as AWS Secrets Manager) and Databricks. The process involves creating a secret in the secret management service, setting up a SecretScope in Databricks, and then defining policies to handle the rotation of the secret automatically.

    In AWS, the aws.secretsmanager.Secret resource represents a secret and aws.secretsmanager.SecretRotation sets up the rotation policy. In Databricks, databricks.SecretScope can be used to create a scope, and databricks.Secret resources to add secrets to that scope.

    Here's a Pulumi Python program that demonstrates how to automate secret rotation for AI workloads using Databricks SecretScopes:

    import pulumi import pulumi_aws as aws import pulumi_databricks as databricks # Create a secret in AWS Secrets Manager secret = aws.secretsmanager.Secret("aiSecret", description="Secret for AI workload", # 'name' is optional; if not provided, AWS generates a name. ) # Define the rotation function for the secret # This requires an existing AWS Lambda function that handles the rotation logic rotation_function = aws.lambda_.Function("rotationFunction") # Placeholder for an existing Lambda function # Set the rotation policy for the secret secret_rotation = aws.secretsmanager.SecretRotation("aiSecretRotation", secret_id=secret.id, rotation_lambda_arn=rotation_function.arn, rotation_rules=aws.secretsmanager.SecretRotationRotationRulesArgs( automatically_after_days=30, ), ) # Create a SecretScope in Databricks databricks_secret_scope = databricks.SecretScope("aiSecretScope", initial_manage_principal="users", # 'name' is optional; if not provided, Databricks generates a name. ) # Add the AWS secret to the Databricks SecretScope databricks_secret = databricks.Secret("aiSecretInScope", string_value=secret.id.apply(lambda secret_id: f"{{secretsmanager:{secret_id}}}"), key="aiSecretKey", scope=databricks_secret_scope.name, ) # Export the Databricks SecretScope name pulumi.export("databricks_secret_scope_name", databricks_secret_scope.name)


    • We create an AWS secret using the aws.secretsmanager.Secret resource. This secret will store sensitive data for your AI workloads.
    • The secret rotation policy is defined using the aws.secretsmanager.SecretRotation resource. It references an existing AWS Lambda function through the rotation_lambda_arn attribute. This function contains the logic to rotate the secret safely.
    • Then, we create a SecretScope in Databricks with the databricks.SecretScope resource. The scope functions as a namespace for related secrets.
    • The secret from AWS Secrets Manager is added to the Databricks SecretScope with the databricks.Secret resource. It uses the ID of the AWS secret, and Databricks references the secret value using the AWS secret manager's syntax.
    • We export the SecretScope's name using pulumi.export so that you can use it for reference in your applications or further automation tasks.

    This program expects that you have configured Pulumi with the necessary cloud provider credentials and installed the required Pulumi providers for AWS and Databricks. The AWS Lambda function for rotating the secret would need to be created beforehand, as it contains specific logic for updating the secret value which depends on the use case.

    Finally, ensure that your Databricks workspace is configured to allow integration with AWS Secrets Manager and has permissions to access the secrets.