Fine-grained Access Control for AI Secrets with Databricks Secret ACLs

Question

Pulumi · Accepted Answer

Fine-grained access control for secrets in Databricks allows you to define precisely who has access to sensitive information such as tokens, passwords, and keys. This is achieved by using Databricks Secret Access Control Lists (ACLs). In Pulumi, we can manage these Databricks secret ACLs using the `pulumi_databricks` package.

Below you'll find a Pulumi program written in Python that creates a secret scope, a secret inside that scope, and then sets up an ACL for a specific principal (like a user or a group) to control the level of permission they have over that secret.

### Step-by-step Explanation

1. **Create a Secret Scope**: A secret scope is a secure container for storing secrets. The secrets stored in the scope are available to jobs, clusters, and notebooks that have access to the scope.

2. **Create a Secret**: Inside the scope, we can create a secret that holds the sensitive information that needs to be protected.

3. **Set Up an ACL**: Finally, we define an Access Control List for the secret which specifies the permissions for a given principal. The principal can be a user or a group, and the permissions decide what operations they can perform on the secret like `READ`, `WRITE`, or `MANAGE`.

Here is the Pulumi program:

```python
import pulumi
import pulumi_databricks as databricks

# Create a secret scope
secret_scope = databricks.SecretScope("ai-secret-scope",
    initial_manage_principal="users")

# Create a secret in the above scope
secret = databricks.Secret("ai-secret",
    string_value="sensitive-value",
    key="super-secret-key",
    scope=secret_scope.name)

# Set up an ACL for specific principal
secret_acl = databricks.SecretAcl("ai-secret-acl",
    principal="user@example.com",
    permission="READ",
    scope=secret_scope.name)

# Export the secret ID and Secret ACL ID
pulumi.export("secret_id", secret.id)
pulumi.export("secret_acl_id", secret_acl.id)
```

### How to Use This Program

This Pulumi program would be part of a larger codebase where you manage your Databricks environment. Let's break down how you'd use this program:

- Install Pulumi and the appropriate provider package (`pulumi_databricks`).
- Write this code in a file, say `main.py`.
- Use `pulumi up` from the command line in the same directory as your `main.py` to deploy this infrastructure as code.

### Important Things to Note

- You must have Databricks and Pulumi set up and configured before running this program.
- Replace `"user@example.com"` with the actual principal you wish to grant access to.
- The `initial_manage_principal` is set to `"users"` which means all users have the ability to manage the secrets within this scope. You may want to restrict this to a specific user or group for additional security.

This code will ensure that you can version control and automate the creation of secret management policies within Databricks, which is crucial for maintaining a secure and well-governed data platform.