Implementing secret scopes for securing sensitive data access in Databricks

Question

Pulumi · Accepted Answer

When managing sensitive data, it's crucial to ensure that access is properly secured. In Databricks, one way to achieve this is by using secret scopes. Secret scopes are used to store and reference secrets securely within Databricks workspaces. Here's how you can implement secret scopes using Pulumi:

1. **databricks.SecretScope**: This resource allows you to create a secret scope, a designated container for secrets. You can specify the backend type, like Databricks or Azure Key Vault, for the storage of these secrets.

2. **databricks.Secret**: Within your secret scope, you can then define individual secrets, such as database passwords or API keys, using this resource. Secrets are referenced within your Databricks workspaces and notebooks but their actual values are hidden and stored securely.

3. **databricks.Permissions**: To control access to the secret scopes and secrets, you use permissions. This resource lets you set who can access these secrets at a granular level, ensuring that only authorized users or processes can retrieve them.

Below is a TypeScript program that creates a Databricks secret scope with a secret and sets permissions for access. Please ensure you have configured your Pulumi with the appropriate Databricks provider settings.

```typescript
import * as databricks from "@pulumi/databricks";

// Create a new secret scope
const mySecretScope = new databricks.SecretScope("my-secret-scope", {
    name: "my-secret-scope",
    backendType: "DATABRICKS", // or "AZURE_KEYVAULT"
    // If using Azure KeyVault, you would include `keyvaultMetadata` here.
});

// Add a secret to the scope
const mySecret = new databricks.Secret("my-secret", {
    key: "my-secret-key",
    stringValue: "my-secret-value", // This should be the actual secret content
    scope: mySecretScope.name,
});

// Set permissions on the secret scope
const scopePermissions = new databricks.Permissions("my-secret-scope-permissions", {
    // Replace with appropriate object type and ID
    objectType: "secretscope",
    objectId: mySecretScope.id,
    accessControls: [{
        // Replace with the actual user or group name
        userName: "example-user",
        permissionLevel: "MANAGE", // Choices include "READ", "WRITE", "MANAGE"
    }],
});

export const secretScopeName = mySecretScope.name;
export const secretId = mySecret.id;
```

In this program:
- We create a secret scope named `my-secret-scope`.
- We add a secret to the scope with a key `my-secret-key` and a value `my-secret-value`.
- We set permissions to allow the user `example-user` to manage the secret scope.

Please make sure to replace `"my-secret-value"` with the actual sensitive data you intend to secure, and manage it securely, maybe through environment variables or a secure store, not embedded directly in your code.

The `export` statements at the end of the program output the secret scope name and secret ID, which can be useful for auditing, tracking, and referencing in other parts of your Pulumi program.

*Note: The above program simplifies sensitive data handling by showing a hardcoded string. In practice, you should use Pulumi Config to handle sensitive data securely, avoiding placing it directly in your source code.*

For more information on working with Databricks resources in Pulumi, you can consult the [Pulumi Registry](https://www.pulumi.com/registry/packages/databricks/).