Secure Multi-User Data Science Workspaces with Databricks Entitlements

Question

Pulumi · Accepted Answer

To create secure multi-user data science workspaces using Databricks on Pulumi, we will utilize various Databricks resources, including entitlements, to control permissions and access within the workspaces.

The core resources will include:

1. **Workspaces (`MwsWorkspaces`)** - The workspace is an environment for accessing all Databricks assets. Each workspace is an isolated environment with its own set of clusters, notebooks, jobs, and databases.

2. **Permissions (`Permissions`)** - Manages access control for various Databricks assets. You would use this to assign permissions to users within your workspace.

3. **Entitlements (`Entitlements`)** - Controls specific user entitlements that allow or deny certain actions on the platform, such as creating clusters or accessing the workspace.

4. **Group Membership (`GroupMember`)** - Allows you to manage group memberships. In Databricks, groups are collections of users that you can manage collectively.

5. **User Roles (`UserRole`)** - Associates a Databricks user with a role that comes with an associated set of permissions.

Before we proceed with the Pulumi code, make sure you have the Pulumi CLI installed, an active Pulumi account, the appropriate Databricks and cloud provider credentials set up, and have selected the appropriate Pulumi stack you want to deploy to.

Below is a sample Pulumi program written in Python that creates a multi-user data science workspace with defined entitlements for two hypothetical users, ensuring a secure workspace environment. Please note that the code assumes that appropriate service principals and permissions are already in place for the account that you are using.

```python
import pulumi
import pulumi_databricks as databricks

# Create a Databricks workspace
workspace = databricks.MwsWorkspaces("myWorkspace",
    # Ensure that the account ID and other sensitive inputs are fetched from secure storage or environment variables
    account_id=pulumi.Config('databricks').require_secret('accountId'),
    cloud="aws",  # Specify the cloud provider where Databricks is hosted, in this case AWS
    workspace_name="my-secure-workspace",
    pricing_tier="premium",  # Choose an appropriate pricing tier for your workspace
    aws_region="us-west-2"  # Specify the AWS region for your workspace
)

# Define entitlements for user A
user_a_entitlements = databricks.Entitlements("userAEntitlements",
    user_id="userA@example.com",  # Replace with the actual user ID of user A
    allow_cluster_create=True,
    workspace_access=True  # Granting workspace access to user A
)

# Define entitlements for user B
user_b_entitlements = databricks.Entitlements("userBEntitlements",
    user_id="userB@example.com",  # Replace with the actual user ID of user B
    allow_cluster_create=False,  # User B will not be entitled to create clusters
    workspace_access=True  # Granting workspace access to user B
)

# Assign permissions within the workspace (e.g., using built-in roles)
user_a_permissions = databricks.Permissions("userAPermissions",
    object_type="workspace",
    access_controls=[
        {
            "user_name": user_a_entitlements.user_id,
            "permission_level": "CAN_MANAGE"
        }
    ]
)

# Similarly, you can create groups, add users to groups, or assign group-based permissions

pulumi.export("workspaceUrl", workspace.workspace_url)  # Export the workspace URL for access
```

In this program, we established a secure workspace on Databricks and assigned different entitlements and permissions to two hypothetical users. The `MwsWorkspaces` resource creates a new workspace environment for our data assets and users. We use the `Entitlements` resource to determine what actions each user can perform within the Databricks environment.

Finally, the `Permissions` resource allows us to define fine-grained access control policies per user or group. You can extend these with additional configurations to further refine your environment.

In a production setup, you would typically use Pulumi's `Config` class to handle secrets and sensitive data securely. Always avoid hardcoding sensitive information like the account IDs or user names in your code.

Please adjust the user IDs and entitlement settings based on the specific requirements of your team and organizational policies. Ensure that you have the correct permissions and the necessary service principal and user IDs created within Databricks before running this code.