Controlling Permissions for Azure Databricks Collaborative Environments

Question

Pulumi · Accepted Answer

Controlling permissions for collaborative environments like Azure Databricks involves managing access to different resources within the workspace. Azure Databricks uses a RBAC (Role-Based Access Control) system, where access is granted based on the roles assigned to users, groups, and service principals. Unfortunately, Pulumi does not directly expose advanced permission settings for Databricks as part of its resource providers at the time of this writing.

However, I can show you how to provision an Azure Databricks workspace using Pulumi's `azure-native` package and provide basic guidance on where to look next to manage permissions within the Databricks environment.

To create an Azure Databricks workspace, you would define resources in a Pulumi program which outline your cloud infrastructure. This includes the Databricks workspace and any necessary supporting infrastructure like a resource group.

Here's a Pulumi Python program example that provisions an Azure Databricks workspace:

```python
import pulumi
from pulumi_azure_native import resources
from pulumi_azure_native import databricks

# Create an Azure Resource Group
resource_group = resources.ResourceGroup('resource_group')

# Create an Azure Databricks workspace in the resource group
databricks_workspace = databricks.Workspace('databricks_workspace',
    resource_group_name=resource_group.name,
    location=resource_group.location,
    sku=databricks.WorkspaceSkuArgs(
        name="standard"
    ),
    tags={
        "Environment": "dev"
    }
)

# Export the Azure Databricks Workspace URL
pulumi.export('databricks_workspace_url', databricks_workspace.workspace_url)
```

In this program:

1. The `pulumi_azure_native` package is imported, which contains all the necessary classes to work with Azure resources.
2. A new Azure resource group is provisioned using the `ResourceGroup` class, providing a namespace in which our Azure Databricks workspace will live.
3. The `Workspace` class from `pulumi_azure_native.databricks` is used to provision an Azure Databricks workspace within the resource group we created earlier. The SKU (which provides the pricing tier and capabilities) is specified as "standard".
4. We tag the workspace with "Environment: dev" to indicate that this workspace is allocated for development purposes.
5. Finally, we export the URL of the Azure Databricks workspace so it can be accessed after deployment.

After deploying your Databricks workspace, you can manage access within the Azure portal by navigating to the Azure Databricks workspace and using the Access Control (IAM) settings. There, you can assign roles to users, groups, and service principals according to your organization's policies.

To set more granular permissions within Databricks itself, such as on notebooks, clusters, jobs, etc., you would typically use the Databricks REST API or the Databricks CLI, implemented as part of a custom provisioning script or automation tool. As of now, this level of granular permission control is not directly available through Pulumi, but you can manage these settings post-deployment via the Databricks UI or API.