Inter-Workspace Communication via Databricks MWS Networks

Question

Pulumi · Accepted Answer

To enable inter-workspace communication via Databricks Machine Learning Workspace (MWS) Networks, you will need to define network, workspace, and VPC endpoint resources. The following program will create a MWS Network and a MWS Workspace within that network in Databricks.

The `databricks.MwsNetworks` resource represents a network in Databricks that can be attached to one or more workspaces. This network allows for secure communication within the Databricks environment and can connect to your VPC for resource access within your cloud provider's network.

The `databricks.MwsWorkspaces` resource represents a Databricks workspace, which is an environment for accessing Databricks services and managing their workloads. A workspace can be associated with a MWS Network to handle networking related to the workspace's tasks.

The `databricks.MwsVpcEndpoint` resource allows you to create VPC endpoints for the network, enabling secure connections to other VPC resources.

This Pulumi program below sets up a simple Databricks MWS Network and a Workspace attached to it. It assumes you already have a Pulumi project setup and your Databricks provider is properly configured with the required access credentials.

```python
import pulumi
import pulumi_databricks as databricks

# Replace these variables with your own information
aws_account_id = 'YOUR_AWS_ACCOUNT_ID'
aws_region = 'YOUR_AWS_REGION'
subnet_ids = ['YOUR_SUBNET_ID_LIST']  # List of subnet IDs

# Create a Databricks MWS Network
mws_network = databricks.MwsNetworks("mws-network",
    vpc_id="YOUR_VPC_ID",  # Your own VPC ID
    subnet_ids=subnet_ids,  # Subnets for the network
    security_group_ids=["YOUR_SECURITY_GROUP_ID"],  # Security group for the network
    network_name="my-network"  # A name for the network
)

# Create a Databricks MWS Workspace within the network
mws_workspace = databricks.MwsWorkspaces("mws-workspace",
    account_id=aws_account_id,
    aws_region=aws_region,
    network_id=mws_network.network_id,  # Reference to the network created above
    workspace_name="my-workspace",  # Name for the workspace
    pricing_tier="premium"  # Pricing tier for the workspace
)

# Export the IDs of the created resources
pulumi.export("network_id", mws_network.network_id)
pulumi.export("workspace_id", mws_workspace.workspace_id)
```

Before you run this program, ensure the following:

1. Replace `YOUR_AWS_ACCOUNT_ID`, `YOUR_AWS_REGION`, `YOUR_SUBNET_ID_LIST`, `YOUR_VPC_ID`, and `YOUR_SECURITY_GROUP_ID` with your actual AWS account details.
2. You have followed the Databricks provider setup instructions for Pulumi, which includes setting up the appropriate credentials for authentication.

Once this infrastructure is provisioned, we can secure intra-network communication and allow integration with other AWS services through the created VPC endpoints. This setup is fundamental for environments where data security and networking isolation are crucial for inter-workspace communication within Databricks.