1. Inter-Workspace Communication via Databricks MWS Networks


    To enable inter-workspace communication via Databricks Machine Learning Workspace (MWS) Networks, you will need to define network, workspace, and VPC endpoint resources. The following program will create a MWS Network and a MWS Workspace within that network in Databricks.

    The databricks.MwsNetworks resource represents a network in Databricks that can be attached to one or more workspaces. This network allows for secure communication within the Databricks environment and can connect to your VPC for resource access within your cloud provider's network.

    The databricks.MwsWorkspaces resource represents a Databricks workspace, which is an environment for accessing Databricks services and managing their workloads. A workspace can be associated with a MWS Network to handle networking related to the workspace's tasks.

    The databricks.MwsVpcEndpoint resource allows you to create VPC endpoints for the network, enabling secure connections to other VPC resources.

    This Pulumi program below sets up a simple Databricks MWS Network and a Workspace attached to it. It assumes you already have a Pulumi project setup and your Databricks provider is properly configured with the required access credentials.

    import pulumi import pulumi_databricks as databricks # Replace these variables with your own information aws_account_id = 'YOUR_AWS_ACCOUNT_ID' aws_region = 'YOUR_AWS_REGION' subnet_ids = ['YOUR_SUBNET_ID_LIST'] # List of subnet IDs # Create a Databricks MWS Network mws_network = databricks.MwsNetworks("mws-network", vpc_id="YOUR_VPC_ID", # Your own VPC ID subnet_ids=subnet_ids, # Subnets for the network security_group_ids=["YOUR_SECURITY_GROUP_ID"], # Security group for the network network_name="my-network" # A name for the network ) # Create a Databricks MWS Workspace within the network mws_workspace = databricks.MwsWorkspaces("mws-workspace", account_id=aws_account_id, aws_region=aws_region, network_id=mws_network.network_id, # Reference to the network created above workspace_name="my-workspace", # Name for the workspace pricing_tier="premium" # Pricing tier for the workspace ) # Export the IDs of the created resources pulumi.export("network_id", mws_network.network_id) pulumi.export("workspace_id", mws_workspace.workspace_id)

    Before you run this program, ensure the following:

    2. You have followed the Databricks provider setup instructions for Pulumi, which includes setting up the appropriate credentials for authentication.

    Once this infrastructure is provisioned, we can secure intra-network communication and allow integration with other AWS services through the created VPC endpoints. This setup is fundamental for environments where data security and networking isolation are crucial for inter-workspace communication within Databricks.