Private Networking for Secure Databricks Deployments

Question

Pulumi · Accepted Answer

When configuring private networking for secure Databricks deployments on the cloud, it is generally preferred to create an isolated network where your Databricks clusters will operate. This private network will not be accessible from the public internet, ensuring that your data processing tasks remain secure.

To establish private networking for Databricks, you typically need:

- A Virtual Network (VNet) which acts as your own private cloud network.
- Subnets, which are segments within your VNet, to organize resources and manage traffic internally.
- Network Security Groups (NSGs), which are used to define inbound and outbound security rules for your network.
- A Private DNS Zone, which allows you to use your own domain names within your private network, rather than public IP addresses or Azure DNS names.

Let's take an example of creating a secure network on Azure for Databricks deployments using Pulumi in Python.

Below is a Pulumi program that sets up private networking for Databricks on Azure. It creates a VNet, subnets, and network security groups, and demonstrates the use of associated security rules. We will also create a Private DNS zone for internal name resolution of services.

```python
import pulumi
import pulumi_azure_native as azure

# Create a resource group for grouping all our network resources
resource_group = azure.resources.ResourceGroup('databricks-vnet-rg')

# Create a virtual network (VNet) where we will add Databricks and associated resources
vnet = azure.network.VirtualNetwork('databricks-vnet',
    resource_group_name=resource_group.name,
    location=resource_group.location,
    address_space=azure.network.AddressSpaceArgs(
        address_prefixes=['10.0.0.0/16'],
    )
)

# Create a subnet for Databricks Cluster
databricks_subnet = azure.network.Subnet('databricks-subnet',
    resource_group_name=resource_group.name,
    virtual_network_name=vnet.name,
    address_prefix='10.0.1.0/24',
    # Network Security Group to be associated later
    delegations=[azure.network.DelegationArgs(
        name='databricks-delegation',
        service_name='Microsoft.Databricks/workspaces',
    )]
)

# Create a network security group for the Databricks subnet
nsg = azure.network.NetworkSecurityGroup('databricks-nsg',
    resource_group_name=resource_group.name,
    location=resource_group.location
)

# Create a security rule that allows inbound traffic on port 443 (HTTPS) for secure communication
security_rule = azure.network.SecurityRule('databricks-https-rule',
    resource_group_name=resource_group.name,
    network_security_group_name=nsg.name,
    priority=100,
    direction='Inbound',
    access='Allow',
    protocol='Tcp',
    source_port_range='*',
    destination_port_range='443',
    source_address_prefix='*',
    destination_address_prefix='*',
    description='Allow HTTPS Inbound'
)

# Associate NSG with Subnet
subnet_update = azure.network.Subnet('databricks-subnet-nsg-association',
    resource_group_name=resource_group.name,
    virtual_network_name=vnet.name,
    subnet_name=databricks_subnet.name,
    network_security_group=azure.network.NetworkSecurityGroupArgs(
        id=nsg.id,
    )
)

# Create a private DNS zone for internal communication within VNet
private_dns_zone = azure.network.PrivateZone('databricks-private-dns-zone',
    resource_group_name=resource_group.name,
    location='global',
    private_zone_name='privatedns.databricks.internal',
)

# Exports the VNet and subnet IDs to be used later, such as by a Databricks workspace
pulumi.export('virtual_network_id', vnet.id)
pulumi.export('databricks_subnet_id', databricks_subnet.id)
```

In this program:

- We define a resource group in Azure to contain all the networking resources.
- A virtual network (VNet) called `databricks-vnet` is created with a specified address space.
- Within the VNet, we establish a subnet called `databricks-subnet` dedicated to Databricks clusters.
- A network security group (NSG) is created to define and enforce security rules for the subnet.
- We define a security rule within the NSG to allow inbound HTTPS traffic on port 443, ensuring secure communication.
- The NSG is associated with the Databricks subnet to apply the security rules.
- A private DNS zone named `privatedns.databricks.internal` is configured for friendly DNS names within the VNet.

With this setup, you have a secure, private network to deploy Databricks workspaces and clusters that are not directly exposed to the internet, meeting best practices for privacy and security.