databricks.MwsNetworks

Databricks on AWS usage

Note Initialize provider with alias = "mws", host = "https://accounts.cloud.databricks.com" and use provider = databricks.mws

Use this resource to configure VPC & subnets for new workspaces within AWS. It is essential to understand that this will require you to configure your provider separately for the multiple workspaces resources.

• Databricks must have access to at least two subnets for each workspace, with each subnet in a different Availability Zone. You cannot specify more than one Databricks workspace subnet per Availability Zone in the Create network configuration API call. You can have more than one subnet per Availability Zone as part of your network setup, but you can choose only one subnet per Availability Zone for the Databricks workspace.
• Databricks assigns two IP addresses per node, one for management traffic and one for Spark applications. The total number of instances for each subnet is equal to half of the available IP addresses.
• Each subnet must have a netmask between /17 and /25.
• Subnets must be private.
• Subnets must have outbound access to the public network using a aws_nat_gateway, or other similar customer-managed appliance infrastructure.
• The NAT gateway must be set up in its subnet (public_subnets in the example below) that routes quad-zero (0.0.0.0/0) traffic to an internet gateway or other customer-managed appliance infrastructure.

Note The NAT gateway needs only one IP address per AZ. Hence, the public subnet only needs two IP addresses. In order to limit the number of IP addresses in the public subnet, you can specify a secondary CIDR block (cidr_block_public) using the argument secondary_cidr_blocks then pass it to the public_subnets argument. Please review the IPv4 CIDR block association restrictions when choosing the secondary cidr block.

Please follow this complete runnable example & subnet for new workspaces within GCP. It is essential to understand that this will require you to configure your provider separately for the multiple workspaces resources.

• Databricks must have access to a subnet in the same region as the workspace, of which IP range will be used to allocate your workspace’s GKE cluster nodes.
• The subnet must have a netmask between /29 and /9.
• Databricks must have access to 2 secondary IP ranges, one between /21 to /9 for workspace’s GKE cluster pods, and one between /27 to /16 for workspace’s GKE cluster services.
• Subnet must have outbound access to the public network using a gcp_compute_router_nat or other similar customer-managed appliance infrastructure.

Please follow this complete runnable example] private_subnets = [cidrsubnet(var.cidr_block, 3, 1), cidrsubnet(var.cidr_block, 3, 2)]

default_security_group_egress = [{ cidr_blocks = “0.0.0.0/0” }]

default_security_group_ingress = [{ description = “Allow all internal TCP and UDP” self = true }] }

resource “databricks.MwsNetworks” “this” { provider = databricks.mws account_id = var.databricks_account_id network_name = “${local.prefix}-network” security_group_ids = [module.vpc.default_security_group_id] subnet_ids = module.vpc.private_subnets vpc_id = module.vpc.vpc_id }

In order to create a VPC [that leverages AWS PrivateLink](https://docs.databricks.com/administration-guide/cloud-configurations/aws/privatelink.html) you would need to add the `vpc_endpoint_id` Attributes from [mws_vpc_endpoint](mws_vpc_endpoint.md) resources into the [databricks_mws_networks](databricks_mws_networks.md) resource. For example:

```hcl
resource "databricks_mws_networks" "this" {
  provider           = databricks.mws
  account_id         = var.databricks_account_id
  network_name       = "${local.prefix}-network"
  security_group_ids = [module.vpc.default_security_group_id]
  subnet_ids         = module.vpc.private_subnets
  vpc_id             = module.vpc.vpc_id
  vpc_endpoints {
    dataplane_relay = [databricks_mws_vpc_endpoint.relay.vpc_endpoint_id]
    rest_api        = [databricks_mws_vpc_endpoint.workspace.vpc_endpoint_id]
  }
  depends_on = [aws_vpc_endpoint.workspace, aws_vpc_endpoint.relay]
}

Creating a Databricks on GCP workspace

variable "databricks_account_id" {
  description = "Account Id that could be found in the top right corner of https://accounts.cloud.databricks.com/"
}

resource "google_compute_network" "dbx_private_vpc" {
  project                 = var.google_project
  name                    = "tf-network-${random_string.suffix.result}"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "network-with-private-secondary-ip-ranges" {
  name          = "test-dbx-${random_string.suffix.result}"
  ip_cidr_range = "10.0.0.0/16"
  region        = "us-central1"
  network       = google_compute_network.dbx_private_vpc.id
  secondary_ip_range {
    range_name    = "pods"
    ip_cidr_range = "10.1.0.0/16"
  }
  secondary_ip_range {
    range_name    = "svc"
    ip_cidr_range = "10.2.0.0/20"
  }
  private_ip_google_access = true
}

resource "google_compute_router" "router" {
  name    = "my-router-${random_string.suffix.result}"
  region  = google_compute_subnetwork.network-with-private-secondary-ip-ranges.region
  network = google_compute_network.dbx_private_vpc.id
}

resource "google_compute_router_nat" "nat" {
  name                               = "my-router-nat-${random_string.suffix.result}"
  router                             = google_compute_router.router.name
  region                             = google_compute_router.router.region
  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
}

resource "databricks_mws_networks" "this" {
  account_id   = var.databricks_account_id
  network_name = "test-demo-${random_string.suffix.result}"
  gcp_network_info {
    network_project_id    = var.google_project
    vpc_id                = google_compute_network.dbx_private_vpc.name
    subnet_id             = google_compute_subnetwork.network_with_private_secondary_ip_ranges.name
    subnet_region         = google_compute_subnetwork.network_with_private_secondary_ip_ranges.region
    pod_ip_range_name     = "pods"
    service_ip_range_name = "svc"
  }
}

In order to create a VPC that leverages GCP Private Service Connect you would need to add the vpc_endpoint_id Attributes from mws_vpc_endpoint resources into the databricks.MwsNetworks resource. For example:

resource "databricks_mws_networks" "this" {
  account_id   = var.databricks_account_id
  network_name = "test-demo-${random_string.suffix.result}"
  gcp_network_info {
    network_project_id    = var.google_project
    vpc_id                = google_compute_network.dbx_private_vpc.name
    subnet_id             = google_compute_subnetwork.network_with_private_secondary_ip_ranges.name
    subnet_region         = google_compute_subnetwork.network_with_private_secondary_ip_ranges.region
    pod_ip_range_name     = "pods"
    service_ip_range_name = "svc"
  }
  vpc_endpoints {
    dataplane_relay = [databricks_mws_vpc_endpoint.relay.vpc_endpoint_id]
    rest_api        = [databricks_mws_vpc_endpoint.workspace.vpc_endpoint_id]
  }
}

Modifying networks on running workspaces (AWS only)

Due to specifics of platform APIs, changing any attribute of network configuration would cause databricks.MwsNetworks to be re-created - deleted & added again with special case for running workspaces. Once network configuration is attached to a running databricks_mws_workspaces, you cannot delete it and pulumi up would result in INVALID_STATE: Unable to delete, Network is being used by active workspace X error. In order to modify any attributes of a network, you have to perform three different pulumi up steps:

1. Create a new databricks.MwsNetworks resource.
2. Update the databricks.MwsWorkspaces to point to the new network_id.
3. Delete the old databricks.MwsNetworks resource.

Related Resources

The following resources are used in the same context:

• Provisioning Databricks on AWS guide.
• Provisioning Databricks on AWS with PrivateLink guide.
• Provisioning AWS Databricks E2 with a Hub & Spoke firewall for data exfiltration protection guide.
• Provisioning Databricks on GCP guide.
• Provisioning Databricks workspaces on GCP with Private Service Connect guide.
• databricks.MwsVpcEndpoint resources with Databricks such that they can be used as part of a databricks.MwsNetworks configuration.
• databricks.MwsPrivateAccessSettings to create a Private Access Setting that can be used as part of a databricks.MwsWorkspaces resource to create a Databricks Workspace that leverages AWS PrivateLink or GCP Private Service Connect.
• databricks.MwsWorkspaces to set up workspaces in E2 architecture on AWS.

Create MwsNetworks Resource

Resources are created with functions called constructors. To learn more about declaring and configuring resources, see Resources.

Constructor syntax

new MwsNetworks(name: string, args: MwsNetworksArgs, opts?: CustomResourceOptions);

@overload
def MwsNetworks(resource_name: str,
                args: MwsNetworksArgs,
                opts: Optional[ResourceOptions] = None)

@overload
def MwsNetworks(resource_name: str,
                opts: Optional[ResourceOptions] = None,
                account_id: Optional[str] = None,
                network_name: Optional[str] = None,
                creation_time: Optional[int] = None,
                error_messages: Optional[Sequence[MwsNetworksErrorMessageArgs]] = None,
                gcp_network_info: Optional[MwsNetworksGcpNetworkInfoArgs] = None,
                network_id: Optional[str] = None,
                security_group_ids: Optional[Sequence[str]] = None,
                subnet_ids: Optional[Sequence[str]] = None,
                vpc_endpoints: Optional[MwsNetworksVpcEndpointsArgs] = None,
                vpc_id: Optional[str] = None,
                vpc_status: Optional[str] = None,
                workspace_id: Optional[int] = None)

func NewMwsNetworks(ctx *Context, name string, args MwsNetworksArgs, opts ...ResourceOption) (*MwsNetworks, error)

public MwsNetworks(string name, MwsNetworksArgs args, CustomResourceOptions? opts = null)

public MwsNetworks(String name, MwsNetworksArgs args)
public MwsNetworks(String name, MwsNetworksArgs args, CustomResourceOptions options)

type: databricks:MwsNetworks
properties: # The arguments to resource properties.
options: # Bag of options to control resource's behavior.

Parameters

Example

The following reference example uses placeholder values for all input properties.

var mwsNetworksResource = new Databricks.MwsNetworks("mwsNetworksResource", new()
{
    AccountId = "string",
    NetworkName = "string",
    CreationTime = 0,
    ErrorMessages = new[]
    {
        new Databricks.Inputs.MwsNetworksErrorMessageArgs
        {
            ErrorMessage = "string",
            ErrorType = "string",
        },
    },
    GcpNetworkInfo = new Databricks.Inputs.MwsNetworksGcpNetworkInfoArgs
    {
        NetworkProjectId = "string",
        PodIpRangeName = "string",
        ServiceIpRangeName = "string",
        SubnetId = "string",
        SubnetRegion = "string",
        VpcId = "string",
    },
    NetworkId = "string",
    SecurityGroupIds = new[]
    {
        "string",
    },
    SubnetIds = new[]
    {
        "string",
    },
    VpcEndpoints = new Databricks.Inputs.MwsNetworksVpcEndpointsArgs
    {
        DataplaneRelays = new[]
        {
            "string",
        },
        RestApis = new[]
        {
            "string",
        },
    },
    VpcId = "string",
    VpcStatus = "string",
    WorkspaceId = 0,
});

example, err := databricks.NewMwsNetworks(ctx, "mwsNetworksResource", &databricks.MwsNetworksArgs{
	AccountId:    pulumi.String("string"),
	NetworkName:  pulumi.String("string"),
	CreationTime: pulumi.Int(0),
	ErrorMessages: databricks.MwsNetworksErrorMessageArray{
		&databricks.MwsNetworksErrorMessageArgs{
			ErrorMessage: pulumi.String("string"),
			ErrorType:    pulumi.String("string"),
		},
	},
	GcpNetworkInfo: &databricks.MwsNetworksGcpNetworkInfoArgs{
		NetworkProjectId:   pulumi.String("string"),
		PodIpRangeName:     pulumi.String("string"),
		ServiceIpRangeName: pulumi.String("string"),
		SubnetId:           pulumi.String("string"),
		SubnetRegion:       pulumi.String("string"),
		VpcId:              pulumi.String("string"),
	},
	NetworkId: pulumi.String("string"),
	SecurityGroupIds: pulumi.StringArray{
		pulumi.String("string"),
	},
	SubnetIds: pulumi.StringArray{
		pulumi.String("string"),
	},
	VpcEndpoints: &databricks.MwsNetworksVpcEndpointsArgs{
		DataplaneRelays: pulumi.StringArray{
			pulumi.String("string"),
		},
		RestApis: pulumi.StringArray{
			pulumi.String("string"),
		},
	},
	VpcId:       pulumi.String("string"),
	VpcStatus:   pulumi.String("string"),
	WorkspaceId: pulumi.Int(0),
})

var mwsNetworksResource = new MwsNetworks("mwsNetworksResource", MwsNetworksArgs.builder()        
    .accountId("string")
    .networkName("string")
    .creationTime(0)
    .errorMessages(MwsNetworksErrorMessageArgs.builder()
        .errorMessage("string")
        .errorType("string")
        .build())
    .gcpNetworkInfo(MwsNetworksGcpNetworkInfoArgs.builder()
        .networkProjectId("string")
        .podIpRangeName("string")
        .serviceIpRangeName("string")
        .subnetId("string")
        .subnetRegion("string")
        .vpcId("string")
        .build())
    .networkId("string")
    .securityGroupIds("string")
    .subnetIds("string")
    .vpcEndpoints(MwsNetworksVpcEndpointsArgs.builder()
        .dataplaneRelays("string")
        .restApis("string")
        .build())
    .vpcId("string")
    .vpcStatus("string")
    .workspaceId(0)
    .build());

mws_networks_resource = databricks.MwsNetworks("mwsNetworksResource",
    account_id="string",
    network_name="string",
    creation_time=0,
    error_messages=[databricks.MwsNetworksErrorMessageArgs(
        error_message="string",
        error_type="string",
    )],
    gcp_network_info=databricks.MwsNetworksGcpNetworkInfoArgs(
        network_project_id="string",
        pod_ip_range_name="string",
        service_ip_range_name="string",
        subnet_id="string",
        subnet_region="string",
        vpc_id="string",
    ),
    network_id="string",
    security_group_ids=["string"],
    subnet_ids=["string"],
    vpc_endpoints=databricks.MwsNetworksVpcEndpointsArgs(
        dataplane_relays=["string"],
        rest_apis=["string"],
    ),
    vpc_id="string",
    vpc_status="string",
    workspace_id=0)

const mwsNetworksResource = new databricks.MwsNetworks("mwsNetworksResource", {
    accountId: "string",
    networkName: "string",
    creationTime: 0,
    errorMessages: [{
        errorMessage: "string",
        errorType: "string",
    }],
    gcpNetworkInfo: {
        networkProjectId: "string",
        podIpRangeName: "string",
        serviceIpRangeName: "string",
        subnetId: "string",
        subnetRegion: "string",
        vpcId: "string",
    },
    networkId: "string",
    securityGroupIds: ["string"],
    subnetIds: ["string"],
    vpcEndpoints: {
        dataplaneRelays: ["string"],
        restApis: ["string"],
    },
    vpcId: "string",
    vpcStatus: "string",
    workspaceId: 0,
});

type: databricks:MwsNetworks
properties:
    accountId: string
    creationTime: 0
    errorMessages:
        - errorMessage: string
          errorType: string
    gcpNetworkInfo:
        networkProjectId: string
        podIpRangeName: string
        serviceIpRangeName: string
        subnetId: string
        subnetRegion: string
        vpcId: string
    networkId: string
    networkName: string
    securityGroupIds:
        - string
    subnetIds:
        - string
    vpcEndpoints:
        dataplaneRelays:
            - string
        restApis:
            - string
    vpcId: string
    vpcStatus: string
    workspaceId: 0

MwsNetworks Resource Properties

To learn more about resource properties and how to use them, see Inputs and Outputs in the Architecture and Concepts docs.

Inputs

The MwsNetworks resource accepts the following input properties:

Outputs

All input properties are implicitly available as output properties. Additionally, the MwsNetworks resource produces the following output properties:

Look up Existing MwsNetworks Resource

Get an existing MwsNetworks resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.

public static get(name: string, id: Input<ID>, state?: MwsNetworksState, opts?: CustomResourceOptions): MwsNetworks

@staticmethod
def get(resource_name: str,
        id: str,
        opts: Optional[ResourceOptions] = None,
        account_id: Optional[str] = None,
        creation_time: Optional[int] = None,
        error_messages: Optional[Sequence[MwsNetworksErrorMessageArgs]] = None,
        gcp_network_info: Optional[MwsNetworksGcpNetworkInfoArgs] = None,
        network_id: Optional[str] = None,
        network_name: Optional[str] = None,
        security_group_ids: Optional[Sequence[str]] = None,
        subnet_ids: Optional[Sequence[str]] = None,
        vpc_endpoints: Optional[MwsNetworksVpcEndpointsArgs] = None,
        vpc_id: Optional[str] = None,
        vpc_status: Optional[str] = None,
        workspace_id: Optional[int] = None) -> MwsNetworks

func GetMwsNetworks(ctx *Context, name string, id IDInput, state *MwsNetworksState, opts ...ResourceOption) (*MwsNetworks, error)

public static MwsNetworks Get(string name, Input<string> id, MwsNetworksState? state, CustomResourceOptions? opts = null)

public static MwsNetworks get(String name, Output<String> id, MwsNetworksState state, CustomResourceOptions options)

Resource lookup is not supported in YAML

Supporting Types

MwsNetworksErrorMessage, MwsNetworksErrorMessageArgs

MwsNetworksGcpNetworkInfo, MwsNetworksGcpNetworkInfoArgs

MwsNetworksVpcEndpoints, MwsNetworksVpcEndpointsArgs

Import

-> Note Importing this resource is not currently supported.

To learn more about importing existing cloud resources, see Importing resources.

Package Details

Repository

databricks pulumi/pulumi-databricks

License

Apache-2.0

Notes

This Pulumi package is based on the databricks Terraform Provider.