1. Answers
  2. Setting up Multi-Tenant AI Environment in Databricks Workspace

How do I set up a multi-tenant AI environment in a Databricks workspace?

In this guide, we will set up a multi-tenant AI environment within an Azure Databricks workspace using Pulumi. This setup will include creating a Databricks workspace, configuring it for multiple tenants, and setting up necessary resources such as clusters and users.

Key Points

  • Create an Azure Databricks workspace.
  • Configure the workspace for multi-tenancy by setting up clusters and users.
  • Ensure each tenant has isolated resources within the workspace.
import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure-native";
import * as databricks from "@pulumi/databricks";

// Define the resource group
const resourceGroup = new azure.resources.ResourceGroup("resourceGroup", {
    location: "East US",
});

// Create an Azure Databricks workspace
const databricksWorkspace = new azure.databricks.Workspace("databricksWorkspace", {
    resourceGroupName: resourceGroup.name,
    location: resourceGroup.location,
    sku: {
        name: "standard",
    },
    managedResourceGroupId: resourceGroup.id,
    parameters: {
        enableNoPublicIp: {
            value: true,
        },
        publicIpName: {
            value: "databricksPublicIp",
        },
    },
});

// Create Databricks users for tenants
const tenant1User = new databricks.User("tenant1User", {
    userName: "tenant1@example.com",
    displayName: "Tenant 1 User",
    active: true,
    workspaceAccess: true,
});

const tenant2User = new databricks.User("tenant2User", {
    userName: "tenant2@example.com",
    displayName: "Tenant 2 User",
    active: true,
    workspaceAccess: true,
});

// Create clusters for each tenant
const tenant1Cluster = new databricks.Cluster("tenant1Cluster", {
    clusterName: "tenant1-cluster",
    sparkVersion: "7.3.x-scala2.12",
    nodeTypeId: "Standard_DS3_v2",
    numWorkers: 2,
    autoterminationMinutes: 20,
    sparkConf: {
        "spark.databricks.cluster.profile": "singleNode",
    },
});

const tenant2Cluster = new databricks.Cluster("tenant2Cluster", {
    clusterName: "tenant2-cluster",
    sparkVersion: "7.3.x-scala2.12",
    nodeTypeId: "Standard_DS3_v2",
    numWorkers: 2,
    autoterminationMinutes: 20,
    sparkConf: {
        "spark.databricks.cluster.profile": "singleNode",
    },
});

// Export the workspace URL
export const workspaceUrl = databricksWorkspace.workspaceUrl;

Summary

In this setup, we created an Azure Databricks workspace and configured it for multi-tenancy by adding two users and two clusters, one for each tenant. This ensures that each tenant has isolated resources within the workspace, supporting a multi-tenant AI environment.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up