How do I set up a multi-tenant AI environment in a Databricks workspace?
In this guide, we will set up a multi-tenant AI environment within an Azure Databricks workspace using Pulumi. This setup will include creating a Databricks workspace, configuring it for multiple tenants, and setting up necessary resources such as clusters and users.
Key Points
- Create an Azure Databricks workspace.
- Configure the workspace for multi-tenancy by setting up clusters and users.
- Ensure each tenant has isolated resources within the workspace.
import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure-native";
import * as databricks from "@pulumi/databricks";
// Define the resource group
const resourceGroup = new azure.resources.ResourceGroup("resourceGroup", {
location: "East US",
});
// Create an Azure Databricks workspace
const databricksWorkspace = new azure.databricks.Workspace("databricksWorkspace", {
resourceGroupName: resourceGroup.name,
location: resourceGroup.location,
sku: {
name: "standard",
},
managedResourceGroupId: resourceGroup.id,
parameters: {
enableNoPublicIp: {
value: true,
},
publicIpName: {
value: "databricksPublicIp",
},
},
});
// Create Databricks users for tenants
const tenant1User = new databricks.User("tenant1User", {
userName: "tenant1@example.com",
displayName: "Tenant 1 User",
active: true,
workspaceAccess: true,
});
const tenant2User = new databricks.User("tenant2User", {
userName: "tenant2@example.com",
displayName: "Tenant 2 User",
active: true,
workspaceAccess: true,
});
// Create clusters for each tenant
const tenant1Cluster = new databricks.Cluster("tenant1Cluster", {
clusterName: "tenant1-cluster",
sparkVersion: "7.3.x-scala2.12",
nodeTypeId: "Standard_DS3_v2",
numWorkers: 2,
autoterminationMinutes: 20,
sparkConf: {
"spark.databricks.cluster.profile": "singleNode",
},
});
const tenant2Cluster = new databricks.Cluster("tenant2Cluster", {
clusterName: "tenant2-cluster",
sparkVersion: "7.3.x-scala2.12",
nodeTypeId: "Standard_DS3_v2",
numWorkers: 2,
autoterminationMinutes: 20,
sparkConf: {
"spark.databricks.cluster.profile": "singleNode",
},
});
// Export the workspace URL
export const workspaceUrl = databricksWorkspace.workspaceUrl;
Summary
In this setup, we created an Azure Databricks workspace and configured it for multi-tenancy by adding two users and two clusters, one for each tenant. This ensures that each tenant has isolated resources within the workspace, supporting a multi-tenant AI environment.
Deploy this code
Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.
Sign upNew to Pulumi?
Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.
Sign upThank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.