Customizing Databricks workspace environment variables

Question

Pulumi · Accepted Answer

When working with Databricks on Pulumi, you can customize the workspace environment, including setting environment variables. This might involve configuring the workspace itself or associated resources like storage configurations, network settings, or permissions, depending on your requirements.

Below is a TypeScript program that uses the Pulumi `databricks` provider to create and configure a Databricks workspace. This program showcases how to establish a Databricks workspace and apply custom configurations, which might include environment variables based on your specific needs. We'll use the `databricks.WorkspaceConf` resource for this purpose.

Please remember to install the necessary Pulumi package by running `npm install @pulumi/databricks` before executing this code.

### Program Explanation

First, we import the required modules from the `@pulumi/databricks` package. Then, we instantiate the `WorkspaceConf` resource, which is used to apply custom configurations to a Databricks workspace. You can set environment variables as key-value pairs in the `customConfig` object, although this part is not explicitly exposed in the Pulumi Databricks provider. Environment variables are typically set within the Databricks UI or through initialization scripts when clusters are launched. However, the custom configurations could include Spark defaults, access control settings, or other features tweakable at the workspace level.

The example does not set specific environment variables as their management is mostly done through the Databricks UI, API, or CLI, not directly through Pulumi. For the sake of demonstration, we will add a placeholder for `customConfig` where you can specify workspace-level configurations.

Here's the program:

```typescript
import * as databricks from "@pulumi/databricks";

// Ensure to replace these placeholder values with actual parameters relevant to your environment.
const workspaceName = "my-databricks-workspace";
const resourceGroupName = "my-resource-group";
const location = "West US 2"; // Choose the Azure region that best suits your needs.

// Create a Databricks workspace.
const workspace = new databricks.Workspace(workspaceName, {
    location: location,
    resourceGroupName: resourceGroupName,
    sku: "standard" // For example, choose from standard, premium, or other available SKUs.
});

// Apply custom configurations to the Databricks workspace.
const workspaceConf = new databricks.WorkspaceConf(workspaceName, {
    customConfig: {
        // Replace these placeholder configurations with actual custom configurations.
        // Environment variables are set through the Databricks UI, API, or CLI.
        // For example: "spark.databricks.cluster.profile": "singleNode",
        // "spark.master": "local[*]"
        // Add or modify configurations as required for your environment.
    }
}, { dependsOn: [workspace] }); // Ensure the WorkspaceConf is applied after the workspace is provisioned.

// Export the Databricks workspace URL for easy access.
export const workspaceUrl = workspace.url;
```

### After Execution

After running this Pulumi program, you'll have provisioned a new Databricks workspace with the specified configuration. The export statement at the end exposes the workspace URL which you can use to access the Databricks workspace through a web browser.

For more complex setups and interactions with other Azure resources, you can expand upon this program by integrating network settings, storage configurations, and more. Always refer to the [Pulumi Databricks documentation](https://www.pulumi.com/registry/packages/databricks/api-docs/) for the latest and most detailed information.