Cross-regional Databricks SQL Configuration with SQLGlobalConfig

Question

Pulumi · Accepted Answer

In order to set up a cross-regional Databricks SQL configuration, you'll be working with Databricks SQL Global Configuration (SQLGlobalConfig). The `databricks.SqlGlobalConfig` resource in Pulumi allows you to define configurations that apply to Databricks SQL endpoints. This is used to manage settings like security policies, SQL configuration parameters, and other global settings that affect how your SQL endpoints operate.

To demonstrate how to configure this in Pulumi using Python, we'll write a program that sets up a SQL global configuration with some example settings. Here's what the program will do:

- Set a security policy for SQL endpoints to ensure they adhere to organizational standards.
- Define SQL configuration parameters that apply globally to manage the behavior of SQL endpoints.
- Assign a data access configuration that determines how data is accessed within the SQL endpoints.

Let's get to it! Here's the Pulumi program written in Python that accomplishes the above configuration:

```python
import pulumi
import pulumi_databricks as databricks

# Instantiate a Databricks SQL Global Config resource
# Modify the securityPolicy, sqlConfigParams, and dataAccessConfig as per your organization's requirements.
sql_global_config = databricks.SqlGlobalConfig("sql-global-config",
    security_policy="CUSTOM_MANAGED",  # Can be "DATA_CONTROLLED" OR "CUSTOM_MANAGED"
    sql_config_params={
        "queryExecutionTimeoutInSeconds": "120",
        "resultCache": "enabled"
    },
    data_access_config={
        "storageAccess": {
            "s3": {
                "enableIAM": True,
                "bucketName": "example-bucket"
            }
        }
    },
)

# Output the sql_global_config ID
pulumi.export("sqlGlobalConfigId", sql_global_config.id)
```

The program starts by importing the required modules. We're importing `pulumi` for basic Pulumi functionality and `pulumi_databricks` because we're working with the Pulumi Databricks provider.

Next, it declares a `databricks.SqlGlobalConfig` resource named `sql-global-config`. In this resource:

- `security_policy` is set to `CUSTOM_MANAGED` or can be `DATA_CONTROLLED` depending on whether you want the security policy to be managed by you or controlled by data access configurations, respectively.
- `sql_config_params` defines global settings for SQL-related configurations. Here, we've set a query execution timeout and enabled a result cache.
- `data_access_config` dictates the global data access settings, which includes how storage (like AWS S3 in this case) is accessed. This example enables IAM for S3 and specifies a bucket name.

The configuration settings used in the program should be changed according to your own requirements and organizational standards.

Finally, the program exports the `id` of the `SqlGlobalConfig` resource so that you can easily reference it elsewhere in your Pulumi stack or retrieve it using the CLI after deployment.

Remember to configure your Pulumi for Databricks by doing `pulumi config set databricks:token <YOUR DATABRICKS_TOKEN>` and setting up any other necessary configuration such as the Databricks workspace URL.

By running this program with Pulumi, it will configure the global settings for Databricks SQL across all regions where your Databricks workspaces are located, thereby implementing a cross-regional configuration.