Cross-regional Databricks SQL Configuration with SQLGlobalConfig
PythonIn order to set up a cross-regional Databricks SQL configuration, you'll be working with Databricks SQL Global Configuration (SQLGlobalConfig). The
databricks.SqlGlobalConfig
resource in Pulumi allows you to define configurations that apply to Databricks SQL endpoints. This is used to manage settings like security policies, SQL configuration parameters, and other global settings that affect how your SQL endpoints operate.To demonstrate how to configure this in Pulumi using Python, we'll write a program that sets up a SQL global configuration with some example settings. Here's what the program will do:
- Set a security policy for SQL endpoints to ensure they adhere to organizational standards.
- Define SQL configuration parameters that apply globally to manage the behavior of SQL endpoints.
- Assign a data access configuration that determines how data is accessed within the SQL endpoints.
Let's get to it! Here's the Pulumi program written in Python that accomplishes the above configuration:
import pulumi import pulumi_databricks as databricks # Instantiate a Databricks SQL Global Config resource # Modify the securityPolicy, sqlConfigParams, and dataAccessConfig as per your organization's requirements. sql_global_config = databricks.SqlGlobalConfig("sql-global-config", security_policy="CUSTOM_MANAGED", # Can be "DATA_CONTROLLED" OR "CUSTOM_MANAGED" sql_config_params={ "queryExecutionTimeoutInSeconds": "120", "resultCache": "enabled" }, data_access_config={ "storageAccess": { "s3": { "enableIAM": True, "bucketName": "example-bucket" } } }, ) # Output the sql_global_config ID pulumi.export("sqlGlobalConfigId", sql_global_config.id)
The program starts by importing the required modules. We're importing
pulumi
for basic Pulumi functionality andpulumi_databricks
because we're working with the Pulumi Databricks provider.Next, it declares a
databricks.SqlGlobalConfig
resource namedsql-global-config
. In this resource:security_policy
is set toCUSTOM_MANAGED
or can beDATA_CONTROLLED
depending on whether you want the security policy to be managed by you or controlled by data access configurations, respectively.sql_config_params
defines global settings for SQL-related configurations. Here, we've set a query execution timeout and enabled a result cache.data_access_config
dictates the global data access settings, which includes how storage (like AWS S3 in this case) is accessed. This example enables IAM for S3 and specifies a bucket name.
The configuration settings used in the program should be changed according to your own requirements and organizational standards.
Finally, the program exports the
id
of theSqlGlobalConfig
resource so that you can easily reference it elsewhere in your Pulumi stack or retrieve it using the CLI after deployment.Remember to configure your Pulumi for Databricks by doing
pulumi config set databricks:token <YOUR DATABRICKS_TOKEN>
and setting up any other necessary configuration such as the Databricks workspace URL.By running this program with Pulumi, it will configure the global settings for Databricks SQL across all regions where your Databricks workspaces are located, thereby implementing a cross-regional configuration.