Enforcing Resource Quotas with Databricks Cluster Policies

Question

Pulumi · Accepted Answer

Enforcing Resource Quotas with Databricks Cluster Policies involves creating rules that limit the creation of clusters to ensure they stay within certain bounds such as maximum number of clusters per user, specific node types, or maximum number of workers. These policy definitions provide a way to manage resources effectively and avoid resource overuse.

In the context of working with Pulumi and Databricks, you would use the `databricks.ClusterPolicy` resource to define these policies. This resource type allows you to specify constraints and rules for cluster creation.

The following program demonstrates how to create a Databricks cluster policy using Pulumi. This policy specifies constraints such as the maximum number of workers, the allowed node types, and the maximum number of clusters per user.

```python
import pulumi
import pulumi_databricks as databricks

# Define a new Databricks Cluster Policy
cluster_policy = databricks.ClusterPolicy("myClusterPolicy",
    # The definition specifies the actual constraints and rules for creation of Databricks clusters.
    definition={
        # Specify the maximum number of workers for the cluster.
        "spark_conf.spark.databricks.cluster.profile": {
            "type": "fixed",
            "value": "singleNode"
        },
        # Define allowed node types for the cluster.
        "node_type_id": {
            "type": "allowlist",
            "value": ["Standard_DS3_v2", "Standard_DS4_v2"]
        },
        # Specify the maximum number of clusters a single user can create.
        "custom_tags.MaximumClustersPerUser": {
            "type": "fixed",
            "value": "1"
        }
    },
    # Optionally, you can provide a user-friendly description for the policy.
    description="This policy enforces constraints to limit resource usage."
)

# Export the ID of the cluster policy which can be used to associate clusters with this policy.
pulumi.export("cluster_policy_id", cluster_policy.id)
```

In this Pulumi program:
- We import the necessary Pulumi packages.
- We define a `ClusterPolicy` resource with the name `myClusterPolicy`.
- The `definition` parameter of the `ClusterPolicy` resource is a dictionary where:
  - `"spark_conf.spark.databricks.cluster.profile"` limits the cluster to be a single node cluster.
  - `"node_type_id"` provides an allowlist of node types that can be used. In this case, only `Standard_DS3_v2` and `Standard_DS4_v2` are allowed.
  - `"custom_tags.MaximumClustersPerUser"` sets a limit of 1 cluster per user.
- An optional description is provided to better explain the purpose of the policy.
- Finally, we export the ID of the policy. This ID can be used in other Pulumi programs or resources to apply this cluster policy to Databricks clusters.

You can refer to the [`databricks.ClusterPolicy` documentation](https://www.pulumi.com/registry/packages/databricks/api-docs/clusterpolicy/) for more details on the properties and usage of this resource.