1. Enforcing Resource Quotas with Databricks Cluster Policies


    Enforcing Resource Quotas with Databricks Cluster Policies involves creating rules that limit the creation of clusters to ensure they stay within certain bounds such as maximum number of clusters per user, specific node types, or maximum number of workers. These policy definitions provide a way to manage resources effectively and avoid resource overuse.

    In the context of working with Pulumi and Databricks, you would use the databricks.ClusterPolicy resource to define these policies. This resource type allows you to specify constraints and rules for cluster creation.

    The following program demonstrates how to create a Databricks cluster policy using Pulumi. This policy specifies constraints such as the maximum number of workers, the allowed node types, and the maximum number of clusters per user.

    import pulumi import pulumi_databricks as databricks # Define a new Databricks Cluster Policy cluster_policy = databricks.ClusterPolicy("myClusterPolicy", # The definition specifies the actual constraints and rules for creation of Databricks clusters. definition={ # Specify the maximum number of workers for the cluster. "spark_conf.spark.databricks.cluster.profile": { "type": "fixed", "value": "singleNode" }, # Define allowed node types for the cluster. "node_type_id": { "type": "allowlist", "value": ["Standard_DS3_v2", "Standard_DS4_v2"] }, # Specify the maximum number of clusters a single user can create. "custom_tags.MaximumClustersPerUser": { "type": "fixed", "value": "1" } }, # Optionally, you can provide a user-friendly description for the policy. description="This policy enforces constraints to limit resource usage." ) # Export the ID of the cluster policy which can be used to associate clusters with this policy. pulumi.export("cluster_policy_id", cluster_policy.id)

    In this Pulumi program:

    • We import the necessary Pulumi packages.
    • We define a ClusterPolicy resource with the name myClusterPolicy.
    • The definition parameter of the ClusterPolicy resource is a dictionary where:
      • "spark_conf.spark.databricks.cluster.profile" limits the cluster to be a single node cluster.
      • "node_type_id" provides an allowlist of node types that can be used. In this case, only Standard_DS3_v2 and Standard_DS4_v2 are allowed.
      • "custom_tags.MaximumClustersPerUser" sets a limit of 1 cluster per user.
    • An optional description is provided to better explain the purpose of the policy.
    • Finally, we export the ID of the policy. This ID can be used in other Pulumi programs or resources to apply this cluster policy to Databricks clusters.

    You can refer to the databricks.ClusterPolicy documentation for more details on the properties and usage of this resource.