databricks.Cluster

Runtime version of the cluster. Any supported databricks.getSparkVersion id. We advise using Cluster Policies to restrict the list of versions for simplicity while maintaining enough control.

Whether to use policy default values for missing cluster attributes.

Automatically terminate the cluster after being inactive for this time in minutes. If specified, the threshold must be between 10 and 10000 minutes. You can also set this value to 0 to explicitly disable automatic termination. Defaults to 60. We highly recommend having this setting present for Interactive/BI clusters.

Cluster name, which doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS EC2 instances and EBS volumes) with these tags in addition to default_tags. If a custom cluster tag has the same name as a default cluster tag, the custom tag is prefixed with an x_ when it is propagated.

Select the security features of the cluster. Unity Catalog requires SINGLE_USER or USER_ISOLATION mode. LEGACY_PASSTHROUGH for passthrough cluster and LEGACY_TABLE_ACL for Table ACL cluster. If omitted, no security features are enabled. In the Databricks UI, this has been recently been renamed Access Mode and USER_ISOLATION has been renamed Shared, but use these terms here.

similar to instance_pool_id, but for driver node. If omitted, and instance_pool_id is specified, then the driver will be allocated from that pool.

The node type of the Spark driver. This field is optional; if unset, API will set the driver node type to the same value as node_type_id defined above.

If you don’t want to allocate a fixed number of EBS volumes at cluster creation time, use autoscaling local storage. With autoscaling local storage, Databricks monitors the amount of free disk space available on your cluster’s Spark workers. If a worker begins to run too low on disk, Databricks automatically attaches a new EBS volume to the worker before it runs out of disk space. EBS volumes are attached up to a limit of 5 TB of total disk space per instance (including the instance’s local storage). To scale down EBS usage, make sure you have autotermination_minutes and autoscale attributes set. More documentation available at cluster configuration page.

Some instance types you use to run clusters may have locally attached disks. Databricks may store shuffle data or temporary data on these locally attached disks. To ensure that all data at rest is encrypted for all storage types, including shuffle data stored temporarily on your cluster’s local disks, you can enable local disk encryption. When local disk encryption is enabled, Databricks generates an encryption key locally unique to each cluster node and uses it to encrypt all data stored on local disks. The scope of the key is local to each cluster node and is destroyed along with the cluster node itself. During its lifetime, the key resides in memory for encryption and decryption and is stored encrypted on the disk. Your workloads may run more slowly because of the performance impact of reading and writing encrypted data to and from local volumes. This feature is not available for all Azure Databricks subscriptions. Contact your Microsoft or Databricks account representative to request access.

An optional token to guarantee the idempotency of cluster creation requests. If an active cluster with the provided token already exists, the request will not create a new cluster, but it will return the existing running cluster's ID instead. If you specify the idempotency token, upon failure, you can retry until the request succeeds. Databricks platform guarantees to launch exactly one cluster with that idempotency token. This token should have at most 64 characters.

To reduce cluster start time, you can attach a cluster to a predefined pool of idle instances. When attached to a pool, a cluster allocates its driver and worker nodes from the pool. If the pool does not have sufficient idle resources to accommodate the cluster’s request, it expands by allocating new instances from the instance provider. When an attached cluster changes its state to TERMINATED, the instances it used are returned to the pool and reused by a different cluster.

boolean value specifying if the cluster is pinned (not pinned by default). You must be a Databricks administrator to use this. The pinned clusters' maximum number is limited to 100, so apply may fail if you have more than that (this number may change over time, so check Databricks documentation for actual number).

The following example demonstrates how to create an autoscaling cluster with Delta Cache enabled:

import * as pulumi from "@pulumi/pulumi";
import * as databricks from "@pulumi/databricks";
const smallest = databricks.getNodeType({
localDisk: true,
});
const latestLts = databricks.getSparkVersion({
longTermSupport: true,
});
const sharedAutoscaling = new databricks.Cluster("sharedAutoscaling", {
clusterName: "Shared Autoscaling",
sparkVersion: latestLts.then(latestLts => latestLts.id),
nodeTypeId: smallest.then(smallest => smallest.id),
autoterminationMinutes: 20,
autoscale: {
minWorkers: 1,
maxWorkers: 50,
},
sparkConf: {
"spark.databricks.io.cache.enabled": true,
"spark.databricks.io.cache.maxDiskUsage": "50g",
"spark.databricks.io.cache.maxMetaDataCache": "1g",
},
});

import pulumi
import pulumi_databricks as databricks

smallest = databricks.get_node_type(local_disk=True)
latest_lts = databricks.get_spark_version(long_term_support=True)
shared_autoscaling = databricks.Cluster("sharedAutoscaling",
    cluster_name="Shared Autoscaling",
    spark_version=latest_lts.id,
    node_type_id=smallest.id,
    autotermination_minutes=20,
    autoscale=databricks.ClusterAutoscaleArgs(
        min_workers=1,
        max_workers=50,
    ),
    spark_conf={
        "spark.databricks.io.cache.enabled": True,
        "spark.databricks.io.cache.maxDiskUsage": "50g",
        "spark.databricks.io.cache.maxMetaDataCache": "1g",
    })

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Databricks = Pulumi.Databricks;

return await Deployment.RunAsync(() => 
{
    var smallest = Databricks.GetNodeType.Invoke(new()
    {
        LocalDisk = true,
    });

    var latestLts = Databricks.GetSparkVersion.Invoke(new()
    {
        LongTermSupport = true,
    });

    var sharedAutoscaling = new Databricks.Cluster("sharedAutoscaling", new()
    {
        ClusterName = "Shared Autoscaling",
        SparkVersion = latestLts.Apply(getSparkVersionResult => getSparkVersionResult.Id),
        NodeTypeId = smallest.Apply(getNodeTypeResult => getNodeTypeResult.Id),
        AutoterminationMinutes = 20,
        Autoscale = new Databricks.Inputs.ClusterAutoscaleArgs
        {
            MinWorkers = 1,
            MaxWorkers = 50,
        },
        SparkConf = 
        {
            { "spark.databricks.io.cache.enabled", true },
            { "spark.databricks.io.cache.maxDiskUsage", "50g" },
            { "spark.databricks.io.cache.maxMetaDataCache", "1g" },
        },
    });

});

package main

import (
	"github.com/pulumi/pulumi-databricks/sdk/go/databricks"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		smallest, err := databricks.GetNodeType(ctx, &databricks.GetNodeTypeArgs{
			LocalDisk: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		latestLts, err := databricks.GetSparkVersion(ctx, &databricks.GetSparkVersionArgs{
			LongTermSupport: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		_, err = databricks.NewCluster(ctx, "sharedAutoscaling", &databricks.ClusterArgs{
			ClusterName:            pulumi.String("Shared Autoscaling"),
			SparkVersion:           *pulumi.String(latestLts.Id),
			NodeTypeId:             *pulumi.String(smallest.Id),
			AutoterminationMinutes: pulumi.Int(20),
			Autoscale: &databricks.ClusterAutoscaleArgs{
				MinWorkers: pulumi.Int(1),
				MaxWorkers: pulumi.Int(50),
			},
			SparkConf: pulumi.Map{
				"spark.databricks.io.cache.enabled":          pulumi.Any(true),
				"spark.databricks.io.cache.maxDiskUsage":     pulumi.Any("50g"),
				"spark.databricks.io.cache.maxMetaDataCache": pulumi.Any("1g"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.databricks.DatabricksFunctions;
import com.pulumi.databricks.inputs.GetNodeTypeArgs;
import com.pulumi.databricks.inputs.GetSparkVersionArgs;
import com.pulumi.databricks.Cluster;
import com.pulumi.databricks.ClusterArgs;
import com.pulumi.databricks.inputs.ClusterAutoscaleArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        final var smallest = DatabricksFunctions.getNodeType(GetNodeTypeArgs.builder()
            .localDisk(true)
            .build());

        final var latestLts = DatabricksFunctions.getSparkVersion(GetSparkVersionArgs.builder()
            .longTermSupport(true)
            .build());

        var sharedAutoscaling = new Cluster("sharedAutoscaling", ClusterArgs.builder()        
            .clusterName("Shared Autoscaling")
            .sparkVersion(latestLts.applyValue(getSparkVersionResult -> getSparkVersionResult.id()))
            .nodeTypeId(smallest.applyValue(getNodeTypeResult -> getNodeTypeResult.id()))
            .autoterminationMinutes(20)
            .autoscale(ClusterAutoscaleArgs.builder()
                .minWorkers(1)
                .maxWorkers(50)
                .build())
            .sparkConf(Map.ofEntries(
                Map.entry("spark.databricks.io.cache.enabled", true),
                Map.entry("spark.databricks.io.cache.maxDiskUsage", "50g"),
                Map.entry("spark.databricks.io.cache.maxMetaDataCache", "1g")
            ))
            .build());

    }
}

resources:
  sharedAutoscaling:
    type: databricks:Cluster
    properties:
      clusterName: Shared Autoscaling
      sparkVersion: ${latestLts.id}
      nodeTypeId: ${smallest.id}
      autoterminationMinutes: 20
      autoscale:
        minWorkers: 1
        maxWorkers: 50
      sparkConf:
        spark.databricks.io.cache.enabled: true
        spark.databricks.io.cache.maxDiskUsage: 50g
        spark.databricks.io.cache.maxMetaDataCache: 1g
variables:
  smallest:
    fn::invoke:
      Function: databricks:getNodeType
      Arguments:
        localDisk: true
  latestLts:
    fn::invoke:
      Function: databricks:getSparkVersion
      Arguments:
        longTermSupport: true

Any supported databricks.getNodeType id. If instance_pool_id is specified, this field is not needed.

Number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

The type of runtime engine to use. If not specified, the runtime engine type is inferred based on the spark_version value. Allowed values include: PHOTON, STANDARD.

The optional user name of the user to assign to an interactive cluster. This field is required when using data_security_mode set to SINGLE_USER or AAD Passthrough for Azure Data Lake Storage (ADLS) with a single-user cluster (i.e., not high-concurrency clusters).

Map with key-value pairs to fine-tune Spark clusters, where you can provide custom Spark configuration properties in a cluster configuration.

Map with environment variable key-value pairs to fine-tune Spark clusters. Key-value pairs of the form (X,Y) are exported (i.e., X='Y') while launching the driver and workers.

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. You can specify up to 10 keys.

Runtime version of the cluster. Any supported databricks.getSparkVersion id. We advise using Cluster Policies to restrict the list of versions for simplicity while maintaining enough control.

Whether to use policy default values for missing cluster attributes.

Automatically terminate the cluster after being inactive for this time in minutes. If specified, the threshold must be between 10 and 10000 minutes. You can also set this value to 0 to explicitly disable automatic termination. Defaults to 60. We highly recommend having this setting present for Interactive/BI clusters.

Cluster name, which doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS EC2 instances and EBS volumes) with these tags in addition to default_tags. If a custom cluster tag has the same name as a default cluster tag, the custom tag is prefixed with an x_ when it is propagated.

Select the security features of the cluster. Unity Catalog requires SINGLE_USER or USER_ISOLATION mode. LEGACY_PASSTHROUGH for passthrough cluster and LEGACY_TABLE_ACL for Table ACL cluster. If omitted, no security features are enabled. In the Databricks UI, this has been recently been renamed Access Mode and USER_ISOLATION has been renamed Shared, but use these terms here.

similar to instance_pool_id, but for driver node. If omitted, and instance_pool_id is specified, then the driver will be allocated from that pool.

The node type of the Spark driver. This field is optional; if unset, API will set the driver node type to the same value as node_type_id defined above.

If you don’t want to allocate a fixed number of EBS volumes at cluster creation time, use autoscaling local storage. With autoscaling local storage, Databricks monitors the amount of free disk space available on your cluster’s Spark workers. If a worker begins to run too low on disk, Databricks automatically attaches a new EBS volume to the worker before it runs out of disk space. EBS volumes are attached up to a limit of 5 TB of total disk space per instance (including the instance’s local storage). To scale down EBS usage, make sure you have autotermination_minutes and autoscale attributes set. More documentation available at cluster configuration page.

Some instance types you use to run clusters may have locally attached disks. Databricks may store shuffle data or temporary data on these locally attached disks. To ensure that all data at rest is encrypted for all storage types, including shuffle data stored temporarily on your cluster’s local disks, you can enable local disk encryption. When local disk encryption is enabled, Databricks generates an encryption key locally unique to each cluster node and uses it to encrypt all data stored on local disks. The scope of the key is local to each cluster node and is destroyed along with the cluster node itself. During its lifetime, the key resides in memory for encryption and decryption and is stored encrypted on the disk. Your workloads may run more slowly because of the performance impact of reading and writing encrypted data to and from local volumes. This feature is not available for all Azure Databricks subscriptions. Contact your Microsoft or Databricks account representative to request access.

An optional token to guarantee the idempotency of cluster creation requests. If an active cluster with the provided token already exists, the request will not create a new cluster, but it will return the existing running cluster's ID instead. If you specify the idempotency token, upon failure, you can retry until the request succeeds. Databricks platform guarantees to launch exactly one cluster with that idempotency token. This token should have at most 64 characters.

To reduce cluster start time, you can attach a cluster to a predefined pool of idle instances. When attached to a pool, a cluster allocates its driver and worker nodes from the pool. If the pool does not have sufficient idle resources to accommodate the cluster’s request, it expands by allocating new instances from the instance provider. When an attached cluster changes its state to TERMINATED, the instances it used are returned to the pool and reused by a different cluster.

boolean value specifying if the cluster is pinned (not pinned by default). You must be a Databricks administrator to use this. The pinned clusters' maximum number is limited to 100, so apply may fail if you have more than that (this number may change over time, so check Databricks documentation for actual number).

The following example demonstrates how to create an autoscaling cluster with Delta Cache enabled:

import * as pulumi from "@pulumi/pulumi";
import * as databricks from "@pulumi/databricks";
const smallest = databricks.getNodeType({
localDisk: true,
});
const latestLts = databricks.getSparkVersion({
longTermSupport: true,
});
const sharedAutoscaling = new databricks.Cluster("sharedAutoscaling", {
clusterName: "Shared Autoscaling",
sparkVersion: latestLts.then(latestLts => latestLts.id),
nodeTypeId: smallest.then(smallest => smallest.id),
autoterminationMinutes: 20,
autoscale: {
minWorkers: 1,
maxWorkers: 50,
},
sparkConf: {
"spark.databricks.io.cache.enabled": true,
"spark.databricks.io.cache.maxDiskUsage": "50g",
"spark.databricks.io.cache.maxMetaDataCache": "1g",
},
});

import pulumi
import pulumi_databricks as databricks

smallest = databricks.get_node_type(local_disk=True)
latest_lts = databricks.get_spark_version(long_term_support=True)
shared_autoscaling = databricks.Cluster("sharedAutoscaling",
    cluster_name="Shared Autoscaling",
    spark_version=latest_lts.id,
    node_type_id=smallest.id,
    autotermination_minutes=20,
    autoscale=databricks.ClusterAutoscaleArgs(
        min_workers=1,
        max_workers=50,
    ),
    spark_conf={
        "spark.databricks.io.cache.enabled": True,
        "spark.databricks.io.cache.maxDiskUsage": "50g",
        "spark.databricks.io.cache.maxMetaDataCache": "1g",
    })

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Databricks = Pulumi.Databricks;

return await Deployment.RunAsync(() => 
{
    var smallest = Databricks.GetNodeType.Invoke(new()
    {
        LocalDisk = true,
    });

    var latestLts = Databricks.GetSparkVersion.Invoke(new()
    {
        LongTermSupport = true,
    });

    var sharedAutoscaling = new Databricks.Cluster("sharedAutoscaling", new()
    {
        ClusterName = "Shared Autoscaling",
        SparkVersion = latestLts.Apply(getSparkVersionResult => getSparkVersionResult.Id),
        NodeTypeId = smallest.Apply(getNodeTypeResult => getNodeTypeResult.Id),
        AutoterminationMinutes = 20,
        Autoscale = new Databricks.Inputs.ClusterAutoscaleArgs
        {
            MinWorkers = 1,
            MaxWorkers = 50,
        },
        SparkConf = 
        {
            { "spark.databricks.io.cache.enabled", true },
            { "spark.databricks.io.cache.maxDiskUsage", "50g" },
            { "spark.databricks.io.cache.maxMetaDataCache", "1g" },
        },
    });

});

package main

import (
	"github.com/pulumi/pulumi-databricks/sdk/go/databricks"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		smallest, err := databricks.GetNodeType(ctx, &databricks.GetNodeTypeArgs{
			LocalDisk: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		latestLts, err := databricks.GetSparkVersion(ctx, &databricks.GetSparkVersionArgs{
			LongTermSupport: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		_, err = databricks.NewCluster(ctx, "sharedAutoscaling", &databricks.ClusterArgs{
			ClusterName:            pulumi.String("Shared Autoscaling"),
			SparkVersion:           *pulumi.String(latestLts.Id),
			NodeTypeId:             *pulumi.String(smallest.Id),
			AutoterminationMinutes: pulumi.Int(20),
			Autoscale: &databricks.ClusterAutoscaleArgs{
				MinWorkers: pulumi.Int(1),
				MaxWorkers: pulumi.Int(50),
			},
			SparkConf: pulumi.Map{
				"spark.databricks.io.cache.enabled":          pulumi.Any(true),
				"spark.databricks.io.cache.maxDiskUsage":     pulumi.Any("50g"),
				"spark.databricks.io.cache.maxMetaDataCache": pulumi.Any("1g"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.databricks.DatabricksFunctions;
import com.pulumi.databricks.inputs.GetNodeTypeArgs;
import com.pulumi.databricks.inputs.GetSparkVersionArgs;
import com.pulumi.databricks.Cluster;
import com.pulumi.databricks.ClusterArgs;
import com.pulumi.databricks.inputs.ClusterAutoscaleArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        final var smallest = DatabricksFunctions.getNodeType(GetNodeTypeArgs.builder()
            .localDisk(true)
            .build());

        final var latestLts = DatabricksFunctions.getSparkVersion(GetSparkVersionArgs.builder()
            .longTermSupport(true)
            .build());

        var sharedAutoscaling = new Cluster("sharedAutoscaling", ClusterArgs.builder()        
            .clusterName("Shared Autoscaling")
            .sparkVersion(latestLts.applyValue(getSparkVersionResult -> getSparkVersionResult.id()))
            .nodeTypeId(smallest.applyValue(getNodeTypeResult -> getNodeTypeResult.id()))
            .autoterminationMinutes(20)
            .autoscale(ClusterAutoscaleArgs.builder()
                .minWorkers(1)
                .maxWorkers(50)
                .build())
            .sparkConf(Map.ofEntries(
                Map.entry("spark.databricks.io.cache.enabled", true),
                Map.entry("spark.databricks.io.cache.maxDiskUsage", "50g"),
                Map.entry("spark.databricks.io.cache.maxMetaDataCache", "1g")
            ))
            .build());

    }
}

resources:
  sharedAutoscaling:
    type: databricks:Cluster
    properties:
      clusterName: Shared Autoscaling
      sparkVersion: ${latestLts.id}
      nodeTypeId: ${smallest.id}
      autoterminationMinutes: 20
      autoscale:
        minWorkers: 1
        maxWorkers: 50
      sparkConf:
        spark.databricks.io.cache.enabled: true
        spark.databricks.io.cache.maxDiskUsage: 50g
        spark.databricks.io.cache.maxMetaDataCache: 1g
variables:
  smallest:
    fn::invoke:
      Function: databricks:getNodeType
      Arguments:
        localDisk: true
  latestLts:
    fn::invoke:
      Function: databricks:getSparkVersion
      Arguments:
        longTermSupport: true

Any supported databricks.getNodeType id. If instance_pool_id is specified, this field is not needed.

Number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

The type of runtime engine to use. If not specified, the runtime engine type is inferred based on the spark_version value. Allowed values include: PHOTON, STANDARD.

The optional user name of the user to assign to an interactive cluster. This field is required when using data_security_mode set to SINGLE_USER or AAD Passthrough for Azure Data Lake Storage (ADLS) with a single-user cluster (i.e., not high-concurrency clusters).

Map with key-value pairs to fine-tune Spark clusters, where you can provide custom Spark configuration properties in a cluster configuration.

Map with environment variable key-value pairs to fine-tune Spark clusters. Key-value pairs of the form (X,Y) are exported (i.e., X='Y') while launching the driver and workers.

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. You can specify up to 10 keys.

Runtime version of the cluster. Any supported databricks.getSparkVersion id. We advise using Cluster Policies to restrict the list of versions for simplicity while maintaining enough control.

Whether to use policy default values for missing cluster attributes.

Automatically terminate the cluster after being inactive for this time in minutes. If specified, the threshold must be between 10 and 10000 minutes. You can also set this value to 0 to explicitly disable automatic termination. Defaults to 60. We highly recommend having this setting present for Interactive/BI clusters.

Cluster name, which doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS EC2 instances and EBS volumes) with these tags in addition to default_tags. If a custom cluster tag has the same name as a default cluster tag, the custom tag is prefixed with an x_ when it is propagated.

Select the security features of the cluster. Unity Catalog requires SINGLE_USER or USER_ISOLATION mode. LEGACY_PASSTHROUGH for passthrough cluster and LEGACY_TABLE_ACL for Table ACL cluster. If omitted, no security features are enabled. In the Databricks UI, this has been recently been renamed Access Mode and USER_ISOLATION has been renamed Shared, but use these terms here.

similar to instance_pool_id, but for driver node. If omitted, and instance_pool_id is specified, then the driver will be allocated from that pool.

The node type of the Spark driver. This field is optional; if unset, API will set the driver node type to the same value as node_type_id defined above.

If you don’t want to allocate a fixed number of EBS volumes at cluster creation time, use autoscaling local storage. With autoscaling local storage, Databricks monitors the amount of free disk space available on your cluster’s Spark workers. If a worker begins to run too low on disk, Databricks automatically attaches a new EBS volume to the worker before it runs out of disk space. EBS volumes are attached up to a limit of 5 TB of total disk space per instance (including the instance’s local storage). To scale down EBS usage, make sure you have autotermination_minutes and autoscale attributes set. More documentation available at cluster configuration page.

Some instance types you use to run clusters may have locally attached disks. Databricks may store shuffle data or temporary data on these locally attached disks. To ensure that all data at rest is encrypted for all storage types, including shuffle data stored temporarily on your cluster’s local disks, you can enable local disk encryption. When local disk encryption is enabled, Databricks generates an encryption key locally unique to each cluster node and uses it to encrypt all data stored on local disks. The scope of the key is local to each cluster node and is destroyed along with the cluster node itself. During its lifetime, the key resides in memory for encryption and decryption and is stored encrypted on the disk. Your workloads may run more slowly because of the performance impact of reading and writing encrypted data to and from local volumes. This feature is not available for all Azure Databricks subscriptions. Contact your Microsoft or Databricks account representative to request access.

An optional token to guarantee the idempotency of cluster creation requests. If an active cluster with the provided token already exists, the request will not create a new cluster, but it will return the existing running cluster's ID instead. If you specify the idempotency token, upon failure, you can retry until the request succeeds. Databricks platform guarantees to launch exactly one cluster with that idempotency token. This token should have at most 64 characters.

To reduce cluster start time, you can attach a cluster to a predefined pool of idle instances. When attached to a pool, a cluster allocates its driver and worker nodes from the pool. If the pool does not have sufficient idle resources to accommodate the cluster’s request, it expands by allocating new instances from the instance provider. When an attached cluster changes its state to TERMINATED, the instances it used are returned to the pool and reused by a different cluster.

boolean value specifying if the cluster is pinned (not pinned by default). You must be a Databricks administrator to use this. The pinned clusters' maximum number is limited to 100, so apply may fail if you have more than that (this number may change over time, so check Databricks documentation for actual number).

The following example demonstrates how to create an autoscaling cluster with Delta Cache enabled:

import * as pulumi from "@pulumi/pulumi";
import * as databricks from "@pulumi/databricks";
const smallest = databricks.getNodeType({
localDisk: true,
});
const latestLts = databricks.getSparkVersion({
longTermSupport: true,
});
const sharedAutoscaling = new databricks.Cluster("sharedAutoscaling", {
clusterName: "Shared Autoscaling",
sparkVersion: latestLts.then(latestLts => latestLts.id),
nodeTypeId: smallest.then(smallest => smallest.id),
autoterminationMinutes: 20,
autoscale: {
minWorkers: 1,
maxWorkers: 50,
},
sparkConf: {
"spark.databricks.io.cache.enabled": true,
"spark.databricks.io.cache.maxDiskUsage": "50g",
"spark.databricks.io.cache.maxMetaDataCache": "1g",
},
});

import pulumi
import pulumi_databricks as databricks

smallest = databricks.get_node_type(local_disk=True)
latest_lts = databricks.get_spark_version(long_term_support=True)
shared_autoscaling = databricks.Cluster("sharedAutoscaling",
    cluster_name="Shared Autoscaling",
    spark_version=latest_lts.id,
    node_type_id=smallest.id,
    autotermination_minutes=20,
    autoscale=databricks.ClusterAutoscaleArgs(
        min_workers=1,
        max_workers=50,
    ),
    spark_conf={
        "spark.databricks.io.cache.enabled": True,
        "spark.databricks.io.cache.maxDiskUsage": "50g",
        "spark.databricks.io.cache.maxMetaDataCache": "1g",
    })

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Databricks = Pulumi.Databricks;

return await Deployment.RunAsync(() => 
{
    var smallest = Databricks.GetNodeType.Invoke(new()
    {
        LocalDisk = true,
    });

    var latestLts = Databricks.GetSparkVersion.Invoke(new()
    {
        LongTermSupport = true,
    });

    var sharedAutoscaling = new Databricks.Cluster("sharedAutoscaling", new()
    {
        ClusterName = "Shared Autoscaling",
        SparkVersion = latestLts.Apply(getSparkVersionResult => getSparkVersionResult.Id),
        NodeTypeId = smallest.Apply(getNodeTypeResult => getNodeTypeResult.Id),
        AutoterminationMinutes = 20,
        Autoscale = new Databricks.Inputs.ClusterAutoscaleArgs
        {
            MinWorkers = 1,
            MaxWorkers = 50,
        },
        SparkConf = 
        {
            { "spark.databricks.io.cache.enabled", true },
            { "spark.databricks.io.cache.maxDiskUsage", "50g" },
            { "spark.databricks.io.cache.maxMetaDataCache", "1g" },
        },
    });

});

package main

import (
	"github.com/pulumi/pulumi-databricks/sdk/go/databricks"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		smallest, err := databricks.GetNodeType(ctx, &databricks.GetNodeTypeArgs{
			LocalDisk: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		latestLts, err := databricks.GetSparkVersion(ctx, &databricks.GetSparkVersionArgs{
			LongTermSupport: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		_, err = databricks.NewCluster(ctx, "sharedAutoscaling", &databricks.ClusterArgs{
			ClusterName:            pulumi.String("Shared Autoscaling"),
			SparkVersion:           *pulumi.String(latestLts.Id),
			NodeTypeId:             *pulumi.String(smallest.Id),
			AutoterminationMinutes: pulumi.Int(20),
			Autoscale: &databricks.ClusterAutoscaleArgs{
				MinWorkers: pulumi.Int(1),
				MaxWorkers: pulumi.Int(50),
			},
			SparkConf: pulumi.Map{
				"spark.databricks.io.cache.enabled":          pulumi.Any(true),
				"spark.databricks.io.cache.maxDiskUsage":     pulumi.Any("50g"),
				"spark.databricks.io.cache.maxMetaDataCache": pulumi.Any("1g"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.databricks.DatabricksFunctions;
import com.pulumi.databricks.inputs.GetNodeTypeArgs;
import com.pulumi.databricks.inputs.GetSparkVersionArgs;
import com.pulumi.databricks.Cluster;
import com.pulumi.databricks.ClusterArgs;
import com.pulumi.databricks.inputs.ClusterAutoscaleArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        final var smallest = DatabricksFunctions.getNodeType(GetNodeTypeArgs.builder()
            .localDisk(true)
            .build());

        final var latestLts = DatabricksFunctions.getSparkVersion(GetSparkVersionArgs.builder()
            .longTermSupport(true)
            .build());

        var sharedAutoscaling = new Cluster("sharedAutoscaling", ClusterArgs.builder()        
            .clusterName("Shared Autoscaling")
            .sparkVersion(latestLts.applyValue(getSparkVersionResult -> getSparkVersionResult.id()))
            .nodeTypeId(smallest.applyValue(getNodeTypeResult -> getNodeTypeResult.id()))
            .autoterminationMinutes(20)
            .autoscale(ClusterAutoscaleArgs.builder()
                .minWorkers(1)
                .maxWorkers(50)
                .build())
            .sparkConf(Map.ofEntries(
                Map.entry("spark.databricks.io.cache.enabled", true),
                Map.entry("spark.databricks.io.cache.maxDiskUsage", "50g"),
                Map.entry("spark.databricks.io.cache.maxMetaDataCache", "1g")
            ))
            .build());

    }
}

resources:
  sharedAutoscaling:
    type: databricks:Cluster
    properties:
      clusterName: Shared Autoscaling
      sparkVersion: ${latestLts.id}
      nodeTypeId: ${smallest.id}
      autoterminationMinutes: 20
      autoscale:
        minWorkers: 1
        maxWorkers: 50
      sparkConf:
        spark.databricks.io.cache.enabled: true
        spark.databricks.io.cache.maxDiskUsage: 50g
        spark.databricks.io.cache.maxMetaDataCache: 1g
variables:
  smallest:
    fn::invoke:
      Function: databricks:getNodeType
      Arguments:
        localDisk: true
  latestLts:
    fn::invoke:
      Function: databricks:getSparkVersion
      Arguments:
        longTermSupport: true

Any supported databricks.getNodeType id. If instance_pool_id is specified, this field is not needed.

Number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

The type of runtime engine to use. If not specified, the runtime engine type is inferred based on the spark_version value. Allowed values include: PHOTON, STANDARD.

The optional user name of the user to assign to an interactive cluster. This field is required when using data_security_mode set to SINGLE_USER or AAD Passthrough for Azure Data Lake Storage (ADLS) with a single-user cluster (i.e., not high-concurrency clusters).

Map with key-value pairs to fine-tune Spark clusters, where you can provide custom Spark configuration properties in a cluster configuration.

Map with environment variable key-value pairs to fine-tune Spark clusters. Key-value pairs of the form (X,Y) are exported (i.e., X='Y') while launching the driver and workers.

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. You can specify up to 10 keys.

Runtime version of the cluster. Any supported databricks.getSparkVersion id. We advise using Cluster Policies to restrict the list of versions for simplicity while maintaining enough control.

Whether to use policy default values for missing cluster attributes.

Automatically terminate the cluster after being inactive for this time in minutes. If specified, the threshold must be between 10 and 10000 minutes. You can also set this value to 0 to explicitly disable automatic termination. Defaults to 60. We highly recommend having this setting present for Interactive/BI clusters.

Cluster name, which doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS EC2 instances and EBS volumes) with these tags in addition to default_tags. If a custom cluster tag has the same name as a default cluster tag, the custom tag is prefixed with an x_ when it is propagated.

Select the security features of the cluster. Unity Catalog requires SINGLE_USER or USER_ISOLATION mode. LEGACY_PASSTHROUGH for passthrough cluster and LEGACY_TABLE_ACL for Table ACL cluster. If omitted, no security features are enabled. In the Databricks UI, this has been recently been renamed Access Mode and USER_ISOLATION has been renamed Shared, but use these terms here.

similar to instance_pool_id, but for driver node. If omitted, and instance_pool_id is specified, then the driver will be allocated from that pool.

The node type of the Spark driver. This field is optional; if unset, API will set the driver node type to the same value as node_type_id defined above.

If you don’t want to allocate a fixed number of EBS volumes at cluster creation time, use autoscaling local storage. With autoscaling local storage, Databricks monitors the amount of free disk space available on your cluster’s Spark workers. If a worker begins to run too low on disk, Databricks automatically attaches a new EBS volume to the worker before it runs out of disk space. EBS volumes are attached up to a limit of 5 TB of total disk space per instance (including the instance’s local storage). To scale down EBS usage, make sure you have autotermination_minutes and autoscale attributes set. More documentation available at cluster configuration page.

Some instance types you use to run clusters may have locally attached disks. Databricks may store shuffle data or temporary data on these locally attached disks. To ensure that all data at rest is encrypted for all storage types, including shuffle data stored temporarily on your cluster’s local disks, you can enable local disk encryption. When local disk encryption is enabled, Databricks generates an encryption key locally unique to each cluster node and uses it to encrypt all data stored on local disks. The scope of the key is local to each cluster node and is destroyed along with the cluster node itself. During its lifetime, the key resides in memory for encryption and decryption and is stored encrypted on the disk. Your workloads may run more slowly because of the performance impact of reading and writing encrypted data to and from local volumes. This feature is not available for all Azure Databricks subscriptions. Contact your Microsoft or Databricks account representative to request access.

An optional token to guarantee the idempotency of cluster creation requests. If an active cluster with the provided token already exists, the request will not create a new cluster, but it will return the existing running cluster's ID instead. If you specify the idempotency token, upon failure, you can retry until the request succeeds. Databricks platform guarantees to launch exactly one cluster with that idempotency token. This token should have at most 64 characters.

To reduce cluster start time, you can attach a cluster to a predefined pool of idle instances. When attached to a pool, a cluster allocates its driver and worker nodes from the pool. If the pool does not have sufficient idle resources to accommodate the cluster’s request, it expands by allocating new instances from the instance provider. When an attached cluster changes its state to TERMINATED, the instances it used are returned to the pool and reused by a different cluster.

boolean value specifying if the cluster is pinned (not pinned by default). You must be a Databricks administrator to use this. The pinned clusters' maximum number is limited to 100, so apply may fail if you have more than that (this number may change over time, so check Databricks documentation for actual number).

The following example demonstrates how to create an autoscaling cluster with Delta Cache enabled:

import * as pulumi from "@pulumi/pulumi";
import * as databricks from "@pulumi/databricks";
const smallest = databricks.getNodeType({
localDisk: true,
});
const latestLts = databricks.getSparkVersion({
longTermSupport: true,
});
const sharedAutoscaling = new databricks.Cluster("sharedAutoscaling", {
clusterName: "Shared Autoscaling",
sparkVersion: latestLts.then(latestLts => latestLts.id),
nodeTypeId: smallest.then(smallest => smallest.id),
autoterminationMinutes: 20,
autoscale: {
minWorkers: 1,
maxWorkers: 50,
},
sparkConf: {
"spark.databricks.io.cache.enabled": true,
"spark.databricks.io.cache.maxDiskUsage": "50g",
"spark.databricks.io.cache.maxMetaDataCache": "1g",
},
});

import pulumi
import pulumi_databricks as databricks

smallest = databricks.get_node_type(local_disk=True)
latest_lts = databricks.get_spark_version(long_term_support=True)
shared_autoscaling = databricks.Cluster("sharedAutoscaling",
    cluster_name="Shared Autoscaling",
    spark_version=latest_lts.id,
    node_type_id=smallest.id,
    autotermination_minutes=20,
    autoscale=databricks.ClusterAutoscaleArgs(
        min_workers=1,
        max_workers=50,
    ),
    spark_conf={
        "spark.databricks.io.cache.enabled": True,
        "spark.databricks.io.cache.maxDiskUsage": "50g",
        "spark.databricks.io.cache.maxMetaDataCache": "1g",
    })

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Databricks = Pulumi.Databricks;

return await Deployment.RunAsync(() => 
{
    var smallest = Databricks.GetNodeType.Invoke(new()
    {
        LocalDisk = true,
    });

    var latestLts = Databricks.GetSparkVersion.Invoke(new()
    {
        LongTermSupport = true,
    });

    var sharedAutoscaling = new Databricks.Cluster("sharedAutoscaling", new()
    {
        ClusterName = "Shared Autoscaling",
        SparkVersion = latestLts.Apply(getSparkVersionResult => getSparkVersionResult.Id),
        NodeTypeId = smallest.Apply(getNodeTypeResult => getNodeTypeResult.Id),
        AutoterminationMinutes = 20,
        Autoscale = new Databricks.Inputs.ClusterAutoscaleArgs
        {
            MinWorkers = 1,
            MaxWorkers = 50,
        },
        SparkConf = 
        {
            { "spark.databricks.io.cache.enabled", true },
            { "spark.databricks.io.cache.maxDiskUsage", "50g" },
            { "spark.databricks.io.cache.maxMetaDataCache", "1g" },
        },
    });

});

package main

import (
	"github.com/pulumi/pulumi-databricks/sdk/go/databricks"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		smallest, err := databricks.GetNodeType(ctx, &databricks.GetNodeTypeArgs{
			LocalDisk: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		latestLts, err := databricks.GetSparkVersion(ctx, &databricks.GetSparkVersionArgs{
			LongTermSupport: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		_, err = databricks.NewCluster(ctx, "sharedAutoscaling", &databricks.ClusterArgs{
			ClusterName:            pulumi.String("Shared Autoscaling"),
			SparkVersion:           *pulumi.String(latestLts.Id),
			NodeTypeId:             *pulumi.String(smallest.Id),
			AutoterminationMinutes: pulumi.Int(20),
			Autoscale: &databricks.ClusterAutoscaleArgs{
				MinWorkers: pulumi.Int(1),
				MaxWorkers: pulumi.Int(50),
			},
			SparkConf: pulumi.Map{
				"spark.databricks.io.cache.enabled":          pulumi.Any(true),
				"spark.databricks.io.cache.maxDiskUsage":     pulumi.Any("50g"),
				"spark.databricks.io.cache.maxMetaDataCache": pulumi.Any("1g"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.databricks.DatabricksFunctions;
import com.pulumi.databricks.inputs.GetNodeTypeArgs;
import com.pulumi.databricks.inputs.GetSparkVersionArgs;
import com.pulumi.databricks.Cluster;
import com.pulumi.databricks.ClusterArgs;
import com.pulumi.databricks.inputs.ClusterAutoscaleArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        final var smallest = DatabricksFunctions.getNodeType(GetNodeTypeArgs.builder()
            .localDisk(true)
            .build());

        final var latestLts = DatabricksFunctions.getSparkVersion(GetSparkVersionArgs.builder()
            .longTermSupport(true)
            .build());

        var sharedAutoscaling = new Cluster("sharedAutoscaling", ClusterArgs.builder()        
            .clusterName("Shared Autoscaling")
            .sparkVersion(latestLts.applyValue(getSparkVersionResult -> getSparkVersionResult.id()))
            .nodeTypeId(smallest.applyValue(getNodeTypeResult -> getNodeTypeResult.id()))
            .autoterminationMinutes(20)
            .autoscale(ClusterAutoscaleArgs.builder()
                .minWorkers(1)
                .maxWorkers(50)
                .build())
            .sparkConf(Map.ofEntries(
                Map.entry("spark.databricks.io.cache.enabled", true),
                Map.entry("spark.databricks.io.cache.maxDiskUsage", "50g"),
                Map.entry("spark.databricks.io.cache.maxMetaDataCache", "1g")
            ))
            .build());

    }
}

resources:
  sharedAutoscaling:
    type: databricks:Cluster
    properties:
      clusterName: Shared Autoscaling
      sparkVersion: ${latestLts.id}
      nodeTypeId: ${smallest.id}
      autoterminationMinutes: 20
      autoscale:
        minWorkers: 1
        maxWorkers: 50
      sparkConf:
        spark.databricks.io.cache.enabled: true
        spark.databricks.io.cache.maxDiskUsage: 50g
        spark.databricks.io.cache.maxMetaDataCache: 1g
variables:
  smallest:
    fn::invoke:
      Function: databricks:getNodeType
      Arguments:
        localDisk: true
  latestLts:
    fn::invoke:
      Function: databricks:getSparkVersion
      Arguments:
        longTermSupport: true

Any supported databricks.getNodeType id. If instance_pool_id is specified, this field is not needed.

Number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

The type of runtime engine to use. If not specified, the runtime engine type is inferred based on the spark_version value. Allowed values include: PHOTON, STANDARD.

The optional user name of the user to assign to an interactive cluster. This field is required when using data_security_mode set to SINGLE_USER or AAD Passthrough for Azure Data Lake Storage (ADLS) with a single-user cluster (i.e., not high-concurrency clusters).

Map with key-value pairs to fine-tune Spark clusters, where you can provide custom Spark configuration properties in a cluster configuration.

Map with environment variable key-value pairs to fine-tune Spark clusters. Key-value pairs of the form (X,Y) are exported (i.e., X='Y') while launching the driver and workers.

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. You can specify up to 10 keys.

Runtime version of the cluster. Any supported databricks.getSparkVersion id. We advise using Cluster Policies to restrict the list of versions for simplicity while maintaining enough control.

Whether to use policy default values for missing cluster attributes.

Automatically terminate the cluster after being inactive for this time in minutes. If specified, the threshold must be between 10 and 10000 minutes. You can also set this value to 0 to explicitly disable automatic termination. Defaults to 60. We highly recommend having this setting present for Interactive/BI clusters.

Cluster name, which doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS EC2 instances and EBS volumes) with these tags in addition to default_tags. If a custom cluster tag has the same name as a default cluster tag, the custom tag is prefixed with an x_ when it is propagated.

Select the security features of the cluster. Unity Catalog requires SINGLE_USER or USER_ISOLATION mode. LEGACY_PASSTHROUGH for passthrough cluster and LEGACY_TABLE_ACL for Table ACL cluster. If omitted, no security features are enabled. In the Databricks UI, this has been recently been renamed Access Mode and USER_ISOLATION has been renamed Shared, but use these terms here.

similar to instance_pool_id, but for driver node. If omitted, and instance_pool_id is specified, then the driver will be allocated from that pool.

The node type of the Spark driver. This field is optional; if unset, API will set the driver node type to the same value as node_type_id defined above.

If you don’t want to allocate a fixed number of EBS volumes at cluster creation time, use autoscaling local storage. With autoscaling local storage, Databricks monitors the amount of free disk space available on your cluster’s Spark workers. If a worker begins to run too low on disk, Databricks automatically attaches a new EBS volume to the worker before it runs out of disk space. EBS volumes are attached up to a limit of 5 TB of total disk space per instance (including the instance’s local storage). To scale down EBS usage, make sure you have autotermination_minutes and autoscale attributes set. More documentation available at cluster configuration page.

Some instance types you use to run clusters may have locally attached disks. Databricks may store shuffle data or temporary data on these locally attached disks. To ensure that all data at rest is encrypted for all storage types, including shuffle data stored temporarily on your cluster’s local disks, you can enable local disk encryption. When local disk encryption is enabled, Databricks generates an encryption key locally unique to each cluster node and uses it to encrypt all data stored on local disks. The scope of the key is local to each cluster node and is destroyed along with the cluster node itself. During its lifetime, the key resides in memory for encryption and decryption and is stored encrypted on the disk. Your workloads may run more slowly because of the performance impact of reading and writing encrypted data to and from local volumes. This feature is not available for all Azure Databricks subscriptions. Contact your Microsoft or Databricks account representative to request access.

An optional token to guarantee the idempotency of cluster creation requests. If an active cluster with the provided token already exists, the request will not create a new cluster, but it will return the existing running cluster's ID instead. If you specify the idempotency token, upon failure, you can retry until the request succeeds. Databricks platform guarantees to launch exactly one cluster with that idempotency token. This token should have at most 64 characters.

To reduce cluster start time, you can attach a cluster to a predefined pool of idle instances. When attached to a pool, a cluster allocates its driver and worker nodes from the pool. If the pool does not have sufficient idle resources to accommodate the cluster’s request, it expands by allocating new instances from the instance provider. When an attached cluster changes its state to TERMINATED, the instances it used are returned to the pool and reused by a different cluster.

boolean value specifying if the cluster is pinned (not pinned by default). You must be a Databricks administrator to use this. The pinned clusters' maximum number is limited to 100, so apply may fail if you have more than that (this number may change over time, so check Databricks documentation for actual number).

The following example demonstrates how to create an autoscaling cluster with Delta Cache enabled:

import * as pulumi from "@pulumi/pulumi";
import * as databricks from "@pulumi/databricks";
const smallest = databricks.getNodeType({
localDisk: true,
});
const latestLts = databricks.getSparkVersion({
longTermSupport: true,
});
const sharedAutoscaling = new databricks.Cluster("sharedAutoscaling", {
clusterName: "Shared Autoscaling",
sparkVersion: latestLts.then(latestLts => latestLts.id),
nodeTypeId: smallest.then(smallest => smallest.id),
autoterminationMinutes: 20,
autoscale: {
minWorkers: 1,
maxWorkers: 50,
},
sparkConf: {
"spark.databricks.io.cache.enabled": true,
"spark.databricks.io.cache.maxDiskUsage": "50g",
"spark.databricks.io.cache.maxMetaDataCache": "1g",
},
});

import pulumi
import pulumi_databricks as databricks

smallest = databricks.get_node_type(local_disk=True)
latest_lts = databricks.get_spark_version(long_term_support=True)
shared_autoscaling = databricks.Cluster("sharedAutoscaling",
    cluster_name="Shared Autoscaling",
    spark_version=latest_lts.id,
    node_type_id=smallest.id,
    autotermination_minutes=20,
    autoscale=databricks.ClusterAutoscaleArgs(
        min_workers=1,
        max_workers=50,
    ),
    spark_conf={
        "spark.databricks.io.cache.enabled": True,
        "spark.databricks.io.cache.maxDiskUsage": "50g",
        "spark.databricks.io.cache.maxMetaDataCache": "1g",
    })

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Databricks = Pulumi.Databricks;

return await Deployment.RunAsync(() => 
{
    var smallest = Databricks.GetNodeType.Invoke(new()
    {
        LocalDisk = true,
    });

    var latestLts = Databricks.GetSparkVersion.Invoke(new()
    {
        LongTermSupport = true,
    });

    var sharedAutoscaling = new Databricks.Cluster("sharedAutoscaling", new()
    {
        ClusterName = "Shared Autoscaling",
        SparkVersion = latestLts.Apply(getSparkVersionResult => getSparkVersionResult.Id),
        NodeTypeId = smallest.Apply(getNodeTypeResult => getNodeTypeResult.Id),
        AutoterminationMinutes = 20,
        Autoscale = new Databricks.Inputs.ClusterAutoscaleArgs
        {
            MinWorkers = 1,
            MaxWorkers = 50,
        },
        SparkConf = 
        {
            { "spark.databricks.io.cache.enabled", true },
            { "spark.databricks.io.cache.maxDiskUsage", "50g" },
            { "spark.databricks.io.cache.maxMetaDataCache", "1g" },
        },
    });

});

package main

import (
	"github.com/pulumi/pulumi-databricks/sdk/go/databricks"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		smallest, err := databricks.GetNodeType(ctx, &databricks.GetNodeTypeArgs{
			LocalDisk: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		latestLts, err := databricks.GetSparkVersion(ctx, &databricks.GetSparkVersionArgs{
			LongTermSupport: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		_, err = databricks.NewCluster(ctx, "sharedAutoscaling", &databricks.ClusterArgs{
			ClusterName:            pulumi.String("Shared Autoscaling"),
			SparkVersion:           *pulumi.String(latestLts.Id),
			NodeTypeId:             *pulumi.String(smallest.Id),
			AutoterminationMinutes: pulumi.Int(20),
			Autoscale: &databricks.ClusterAutoscaleArgs{
				MinWorkers: pulumi.Int(1),
				MaxWorkers: pulumi.Int(50),
			},
			SparkConf: pulumi.Map{
				"spark.databricks.io.cache.enabled":          pulumi.Any(true),
				"spark.databricks.io.cache.maxDiskUsage":     pulumi.Any("50g"),
				"spark.databricks.io.cache.maxMetaDataCache": pulumi.Any("1g"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.databricks.DatabricksFunctions;
import com.pulumi.databricks.inputs.GetNodeTypeArgs;
import com.pulumi.databricks.inputs.GetSparkVersionArgs;
import com.pulumi.databricks.Cluster;
import com.pulumi.databricks.ClusterArgs;
import com.pulumi.databricks.inputs.ClusterAutoscaleArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        final var smallest = DatabricksFunctions.getNodeType(GetNodeTypeArgs.builder()
            .localDisk(true)
            .build());

        final var latestLts = DatabricksFunctions.getSparkVersion(GetSparkVersionArgs.builder()
            .longTermSupport(true)
            .build());

        var sharedAutoscaling = new Cluster("sharedAutoscaling", ClusterArgs.builder()        
            .clusterName("Shared Autoscaling")
            .sparkVersion(latestLts.applyValue(getSparkVersionResult -> getSparkVersionResult.id()))
            .nodeTypeId(smallest.applyValue(getNodeTypeResult -> getNodeTypeResult.id()))
            .autoterminationMinutes(20)
            .autoscale(ClusterAutoscaleArgs.builder()
                .minWorkers(1)
                .maxWorkers(50)
                .build())
            .sparkConf(Map.ofEntries(
                Map.entry("spark.databricks.io.cache.enabled", true),
                Map.entry("spark.databricks.io.cache.maxDiskUsage", "50g"),
                Map.entry("spark.databricks.io.cache.maxMetaDataCache", "1g")
            ))
            .build());

    }
}

resources:
  sharedAutoscaling:
    type: databricks:Cluster
    properties:
      clusterName: Shared Autoscaling
      sparkVersion: ${latestLts.id}
      nodeTypeId: ${smallest.id}
      autoterminationMinutes: 20
      autoscale:
        minWorkers: 1
        maxWorkers: 50
      sparkConf:
        spark.databricks.io.cache.enabled: true
        spark.databricks.io.cache.maxDiskUsage: 50g
        spark.databricks.io.cache.maxMetaDataCache: 1g
variables:
  smallest:
    fn::invoke:
      Function: databricks:getNodeType
      Arguments:
        localDisk: true
  latestLts:
    fn::invoke:
      Function: databricks:getSparkVersion
      Arguments:
        longTermSupport: true

Any supported databricks.getNodeType id. If instance_pool_id is specified, this field is not needed.

Number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

The type of runtime engine to use. If not specified, the runtime engine type is inferred based on the spark_version value. Allowed values include: PHOTON, STANDARD.

The optional user name of the user to assign to an interactive cluster. This field is required when using data_security_mode set to SINGLE_USER or AAD Passthrough for Azure Data Lake Storage (ADLS) with a single-user cluster (i.e., not high-concurrency clusters).

Map with key-value pairs to fine-tune Spark clusters, where you can provide custom Spark configuration properties in a cluster configuration.

Map with environment variable key-value pairs to fine-tune Spark clusters. Key-value pairs of the form (X,Y) are exported (i.e., X='Y') while launching the driver and workers.

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. You can specify up to 10 keys.

Runtime version of the cluster. Any supported databricks.getSparkVersion id. We advise using Cluster Policies to restrict the list of versions for simplicity while maintaining enough control.

Whether to use policy default values for missing cluster attributes.

Automatically terminate the cluster after being inactive for this time in minutes. If specified, the threshold must be between 10 and 10000 minutes. You can also set this value to 0 to explicitly disable automatic termination. Defaults to 60. We highly recommend having this setting present for Interactive/BI clusters.

Cluster name, which doesn’t have to be unique. If not specified at creation, the cluster name will be an empty string.

Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS EC2 instances and EBS volumes) with these tags in addition to default_tags. If a custom cluster tag has the same name as a default cluster tag, the custom tag is prefixed with an x_ when it is propagated.

Select the security features of the cluster. Unity Catalog requires SINGLE_USER or USER_ISOLATION mode. LEGACY_PASSTHROUGH for passthrough cluster and LEGACY_TABLE_ACL for Table ACL cluster. If omitted, no security features are enabled. In the Databricks UI, this has been recently been renamed Access Mode and USER_ISOLATION has been renamed Shared, but use these terms here.

similar to instance_pool_id, but for driver node. If omitted, and instance_pool_id is specified, then the driver will be allocated from that pool.

The node type of the Spark driver. This field is optional; if unset, API will set the driver node type to the same value as node_type_id defined above.

If you don’t want to allocate a fixed number of EBS volumes at cluster creation time, use autoscaling local storage. With autoscaling local storage, Databricks monitors the amount of free disk space available on your cluster’s Spark workers. If a worker begins to run too low on disk, Databricks automatically attaches a new EBS volume to the worker before it runs out of disk space. EBS volumes are attached up to a limit of 5 TB of total disk space per instance (including the instance’s local storage). To scale down EBS usage, make sure you have autotermination_minutes and autoscale attributes set. More documentation available at cluster configuration page.

Some instance types you use to run clusters may have locally attached disks. Databricks may store shuffle data or temporary data on these locally attached disks. To ensure that all data at rest is encrypted for all storage types, including shuffle data stored temporarily on your cluster’s local disks, you can enable local disk encryption. When local disk encryption is enabled, Databricks generates an encryption key locally unique to each cluster node and uses it to encrypt all data stored on local disks. The scope of the key is local to each cluster node and is destroyed along with the cluster node itself. During its lifetime, the key resides in memory for encryption and decryption and is stored encrypted on the disk. Your workloads may run more slowly because of the performance impact of reading and writing encrypted data to and from local volumes. This feature is not available for all Azure Databricks subscriptions. Contact your Microsoft or Databricks account representative to request access.

An optional token to guarantee the idempotency of cluster creation requests. If an active cluster with the provided token already exists, the request will not create a new cluster, but it will return the existing running cluster's ID instead. If you specify the idempotency token, upon failure, you can retry until the request succeeds. Databricks platform guarantees to launch exactly one cluster with that idempotency token. This token should have at most 64 characters.

To reduce cluster start time, you can attach a cluster to a predefined pool of idle instances. When attached to a pool, a cluster allocates its driver and worker nodes from the pool. If the pool does not have sufficient idle resources to accommodate the cluster’s request, it expands by allocating new instances from the instance provider. When an attached cluster changes its state to TERMINATED, the instances it used are returned to the pool and reused by a different cluster.

boolean value specifying if the cluster is pinned (not pinned by default). You must be a Databricks administrator to use this. The pinned clusters' maximum number is limited to 100, so apply may fail if you have more than that (this number may change over time, so check Databricks documentation for actual number).

The following example demonstrates how to create an autoscaling cluster with Delta Cache enabled:

import * as pulumi from "@pulumi/pulumi";
import * as databricks from "@pulumi/databricks";
const smallest = databricks.getNodeType({
localDisk: true,
});
const latestLts = databricks.getSparkVersion({
longTermSupport: true,
});
const sharedAutoscaling = new databricks.Cluster("sharedAutoscaling", {
clusterName: "Shared Autoscaling",
sparkVersion: latestLts.then(latestLts => latestLts.id),
nodeTypeId: smallest.then(smallest => smallest.id),
autoterminationMinutes: 20,
autoscale: {
minWorkers: 1,
maxWorkers: 50,
},
sparkConf: {
"spark.databricks.io.cache.enabled": true,
"spark.databricks.io.cache.maxDiskUsage": "50g",
"spark.databricks.io.cache.maxMetaDataCache": "1g",
},
});

import pulumi
import pulumi_databricks as databricks

smallest = databricks.get_node_type(local_disk=True)
latest_lts = databricks.get_spark_version(long_term_support=True)
shared_autoscaling = databricks.Cluster("sharedAutoscaling",
    cluster_name="Shared Autoscaling",
    spark_version=latest_lts.id,
    node_type_id=smallest.id,
    autotermination_minutes=20,
    autoscale=databricks.ClusterAutoscaleArgs(
        min_workers=1,
        max_workers=50,
    ),
    spark_conf={
        "spark.databricks.io.cache.enabled": True,
        "spark.databricks.io.cache.maxDiskUsage": "50g",
        "spark.databricks.io.cache.maxMetaDataCache": "1g",
    })

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Databricks = Pulumi.Databricks;

return await Deployment.RunAsync(() => 
{
    var smallest = Databricks.GetNodeType.Invoke(new()
    {
        LocalDisk = true,
    });

    var latestLts = Databricks.GetSparkVersion.Invoke(new()
    {
        LongTermSupport = true,
    });

    var sharedAutoscaling = new Databricks.Cluster("sharedAutoscaling", new()
    {
        ClusterName = "Shared Autoscaling",
        SparkVersion = latestLts.Apply(getSparkVersionResult => getSparkVersionResult.Id),
        NodeTypeId = smallest.Apply(getNodeTypeResult => getNodeTypeResult.Id),
        AutoterminationMinutes = 20,
        Autoscale = new Databricks.Inputs.ClusterAutoscaleArgs
        {
            MinWorkers = 1,
            MaxWorkers = 50,
        },
        SparkConf = 
        {
            { "spark.databricks.io.cache.enabled", true },
            { "spark.databricks.io.cache.maxDiskUsage", "50g" },
            { "spark.databricks.io.cache.maxMetaDataCache", "1g" },
        },
    });

});

package main

import (
	"github.com/pulumi/pulumi-databricks/sdk/go/databricks"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		smallest, err := databricks.GetNodeType(ctx, &databricks.GetNodeTypeArgs{
			LocalDisk: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		latestLts, err := databricks.GetSparkVersion(ctx, &databricks.GetSparkVersionArgs{
			LongTermSupport: pulumi.BoolRef(true),
		}, nil)
		if err != nil {
			return err
		}
		_, err = databricks.NewCluster(ctx, "sharedAutoscaling", &databricks.ClusterArgs{
			ClusterName:            pulumi.String("Shared Autoscaling"),
			SparkVersion:           *pulumi.String(latestLts.Id),
			NodeTypeId:             *pulumi.String(smallest.Id),
			AutoterminationMinutes: pulumi.Int(20),
			Autoscale: &databricks.ClusterAutoscaleArgs{
				MinWorkers: pulumi.Int(1),
				MaxWorkers: pulumi.Int(50),
			},
			SparkConf: pulumi.Map{
				"spark.databricks.io.cache.enabled":          pulumi.Any(true),
				"spark.databricks.io.cache.maxDiskUsage":     pulumi.Any("50g"),
				"spark.databricks.io.cache.maxMetaDataCache": pulumi.Any("1g"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.databricks.DatabricksFunctions;
import com.pulumi.databricks.inputs.GetNodeTypeArgs;
import com.pulumi.databricks.inputs.GetSparkVersionArgs;
import com.pulumi.databricks.Cluster;
import com.pulumi.databricks.ClusterArgs;
import com.pulumi.databricks.inputs.ClusterAutoscaleArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        final var smallest = DatabricksFunctions.getNodeType(GetNodeTypeArgs.builder()
            .localDisk(true)
            .build());

        final var latestLts = DatabricksFunctions.getSparkVersion(GetSparkVersionArgs.builder()
            .longTermSupport(true)
            .build());

        var sharedAutoscaling = new Cluster("sharedAutoscaling", ClusterArgs.builder()        
            .clusterName("Shared Autoscaling")
            .sparkVersion(latestLts.applyValue(getSparkVersionResult -> getSparkVersionResult.id()))
            .nodeTypeId(smallest.applyValue(getNodeTypeResult -> getNodeTypeResult.id()))
            .autoterminationMinutes(20)
            .autoscale(ClusterAutoscaleArgs.builder()
                .minWorkers(1)
                .maxWorkers(50)
                .build())
            .sparkConf(Map.ofEntries(
                Map.entry("spark.databricks.io.cache.enabled", true),
                Map.entry("spark.databricks.io.cache.maxDiskUsage", "50g"),
                Map.entry("spark.databricks.io.cache.maxMetaDataCache", "1g")
            ))
            .build());

    }
}

resources:
  sharedAutoscaling:
    type: databricks:Cluster
    properties:
      clusterName: Shared Autoscaling
      sparkVersion: ${latestLts.id}
      nodeTypeId: ${smallest.id}
      autoterminationMinutes: 20
      autoscale:
        minWorkers: 1
        maxWorkers: 50
      sparkConf:
        spark.databricks.io.cache.enabled: true
        spark.databricks.io.cache.maxDiskUsage: 50g
        spark.databricks.io.cache.maxMetaDataCache: 1g
variables:
  smallest:
    fn::invoke:
      Function: databricks:getNodeType
      Arguments:
        localDisk: true
  latestLts:
    fn::invoke:
      Function: databricks:getSparkVersion
      Arguments:
        longTermSupport: true

Any supported databricks.getNodeType id. If instance_pool_id is specified, this field is not needed.

Number of worker nodes that this cluster should have. A cluster has one Spark driver and num_workers executors for a total of num_workers + 1 Spark nodes.

The type of runtime engine to use. If not specified, the runtime engine type is inferred based on the spark_version value. Allowed values include: PHOTON, STANDARD.

The optional user name of the user to assign to an interactive cluster. This field is required when using data_security_mode set to SINGLE_USER or AAD Passthrough for Azure Data Lake Storage (ADLS) with a single-user cluster (i.e., not high-concurrency clusters).

Map with key-value pairs to fine-tune Spark clusters, where you can provide custom Spark configuration properties in a cluster configuration.

Map with environment variable key-value pairs to fine-tune Spark clusters. Key-value pairs of the form (X,Y) are exported (i.e., X='Y') while launching the driver and workers.

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. You can specify up to 10 keys.

(map) Tags that are added by Databricks by default, regardless of any custom_tags that may have been added. These include: Vendor: Databricks, Creator: <username_of_creator>, ClusterName: <name_of_cluster>, ClusterId: <id_of_cluster>, Name: , and any workspace and pool tags.

The provider-assigned unique ID for this managed resource.

(string) State of the cluster.

(map) Tags that are added by Databricks by default, regardless of any custom_tags that may have been added. These include: Vendor: Databricks, Creator: <username_of_creator>, ClusterName: <name_of_cluster>, ClusterId: <id_of_cluster>, Name: , and any workspace and pool tags.

The provider-assigned unique ID for this managed resource.

(string) State of the cluster.

(map) Tags that are added by Databricks by default, regardless of any custom_tags that may have been added. These include: Vendor: Databricks, Creator: <username_of_creator>, ClusterName: <name_of_cluster>, ClusterId: <id_of_cluster>, Name: , and any workspace and pool tags.

The provider-assigned unique ID for this managed resource.

(string) State of the cluster.

(map) Tags that are added by Databricks by default, regardless of any custom_tags that may have been added. These include: Vendor: Databricks, Creator: <username_of_creator>, ClusterName: <name_of_cluster>, ClusterId: <id_of_cluster>, Name: , and any workspace and pool tags.

The provider-assigned unique ID for this managed resource.

(string) State of the cluster.

(map) Tags that are added by Databricks by default, regardless of any custom_tags that may have been added. These include: Vendor: Databricks, Creator: <username_of_creator>, ClusterName: <name_of_cluster>, ClusterId: <id_of_cluster>, Name: , and any workspace and pool tags.

The provider-assigned unique ID for this managed resource.

(string) State of the cluster.

(map) Tags that are added by Databricks by default, regardless of any custom_tags that may have been added. These include: Vendor: Databricks, Creator: <username_of_creator>, ClusterName: <name_of_cluster>, ClusterId: <id_of_cluster>, Name: , and any workspace and pool tags.

The provider-assigned unique ID for this managed resource.

(string) State of the cluster.

Deprecated:

Please use 'availability' instead.

Deprecated:

Please use 'availability' instead.

Deprecated:

Please use 'availability' instead.

Deprecated:

Please use 'availability' instead.

Deprecated:

Please use 'availability' instead.

Deprecated:

Please use 'availability' instead.

Deprecated:

For init scripts use 'volumes', 'workspace' or cloud storage location instead of 'dbfs'.

Deprecated:

For init scripts use 'volumes', 'workspace' or cloud storage location instead of 'dbfs'.

Deprecated:

For init scripts use 'volumes', 'workspace' or cloud storage location instead of 'dbfs'.

Deprecated:

For init scripts use 'volumes', 'workspace' or cloud storage location instead of 'dbfs'.

Deprecated:

For init scripts use 'volumes', 'workspace' or cloud storage location instead of 'dbfs'.

Deprecated:

For init scripts use 'volumes', 'workspace' or cloud storage location instead of 'dbfs'.