dataproc

This provider is a derived work of the Terraform Provider distributed under MPL 2.0. If you encounter a bug or missing feature, first check the pulumi/pulumi-gcp repo; however, if that doesn’t turn up anything, please consult the source terraform-providers/terraform-provider-google repo.

class pulumi_gcp.dataproc.AutoscalingPolicy(resource_name, opts=None, basic_algorithm=None, location=None, policy_id=None, project=None, secondary_worker_config=None, worker_config=None, __props__=None, __name__=None, __opts__=None)

Create a AutoscalingPolicy resource with the given unique name, props, and options.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • project (pulumi.Input[str]) – The ID of the project in which the resource belongs. If it is not provided, the provider project is used.

The basic_algorithm object supports the following:

  • cooldownPeriod (pulumi.Input[str])

  • yarnConfig (pulumi.Input[dict])

    • gracefulDecommissionTimeout (pulumi.Input[str])

    • scaleDownFactor (pulumi.Input[float])

    • scaleDownMinWorkerFraction (pulumi.Input[float])

    • scaleUpFactor (pulumi.Input[float])

    • scaleUpMinWorkerFraction (pulumi.Input[float])

The secondary_worker_config object supports the following:

  • max_instances (pulumi.Input[float])

  • minInstances (pulumi.Input[float])

  • weight (pulumi.Input[float])

The worker_config object supports the following:

  • max_instances (pulumi.Input[float])

  • minInstances (pulumi.Input[float])

  • weight (pulumi.Input[float])

project = None

The ID of the project in which the resource belongs. If it is not provided, the provider project is used.

static get(resource_name, id, opts=None, basic_algorithm=None, location=None, name=None, policy_id=None, project=None, secondary_worker_config=None, worker_config=None)

Get an existing AutoscalingPolicy resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • project (pulumi.Input[str]) – The ID of the project in which the resource belongs. If it is not provided, the provider project is used.

The basic_algorithm object supports the following:

  • cooldownPeriod (pulumi.Input[str])

  • yarnConfig (pulumi.Input[dict])

    • gracefulDecommissionTimeout (pulumi.Input[str])

    • scaleDownFactor (pulumi.Input[float])

    • scaleDownMinWorkerFraction (pulumi.Input[float])

    • scaleUpFactor (pulumi.Input[float])

    • scaleUpMinWorkerFraction (pulumi.Input[float])

The secondary_worker_config object supports the following:

  • max_instances (pulumi.Input[float])

  • minInstances (pulumi.Input[float])

  • weight (pulumi.Input[float])

The worker_config object supports the following:

  • max_instances (pulumi.Input[float])

  • minInstances (pulumi.Input[float])

  • weight (pulumi.Input[float])

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

class pulumi_gcp.dataproc.Cluster(resource_name, opts=None, cluster_config=None, labels=None, name=None, project=None, region=None, __props__=None, __name__=None, __opts__=None)

Manages a Cloud Dataproc cluster resource within GCP. For more information see the official dataproc documentation.

!> Warning: Due to limitations of the API, all arguments except labels,cluster_config.worker_config.num_instances and cluster_config.preemptible_worker_config.num_instances are non-updatable. Changing others will cause recreation of the whole cluster!

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • cluster_config (pulumi.Input[dict]) – Allows you to configure various aspects of the cluster. Structure defined below.

  • labels (pulumi.Input[dict]) – The list of labels (key/value pairs) to be applied to instances in the cluster. GCP generates some itself including goog-dataproc-cluster-name which is the name of the cluster.

  • name (pulumi.Input[str]) – The name of the cluster, unique within the project and zone.

  • project (pulumi.Input[str]) – The ID of the project in which the cluster will exist. If it is not provided, the provider project is used.

  • region (pulumi.Input[str]) – The region in which the cluster and associated nodes will be created in. Defaults to global.

The cluster_config object supports the following:

  • autoscalingConfig (pulumi.Input[dict])

    • policyUri (pulumi.Input[str])

  • bucket (pulumi.Input[str])

  • encryptionConfig (pulumi.Input[dict])

    • kmsKeyName (pulumi.Input[str])

  • gceClusterConfig (pulumi.Input[dict])

    • internalIpOnly (pulumi.Input[bool])

    • metadata (pulumi.Input[dict])

    • network (pulumi.Input[str])

    • service_account (pulumi.Input[str])

    • serviceAccountScopes (pulumi.Input[list])

    • subnetwork (pulumi.Input[str])

    • tags (pulumi.Input[list])

    • zone (pulumi.Input[str])

  • initializationActions (pulumi.Input[list])

    • script (pulumi.Input[str])

    • timeout_sec (pulumi.Input[float])

  • masterConfig (pulumi.Input[dict])

    • accelerators (pulumi.Input[list])

      • acceleratorCount (pulumi.Input[float])

      • acceleratorType (pulumi.Input[str])

    • diskConfig (pulumi.Input[dict])

      • bootDiskSizeGb (pulumi.Input[float])

      • bootDiskType (pulumi.Input[str])

      • numLocalSsds (pulumi.Input[float])

    • imageUri (pulumi.Input[str])

    • instanceNames (pulumi.Input[list])

    • machine_type (pulumi.Input[str])

    • min_cpu_platform (pulumi.Input[str])

    • numInstances (pulumi.Input[float])

  • preemptibleWorkerConfig (pulumi.Input[dict])

    • diskConfig (pulumi.Input[dict])

      • bootDiskSizeGb (pulumi.Input[float])

      • bootDiskType (pulumi.Input[str])

      • numLocalSsds (pulumi.Input[float])

    • instanceNames (pulumi.Input[list])

    • numInstances (pulumi.Input[float])

  • softwareConfig (pulumi.Input[dict])

    • imageVersion (pulumi.Input[str])

    • optionalComponents (pulumi.Input[list])

    • overrideProperties (pulumi.Input[dict])

    • properties (pulumi.Input[dict])

  • stagingBucket (pulumi.Input[str])

  • worker_config (pulumi.Input[dict])

    • accelerators (pulumi.Input[list])

      • acceleratorCount (pulumi.Input[float])

      • acceleratorType (pulumi.Input[str])

    • diskConfig (pulumi.Input[dict])

      • bootDiskSizeGb (pulumi.Input[float])

      • bootDiskType (pulumi.Input[str])

      • numLocalSsds (pulumi.Input[float])

    • imageUri (pulumi.Input[str])

    • instanceNames (pulumi.Input[list])

    • machine_type (pulumi.Input[str])

    • min_cpu_platform (pulumi.Input[str])

    • numInstances (pulumi.Input[float])

cluster_config = None

Allows you to configure various aspects of the cluster. Structure defined below.

  • autoscalingConfig (dict)

    • policyUri (str)

  • bucket (str)

  • encryptionConfig (dict)

    • kmsKeyName (str)

  • gceClusterConfig (dict)

    • internalIpOnly (bool)

    • metadata (dict)

    • network (str)

    • service_account (str)

    • serviceAccountScopes (list)

    • subnetwork (str)

    • tags (list)

    • zone (str)

  • initializationActions (list)

    • script (str)

    • timeout_sec (float)

  • masterConfig (dict)

    • accelerators (list)

      • acceleratorCount (float)

      • acceleratorType (str)

    • diskConfig (dict)

      • bootDiskSizeGb (float)

      • bootDiskType (str)

      • numLocalSsds (float)

    • imageUri (str)

    • instanceNames (list)

    • machine_type (str)

    • min_cpu_platform (str)

    • numInstances (float)

  • preemptibleWorkerConfig (dict)

    • diskConfig (dict)

      • bootDiskSizeGb (float)

      • bootDiskType (str)

      • numLocalSsds (float)

    • instanceNames (list)

    • numInstances (float)

  • softwareConfig (dict)

    • imageVersion (str)

    • optionalComponents (list)

    • overrideProperties (dict)

    • properties (dict)

  • stagingBucket (str)

  • worker_config (dict)

    • accelerators (list)

      • acceleratorCount (float)

      • acceleratorType (str)

    • diskConfig (dict)

      • bootDiskSizeGb (float)

      • bootDiskType (str)

      • numLocalSsds (float)

    • imageUri (str)

    • instanceNames (list)

    • machine_type (str)

    • min_cpu_platform (str)

    • numInstances (float)

labels = None

The list of labels (key/value pairs) to be applied to instances in the cluster. GCP generates some itself including goog-dataproc-cluster-name which is the name of the cluster.

name = None

The name of the cluster, unique within the project and zone.

project = None

The ID of the project in which the cluster will exist. If it is not provided, the provider project is used.

region = None

The region in which the cluster and associated nodes will be created in. Defaults to global.

static get(resource_name, id, opts=None, cluster_config=None, labels=None, name=None, project=None, region=None)

Get an existing Cluster resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • cluster_config (pulumi.Input[dict]) – Allows you to configure various aspects of the cluster. Structure defined below.

  • labels (pulumi.Input[dict]) – The list of labels (key/value pairs) to be applied to instances in the cluster. GCP generates some itself including goog-dataproc-cluster-name which is the name of the cluster.

  • name (pulumi.Input[str]) – The name of the cluster, unique within the project and zone.

  • project (pulumi.Input[str]) – The ID of the project in which the cluster will exist. If it is not provided, the provider project is used.

  • region (pulumi.Input[str]) – The region in which the cluster and associated nodes will be created in. Defaults to global.

The cluster_config object supports the following:

  • autoscalingConfig (pulumi.Input[dict])

    • policyUri (pulumi.Input[str])

  • bucket (pulumi.Input[str])

  • encryptionConfig (pulumi.Input[dict])

    • kmsKeyName (pulumi.Input[str])

  • gceClusterConfig (pulumi.Input[dict])

    • internalIpOnly (pulumi.Input[bool])

    • metadata (pulumi.Input[dict])

    • network (pulumi.Input[str])

    • service_account (pulumi.Input[str])

    • serviceAccountScopes (pulumi.Input[list])

    • subnetwork (pulumi.Input[str])

    • tags (pulumi.Input[list])

    • zone (pulumi.Input[str])

  • initializationActions (pulumi.Input[list])

    • script (pulumi.Input[str])

    • timeout_sec (pulumi.Input[float])

  • masterConfig (pulumi.Input[dict])

    • accelerators (pulumi.Input[list])

      • acceleratorCount (pulumi.Input[float])

      • acceleratorType (pulumi.Input[str])

    • diskConfig (pulumi.Input[dict])

      • bootDiskSizeGb (pulumi.Input[float])

      • bootDiskType (pulumi.Input[str])

      • numLocalSsds (pulumi.Input[float])

    • imageUri (pulumi.Input[str])

    • instanceNames (pulumi.Input[list])

    • machine_type (pulumi.Input[str])

    • min_cpu_platform (pulumi.Input[str])

    • numInstances (pulumi.Input[float])

  • preemptibleWorkerConfig (pulumi.Input[dict])

    • diskConfig (pulumi.Input[dict])

      • bootDiskSizeGb (pulumi.Input[float])

      • bootDiskType (pulumi.Input[str])

      • numLocalSsds (pulumi.Input[float])

    • instanceNames (pulumi.Input[list])

    • numInstances (pulumi.Input[float])

  • softwareConfig (pulumi.Input[dict])

    • imageVersion (pulumi.Input[str])

    • optionalComponents (pulumi.Input[list])

    • overrideProperties (pulumi.Input[dict])

    • properties (pulumi.Input[dict])

  • stagingBucket (pulumi.Input[str])

  • worker_config (pulumi.Input[dict])

    • accelerators (pulumi.Input[list])

      • acceleratorCount (pulumi.Input[float])

      • acceleratorType (pulumi.Input[str])

    • diskConfig (pulumi.Input[dict])

      • bootDiskSizeGb (pulumi.Input[float])

      • bootDiskType (pulumi.Input[str])

      • numLocalSsds (pulumi.Input[float])

    • imageUri (pulumi.Input[str])

    • instanceNames (pulumi.Input[list])

    • machine_type (pulumi.Input[str])

    • min_cpu_platform (pulumi.Input[str])

    • numInstances (pulumi.Input[float])

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

class pulumi_gcp.dataproc.ClusterIAMBinding(resource_name, opts=None, cluster=None, condition=None, members=None, project=None, region=None, role=None, __props__=None, __name__=None, __opts__=None)

Three different resources help you manage IAM policies on dataproc clusters. Each of these resources serves a different use case:

  • dataproc.ClusterIAMPolicy: Authoritative. Sets the IAM policy for the cluster and replaces any existing policy already attached.

  • dataproc.ClusterIAMBinding: Authoritative for a given role. Updates the IAM policy to grant a role to a list of members. Other roles within the IAM policy for the cluster are preserved.

  • dataproc.ClusterIAMMember: Non-authoritative. Updates the IAM policy to grant a role to a new member. Other members for the role for the cluster are preserved.

Note: dataproc.ClusterIAMPolicy cannot be used in conjunction with dataproc.ClusterIAMBinding and dataproc.ClusterIAMMember or they will fight over what your policy should be. In addition, be careful not to accidentally unset ownership of the cluster as dataproc.ClusterIAMPolicy replaces the entire policy.

Note: dataproc.ClusterIAMBinding resources can be used in conjunction with dataproc.ClusterIAMMember resources only if they do not grant privilege to the same role.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • cluster (pulumi.Input[str]) – The name or relative resource id of the cluster to manage IAM policies for.

  • project (pulumi.Input[str]) – The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • role (pulumi.Input[str]) – The role that should be applied. Only one dataproc.ClusterIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

The condition object supports the following:

  • description (pulumi.Input[str])

  • expression (pulumi.Input[str])

  • title (pulumi.Input[str])

cluster = None

The name or relative resource id of the cluster to manage IAM policies for.

etag = None

(Computed) The etag of the clusters’s IAM policy.

project = None

The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

region = None

The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

role = None

The role that should be applied. Only one dataproc.ClusterIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

static get(resource_name, id, opts=None, cluster=None, condition=None, etag=None, members=None, project=None, region=None, role=None)

Get an existing ClusterIAMBinding resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • cluster (pulumi.Input[str]) – The name or relative resource id of the cluster to manage IAM policies for.

  • etag (pulumi.Input[str]) – (Computed) The etag of the clusters’s IAM policy.

  • project (pulumi.Input[str]) – The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • role (pulumi.Input[str]) – The role that should be applied. Only one dataproc.ClusterIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

The condition object supports the following:

  • description (pulumi.Input[str])

  • expression (pulumi.Input[str])

  • title (pulumi.Input[str])

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

class pulumi_gcp.dataproc.ClusterIAMMember(resource_name, opts=None, cluster=None, condition=None, member=None, project=None, region=None, role=None, __props__=None, __name__=None, __opts__=None)

Three different resources help you manage IAM policies on dataproc clusters. Each of these resources serves a different use case:

  • dataproc.ClusterIAMPolicy: Authoritative. Sets the IAM policy for the cluster and replaces any existing policy already attached.

  • dataproc.ClusterIAMBinding: Authoritative for a given role. Updates the IAM policy to grant a role to a list of members. Other roles within the IAM policy for the cluster are preserved.

  • dataproc.ClusterIAMMember: Non-authoritative. Updates the IAM policy to grant a role to a new member. Other members for the role for the cluster are preserved.

Note: dataproc.ClusterIAMPolicy cannot be used in conjunction with dataproc.ClusterIAMBinding and dataproc.ClusterIAMMember or they will fight over what your policy should be. In addition, be careful not to accidentally unset ownership of the cluster as dataproc.ClusterIAMPolicy replaces the entire policy.

Note: dataproc.ClusterIAMBinding resources can be used in conjunction with dataproc.ClusterIAMMember resources only if they do not grant privilege to the same role.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • cluster (pulumi.Input[str]) – The name or relative resource id of the cluster to manage IAM policies for.

  • project (pulumi.Input[str]) – The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • role (pulumi.Input[str]) – The role that should be applied. Only one dataproc.ClusterIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

The condition object supports the following:

  • description (pulumi.Input[str])

  • expression (pulumi.Input[str])

  • title (pulumi.Input[str])

cluster = None

The name or relative resource id of the cluster to manage IAM policies for.

etag = None

(Computed) The etag of the clusters’s IAM policy.

project = None

The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

region = None

The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

role = None

The role that should be applied. Only one dataproc.ClusterIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

static get(resource_name, id, opts=None, cluster=None, condition=None, etag=None, member=None, project=None, region=None, role=None)

Get an existing ClusterIAMMember resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • cluster (pulumi.Input[str]) – The name or relative resource id of the cluster to manage IAM policies for.

  • etag (pulumi.Input[str]) – (Computed) The etag of the clusters’s IAM policy.

  • project (pulumi.Input[str]) – The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • role (pulumi.Input[str]) – The role that should be applied. Only one dataproc.ClusterIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

The condition object supports the following:

  • description (pulumi.Input[str])

  • expression (pulumi.Input[str])

  • title (pulumi.Input[str])

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

class pulumi_gcp.dataproc.ClusterIAMPolicy(resource_name, opts=None, cluster=None, policy_data=None, project=None, region=None, __props__=None, __name__=None, __opts__=None)

Three different resources help you manage IAM policies on dataproc clusters. Each of these resources serves a different use case:

  • dataproc.ClusterIAMPolicy: Authoritative. Sets the IAM policy for the cluster and replaces any existing policy already attached.

  • dataproc.ClusterIAMBinding: Authoritative for a given role. Updates the IAM policy to grant a role to a list of members. Other roles within the IAM policy for the cluster are preserved.

  • dataproc.ClusterIAMMember: Non-authoritative. Updates the IAM policy to grant a role to a new member. Other members for the role for the cluster are preserved.

Note: dataproc.ClusterIAMPolicy cannot be used in conjunction with dataproc.ClusterIAMBinding and dataproc.ClusterIAMMember or they will fight over what your policy should be. In addition, be careful not to accidentally unset ownership of the cluster as dataproc.ClusterIAMPolicy replaces the entire policy.

Note: dataproc.ClusterIAMBinding resources can be used in conjunction with dataproc.ClusterIAMMember resources only if they do not grant privilege to the same role.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • cluster (pulumi.Input[str]) – The name or relative resource id of the cluster to manage IAM policies for.

  • policy_data (pulumi.Input[str]) – The policy data generated by a organizations.getIAMPolicy data source.

  • project (pulumi.Input[str]) – The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

cluster = None

The name or relative resource id of the cluster to manage IAM policies for.

etag = None

(Computed) The etag of the clusters’s IAM policy.

policy_data = None

The policy data generated by a organizations.getIAMPolicy data source.

project = None

The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

region = None

The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

static get(resource_name, id, opts=None, cluster=None, etag=None, policy_data=None, project=None, region=None)

Get an existing ClusterIAMPolicy resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • cluster (pulumi.Input[str]) – The name or relative resource id of the cluster to manage IAM policies for.

  • etag (pulumi.Input[str]) – (Computed) The etag of the clusters’s IAM policy.

  • policy_data (pulumi.Input[str]) – The policy data generated by a organizations.getIAMPolicy data source.

  • project (pulumi.Input[str]) – The project in which the cluster belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the cluster belongs. If it is not provided, this provider will use the provider default.

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

class pulumi_gcp.dataproc.Job(resource_name, opts=None, force_delete=None, hadoop_config=None, hive_config=None, labels=None, pig_config=None, placement=None, project=None, pyspark_config=None, reference=None, region=None, scheduling=None, spark_config=None, sparksql_config=None, __props__=None, __name__=None, __opts__=None)

Manages a job resource within a Dataproc cluster within GCE. For more information see the official dataproc documentation.

!> Note: This resource does not support ‘update’ and changing any attributes will cause the resource to be recreated.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • force_delete (pulumi.Input[bool]) – By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.

  • labels (pulumi.Input[dict]) – The list of labels (key/value pairs) to add to the job.

  • project (pulumi.Input[str]) – The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.

  • region (pulumi.Input[str]) – The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.

The hadoop_config object supports the following:

  • archiveUris (pulumi.Input[list])

  • args (pulumi.Input[list])

  • fileUris (pulumi.Input[list])

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • mainClass (pulumi.Input[str])

  • mainJarFileUri (pulumi.Input[str])

  • properties (pulumi.Input[dict])

The hive_config object supports the following:

  • continueOnFailure (pulumi.Input[bool])

  • jarFileUris (pulumi.Input[list])

  • properties (pulumi.Input[dict])

  • queryFileUri (pulumi.Input[str])

  • queryLists (pulumi.Input[list])

  • scriptVariables (pulumi.Input[dict])

The pig_config object supports the following:

  • continueOnFailure (pulumi.Input[bool])

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • properties (pulumi.Input[dict])

  • queryFileUri (pulumi.Input[str])

  • queryLists (pulumi.Input[list])

  • scriptVariables (pulumi.Input[dict])

The placement object supports the following:

  • clusterName (pulumi.Input[str])

  • clusterUuid (pulumi.Input[str])

The pyspark_config object supports the following:

  • archiveUris (pulumi.Input[list])

  • args (pulumi.Input[list])

  • fileUris (pulumi.Input[list])

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • mainPythonFileUri (pulumi.Input[str])

  • properties (pulumi.Input[dict])

  • pythonFileUris (pulumi.Input[list])

The reference object supports the following:

  • job_id (pulumi.Input[str])

The scheduling object supports the following:

  • maxFailuresPerHour (pulumi.Input[float])

The spark_config object supports the following:

  • archiveUris (pulumi.Input[list])

  • args (pulumi.Input[list])

  • fileUris (pulumi.Input[list])

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • mainClass (pulumi.Input[str])

  • mainJarFileUri (pulumi.Input[str])

  • properties (pulumi.Input[dict])

The sparksql_config object supports the following:

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • properties (pulumi.Input[dict])

  • queryFileUri (pulumi.Input[str])

  • queryLists (pulumi.Input[list])

  • scriptVariables (pulumi.Input[dict])

driver_controls_files_uri = None

If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.

driver_output_resource_uri = None

A URI pointing to the location of the stdout of the job’s driver program.

force_delete = None

By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.

labels = None

The list of labels (key/value pairs) to add to the job.

project = None

The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.

region = None

The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.

static get(resource_name, id, opts=None, driver_controls_files_uri=None, driver_output_resource_uri=None, force_delete=None, hadoop_config=None, hive_config=None, labels=None, pig_config=None, placement=None, project=None, pyspark_config=None, reference=None, region=None, scheduling=None, spark_config=None, sparksql_config=None, status=None)

Get an existing Job resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • driver_controls_files_uri (pulumi.Input[str]) – If present, the location of miscellaneous control files which may be used as part of job setup and handling. If not present, control files may be placed in the same location as driver_output_uri.

  • driver_output_resource_uri (pulumi.Input[str]) – A URI pointing to the location of the stdout of the job’s driver program.

  • force_delete (pulumi.Input[bool]) – By default, you can only delete inactive jobs within Dataproc. Setting this to true, and calling destroy, will ensure that the job is first cancelled before issuing the delete.

  • labels (pulumi.Input[dict]) – The list of labels (key/value pairs) to add to the job.

  • project (pulumi.Input[str]) – The project in which the cluster can be found and jobs subsequently run against. If it is not provided, the provider project is used.

  • region (pulumi.Input[str]) – The Cloud Dataproc region. This essentially determines which clusters are available for this job to be submitted to. If not specified, defaults to global.

The hadoop_config object supports the following:

  • archiveUris (pulumi.Input[list])

  • args (pulumi.Input[list])

  • fileUris (pulumi.Input[list])

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • mainClass (pulumi.Input[str])

  • mainJarFileUri (pulumi.Input[str])

  • properties (pulumi.Input[dict])

The hive_config object supports the following:

  • continueOnFailure (pulumi.Input[bool])

  • jarFileUris (pulumi.Input[list])

  • properties (pulumi.Input[dict])

  • queryFileUri (pulumi.Input[str])

  • queryLists (pulumi.Input[list])

  • scriptVariables (pulumi.Input[dict])

The pig_config object supports the following:

  • continueOnFailure (pulumi.Input[bool])

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • properties (pulumi.Input[dict])

  • queryFileUri (pulumi.Input[str])

  • queryLists (pulumi.Input[list])

  • scriptVariables (pulumi.Input[dict])

The placement object supports the following:

  • clusterName (pulumi.Input[str])

  • clusterUuid (pulumi.Input[str])

The pyspark_config object supports the following:

  • archiveUris (pulumi.Input[list])

  • args (pulumi.Input[list])

  • fileUris (pulumi.Input[list])

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • mainPythonFileUri (pulumi.Input[str])

  • properties (pulumi.Input[dict])

  • pythonFileUris (pulumi.Input[list])

The reference object supports the following:

  • job_id (pulumi.Input[str])

The scheduling object supports the following:

  • maxFailuresPerHour (pulumi.Input[float])

The spark_config object supports the following:

  • archiveUris (pulumi.Input[list])

  • args (pulumi.Input[list])

  • fileUris (pulumi.Input[list])

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • mainClass (pulumi.Input[str])

  • mainJarFileUri (pulumi.Input[str])

  • properties (pulumi.Input[dict])

The sparksql_config object supports the following:

  • jarFileUris (pulumi.Input[list])

  • loggingConfig (pulumi.Input[dict])

    • driverLogLevels (pulumi.Input[dict])

  • properties (pulumi.Input[dict])

  • queryFileUri (pulumi.Input[str])

  • queryLists (pulumi.Input[list])

  • scriptVariables (pulumi.Input[dict])

The status object supports the following:

  • details (pulumi.Input[str])

  • state (pulumi.Input[str])

  • stateStartTime (pulumi.Input[str])

  • substate (pulumi.Input[str])

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

class pulumi_gcp.dataproc.JobIAMBinding(resource_name, opts=None, condition=None, job_id=None, members=None, project=None, region=None, role=None, __props__=None, __name__=None, __opts__=None)

Three different resources help you manage IAM policies on dataproc jobs. Each of these resources serves a different use case:

  • dataproc.JobIAMPolicy: Authoritative. Sets the IAM policy for the job and replaces any existing policy already attached.

  • dataproc.JobIAMBinding: Authoritative for a given role. Updates the IAM policy to grant a role to a list of members. Other roles within the IAM policy for the job are preserved.

  • dataproc.JobIAMMember: Non-authoritative. Updates the IAM policy to grant a role to a new member. Other members for the role for the job are preserved.

Note: dataproc.JobIAMPolicy cannot be used in conjunction with dataproc.JobIAMBinding and dataproc.JobIAMMember or they will fight over what your policy should be. In addition, be careful not to accidentally unset ownership of the job as dataproc.JobIAMPolicy replaces the entire policy.

Note: dataproc.JobIAMBinding resources can be used in conjunction with dataproc.JobIAMMember resources only if they do not grant privilege to the same role.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • project (pulumi.Input[str]) – The project in which the job belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the job belongs. If it is not provided, this provider will use the provider default.

  • role (pulumi.Input[str]) – The role that should be applied. Only one dataproc.JobIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

The condition object supports the following:

  • description (pulumi.Input[str])

  • expression (pulumi.Input[str])

  • title (pulumi.Input[str])

etag = None

(Computed) The etag of the jobs’s IAM policy.

project = None

The project in which the job belongs. If it is not provided, this provider will use the provider default.

region = None

The region in which the job belongs. If it is not provided, this provider will use the provider default.

role = None

The role that should be applied. Only one dataproc.JobIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

static get(resource_name, id, opts=None, condition=None, etag=None, job_id=None, members=None, project=None, region=None, role=None)

Get an existing JobIAMBinding resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • etag (pulumi.Input[str]) – (Computed) The etag of the jobs’s IAM policy.

  • project (pulumi.Input[str]) – The project in which the job belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the job belongs. If it is not provided, this provider will use the provider default.

  • role (pulumi.Input[str]) – The role that should be applied. Only one dataproc.JobIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

The condition object supports the following:

  • description (pulumi.Input[str])

  • expression (pulumi.Input[str])

  • title (pulumi.Input[str])

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

class pulumi_gcp.dataproc.JobIAMMember(resource_name, opts=None, condition=None, job_id=None, member=None, project=None, region=None, role=None, __props__=None, __name__=None, __opts__=None)

Three different resources help you manage IAM policies on dataproc jobs. Each of these resources serves a different use case:

  • dataproc.JobIAMPolicy: Authoritative. Sets the IAM policy for the job and replaces any existing policy already attached.

  • dataproc.JobIAMBinding: Authoritative for a given role. Updates the IAM policy to grant a role to a list of members. Other roles within the IAM policy for the job are preserved.

  • dataproc.JobIAMMember: Non-authoritative. Updates the IAM policy to grant a role to a new member. Other members for the role for the job are preserved.

Note: dataproc.JobIAMPolicy cannot be used in conjunction with dataproc.JobIAMBinding and dataproc.JobIAMMember or they will fight over what your policy should be. In addition, be careful not to accidentally unset ownership of the job as dataproc.JobIAMPolicy replaces the entire policy.

Note: dataproc.JobIAMBinding resources can be used in conjunction with dataproc.JobIAMMember resources only if they do not grant privilege to the same role.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • project (pulumi.Input[str]) – The project in which the job belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the job belongs. If it is not provided, this provider will use the provider default.

  • role (pulumi.Input[str]) – The role that should be applied. Only one dataproc.JobIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

The condition object supports the following:

  • description (pulumi.Input[str])

  • expression (pulumi.Input[str])

  • title (pulumi.Input[str])

etag = None

(Computed) The etag of the jobs’s IAM policy.

project = None

The project in which the job belongs. If it is not provided, this provider will use the provider default.

region = None

The region in which the job belongs. If it is not provided, this provider will use the provider default.

role = None

The role that should be applied. Only one dataproc.JobIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

static get(resource_name, id, opts=None, condition=None, etag=None, job_id=None, member=None, project=None, region=None, role=None)

Get an existing JobIAMMember resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • etag (pulumi.Input[str]) – (Computed) The etag of the jobs’s IAM policy.

  • project (pulumi.Input[str]) – The project in which the job belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the job belongs. If it is not provided, this provider will use the provider default.

  • role (pulumi.Input[str]) – The role that should be applied. Only one dataproc.JobIAMBinding can be used per role. Note that custom roles must be of the format [projects|organizations]/{parent-name}/roles/{role-name}.

The condition object supports the following:

  • description (pulumi.Input[str])

  • expression (pulumi.Input[str])

  • title (pulumi.Input[str])

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

class pulumi_gcp.dataproc.JobIAMPolicy(resource_name, opts=None, job_id=None, policy_data=None, project=None, region=None, __props__=None, __name__=None, __opts__=None)

Three different resources help you manage IAM policies on dataproc jobs. Each of these resources serves a different use case:

  • dataproc.JobIAMPolicy: Authoritative. Sets the IAM policy for the job and replaces any existing policy already attached.

  • dataproc.JobIAMBinding: Authoritative for a given role. Updates the IAM policy to grant a role to a list of members. Other roles within the IAM policy for the job are preserved.

  • dataproc.JobIAMMember: Non-authoritative. Updates the IAM policy to grant a role to a new member. Other members for the role for the job are preserved.

Note: dataproc.JobIAMPolicy cannot be used in conjunction with dataproc.JobIAMBinding and dataproc.JobIAMMember or they will fight over what your policy should be. In addition, be careful not to accidentally unset ownership of the job as dataproc.JobIAMPolicy replaces the entire policy.

Note: dataproc.JobIAMBinding resources can be used in conjunction with dataproc.JobIAMMember resources only if they do not grant privilege to the same role.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • policy_data (pulumi.Input[str]) – The policy data generated by a organizations.getIAMPolicy data source.

  • project (pulumi.Input[str]) – The project in which the job belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the job belongs. If it is not provided, this provider will use the provider default.

etag = None

(Computed) The etag of the jobs’s IAM policy.

policy_data = None

The policy data generated by a organizations.getIAMPolicy data source.

project = None

The project in which the job belongs. If it is not provided, this provider will use the provider default.

region = None

The region in which the job belongs. If it is not provided, this provider will use the provider default.

static get(resource_name, id, opts=None, etag=None, job_id=None, policy_data=None, project=None, region=None)

Get an existing JobIAMPolicy resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • etag (pulumi.Input[str]) – (Computed) The etag of the jobs’s IAM policy.

  • policy_data (pulumi.Input[str]) – The policy data generated by a organizations.getIAMPolicy data source.

  • project (pulumi.Input[str]) – The project in which the job belongs. If it is not provided, this provider will use the provider default.

  • region (pulumi.Input[str]) – The region in which the job belongs. If it is not provided, this provider will use the provider default.

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str