dataflow

This provider is a derived work of the Terraform Provider distributed under MPL 2.0. If you encounter a bug or missing feature, first check the pulumi/pulumi-gcp repo; however, if that doesn’t turn up anything, please consult the source terraform-providers/terraform-provider-google repo.

class pulumi_gcp.dataflow.Job(resource_name, opts=None, ip_configuration=None, labels=None, machine_type=None, max_workers=None, name=None, network=None, on_delete=None, parameters=None, project=None, region=None, service_account_email=None, subnetwork=None, temp_gcs_location=None, template_gcs_path=None, zone=None, __props__=None, __name__=None, __opts__=None)

Creates a job on Dataflow, which is an implementation of Apache Beam running on Google Compute Engine. For more information see the official documentation for Beam and Dataflow.

There are many types of Dataflow jobs. Some Dataflow jobs run constantly, getting new data from (e.g.) a GCS bucket, and outputting data continuously. Some jobs process a set amount of data then terminate. All jobs can fail while running due to programming errors or other issues. In this way, Dataflow jobs are different from most other resources.

The Dataflow resource is considered ‘existing’ while it is in a nonterminal state. If it reaches a terminal state (e.g. ‘FAILED’, ‘COMPLETE’, ‘CANCELLED’), it will be recreated on the next ‘apply’. This is as expected for jobs which run continuously, but may surprise users who use this resource for other kinds of Dataflow jobs.

A Dataflow job which is ‘destroyed’ may be “cancelled” or “drained”. If “cancelled”, the job terminates - any data written remains where it is, but no new data will be processed. If “drained”, no new data will enter the pipeline, but any data currently in the pipeline will finish being processed. The default is “cancelled”, but if a user sets on_delete to "drain" in the configuration, you may experience a long wait for your destroy to complete.

Parameters
  • resource_name (str) – The name of the resource.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • ip_configuration (pulumi.Input[str]) – The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PUBLIC".

  • labels (pulumi.Input[dict]) – User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page.

  • machine_type (pulumi.Input[str]) – The machine type to use for the job.

  • max_workers (pulumi.Input[float]) – The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.

  • name (pulumi.Input[str]) – A unique name for the resource, required by Dataflow.

  • network (pulumi.Input[str]) – The network to which VMs will be assigned. If it is not provided, “default” will be used.

  • on_delete (pulumi.Input[str]) – One of “drain” or “cancel”. Specifies behavior of deletion during a destroy. See above note.

  • parameters (pulumi.Input[dict]) – Key/Value pairs to be passed to the Dataflow job (as used in the template).

  • project (pulumi.Input[str]) – The project in which the resource belongs. If it is not provided, the provider project is used.

  • service_account_email (pulumi.Input[str]) – The Service Account email used to create the job.

  • subnetwork (pulumi.Input[str]) – The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.

  • temp_gcs_location (pulumi.Input[str]) – A writeable location on GCS for the Dataflow job to dump its temporary data.

  • template_gcs_path (pulumi.Input[str]) – The GCS path to the Dataflow job template.

  • zone (pulumi.Input[str]) – The zone in which the created job should run. If it is not provided, the provider zone is used.

ip_configuration = None

The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PUBLIC".

labels = None

User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page.

machine_type = None

The machine type to use for the job.

max_workers = None

The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.

name = None

A unique name for the resource, required by Dataflow.

network = None

The network to which VMs will be assigned. If it is not provided, “default” will be used.

on_delete = None

One of “drain” or “cancel”. Specifies behavior of deletion during a destroy. See above note.

parameters = None

Key/Value pairs to be passed to the Dataflow job (as used in the template).

project = None

The project in which the resource belongs. If it is not provided, the provider project is used.

service_account_email = None

The Service Account email used to create the job.

state = None

The current state of the resource, selected from the JobState enum

subnetwork = None

The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.

temp_gcs_location = None

A writeable location on GCS for the Dataflow job to dump its temporary data.

template_gcs_path = None

The GCS path to the Dataflow job template.

zone = None

The zone in which the created job should run. If it is not provided, the provider zone is used.

static get(resource_name, id, opts=None, ip_configuration=None, labels=None, machine_type=None, max_workers=None, name=None, network=None, on_delete=None, parameters=None, project=None, region=None, service_account_email=None, state=None, subnetwork=None, temp_gcs_location=None, template_gcs_path=None, zone=None)

Get an existing Job resource’s state with the given name, id, and optional extra properties used to qualify the lookup.

Parameters
  • resource_name (str) – The unique name of the resulting resource.

  • id (str) – The unique provider ID of the resource to lookup.

  • opts (pulumi.ResourceOptions) – Options for the resource.

  • ip_configuration (pulumi.Input[str]) – The configuration for VM IPs. Options are "WORKER_IP_PUBLIC" or "WORKER_IP_PUBLIC".

  • labels (pulumi.Input[dict]) –

    User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page.

  • machine_type (pulumi.Input[str]) – The machine type to use for the job.

  • max_workers (pulumi.Input[float]) – The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.

  • name (pulumi.Input[str]) – A unique name for the resource, required by Dataflow.

  • network (pulumi.Input[str]) – The network to which VMs will be assigned. If it is not provided, “default” will be used.

  • on_delete (pulumi.Input[str]) – One of “drain” or “cancel”. Specifies behavior of deletion during a destroy. See above note.

  • parameters (pulumi.Input[dict]) – Key/Value pairs to be passed to the Dataflow job (as used in the template).

  • project (pulumi.Input[str]) – The project in which the resource belongs. If it is not provided, the provider project is used.

  • service_account_email (pulumi.Input[str]) – The Service Account email used to create the job.

  • state (pulumi.Input[str]) –

    The current state of the resource, selected from the JobState enum

  • subnetwork (pulumi.Input[str]) – The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.

  • temp_gcs_location (pulumi.Input[str]) – A writeable location on GCS for the Dataflow job to dump its temporary data.

  • template_gcs_path (pulumi.Input[str]) – The GCS path to the Dataflow job template.

  • zone (pulumi.Input[str]) – The zone in which the created job should run. If it is not provided, the provider zone is used.

translate_output_property(prop)

Provides subclasses of Resource an opportunity to translate names of output properties into a format of their choosing before writing those properties to the resource object.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str

translate_input_property(prop)

Provides subclasses of Resource an opportunity to translate names of input properties into a format of their choosing before sending those properties to the Pulumi engine.

Parameters

prop (str) – A property name.

Returns

A potentially transformed property name.

Return type

str