Module dataflow

@pulumi/gcp > dataflow

This provider is a derived work of the Terraform Provider distributed under MPL 2.0. If you encounter a bug or missing feature, first check the pulumi/pulumi-gcp repo; however, if that doesn’t turn up anything, please consult the source terraform-providers/terraform-provider-google repo.

class Job

extends CustomResource

Creates a job on Dataflow, which is an implementation of Apache Beam running on Google Compute Engine. For more information see the official documentation for Beam and Dataflow.

Example Usage

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";

const bigDataJob = new gcp.dataflow.Job("bigDataJob", {
    parameters: {
        baz: "qux",
        foo: "bar",
    },
    tempGcsLocation: "gs://my-bucket/tmp_dir",
    templateGcsPath: "gs://my-bucket/templates/template_file",
});

Note on “destroy” / “apply”

There are many types of Dataflow jobs. Some Dataflow jobs run constantly, getting new data from (e.g.) a GCS bucket, and outputting data continuously. Some jobs process a set amount of data then terminate. All jobs can fail while running due to programming errors or other issues. In this way, Dataflow jobs are different from most other resources.

The Dataflow resource is considered ‘existing’ while it is in a nonterminal state. If it reaches a terminal state (e.g. ‘FAILED’, ‘COMPLETE’, ‘CANCELLED’), it will be recreated on the next ‘apply’. This is as expected for jobs which run continously, but may surprise users who use this resource for other kinds of Dataflow jobs.

A Dataflow job which is ‘destroyed’ may be “cancelled” or “drained”. If “cancelled”, the job terminates - any data written remains where it is, but no new data will be processed. If “drained”, no new data will enter the pipeline, but any data currently in the pipeline will finish being processed. The default is “cancelled”, but if a user sets onDelete to "drain" in the configuration, you may experience a long wait for your destroy to complete.

This content is derived from https://github.com/terraform-providers/terraform-provider-google/blob/master/website/docs/r/dataflow_job.html.markdown.

constructor

new Job(name: string, args: JobArgs, opts?: pulumi.CustomResourceOptions)

Create a Job resource with the given unique name, arguments, and options.

  • name The unique name of the resource.
  • args The arguments to use to populate this resource's properties.
  • opts A bag of options that control this resource's behavior.

method get

public static get(name: string, id: pulumi.Input<pulumi.ID>, state?: JobState, opts?: pulumi.CustomResourceOptions): Job

Get an existing Job resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.

method getProvider

getProvider(moduleMember: string): ProviderResource | undefined

method isInstance

public static isInstance(obj: any): boolean

Returns true if the given object is an instance of Job. This is designed to work even when multiple copies of the Pulumi SDK have been loaded into the same process.

property id

id: Output<ID>;

id is the provider-assigned unique ID for this managed resource. It is set during deployments and may be missing (undefined) during planning phases.

property labels

public labels: pulumi.Output<{[key: string]: any} | undefined>;

User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page.

property machineType

public machineType: pulumi.Output<string | undefined>;

The machine type to use for the job.

property maxWorkers

public maxWorkers: pulumi.Output<number | undefined>;

The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.

property name

public name: pulumi.Output<string>;

A unique name for the resource, required by Dataflow.

property network

public network: pulumi.Output<string | undefined>;

The network to which VMs will be assigned. If it is not provided, “default” will be used.

property onDelete

public onDelete: pulumi.Output<string | undefined>;

One of “drain” or “cancel”. Specifies behavior of deletion during a destroy. See above note.

property parameters

public parameters: pulumi.Output<{[key: string]: any} | undefined>;

Key/Value pairs to be passed to the Dataflow job (as used in the template).

property project

public project: pulumi.Output<string>;

The project in which the resource belongs. If it is not provided, the provider project is used.

property region

public region: pulumi.Output<string | undefined>;

property serviceAccountEmail

public serviceAccountEmail: pulumi.Output<string | undefined>;

The Service Account email used to create the job.

property state

public state: pulumi.Output<string>;

The current state of the resource, selected from the JobState enum

property subnetwork

public subnetwork: pulumi.Output<string | undefined>;

The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.

property tempGcsLocation

public tempGcsLocation: pulumi.Output<string>;

A writeable location on GCS for the Dataflow job to dump its temporary data.

property templateGcsPath

public templateGcsPath: pulumi.Output<string>;

The GCS path to the Dataflow job template.

property urn

urn: Output<URN>;

urn is the stable logical URN used to distinctly address a resource, both before and after deployments.

property zone

public zone: pulumi.Output<string | undefined>;

The zone in which the created job should run. If it is not provided, the provider zone is used.

interface JobArgs

The set of arguments for constructing a Job resource.

property labels

labels?: pulumi.Input<{[key: string]: any}>;

User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page.

property machineType

machineType?: pulumi.Input<string>;

The machine type to use for the job.

property maxWorkers

maxWorkers?: pulumi.Input<number>;

The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.

property name

name?: pulumi.Input<string>;

A unique name for the resource, required by Dataflow.

property network

network?: pulumi.Input<string>;

The network to which VMs will be assigned. If it is not provided, “default” will be used.

property onDelete

onDelete?: pulumi.Input<string>;

One of “drain” or “cancel”. Specifies behavior of deletion during a destroy. See above note.

property parameters

parameters?: pulumi.Input<{[key: string]: any}>;

Key/Value pairs to be passed to the Dataflow job (as used in the template).

property project

project?: pulumi.Input<string>;

The project in which the resource belongs. If it is not provided, the provider project is used.

property region

region?: pulumi.Input<string>;

property serviceAccountEmail

serviceAccountEmail?: pulumi.Input<string>;

The Service Account email used to create the job.

property subnetwork

subnetwork?: pulumi.Input<string>;

The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.

property tempGcsLocation

tempGcsLocation: pulumi.Input<string>;

A writeable location on GCS for the Dataflow job to dump its temporary data.

property templateGcsPath

templateGcsPath: pulumi.Input<string>;

The GCS path to the Dataflow job template.

property zone

zone?: pulumi.Input<string>;

The zone in which the created job should run. If it is not provided, the provider zone is used.

interface JobState

Input properties used for looking up and filtering Job resources.

property labels

labels?: pulumi.Input<{[key: string]: any}>;

User labels to be specified for the job. Keys and values should follow the restrictions specified in the labeling restrictions page.

property machineType

machineType?: pulumi.Input<string>;

The machine type to use for the job.

property maxWorkers

maxWorkers?: pulumi.Input<number>;

The number of workers permitted to work on the job. More workers may improve processing speed at additional cost.

property name

name?: pulumi.Input<string>;

A unique name for the resource, required by Dataflow.

property network

network?: pulumi.Input<string>;

The network to which VMs will be assigned. If it is not provided, “default” will be used.

property onDelete

onDelete?: pulumi.Input<string>;

One of “drain” or “cancel”. Specifies behavior of deletion during a destroy. See above note.

property parameters

parameters?: pulumi.Input<{[key: string]: any}>;

Key/Value pairs to be passed to the Dataflow job (as used in the template).

property project

project?: pulumi.Input<string>;

The project in which the resource belongs. If it is not provided, the provider project is used.

property region

region?: pulumi.Input<string>;

property serviceAccountEmail

serviceAccountEmail?: pulumi.Input<string>;

The Service Account email used to create the job.

property state

state?: pulumi.Input<string>;

The current state of the resource, selected from the JobState enum

property subnetwork

subnetwork?: pulumi.Input<string>;

The subnetwork to which VMs will be assigned. Should be of the form “regions/REGION/subnetworks/SUBNETWORK”.

property tempGcsLocation

tempGcsLocation?: pulumi.Input<string>;

A writeable location on GCS for the Dataflow job to dump its temporary data.

property templateGcsPath

templateGcsPath?: pulumi.Input<string>;

The GCS path to the Dataflow job template.

property zone

zone?: pulumi.Input<string>;

The zone in which the created job should run. If it is not provided, the provider zone is used.