Serverless Machine Learning Workflow Orchestration with GCP Cloud Build

Question

Pulumi · Accepted Answer

To orchestrate a serverless machine learning workflow on Google Cloud Platform (GCP) with Cloud Build, we'll deploy a workflow using Google Cloud Workflows to manage the sequence of processes, a Google Cloud Build Trigger to react to changes such as code commits, and potentially use Google Cloud ML Engine for training machine learning models.

We will use the following Pulumi resources:
- `gcp.workflows.Workflow`: To create the serverless workflow that will orchestrate our machine learning tasks.
- `gcp.cloudbuild.Trigger`: To set up a Cloud Build trigger that initiates our workflow whenever certain conditions are met, such as pushing a new commit to a specified branch of a repository.
- `gcp.ml.EngineModel`: If we need to manage a machine learning model on GCP, this resource will allow us to deploy the model.

Here's an outline of what we'll do with the Pulumi program:
1. Define a `Workflow` resource to create a workflow that specifies the steps of the machine learning job.
2. Create a `Trigger` resource to watch for code commits and start the workflow.
3. Optionally, use the `EngineModel` resource to manage the machine learning model if required.

Let's proceed with a program written in Python.

```python
import pulumi
import pulumi_gcp as gcp

# Define the serverless workflow using Google Cloud Workflows.
workflow = gcp.workflows.Workflow("ml-workflow",
    region="us-central1",
    description="Orchestrates serverless machine learning workflow",
    source_contents="""
        - getCurrentTime:
            call: http.get
            args:
                url: https://us-central1-workflowsample.cloudfunctions.net/datetime
            result: currentTime
        - trainModel:
            call: http.post
            args:
                url: https://ml.googleapis.com/v1/projects/myProject/models
                body:
                    name: "MyModel"
                    description: "Train a new model"
            result: trainingOutput
        - checkTrainingStatus:
            call: http.get
            args:
                url: ${"https://ml.googleapis.com/v1/projects/myProject/models/" + sys.get_env("MODEL_ID")}
            result: modelStatus
    """)

# Set up a Cloud Build trigger that will start the workflow whenever a push to the main branch is detected.
trigger = gcp.cloudbuild.Trigger("ml-trigger",
    description="Trigger for machine learning workflow",
    trigger_template=gcp.cloudbuild.TriggerTriggerTemplateArgs(
        branch_name="main",
        project="my-gcp-project",
        repo_name="my-repo",
    ),
    filename="cloudbuild.yaml"
)

# Optionally, you can manage your machine learning model with Cloud ML Engine.
# This is a placeholder for creating or managing a model, and additional steps may be required.
model = gcp.ml.EngineModel("ml-model",
    description="My machine learning model",
    # other model options...
)

# Export the URL of the workflow and the name of the Cloud Build trigger.
pulumi.export("workflow_url", workflow.self_link)
pulumi.export("trigger_name", trigger.name)
```

In this program, we create a workflow that consists of three steps:
- `getCurrentTime`: This step calls a cloud function to get the current time.
- `trainModel`: This step sends a POST request to start training a machine learning model.
- `checkTrainingStatus`: This step checks the status of the training job by sending a GET request.

The Cloud Build trigger is set up to respond to pushes to the main branch of a specified repository. The `filename` parameter refers to a configuration file (`cloudbuild.yaml`) that would be present in the root of the repository and would define build instructions.

The `EngineModel` resource is used to create and manage the machine learning model on GCP. We have only included the description for this resource, and additional configuration would be necessary depending on the requirements of the machine learning task.

When you run this program with Pulumi, it will deploy these resources on GCP, setting up an automated, serverless workflow for machine learning tasks.