Scheduling Recurring AI Workload Executions Using GCP Workflows

Question

Pulumi · Accepted Answer

To schedule recurring Artificial Intelligence (AI) workload executions on Google Cloud, you can use a combination of services: Google Cloud Workflows for orchestrating your AI pipeline, and Google Cloud Scheduler for scheduling the workflow executions. The combination of these services allows you to define complex workloads with conditional logic, retries, and error handling, and then run them on a recurring schedule.

Here's a breakdown of how you can use these services in Pulumi with Python:

1. **Google Cloud Workflows**: Create a workflow that defines the steps of your AI workload. Workflows allow you to orchestrate API calls to various Google Cloud services such as AI Platform, Compute Engine, or any other service that you need for your AI workload.

2. **Google Cloud Scheduler**: Set up a Cloud Scheduler job to trigger your workflow at a specified interval. Cloud Scheduler acts as a fully managed cron job service that can call HTTP endpoints or post messages to Pub/Sub topics on schedules you define.

In this Pulumi Python program, we will define a Workflow that represents the AI workload and a Cloud Scheduler job that triggers this workflow based on a schedule.

### Prerequisites

Before you use the program, you need to have the following prerequisites:

- A Google Cloud project with billing enabled
- The appropriate permissions to create Workflows and Scheduler jobs
- `pulumi-gcp` package installed in your Pulumi project
- Google Cloud SDK (`gcloud`) installed and configured for the desired project and region

Here is a Pulumi program in Python that sets up a scheduled execution for an AI workload using GCP Workflows and Cloud Scheduler:

```python
import pulumi
import pulumi_gcp as gcp

# Replace these variables with your own configuration
project = 'your-gcp-project-id'
location = 'your-region'  # e.g., 'us-central1'
schedule = '* * * * *'  # This cron schedule runs every minute. Customize as needed.
workflow_name = 'ai-workflow'
scheduler_job_name = 'ai-workflow-scheduler-job'

# Define a GCP Workflow to orchestrate your recurring AI workload execution.
ai_workflow = gcp.workflows.Workflow('aiWorkflow',
    project=project,
    location=location,
    description='Workflow that orchestrates my recurring AI tasks',
    source_contents="""
        # You can define your workflow steps here to run AI tasks. For example:
        - getCurrentTime:
            call: http.get
            args:
                url: https://us-central1-workflowsample.cloudfunctions.net/datetime
            result: currentTime
    """
)

# Create a GCP Cloud Scheduler job that will trigger the workflow on a schedule.
scheduler_job = gcp.cloudscheduler.Job('aiWorkflowSchedulerJob',
    project=project,
    location=location,
    description='Scheduler job for triggering the AI workflow',
    schedule=schedule,
    http_target=gcp.cloudscheduler.JobHttpTargetArgs(
        uri=ai_workflow.service_account.apply(lambda sa: f"https://{location}-{project}.cloudfunctions.net/{sa}"),
        http_method='GET',
        oidc_token=gcp.cloudscheduler.JobHttpTargetOidcTokenArgs(
            service_account_email=ai_workflow.service_account
        ),
    )
)

# Export the URL of the Cloud Scheduler job
pulumi.export('scheduler_job_url', scheduler_job.http_target.uri)
```

### Explanation:

- The `ai_workflow` is a representation of your AI workload, defined as a series of steps in a workflow using GCP Workflows. The actual steps for your use case need to replace the dummy HTTP GET request shown in the `source_contents`.
  
- The `scheduler_job` creates a job in Cloud Scheduler that calls the Workflow endpoint on a recurring schedule defined in the `schedule` variable.
  
- The `http_target.uri` is dynamically built using `.apply()` on the `service_account` attribute of the workflow, constructing the endpoint that the scheduler will trigger.

- The OIDC token (`oidc_token`) is used to authenticate the requests from Scheduler to Workflow.
  
- Finally, `pulumi.export` is used to output the URL of the scheduler job so you can check it after deployment.

To run this program, you'll need to specify your Google Cloud project ID and preferred region. You will also need to adapt the workflow definition based on the specific tasks you want to perform in your AI workload. The schedule is set in a standard cron format; the example here runs every minute, but you should adjust it according to your needs.

Once you have altered the program with your configuration, run it using the `pulumi up` command in your terminal or command prompt, and Pulumi will handle the provisioning of the resources.