Scheduled AI Pipeline Execution with CloudWatch EventRule

Question

Pulumi · Accepted Answer

To schedule an AI pipeline execution in AWS, you'll use an AWS CloudWatch `EventRule` to trigger your pipeline on a scheduled basis. Imagine you have an AWS SageMaker Pipeline for your AI workload that you want to execute periodically. You can achieve this by defining a CloudWatch Event Rule that runs on a cron-like schedule and targets the SageMaker Pipeline.

Below is a Pulumi program in Python that creates the necessary resources to schedule an AI pipeline execution using AWS SageMaker and CloudWatch. We'll define the following:

1. **SageMaker Pipeline**: This represents your AI pipeline. For the sake of this example, we'll assume that you already have an existing SageMaker Pipeline and we will use its ARN.

2. **CloudWatch Event Rule**: This is the rule that triggers on the schedule you define. In this example, we'll set it to trigger daily.

3. **CloudWatch Event Target**: This connects the Event Rule to the SageMaker Pipeline, effectively setting the pipeline to be the target of scheduled events.

4. **IAM Role**: IAM roles are a secure way to grant permissions that your Event Rule will assume when executing the SageMaker Pipeline.

Let's write a Pulumi program for this:

```python
import pulumi
import pulumi_aws as aws

# Configuration for the schedule in standard cron format - this example runs at 6 AM UTC every day.
# You can adjust the schedule as needed.
schedule_expression = "cron(0 6 * * ? *)"

# IAM Role that the Event Rule will assume to trigger the SageMaker Pipeline execution.
cloudwatch_event_role = aws.iam.Role("CloudWatchEventRole",
    assume_role_policy={
        "Version": "2012-10-17",
        "Statement": [{
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Principal": {
                "Service": "events.amazonaws.com"
            },
        }],
    }
)

# Inline policy to grant the necessary permissions to execute the SageMaker Pipeline.
policy = aws.iam.Policy("CloudWatchEventsSageMakerPolicy",
    policy={
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": ["sagemaker:StartPipelineExecution"],
            "Resource": "arn:aws:sagemaker:*:*:pipeline/*", # Replace with the actual ARN of your SageMaker Pipeline
        }],
    }
)

# Attach the policy to the role.
role_policy_attachment = aws.iam.RolePolicyAttachment("CloudWatchEventsSageMakerPolicyAttachment",
    role=cloudwatch_event_role.name,
    policy_arn=policy.arn
)

# Define the CloudWatch Event Rule with the schedule.
event_rule = aws.cloudwatch.EventRule("ScheduledAIPipelineExecutionEventRule",
    schedule_expression=schedule_expression,
    description="Triggers the AI Pipeline execution on a schedule",
)

# Define the CloudWatch Event Target to invoke the SageMaker Pipeline.
event_target = aws.cloudwatch.EventTarget("ScheduledAIPipelineExecutionEventTarget",
    rule=event_rule.name,
    arn="arn:aws:sagemaker:*:*:pipeline/*", # Replace with the actual ARN of your SageMaker Pipeline
    role_arn=cloudwatch_event_role.arn,
)

# Export the Event Rule ARN so that it's easy to retrieve and reference if needed.
pulumi.export("event_rule_arn", event_rule.arn)
```

Here's what each part of the Pulumi program above is doing:

- **IAM Role (`cloudwatch_event_role`)**: The first resource created is an IAM role that the CloudWatch Event Rule will assume when it triggers the SageMaker Pipeline. The role's trust relationship policy document allows the `events.amazonaws.com` service to assume the role.

- **IAM Policy (`policy`)**: We then define an IAM policy that grants permission to start the SageMaker Pipeline execution. The policy is attached to the role from the previous step.

- **CloudWatch Event Rule (`event_rule`)**: The event rule specifies the schedule you want to trigger your pipeline using a cron expression. This example uses `schedule_expression` to run the event daily at 6 AM UTC.

- **CloudWatch Event Target (`event_target`)**: This target is associated with the CloudWatch Event Rule and specifies the SageMaker Pipeline ARN to invoke when the rule triggers. It also uses the role we created earlier to run with the proper permissions.

- **Export**: Finally, at the end of the program, we export the ARN of the created CloudWatch Event Rule so you can reference it easily outside of Pulumi.

After running this Pulumi program, you'll have established a CloudWatch Event Rule that triggers a SageMaker Pipeline execution according to the defined schedule. This will allow you to automate the recurring execution of your AI pipelines on AWS.

Remember to replace `"arn:aws:sagemaker:*:*:pipeline/*"` with the actual ARN of your SageMaker Pipeline. The actual ARN is typically provided when you create a SageMaker Pipeline or can be found in the AWS Management Console under Amazon SageMaker services.