1. Answers
  2. Automating Kubernetes Jobs with External Events

How do I trigger Kubernetes jobs in response to external events?

Automating Kubernetes Jobs with External Events

In this example, you’ll learn how to set up and configure Kubernetes Jobs to be triggered by external events. We’ll utilize Kubernetes resources to create a job that can be triggered whenever a new message appears in an AWS SQS queue. We’ll use aws_sqs_queue to manage the event sources and kubernetes_job to run the desired jobs.

The steps involved are as follows:

  1. Create an SQS queue to serve as the event source.
  2. Create an IAM Role and policy for the Kubernetes service account to access the SQS queue.
  3. Create a Kubernetes job which will be triggered by the messages in the queue.

Let’s dive into the example:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as kubernetes from "@pulumi/kubernetes";

// Define a new SQS Queue
const exampleQueue = new aws.sqs.Queue("example_queue", {name: "example-queue"});
// Define an IAM Role that allows Kubernetes to access the SQS queue
const k8sSqsRole = new aws.iam.Role("k8s_sqs_role", {
    name: "k8s-sqs-role",
    assumeRolePolicy: `  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": "sts:AssumeRole",
        "Effect": "Allow",
        "Principal": {
          "Service": "eks.amazonaws.com"
        }
      }
    ]
  }
`,
});
// Attach necessary policies to the IAM Role
const sqsReadPolicy = new aws.iam.Policy("sqs_read_policy", {
    name: "sqs-read-policy",
    description: "Policy for SQS read access from Kubernetes",
    policy: pulumi.interpolate`  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "sqs:ReceiveMessage",
          "sqs:DeleteMessage",
          "sqs:GetQueueAttributes"
        ],
        "Resource": "${exampleQueue.arn}"
      }
    ]
  }
`,
});
// Attach the policy to the IAM role
const rolePolicyAttachment = new aws.iam.RolePolicyAttachment("role_policy_attachment", {
    role: k8sSqsRole.name,
    policyArn: sqsReadPolicy.arn,
});
// Creating a Kubernetes namespace
const exampleNs = new kubernetes.core.v1.Namespace("example_ns", {metadata: {
    name: "example-namespace",
}});
// Create the Kubernetes Service Account with the attached IAM role
const exampleSa = new kubernetes.core.v1.ServiceAccount("example_sa", {metadata: {
    name: "example-sa",
    namespace: exampleNs.metadata.apply(metadata => metadata.name),
    annotations: {
        "eks.amazonaws.com/role-arn": k8sSqsRole.arn,
    },
}});
// Define a Kubernetes Job
const exampleJob = new kubernetes.batch.v1.Job("example_job", {
    metadata: {
        name: "example-job",
        namespace: exampleNs.metadata.apply(metadata => metadata.name),
    },
    spec: {
        template: {
            metadata: {
                name: "example-job-template",
            },
            spec: {
                serviceAccountName: exampleSa.metadata.apply(metadata => metadata.name),
                containers: [{
                    name: "example-container",
                    image: "amazonlinux:latest",
                    command: [
                        "sh",
                        "-c",
                        "echo 'Job triggered by SQS event'; sleep 30",
                    ],
                }],
                restartPolicy: "Never",
            },
        },
    },
});

Key Points:

  • SQS Queue: Used as an event source.
  • IAM Role and Policy: Grants Kubernetes access to the SQS queue.
  • Kubernetes Job: The job configuration that is executed when triggered.

Summary:

You have now configured a system where Kubernetes jobs can be triggered by external events using SQS. The code provided sets up an SQS queue, configures the necessary IAM roles and policies, and creates a Kubernetes job that runs when a message is detected in the queue. This setup ensures an automated response to events, enhancing the automation capabilities within your Kubernetes cluster.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up