1. Answers
  2. How do I autoscale ECS with SQS messages?

How do I autoscale ECS with SQS messages?

To autoscale an ECS service based on the number of messages in an SQS queue, you need to set up an ECS cluster, an ECS service, an SQS queue, and an Application Auto Scaling policy that adjusts the ECS service’s desired count based on the SQS queue length.

Here’s a step-by-step guide to achieving this with Pulumi in TypeScript:

  1. Create an ECS Cluster: This will host your ECS services.
  2. Create an SQS Queue: This queue will be used to trigger the scaling actions based on the number of messages.
  3. Create an ECS Service: This service will run your tasks and will be scaled based on the SQS queue length.
  4. Set up Application Auto Scaling: This will adjust the desired count of the ECS service tasks based on the SQS queue length.

Let’s start with the detailed code:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create an ECS Cluster
const cluster = new aws.ecs.Cluster("ecs-cluster");

// Create an SQS Queue
const queue = new aws.sqs.Queue("sqs-queue", {
    visibilityTimeoutSeconds: 300,
});

// Create a Task Definition for the ECS Service
const taskDefinition = new aws.ecs.TaskDefinition("task-definition", {
    family: "service",
    containerDefinitions: JSON.stringify([{
        name: "my-app",
        image: "nginx", // Replace with your application image
        essential: true,
        memory: 512,
        cpu: 256,
        portMappings: [{
            containerPort: 80,
            hostPort: 80,
        }],
    }]),
});

// Create an ECS Service
const service = new aws.ecs.Service("ecs-service", {
    cluster: cluster.arn,
    taskDefinition: taskDefinition.arn,
    desiredCount: 1,
    launchType: "EC2", // Use "FARGATE" if you're using Fargate
    loadBalancers: [],
    deploymentMaximumPercent: 200,
    deploymentMinimumHealthyPercent: 50,
});

// Create an Application Auto Scaling Target
const scalingTarget = new aws.appautoscaling.Target("scaling-target", {
    maxCapacity: 10,
    minCapacity: 1,
    resourceId: pulumi.interpolate`service/${cluster.name}/${service.name}`,
    scalableDimension: "ecs:service:DesiredCount",
    serviceNamespace: "ecs",
});

// Create an Application Auto Scaling Policy
const scalingPolicy = new aws.appautoscaling.Policy("scaling-policy", {
    policyType: "TargetTrackingScaling",
    resourceId: scalingTarget.resourceId,
    scalableDimension: scalingTarget.scalableDimension,
    serviceNamespace: scalingTarget.serviceNamespace,
    targetTrackingScalingPolicyConfiguration: {
        targetValue: 5.0, // Adjust based on your desired queue length threshold
        predefinedMetricSpecification: {
            predefinedMetricType: "SQSQueueMessagesVisible",
            resourceLabel: queue.id,
        },
        scaleInCooldown: 60,
        scaleOutCooldown: 60,
    },
});

// Export the ECS Cluster and SQS Queue URLs
export const clusterName = cluster.name;
export const queueUrl = queue.url;

Explanation

  • ECS Cluster: This is the cluster that will run your ECS services.
  • SQS Queue: This queue will be monitored for the number of messages.
  • ECS Task Definition: Defines the container specifications for your ECS service.
  • ECS Service: Manages running instances of the task definition.
  • Application Auto Scaling Target: Defines the resource to be scaled, in this case, the ECS service.
  • Application Auto Scaling Policy: Defines the scaling policy based on the SQS queue length.

How it works

  1. ECS Cluster: We create an ECS cluster to run our services.
  2. SQS Queue: An SQS queue is created to hold messages that will trigger scaling actions.
  3. ECS Task Definition: Defines the container configuration for the ECS service.
  4. ECS Service: This service runs the tasks defined in the task definition.
  5. Application Auto Scaling Target: This configures the ECS service to be scalable.
  6. Application Auto Scaling Policy: This policy adjusts the number of running tasks based on the number of messages in the SQS queue.

This setup ensures that your ECS service scales up or down based on the load indicated by the number of messages in the SQS queue. Adjust the parameters such as targetValue, maxCapacity, and minCapacity based on your specific requirements.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up