How do I deploy Kafka on AWS ECS?

To use Kafka with Amazon ECS, you need to set up an ECS cluster, define a task definition for Kafka, and then run the Kafka service on ECS. Below, I’ll walk you through the steps using Pulumi with TypeScript.

Steps to Set Up Kafka on Amazon ECS

Set Up an ECS Cluster: Create an ECS cluster where your Kafka containers will run.
Define a Task Definition: Define the ECS task that specifies the Docker container for Kafka.
Run the Kafka Service: Create an ECS service to run the Kafka task.

Detailed Explanation

ECS Cluster: This is the logical grouping of ECS tasks or services.
Task Definition: This is a blueprint for your application. It contains one or more container definitions.
ECS Service: This ensures that the specified number of task instances are running and healthy.

Pulumi Program

Below is a TypeScript program using Pulumi to set up Kafka with Amazon ECS:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create an ECS cluster
const cluster = new aws.ecs.Cluster("kafka-cluster");

// Create an IAM role for the ECS task
const taskRole = new aws.iam.Role("taskRole", {
    assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [{
            Action: "sts:AssumeRole",
            Principal: {
                Service: "ecs-tasks.amazonaws.com",
            },
            Effect: "Allow",
            Sid: "",
        }],
    }),
});

// Attach the necessary policies to the task role
const taskRolePolicyAttachment = new aws.iam.RolePolicyAttachment("taskRolePolicyAttachment", {
    role: taskRole.name,
    policyArn: "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
});

// Define the ECS task definition
const kafkaTaskDefinition = new aws.ecs.TaskDefinition("kafkaTask", {
    family: "kafka-task",
    cpu: "512",
    memory: "1024",
    networkMode: "awsvpc",
    requiresCompatibilities: ["FARGATE"],
    executionRoleArn: taskRole.arn,
    containerDefinitions: JSON.stringify([{
        name: "kafka",
        image: "bitnami/kafka:latest",
        essential: true,
        portMappings: [{
            containerPort: 9092,
            hostPort: 9092,
        }],
        environment: [
            { name: "KAFKA_BROKER_ID", value: "1" },
            { name: "KAFKA_ZOOKEEPER_CONNECT", value: "zookeeper:2181" },
        ],
    }]),
});

// Create an ECS service to run the Kafka task
const kafkaService = new aws.ecs.Service("kafkaService", {
    cluster: cluster.arn,
    taskDefinition: kafkaTaskDefinition.arn,
    desiredCount: 1,
    launchType: "FARGATE",
    networkConfiguration: {
        subnets: ["subnet-12345678"],  // Replace with your subnet ID
        securityGroups: ["sg-12345678"],  // Replace with your security group ID
    },
});

// Export the cluster name and Kafka service name
export const clusterName = cluster.name;
export const kafkaServiceName = kafkaService.name;

Explanation of the Code

Cluster: We create an ECS cluster named kafka-cluster.
IAM Role: We create an IAM role for the ECS task with necessary policies attached.
Task Definition: We define a task with a single container running Kafka. The container uses the Bitnami Kafka image and listens on port 9092.
Service: We create an ECS service to run the Kafka task on Fargate, specifying the desired count of 1, and configuring networking with subnets and security groups.

Running the Program

Install Pulumi CLI: If you haven’t already, install the Pulumi CLI from here.
Configure AWS CLI: Ensure your AWS CLI is configured with the necessary permissions.

Run Pulumi Commands:

pulumi login
pulumi stack init dev
pulumi up

This will create the ECS cluster, task definition, and service, and deploy Kafka on ECS. You can then use the exported clusterName and kafkaServiceName to interact with your Kafka service.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.