How do I provision an AWS MSK for Apache Kafka Cluster?

Overview

In this example, we’ll provision an Amazon Managed Streaming for Apache Kafka (MSK) cluster. This involves creating the necessary resources such as VPC, subnets, security groups, and the MSK cluster itself. Using infrastructure as code helps automate and manage these resources efficiently.

Key Resources

VPC - A Virtual Private Cloud to provide network isolation.
Subnets - Network subnets to host the MSK brokers.
Security Group - To manage inbound and outbound traffic rules.
MSK Cluster - The managed Kafka cluster.

The following code block defines and configures these resources.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create a VPC
const myVpc = new aws.ec2.Vpc("my_vpc", {
    cidrBlock: "10.0.0.0/16",
    tags: {
        Name: "my_vpc",
    },
});
// Create Subnets
const subnetA = new aws.ec2.Subnet("subnet_a", {
    vpcId: myVpc.id,
    cidrBlock: "10.0.1.0/24",
    availabilityZone: "us-west-2a",
    tags: {
        Name: "subnet_a",
    },
});
const subnetB = new aws.ec2.Subnet("subnet_b", {
    vpcId: myVpc.id,
    cidrBlock: "10.0.2.0/24",
    availabilityZone: "us-west-2b",
    tags: {
        Name: "subnet_b",
    },
});
// Create a Security Group
const kafkaSg = new aws.ec2.SecurityGroup("kafka_sg", {
    vpcId: myVpc.id,
    ingress: [{
        fromPort: 2181,
        toPort: 2181,
        protocol: "tcp",
        cidrBlocks: ["0.0.0.0/0"],
    }],
    egress: [{
        fromPort: 0,
        toPort: 0,
        protocol: "-1",
        cidrBlocks: ["0.0.0.0/0"],
    }],
    tags: {
        Name: "kafka_sg",
    },
});
// Create the MSK Cluster
const myKafkaCluster = new aws.msk.Cluster("my_kafka_cluster", {
    clusterName: "my_kafka_cluster",
    kafkaVersion: "2.6.0",
    numberOfBrokerNodes: 2,
    brokerNodeGroupInfo: {
        instanceType: "kafka.m5.large",
        clientSubnets: [
            subnetA.id,
            subnetB.id,
        ],
        securityGroups: [kafkaSg.id],
    },
    tags: {
        Name: "my_kafka_cluster",
    },
});
export const vpcId = myVpc.id;
export const subnetIds = [
    subnetA.id,
    subnetB.id,
];
export const securityGroupId = kafkaSg.id;
export const kafkaClusterArn = myKafkaCluster.arn;

Key Points

VPC: Provides network isolation for the Kafka cluster.
Subnets: Distributes the Kafka brokers across availability zones for high availability.
Security Group: Controls access to the Kafka brokers.
MSK Cluster: Managed Kafka service simplifying operations.

Summary

We provisioned an AWS MSK cluster for Apache Kafka, including setting up a VPC, subnets, security groups, and the Kafka cluster itself. Using infrastructure as code ensures reproducibility and efficient management of our cloud resources.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.