How do I orchestrate data using AWS Managed Streaming for Kafka (MSK)?
In this guide, we will set up data orchestration using AWS Managed Streaming for Kafka (MSK) with Pulumi. We will create an MSK cluster along with the necessary configurations.
Key Points
- Create an MSK cluster.
- Configure the MSK cluster with necessary settings.
- Set up security and networking configurations.
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
// Define the VPC
const vpc = new aws.ec2.Vpc("msk-vpc", {
cidrBlock: "10.0.0.0/16",
enableDnsHostnames: true,
enableDnsSupport: true,
});
// Define subnets
const subnet1 = new aws.ec2.Subnet("msk-subnet-1", {
vpcId: vpc.id,
cidrBlock: "10.0.1.0/24",
availabilityZone: "us-west-2a",
});
const subnet2 = new aws.ec2.Subnet("msk-subnet-2", {
vpcId: vpc.id,
cidrBlock: "10.0.2.0/24",
availabilityZone: "us-west-2b",
});
// Define security group
const securityGroup = new aws.ec2.SecurityGroup("msk-sg", {
vpcId: vpc.id,
description: "Security group for MSK",
ingress: [
{ protocol: "tcp", fromPort: 9092, toPort: 9092, cidrBlocks: ["0.0.0.0/0"] },
],
egress: [
{ protocol: "tcp", fromPort: 0, toPort: 0, cidrBlocks: ["0.0.0.0/0"] },
],
});
// Create the MSK cluster
const mskCluster = new aws.msk.Cluster("msk-cluster", {
clusterName: "example-msk-cluster",
kafkaVersion: "2.8.0",
numberOfBrokerNodes: 2,
brokerNodeGroupInfo: {
instanceType: "kafka.m5.large",
clientSubnets: [subnet1.id, subnet2.id],
securityGroups: [securityGroup.id],
},
encryptionInfo: {
encryptionAtRestKmsKeyArn: aws.kms.getKey({ keyId: "alias/aws/kafka" }).then(k => k.arn),
},
loggingInfo: {
brokerLogs: {
cloudwatchLogs: {
enabled: true,
logGroup: new aws.cloudwatch.LogGroup("msk-log-group").name,
},
},
},
});
// Export the Cluster ARN
export const clusterArn = mskCluster.arn;
Summary
In this guide, we set up an AWS Managed Streaming for Kafka (MSK) cluster using Pulumi. We created a VPC, subnets, and a security group, and then configured the MSK cluster with the necessary settings, including encryption and logging. This setup enables efficient data orchestration and streaming using MSK.
Deploy this code
Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.
Sign upNew to Pulumi?
Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.
Sign upThank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.