Configuring ACLs for Kafka Admin Operations

Introduction

In this guide, we will walk through configuring Access Control Lists (ACLs) for Kafka Admin operations using Pulumi. We will use AWS as our cloud provider, as per the organization’s system prompts. The key services involved include Amazon MSK (Managed Streaming for Apache Kafka) and IAM (Identity and Access Management) for managing permissions.

Step-by-Step Explanation

Step 1: Set Up Your Pulumi Project

1. If you haven’t already, install the Pulumi CLI from Pulumi’s installation guide.
2. Create a new Pulumi project:

   pulumi new aws-typescript
   

3. Follow the prompts to set up your project.

Step 2: Define Your Kafka Cluster

1. In your index.ts file, import the necessary Pulumi and AWS packages:

   import * as pulumi from "@pulumi/pulumi";
   import * as aws from "@pulumi/aws";
   

2. Define your MSK cluster:

   const kafkaCluster = new aws.msk.Cluster("my-cluster", {
       clusterName: "my-cluster",
       kafkaVersion: "2.8.1",
       numberOfBrokerNodes: 3,
       brokerNodeGroupInfo: {
           instanceType: "kafka.m5.large",
           clientSubnets: ["subnet-12345678", "subnet-87654321"],
           securityGroups: ["sg-12345678"],
       },
       encryptionInfo: {
           encryptionAtRestKmsKeyArn: "arn:aws:kms:us-west-2:123456789012:key/abcd-1234-abcd-1234-abcd1234abcd",
       },
   });
   

Step 3: Configure IAM Roles and Policies

1. Create an IAM role for Kafka admin operations:

   const kafkaAdminRole = new aws.iam.Role("kafkaAdminRole", {
       assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
           Service: "kafka.amazonaws.com",
       }),
   });
   

2. Attach the necessary policies to the role:

   const kafkaAdminPolicy = new aws.iam.RolePolicy("kafkaAdminPolicy", {
       role: kafkaAdminRole.id,
       policy: pulumi.output({
           Version: "2012-10-17",
           Statement: [
               {
                   Action: [
                       "kafka:CreateTopic",
                       "kafka:DeleteTopic",
                       "kafka:DescribeTopic",
                       "kafka:ListTopics",
                   ],
                   Effect: "Allow",
                   Resource: "*",
               },
           ],
       }).apply(JSON.stringify),
   });
   

Step 4: Apply ACLs to Kafka Cluster

1. Define the ACLs for the Kafka cluster:

   const kafkaAcl = new aws.msk.ClusterAcl("kafkaAcl", {
       clusterArn: kafkaCluster.arn,
       acl: {
           resourceType: "TOPIC",
           resourceName: "*",
           operation: "ALL",
           permissionType: "ALLOW",
           principal: kafkaAdminRole.arn,
       },
   });
   

Step 5: Deploy Your Changes

1. Run pulumi up to deploy your changes:

   pulumi up
   

2. Confirm the changes and wait for the deployment to complete.

Conclusion

In this guide, we configured ACLs for Kafka Admin operations using Pulumi with AWS. We set up an MSK cluster, created IAM roles and policies, and applied ACLs to manage access control. This setup ensures that only authorized roles can perform admin operations on your Kafka cluster.

Full Code Example

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Define the MSK cluster
const kafkaCluster = new aws.msk.Cluster("my-cluster", {
    clusterName: "my-cluster",
    kafkaVersion: "2.8.1",
    numberOfBrokerNodes: 3,
    brokerNodeGroupInfo: {
        instanceType: "kafka.m5.large",
        clientSubnets: ["subnet-12345678", "subnet-87654321"],
        securityGroups: ["sg-12345678"],
    },
    encryptionInfo: {
        encryptionAtRestKmsKeyArn: "arn:aws:kms:us-west-2:123456789012:key/abcd-1234-abcd-1234-abcd1234abcd",
    },
});

// Create an IAM role for Kafka admin operations
const kafkaAdminRole = new aws.iam.Role("kafkaAdminRole", {
    assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
        Service: "kafka.amazonaws.com",
    }),
});

// Attach the necessary policies to the role
const kafkaAdminPolicy = new aws.iam.RolePolicy("kafkaAdminPolicy", {
    role: kafkaAdminRole.id,
    policy: pulumi.output({
        Version: "2012-10-17",
        Statement: [
            {
                Action: [
                    "kafka:CreateTopic",
                    "kafka:DeleteTopic",
                    "kafka:DescribeTopic",
                    "kafka:ListTopics",
                ],
                Effect: "Allow",
                Resource: "*",
            },
        ],
    }).apply(JSON.stringify),
});

// Define the ACLs for the Kafka cluster
const kafkaAcl = new aws.msk.ClusterPolicy("kafkaAcl", {
    clusterArn: kafkaCluster.arn,
    policy: pulumi.output({
        Version: "2012-10-17",
        Statement: [
            {
                Action: [
                    "kafka:CreateTopic",
                    "kafka:DeleteTopic",
                    "kafka:DescribeTopic",
                    "kafka:ListTopics",
                ],
                Effect: "Allow",
                Resource: "*",
            },
        ],
    }).apply(JSON.stringify),
});