How do I scale Kafka Topics replication and partitioning with Strimzi on Kubernetes?

In this guide, we will demonstrate how to scale Kafka topics for replication and partitioning using Strimzi on Kubernetes. Strimzi is a Kubernetes Operator that simplifies running Apache Kafka on Kubernetes. We will create a Kafka cluster and configure topics with specific replication and partition settings.

Key Points:

Create a Kafka cluster using Strimzi.
Define Kafka topics with desired replication and partition settings.
Deploy these configurations on a Kubernetes cluster.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Define the Kafka cluster using Strimzi
const kafkaCluster = new k8s.apiextensions.CustomResource("kafkaCluster", {
    apiVersion: "kafka.strimzi.io/v1beta2",
    kind: "Kafka",
    metadata: {
        name: "my-kafka-cluster",
        namespace: "kafka",
    },
    spec: {
        kafka: {
            version: "2.8.0",
            replicas: 3,
            listeners: [
                {
                    name: "plain",
                    port: 9092,
                    type: "internal",
                    tls: false,
                },
                {
                    name: "tls",
                    port: 9093,
                    type: "internal",
                    tls: true,
                },
            ],
            config: {
                "offsets.topic.replication.factor": 3,
                "transaction.state.log.replication.factor": 3,
                "transaction.state.log.min.isr": 2,
                "log.message.format.version": "2.8",
            },
            storage: [
                {
                    type: "persistent-claim",
                    size: "100Gi",
                    deleteClaim: false,
                },
            ],
        },
        zookeeper: {
            replicas: 3,
            storage: [
                {
                    type: "persistent-claim",
                    size: "100Gi",
                    deleteClaim: false,
                },
            ],
        },
        entityOperator: {
            topicOperator: {},
            userOperator: {},
        },
    },
});

// Define Kafka topics with replication and partition settings
const kafkaTopic = new k8s.apiextensions.CustomResource("kafkaTopic", {
    apiVersion: "kafka.strimzi.io/v1beta2",
    kind: "KafkaTopic",
    metadata: {
        name: "my-topic",
        namespace: "kafka",
        labels: {
            strimziCluster: "my-kafka-cluster",
        },
    },
    spec: {
        partitions: 10,
        replicas: 3,
        config: {
            "retention.ms": 7200000,
            "segment.bytes": 1073741824,
        },
    },
});

export const kafkaClusterName = kafkaCluster.metadata.name;
export const kafkaTopicName = kafkaTopic.metadata.name;

Summary

In this guide, we created a Kafka cluster using Strimzi on Kubernetes. We then defined a Kafka topic with specific replication and partition settings to ensure scalability and reliability. This setup allows for efficient management of Kafka topics in a Kubernetes environment.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.