How do I use Kubernetes autoscaling with Kafka Strimzi Operator?

In this guide, we will demonstrate how to set up Kubernetes autoscaling with Kafka Strimzi Operator using Pulumi. We will create a Kafka cluster using the Strimzi Operator and configure a Horizontal Pod Autoscaler (HPA) to manage the scaling of the Kafka brokers based on CPU utilization.

Key Points:

Install the Strimzi Kafka Operator.
Deploy a Kafka cluster using Strimzi.
Set up a Horizontal Pod Autoscaler for the Kafka brokers.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Namespace for Strimzi and Kafka resources
const namespace = new k8s.core.v1.Namespace("kafka-namespace", {
    metadata: { name: "kafka" },
});

// Install Strimzi Operator
const strimziOperator = new k8s.helm.v3.Chart("strimzi-kafka-operator", {
    chart: "strimzi-kafka-operator",
    version: "0.23.0",
    fetchOpts: {
        repo: "https://strimzi.io/charts/",
    },
    namespace: namespace.metadata.name,
});

// Kafka Cluster CRD
const kafkaCluster = new k8s.apiextensions.CustomResource("kafka-cluster", {
    apiVersion: "kafka.strimzi.io/v1beta2",
    kind: "Kafka",
    metadata: {
        namespace: namespace.metadata.name,
        name: "my-kafka-cluster",
    },
    spec: {
        kafka: {
            version: "2.8.0",
            replicas: 3,
            listeners: [
                {
                    name: "plain",
                    port: 9092,
                    type: "internal",
                    tls: false,
                },
            ],
            config: {
                "offsets.topic.replication.factor": 3,
                "transaction.state.log.replication.factor": 3,
                "transaction.state.log.min.isr": 2,
                "log.message.format.version": "2.8",
            },
            storage: {
                type: "ephemeral",
            },
        },
        zookeeper: {
            replicas: 3,
            storage: {
                type: "ephemeral",
            },
        },
        entityOperator: {
            topicOperator: {},
            userOperator: {},
        },
    },
});

// Horizontal Pod Autoscaler for Kafka brokers
const kafkaHPA = new k8s.autoscaling.v2beta2.HorizontalPodAutoscaler("kafka-hpa", {
    metadata: {
        namespace: namespace.metadata.name,
    },
    spec: {
        scaleTargetRef: {
            apiVersion: "apps/v1",
            kind: "StatefulSet",
            name: "my-kafka-cluster-kafka",
        },
        minReplicas: 3,
        maxReplicas: 6,
        metrics: [
            {
                type: "Resource",
                resource: {
                    name: "cpu",
                    target: {
                        type: "Utilization",
                        averageUtilization: 80,
                    },
                },
            },
        ],
    },
});

export const kafkaClusterName = kafkaCluster.metadata.name;
export const kafkaHPAName = kafkaHPA.metadata.name;

Summary

In this guide, we installed the Strimzi Kafka Operator, deployed a Kafka cluster, and configured a Horizontal Pod Autoscaler to manage the scaling of Kafka brokers based on CPU utilization. This setup ensures that the Kafka cluster can automatically scale its resources to handle varying workloads efficiently.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.