How do I manage Kafka topics with Strimzi Topic Operator?

In this guide, we will demonstrate how to manage Kafka topics using the Strimzi Topic Operator with Pulumi. The Strimzi Topic Operator manages Kafka topics in a Kubernetes-native way, allowing you to declare Kafka topics as Kubernetes resources.

We will define a Kafka topic resource using Pulumi and deploy it to a Kubernetes cluster where Strimzi is installed. This will automate the creation and management of Kafka topics.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Define the namespace where Strimzi is installed
const strimziNamespace = new k8s.core.v1.Namespace("strimzi-namespace", {
    metadata: {
        name: "strimzi",
    },
});

// Define a KafkaTopic custom resource
const kafkaTopic = new k8s.apiextensions.CustomResource("my-kafka-topic", {
    apiVersion: "kafka.strimzi.io/v1beta2",
    kind: "KafkaTopic",
    metadata: {
        name: "my-topic",
        namespace: strimziNamespace.metadata.name,
    },
    spec: {
        partitions: 3,
        replicas: 2,
        config: {
            "retention.ms": "7200000",
            "segment.bytes": "1073741824",
        },
    },
});

// Export the Kafka topic name
export const topicName = kafkaTopic.metadata.name;

Key Points:

• We first define a Kubernetes namespace where Strimzi is installed.
• We then define a KafkaTopic custom resource using Pulumi’s k8s.apiextensions.CustomResource class.
• The KafkaTopic resource includes specifications such as the number of partitions, replicas, and topic configuration settings.
• Finally, we export the topic name for reference.

Summary:

In this guide, we demonstrated how to manage Kafka topics using the Strimzi Topic Operator with Pulumi. By defining Kafka topics as Kubernetes resources, we can leverage Kubernetes-native tools and workflows to manage Kafka topics efficiently.