Using kubernetes kafka.strimzi.io with service

Question

Pulumi · Accepted Answer

To deploy a Kafka cluster using Strimzi on Kubernetes, you will make use of Custom Resource Definitions (CRDs) that Strimzi provides. Strimzi simplifies the process of running Apache Kafka in a Kubernetes cluster.

Firstly, ensure that you have Strimzi's Custom Resource Definitions applied to your Kubernetes cluster. You can typically do this by applying the installation files from Strimzi's release page directly to your Kubernetes cluster.

Once that's in place, you create a `Kafka` resource which Strimzi will use to set up the Kafka cluster. In the example below, we'll define a Kafka cluster with a single Kafka broker and a single Zookeeper node since Strimzi requires Kafka to work in tandem with Zookeeper for management purposes.

The `Service` resource is provided implicitly by Strimzi within the Kafka broker, but it's important to expose Kafka outside of the cluster if clients need to connect to it. To achieve this, we can use a `Kafka` resource specification where we set the `external` property of the `listeners` to the required type, such as a `NodePort` or `LoadBalancer`.

Let's proceed with the following TypeScript program, which illustrates how you might use Pulumi to deploy a Kafka cluster using Strimzi on Kubernetes:

```typescript
import * as k8s from "@pulumi/kubernetes";

const namespace = new k8s.core.v1.Namespace("kafka-namespace", {
    metadata: {
        name: "kafka",
    },
});

const kafkaCluster = new k8s.apiextensions.CustomResource("kafka-cluster", {
    apiVersion: "kafka.strimzi.io/v1beta2",
    kind: "Kafka",
    metadata: {
        namespace: namespace.metadata.name,
        name: "my-cluster",
    },
    spec: {
        kafka: {
            version: "3.0.0",
            replicas: 1,
            listeners: {
                plain: {},
                // Expose Kafka outside of the cluster using a LoadBalancer
                external: {
                    type: "LoadBalancer",
                },
            },
            config: {
                offsetsTopicReplicationFactor: 1,
                transactionStateLogReplicationFactor: 1,
                transactionStateLogMinIsr: 1,
            },
            storage: {
                type: "jbod",
                volumes: [{ id: 0, type: "persistent-claim", size: "100Gi", deleteClaim: false }],
            },
        },
        zookeeper: {
            replicas: 1,
            storage: {
                type: "persistent-claim",
                size: "100Gi",
                deleteClaim: false,
            },
        },
        entityOperator: {
            topicOperator: {},
            userOperator: {},
        },
    },
}, { dependsOn: [namespace] });

export const kafkaClusterName = kafkaCluster.metadata.name;
```

Here's a breakdown of what the above code does:

- We initiate a new Kubernetes namespace to encapsulate all our Kafka related resources. This helps in maintaining a cleaner cluster and provides easier management of related resources.
- We create a new `CustomResource` of kind `Kafka`, this utilizes the CRDs provided by Strimzi. The `apiVersion` tells the Kubernetes API server which version of the CRD to use.
- The `spec` field of the `CustomResource` defines the properties of the Kafka cluster such as version, the number of replicas, listener configuration, storage configuration, and the entity operator which includes the topic and user operators.

The `listeners` section defines how Kafka clients will communicate with the Kafka brokers. The `plain` listener is for internal cluster communication, and the `external` listener with a type of `LoadBalancer` is meant for traffic outside the Kubernetes cluster. Note that the exact configuration could be modified to suit the security requirements and the specifics of your environment.

It's important to note that adjustments may need to be made based on your actual cluster's capacity, and the storage size would typically depend on your needs and budget.

This Pulumi program assumes you have the necessary access rights to create resources in your Kubernetes cluster, and that Strimzi's CRDs are already installed.

Remember that Pulumi tracks the state of your resources, so running this script will not only create them but allow Pulumi to manage their lifecycle as needed. Please consult Strimzi's documentation to ensure you comply with the necessary prerequisites before deploying a Kafka cluster in your Kubernetes environment.