Using kubernetes kafka.strimzi.io with traefik

Question

Pulumi · Accepted Answer

To set up a Kafka cluster on Kubernetes using `strimzi.io`, and to expose it using Traefik as an Ingress controller, you will go through several steps. Below is an explanation of these steps, followed by the Pulumi TypeScript program that sets up the infrastructure accordingly.

1. **Kubernetes Cluster**: First, you need an existing Kubernetes cluster where you can deploy Kafka and Traefik. For this example, I'm assuming that you already have a cluster setup and the Pulumi program will not cover the cluster creation.

2. **Kafka with Strimzi**: [Strimzi](https://strimzi.io/) provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. Strimzi uses Custom Resource Definitions (CRDs) to manage Kafka clusters and topics within Kubernetes.

3. **Traefik Ingress**: Traefik is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease. It can act as an Ingress controller in a Kubernetes cluster, managing access to services from the outside.

### The Program

The Pulumi program below does the following:

- It installs the Strimzi Kafka operator, which manages Kafka clusters within the Kubernetes cluster.
- Creates a Kafka cluster custom resource, which the Strimzi operator picks up to deploy a Kafka cluster.
- Sets up Traefik as an Ingress controller to manage access to services externally.

Before using the Pulumi code, ensure that you have Pulumi installed, are logged in, and the Kubernetes cluster is configured correctly with `kubectl`.

```typescript
import * as k8s from "@pulumi/kubernetes";

// 1. Install the Strimzi Kafka Operator
const strimziOperator = new k8s.yaml.ConfigFile("strimzi-operator", {
    file: "https://strimzi.io/install/latest?namespace=kafka", // Change 'kafka' to the desired namespace
});

// 2. Create a Kafka Cluster Resource
const kafkaCluster = new k8s.yaml.ConfigFile("kafka-cluster", {
    file: "./kafka-cluster.yaml",
}, { dependsOn: [strimziOperator] });

// The 'kafka-cluster.yaml' defines the Kafka cluster configuration.
// You need to create this YAML file based on your requirements and the Strimzi documentation.

// 3. Install Traefik Ingress Controller
const traefik = new k8s.helm.v3.Chart("traefik", {
    chart: "traefik",
    version: "9.18.2",
    fetchOpts: {
        // Use the correct repository URL for Traefik Helm chart
        repo: "https://helm.traefik.io/traefik",
    },
    values: {
        deployment: {
            // Ensure resources are adequate for production use
            resources: {
                requests: {
                    cpu: "100m",
                    memory: "100Mi"
                },
                limits: {
                    cpu: "300m",
                    memory: "300Mi"
                },
            },
        },
        service: {
            // Specify service types and annotations for cloud provider integrations if necessary
            type: "LoadBalancer",
            annotations: {
                // Add any necessary annotations for the cloud provider
            },
        },
    },
});

// 4. Create an Ingress Resource
// The Ingress resource defines how external HTTP/S traffic should be routed to the Kafka services.
// Typically this might be a UI or a REST proxy for Kafka.

// Save your ingress configuration in 'kafka-ingress.yaml'.
const kafkaIngress = new k8s.yaml.ConfigFile("kafka-ingress", {
    file: "./kafka-ingress.yaml",
}, { dependsOn: [traefik, kafkaCluster] });

// Ensure you have a 'kafka-ingress.yaml' with your desired host rules and paths.

// Export the Traefik LoadBalancer IP or Hostname
export const traefikEndpoint = traefik.getResourceProperty("v1/Service", "traefik", "status")
    .apply(status => status.loadBalancer.ingress[0]);
```

Remember to populate the `kafka-cluster.yaml` and `kafka-ingress.yaml` with the appropriate Kafka cluster configuration and Ingress routing rules respectively. You can find examples and detailed configurations in the [Strimzi documentation](https://strimzi.io/docs/operators/latest/using.html) and the [Traefik documentation](https://doc.traefik.io/traefik/routing/providers/kubernetes-ingress/).

This code will create the described resources in your existing Kubernetes cluster. After running `pulumi up` with this program, the described resources will be deployed to your cluster. You can then check the Ingress's external IP or host by using `kubectl get svc` to see Traefik's LoadBalancer service information or by inspecting the stack output of the Pulumi program. This is the endpoint you'll use for interacting with your Kafka cluster.

Adapt the resource allocations, annotations, and any other configurations as necessary to match your cluster setup and requirements. The actual Kafka deployment and service exposure configuration may differ based on your specific use case and environment.