Using kubernetes kafka.strimzi.io with cilium.io

Question

Pulumi · Accepted Answer

When setting up Apache Kafka on Kubernetes, you can use Strimzi, which makes it easy to run Kafka on Kubernetes clusters by providing container images and Operators for deploying and managing Kafka clusters. Networking within a Kubernetes cluster can be managed by various CNI (Container Network Interface) plugins—one popular choice is Cilium, which provides networking and security by leveraging eBPF.

In this Pulumi program, you'll learn how to create a Kubernetes cluster and then install both Strimzi to manage Kafka and Cilium for networking. The Kubernetes cluster creation step will be abstracted in this demonstration. Still, you can accomplish this using Pulumi with providers such as AWS (EKS), Azure (AKS), or Google Cloud (GKE), or even by using `kubernetes` provider if you've got an existing cluster.

Below, I'll guide you through setting up Strimzi and Cilium using Pulumi's `kubernetes` provider. This program assumes you have a Kubernetes cluster running and accessible via `kubectl` from your local machine.

We will start with configuring the Kubernetes provider, then proceed to install Cilium for network management. After that, we'll proceed to deploy Strimzi to set up Kafka.

```typescript
import * as k8s from "@pulumi/kubernetes";

// Create a Kubernetes Provider using the default kubeconfig credentials.
const provider = new k8s.Provider("k8s", {});

// Deploy Cilium Helm Chart for Kubernetes networking.
// The helm chart will install all necessary resources and configure Cilium as the CNI for the cluster.
// You can customize the values to fit your specific needs by consulting the Cilium Helm chart documentation.
const ciliumRelease = new k8s.helm.v3.Release("cilium", {
  chart: "cilium",
  version: "1.9.1", // Use the version of Cilium that you want to install.
  namespace: "kube-system", // This is the namespace that Cilium will be installed into. It is typical to put network plugins into kube-system.
  repositoryOpts: {
    repo: "https://helm.cilium.io/",
  },
}, { provider });

// Deploy Strimzi Helm Chart for managing Kafka on Kubernetes.
// This deployment will create the CRDs, roles, and operator for managing Kafka clusters.
// Similar to Cilium, you can customize the installation via Helm chart values.
const strimziOperatorRelease = new k8s.helm.v3.Release("strimzi-kafka-operator", {
  chart: "strimzi-kafka-operator",
  version: "0.23.0", // Make sure to specify the version of Strimzi you want to use.
  namespace: "kafka", // We are installing Strimzi in its own namespace called `kafka`.
  repositoryOpts: {
    repo: "https://strimzi.io/charts/",
  },
}, { provider, dependsOn: ciliumRelease });
// Ensure that Cilium is set up before moving on to deploying Strimzi.

// Once the Strimzi Operator is deployed, you can now deploy Kafka clusters by creating custom Kafka cluster resources.

// Export the Strimzi and Cilium status
export const strimziStatus = strimziOperatorRelease.status;
export const ciliumStatus = ciliumRelease.status;
```

This program first sets up a Kubernetes provider using your local kubeconfig file, which allows Pulumi to interact with your Kubernetes cluster. It then proceeds to deploy Cilium using its Helm chart, specifying the version and installation namespace. Cilium acts as the network plugin and provides network security for Kubernetes.

After Cilium setup, this program deploys Strimzi using its Helm chart. It installs the necessary Custom Resource Definitions (CRDs), roles, and the Strimzi Operator in the specified namespace. The Strimzi Operator will manage Kafka resources in your cluster.

The `dependsOn` property ensures that Pulumi will create Cilium before Strimzi, as Cilium must be available to provide networking for the Strimzi components and any Kafka clusters it manages.

Lastly, it exports the status details of both the Cilium and Strimzi Helm releases, allowing you to verify the deployments' status directly from your Pulumi stack outputs.

Please ensure you have Helm installed and the Helm Charts for both Cilium and Strimzi available. For this program to run successfully, your Kubernetes cluster should not have conflicting CNIs (Container Network Interfaces) installed, as Cilium will take over networking responsibilities. Also, ensure that your Kubernetes cluster has the adequate RBAC permissions for installing these components.

To use this program, you will need Pulumi installed and set up. Save this code to a TypeScript file, and run it using the Pulumi CLI:

1. Create a new Pulumi project if you haven't done so.
2. Add this TypeScript code to your `index.ts` file in the Pulumi project.
3. Run `pulumi up` to deploy the Cilium and Strimzi charts to your Kubernetes cluster.

Remember, you will need to manage the Kafka clusters by creating the Kafka resources referencing the CRDs installed by Strimzi. Pulumi enables you to do that as well, but those steps are beyond the initial setup shown here.