Using kubernetes kafka.strimzi.io with helm.fluxcd.io

Question

Pulumi · Accepted Answer

To build a Kubernetes infrastructure that includes Kafka via the Strimzi.io operator and manages it with the Flux CD's Helm operator, you would use Helm to package and deploy the Strimzi Kafka operator, and FluxCD to ensure that the Kafka instance within the Kubernetes cluster is synchronized with the desired state as defined in a Git repository.

Below, I will guide you through creating a Pulumi program using TypeScript to accomplish this. The program will:

Create a Helm Release for the Strimzi Kafka operator using the kubernetes.helm.v3.Chart resource.
Set up Flux CD with the flux.FluxBootstrapGit resource to manage the deployment using GitOps principles.

Pulumi Program Explanation

The program consists of two primary parts:

Helm Chart for Strimzi Kafka Operator: We will create a Helm chart instance for deploying the Strimzi Kafka operator. Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations.
Flux CD Bootstrap: We will set up Flux CD to watch a repository that contains the desired state of your Kafka deployment. Flux CD will apply the changes from the repository to your cluster so that the actual state matches the desired state, thus achieving continuous delivery.

Let's dive into the code:

import * as kubernetes from "@pulumi/kubernetes";

// Define the settings for our Strimzi Kafka operator Helm chart
const strimziKafkaHelmChart = new kubernetes.helm.v3.Chart("strimzi-kafka", {
    chart: "strimzi-kafka-operator",
    version: "0.20.0", // Specify the version you want to deploy
    fetchOpts: {
        repo: "https://strimzi.io/charts/",
    },
});

// The settings and credentials for accessing Git should be put here.
// The namespace where Flux will be installed, and any other Flux components that you need.
// As well as parameters to specify interval between git repo checks and version of the Flux.
const fluxSettings = {
    // For the sake of illustration, these values should be replaced with actual values.
    gitRepoUrl: "https://github.com/your-org/your-kafka-config-repo.git",
    gitBranch: "main",
    gitPath: "path-to-kafka-configs",
    namespace: "flux-system",
    // Secret details to access your private git repository
    gitSecret: {
        name: "flux-git-auth",
        data: {
            identity: "<BASE64_ENCODED_GIT_SSH_PRIVATE_KEY>",
            "identity.pub": "<BASE64_ENCODED_GIT_SSH_PUBLIC_KEY>",
            "known_hosts": "<BASE64_ENCODED_GIT_KNOWN_HOSTS>",
        },
    },
};

// Use the flux.FluxBootstrapGit resource to configure Flux to synchronize with your Git repository
const flux = new kubernetes.apiextensions.CustomResource("flux-git-sync", {
    apiVersion: "fluxcd.io/v1beta1",
    kind: "HelmRepository",
    metadata: {
        namespace: fluxSettings.namespace,
    },
    spec: {
        interval: "1m",
        url: fluxSettings.gitRepoUrl,
        secretRef: {
            name: fluxSettings.gitSecret.name,
        },
    },
}, { dependsOn: [strimziKafkaHelmChart] });

// Store some information about the resources we've created
export const kafkaHelmChartName = strimziKafkaHelmChart.metadata.name;
export const fluxNamespace = fluxSettings.namespace;

Here's what each part does:

Strimzi Kafka Operator: The kubernetes.helm.v3.Chart resource is used to create a Helm release for the Strimzi Kafka operator. We specify the chart name, version, and repository where the Helm chart is located.
Flux Bootstrap: The kubernetes.apiextensions.CustomResource defines a Helm repository custom resource to be managed by Flux. It points to the Git repository where your Kafka configuration is stored.

Conclusion

In the above program, we've set up the Strimzi Kafka operator using a Helm chart and arranged for FluxCD to manage our Kafka configurations as specified in a Git repository. Whenever you push updates to this repository, Flux CD will automatically pick up the changes and update the Kafka configuration in your cluster to match, enabling a GitOps approach to managing your Kafka clusters.

By using Pulumi, you gain the ability to orchestrate complex Kubernetes setups in a straightforward, version-controlled way. Each commit to your Pulumi code is a potential change in your infrastructure, which can be reviewed and audited before deployment.

Remember to replace placeholder values such as <BASE64_ENCODED_GIT_SSH_PRIVATE_KEY> with actual encoded values for your Git credentials to ensure secure access to your repository. This program assumes that you have the necessary privileges on the Kubernetes cluster and access to the Git repository. The exact versions of Helm charts, as well as the namespace and other specific details, will vary based on your specific setup and should be adjusted to fit your needs.