How do I deploy Strimzi Kafka on Kubernetes with Fluent Bit for log processing?

In this guide, we will deploy Strimzi Kafka on a Kubernetes cluster and set up Fluent Bit for log processing. Strimzi provides a way to run Apache Kafka on Kubernetes, while Fluent Bit is a lightweight log processor and forwarder.

We will use Pulumi to define and deploy the necessary Kubernetes resources. The resources include Kafka and Zookeeper clusters managed by Strimzi, and a Fluent Bit DaemonSet to collect and forward logs.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a namespace for Kafka and Fluent Bit
const namespace = new k8s.core.v1.Namespace("logging", {
    metadata: {
        name: "logging",
    },
});

// Deploy the Strimzi Kafka operator
const strimziOperator = new k8s.helm.v3.Chart("strimzi", {
    chart: "strimzi-kafka-operator",
    version: "0.25.0",
    fetchOpts: {
        repo: "https://strimzi.io/charts/",
    },
    namespace: namespace.metadata.name,
});

// Define a Kafka cluster
const kafkaCluster = new k8s.apiextensions.CustomResource("kafkaCluster", {
    apiVersion: "kafka.strimzi.io/v1beta2",
    kind: "Kafka",
    metadata: {
        name: "my-cluster",
        namespace: namespace.metadata.name,
    },
    spec: {
        kafka: {
            version: "2.8.0",
            replicas: 3,
            listeners: [
                {
                    name: "plain",
                    port: 9092,
                    type: "internal",
                    tls: false,
                },
                {
                    name: "tls",
                    port: 9093,
                    type: "internal",
                    tls: true,
                },
            ],
            config: {
                "offsets.topic.replication.factor": 3,
                "transaction.state.log.replication.factor": 3,
                "transaction.state.log.min.isr": 2,
                "log.message.format.version": "2.8",
            },
            storage: {
                type: "jbod",
                volumes: [
                    {
                        id: 0,
                        type: "persistent-claim",
                        size: "100Gi",
                        deleteClaim: false,
                    },
                ],
            },
        },
        zookeeper: {
            replicas: 3,
            storage: {
                type: "persistent-claim",
                size: "100Gi",
                deleteClaim: false,
            },
        },
        entityOperator: {
            topicOperator: {},
            userOperator: {},
        },
    },
});

// Deploy Fluent Bit as a DaemonSet
const fluentBitConfigMap = new k8s.core.v1.ConfigMap("fluent-bit-config", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "fluent-bit-config",
    },
    data: {
        "fluent-bit.conf": `
[SERVICE]
    Flush        1
    Log_Level    info

[INPUT]
    Name         tail
    Path         /var/log/containers/*.log
    Parser       docker
    Tag          kube.*

[FILTER]
    Name         kubernetes
    Match        kube.*
    Kube_URL     https://kubernetes.default.svc:443
    Merge_Log    On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off

[OUTPUT]
    Name         kafka
    Match        *
    Brokers      my-cluster-kafka-bootstrap.logging:9092
    Topics       logs
    Format       json
    Timestamp_Key @timestamp
    Timestamp_Format iso8601
        `,
    },
});

const fluentBitDaemonSet = new k8s.apps.v1.DaemonSet("fluent-bit", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "fluent-bit",
    },
    spec: {
        selector: {
            matchLabels: {
                "app.kubernetes.io/name": "fluent-bit",
            },
        },
        template: {
            metadata: {
                labels: {
                    "app.kubernetes.io/name": "fluent-bit",
                },
            },
            spec: {
                containers: [
                    {
                        name: "fluent-bit",
                        image: "fluent/fluent-bit:1.8.3",
                        volumeMounts: [
                            {
                                name: "varlog",
                                mountPath: "/var/log",
                            },
                            {
                                name: "config",
                                mountPath: "/fluent-bit/etc/fluent-bit.conf",
                                subPath: "fluent-bit.conf",
                            },
                        ],
                    },
                ],
                volumes: [
                    {
                        name: "varlog",
                        hostPath: {
                            path: "/var/log",
                        },
                    },
                    {
                        name: "config",
                        configMap: {
                            name: "fluent-bit-config",
                        },
                    },
                ],
            },
        },
    },
});

Key Points

We created a namespace named logging to isolate our Kafka and Fluent Bit resources.
We deployed the Strimzi Kafka operator using a Helm chart and defined a Kafka cluster with Zookeeper.
We set up Fluent Bit as a DaemonSet to collect logs from Kubernetes nodes and forward them to our Kafka cluster.

Summary

In this guide, we demonstrated how to deploy Strimzi Kafka on a Kubernetes cluster and set up Fluent Bit for log processing using Pulumi. We used a Helm chart to deploy the Strimzi Kafka operator, defined a Kafka cluster, and configured Fluent Bit to collect and forward logs to Kafka.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.