1. Answers
  2. Deploy Fluent Bit with ClickHouse on Kubernetes

How do I deploy Fluent Bit with ClickHouse on Kubernetes?

In this example, we will deploy Fluent Bit with ClickHouse on a Kubernetes cluster using Pulumi. Fluent Bit will be configured to collect logs from the Kubernetes cluster and forward them to ClickHouse for storage and analysis.

Key Points:

  • We will create a Kubernetes namespace for Fluent Bit and ClickHouse.
  • We will deploy ClickHouse using a StatefulSet.
  • We will deploy Fluent Bit as a DaemonSet to collect logs from all nodes.
  • Fluent Bit will be configured to forward logs to ClickHouse.
import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a namespace for Fluent Bit and ClickHouse
const namespace = new k8s.core.v1.Namespace("logging", {
    metadata: { name: "logging" },
});

// Deploy ClickHouse StatefulSet
const clickhouseStatefulSet = new k8s.apps.v1.StatefulSet("clickhouse", {
    metadata: { namespace: namespace.metadata.name },
    spec: {
        serviceName: "clickhouse",
        replicas: 1,
        selector: { matchLabels: { app: "clickhouse" } },
        template: {
            metadata: { labels: { app: "clickhouse" } },
            spec: {
                containers: [{
                    name: "clickhouse",
                    image: "yandex/clickhouse-server:latest",
                    ports: [{ containerPort: 8123 }],
                    volumeMounts: [{ name: "clickhouse-storage", mountPath: "/var/lib/clickhouse" }],
                }],
            },
        },
        volumeClaimTemplates: [{
            metadata: { name: "clickhouse-storage" },
            spec: {
                accessModes: ["ReadWriteOnce"],
                resources: { requests: { storage: "10Gi" } },
            },
        }],
    },
});

// Deploy Fluent Bit DaemonSet
const fluentBitDaemonSet = new k8s.apps.v1.DaemonSet("fluent-bit", {
    metadata: { namespace: namespace.metadata.name },
    spec: {
        selector: { matchLabels: { app: "fluent-bit" } },
        template: {
            metadata: { labels: { app: "fluent-bit" } },
            spec: {
                containers: [{
                    name: "fluent-bit",
                    image: "fluent/fluent-bit:latest",
                    ports: [{ containerPort: 2020 }],
                    volumeMounts: [{ name: "varlog", mountPath: "/var/log" }],
                    env: [
                        { name: "FLUENT_BIT_OUTPUT_HOST", value: "clickhouse.logging.svc.cluster.local" },
                        { name: "FLUENT_BIT_OUTPUT_PORT", value: "8123" },
                    ],
                }],
                volumes: [{ name: "varlog", hostPath: { path: "/var/log" } }],
            },
        },
    },
});

Conclusion

In this example, we created a namespace, deployed ClickHouse using a StatefulSet, and deployed Fluent Bit using a DaemonSet. Fluent Bit is configured to collect logs from all nodes and forward them to ClickHouse. This setup allows for efficient log collection and storage, enabling powerful log analysis capabilities.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up