1. Answers
  2. Deploy Fluent Bit with ClickHouse on Kubernetes

How Do I Deploy Fluent Bit With ClickHouse on Kubernetes?

Introduction

Deploying Fluent Bit with ClickHouse on Kubernetes enables efficient log management and analysis. In this guide, we will use Pulumi to automate the deployment process. Our objective is to set up Fluent Bit to collect logs from the Kubernetes cluster and forward them to ClickHouse, where they can be stored and analyzed. This setup ensures that log data is efficiently captured and available for real-time analysis.

Key Points:

  • Namespace Creation: We will create a dedicated Kubernetes namespace for managing Fluent Bit and ClickHouse resources.
  • ClickHouse Deployment: ClickHouse will be deployed using a StatefulSet, ensuring data persistence and high availability.
  • Fluent Bit Deployment: Fluent Bit will be deployed as a DaemonSet to ensure log collection from all nodes within the cluster.
  • Log Forwarding: Fluent Bit will be configured to forward logs directly to ClickHouse for storage and further analysis.

Step-by-Step Deployment Process

  1. Create a Namespace: Start by creating a new namespace in Kubernetes to isolate the Fluent Bit and ClickHouse resources. This helps in managing resources and permissions effectively.

  2. Deploy ClickHouse:

    • Use a StatefulSet to deploy ClickHouse. This ensures that the database maintains its state across restarts.
    • Configure the StatefulSet to use persistent storage by defining a volumeClaimTemplate.

  3. Deploy Fluent Bit:

    • Deploy Fluent Bit as a DaemonSet to ensure it runs on every node in the cluster.
    • Configure Fluent Bit to collect logs from the /var/log directory of each node.
    • Set environment variables to define ClickHouse as the log output destination.

  4. Configure Log Forwarding:

    • Ensure that Fluent Bit is set to forward logs to the ClickHouse service endpoint within the Kubernetes cluster.
import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a namespace for Fluent Bit and ClickHouse
const namespace = new k8s.core.v1.Namespace("logging", {
    metadata: { name: "logging" },
});

// Deploy ClickHouse StatefulSet
const clickhouseStatefulSet = new k8s.apps.v1.StatefulSet("clickhouse", {
    metadata: { namespace: namespace.metadata.name },
    spec: {
        serviceName: "clickhouse",
        replicas: 1,
        selector: { matchLabels: { app: "clickhouse" } },
        template: {
            metadata: { labels: { app: "clickhouse" } },
            spec: {
                containers: [{
                    name: "clickhouse",
                    image: "yandex/clickhouse-server:latest",
                    ports: [{ containerPort: 8123 }],
                    volumeMounts: [{ name: "clickhouse-storage", mountPath: "/var/lib/clickhouse" }],
                }],
            },
        },
        volumeClaimTemplates: [{
            metadata: { name: "clickhouse-storage" },
            spec: {
                accessModes: ["ReadWriteOnce"],
                resources: { requests: { storage: "10Gi" } },
            },
        }],
    },
});

// Deploy Fluent Bit DaemonSet
const fluentBitDaemonSet = new k8s.apps.v1.DaemonSet("fluent-bit", {
    metadata: { namespace: namespace.metadata.name },
    spec: {
        selector: { matchLabels: { app: "fluent-bit" } },
        template: {
            metadata: { labels: { app: "fluent-bit" } },
            spec: {
                containers: [{
                    name: "fluent-bit",
                    image: "fluent/fluent-bit:latest",
                    ports: [{ containerPort: 2020 }],
                    volumeMounts: [{ name: "varlog", mountPath: "/var/log" }],
                    env: [
                        { name: "FLUENT_BIT_OUTPUT_HOST", value: "clickhouse.logging.svc.cluster.local" },
                        { name: "FLUENT_BIT_OUTPUT_PORT", value: "8123" },
                    ],
                }],
                volumes: [{ name: "varlog", hostPath: { path: "/var/log" } }],
            },
        },
    },
});

Conclusion

In this setup, we successfully created a namespace and deployed ClickHouse using a StatefulSet for persistent storage. Fluent Bit was deployed as a DaemonSet to collect logs from all nodes and forward them to ClickHouse. This configuration allows for efficient log collection and powerful analysis capabilities, providing valuable insights into your Kubernetes cluster’s performance and health.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up