How do I aggregate and manage Kubernetes logs with Fluentd?

In this guide, we will set up Fluentd to aggregate and manage logs from a Kubernetes cluster using Pulumi. Fluentd is an open-source data collector that helps you unify data collection and consumption for better use and understanding of data.

We will create a Kubernetes Namespace for Fluentd, deploy Fluentd as a DaemonSet, and configure it to collect logs from all containers in the cluster.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a Kubernetes Namespace for Fluentd
const namespace = new k8s.core.v1.Namespace("fluentd-namespace", {
    metadata: { name: "fluentd" },
});

// Define the Fluentd DaemonSet
const fluentdDaemonSet = new k8s.apps.v1.DaemonSet("fluentd-daemonset", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "fluentd",
    },
    spec: {
        selector: {
            matchLabels: {
                name: "fluentd",
            },
        },
        template: {
            metadata: {
                labels: {
                    name: "fluentd",
                },
            },
            spec: {
                containers: [
                    {
                        name: "fluentd",
                        image: "fluent/fluentd:v1.11.5-debian-1.0",
                        ports: [
                            {
                                containerPort: 24224,
                                name: "forward",
                            },
                        ],
                        volumeMounts: [
                            {
                                name: "varlog",
                                mountPath: "/var/log",
                            },
                            {
                                name: "varlibdockercontainers",
                                mountPath: "/var/lib/docker/containers",
                                readOnly: true,
                            },
                        ],
                    },
                ],
                volumes: [
                    {
                        name: "varlog",
                        hostPath: {
                            path: "/var/log",
                        },
                    },
                    {
                        name: "varlibdockercontainers",
                        hostPath: {
                            path: "/var/lib/docker/containers",
                        },
                    },
                ],
            },
        },
    },
});

// Export the namespace and DaemonSet names
export const namespaceName = namespace.metadata.name;
export const daemonSetName = fluentdDaemonSet.metadata.name;

Key Points

Namespace: We created a Kubernetes Namespace named fluentd for organizing Fluentd resources.
DaemonSet: We deployed Fluentd as a DaemonSet to ensure a Fluentd Pod runs on each node in the cluster.
Volumes: We mounted the host’s /var/log and /var/lib/docker/containers directories into the Fluentd container to collect logs from these paths.

Summary

We successfully set up Fluentd to aggregate and manage logs from a Kubernetes cluster using Pulumi. The Fluentd DaemonSet ensures that logs from all containers are collected and managed efficiently.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.