Centralizing subscription audit logging with Fluentd in Kubernetes

Question

Pulumi · Accepted Answer

To centralize subscription audit logging with Fluentd in Kubernetes, you need to achieve a few things:

Install Fluentd: This component is responsible for collecting logs from different sources. You usually run it as a DaemonSet in Kubernetes, which ensures that a Fluentd pod runs on every Kubernetes node.
Enable Audit Logging: Kubernetes must be configured to emit audit logs, which contain records of events happening in the Kubernetes API server.
Configure Fluentd: Fluentd must be set up to read these audit logs and then ship them to a centralized log storage solution.

In the Pulumi program below, I will create a Fluentd configuration to collect Kubernetes audit logs and then ship them. For demonstration purposes, I'll assume that the Fluentd daemon is configured to send logs to an Elasticsearch instance.

Please note, before running the Pulumi program, ensure that:

You have a Kubernetes cluster available and kubeconfig is set up to interact with your cluster.
You have the necessary permissions to create and manage resources in the cluster.
Elasticsearch or an alternative log storage service is already running, and you have the endpoint details.

Let's proceed with the needed Pulumi TypeScript program to set up Fluentd for centralizing audit logs:

import * as k8s from '@pulumi/kubernetes';

const fluentdConfigMapData = `
<match **>
  @type elasticsearch
  @id out_es
  @log_level info
  host "elasticsearch-logging.default.svc.cluster.local" // Use your Elasticsearch service address here
  port 9200
  index_name fluentd
  <buffer>
    @type file
    path /var/log/fluentd-buffers/kubernetes.system.buffer
    flush_interval 5s
  </buffer>
</match>
`;

// Create a ConfigMap for Fluentd configuration
const fluentdConfigMap = new k8s.core.v1.ConfigMap("fluentd-config", {
    metadata: {
        namespace: "kube-system",
    },
    data: {
        "fluent.conf": fluentdConfigMapData,
        "k8s-audit-log.conf": "@include kubernetes.audit.log",
    },
});

// Create a Fluentd DaemonSet to collect audit logs
const fluentdDaemonSet = new k8s.apps.v1.DaemonSet("fluentd-ds", {
    metadata: {
        name: "fluentd",
        namespace: "kube-system",
    },
    spec: {
        selector: {
            matchLabels: {
                name: "fluentd",
            },
        },
        template: {
            metadata: {
                labels: {
                    name: "fluentd",
                },
            },
            spec: {
                containers: [{
                    name: "fluentd",
                    image: "fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch", // Use the appropriate Fluentd image
                    env: [
                        {
                            name: "FLUENT_ELASTICSEARCH_HOST",
                            value: "elasticsearch-logging.default.svc.cluster.local", // Again, use your actual Elasticsearch host
                        },
                        {
                            name: "FLUENT_ELASTICSEARCH_PORT",
                            value: "9200",
                        },
                    ],
                    volumeMounts: [
                        {
                            name: "fluentd-conf",
                            mountPath: "/fluentd/etc",
                        },
                        // Mounting the directory on the host where audit logs are stored, if applicable
                        {
                            name: "varlog",
                            mountPath: "/var/log",
                        },
                        {
                            name: "varlibdockercontainers",
                            mountPath: "/var/lib/docker/containers",
                            readOnly: true,
                        },
                    ],
                }],
                volumes: [
                    {
                        name: "fluentd-conf",
                        configMap: {
                            name: fluentdConfigMap.metadata.name,
                        },
                    },
                    // Host volume where audit logs are stored to be collected by Fluentd
                    {
                        name: "varlog",
                        hostPath: {
                            path: "/var/log",
                        },
                    },
                    {
                        name: "varlibdockercontainers",
                        hostPath: {
                            path: "/var/lib/docker/containers",
                        },
                    },
                ],
            },
        },
    },
});

This program does the following:

Creates a ConfigMap: This is where the configuration for Fluentd is stored. The fluent.conf section includes the settings for connecting to Elasticsearch and the details needed to process and buffer logs.
Sets Up a DaemonSet: This will ensure Fluentd runs on each node of the cluster. Fluentd pods will use the configuration defined in the ConfigMap and mount the necessary volumes to access host paths that may contain logs.

Make sure to replace placeholders for Elasticsearch service addresses and any other configurations specific to your setup before running the program. Also, ensure Elasticsearch is up and reachable from within your Kubernetes cluster.

Once applied, this Pulumi program will deploy Fluentd to your Kubernetes cluster, and Fluentd will begin collecting and transmitting logs to your centralized Elasticsearch instance. You can then use Kibana or another Elasticsearch-compatible tool to visualize and query the logs.