Using kubernetes storage.k8s.io with fluentbit.fluent.io

Question

Pulumi · Accepted Answer

To integrate Kubernetes `storage.k8s.io` resources with `fluentbit.fluent.io`, we might want to set up a Fluent Bit logging system that utilizes a persistent storage class for log data storage. Fluent Bit is an open-source log processor and forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. It's designed to be lightweight and have a small footprint.

In Kubernetes, a `StorageClass` resource allows administrators to define different classes of storage based on the underlying storage provider. When a user requests persistent storage, Kubernetes will use the `StorageClass` to dynamically provision the storage based on the class specified in the PersistentVolumeClaim (PVC).

To accomplish this integration in Pulumi using TypeScript, we will perform the following steps:

1. Create a `StorageClass` to define the type of storage we want for our logs.
2. Create a `PersistentVolumeClaim` that requests storage of a certain size from the `StorageClass`.
3. Deploy Fluent Bit as a DaemonSet or Deployment, configured to use the PVC for storing logs.

Here's how you can create a Pulumi program in TypeScript to set up Fluent Bit with a persistent storage class in Kubernetes:

```typescript
import * as k8s from "@pulumi/kubernetes";

// Create a new Kubernetes StorageClass
const storageClass = new k8s.storage.v1.StorageClass("fluentbit-storage", {
    metadata: {
        name: "fluentbit-storage",
    },
    provisioner: "kubernetes.io/aws-ebs", // Replace with the appropriate provisioner for your cloud provider
    parameters: {
        type: "gp2", // General purpose SSD for AWS EBS
    },
    reclaimPolicy: "Retain", // Retain the volume after the PVC is deleted
    allowVolumeExpansion: true, // Allow the volume size to be increased if needed
});

// Create a PersistentVolumeClaim for Fluent Bit logs
const pvc = new k8s.core.v1.PersistentVolumeClaim("fluentbit-pvc", {
    metadata: {
        name: "fluentbit-pvc",
    },
    spec: {
        accessModes: ["ReadWriteOnce"], // Single node read-write
        storageClassName: storageClass.metadata.name,
        resources: {
            requests: {
                storage: "10Gi", // Request 10Gi storage
            },
        },
    },
});

// Deploy Fluent Bit using a DaemonSet
const fluentBitDaemonSet = new k8s.apps.v1.DaemonSet("fluentbit-ds", {
    metadata: {
        name: "fluentbit",
    },
    spec: {
        selector: { matchLabels: { name: "fluentbit" } },
        template: {
            metadata: { labels: { name: "fluentbit" } },
            spec: {
                containers: [{
                    name: "fluent-bit",
                    image: "fluent/fluent-bit:1.8", // Use the relevant Fluent Bit image
                    volumeMounts: [{
                        name: "fluentbit-storage",
                        mountPath: "/var/log/fluentbit-logs", // Mount path in the container for log storage
                    }],
                }],
                volumes: [{
                    name: "fluentbit-storage",
                    persistentVolumeClaim: {
                        claimName: pvc.metadata.name,
                    },
                }],
            },
        },
    },
});

// Export the Fluent Bit DaemonSet name
export const fluentBitName = fluentBitDaemonSet.metadata.name;
```

Explanation:

- The `StorageClass` named `fluentbit-storage` is defined with AWS `gp2` storage type. You need to substitute the provisioner and parameters based on your cloud provider and storage type you wish to use.
- The `PersistentVolumeClaim` named `fluentbit-pvc` requests 10Gi storage from the `fluentbit-storage` StorageClass and is set with `ReadWriteOnce` access mode, meaning it can be mounted as read-write by a single node.
- A Fluent Bit DaemonSet named `fluentbit-ds` is created to run on all nodes. It is configured with a volume mount that points to `/var/log/fluentbit-logs` where Fluent Bit can write its logs.
- The Fluent Bit container image version used is `1.8`, but you should use a specific version suitable for your needs.
- Finally, the program exports the Fluent Bit DaemonSet's name as a stack output.

This Pulumi program will set up Fluent Bit with persistent storage on your Kubernetes cluster. You can apply this configuration by running `pulumi up` after setting up your Pulumi and Kubernetes environment.