Using kubernetes fluentbit.fluent.io with minio.min.io

Question

Pulumi · Accepted Answer

To use Fluent Bit with MinIO on Kubernetes, you need to achieve several things:

1. Deploy MinIO as a storage backend to store your logs.
2. Deploy Fluent Bit as a log processor and forwarder, configured to send logs to MinIO.

To do this, you'll define Kubernetes manifests for both Fluent Bit and MinIO. Fluent Bit will be configured to forward logs to MinIO using the S3 output plugin.

Below is a basic Pulumi program that defines a MinIO deployment and a Fluent Bit DaemonSet with the necessary ConfigMap. This program assumes that you already have a Kubernetes cluster running and Pulumi installed and configured.

### MinIO Setup:

MinIO is an object storage server compatible with Amazon S3 APIs. You'll deploy MinIO using a Deployment and a Service for access. For simplicity, the setup uses a `ClusterIP` Service, which makes MinIO accessible only within the cluster. For production, consider using a `LoadBalancer` service or Ingress for secure external access and configuring MinIO with persistent volumes for data durability.

### Fluent Bit Setup:

Fluent Bit is a log processor and forwarder that allows you to collect data like logs from different sources and send them to multiple destinations. It is designed with a pluggable and flexible architecture. In this program, you'll configure Fluent Bit to watch for container logs and use the S3 output plugin to send the logs to the deployed MinIO service.

### Prerequisites:

Before running this Pulumi program, make sure to have the following:

- A Kubernetes cluster.
- `kubectl` configured to interact with your cluster.
- Pulumi CLI installed and setup to manage resources on the Kubernetes cluster.

Now, let's proceed with the Pulumi TypeScript program:

```typescript
import * as k8s from "@pulumi/kubernetes";

// Create a Kubernetes Namespace for MinIO.
const minioNamespace = new k8s.core.v1.Namespace("minio-namespace", {
    metadata: {
        name: "minio",
    },
});

// Deploy MinIO using a Kubernetes Deployment.
const minioDeployment = new k8s.apps.v1.Deployment("minio-deployment", {
    metadata: {
        namespace: minioNamespace.metadata.name,
    },
    spec: {
        selector: {
            matchLabels: {
                app: "minio",
            },
        },
        replicas: 1,
        template: {
            metadata: {
                labels: {
                    app: "minio",
                },
            },
            spec: {
                containers: [{
                    name: "minio",
                    image: "minio/minio:latest",
                    args: ["server", "/data"],
                    ports: [{ containerPort: 9000 }],
                    env: [
                        // Default access key and secret key for MinIO.
                        // For production, use more secure keys and consider using Kubernetes Secrets.
                        { name: "MINIO_ACCESS_KEY", value: "minioaccesskey" },
                        { name: "MINIO_SECRET_KEY", value: "miniosecretkey" },
                    ],
                    volumeMounts: [{
                        name: "storage",
                        mountPath: "/data",
                    }],
                }],
                volumes: [{
                    name: "storage",
                    emptyDir: {}, // For production, this should be a persistent volume claim
                }],
            },
        },
    },
}, { dependsOn: [minioNamespace] });

// Expose MinIO with a Kubernetes Service.
const minioService = new k8s.core.v1.Service("minio-service", {
    metadata: {
        namespace: minioNamespace.metadata.name,
    },
    spec: {
        type: "ClusterIP",
        ports: [{ port: 9000, targetPort: 9000 }],
        selector: {
            app: "minio",
        },
    },
}, { dependsOn: [minioDeployment] });

// Create a Kubernetes ConfigMap that contains the Fluent Bit configuration.
const fluentBitConfigMap = new k8s.core.v1.ConfigMap("fluentbit-config", {
    metadata: {
        namespace: minioNamespace.metadata.name,
    },
    data: {
        // Configuration for Fluent Bit. This includes input, filter, and output sections.
        "fluent-bit.conf": `
            [SERVICE]
                Flush         1
                Log_Level     info
                Parsers_File  parsers.conf
                HTTP_Server   On
                HTTP_Listen   0.0.0.0
                HTTP_Port     2020

[INPUT]
                Name          tail
                Path          /var/log/containers/*.log
                Parser        docker
                Tag           kube.*

[FILTER]
                Name          kubernetes
                Match         kube.*

[OUTPUT]
                Name          s3
                Match         kube.*
                bucket        fluentbit
                region        custom
                s3_key_format /$TAG/%Y/%m/%d/%H/%M/%S
                endpoint      http://minio-service.minio:9000
                access_key_id minioaccesskey
                secret_access_key miniosecretkey
                total_file_size 250M
        `,
        "parsers.conf": `
            [PARSER]
                Name        docker
                Format      json
                Time_Key    time
                Time_Format %Y-%m-%dT%H:%M:%S.%L
                Time_Keep   On
        `,
    },
});

// Deploy Fluent Bit as a DaemonSet to collect all the logs.
const fluentBitDaemonSet = new k8s.apps.v1.DaemonSet("fluentbit-daemonset", {
    metadata: {
        namespace: minioNamespace.metadata.name,
    },
    spec: {
        selector: {
            matchLabels: {
                name: "fluent-bit",
            },
        },
        template: {
            metadata: {
                labels: {
                    name: "fluent-bit",
                },
            },
            spec: {
                containers: [{
                    name: "fluent-bit",
                    image: "fluent/fluent-bit:1.8",
                    volumeMounts: [
                        {
                            name: "varlog",
                            mountPath: "/var/log",
                        },
                        {
                            name: "varlibdockercontainers",
                            mountPath: "/var/lib/docker/containers",
                            readOnly: true,
                        },
                        {
                            name: "fluent-bit-config",
                            mountPath: "/fluent-bit/etc/",
                        },
                    ],
                }],
                volumes: [
                    {
                        name: "varlog",
                        hostPath: { path: "/var/log" },
                    },
                    {
                        name: "varlibdockercontainers",
                        hostPath: { path: "/var/lib/docker/containers" },
                    },
                    {
                        name: "fluent-bit-config",
                        configMap: { name: fluentBitConfigMap.metadata.name },
                    },
                ],
            },
        },
    },
}, { dependsOn: [fluentBitConfigMap] });

// Export the MinIO Service endpoint for accessing the MinIO dashboard.
export const minioEndpoint = minioService.metadata.name.apply(name => `http://${name}.minio:9000`);
```

This program consists of several important parts:

- The `minioNamespace` variable defines a `Namespace` where all the MinIO and Fluent Bit resources will reside.
- The `minioDeployment` and `minioService` variables define the MinIO server and its service in the cluster. Here, MinIO is configured with default access and secret keys—replace these with your own keys and consider using `Secrets` for production environments.
- The `fluentBitConfigMap` variable defines a `ConfigMap` that contains the Fluent Bit configuration, which will be mounted into the Fluent Bit pods.
- The `fluentBitDaemonSet` variable defines the DaemonSet that will run Fluent Bit on each node in the cluster to collect logs and send them to MinIO.

After deploying this Pulumi program, you'll have Fluent Bit forwarding logs to MinIO, where you can access and manage them.

Please note that you should adjust the resource definitions and configurations based on your specific requirements and best practices, such as using Persistent Volumes for MinIO storage, managing secrets securely, setting resource limits for the pods, and configuring proper access control.

To apply this Pulumi program:
1. Save the code in a file named `index.ts`.
2. Run the following command to create a new Pulumi project:
   ```
   pulumi new typescript --force
   ```
3. Replace the content of `index.ts` with the code from above.
4. Run `pulumi up` to create the resources in the cluster. Follow the CLI instructions to perform the deployment.

Please review Kubernetes and Pulumi documentation to understand how the configurations work and how they can be modified to suit different environments and requirements.