Using kubernetes iamauthenticator.k8s.aws with fluentbit.fluent.io

Question

Pulumi · Accepted Answer

To use the AWS IAM Authenticator with Kubernetes alongside Fluent Bit for log processing, you will integrate several components.

1. **AWS IAM Authenticator**: It is used for cluster authentication. AWS IAM Authenticator is a tool to use AWS IAM credentials to authenticate to a Kubernetes cluster. In EKS, the IAM Authenticator is managed by AWS, so you don't have to install it yourself. Instead, you ensure that the `aws-auth` ConfigMap is correctly configured so that the Kubernetes cluster knows which IAM users or roles can access it.

2. **Kubernetes**: As the orchestrator that will manage your containers.

3. **Fluent Bit**: It is a popular open-source log processor and forwarder, which allows you to collect data/logs from different sources, enrich them with filters, and send them to multiple destinations. It's part of the Fluentd ecosystem and is designed for high performance.

In Pulumi, you would typically use the `aws` or `aws-native` provider to interact with AWS resources, including EKS. As for Kubernetes resources, you would use the `kubernetes` provider to define and manage the deployment of Fluent Bit.

Below is a detailed Pulumi program, written in TypeScript, showing how to set up an EKS cluster and deploy Fluent Bit with AWS IAM as the authenticator. Note that the Fluent Bit set up as defined in this program will collect logs from the Kubernetes cluster and print them for demo purposes. In a real-world scenario, you would configure Fluent Bit to forward logs to your logging backend of choice.

```typescript
import * as aws from '@pulumi/aws';
import * as eks from '@pulumi/eks';
import * as k8s from '@pulumi/kubernetes';

// Create an EKS cluster.
const cluster = new eks.Cluster('my-cluster', {
    // Specify the IAM role to attach to the cluster.
    // You'd set up your role with the necessary permissions here.
    roleMappings: [
        {
            roleArn: 'arn:aws:iam::123456789012:role/EKSClusterRole', // Replace with the correct IAM role ARN.
            groups: ['system:masters'],
            username: 'pulumi:admin',
        },
    ],
    // ... other cluster config
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

// Set up Fluent Bit as a Kubernetes DaemonSet.
const fluentbitNamespace = new k8s.core.v1.Namespace('fluentbit-ns', {
    metadata: {
        name: 'fluentbit',
    },
}, { provider: cluster.provider });

const fluentbitConfigMap = new k8s.core.v1.ConfigMap('fluentbit-config', {
    metadata: {
        namespace: fluentbitNamespace.metadata.name,
    },
    data: {
        'fluent-bit.conf': `
            [SERVICE]
                Flush        1
                Log_Level    info
                Daemon       off
                Parsers_File parsers.conf

[INPUT]
                Name cpu
                Tag  cpu.local
                # ... other input config

[OUTPUT]
                Name  stdout
                Match *
        `,
        'parsers.conf': `
            [PARSER]
                Name        json
                Format      json
                Time_Key    time
                Time_Format %d/%b/%Y:%H:%M:%S %z
        `,
    },
}, { provider: cluster.provider });

const fluentbitDaemonSet = new k8s.apps.v1.DaemonSet('fluentbit', {
    metadata: {
        namespace: fluentbitNamespace.metadata.name,
    },
    spec: {
        selector: {
            matchLabels: {
                app: 'fluentbit',
            },
        },
        template: {
            metadata: {
                labels: {
                    app: 'fluentbit',
                },
            },
            spec: {
                // This service account would be linked to the AWS IAM role that provides access to AWS services
                serviceAccountName: 'fluentbit-service-account',
                containers: [
                    {
                        name: 'fluentbit',
                        image: 'fluent/fluent-bit:1.8',
                        // Given Fluent Bit's extensive configuration options, use a ConfigMap to manage its configuration
                        volumeMounts: [
                            { 
                                name: 'fluentbit-config',
                                mountPath: '/fluent-bit/etc/',
                            },
                        ],
                    },
                ],
                volumes: [
                    {
                        name: 'fluentbit-config',
                        configMap: {
                            name: fluentbitConfigMap.metadata.name,
                        },
                    },
                ],
            },
        },
    },
}, { provider: cluster.provider });

// Expose useful references, such as the Fluent Bit namespace name
export const fluentbitNamespaceName = fluentbitNamespace.metadata.name;
```

In this program:

- We start by creating an Amazon EKS cluster, specifying an IAM role ARN that has proper permissions to manage the cluster and the worker nodes.
- After the cluster is provisioned, we get a `kubeconfig` that can be used to interact with the cluster.
- We then define a Kubernetes namespace for Fluent Bit to keep our resources organized.
- Next, we define a `ConfigMap` for Fluent Bit's configuration—this allows Fluent Bit to know what data to process and where to send it.
- We continue by setting up Fluent Bit as a `DaemonSet`. This ensures that Fluent Bit runs on each node in the EKS cluster.
- We mount the earlier defined `ConfigMap` into the Fluent Bit pods, which allows Fluent Bit to access its configuration files.
- Finally, we export the Fluent Bit namespace name for ease of reference.

This program provides the required setup for Fluent Bit with proper AWS IAM integration for an EKS cluster. Adaptations would be needed for different use cases, such as custom parsers or different output sinks for the logs collected by Fluent Bit.