Using kubernetes postgresql.cnpg.io with fluentbit.fluent.io

Question

Pulumi · Accepted Answer

In order to set up a Kubernetes cluster with a PostgreSQL database using postgresql.cnpg.io and logging using fluentbit.fluent.io, we will need to do several things:

Provision a Kubernetes cluster.
Deploy a PostgreSQL instance within the Kubernetes cluster.
Deploy Fluent Bit as a logging agent to aggregate and forward logs.

For this program, I'll assume you have a Kubernetes cluster already running. If not, you can provision one using Pulumi's digitalocean.KubernetesCluster for DigitalOcean, azure-native.kubernetes.ConnectedCluster for Azure, or equivalent resources from other providers.

In Kubernetes, applications like PostgreSQL and Fluent Bit can be typically packaged and deployed using Helm charts. Helm is a package manager for Kubernetes that allows you to define, install, and upgrade even the most complex Kubernetes applications.

Let's set up a PostgreSQL deployment using a Helm chart from the postgresql.cnpg.io repository and a Fluent Bit deployment from the fluentbit.fluent.io repository.

Here's how you can set up PostgreSQL and Fluent Bit in your Kubernetes cluster using Pulumi with TypeScript:

import * as k8s from "@pulumi/kubernetes";

// Create a Pulumi Kubernetes provider that uses the current context of kubectl.
// Assumes that you have configured this using `kubectl` command line tool and have the right access setup
const provider = new k8s.Provider("k8s-provider", {
    kubeconfig: <Your kubeconfig here> // You can directly pass the kubeconfig or use a Pulumi config
});

// Deploy PostgreSQL using a Helm Chart
const postgresqlChart = new k8s.helm.v3.Chart("postgresql", {
    // Helm Chart configuration
    chart: "postgresql",
    version: "10.x.x", // Replace with the version you wish to deploy
    fetchOpts:{
        repo: "https://charts.bitnami.com/bitnami", // The repository containing the postgresql chart
    },
    // values for the PostgreSQL Helm Chart to configure the deployment
    values: {
        global: {
            storageClass: "standard-rwo", // Specify the storage class for the PostgreSQL PV
        },
        image: {
            registry: "docker.io",
            repository: "bitnami/postgresql",
            tag: "13.1.0", // Replace with the preferred PostgreSQL version
        },
        postgresqlUsername: "pulumi_user",
        postgresqlPassword: "pulumi_password", // Replace with your secure password or use Pulumi secrets
        postgresqlDatabase: "pulumi_db",
    },
}, { provider }); // Ensuring this Helm chart is installed using the k8s provider we configured

// Deploy Fluent Bit to collect logs using a Helm chart
const fluentBitChart = new k8s.helm.v3.Chart("fluentbit", {
    // Helm Chart configuration
    chart: "fluent-bit",
    version: "0.16.2", // Replace with the version you wish to deploy
    fetchOpts:{
        repo: "https://fluent.github.io/helm-charts", // The repository containing the fluent-bit chart
    },
    // Configuration values for the Fluent Bit Helm Chart.
    values: {
        backend: {
            type: "es", // Choose the backend type (e.g., Elasticsearch, standard output)
            es: {
                host: "elasticsearch-client", // Replace with your Elasticsearch service name and namespace
                port: 9200
            }
        },
        onPremise: {
            storage: {
                storagePath: "/var/log/fluentbit-buffers", // Local path for Fluent Bit logs buffering
            },
        },
    },
}, { provider }); // Ensuring this Helm chart is installed using the k8s provider we configured

// Optional: Export the PostgreSQL service URL
export const postgresqlServiceUrl = postgresqlChart.getResource("v1/Service", "postgresql").status.apply(s => `postgresql://${s.loadBalancer.ingress[0].ip}`);

Please note that you will need to replace <Your kubeconfig here> with the actual kubeconfig or a reference to it, and you will need to replace placeholders such as pulumi_password with your desired configuration values. For sensitive values like passwords, you should use Pulumi's secrets management to keep them secure.

Also, adjust the backend configuration for Fluent Bit depending on where you want to send the logs. In this example, I showed how to configure for Elasticsearch, but you can configure it to fit your actual logging infrastructure.

You can install these Helm charts into your running Kubernetes cluster by running pulumi up. This will create all the required resources for PostgreSQL and Fluent Bit within the cluster.