1. Answers
  2. Setting Up a Grafana Logs Instance for Kubernetes Events

How do I set up a Grafana Logs instance for Kubernetes events?

In this guide, we will set up a Grafana Logs instance using Grafana Loki to collect and visualize Kubernetes events. Grafana Loki is a log aggregation system designed to store and query logs from Kubernetes applications. We will use Pulumi to automate the deployment of Grafana Loki and the necessary Kubernetes resources.

Key Points:

  1. Grafana Loki: A log aggregation system that integrates with Grafana for log visualization.
  2. Kubernetes Events: Events generated by Kubernetes components, useful for monitoring and debugging.
  3. Pulumi: An infrastructure as code tool to automate the deployment of cloud resources.

Steps:

  1. Deploy a Grafana Loki instance in your Kubernetes cluster.
  2. Configure Loki to collect Kubernetes events.
  3. Set up Grafana to visualize the logs collected by Loki.

Below is the Pulumi program to set up a Grafana Logs instance for Kubernetes events:

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a namespace for Loki
const namespace = new k8s.core.v1.Namespace("loki-namespace", {
    metadata: { name: "loki" }
});

// Deploy the Loki StatefulSet
const lokiStatefulSet = new k8s.apps.v1.StatefulSet("loki-statefulset", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "loki"
    },
    spec: {
        serviceName: "loki",
        replicas: 1,
        selector: {
            matchLabels: { app: "loki" }
        },
        template: {
            metadata: {
                labels: { app: "loki" }
            },
            spec: {
                containers: [{
                    name: "loki",
                    image: "grafana/loki:2.2.0",
                    ports: [{ containerPort: 3100 }],
                    volumeMounts: [{
                        name: "loki-storage",
                        mountPath: "/loki"
                    }],
                    args: [
                        "-config.file=/etc/loki/local-config.yaml"
                    ]
                }]
            }
        },
        volumeClaimTemplates: [{
            metadata: {
                name: "loki-storage"
            },
            spec: {
                accessModes: ["ReadWriteOnce"],
                resources: {
                    requests: {
                        storage: "10Gi"
                    }
                }
            }
        }]
    }
});

// Create a Service for Loki
const lokiService = new k8s.core.v1.Service("loki-service", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "loki"
    },
    spec: {
        selector: { app: "loki" },
        ports: [{
            port: 3100,
            targetPort: 3100
        }]
    }
});

// Deploy the Promtail DaemonSet to collect Kubernetes events
const promtailDaemonSet = new k8s.apps.v1.DaemonSet("promtail-daemonset", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "promtail"
    },
    spec: {
        selector: {
            matchLabels: { app: "promtail" }
        },
        template: {
            metadata: {
                labels: { app: "promtail" }
            },
            spec: {
                containers: [{
                    name: "promtail",
                    image: "grafana/promtail:2.2.0",
                    args: [
                        "-config.file=/etc/promtail/config.yaml"
                    ],
                    volumeMounts: [{
                        name: "config",
                        mountPath: "/etc/promtail"
                    }, {
                        name: "varlog",
                        mountPath: "/var/log"
                    }, {
                        name: "varlibdockercontainers",
                        mountPath: "/var/lib/docker/containers",
                        readOnly: true
                    }]
                }],
                volumes: [{
                    name: "config",
                    configMap: {
                        name: "promtail-config"
                    }
                }, {
                    name: "varlog",
                    hostPath: {
                        path: "/var/log"
                    }
                }, {
                    name: "varlibdockercontainers",
                    hostPath: {
                        path: "/var/lib/docker/containers"
                    }
                }]
            }
        }
    }
});

// Create a ConfigMap for Promtail configuration
const promtailConfigMap = new k8s.core.v1.ConfigMap("promtail-config", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "promtail-config"
    },
    data: {
        "config.yaml": `
            server:
              http_listen_port: 9080
              grpc_listen_port: 0

            positions:
              filename: /tmp/positions.yaml

            clients:
              - url: http://loki:3100/loki/api/v1/push

            scrape_configs:
              - job_name: system
                static_configs:
                  - targets:
                      - localhost
                    labels:
                      job: varlogs
                      __path__: /var/log/*log
              - job_name: kubernetes
                kubernetes_sd_configs:
                  - role: node
                relabel_configs:
                  - source_labels: [__meta_kubernetes_node_name]
                    target_label: __host__
        `
    }
});

Summary

In this setup, we:

  • Created a namespace for Grafana Loki.
  • Deployed a Loki StatefulSet to collect logs.
  • Created a Service to expose the Loki instance.
  • Deployed a Promtail DaemonSet to collect Kubernetes events and send them to Loki.
  • Configured Promtail using a ConfigMap to scrape logs from Kubernetes nodes and system logs.

This setup allows you to collect and visualize Kubernetes events using Grafana Loki, providing valuable insights for monitoring and debugging your Kubernetes cluster.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up