How do I set up a Grafana Logs instance for Kubernetes events?
In this guide, we will set up a Grafana Logs instance using Grafana Loki to collect and visualize Kubernetes events. Grafana Loki is a log aggregation system designed to store and query logs from Kubernetes applications. We will use Pulumi to automate the deployment of Grafana Loki and the necessary Kubernetes resources.
Key Points:
- Grafana Loki: A log aggregation system that integrates with Grafana for log visualization.
- Kubernetes Events: Events generated by Kubernetes components, useful for monitoring and debugging.
- Pulumi: An infrastructure as code tool to automate the deployment of cloud resources.
Steps:
- Deploy a Grafana Loki instance in your Kubernetes cluster.
- Configure Loki to collect Kubernetes events.
- Set up Grafana to visualize the logs collected by Loki.
Below is the Pulumi program to set up a Grafana Logs instance for Kubernetes events:
import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
// Create a namespace for Loki
const namespace = new k8s.core.v1.Namespace("loki-namespace", {
metadata: { name: "loki" }
});
// Deploy the Loki StatefulSet
const lokiStatefulSet = new k8s.apps.v1.StatefulSet("loki-statefulset", {
metadata: {
namespace: namespace.metadata.name,
name: "loki"
},
spec: {
serviceName: "loki",
replicas: 1,
selector: {
matchLabels: { app: "loki" }
},
template: {
metadata: {
labels: { app: "loki" }
},
spec: {
containers: [{
name: "loki",
image: "grafana/loki:2.2.0",
ports: [{ containerPort: 3100 }],
volumeMounts: [{
name: "loki-storage",
mountPath: "/loki"
}],
args: [
"-config.file=/etc/loki/local-config.yaml"
]
}]
}
},
volumeClaimTemplates: [{
metadata: {
name: "loki-storage"
},
spec: {
accessModes: ["ReadWriteOnce"],
resources: {
requests: {
storage: "10Gi"
}
}
}
}]
}
});
// Create a Service for Loki
const lokiService = new k8s.core.v1.Service("loki-service", {
metadata: {
namespace: namespace.metadata.name,
name: "loki"
},
spec: {
selector: { app: "loki" },
ports: [{
port: 3100,
targetPort: 3100
}]
}
});
// Deploy the Promtail DaemonSet to collect Kubernetes events
const promtailDaemonSet = new k8s.apps.v1.DaemonSet("promtail-daemonset", {
metadata: {
namespace: namespace.metadata.name,
name: "promtail"
},
spec: {
selector: {
matchLabels: { app: "promtail" }
},
template: {
metadata: {
labels: { app: "promtail" }
},
spec: {
containers: [{
name: "promtail",
image: "grafana/promtail:2.2.0",
args: [
"-config.file=/etc/promtail/config.yaml"
],
volumeMounts: [{
name: "config",
mountPath: "/etc/promtail"
}, {
name: "varlog",
mountPath: "/var/log"
}, {
name: "varlibdockercontainers",
mountPath: "/var/lib/docker/containers",
readOnly: true
}]
}],
volumes: [{
name: "config",
configMap: {
name: "promtail-config"
}
}, {
name: "varlog",
hostPath: {
path: "/var/log"
}
}, {
name: "varlibdockercontainers",
hostPath: {
path: "/var/lib/docker/containers"
}
}]
}
}
}
});
// Create a ConfigMap for Promtail configuration
const promtailConfigMap = new k8s.core.v1.ConfigMap("promtail-config", {
metadata: {
namespace: namespace.metadata.name,
name: "promtail-config"
},
data: {
"config.yaml": `
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
- job_name: kubernetes
kubernetes_sd_configs:
- role: node
relabel_configs:
- source_labels: [__meta_kubernetes_node_name]
target_label: __host__
`
}
});
Summary
In this setup, we:
- Created a namespace for Grafana Loki.
- Deployed a Loki StatefulSet to collect logs.
- Created a Service to expose the Loki instance.
- Deployed a Promtail DaemonSet to collect Kubernetes events and send them to Loki.
- Configured Promtail using a ConfigMap to scrape logs from Kubernetes nodes and system logs.
This setup allows you to collect and visualize Kubernetes events using Grafana Loki, providing valuable insights for monitoring and debugging your Kubernetes cluster.
Deploy this code
Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.
Sign upNew to Pulumi?
Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.
Sign upThank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.