How do I create interactive log analysis dashboards with Grafana and Loki?
In this guide, we will create interactive log analysis dashboards using Grafana and Loki on Kubernetes. Grafana is a popular open-source platform for monitoring and observability, while Loki is a log aggregation system designed to work with Grafana. We will deploy both Grafana and Loki on a Kubernetes cluster using Pulumi.
Steps:
- Provision a Kubernetes cluster: We will assume you already have a Kubernetes cluster running.
- Deploy Loki: We will deploy Loki as a log aggregation system.
- Deploy Grafana: We will deploy Grafana to visualize logs collected by Loki.
- Configure Grafana to use Loki: We will configure Grafana to use Loki as a data source.
import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
// Define the Loki deployment
const lokiDeployment = new k8s.apps.v1.Deployment("loki-deployment", {
spec: {
selector: { matchLabels: { app: "loki" } },
replicas: 1,
template: {
metadata: { labels: { app: "loki" } },
spec: {
containers: [{
name: "loki",
image: "grafana/loki:2.4.1",
ports: [{ containerPort: 3100 }],
args: [
"-config.file=/etc/loki/local-config.yaml"
],
volumeMounts: [{
name: "config",
mountPath: "/etc/loki"
}]
}],
volumes: [{
name: "config",
configMap: {
name: "loki-config"
}
}]
}
}
}
});
// Define the Loki service
const lokiService = new k8s.core.v1.Service("loki-service", {
spec: {
selector: { app: "loki" },
ports: [{ port: 3100, targetPort: 3100 }]
}
});
// Define the Loki config map
const lokiConfigMap = new k8s.core.v1.ConfigMap("loki-config", {
metadata: { name: "loki-config" },
data: {
"local-config.yaml": `
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
max_transfer_retries: 0
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h
storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
shared_store: filesystem
filesystem:
directory: /loki/chunks
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
`
}
});
// Define the Grafana deployment
const grafanaDeployment = new k8s.apps.v1.Deployment("grafana-deployment", {
spec: {
selector: { matchLabels: { app: "grafana" } },
replicas: 1,
template: {
metadata: { labels: { app: "grafana" } },
spec: {
containers: [{
name: "grafana",
image: "grafana/grafana:8.3.3",
ports: [{ containerPort: 3000 }],
env: [{
name: "GF_SECURITY_ADMIN_PASSWORD",
value: "admin"
}]
}]
}
}
}
});
// Define the Grafana service
const grafanaService = new k8s.core.v1.Service("grafana-service", {
spec: {
selector: { app: "grafana" },
ports: [{ port: 80, targetPort: 3000 }]
}
});
// Export the URLs to access Grafana and Loki
export const grafanaUrl = grafanaService.status.loadBalancer.ingress[0].hostname;
export const lokiUrl = lokiService.status.loadBalancer.ingress[0].hostname;
Key Points:
- Loki Deployment: We deploy Loki using a Kubernetes Deployment resource, specifying the necessary container image and configuration.
- Loki Service: We expose Loki using a Kubernetes Service resource.
- Grafana Deployment: We deploy Grafana using a Kubernetes Deployment resource, setting the admin password via environment variables.
- Grafana Service: We expose Grafana using a Kubernetes Service resource.
- Configuration: We use a ConfigMap to store Loki’s configuration.
Summary:
In this guide, we deployed Grafana and Loki on a Kubernetes cluster using Pulumi. We configured Loki to aggregate logs and Grafana to visualize them. The services were exposed to allow access to Grafana and Loki. This setup provides an interactive log analysis dashboard using Grafana and Loki.
Deploy this code
Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.
Sign upNew to Pulumi?
Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.
Sign upThank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.