How do I configure Prometheus to scrape Kubernetes cluster metrics?

In this guide, we will demonstrate how to configure Prometheus to scrape metrics from a Kubernetes cluster using Pulumi. Prometheus is an open-source systems monitoring and alerting toolkit, and it can scrape metrics from various sources, including Kubernetes clusters.

We will use Pulumi to define and deploy the necessary resources, including the Prometheus server and the configuration to scrape metrics from the Kubernetes cluster.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a namespace for Prometheus
const prometheusNamespace = new k8s.core.v1.Namespace("prometheus", {
    metadata: { name: "prometheus" },
});

// Define the Prometheus configuration
const prometheusConfigMap = new k8s.core.v1.ConfigMap("prometheus-config", {
    metadata: {
        namespace: prometheusNamespace.metadata.name,
        name: "prometheus-config",
    },
    data: {
        "prometheus.yml": `
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https
  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: instance
        replacement: $1
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
        replacement: $1
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
        regex: (.+)
        replacement: $1
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
        regex: (.+)
        replacement: $1
`,
    },
});

// Deploy Prometheus using a Deployment
const prometheusDeployment = new k8s.apps.v1.Deployment("prometheus-deployment", {
    metadata: {
        namespace: prometheusNamespace.metadata.name,
        name: "prometheus",
    },
    spec: {
        replicas: 1,
        selector: {
            matchLabels: { app: "prometheus" },
        },
        template: {
            metadata: {
                labels: { app: "prometheus" },
            },
            spec: {
                containers: [
                    {
                        name: "prometheus",
                        image: "prom/prometheus:v2.26.0",
                        ports: [{ containerPort: 9090 }],
                        volumeMounts: [
                            {
                                name: "config-volume",
                                mountPath: "/etc/prometheus",
                            },
                        ],
                    },
                ],
                volumes: [
                    {
                        name: "config-volume",
                        configMap: {
                            name: prometheusConfigMap.metadata.name,
                        },
                    },
                ],
            },
        },
    },
});

// Expose Prometheus using a Service
const prometheusService = new k8s.core.v1.Service("prometheus-service", {
    metadata: {
        namespace: prometheusNamespace.metadata.name,
        name: "prometheus",
    },
    spec: {
        selector: {
            app: "prometheus",
        },
        ports: [
            {
                port: 9090,
                targetPort: 9090,
            },
        ],
    },
});

// Export the Prometheus URL
export const prometheusUrl = pulumi.interpolate`http://${prometheusService.metadata.name}.${prometheusService.metadata.namespace}.svc.cluster.local:9090`;

Key Points

Namespace Creation: We create a namespace for Prometheus to organize its resources.
Configuration: We define a ConfigMap with the Prometheus configuration to scrape Kubernetes cluster metrics.
Deployment: We deploy Prometheus using a Kubernetes Deployment, specifying the container image and mounting the configuration.
Service Exposure: We expose Prometheus using a Kubernetes Service to make it accessible within the cluster.
Export URL: We export the Prometheus URL to access the Prometheus dashboard.

Summary

In this guide, we configured Prometheus to scrape metrics from a Kubernetes cluster using Pulumi. We created a namespace, defined the Prometheus configuration, deployed Prometheus, exposed it using a service, and exported the Prometheus URL for easy access. This setup allows you to monitor your Kubernetes cluster metrics effectively.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.