How do I configure Prometheus to scrape application metrics using Pod annotations?

To configure Prometheus to scrape application metrics using Pod annotations, you need to set up Prometheus in your Kubernetes cluster and annotate the pods with the necessary metadata so that Prometheus can automatically discover and scrape metrics from them.

Here’s a step-by-step guide on how to achieve this using Pulumi and TypeScript:

Set up Prometheus in your Kubernetes cluster: You can use the Prometheus Operator to deploy and manage Prometheus instances in your Kubernetes cluster. The Prometheus Operator simplifies the configuration of Prometheus and its related resources.
Annotate your Pods: Add Prometheus-specific annotations to your Pods so that Prometheus can discover and scrape metrics from them.

Step-by-Step Guide

Install Prometheus Operator: First, you need to install the Prometheus Operator in your Kubernetes cluster. You can do this using the kubernetes package in Pulumi.
Create a ServiceMonitor: A ServiceMonitor resource defines how Prometheus should scrape metrics from your services. You will create a ServiceMonitor that selects the Pods based on the annotations.
Annotate your Pods: Add the necessary annotations to your Pod templates so that Prometheus can discover and scrape metrics from them.

Here’s how you can do this using Pulumi and TypeScript:

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a namespace for Prometheus
const prometheusNamespace = new k8s.core.v1.Namespace("prometheus", {
    metadata: {
        name: "prometheus",
    },
});

// Install the Prometheus Operator
const prometheusOperator = new k8s.helm.v3.Chart("prometheus-operator", {
    chart: "kube-prometheus-stack",
    version: "16.5.0",
    fetchOpts: {
        repo: "https://prometheus-community.github.io/helm-charts",
    },
    namespace: prometheusNamespace.metadata.name,
});

// Create a ServiceMonitor to scrape metrics from Pods with specific annotations
const serviceMonitor = new k8s.apiextensions.CustomResource("serviceMonitor", {
    apiVersion: "monitoring.coreos.com/v1",
    kind: "ServiceMonitor",
    metadata: {
        name: "example-service-monitor",
        namespace: prometheusNamespace.metadata.name,
    },
    spec: {
        selector: {
            matchLabels: {
                app: "example-app",
            },
        },
        endpoints: [
            {
                port: "metrics",
                interval: "30s",
            },
        ],
    },
});

// Annotate your Pods with Prometheus-specific annotations
const appLabels = { app: "example-app" };
const deployment = new k8s.apps.v1.Deployment("example-app", {
    metadata: {
        labels: appLabels,
    },
    spec: {
        selector: { matchLabels: appLabels },
        replicas: 1,
        template: {
            metadata: {
                labels: appLabels,
                annotations: {
                    "prometheus.io/scrape": "true",
                    "prometheus.io/port": "9090",
                    "prometheus.io/path": "/metrics",
                },
            },
            spec: {
                containers: [
                    {
                        name: "example-app",
                        image: "example/app:latest",
                        ports: [{ containerPort: 9090 }],
                    },
                ],
            },
        },
    },
});

Explanation

Namespace Creation: We create a namespace for Prometheus to isolate its resources.
Prometheus Operator: We use the Prometheus Operator Helm chart to deploy Prometheus and its related components.
ServiceMonitor: We create a ServiceMonitor resource that tells Prometheus to scrape metrics from Pods with the label app: example-app.
Pod Annotations: We annotate the Pods with prometheus.io/scrape, prometheus.io/port, and prometheus.io/path so that Prometheus knows where to find the metrics.

This setup ensures that Prometheus can automatically discover and scrape metrics from your annotated Pods. You can customize the annotations and ServiceMonitor specifications as needed for your specific use case.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.