1. Answers
  2. Persistent storage for Prometheus on Kubernetes

How do I set up persistent storage for Prometheus on Kubernetes?

To enable Prometheus persistent storage for long-term metric retention, we’ll use Kubernetes to deploy Prometheus with a persistent volume. This will ensure that the metrics data is retained even if the Prometheus pod is restarted. We’ll use a combination of Kubernetes resources such as PersistentVolume, PersistentVolumeClaim, and StatefulSet.

Here’s a detailed explanation of the resources we’ll use:

  1. PersistentVolume (PV): This is a piece of storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node.
  2. PersistentVolumeClaim (PVC): This is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources.
  3. StatefulSet: This is the workload API object used to manage stateful applications. It manages the deployment and scaling of a set of pods, and provides guarantees about the ordering and uniqueness of these pods.

We’ll create a PersistentVolume and PersistentVolumeClaim for Prometheus, and then deploy Prometheus using a StatefulSet that references the PVC for storage.

Below is the Pulumi program written in TypeScript to achieve this:

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create a PersistentVolume
const pv = new k8s.core.v1.PersistentVolume("prometheus-pv", {
    spec: {
        capacity: {
            storage: "10Gi",
        },
        accessModes: ["ReadWriteOnce"],
        persistentVolumeReclaimPolicy: "Retain",
        hostPath: {
            path: "/mnt/data",
        },
    },
});

// Create a PersistentVolumeClaim
const pvc = new k8s.core.v1.PersistentVolumeClaim("prometheus-pvc", {
    spec: {
        accessModes: ["ReadWriteOnce"],
        resources: {
            requests: {
                storage: "10Gi",
            },
        },
    },
});

// Deploy Prometheus using a StatefulSet
const prometheusStatefulSet = new k8s.apps.v1.StatefulSet("prometheus", {
    spec: {
        serviceName: "prometheus",
        replicas: 1,
        selector: {
            matchLabels: {
                app: "prometheus",
            },
        },
        template: {
            metadata: {
                labels: {
                    app: "prometheus",
                },
            },
            spec: {
                containers: [
                    {
                        name: "prometheus",
                        image: "prom/prometheus:v2.30.3",
                        ports: [
                            {
                                containerPort: 9090,
                                name: "web",
                            },
                        ],
                        volumeMounts: [
                            {
                                name: "prometheus-storage",
                                mountPath: "/prometheus",
                            },
                        ],
                    },
                ],
            },
        },
        volumeClaimTemplates: [
            {
                metadata: {
                    name: "prometheus-storage",
                },
                spec: {
                    accessModes: ["ReadWriteOnce"],
                    resources: {
                        requests: {
                            storage: "10Gi",
                        },
                    },
                },
            },
        ],
    },
});

// Export the Prometheus URL
export const prometheusUrl = pulumi.interpolate`http://${prometheusStatefulSet.spec.template.spec.containers[0].name}:9090`;

Explanation:

  1. PersistentVolume (PV): We create a PV with a storage capacity of 10Gi and an access mode of ReadWriteOnce. The hostPath specifies the path on the host where the data will be stored.
  2. PersistentVolumeClaim (PVC): We create a PVC that requests 10Gi of storage with an access mode of ReadWriteOnce.
  3. StatefulSet: We deploy Prometheus using a StatefulSet. The StatefulSet ensures that Prometheus has persistent storage by referencing the PVC. The volumeClaimTemplates section in the StatefulSet ensures that each replica of the StatefulSet gets its own PVC.

This setup ensures that Prometheus has persistent storage for long-term metric retention. You can adjust the storage size and other configurations as needed.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up