Configuring Prometheus to scrape metrics from Istio sidecars

Question

Pulumi · Accepted Answer

Prometheus is a popular open-source monitoring tool widely used in the Kubernetes ecosystem. It is designed to collect and store metrics as time series data, providing a powerful query language to analyze them as well as alerting capabilities.

Istio is a service mesh that provides traffic management, observability, and security to microservices. Istio sidecars are proxies deployed alongside microservices in Kubernetes pods, and they generate a wealth of metrics that provide insights into the behavior and performance of your microservices.

To enable Prometheus to scrape metrics from Istio sidecars, you need to perform the following steps:

1. Deploy Prometheus in your Kubernetes cluster.
2. Configure Prometheus with the proper scraping jobs to collect metrics from Istio.

The following Pulumi program demonstrates how to accomplish these two steps. It uses the Pulumi Kubernetes provider to deploy Prometheus into the cluster and configures it to scrape metrics from Istio sidecars.

First, make sure you have Pulumi installed and configured to access your Kubernetes cluster.

Let's walk through the code:

```typescript
import * as k8s from '@pulumi/kubernetes';

// Create a Kubernetes Namespace for Prometheus.
const namespace = new k8s.core.v1.Namespace('prometheus', {
    metadata: { name: 'prometheus' },
});

// Deploy Prometheus in the `prometheus` namespace.
const prometheusDeployment = new k8s.apps.v1.Deployment('prometheus-deployment', {
    metadata: {
        namespace: namespace.metadata.name,
        labels: { app: 'prometheus-server' },
    },
    spec: {
        replicas: 1,
        selector: {
            matchLabels: { app: 'prometheus-server' },
        },
        template: {
            metadata: { labels: { app: 'prometheus-server' } },
            spec: {
                containers: [{
                    name: 'prometheus',
                    image: 'prom/prometheus:v2.30.0',
                    args: ['--config.file=/etc/prometheus/prometheus.yml'],
                    ports: [{ containerPort: 9090 }],
                    volumeMounts: [{
                        name: 'prometheus-config-volume',
                        mountPath: '/etc/prometheus',
                    }],
                }],
                volumes: [{
                    name: 'prometheus-config-volume',
                    configMap: {
                        name: 'prometheus-config',
                    },
                }],
            },
        },
    },
});

// Define the Prometheus configuration as a Kubernetes ConfigMap.
const prometheusConfigMap = new k8s.core.v1.ConfigMap('prometheus-config', {
    metadata: { namespace: namespace.metadata.name },
    data: {
        'prometheus.yml': `
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'istio-mesh'
    scrape_interval: 5s
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_label_app]
        action: keep
        regex: 'istio-ingressgateway|istio-pilot|prometheus|grafana|...other_istio_components...'
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        action: keep
        regex: '.*-envoy-prom'
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: job
        regex: '(.+)-envoy'
        replacement: 'istio-$1'
`.trim(),
    },
}, { dependsOn: [namespace] });

// Expose Prometheus using a Kubernetes Service.
const service = new k8s.core.v1.Service('prometheus-service', {
    metadata: {
        namespace: namespace.metadata.name,
        labels: prometheusDeployment.metadata.labels,
    },
    spec: {
        type: 'LoadBalancer',
        ports: [{ port: 9090, targetPort: 9090 }],
        selector: { app: 'prometheus-server' },
    },
}, { dependsOn: [prometheusDeployment] });

// Export the Prometheus service URL
export const prometheusServiceUrl = service.status.loadBalancer.ingress[0].ip;
```

In this program, we set up a `Namespace` for Prometheus, a `ConfigMap` that includes the Prometheus scraping configuration, and a `Deployment` to run Prometheus itself.

The scraping configuration specified in the `ConfigMap` is provided in the `prometheus.yml` entry. It includes a single scraping job named `istio-mesh`, which instructs Prometheus to scrape metrics from Istio sidecars periodically. The job targets specifically the endpoints of services with labels related to Istio components and pods with container ports named with the `-envoy-prom` suffix, which are the standard names for Istio sidecar proxies' Prometheus metrics endpoints.

The key part here is the `scrape_configs` section, which defines the scraping rules for Prometheus. We have a job that targets certain labels (`istio-ingressgateway`, `istio-pilot`, etc.) and relabels the metrics to be identified as Istio components.

Lastly, we expose Prometheus using a `Service` with `LoadBalancer` type, making it accessible outside the Kubernetes cluster. The service URL is exported so that you can easily access the Prometheus web interface.

To apply this Pulumi program to your cluster, save the code to a file named `index.ts`, and run `pulumi up` from the same directory. Make sure Pulumi CLI is installed and configured to connect to your Kubernetes cluster.

For more information on the Pulumi Kubernetes provider, visit [Pulumi's Kubernetes documentation](https://www.pulumi.com/docs/reference/pkg/kubernetes/). For more about configuring Prometheus with Istio, check out the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) and the [Istio documentation](https://istio.io/latest/docs/ops/integrations/prometheus/).