Auto-instrumenting Kubernetes Pods with OpenTelemetry Sidecars

Question

Pulumi · Accepted Answer

To auto-instrument Kubernetes pods with OpenTelemetry sidecars, you'll want to take advantage of the Kubernetes Admission Controllers and specifically use a MutatingAdmissionWebhook. This webhook can alter the pod specifications on-the-fly to include OpenTelemetry sidecars without you having to modify each pod's specification manually.

The Pulumi code will consist of the following major parts:
1. Define a namespace for the OpenTelemetry components.
2. Deploy the OpenTelemetry Collector as a sidecar configuration.
3. Set up a MutatingAdmissionWebhook configuration that specifies how pods should be modified to include the OpenTelemetry sidecar.

Here's how you would create this setup using Pulumi with the Kubernetes provider:

```python
import pulumi
import pulumi_kubernetes as k8s

# Create a namespace for the OpenTelemetry components
otel_namespace = k8s.core.v1.Namespace("otel-namespace",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="open-telemetry",
    )
)

# The OpenTelemetry Collector sidecar configuration
otel_collector_sidecar = k8s.core.v1.ConfigMap("otel-collector-sidecar",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="otel-collector-sidecar-config",
        namespace=otel_namespace.metadata["name"],
    ),
    data={
        "collector.yaml": """
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: open-telemetry
spec:
  selector:
    matchLabels:
      app: otel-collector
  replicas: 1
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector:latest
        # Your OpenTelemetry configuration goes here under `config`.
        command:
          - "/otelcol"
          - "--config=/conf/collector.yaml"
        volumeMounts:
          - name: otel-collector-config-vol
            mountPath: /conf
      volumes:
        - name: otel-collector-config-vol
          configMap:
            name: otel-collector-sidecar-config
"""
    }
)

# The Mutating Webhook Configuration to inject the sidecar
otel_mutating_webhook_configuration = k8s.admissionregistration.v1.MutatingWebhookConfiguration("otel-mutating-webhook-configuration",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="opentelemetry-auto-instrumentation",
    ),
    webhooks=[k8s.admissionregistration.v1.MutatingWebhookArgs(
        name="otel-injection-webhook.open-telemetry.svc",
        client_config=k8s.admissionregistration.v1.WebhookClientConfigArgs(
            service=k8s.admissionregistration.v1.ServiceReferenceArgs(
                namespace=otel_namespace.metadata["name"],
                name="otel-collector",
                path="/inject",
            ),
        ),
        rules=[k8s.admissionregistration.v1.RuleWithOperationsArgs(
            operations=["CREATE"],
            rule=k8s.admissionregistration.v1.RuleArgs(
                api_groups=[""],
                api_versions=["v1"],
                resources=["pods"],
            ),
        )],
        side_effects="None",
        admission_review_versions=["v1"],
        reinvocation_policy="Never",
    )]
)

pulumi.export('namespace', otel_namespace.metadata["name"])
pulumi.export('config_map', otel_collector_sidecar.metadata["name"])
pulumi.export('webhook_configuration', otel_mutating_webhook_configuration.metadata["name"])
```

Let's break down the above code:

- We start by creating a dedicated namespace for the OpenTelemetry components.
- We define a `ConfigMap` that holds the OpenTelemetry Collector configuration. This configuration will be used by the sidecar.
- We create a `MutatingWebhookConfiguration` resource that describes when and how pods should be modified to include the OpenTelemetry sidecar. The `webhooks` field specifies the criteria for selecting requests and how they should be handled.

Keep in mind that in a real-world scenario, you would also need to define the required Role and RoleBinding resources that grant the webhook service account permissions to mutate pods within the cluster. Also, minikube or a Kubernetes cluster hosted on a cloud provider might require additional setup such as domain name configuration, certificate manager setup for SSL, and exposing the OpenTelemetry collector service externally if needed.

This Pulumi program sets up the necessary resources to auto-instrument Kubernetes pods with OpenTelemetry sidecars, but the actual implementation of the webhook service and its logic to inject the sidecar container into the pod specifications is something that you would need to implement as a separate piece of software running within your cluster.