1. Auto-instrumenting Kubernetes Pods with OpenTelemetry Sidecars


    To auto-instrument Kubernetes pods with OpenTelemetry sidecars, you'll want to take advantage of the Kubernetes Admission Controllers and specifically use a MutatingAdmissionWebhook. This webhook can alter the pod specifications on-the-fly to include OpenTelemetry sidecars without you having to modify each pod's specification manually.

    The Pulumi code will consist of the following major parts:

    1. Define a namespace for the OpenTelemetry components.
    2. Deploy the OpenTelemetry Collector as a sidecar configuration.
    3. Set up a MutatingAdmissionWebhook configuration that specifies how pods should be modified to include the OpenTelemetry sidecar.

    Here's how you would create this setup using Pulumi with the Kubernetes provider:

    import pulumi import pulumi_kubernetes as k8s # Create a namespace for the OpenTelemetry components otel_namespace = k8s.core.v1.Namespace("otel-namespace", metadata=k8s.meta.v1.ObjectMetaArgs( name="open-telemetry", ) ) # The OpenTelemetry Collector sidecar configuration otel_collector_sidecar = k8s.core.v1.ConfigMap("otel-collector-sidecar", metadata=k8s.meta.v1.ObjectMetaArgs( name="otel-collector-sidecar-config", namespace=otel_namespace.metadata["name"], ), data={ "collector.yaml": """ apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector namespace: open-telemetry spec: selector: matchLabels: app: otel-collector replicas: 1 template: metadata: labels: app: otel-collector spec: containers: - name: otel-collector image: otel/opentelemetry-collector:latest # Your OpenTelemetry configuration goes here under `config`. command: - "/otelcol" - "--config=/conf/collector.yaml" volumeMounts: - name: otel-collector-config-vol mountPath: /conf volumes: - name: otel-collector-config-vol configMap: name: otel-collector-sidecar-config """ } ) # The Mutating Webhook Configuration to inject the sidecar otel_mutating_webhook_configuration = k8s.admissionregistration.v1.MutatingWebhookConfiguration("otel-mutating-webhook-configuration", metadata=k8s.meta.v1.ObjectMetaArgs( name="opentelemetry-auto-instrumentation", ), webhooks=[k8s.admissionregistration.v1.MutatingWebhookArgs( name="otel-injection-webhook.open-telemetry.svc", client_config=k8s.admissionregistration.v1.WebhookClientConfigArgs( service=k8s.admissionregistration.v1.ServiceReferenceArgs( namespace=otel_namespace.metadata["name"], name="otel-collector", path="/inject", ), ), rules=[k8s.admissionregistration.v1.RuleWithOperationsArgs( operations=["CREATE"], rule=k8s.admissionregistration.v1.RuleArgs( api_groups=[""], api_versions=["v1"], resources=["pods"], ), )], side_effects="None", admission_review_versions=["v1"], reinvocation_policy="Never", )] ) pulumi.export('namespace', otel_namespace.metadata["name"]) pulumi.export('config_map', otel_collector_sidecar.metadata["name"]) pulumi.export('webhook_configuration', otel_mutating_webhook_configuration.metadata["name"])

    Let's break down the above code:

    • We start by creating a dedicated namespace for the OpenTelemetry components.
    • We define a ConfigMap that holds the OpenTelemetry Collector configuration. This configuration will be used by the sidecar.
    • We create a MutatingWebhookConfiguration resource that describes when and how pods should be modified to include the OpenTelemetry sidecar. The webhooks field specifies the criteria for selecting requests and how they should be handled.

    Keep in mind that in a real-world scenario, you would also need to define the required Role and RoleBinding resources that grant the webhook service account permissions to mutate pods within the cluster. Also, minikube or a Kubernetes cluster hosted on a cloud provider might require additional setup such as domain name configuration, certificate manager setup for SSL, and exposing the OpenTelemetry collector service externally if needed.

    This Pulumi program sets up the necessary resources to auto-instrument Kubernetes pods with OpenTelemetry sidecars, but the actual implementation of the webhook service and its logic to inject the sidecar container into the pod specifications is something that you would need to implement as a separate piece of software running within your cluster.