Centralized Logging in Kubernetes using OpenTelemetry

Question

Pulumi · Accepted Answer

To achieve centralized logging in a Kubernetes cluster using OpenTelemetry, you'll need to set up an OpenTelemetry Collector within your cluster. This collector gathers logs, metrics, and traces and forwards them to a centralized backend for logging and analysis.

To integrate OpenTelemetry with Kubernetes, follow these steps:

Deploy an OpenTelemetry Collector as a Deployment or DaemonSet within your Kubernetes cluster.
Configure the Collector with Receivers, Processors, Exporters, and Extensions. Receivers accept data from various sources; Processors allow for data transformation/modification; Exporters send the data to a backend like Jaeger, Prometheus, or any other supported backend; Extensions provide capabilities like health checking.
Use Kubernetes Service to expose the Collector to receive data.
Configure your applications to send logs to the Collector.
Configure backends and storage to maintain and visualize the logs.

Below is a basic Pulumi program that deploys an OpenTelemetry Collector to a Kubernetes cluster. To keep it simple and clear, let's assume that the Collector only receives logs and exports them to a hypothetical logging backend. In your case, this logging backend would be configured according to the specifics of the service you're using (for example, Splunk, Elastic, Datadog, etc.).

Here's a sample Pulumi program to deploy an OpenTelemetry Collector to your Kubernetes cluster:

import pulumi
import pulumi_kubernetes as kubernetes

# Define the Kubernetes provider, replace <context> with your Kubernetes cluster context
k8s_provider = kubernetes.Provider('k8s', kubeconfig='<kubeconfig>')

# Define the OpenTelemetry Collector configuration as Kubernetes ConfigMap
otel_collector_config = kubernetes.core.v1.ConfigMap(
    'otel-collector-conf',
    api_version='v1',
    kind='ConfigMap',
    metadata={
        'name': 'otel-collector-conf',
        'namespace': 'logging'
    },
    data={'otel-collector-config': """
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]
    """},
    opts=pulumi.ResourceOptions(provider=k8s_provider)
)

# Deploy the OpenTelemetry Collector
otel_collector_deployment = kubernetes.apps.v1.Deployment(
    'otel-collector-deploy',
    api_version='apps/v1',
    kind='Deployment',
    metadata={
        'name': 'otel-collector',
        'namespace': 'logging'
    },
    spec={
        'replicas': 2,
        'selector': {
            'matchLabels': {
                'app': 'otel-collector'
            }
        },
        'template': {
            'metadata': {
                'labels': {
                    'app': 'otel-collector'
                }
            },
            'spec': {
                'containers': [{
                    'name': 'otel-collector',
                    'image': 'otel/opentelemetry-collector:latest',
                    'command': ["otelcol --config=/conf/otel-collector-config.yaml"],
                    'volumeMounts': [{
                        'mountPath': '/conf',
                        'name': 'otel-collector-conf-volume',
                    }]
                }],
                'volumes': [{
                    'name': 'otel-collector-conf-volume',
                    'configMap': {
                        'name': 'otel-collector-conf'
                    }
                }]
            }
        }
    },
    opts=pulumi.ResourceOptions(provider=k8s_provider)
)

# Expose the OpenTelemetry Collector with a Kubernetes Service
otel_collector_service = kubernetes.core.v1.Service(
    'otel-collector-svc',
    api_version='v1',
    kind='Service',
    metadata={
        'name': 'otel-collector',
        'namespace': 'logging'
    },
    spec={
        'selector': {
            'app': 'otel-collector'
        },
        'ports': [{
            'protocol': 'TCP',
            'port': 4317,
            'targetPort': 4317,
            'name': 'grpc'
        }, {
            'protocol': 'TCP',
            'port': 55681,
            'targetPort': 55681,
            'name': 'http'
        }]
    },
    opts=pulumi.ResourceOptions(provider=k8s_provider)
)

# Export the Collector Service endpoint for easy access
pulumi.export('collector_service_endpoint', pulumi.Output.concat("http://", otel_collector_service.metadata.apply(lambda metadata: metadata['name']), ':', otel_collector_service.spec.apply(lambda spec: spec['ports'][0]['port'])))

In this program, we did the following:

Created a Kubernetes provider to interact with the desired Kubernetes cluster.
Defined a ConfigMap to store the OpenTelemetry Collector configuration.
Deployed an OpenTelemetry Collector to the cluster as a Kubernetes Deployment.
Exposed the Collector using a Kubernetes Service so that it can receive data on specific ports.
Exported the service endpoint for easy access and interaction.

The OpenTelemetry Collector configuration is critical; it defines how the Collector processes and exports the data. In the exporters section, replace the logging exporter with the one that sends data to your centralized backend.

Make sure you adjust the program based on your requirements and the specific backend service you intend to use for logging.

To use this code, replace <kubeconfig> with the path to your Kubernetes cluster's kubeconfig file. Then, deploy this code using the Pulumi CLI tools. Remember, you might need to configure RBAC permissions accordingly to allow the Collector to scrape and export logs from the pods within the cluster.