Using kubernetes autoscaling with opentelemetry.io

Question

Pulumi · Accepted Answer

Autoscaling in Kubernetes is a key feature that allows you to automatically adjust the number of running pods in a Deployment, StatefulSet, or ReplicaSet based on current resource utilization or other select metrics. OpenTelemetry, on the other hand, is an observability framework used to gather telemetry data (metrics, logs, and traces) from your applications and services.

When you refer to using Kubernetes autoscaling with OpenTelemetry, you are likely interested in configuring Kubernetes to autoscale based on metrics obtained from services instrumented with OpenTelemetry.

Here we'll use Pulumi to define a `HorizontalPodAutoscaler` resource in a Kubernetes cluster which targets a specific deployment or other scalable resources. The autoscaler will scale these resources based on monitored CPU utilization or custom metrics defined.

The example below shows how to define a `HorizontalPodAutoscaler` targeting a Kubernetes deployment. While OpenTelemetry-specific setup isn't covered here, the assumption is that you have OpenTelemetry instrumented in your cluster and custom metrics setup that could be used by the autoscaler in conjunction with or in place of standard CPU utilization.

Let's walk through how you can use Pulumi to set up autoscaling in Kubernetes:

```typescript
import * as k8s from "@pulumi/kubernetes";

// This is a placeholder for where you would define your Kubernetes Deployment.
// You should have OpenTelemetry instrumented for your application/deployment
// which collects the metrics for autoscaling.

// The following code demonstrates how to create a HorizontalPodAutoscaler
// which will automatically scale your application in response to metrics like CPU usage.

const appName = "my-app"; // Change this to the name of your application.

// Set minimum and maximum number of replicas
const minReplicas = 2;
const maxReplicas = 10;

// Create a HorizontalPodAutoscaler to automatically scale the deployment based on CPU utilization
const hpa = new k8s.autoscaling.v1.HorizontalPodAutoscaler(appName + "-hpa", {
  metadata: {
    name: appName // Typically should be the same as your application/deployment name for clarity.
  },
  spec: {
    scaleTargetRef: {
      apiVersion: "apps/v1",
      kind: "Deployment",
      name: appName // This should be the name of the Deployment you want to autoscale.
    },
    minReplicas: minReplicas,
    maxReplicas: maxReplicas,
    targetCPUUtilizationPercentage: 50, // Target CPU utilization percentage for scaling. Adjust as necessary.
  }
});

// Export the name of the HPA resource we just created
export const horizontalPodAutoscalerName = hpa.metadata.name;
```

In this code:

- We define a new `HorizontalPodAutoscaler` resource with a minimum and maximum number of replicas.
- The `scaleTargetRef` field points to the Deployment we want to scale automatically—it should match the name of your existing deployment.
- `targetCPUUtilizationPercentage` is the desired target CPU utilization percentage that triggers the scaling action. When the average CPU utilization exceeds this threshold, Kubernetes will start to scale out the number of pods; when it is below, it will scale in.
- We export the name of the HorizontalPodAutoscaler as a stack output, which can be useful for CI/CD pipelines or monitoring.

Deploying this code using Pulumi will create a HorizontalPodAutoscaler in your cluster which automatically adjusts the size of the deployment named `my-app`.

You would typically combine this setup with OpenTelemetry by configuring your `HorizontalPodAutoscaler` to use custom metrics that OpenTelemetry collects. The above example uses CPU utilization for simplicity, but in a more advanced setup, you might define a custom metrics API and channel metrics from OpenTelemetry into Kubernetes' scaling mechanisms.

For setting up a comprehensive autoscaling policy using custom metrics from OpenTelemetry, you can refer to the Kubernetes documentation on [custom metrics](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics) and OpenTelemetry integration guides available on their [official site](https://opentelemetry.io/docs/) for more details about how to configure the agent, collector, and relevant metric sources.