Visualizing Auto-scaling Metrics for AI Workloads in Kubernetes

Question

Pulumi · Accepted Answer

To visualize auto-scaling metrics for AI workloads in Kubernetes, we can configure Horizontal Pod Autoscaler (HPA) resources to automatically scale the number of pods in a deployment based on observed CPU utilization or custom metrics. We would then use a monitoring tool that integrates with Kubernetes, such as Prometheus, to collect and visualize these metrics.

The following steps will be taken in the Pulumi program:
1. Deploy an AI workload in a Kubernetes Deployment.
2. Configure a HorizontalPodAutoscaler to automatically scale this workload.
3. Install Prometheus for metric collection.
4. Set up Prometheus to scrape HPA metrics.
5. Visualize the metrics via the Prometheus web UI.

In this example, we use the `pulumi_kubernetes` package that provides resources to interact with Kubernetes API objects, including Deployments and HorizontalPodAutoscalers.

Here's how you could write a Pulumi program in Python to achieve this:

```python
import pulumi
import pulumi_kubernetes as k8s

# Define a Kubernetes Deployment for the AI workload.
ai_workload = k8s.apps.v1.Deployment(
    "ai-workload",
    spec={
        "selector": {"matchLabels": {"app": "ai-workload"}},
        "replicas": 1,
        "template": {
            "metadata": {"labels": {"app": "ai-workload"}},
            "spec": {
                "containers": [
                    {
                        "name": "ai-container",
                        "image": "ai-application-image:latest",  # Replace with your actual AI application image
                        "resources": {
                            "requests": {"cpu": "500m"},
                            "limits": {"cpu": "1000m"},
                        },
                    }
                ],
            },
        },
    }
)

# Create a HorizontalPodAutoscaler to automatically scale the AI workload.
ai_workload_hpa = k8s.autoscaling.v1.HorizontalPodAutoscaler(
    "ai-workload-hpa",
    spec={
        "scaleTargetRef": {
            "apiVersion": "apps/v1",
            "kind": "Deployment",
            "name": ai_workload.metadata["name"],
        },
        "minReplicas": 1,
        "maxReplicas": 10,
        "targetCPUUtilizationPercentage": 80,
    },
    # Metadata used to match HPA with corresponding deployment.
    metadata={"labels": ai_workload.metadata["labels"]},
)

# Install Prometheus using the kube-prometheus-stack Helm chart.
# Here we stub the resource as 'prometheus_chart', assuming you would install it
# via Helm CLI or Pulumi's Helm support.
prometheus_chart = k8s.helm.v3.Chart(
    "prometheus",
    k8s.helm.v3.ChartOpts(
        chart="kube-prometheus-stack",
        version="15.2.3",  # Use the version that suits your requirements
        fetch_opts=k8s.helm.v3.FetchOpts(
            repo="https://prometheus-community.github.io/helm-charts",
        ),
    ),
)

# Export relevant endpoints for accessing Prometheus and AI application.
pulumi.export("ai_workload_name", ai_workload.metadata["name"])
pulumi.export("prometheus_ui", prometheus_chart.resources.apply(
    lambda resources: [
        r["metadata"]["name"] for r in resources if r["kind"] == "Service" and "prometheus" in r["metadata"]["name"]
    ]
))
```

In the code above:
* We define a Kubernetes Deployment with the given container image of the AI workload.
* An HPA resource is set to monitor the CPU utilization of the AI workload and scale the number of replicas up to 10 when it reaches 80%.
* We include the installation of Prometheus using the kube-prometheus-stack Helm chart. This will automatically set up Prometheus to scrape metrics from our workload and Kubernetes components. In a real-world situation, you'll need to ensure Prometheus is properly configured to scrape metrics from your workload.

Please make sure to replace `ai-application-image:latest` with your actual Docker image for the AI workload.

After applying this program with Pulumi, Prometheus will be installed in your cluster and start scraping metrics, which will include HPA metrics by default. You can access the Prometheus UI to query and visualize these metrics. The URL for the Prometheus UI is exported by the program, making it easy to access.