Auto-Scaling AI Workload Management with Traefik in Kubernetes

Question

Pulumi · Accepted Answer

Auto-scaling is a method that dynamically adjusts the number of running instances of an application based on the workload. In Kubernetes, this can be achieved by implementing a Horizontal Pod Autoscaler (HPA), which automatically scales the number of pods in a deployment or replicaset based on observed CPU utilization or on some other, user-defined metrics.

To manage an AI workload in Kubernetes, you might have different pods running AI models that need to scale based on the number of incoming requests or the computational load. Traefik, a modern HTTP reverse proxy and load balancer, can be used to route traffic to these pods and also can be integrated with Kubernetes to automatically discover the required services.

Below is a Pulumi program written in Python that sets up an auto-scaling environment within a Kubernetes cluster, including the deployment of a sample application and Traefik as the ingress controller. The program assumes you have a Kubernetes cluster up and running and that you have configured Pulumi to use your Kubernetes cluster context.

The main resources used in the Pulumi program are:
- `Deployment`: Defines a desired state for the pod replicas managed by Kubernetes. In this context, we use it to deploy the sample application that will process the AI workload.
- `Service`: An abstraction layer which defines a logical set of pods and a policy by which to access them. This might be the backend pods serving the AI application.
- `Ingress`: Exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.
- `HorizontalPodAutoscaler`: Automatically scales the number of pods in a replication controller, deployment, or stateful set based on observed CPU utilization or custom metrics.

Let's create the Pulumi program step by step:

```python
import pulumi
import pulumi_kubernetes as k8s

# Create a Kubernetes Deployment for the AI application
ai_app_labels = {"app": "ai-workload"}
ai_app_deployment = k8s.apps.v1.Deployment(
    "ai-app-deployment",
    spec=k8s.apps.v1.DeploymentSpecArgs(
        replicas=1,  # Start with one replica
        selector=k8s.meta.v1.LabelSelectorArgs(match_labels=ai_app_labels),
        template=k8s.core.v1.PodTemplateSpecArgs(
            metadata=k8s.meta.v1.ObjectMetaArgs(labels=ai_app_labels),
            spec=k8s.core.v1.PodSpecArgs(
                containers=[k8s.core.v1.ContainerArgs(
                    name="ai-app",
                    image="YOUR_AI_APP_IMAGE", # Replace with your AI application's image
                    resources=k8s.core.v1.ResourceRequirementsArgs(
                        requests={"cpu": "500m"},  # Minimum resources required
                        limits={"cpu": "1000m"},  # Maximum resources the app can use
                    ),
                )],
            ),
        ),
    ),
)

# Create a Kubernetes Service to expose the AI application
ai_app_service = k8s.core.v1.Service(
    "ai-app-service",
    spec=k8s.core.v1.ServiceSpecArgs(
        selector=ai_app_labels,
        ports=[k8s.core.v1.ServicePortArgs(
            port=80,
            target_port=8080,
        )],
        type="ClusterIP",  # Internal cluster service only
    ),
)

# Deploy Traefik as an Ingress controller, pre-configured to use the service created
traefik_ingress = k8s.networking.v1.Ingress(
    "traefik-ingress",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        annotations={"kubernetes.io/ingress.class": "traefik"},
    ),
    spec=k8s.networking.v1.IngressSpecArgs(
        rules=[k8s.networking.v1.IngressRuleArgs(
            http=k8s.networking.v1.HTTPIngressRuleValueArgs(
                paths=[k8s.networking.v1.HTTPIngressPathArgs(
                    path="/ai",
                    backend=k8s.networking.v1.IngressBackendArgs(
                        service=k8s.networking.v1.IngressServiceBackendArgs(
                            name=ai_app_service.metadata.name,
                            port=k8s.networking.v1.ServiceBackendPortArgs(number=80),
                        ),
                    ),
                )],
            ),
        )],
    ),
)

# Auto-scaling: Create a Horizontal Pod Autoscaler (HPA) to scale the AI application
ai_app_hpa = k8s.autoscaling.v2beta2.HorizontalPodAutoscaler(
    "ai-app-hpa",
    spec=k8s.autoscaling.v2beta2.HorizontalPodAutoscalerSpecArgs(
        scale_target_ref=k8s.autoscaling.v2beta2.CrossVersionObjectReferenceArgs(
            api_version="apps/v1",
            kind="Deployment",
            name=ai_app_deployment.metadata.name,
        ),
        min_replicas=1,  # Minimum number of replicas
        max_replicas=10,  # Maximum number of replicas
        metrics=[k8s.autoscaling.v2beta2.MetricSpecArgs(
            type="Resource",
            resource=k8s.autoscaling.v2beta2.ResourceMetricSourceArgs(
                name="cpu",
                target=k8s.autoscaling.v2beta2.MetricTargetArgs(
                    type="Utilization",
                    average_utilization=80,  # Target CPU utilization before scaling
                ),
            ),
        )],
    ),
)

# Export the URL to access the AI application through Traefik
pulumi.export("ai_app_url", traefik_ingress.status.load_balancer.ingress[0].hostname.apply(lambda hostname: f"http://{hostname}/ai"))
```

This program defines the necessary Kubernetes objects to deploy an AI app and auto-scale it based on CPU utilization. It starts with a single replica and scales up to a maximum of 10 replicas if the average CPU utilization exceeds 80%. The traffic routing is managed by Traefik, which acts as an ingress controller listening for incoming requests on the `/ai` path.

Remember to replace `YOUR_AI_APP_IMAGE` with the actual container image for your AI application, and adjust the CPU requests and limits as needed for your specific workload.

Finally, the `pulumi.export` line provides an output after the deployment is complete, giving you the URL where the AI application will be accessible. This URL is dynamically obtained from the load balancer's ingress hostname with the correct path routed through Traefik.