Kubernetes Ingress Controllers for AI Model Serving

Question

Pulumi · Accepted Answer

In Kubernetes, an Ingress Controller is a component that manages access to services within a Kubernetes cluster from the outside world. When serving AI models, you would typically use an Ingress to expose your model serving application (often hosted inside a Kubernetes Service), enabling external traffic to reach it.

The `kubernetes-ingress-nginx.IngressController` resource can be used to manage an NGINX Ingress Controller which is a popular choice for this purpose. NGINX acts as a reverse proxy, directing HTTP(S) traffic to the appropriate backend services based on the rules specified in the Ingress resource.

In addition to the Ingress Controller, you will also need an Ingress resource to define the routing rules and a Service to actually expose your AI model serving application.

Here's how you could use Pulumi to set this up in Python:

1. Define a Deployment for your AI model serving application.
2. Create a Service to expose the Deployment.
3. Install the NGINX Ingress Controller.
4. Create an Ingress to define how traffic should be routed to your Service.

Let's walk through each step in detail with a Pulumi program:

```python
import pulumi
import pulumi_kubernetes as k8s

# Step 1: Define a Deployment for your AI model serving application.
# Replace `container_image` with the image of your AI model serving app.
app_labels = {"app": "ai-model-serving"}
app_deployment = k8s.apps.v1.Deployment(
    "ai-model-serving-deployment",
    spec=k8s.apps.v1.DeploymentSpecArgs(
        selector=k8s.meta.v1.LabelSelectorArgs(match_labels=app_labels),
        replicas=1,
        template=k8s.core.v1.PodTemplateSpecArgs(
            metadata=k8s.meta.v1.ObjectMetaArgs(labels=app_labels),
            spec=k8s.core.v1.PodSpecArgs(
                containers=[k8s.core.v1.ContainerArgs(
                    name="ai-model-serving-container",
                    image="container_image"  # TODO: Replace with your container image
                )],
            ),
        ),
    )
)

# Step 2: Create a Service to expose the model serving application.
model_service = k8s.core.v1.Service(
    "ai-model-service",
    spec=k8s.core.v1.ServiceSpecArgs(
        selector=app_labels,
        ports=[k8s.core.v1.ServicePortArgs(
            port=80,
            target_port=8000  # Replace 8000 with the port your app serves traffic on
        )],
    )
)

# Step 3: Install the NGINX Ingress Controller using the Pulumi resource.
# In a real-world scenario, you might install it using Helm, or Pulumi's HelmRelease.
# Here we assume the Ingress Controller is already installed and focus on the Ingress resource.

# Step 4: Create an Ingress resource to define routing rules.
ai_model_ingress = k8s.networking.v1.Ingress(
    "ai-model-ingress",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        annotations={"kubernetes.io/ingress.class": "nginx"}
    ),
    spec=k8s.networking.v1.IngressSpecArgs(
        rules=[k8s.networking.v1.IngressRuleArgs(
            http=k8s.networking.v1.HTTPIngressRuleValueArgs(
                paths=[k8s.networking.v1.HTTPIngressPathArgs(
                    path="/model",  # Path to access the model serving endpoint
                    path_type="Prefix",
                    backend=k8s.networking.v1.IngressBackendArgs(
                        service=k8s.networking.v1.IngressServiceBackendArgs(
                            name=model_service.metadata.name,
                            port=k8s.networking.v1.ServiceBackendPortArgs(
                                number=80
                            )
                        )
                    )
                )]
            )
        )]
    )
)

# Export the model serving endpoint URL
pulumi.export('model_serving_url', ai_model_ingress.status.apply(
    lambda status: status.load_balancer.ingress[0].hostname if status.load_balancer else "Ingress not yet allocated"
))
```

Explanation:

1. **Deployment**: You create a Kubernetes Deployment to manage your pods. These pods run the containerized AI model serving application. You must replace `container_image` with the image URL of your actual application.

2. **Service**: This Service makes your pods accessible within the Kubernetes cluster. It defines a port mapping; external traffic to port 80 is directed to the target port where your pods are serving traffic (in this case, port 8000 is used as an example).

3. **NGINX Ingress Controller**: A controller that uses NGINX as a reverse proxy and load balancer. It must be installed in your cluster separately, perhaps using a managed solution from your cloud provider or via a tool like Helm.

4. **Ingress Resource**: The Ingress resource defines how external HTTP(S) traffic should access services within your cluster. Here, traffic coming to the path `/model` will be routed to the Service created in step 2.

Once this code is run using Pulumi, the `model_serving_url` output will give you the URL to access your AI model serving application, although be aware it could take some time for your cloud provider to allocate a public endpoint.

Always make sure to configure Pulumi with the right Kubernetes context or credentials before attempting to deploy this code.