Version-controlled ML Model Serving on Kubernetes

Question

Pulumi · Accepted Answer

To serve a machine learning (ML) model on Kubernetes with version control, you'll want to package your ML model inside a Docker container, deploy it on a Kubernetes cluster, and manage different versions using Kubernetes' native resources along with the help of a registry that stores versions of your models.

Here is an overview of the process:

1. **Create a Docker Image:** Package your trained ML model into a Docker image. This image includes the model files and a web server (such as Flask, FastAPI, etc.) that defines endpoints to interact with the model.

2. **Push to a Registry:** Push the Docker image to a container registry such as Docker Hub, AWS ECR, Google GCR, or Azure Container Registry.

3. **Kubernetes Deployment:** Define a Kubernetes `Deployment` resource that specifies your Docker image and the desired number of replicas. Each `Deployment` is a version of your application.

4. **Kubernetes Service:** Expose your Deployment as a `Service` which provides a stable endpoint for model inference.

5. **Version Control:** Manage versions of your `Deployment` by applying updates to the Docker image tag and rolling out updates via Kubernetes. Use annotations or labels to tag deployments with version information, and automate this process with CI/CD pipelines.

6. **(Optional) Kubernetes Ingress:** Define an `Ingress` resource to manage external access to the ML model services, including routing and SSL termination.

Below is a Pulumi program in Python that outlines how to create these resources on Kubernetes:

```python
import pulumi
import pulumi_kubernetes as k8s

# Configuration for the Docker image of the ML model
# Replace 'your-docker-image' and 'your-image-tag' with your Docker image details
image_name = 'your-docker-image:latest'
model_service_name = 'model-service'

# Define the Kubernetes Deployment for the ML model
model_deployment = k8s.apps.v1.Deployment(
    "model-deployment",
    spec={
        "selector": {"matchLabels": {"app": model_service_name}},
        "replicas": 1,
        "template": {
            "metadata": {"labels": {"app": model_service_name}},
            "spec": {
                "containers": [{
                    "name": model_service_name,
                    "image": image_name,
                    "ports": [{"containerPort": 8080}],
                }],
            },
        },
    })

# Define the Kubernetes Service for stable networking
model_service = k8s.core.v1.Service(
    "model-service",
    spec={
        "selector": {"app": model_service_name},
        "ports": [{"port": 80, "targetPort": 8080}],
        "type": "LoadBalancer",
    })

# Export the model service endpoint for easy access
pulumi.export('model_service_endpoint', model_service.status.apply(lambda status: status['load_balancer']['ingress'][0]['ip']))
```

### Explanation:
- We import the necessary Pulumi libraries for Kubernetes interaction.
- We define the name and tag of your Docker image hosting the ML model.
- We create a Kubernetes Deployment named `model-deployment` using a single replica (which can be scaled as needed).
  - The Deployment specifies the Docker image and the container port (8080 in this case) where the model serving API is running.
- We create a Kubernetes Service named `model-service`, which will be of type `LoadBalancer` for this example, to expose our ML model outside of the Kubernetes cluster on port 80 mapping to the target container port 8080.
- Finally, we export the service endpoint, so that you can easily access the ML model's API once it's running.

Make sure to replace `'your-docker-image:latest'` with your actual Docker image repository and tag, and ensure the ports are correct for your application.

For version control, whenever you have a new version of the model you want to deploy:
- Build and push a new Docker image with a different tag to your container registry.
- Update the `image_name` variable in the Pulumi program with the new Docker image tag.
- Re-run `pulumi up` to perform a rolling update of your service.

Kubernetes will manage the release of the new version without downtime. By integrating this process into a CI/CD system, every time you push a new image tag, the Pulumi program can be automatically updated and applied, effectively giving you version control of your model serving.