How do I run LLM inference workloads on Kubernetes Core?

In this guide, we will deploy an LLM (Large Language Model) inference workload on a Kubernetes cluster. This involves creating a Kubernetes deployment and service to run a containerized LLM model. We will use Pulumi to define and manage our Kubernetes resources.

Key Points

Kubernetes Deployment: Defines the LLM inference workload as a set of replicated pods.
Kubernetes Service: Exposes the deployment to external traffic, allowing inference requests to reach the model.

Detailed Steps

Create a Kubernetes Deployment: This will manage the pods running the LLM model.
Create a Kubernetes Service: This will expose the deployment, allowing external access to the model for inference.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Define the namespace
const namespace = new k8s.core.v1.Namespace("llm-inference-namespace", {
    metadata: { name: "llm-inference" },
});

// Define the deployment for the LLM model
const llmDeployment = new k8s.apps.v1.Deployment("llm-deployment", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "llm-deployment",
    },
    spec: {
        replicas: 3, // Number of pod replicas
        selector: { matchLabels: { app: "llm-model" } },
        template: {
            metadata: { labels: { app: "llm-model" } },
            spec: {
                containers: [{
                    name: "llm-container",
                    image: "your-llm-model-image:latest", // Replace with your LLM model image
                    ports: [{ containerPort: 80 }],
                    resources: {
                        requests: {
                            cpu: "500m",
                            memory: "1Gi",
                        },
                        limits: {
                            cpu: "1",
                            memory: "2Gi",
                        },
                    },
                }],
            },
        },
    },
});

// Define the service to expose the deployment
const llmService = new k8s.core.v1.Service("llm-service", {
    metadata: {
        namespace: namespace.metadata.name,
        name: "llm-service",
    },
    spec: {
        type: "LoadBalancer",
        selector: { app: "llm-model" },
        ports: [{ port: 80, targetPort: 80 }],
    },
});

// Export the service's IP address
export const serviceIP = llmService.status.loadBalancer.ingress[0].ip;

Conclusion

In this example, we created a Kubernetes namespace, deployment, and service using Pulumi. The deployment manages the pods running the LLM model, while the service exposes these pods to external traffic, allowing inference requests to be processed. This setup ensures that the LLM model is highly available and scalable within the Kubernetes cluster.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.