Secure LLM Access with Kubernetes Ingress TLS

Question

Pulumi · Accepted Answer

To secure LLM (Language Model, such as the one we are using right now) access with Kubernetes, you can set up an Ingress resource that uses TLS to encrypt the connections to the backend services. You will need a Kubernetes cluster with an Ingress controller installed (such as nginx or Traefik), a domain name with DNS set up to point to your Ingress controller's external IP, and a TLS certificate you can use for encryption.

In this example, I'll guide you through setting up an Ingress resource that routes traffic to a hypothetical LLM service running on Kubernetes. To make the connection secure, we'll use a TLS certificate. A common way to manage TLS certificates in Kubernetes is cert-manager, a tool that automates the management and issuance of TLS certificates from various issuing sources. It will ensure that certificates are valid and up to date, and will attempt to renew certificates at a configured time before expiration.

You will also need to create a Secret resource that contains your TLS certificate and private key. Here is a Pulumi program written in Python that sets up:

1. An Ingress resource configured to provide secure access over HTTPS by using a TLS certificate.
2. A cert-manager Issuer resource, a Kubernetes resource that represents a certificate authority from which to obtain TLS certificates. This is optional if you already have a certificate ready to use.
3. A Secret resource that will hold your TLS certificate.

Ensure you have the necessary Pulumi and Kubernetes setup on your local machine before running this program.

```python
import pulumi
import pulumi_kubernetes as k8s
from pulumi_kubernetes.networking.v1 import Ingress

# Set up the provider for the Kubernetes cluster if it is not configured
# with the current context of kubectl; otherwise, this is not necessary.
# provider = k8s.Provider("provider", kubeconfig="your-kubeconfig-file")

# Create a Kubernetes Secret for the TLS certificate. This assumes you have
# certificate files available; if not, you might use cert-manager to obtain them.
tls_secret = k8s.core.v1.Secret(
    "tls-secret",
    metadata={
        "name": "llm-tls-cert"
    },
    type="kubernetes.io/tls",
    data={
        "tls.crt": "base64-encoded-certificate",  # Replace with your base64-encoded TLS certificate
        "tls.key": "base64-encoded-key"           # Replace with your base64-encoded private key
    }
    # Uncomment the following line if you are using a custom provider
    # , opts=pulumi.ResourceOptions(provider=provider)
)

# Define an Ingress to manage external access to services in a cluster
# via HTTP/HTTPS using TLS termination on the Ingress. 
ingress = Ingress(
    "llm-ingress",
    metadata={
        "name": "llm-ingress",
        "annotations": {
            # Add any necessary annotations here; e.g.,
            # for nginx-ingress controller or cert-manager
            "kubernetes.io/ingress.class": "nginx"
            # "cert-manager.io/cluster-issuer": "letsencrypt-prod" (if using cert-manager)
        }
    },
    spec={
        "tls": [
            {
                "hosts": ["your-domain.com"],  # Replace with your domain
                "secretName": tls_secret.metadata["name"]
            }
        ],
        "rules": [
            {
                "host": "your-domain.com",  # Replace with your domain
                "http": {
                    "paths": [
                        {
                            "path": "/",
                            "pathType": "Prefix",
                            "backend": {
                                "service": {
                                    "name": "llm-service",  # Name of your LLM service
                                    "port": {
                                        "number": 443  # Port exposed by your LLM service
                                    }
                                }
                            },
                        }
                    ]
                }
            }
        ]
    }
    # Uncomment the following line if you are using a custom provider
    # , opts=pulumi.ResourceOptions(provider=provider)
)

# Export the Ingress endpoint
pulumi.export("ingress_endpoint", ingress.metadata["name"])
```

In this program:
- We define a `Secret` to store the TLS certificate and key. This Secret will be used by the Ingress to serve your domain over HTTPS securely.
- We create an Ingress resource with TLS enabled, specifying our `Secret` in the `tls` section. The rules in the Ingress tell Kubernetes how to route the traffic to your LLM application service.
- Annotations can be applied to the Ingress to customize behavior depending on your Ingress controller. Here, I've included a placeholder for where these annotations would go.

Before running the above program, make sure to replace the placeholder values with actual values suitable for your setup, such as:
- `your-kubeconfig-file` with the path to your kubeconfig if you're using a custom provider.
- `base64-encoded-certificate` with the actual base64-encoded contents of your TLS certificate for your domain.
- `base64-encoded-key` with the actual base64-encoded contents of your TLS key for your domain.
- `your-domain.com` with the domain you have configured for your LLM service.
- `llm-service` and `443` with the actual name and port of your LLM application service within your Kubernetes cluster.

Please, be mindful of securing your key file and certificate, handle them securely, and do not expose them in version control.

Running this program with `pulumi up` will apply these resources to your Kubernetes cluster. Ensure you have the proper permissions to create and manage these resources in the cluster.

After successfully applying, your LLM application will be accessible securely via HTTPS using the domain you configured, with traffic routed through the Kubernetes Ingress controller with TLS termination.