Pulumi Blog: Gke

AI/ML on Kubernetes: Deploying Models with Pulumi on Google Cloud

Sara Huddleston — Mon, 24 Mar 2025 07:32:19 +0000

Kubernetes has transformed cloud infrastructure by enabling scalable, containerized applications. While it initially gained traction for managing web applications and microservices, its capabilities now extend to AI/ML workloads, making it the go-to platform for data scientists and machine learning engineers.

Running AI/ML workloads on Kubernetes presents unique challenges, including:

Specialized hardware requirements (e.g., GPUs, TPUs)
Scalability for model training and inference
Complex data pipelines that integrate various cloud services
Infrastructure automation for seamless deployment

Google Cloud Kubernetes (GKE) provides a robust foundation for AI/ML workloads, but managing infrastructure manually can be cumbersome. This is where Pulumi comes in—enabling Infrastructure as Code (IaC) to automate and simplify AI/ML infrastructure on Kubernetes.

Pulumi: Automating AI/ML Infrastructure on Google Cloud

Pulumi is a modern Infrastructure as Code (IaC) tool that allows teams to define and manage cloud infrastructure using general-purpose programming languages like Python, TypeScript, and Go. This approach is particularly beneficial for AI/ML teams, as Python is already the dominant language in data science and machine learning.

With Pulumi, you can:

Provision and scale Kubernetes clusters on Google Cloud automatically.
Define AI/ML environments as code, making deployments repeatable and version-controlled.
Integrate infrastructure with machine learning pipelines, reducing operational overhead.

Deploying AI/ML Workloads on Kubernetes

Below, we explore two use cases for running AI/ML workloads on Google Kubernetes Engine (GKE) using Pulumi.

Use Case 1: Deploying a Large Language Model (LLM) with Retrieval Augmented Generation (RAG)

Large Language Models (LLMs) like GPT-3, Whisper, and DALL-E require significant infrastructure for training and inference. Retrieval Augmented Generation (RAG) enhances LLMs by integrating external knowledge sources, improving accuracy and relevance.

Using Pulumi, you can automate the deployment of an open-source LLM with RAG on Kubernetes.

Step 1: Set Up a Kubernetes Cluster on Google Cloud

import pulumi
import pulumi_gcp as gcp

# Create a GKE cluster for AI/ML workloads
cluster = gcp.container.Cluster("ml-cluster",
 location="us-central1",
 initial_node_count=3,
 node_version="1.23",
 min_master_version="1.23")

# Create a node pool optimized for ML workloads
node_pool = gcp.container.NodePool("ml-node-pool",
 cluster=cluster.name,
 node_config=gcp.container.NodePoolNodeConfigArgs(
 machine_type="n1-standard-4",
 oauth_scopes=[
 "https://www.googleapis.com/auth/devstorage.read_only",
 "https://www.googleapis.com/auth/logging.write",
 ],
 labels={"team": "ml"},
 shielded_instance_config=gcp.container.NodePoolNodeConfigShieldedInstanceConfigArgs(
 enable_secure_boot=True,
 enable_integrity_monitoring=True,
 ),
 ),
 initial_node_count=2,
 location="us-central1",
 version="1.23")
...

This Pulumi script:

Provisions a Kubernetes cluster on Google Cloud.
Creates a dedicated node pool optimized for AI/ML workloads.
Ensures security best practices for machine learning environments.

Step 2: Deploy the LLM with RAG Model

Once the cluster is set up, we can deploy the LLM with RAG model using Pulumi’s Kubernetes provider:

Define Kubernetes Deployments for the LLM and RAG model.
Package models as Docker containers and deploy them to the cluster.
Configure Kubernetes Services to expose APIs for model inference.
Use ConfigMaps and Secrets to manage parameters and credentials.

By defining these resources in Pulumi, deployments become fully automated, repeatable, and scalable.

Use Case 2: Training and Serving Custom Machine Learning Models

Beyond pre-trained LLMs, Kubernetes is ideal for training and serving custom AI/ML models. Pulumi can help automate every stage of the ML lifecycle.

Step 1: Set Up the Model Training Environment

Using Pulumi, we define a training environment with:

A Kubernetes Deployment for training jobs (GPU-enabled).
Persistent Volume Claims (PVCs) for storing training data and model artifacts.
Monitoring tools for tracking performance.

Step 2: Deploy and Serve the Trained Model

Once the model is trained, Pulumi can be used to:

Deploy the trained model as a Kubernetes Deployment.
Expose the model via a Kubernetes Service (REST API or gRPC).
Add autoscaling rules for dynamic inference scaling.

Pulumi allows teams to manage the entire AI/ML pipeline in a structured and automated way.

Try Jay’s demo code on Creating an AI Training Platform on GKE with Pulumi.

Why Use Pulumi for AI/ML on Kubernetes?

Pulumi provides several advantages for AI/ML teams running workloads on Kubernetes:

1. Use General-Purpose Languages for Infrastructure as Code

Most AI/ML engineers already work with Python or Go, and Pulumi lets them manage infrastructure using the same language.
No need to learn YAML or Kubernetes manifests—define everything programmatically.

2. Automate AI/ML Workflows

Define infrastructure, training jobs, and model serving in one unified IaC framework.
Ensure consistency across development, staging, and production environments.

3. Improve Scalability and Cost Efficiency

Pulumi integrates with Google Cloud AI services for optimized compute resources.
Automate autoscaling and resource allocation for AI/ML workloads.

4. Increase Security and Compliance

Manage credentials and secrets securely with Pulumi ESC (Secrets Management).
Apply policy-as-code to enforce security best practices.

Get Started with AI/ML on Kubernetes with Pulumi

Pulumi makes it easy to deploy, scale, and manage AI/ML workloads on Kubernetes, leveraging Google Cloud’s AI infrastructure. Whether you’re serving LLMs, training custom models, or automating ML pipelines, Pulumi provides a developer-friendly, scalable, and secure solution.

By combining Kubernetes, Google Cloud, and Pulumi, you can accelerate AI/ML innovation while reducing infrastructure complexity.

Multicloud Kubernetes: Running Apps Across EKS, AKS, and GKE

Mike Metral — Wed, 14 Aug 2019 00:00:00 +0000

Kubernetes clusters from the managed platforms of AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and GCP Google Kubernetes Engine (GKE) all vary in configuration, management, and resource properties. This variance creates unnecessary complexity in cluster provisioning and application deployments, as well as for CI/CD and testing.

Additionally, if you wanted to deploy the same app across multiple clusters for specific use cases or test scenarios across providers, subtleties such as LoadBalancer outputs and cluster connection settings can be a nuisance to manage.

In this post, we’ll see how to use Pulumi to deploy the kuard app across EKS, AKS, GKE and a local Kubernetes cluster, such as Docker Desktop or a self-managed cluster. We’ll spin up the clusters in each provider, launch the app, and manage both cluster and app using the TypeScript programming language.

View the full example and code.

![](multicloud.png)

Cluster Provisioning

Provisioning Kubernetes clusters and their IaaS resources is made simple through Pulumi’s various SDKs for the cloud providers:

Crosswalk for AWS further allows us to leverage the Pulumi libraries of common infrastructure for AWS to simplify cloud resource instantiation and management while gaining best-practices as defaults. Check out the pulumi/awsx SDK to get started.

For local clusters such as those that are self-managed, or provisioned by a tool like Docker Desktop, Pulumi can still deploy workloads to these these systems given that the pulumi/kubernetes workload SDK only requires a valid kubeconfig file. For more information on Pulumi’s Kubernetes support, check out the Kubernetes reference page.

We will use the cloud SDKs to provision the managed Kubernetes clusters. Given that we’re working with real code, we are afforded developer benefits such as: code linting, type checking, IDE hints and completion, abstractions and inheritance.

Leveraging these development features creates the opportunity to encapsulate the finer-grained details and settings, and expose the capability to create clusters as simple as the following code:

Workload Deployment

Once the clusters are provisioned, we can leverage the pulumi/kubernetes SDK to manage the Kubernetes workloads that will be deployed into the clusters.

The pulumi/kubernetes SDK uses the official Kubernetes client-go library to interact with Kubernetes. Therefore, Pulumi can work pretty much anywhere kubectl works, even if Pulumi was not used to create the cluster.

Summary

As shown in the code samples, it becomes relatively easy to provision and manage Kubernetes clusters across multiple clouds, as well as deploy workloads to the cluster regardless if they are managed by a cloud provider, or self-managed.

The various SDKS allow you to leverage industry standard best-practices and defaults, in addition to allowing you to further configure and customize how your clusters and apps are managed.

Testing apps across various providers in this form allows you to abstract away the details of provider specific implementations, and focus on how your app operates in the various contexts.

Learn More

If you’d like to learn about Pulumi and how to manage your infrastructure and Kubernetes multi-cloud capabilities through code, get started today. Pulumi is open source and free to use.

For further examples on how to use Pulumi to create Kubernetes clusters, or deploy workloads to a cluster, check out the rest of the Kubernetes tutorials.

As always, you can check out our code on GitHub, follow us on Twitter, subscribe to our YouTube channel, or join our Community Slack channel if you have any questions, need support, or just want to say hello.

If you’d like to chat with our team, or get hands-on assistance with migrating your existing configuration code to Pulumi, please don’t hesitate to drop us a line.

We also encourage you to watch Pulumi team member Levi Blackstone demo this post in an episode of the Kubernetes Community Meeting.

Create Secure Jupyter Notebooks on Kubernetes using Pulumi

Nishi Davidson — Thu, 30 May 2019 00:00:00 +0000

In this post, we will work through an example that shows how to use Pulumi to create Jupyter Notebooks on Kubernetes. Having worked on Kubernetes since 2015, a couple of critical benefits jump out that may resonate with you as well:

You write everything in code - TypeScript in our example here.
You need not initialize Tiller or Helm to work with existing Helm charts like nginx-ingress-controller that we use here.
The security patterns in Helm and Tiller are no longer concerns, rather you get to focus on the RBAC of the actual service which is Jupyter-notebook in this example.
You accomplish more with less YAML and iteratively work towards your use cases.

Prerequisites

Install Pulumi
Install Node.js version 6 or later
Install a package manager for Node.js, such as npm or Yarn
Install Google Cloud SDK
Install Helm and only initialize with helm init —client-only

We will work this example on a GKE cluster so lets first configure Google Cloud Auth:

$ gcloud auth login
$ gcloud config set project <YOUR_GCP_PROJECT_HERE>
$ gcloud auth application-default login

Step 1: Create a Pulumi Project and Stack with a Pulumi TypeScript Template

$ mkdir gke-jupyter-notebook && cd gke-jupyter-notebook
$ pulumi new typescript
$ npm install --save @pulumi/kubernetes @pulumi/gcp

Step 2: Create a GKE Cluster

To create a GKE cluster, simply update the following code in index.ts file and run pulumi up.

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import { readFileSync, fstat } from "fs";

const name = "jupyter-notebook";

/*
 * STEP 2: Create a GKE Cluster
 */
const engineVersion = gcp.container.getEngineVersions().then(v => v.latestMasterVersion);
const cluster = new gcp.container.Cluster(name, {
 initialNodeCount: 2,
 minMasterVersion: engineVersion,
 nodeVersion: engineVersion,
 nodeConfig: {
 machineType: "n1-standard-1",
 oauthScopes: [
 "https://www.googleapis.com/auth/compute",
 "https://www.googleapis.com/auth/devstorage.read_only",
 "https://www.googleapis.com/auth/logging.write",
 "https://www.googleapis.com/auth/monitoring"
 ],
 },
});

// Export the Cluster name
export const clusterName = cluster.name;

// Manufacture a GKE-style kubeconfig.
export const kubeconfig = pulumi.
 all([ cluster.name, cluster.endpoint, cluster.masterAuth ]).
 apply(([ name, endpoint, masterAuth ]) => {
 const context = `${gcp.config.project}_${gcp.config.zone}_${name}`;
 return `apiVersion: v1
clusters:
- cluster:
 certificate-authority-data: ${masterAuth.clusterCaCertificate}
 server: https://${endpoint}
 name: ${context}
contexts:
- context:
 cluster: ${context}
 user: ${context}
 name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
 user:
 auth-provider:
 config:
 cmd-args: config config-helper --format=json
 cmd-path: gcloud
 expiry-key: '{.credential.token_expiry}'
 token-key: '{.credential.access_token}'
 name: gcp
`;
 });

// Create a Kubernetes provider instance that uses our cluster from above.
const clusterProvider = new k8s.Provider(name, { kubeconfig: kubeconfig });

Step 3: Create a NGINX-Ingress-Controller to Generate Ingresses

You can use the default L7 load balancer in Google Cloud but here we add an NGINX-Ingress-Controller to the cluster. With Pulumi, you write four lines of typescript to reuse the stable NGINX ingress controller helm chart and have the controller, default backends, RBAC, Service account, Config map all running within seconds.

Add the following lines of code in index.ts file and run pulumi up once again.

/*
 * STEP 3: Create NGINX Ingress Controller in GKE
 */

const nginxingresscntlr = new k8s.helm.v2.Chart("nginxingresscontroller", {
 repo: "stable",
 chart: "nginx-ingress",
 version: "0.24.1",
 values: {},
}, { providers: { kubernetes: clusterProvider } });

Step 4: Create a Jupyter Notebook Deployment and Service with Type NodePort

Bringing up Jupyter notebook deployment and service requires adding the following lines of code in index.ts file and running pulumi up to apply the changes.

/*
 * STEP 4: Create Jupyter notebook deployment and service in the GKE cluster
 */

const appName = "jupyter-notebook";
const appLabels = { app: appName };
const jupyterNotebook = new k8s.apps.v1beta1.Deployment(appName, {
 metadata: { name: appName, labels: appLabels },
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [
 {
 name: appName,
 image: "jupyter/tensorflow-notebook",
 ports: [{ containerPort: 8888 }],
 command: ["start-notebook.sh"],
 args: ["--NotebookApp.token=''"]
 }]
 }
 }
 }
}, { provider: clusterProvider });

const jupyterService = new k8s.core.v1.Service(appName, {
 metadata: {
 name: appName,
 labels: appLabels
 },
 spec: {
 type: "NodePort",
 selector: appLabels,
 ports: [{ protocol: "TCP", nodePort: 30040, port: 8888, targetPort: 8888 }],
 }
}, { provider: clusterProvider });

Step 5: Create a Secret that is used with your Jupyter Notebook Domain Name

We first create a local auth.txt file with the password using the following command:

htpasswd -c auth.txt jupyter

We then read this file synchronously, convert it to base64 and add it as a secret in the GKE cluster. We use this secret as the TLS password to access the Jupyter notebook ingress endpoint accessible from the domain name defined in the host section of the ingress declaration. The annotations in the ingress declarations are required to enable this behavior on the ingress object.

/*
 * STEP 5: Create a secret to enable "basic-auth" for your Jupyter
 * notebook ingress and add it to the ingress declaration in the
 * GKE cluster
 */

const authContents = (readFileSync("<path-to-auth.txt-file>")).toString()

function toBase64(s: string): string {
 return Buffer.from(s).toString("base64");
}

const authContents_base64 = toBase64(authContents);

const jupyternotebooksecret = new k8s.core.v1.Secret("jupyter-notebook-tls", {
 metadata: { name: "basic-auth", namespace: "default" },
 type: "Opaque",
 data: {
 auth: authContents_base64,
 }
})

export const jupyternotebookingress = new k8s.extensions.v1beta1.Ingress(appName, {
 metadata: {
 name: appName,
 labels: appLabels,
 annotations: {
 "kubernetes.io/tls-acme": "true",
 "kubernetes.io/ingress.class": "nginx",
 "nginx.ingress.kubernetes.io/auth-type": "basic",
 "nginx.ingress.kubernetes.io/auth-secret": "basic-auth",
 "nginx.ingress.kubernetes.io/auth-realm": "Authentication Required - jupyter"
 },
 },
 spec: {
 rules: [{
 host: "nishidavidson.com",
 http: {
 paths: [{
 path: "/",
 backend: { serviceName: "jupyter-notebook", servicePort: 8888 }
 }]
 }
 }],
 tls: [{
 secretName: "jupyter-notebook-tls",
 hosts: ["nishidavidson.com"]
 }],
 }
}, { provider: clusterProvider });

Once complete, you will see the GKE cluster components show up as follows:

Open a browser to access the jupyter-notebook Service using the domain name declared in the host section of your ingress code above. In our example, I used the domain name nishidavidson.com.

When you open your DNS, you will be asked for the “username: jupyter” and “password” for the secret you created in auth.txt file as shown below:

As soon as the password is accepted, You should now be able to go to your website from anywhere in the world and access your password-protected Jupyter notebook running on GKE on secure SSL connection.

Success! We worked through a simple example of creating a GKE cluster, an NGINX ingress controller and stood up our password protected Jupyter notebook Ingress, Service, and Deployment with a simple secret for authentication.

You can get started today using more solutions on Kubernetes from our Pulumi examples repository.