Using kubernetes karpenter.sh with cloud.google.com

Question

Pulumi · Accepted Answer

To use Kubernetes [karpenter.sh](https://karpenter.sh/) with Google Cloud, you will need to have a Kubernetes cluster running on Google Kubernetes Engine (GKE). Karpenter is an open-source project designed to automatically provision new nodes in your Kubernetes cluster when needed. It aims to optimize resource allocation and can quickly scale up or down depending on the workload.

Below is a step-by-step guide and Pulumi TypeScript program that will:

1. Create a GKE cluster on Google Cloud.
2. Assume you have configured `kubectl` to interact with your GKE cluster and you can install Karpenter on it outside of Pulumi.

### Detailed Explanation:

Before we can install Karpenter, we need a running Kubernetes cluster. In Google Cloud, this is typically done using Google Kubernetes Engine (GKE). With Pulumi, we can define infrastructure as code using TypeScript to provision a GKE cluster. Here's how we will achieve our goal:

- Import the necessary Google Cloud Pulumi package to create a GKE cluster.
- Set up the configuration for the GKE cluster, like the machine type, number of nodes, etc.
- Create a new GKE cluster resource using Pulumi.
- Export any necessary outputs, such as the cluster name or kubeconfig, that you will need to interact with the cluster.

After running this Pulumi program and successfully creating a GKE cluster, you will then apply the Karpenter manifests to your cluster using `kubectl`. The Pulumi program does not directly handle Karpenter's installation as it's a Kubernetes workload that should be managed separately from infrastructure provisioning, typically by a separate continuous deployment flow or manually with `kubectl`.

Now, let's create the GKE cluster with Pulumi:

```typescript
import * as gcp from "@pulumi/gcp";

// Create a GKE cluster
const cluster = new gcp.container.Cluster("karpenter-cluster", {
    initialNodeCount: 3,
    minMasterVersion: "latest",
    nodeVersion: "latest",
    nodeConfig: {
        machineType: "e2-medium",
        preemptible: true,
        oauthScopes: [
            "https://www.googleapis.com/auth/compute",
            "https://www.googleapis.com/auth/devstorage.read_only",
            "https://www.googleapis.com/auth/logging.write",
            "https://www.googleapis.com/auth/monitoring"
        ],
    },
});

// Export the Cluster name
export const clusterName = cluster.name;

// Additionally, you can export the kubeconfig to interact with your cluster
const kubeconfig = cluster.endpoint.apply(endpoint =>
    `apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${cluster.masterAuth[0].clusterCaCertificate}
    server: https://${endpoint}
  name: gke-cluster
contexts:
- context:
    cluster: gke-cluster
    user: gke-admin
  name: gke-context
current-context: gke-context
kind: Config
preferences: {}
users:
- name: gke-admin
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp`
);

// Export the kubeconfig
export const k8sConfig = kubeconfig;

// After running `pulumi up`, you'll need to setup kubectl with the kubeconfig of the newly created cluster:
// ```
// pulumi stack output k8sConfig --show-secrets > kubeconfig.yaml
// export KUBECONFIG=./kubeconfig.yaml
// ```

// Then install Karpenter following the instructions at https://karpenter.sh/docs/getting-started/
```

This program defines a new cluster with three nodes of `e2-medium` type, which are instances suitable for small to medium workloads and are also preemptible, meaning they can be cheaper but less reliable as they can be terminated if Google needs the capacity back.

The `oauthScopes` allow the nodes to interact with Google Cloud services such as compute, storage, logging, and monitoring.

After creating the cluster with `pulumi up`, the program outputs the cluster's name and kubeconfig, which you'll use to configure `kubectl`. This allows you to interact with your cluster and install Karpenter or any other Kubernetes resources.

Lastly, follow the [Karpenter installation guide](https://karpenter.sh/docs/getting-started/) to install Karpenter on your GKE cluster. This involves applying Kubernetes manifests to your cluster, which will set up Karpenter's custom resources and controllers.

Keep in mind that node provisioning by Karpenter in your cluster requires additional setup, like IAM role configuration and installation of the Karpenter controller. Ensure you follow the Karpenter documentation carefully to set up these components.