Using kubernetes karpenter.sh with kustomize.toolkit.fluxcd.io

Question

Pulumi · Accepted Answer

Karpenter is an open-source, flexible, high-performance Kubernetes cluster autoscaler built by AWS that can run in any Kubernetes cluster regardless of the cloud platform. Karpenter works by observing the aggregate resource request of unscheduled pods and making decisions to launch or terminate nodes to minimize scheduling latencies.

Flux is a set of continuous and progressive delivery solutions for Kubernetes, and is part of the CNCF incubating projects. The Flux CDK (Continuous Delivery Kubernetes) provides a set of APIs and controllers to deploy applications and keep Kubernetes clusters in sync with sources of configuration (like Git repositories).

In your case, you want to combine Karpenter with Kustomize (part of the Flux toolkit). Kustomize is a standalone tool to customize Kubernetes objects through a file called kustomization.yaml, and Flu CD integrates tightly with it.

Below is a Pulumi program that demonstrates how you might set up Karpenter along with Flux and Kustomize in a Kubernetes cluster using TypeScript. Please note that the code assumes you have an existing Kubernetes cluster and appropriate cloud provider credentials are configured for Pulumi.

import * as kubernetes from "@pulumi/kubernetes";

// Create a Flux Kustomization resource.
const fluxKustomization = new kubernetes.apiextensions.CustomResource("flux-kustomization", {
    apiVersion: "kustomize.toolkit.fluxcd.io/v1beta1",
    kind: "Kustomization",
    metadata: { 
        name: "my-kustomization",
        namespace: "flux-system" // Assuming you have a 'flux-system' namespace.
    },
    spec: {
        interval: "10m",
        path: "./deployments/production", // Path to the directory containing Kustomize files within the Git repo.
        prune: true,
        sourceRef: {
            kind: "GitRepository",
            name: "flux-system", // A GitRepository source object name for your repository.
        },
        validation: "client", // Or set to 'server' for server-side validation.
    }
});

// Create a Karpenter IAM Role for a cluster named `my-cluster` (you'd change accordingly)
// Note: This step would generally require more setup than shown,
// including creating an IAM role and providing appropriate permissions.
const karpenterNodeRole = new kubernetes.core.v1.ServiceAccount("karpenter-node-role", {
    metadata: {
        name: "karpenter",
        namespace: "karpenter" // Assuming you have a 'karpenter' namespace.
    }
});

// Deploy Karpenter to the cluster
const karpenterChart = new kubernetes.helm.v3.Chart("karpenter", {
    repo: "https://charts.karpenter.sh",
    chart: "karpenter",
    namespace: "karpenter",
    values: {
        serviceAccount: {
            create: false, // We created the ServiceAccount separately above.
            name: "karpenter",
        },
        clusterName: "my-cluster", // Name of your existing cluster.
    },
});

// Export the Flux Kustomization URL to access your configuration.
export const fluxKustomizationUrl = fluxKustomization.metadata.name.apply(name => 
    `https://github.com/<your-org>/<your-repo>/tree/master/${name}`
);

// Export the name of the Karpenter node role for reference.
export const karpenterNodeRoleName = karpenterNodeRole.metadata.name;

This code does the following:

It defines a Flux Kustomization resource, which specifies how your Kubernetes cluster interacts with configurations stored in a Git repository.
It simulates the setup of a Karpenter node IAM role; however, in a real-world scenario you would have to configure IAM roles with the necessary permissions.
It uses the helm.v3.Chart class to deploy the Karpenter Helm chart to the Kubernetes cluster, ensuring that the service account karpenter is used for the Karpenter pod.

The exported URLs at the bottom will give you direct links to view your configurations once applied:

fluxKustomizationUrl provides a link to view the flux kustomization configuration in your Git repository.
karpenterNodeRoleName exports the name of the Karpenter node role for your reference.

Before running this code with Pulumi, ensure you have Helm and the Kubernetes Pulumi provider configured correctly. You'll also need to adapt the exact values and configurations according to your setup, particularly for the IAM role and Git repository references.