Introducing the Pulumi Kubernetes Operator

Posted on

Kubernetes developers and operators work together to manage workloads and to continuously ship software through CI/CD. These users have an affinity for automation and pipelines, and richer integration with Kubernetes is a growing theme across the cloud native ecosystem.

We’re excited to introduce the Pulumi Kubernetes Operator: a Kubernetes controller that deploys cloud infrastructure in Pulumi Stacks for you and your team.

These program stacks include virtual machines, block storage, managed Kubernetes clusters, API resources, serverless functions and more!

Overview

The Pulumi Kubernetes Operator is an extension pattern that enables Kubernetes users to create a Pulumi Stack as a first-class API resource, and use the StackController to drive the updates of the Stack until success.

Check out how to deploy the operator to a Kubernetes cluster using any of Pulumi’s supported languages, or through YAML manifests and kubectl.

You can also get started with Pulumi and create a new managed Kubernetes cluster on Amazon EKS, Google GKE, or Azure AKS if you don’t have an existing cluster.

How it Works

When the operator is deployed, the StackController waits for Stack CustomResources to be created, patched, or deleted in the cluster.

On these events, the Kubernetes controller invokes a reconciliation loop to process the request until it has reached success. For Stack creation or updates, this is a successful pulumi up. For deletions, it is a successful pulumi destroy and pulumi stack rm.

Kubernetes Reconciliation Loop

Stacks can be written in Typescript, Python, .NET, and Go, and span cloud providers such as Amazon AWS, Google GCP, and Microsoft Azure, as well as many cloud native services!

The following example showcases a typical Pulumi program in Typescript, that can be instantiated as a Stack.

import * as aws from "@pulumi/aws";

// Create an AWS resource (S3 Bucket)
const names = [];
for (let i = 0; i < 2; i++) {
    const bucket = new aws.s3.Bucket(`my-bucket-${i}`, {
        acl: "public-read",
    });
    names.push(bucket.id);
}

// Export the name of the buckets
export const bucketNames = names;

You can find more examples on GitHub of the types of infrastructure you can manage with Pulumi.

Check out Kubernetes Controllers work if you’d like to learn more.

Creating a Pulumi Stack in Kubernetes

The Stack CustomResourceDefinition (CRD) encapsulates an existing Pulumi infrastructure project in a Git repo, a specific commit SHA to deploy, and any additional settings to control the Pulumi update run.

These settings include your Pulumi API access token, environment variables, config and secrets, and lifecycle controls for the update.

Deploying an NGINX Stack on Kubernetes

Check out an example of creating a Kubernetes NGINX Deployment as a Stack by the Operator in its cluster.

Choose your preferred language below, or check out how to create Pulumi Stacks using kubectl.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

// Get the Pulumi API token.
const pulumiConfig = new pulumi.Config();
const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")

// Create the API token as a Kubernetes Secret.
const accessToken = new kx.Secret("accesstoken", {
    stringData: { accessToken: pulumiAccessToken },
});

// Create an NGINX deployment in-cluster.
const mystack = new k8s.apiextensions.CustomResource("my-stack", {
    apiVersion: 'pulumi.com/v1alpha1',
    kind: 'Stack',
    spec: {
        accessTokenSecret: accessToken.metadata.name,
        stack: "<YOUR_ORG>/k8s-nginx/dev",
        initOnCreate: true,
        projectRepo: "https://github.com/pulumi/examples",
        repoDir: "kubernetes-ts-nginx",
        commit: "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
        destroyOnFinalize: true,
    }
});
import pulumi
from pulumi_kubernetes import core, apiextensions

# Get the Pulumi API token.
pulumi_config = pulumi.Config()
pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")

# Create the API token as a Kubernetes Secret.
access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token })

# Create an NGINX deployment in-cluster.
my_stack = apiextensions.CustomResource("my-stack",
    api_version="pulumi.com/v1alpha1",
    kind="Stack",
    spec={
        "access_token_secret": access_token.metadata["name"],
        "stack": "<YOUR_ORG>/k8s-nginx/dev",
        "init_on_create": True,
        "project_repo": "https://github.com/pulumi/examples",
        "repo_dir: "kubernetes-ts-nginx",
        "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
        "destroy_on_finalize": True,
    }
)
using Pulumi;
using Pulumi.Kubernetes.ApiExtensions;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;

class StackArgs : CustomResourceArgs
{
    [Input("spec")]
    public Input<StackSpecArgs>? Spec { get; set; }

    public StackArgs() : base("pulumi.com/v1alpha1", "Stack")
    {
    }
}

class StackSpecArgs : ResourceArgs
{
    [Input("accessTokenSecret")]
    public Input<string>? AccessTokenSecret { get; set; }

    [Input("stack")]
    public Input<string>? Stack { get; set; }

    [Input("initOnCreate")]
    public Input<bool>? InitOnCreate { get; set; }

    [Input("projectRepo")]
    public Input<string>? ProjectRepo { get; set; }

    [Input("commit")]
    public Input<string>? Commit { get; set; }

    [Input("destroyOnFinalize")]
    public Input<bool>? DestroyOnFinalize { get; set; }
}

class MyStack : Stack
{
    public MyStack()
    {
        // Get the Pulumi API token.
        var config = new Config();
        var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");

        // Create the API token as a Kubernetes Secret.
        var accessToken = new Secret("accesstoken", new SecretArgs
        {
            StringData =
            {
                {"accessToken", pulumiAccessToken}
            }
        });

        // Create an NGINX deployment in-cluster.
        var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("nginx", new StackArgs
        {
            Spec = new StackSpecArgs
            {
                AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),
                Stack = "<YOUR_ORG>/k8s-nginx/dev",
                InitOnCreate = true,
                ProjectRepo = "https://github.com/pulumi/examples",
                RepoDir = "kubernetes-ts-nginx",
                Commit = "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
                DestroyOnFinalize = true,
            }
        });
    }
}
package main

import (
    "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes"
    apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/apiextensions"
    corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
    metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
    "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
    "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Get the Pulumi API token.
        c := config.New(ctx, "")
        pulumiAccessToken := c.Require("pulumiAccessToken")

        // Create the API token as a Kubernetes Secret.
        accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{
            StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)},
        })
        if err != nil {
            return err
        }

        // Create an NGINX deployment in-cluster.
        _, err = apiextensions.NewCustomResource(ctx, "my-stack", &apiextensions.CustomResourceArgs{
            ApiVersion: pulumi.String("pulumi.com/v1alpha1"),
            Kind:       pulumi.String("Stack"),
            OtherFields: kubernetes.UntypedArgs{
                "spec": map[string]interface{}{
                    "accessTokenSecret": accessToken.Metadata.Name(),
                    "stack":             "<YOUR_ORG>/k8s-nginx/dev",
                    "initOnCreate":      true,
                    "projectRepo":       "https://github.com/pulumi/examples",
                    "repoDir":           "kubernetes-ts-nginx",
                    "commit":            "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
                    "destroyOnFinalize": true,
                },
            },
        }, pulumi.DependsOn([]pulumi.Resource{accessToken}))
        return err
    })
}

Deploying an Amazon EKS Stack

Check out an example of creating a new AWS VPC and Kubernetes cluster on AWS EKS as a Stack by the Operator.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

// Get the Pulumi API token and AWS creds.
const pulumiConfig = new pulumi.Config();
const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")
const awsAccessKeyId = pulumiConfig.require("awsAccessKeyId")
const awsSecretAccessKey = pulumiConfig.requireSecret("awsSecretAccessKey")
const awsSessionToken = pulumiConfig.requireSecret("awsSessionToken")

// Create the creds as Kubernetes Secrets.
const accessToken = new kx.Secret("accesstoken", {
    stringData: { accessToken: pulumiAccessToken},
});
const awsCreds = new kx.Secret("aws-creds", {
	stringData: {
		"AWS_ACCESS_KEY_ID": awsAccessKeyId,
		"AWS_SECRET_ACCESS_KEY": awsSecretAccessKey,
		"AWS_SESSION_TOKEN": awsSessionToken,
	},
});

// Create a Kubernetes cluster on AWS EKS.
const mystack = new k8s.apiextensions.CustomResource("my-stack", {
    apiVersion: 'pulumi.com/v1alpha1',
    kind: 'Stack',
    spec: {
        stack: "<YOUR_ORG>/pulumi-aws-eks/dev",
        projectRepo: "https://github.com/metral/pulumi-aws-eks",
        commit: "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
        accessTokenSecret: accessToken.metadata.name,
        config: {
            "aws:region": "us-west-2",
        },
        envSecrets: [awsCreds.metadata.name],
        initOnCreate: true,
        destroyOnFinalize: true,
    }
});
import pulumi
from pulumi_kubernetes import core, apiextensions

# Get the Pulumi API token.
pulumi_config = pulumi.Config()
pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")
aws_access_key_id = pulumi_config.require("awsAccessKeyId")
aws_secret_access_key = pulumi_config.require_secret("awsSecretAccessKey")
aws_session_token = pulumi_config.require_secret("awsSessionToken")

# Create the creds as Kubernetes Secrets.
access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token })
aws_creds = core.v1.Secret("aws-creds", string_data={
    "AWS_ACCESS_KEY_ID": aws_access_key_id,
    "AWS_SECRET_ACCESS_KEY": aws_secret_access_key,
    "AWS_SESSION_TOKEN": aws_session_token,
})

# Create a Kubernetes cluster on AWS EKS.
my_stack = apiextensions.CustomResource("my-stack",
    api_version="pulumi.com/v1alpha1",
    kind="Stack",
    spec={
        "stack": "<YOUR_ORG>/pulumi-aws-eks/dev",
        "project_repo": "https://github.com/metral/pulumi-aws-eks",
        "commit": "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
        "access_token_secret": access_token.metadata["name"],
        "config": {
            "aws:region": "us-west-2",
        },
        "env_secrets": [aws_creds.metadata["name"]],
        "init_on_create": True,
        "destroy_on_finalize": True,
    }
)
using System;
using Pulumi;
using Pulumi.Kubernetes.ApiExtensions;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;

class StackArgs : CustomResourceArgs
{
    [Input("spec")]
    public Input<StackSpecArgs>? Spec { get; set; }

    public StackArgs() : base("pulumi.com/v1alpha1", "Stack")
    {
    }
}

class StackSpecArgs : ResourceArgs
{
    [Input("accessTokenSecret")]
    public Input<string>? AccessTokenSecret { get; set; }

    [Input("stack")]
    public Input<string>? Stack { get; set; }

    [Input("initOnCreate")]
    public Input<bool>? InitOnCreate { get; set; }

    [Input("projectRepo")]
    public Input<string>? ProjectRepo { get; set; }

    [Input("commit")]
    public Input<string>? Commit { get; set; }

    [Input("destroyOnFinalize")]
    public Input<bool>? DestroyOnFinalize { get; set; }

    [Input("envSecrets")]
    public InputList<String>? EnvSecrets { get; set; }

    [Input("config")]
    public InputMap<String>? Config { get; set; }
}

class MyStack : Stack
{
    public MyStack()
    {
        // Get the Pulumi API token.
        var config = new Config();
        var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");
        var awsAccessKeyId = config.Require("awsAccessKeyId");
        var awsSecretAccessKey = config.RequireSecret("awsSecretAccessKey");
        var awsSessionToken = config.RequireSecret("awsSessionToken");

        // Create the creds as Kubernetes Secrets.
        var accessToken = new Secret("accesstoken", new SecretArgs
        {
            StringData =
            {
                {"accessToken", pulumiAccessToken}
            }
        });
        var awsCreds = new Secret("aws-creds", new SecretArgs
        {
            StringData =
            {
                {"AWS_ACCESS_KEY_ID", awsAccessKeyId},
                {"AWS_SECRET_ACCESS_KEY", awsSecretAccessKey},
                {"AWS_SESSION_TOKEN", awsSessionToken}
            }
        });

        // Create an Kubernetes cluster on AWS EKS.
        var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("my-stack", new StackArgs
        {
            Spec = new StackSpecArgs
            {
                Stack = "<YOUR_ORG>/pulumi-aws-eks/dev",
                ProjectRepo = "https://github.com/metral/pulumi-aws-eks",
                Commit = "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
                AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),
                Config =
                {
                    {"aws:region", "us-west-2"}
                },
                EnvSecrets = {awsCreds.Metadata.Apply(m => m.Name)},
                InitOnCreate = true,
                DestroyOnFinalize = true,
            }
        });
    }
}
package main

import (
    "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes"
    apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/apiextensions"
    corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
    metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
    "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
    "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Get the Pulumi API token and AWS creds.
        config := config.New(ctx, "")
        pulumiAccessToken := config.Require("pulumiAccessToken")
        awsAccessKeyID := config.Require("awsAccessKeyId")
        awsSecretAccessKey := config.Require("awsSecretAccessKey")
        awsSessionToken := config.Require("awsSessionToken")

        // Create the creds as Kubernetes Secrets.
        accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{
            StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)},
        })
        if err != nil {
            return err
        }
        awsCreds, err := corev1.NewSecret(ctx, "aws-creds", &corev1.SecretArgs{
            StringData: pulumi.StringMap{
                "AWS_ACCESS_KEY_ID":     pulumi.String(awsAccessKeyID),
                "AWS_SECRET_ACCESS_KEY": pulumi.String(awsSecretAccessKey),
                "AWS_SESSION_TOKEN":     pulumi.String(awsSessionToken),
            },
        })
        if err != nil {
            return err
        }

        // Create an Kubernetes cluster on AWS EKS.
        _, err = apiextensions.NewCustomResource(ctx, "my-stack",
            &apiextensions.CustomResourceArgs{
                ApiVersion: pulumi.String("pulumi.com/v1alpha1"),
                Kind:       pulumi.String("Stack"),
                OtherFields: kubernetes.UntypedArgs{
                    "spec": map[string]interface{}{
                        "stack":             "<YOUR_ORG>/s3-op-project/dev",
                        "projectRepo":       "https://github.com/metral/pulumi-aws-eks",
                        "commit":            "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
                        "accessTokenSecret": accessToken.Metadata.Name(),
                        "config": map[string]string{
                            "aws:region": "us-west-2",
                        },
                        "envSecrets":        []interface{}{awsCreds.Metadata.Name()},
                        "initOnCreate":      true,
                        "destroyOnFinalize": true,
                    },
                },
            }, pulumi.DependsOn([]pulumi.Resource{accessToken, awsCreds}))

        return nil
    })
}

CI/CD Scenarios

As you build out your Kubernetes workloads, you’ll need to implement update strategies to ship new software easily and reliably. Kubernetes API resources like Deployments are used to model stateless apps and operate their lifecycles e.g. recreate app replicas, or perform a rolling update.

A common approach that most teams rely on is the Blue / Green deployment. This starts with an initial version of the app, and then deploys an updated version alongside the initial version. Once ready, traffic is switched over from the initial version to the new version with no downtime.

Blue Green Kubernetes Deployment

In Kubernetes, Blue/Green is modeled by using a Service that load balances and selects a set of Deployment pods labeled “blue,” and then switching to a different selection of updated, “green” pods.

Similar to the Stack CustomResources in the previous sections, we can deploy a Stack of a Blue/Green Kubernetes app in Pulumi, and step through a sequence of its Git commits similiar to how a CI/CD pipeline does. Additionally, we can use the Pulumi config system to parameterize settings and secrets on how to manage the Stack CustomResource project.

Deploying the active blue and passive green versions of the app can be done on the example Stack with the following steps.

pulumi config set --secret pulumiAccessToken <YOUR_PULUMI_API_TOKEN>
pulumi config set stackProjectRepo https://github.com/metral/pulumi-blue-green
pulumi config set stackCommit b19759220f25476605620fdfffeface39a630246
pulumi up -y

After deployment, we’ll test that the v1 / “blue” endpoint is live and running. The v2 / “green” app will be on standby.

while true; do curl http://34.83.25.150:80/version ; echo; sleep 1; done

{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
...

When we’re ready to switch traffic, the blue/green transition is done by updating the config to the Git new commit of the selector swap, and running an update.

pulumi config set stackCommit f4bf2f7b54ec441d5af374933cca4ef09f2bce24
pulumi up -y

After a few seconds you should start seeing the generated traffic returning the new version in a clean cut-over with no dropped packets.

...
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
...

Check out the full walkthrough for a tutorial, and the video clip below for a demo.

Wrap-Up

The Pulumi Kubernetes Operator helps deploy Pulumi Stacks in your Kubernetes cluster, and naturally fits in with your CI/CD process.

We covered examples like deploying an NGINX Stack in the same cluster as the Operator, creating a new Kubernetes cluster in an AWS EKS Stack, and how Stacks can be abstracted for deployment strategies like Blue/Green.

There are many more examples of deploying infrastructure, containers, serverless, and Kubernetes across cloud providers and cloud native services.

Next Steps

Check out the GitHub repo to experiment deploying the operator and some test Stacks.

Learn more about how Pulumi works with Kubernetes, and Get Started if you’re new.

You can help to shape this experience directly by providing feedback on GitHub. We love to hear from our users!

You can explore more content by checking out PulumiTV on YouTube, work through Kubernetes tutorials to dive deeper, and join the Community Slack to engage with users and the Pulumi team.