Pulumi Blog: Mike Metral

Introducing the Pulumi Kubernetes Operator

Mike Metral — Wed, 12 Aug 2020 00:00:00 +0000

Check out version 2.0 of the Pulumi Kubernetes Operator.

Kubernetes developers and operators work together to manage workloads and to continuously ship software through CI/CD. These users have an affinity for automation and pipelines, and richer integration with Kubernetes is a growing theme across the cloud native ecosystem.

We’re excited to introduce the Pulumi Kubernetes Operator: a Kubernetes controller that deploys cloud infrastructure in Pulumi Stacks for you and your team.

These program stacks include virtual machines, block storage, managed Kubernetes clusters, API resources, serverless functions and more!

Overview

The Pulumi Kubernetes Operator is an extension pattern that enables Kubernetes users to create a Pulumi Stack as a first-class API resource, and use the StackController to drive the updates of the Stack until success.

Check out how to deploy the operator to a Kubernetes cluster using any of Pulumi’s supported languages, or through YAML manifests and kubectl.

You can also get started with Pulumi and create a new managed Kubernetes cluster on Amazon EKS, Google GKE, or Azure AKS if you don’t have an existing cluster.

How it Works

When the operator is deployed, the StackController waits for Stack CustomResources to be created, patched, or deleted in the cluster.

On these events, the Kubernetes controller invokes a reconciliation loop to process the request until it has reached success. For Stack creation or updates, this is a successful pulumi up. For deletions, it is a successful pulumi destroy and pulumi stack rm.

Stacks can be written in Typescript, Python, .NET, and Go, and span cloud providers such as Amazon AWS, Google GCP, and Microsoft Azure, as well as many cloud native services!

The following example showcases a typical Pulumi program in Typescript, that can be instantiated as a Stack.

import * as aws from "@pulumi/aws";

// Create an AWS resource (S3 Bucket)
const names = [];
for (let i = 0; i < 2; i++) {
 const bucket = new aws.s3.Bucket(`my-bucket-${i}`, {
 acl: "public-read",
 });
 names.push(bucket.id);
}

// Export the name of the buckets
export const bucketNames = names;

You can find more examples on GitHub of the types of infrastructure you can manage with Pulumi.

Check out Kubernetes Controllers work if you’d like to learn more.

Creating a Pulumi Stack in Kubernetes

The Stack CustomResourceDefinition (CRD) encapsulates an existing Pulumi infrastructure project in a Git repo, a specific commit SHA to deploy, and any additional settings to control the Pulumi update run.

These settings include your Pulumi API access token, environment variables, config and secrets, and lifecycle controls for the update.

Deploying an NGINX Stack on Kubernetes

Check out an example of creating a Kubernetes NGINX Deployment as a Stack by the Operator in its cluster.

Choose your preferred language below, or check out how to create Pulumi Stacks using kubectl.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

// Get the Pulumi API token.
const pulumiConfig = new pulumi.Config();
const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")

// Create the API token as a Kubernetes Secret.
const accessToken = new kx.Secret("accesstoken", {
 stringData: { accessToken: pulumiAccessToken },
});

// Create an NGINX deployment in-cluster.
const mystack = new k8s.apiextensions.CustomResource("my-stack", {
 apiVersion: 'pulumi.com/v1alpha1',
 kind: 'Stack',
 spec: {
 accessTokenSecret: accessToken.metadata.name,
 stack: "<YOUR_ORG>/k8s-nginx/dev",
 initOnCreate: true,
 projectRepo: "https://github.com/pulumi/examples",
 repoDir: "kubernetes-ts-nginx",
 commit: "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 destroyOnFinalize: true,
 }
});

import pulumi
from pulumi_kubernetes import core, apiextensions

# Get the Pulumi API token.
pulumi_config = pulumi.Config()
pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")

# Create the API token as a Kubernetes Secret.
access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token })

# Create an NGINX deployment in-cluster.
my_stack = apiextensions.CustomResource("my-stack",
 api_version="pulumi.com/v1alpha1",
 kind="Stack",
 spec={
 "access_token_secret": access_token.metadata["name"],
 "stack": "<YOUR_ORG>/k8s-nginx/dev",
 "init_on_create": True,
 "project_repo": "https://github.com/pulumi/examples",
 "repo_dir: "kubernetes-ts-nginx",
 "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 "destroy_on_finalize": True,
 }
)

using Pulumi;
using Pulumi.Kubernetes.ApiExtensions;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;

class StackArgs : CustomResourceArgs
{
 [Input("spec")]
 public Input<StackSpecArgs>? Spec { get; set; }

 public StackArgs() : base("pulumi.com/v1alpha1", "Stack")
 {
 }
}

class StackSpecArgs : ResourceArgs
{
 [Input("accessTokenSecret")]
 public Input<string>? AccessTokenSecret { get; set; }

 [Input("stack")]
 public Input<string>? Stack { get; set; }

 [Input("initOnCreate")]
 public Input<bool>? InitOnCreate { get; set; }

 [Input("projectRepo")]
 public Input<string>? ProjectRepo { get; set; }

 [Input("commit")]
 public Input<string>? Commit { get; set; }

 [Input("destroyOnFinalize")]
 public Input<bool>? DestroyOnFinalize { get; set; }
}

class MyStack : Stack
{
 public MyStack()
 {
 // Get the Pulumi API token.
 var config = new Config();
 var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");

 // Create the API token as a Kubernetes Secret.
 var accessToken = new Secret("accesstoken", new SecretArgs
 {
 StringData =
 {
 {"accessToken", pulumiAccessToken}
 }
 });

 // Create an NGINX deployment in-cluster.
 var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("nginx", new StackArgs
 {
 Spec = new StackSpecArgs
 {
 AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),
 Stack = "<YOUR_ORG>/k8s-nginx/dev",
 InitOnCreate = true,
 ProjectRepo = "https://github.com/pulumi/examples",
 RepoDir = "kubernetes-ts-nginx",
 Commit = "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 DestroyOnFinalize = true,
 }
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes"
 apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Get the Pulumi API token.
 c := config.New(ctx, "")
 pulumiAccessToken := c.Require("pulumiAccessToken")

 // Create the API token as a Kubernetes Secret.
 accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{
 StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)},
 })
 if err != nil {
 return err
 }

 // Create an NGINX deployment in-cluster.
 _, err = apiextensions.NewCustomResource(ctx, "my-stack", &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("pulumi.com/v1alpha1"),
 Kind: pulumi.String("Stack"),
 OtherFields: kubernetes.UntypedArgs{
 "spec": map[string]interface{}{
 "accessTokenSecret": accessToken.Metadata.Name(),
 "stack": "<YOUR_ORG>/k8s-nginx/dev",
 "initOnCreate": true,
 "projectRepo": "https://github.com/pulumi/examples",
 "repoDir": "kubernetes-ts-nginx",
 "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 "destroyOnFinalize": true,
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{accessToken}))
 return err
 })
}

Deploying an Amazon EKS Stack

Check out an example of creating a new AWS VPC and Kubernetes cluster on AWS EKS as a Stack by the Operator.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

// Get the Pulumi API token and AWS creds.
const pulumiConfig = new pulumi.Config();
const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")
const awsAccessKeyId = pulumiConfig.require("awsAccessKeyId")
const awsSecretAccessKey = pulumiConfig.requireSecret("awsSecretAccessKey")
const awsSessionToken = pulumiConfig.requireSecret("awsSessionToken")

// Create the creds as Kubernetes Secrets.
const accessToken = new kx.Secret("accesstoken", {
 stringData: { accessToken: pulumiAccessToken},
});
const awsCreds = new kx.Secret("aws-creds", {
 stringData: {
 "AWS_ACCESS_KEY_ID": awsAccessKeyId,
 "AWS_SECRET_ACCESS_KEY": awsSecretAccessKey,
 "AWS_SESSION_TOKEN": awsSessionToken,
 },
});

// Create a Kubernetes cluster on AWS EKS.
const mystack = new k8s.apiextensions.CustomResource("my-stack", {
 apiVersion: 'pulumi.com/v1alpha1',
 kind: 'Stack',
 spec: {
 stack: "<YOUR_ORG>/pulumi-aws-eks/dev",
 projectRepo: "https://github.com/metral/pulumi-aws-eks",
 commit: "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 accessTokenSecret: accessToken.metadata.name,
 config: {
 "aws:region": "us-west-2",
 },
 envSecrets: [awsCreds.metadata.name],
 initOnCreate: true,
 destroyOnFinalize: true,
 }
});

import pulumi
from pulumi_kubernetes import core, apiextensions

# Get the Pulumi API token.
pulumi_config = pulumi.Config()
pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")
aws_access_key_id = pulumi_config.require("awsAccessKeyId")
aws_secret_access_key = pulumi_config.require_secret("awsSecretAccessKey")
aws_session_token = pulumi_config.require_secret("awsSessionToken")

# Create the creds as Kubernetes Secrets.
access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token })
aws_creds = core.v1.Secret("aws-creds", string_data={
 "AWS_ACCESS_KEY_ID": aws_access_key_id,
 "AWS_SECRET_ACCESS_KEY": aws_secret_access_key,
 "AWS_SESSION_TOKEN": aws_session_token,
})

# Create a Kubernetes cluster on AWS EKS.
my_stack = apiextensions.CustomResource("my-stack",
 api_version="pulumi.com/v1alpha1",
 kind="Stack",
 spec={
 "stack": "<YOUR_ORG>/pulumi-aws-eks/dev",
 "project_repo": "https://github.com/metral/pulumi-aws-eks",
 "commit": "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 "access_token_secret": access_token.metadata["name"],
 "config": {
 "aws:region": "us-west-2",
 },
 "env_secrets": [aws_creds.metadata["name"]],
 "init_on_create": True,
 "destroy_on_finalize": True,
 }
)

using System;
using Pulumi;
using Pulumi.Kubernetes.ApiExtensions;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;

class StackArgs : CustomResourceArgs
{
 [Input("spec")]
 public Input<StackSpecArgs>? Spec { get; set; }

 public StackArgs() : base("pulumi.com/v1alpha1", "Stack")
 {
 }
}

class StackSpecArgs : ResourceArgs
{
 [Input("accessTokenSecret")]
 public Input<string>? AccessTokenSecret { get; set; }

 [Input("stack")]
 public Input<string>? Stack { get; set; }

 [Input("initOnCreate")]
 public Input<bool>? InitOnCreate { get; set; }

 [Input("projectRepo")]
 public Input<string>? ProjectRepo { get; set; }

 [Input("commit")]
 public Input<string>? Commit { get; set; }

 [Input("destroyOnFinalize")]
 public Input<bool>? DestroyOnFinalize { get; set; }

 [Input("envSecrets")]
 public InputList<String>? EnvSecrets { get; set; }

 [Input("config")]
 public InputMap<String>? Config { get; set; }
}

class MyStack : Stack
{
 public MyStack()
 {
 // Get the Pulumi API token.
 var config = new Config();
 var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");
 var awsAccessKeyId = config.Require("awsAccessKeyId");
 var awsSecretAccessKey = config.RequireSecret("awsSecretAccessKey");
 var awsSessionToken = config.RequireSecret("awsSessionToken");

 // Create the creds as Kubernetes Secrets.
 var accessToken = new Secret("accesstoken", new SecretArgs
 {
 StringData =
 {
 {"accessToken", pulumiAccessToken}
 }
 });
 var awsCreds = new Secret("aws-creds", new SecretArgs
 {
 StringData =
 {
 {"AWS_ACCESS_KEY_ID", awsAccessKeyId},
 {"AWS_SECRET_ACCESS_KEY", awsSecretAccessKey},
 {"AWS_SESSION_TOKEN", awsSessionToken}
 }
 });

 // Create an Kubernetes cluster on AWS EKS.
 var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("my-stack", new StackArgs
 {
 Spec = new StackSpecArgs
 {
 Stack = "<YOUR_ORG>/pulumi-aws-eks/dev",
 ProjectRepo = "https://github.com/metral/pulumi-aws-eks",
 Commit = "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),
 Config =
 {
 {"aws:region", "us-west-2"}
 },
 EnvSecrets = {awsCreds.Metadata.Apply(m => m.Name)},
 InitOnCreate = true,
 DestroyOnFinalize = true,
 }
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes"
 apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Get the Pulumi API token and AWS creds.
 config := config.New(ctx, "")
 pulumiAccessToken := config.Require("pulumiAccessToken")
 awsAccessKeyID := config.Require("awsAccessKeyId")
 awsSecretAccessKey := config.Require("awsSecretAccessKey")
 awsSessionToken := config.Require("awsSessionToken")

 // Create the creds as Kubernetes Secrets.
 accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{
 StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)},
 })
 if err != nil {
 return err
 }
 awsCreds, err := corev1.NewSecret(ctx, "aws-creds", &corev1.SecretArgs{
 StringData: pulumi.StringMap{
 "AWS_ACCESS_KEY_ID": pulumi.String(awsAccessKeyID),
 "AWS_SECRET_ACCESS_KEY": pulumi.String(awsSecretAccessKey),
 "AWS_SESSION_TOKEN": pulumi.String(awsSessionToken),
 },
 })
 if err != nil {
 return err
 }

 // Create an Kubernetes cluster on AWS EKS.
 _, err = apiextensions.NewCustomResource(ctx, "my-stack",
 &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("pulumi.com/v1alpha1"),
 Kind: pulumi.String("Stack"),
 OtherFields: kubernetes.UntypedArgs{
 "spec": map[string]interface{}{
 "stack": "<YOUR_ORG>/s3-op-project/dev",
 "projectRepo": "https://github.com/metral/pulumi-aws-eks",
 "commit": "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 "accessTokenSecret": accessToken.Metadata.Name(),
 "config": map[string]string{
 "aws:region": "us-west-2",
 },
 "envSecrets": []interface{}{awsCreds.Metadata.Name()},
 "initOnCreate": true,
 "destroyOnFinalize": true,
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{accessToken, awsCreds}))

 return nil
 })
}

CI/CD Scenarios

As you build out your Kubernetes workloads, you’ll need to implement update strategies to ship new software easily and reliably. Kubernetes API resources like Deployments are used to model stateless apps and operate their lifecycles e.g. recreate app replicas, or perform a rolling update.

A common approach that most teams rely on is the Blue / Green deployment. This starts with an initial version of the app, and then deploys an updated version alongside the initial version. Once ready, traffic is switched over from the initial version to the new version with no downtime.

In Kubernetes, Blue/Green is modeled by using a Service that load balances and selects a set of Deployment pods labeled “blue,” and then switching to a different selection of updated, “green” pods.

Similar to the Stack CustomResources in the previous sections, we can deploy a Stack of a Blue/Green Kubernetes app in Pulumi, and step through a sequence of its Git commits similar to how a CI/CD pipeline does. Additionally, we can use the Pulumi config system to parameterize settings and secrets on how to manage the Stack CustomResource project.

Deploying the active blue and passive green versions of the app can be done on the example Stack with the following steps.

pulumi config set --secret pulumiAccessToken <YOUR_PULUMI_API_TOKEN>
pulumi config set stackProjectRepo https://github.com/metral/pulumi-blue-green
pulumi config set stackCommit b19759220f25476605620fdfffeface39a630246
pulumi up -y

After deployment, we’ll test that the v1 / “blue” endpoint is live and running. The v2 / “green” app will be on standby.

while true; do curl http://34.83.25.150:80/version ; echo; sleep 1; done

{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
...

When we’re ready to switch traffic, the blue/green transition is done by updating the config to the Git new commit of the selector swap, and running an update.

pulumi config set stackCommit f4bf2f7b54ec441d5af374933cca4ef09f2bce24
pulumi up -y

After a few seconds you should start seeing the generated traffic returning the new version in a clean cut-over with no dropped packets.

...
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
...

Check out the full walkthrough for a tutorial, and the video clip below for a demo.

Wrap-Up

The Pulumi Kubernetes Operator helps deploy Pulumi Stacks in your Kubernetes cluster, and naturally fits in with your CI/CD process.

We covered examples like deploying an NGINX Stack in the same cluster as the Operator, creating a new Kubernetes cluster in an AWS EKS Stack, and how Stacks can be abstracted for deployment strategies like Blue/Green.

There are many more examples of deploying infrastructure, containers, serverless, and Kubernetes across cloud providers and cloud native services.

Next Steps

Check out the GitHub repo to experiment deploying the operator and some test Stacks.

Learn more about how Pulumi works with Kubernetes, and Get Started if you’re new.

You can help to shape this experience directly by providing feedback on GitHub. We love to hear from our users!

You can explore more content by checking out PulumiTV on YouTube, work through Kubernetes tutorials to dive deeper, and join the Community Slack to engage with users and the Pulumi team.

What's new in Pulumi 2.0 for Kubernetes

Mike Metral — Thu, 04 Jun 2020 00:00:00 +0000

We recently announced the 2.0 release of Pulumi which includes parity for Node.js (JavaScript, TypeScript), Python, .NET (C#, F#, etc) and Go, and improvements to Kubernetes and dozens of other supported cloud resource providers.

Kubernetes support in Pulumi spans orchestration of clusters and application workloads. Clusters can be managed by cloud providers or self-managed. Workloads use the same Kubernetes API to create and manage API resources in the supported Pulumi languages through packages directly generated from the OpenAPI specification.

Overview

In our march towards v2.0, we’ve been working on improvements and added new features for Kubernetes thanks to the valuable feedback we’ve received from our community of users and customers.

Let’s review some of the major highlights.

Managing Workloads

Users provision and manage Kubernetes workloads in various ways: YAML manifests, Helm charts, operators, and direct API access. Pulumi can help to unify this resource management using a programming language of your choice.

Some of the recent highlights for the pulumi/kubernetes package include:

Added language support for Go and .NET

// Create an EKS cluster in Go.
func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create EKS Cluster
 eksCluster, err := eks.NewCluster(ctx, "eks-cluster", &eks.ClusterArgs{
 RoleArn: pulumi.StringInput(eksRole.Arn),
 VpcConfig: &eks.ClusterVpcConfigArgs{
 PublicAccessCidrs: pulumi.StringArray{
 pulumi.String("0.0.0.0/0"),
 },
 SecurityGroupIds: pulumi.StringArray{
 clusterSg.ID().ToStringOutput(),
 },
 SubnetIds: toPulumiStringArray(subnet.Ids),
 },
 })
 if err != nil {
 return err
 }

 // Create the NodeGroup.
 _, err = eks.NewNodeGroup(ctx, "node-group-2", &eks.NodeGroupArgs{
 ClusterName: eksCluster.Name,
 NodeGroupName: pulumi.String("demo-eks-nodegroup-2"),
 NodeRoleArn: pulumi.StringInput(nodeGroupRole.Arn),
 SubnetIds: toPulumiStringArray(subnet.Ids),
 ScalingConfig: &eks.NodeGroupScalingConfigArgs{
 DesiredSize: pulumi.Int(2),
 MaxSize: pulumi.Int(2),
 MinSize: pulumi.Int(1),
 },
 })
 if err != nil {
 return err
 }

 ctx.Export("kubeconfig", generateKubeconfig(eksCluster.Endpoint,
 eksCluster.CertificateAuthority.Data().Elem(), eksCluster.Name))

 return nil
 })
}

 // Create an EKS cluster in C#
 var cluster = new Eks.Cluster("eks-cluster", new Eks.ClusterArgs
 {
 RoleArn = eksRole.Arn,
 VpcConfig = new ClusterVpcConfigArgs
 {
 PublicAccessCidrs = {"0.0.0.0/0"},
 SecurityGroupIds = {clusterSg.Id},
 SubnetIds = subnetIds,
 },
 });

 // Create the NodeGroup.
 var nodeGroup = new Eks.NodeGroup("node-group", new Eks.NodeGroupArgs
 {
 ClusterName = cluster.Name,
 NodeGroupName = "demo-eks-nodegroup",
 NodeRoleArn = nodeGroupRole.Arn,
 SubnetIds = subnetIds,
 ScalingConfig = new NodeGroupScalingConfigArgs{DesiredSize = 2},
 });

Kubernetes version support now includes the latest v1.18 release, which builds on Pulumi’s support for clusters >= v1.9.0
- Warnings are displayed for Kubernetes resources using deprecated apiVersions.
- Updated the client-go dependency in the provider to take advantage of the latest upstream fixes and changes.
General house cleaning and fixes
- Helm v3 CRD’s are included as expected when the include-crds flag is used.
- Fixed CRD resource patch updates by changing their default merge behavior.

Managing Clusters

Kubernetes clusters come in different shapes and sizes, and with various operational requirements to run. We’ve expanded our support for managed Kubernetes clusters across the respective cloud provider packages.

pulumi/eks
- More examples are provided to cover various EKS-specific scenarios.
- Support for additional security groups for node groups to attach to user-specified rules
- Support for envelope encrypted secrets so that Kubernetes Secrets are encrypted in etcd
- Support for alternative authentication approaches on the cluster and infrastructure created, such as using AWS named profiles
- Create kubeconfig files that are scoped to IAM users or roles – this is used to limit access to the cluster.
- Use the EKS package on clients operating in HTTP proxy environments, such as corporate proxies.
- Default to the latest AMIs for node groups, using the latest release from the AWS SSM parameter store.

Wrap-Up

As our community continues to grow, we’re committed to improving our Kubernetes experience even further. You can help to shape this experience directly by providing feedback on GitHub. We love to hear from our users!

Learn more about how Pulumi works with Kubernetes, and Get Started if you’re new.

You can explore more content by checking out PulumiTV on YouTube, work through Kubernetes tutorials to dive deeper, and join the Community Slack to engage with users and the Pulumi team.

Access Control for Pods on Amazon EKS

Mike Metral — Tue, 02 Jun 2020 00:00:00 +0000

Amazon EKS clusters can use IAM roles and policies for Pods to assign fine-grained access control of AWS services. The AWS IAM entities map into Kubernetes RBAC to configure the permissions of Pods that work with AWS services.

Together, AWS IAM and Kubernetes RBAC enable least-privileged access for your apps, scoped to the appropriate policies and user requirements.

Overview

In <100 lines of code we’ll demonstrate how EKS Pods can use AWS IAM to create fine-grained permissions for apps that integrate with other AWS services.

Pod Access Control
Create an OIDC provider
Create IAM for a S3 app
Deploy a S3 app
Wrap-Up
Next Steps

Pod Access Control

AWS EKS supports using IAM entities in a Pod Service Account by leveraging an OIDC provider connected to the Kubernetes cluster.

Continuing with the example from the AWS blog post, when an S3 app written in Go pushes an object to a bucket with the AWS SDK, it will need write access to S3.

When a Pod is launched with a particular Service Account, the OIDC provider works with Kubernetes to verify the Pod’s identity, and in turn collaborates with the AWS Secure Token Service (STS) to grant the Pod temporary credentials to use with the IAM role.

Create an OIDC provider

Creating an OIDC provider is as simple as toggling the createOidcProvider option in the definition of your EKS cluster.

When enabled, the OIDC provider will be created and associated with the cluster’s OIDC provider URL.

import * as eks from '@pulumi/eks'

// Create an EKS cluster with default settings.
// Create and attach an OIDC provider to the cluster.
const cluster = new eks.Cluster('myCluster', {
 createOidcProvider: true,
});

Create IAM for a S3 app

We’ll use the OIDC provider URL and Amazon Resource Name (ARN) to compose the AssumeRoleWithWebIdentity, and S3 IAM policies that will be attached to a new S3 IAM role.

After the role is configured, a Service Account for the S3 Pod will be created, and annotated with the ARN of the S3 role to bind the two together.

import * as aws from '@pulumi/aws'
import * as k8s from '@pulumi/kubernetes'
import * as pulumi from '@pulumi/pulumi'

// Create a pulumi Kubernetes provider using the cluster's kubeconfig.
const k8sProvider = new k8s.Provider('k8s', {
 kubeconfig: cluster.kubeconfig.apply(JSON.stringify),
});

// Create a k8s namespace in the cluster.
const namespace = new k8s.core.v1.Namespace('apps', undefined, { k8sProvider });

// Get the OIDC provider's URL for the cluster.
const clusterOidcProvider = cluster.core.oidcProvider.url;

// Create the new IAM policy for the Service Account using the AssumeRoleWebWebIdentity action.
const saName = 's3'
const saAssumeRolePolicy = pulumi
 .all([clusterOidcProviderUrl, clusterOidcProvider.arn, appsNamespaceName])
 .apply(([url, arn, namespace]) =>
 aws.iam.getPolicyDocument({
 statements: [
 {
 actions: ['sts:AssumeRoleWithWebIdentity'],
 conditions: [
 {
 test: 'StringEquals',
 values: [`system:serviceaccount:${namespace.metadata.name}:${saName}`],
 variable: `${url.replace('https://', '')}:sub`,
 },
 ],
 effect: 'Allow',
 principals: [{identifiers: [arn], type: 'Federated'}],
 },
 ],
 })
 );

// Create a new IAM role that assumes the AssumeRoleWebWebIdentity policy.
const saRole = new aws.iam.Role(saName, {
 assumeRolePolicy: saAssumeRolePolicy.json,
});

// Attach the IAM role to an AWS S3 access policy.
const saS3Rpa = new aws.iam.RolePolicyAttachment(saName, {
 policyArn: 'arn:aws:iam::aws:policy/AmazonS3FullAccess',
 role: saRole,
});

// Create a Service Account with the IAM role annotated to use with the Pod.
const sa = new k8s.core.v1.ServiceAccount(
 saName,
 {
 metadata: {
 namespace: namespace.metadata.name,
 name: saName,
 annotations: {
 'eks.amazonaws.com/role-arn': saRole.arn,
 },
 },
 },
 { k8sProvider });

Deploy a S3 app

We’ll deploy the S3 app to use the new IAM-backed Service Account.

Once the Pod is running, the Service Account annotation will be automatically managed by a Kubernetes dynamic admission controller run by EKS on your behalf.

The AWS EKS webhook manages Pod identity, and injects STS credentials into the Pod to use with the S3 role.

With the credentials, the app can successfully upload data to S3.

import * as aws from "@pulumi/aws";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const bucket = new aws.s3.Bucket("pod-irsa-job-bucket");
const bucketName = bucket.id;
const regionName = pulumi.output(aws.getRegion({}, {async: true})).name;
const s3Pod = new k8s.core.v1.Pod(podName,
 {
 metadata: {labels: labels, namespace: appsNamespaceName},
 spec: {
 serviceAccountName: sa.metadata.name,
 containers: [
 {
 name: podName,
 image: "amazonlinux:2018.03",
 command: ["sh", "-c",
 pulumi.interpolate`curl -sL -o /s3-echoer https://git.io/JfnGX && chmod +x /s3-echoer && echo This is an in-cluster test | /s3-echoer ${bucketName} && sleep 3600`,
 ],
 env: [
 {name: "AWS_DEFAULT_REGION", value: regionName},
 {name: "ENABLE_IRP", value: "true"},
 ],
 },
 ],
 },
 }, { provider: provider },
);

Wrap-Up

Leveraging AWS IAM for Pod workloads is a secure and effective means of limiting privileged execution, and provides a native experience for users.

Pod IAM can be extended further by also using the Kubernetes RBAC system. This allows configuring permissions for Kubernetes API resources, and handle scenarios such as limiting the namespace an IAM role can use, and what resources can be managed in the namespace.

Next Steps

Learn more about how Pulumi works with Kubernetes, and get started if you’re new.

Check out code examples for the S3 app referenced in this post, along with other access control scenarios for EKS.

Watch the video below for more details on how OIDC and Kubernetes RBAC works in EKS. We demonstrate how to deploy fluentd-cloudwatch with IAM to forward Pod logs to AWS CloudWatch.

You can also follow us on Twitter, subscribe to PulumiTV on YouTube, or join our Community Slack channel if you have any questions.

Multicloud Kubernetes: Running Apps Across EKS, AKS, and GKE

Mike Metral — Wed, 14 Aug 2019 00:00:00 +0000

Kubernetes clusters from the managed platforms of AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and GCP Google Kubernetes Engine (GKE) all vary in configuration, management, and resource properties. This variance creates unnecessary complexity in cluster provisioning and application deployments, as well as for CI/CD and testing.

Additionally, if you wanted to deploy the same app across multiple clusters for specific use cases or test scenarios across providers, subtleties such as LoadBalancer outputs and cluster connection settings can be a nuisance to manage.

In this post, we’ll see how to use Pulumi to deploy the kuard app across EKS, AKS, GKE and a local Kubernetes cluster, such as Docker Desktop or a self-managed cluster. We’ll spin up the clusters in each provider, launch the app, and manage both cluster and app using the TypeScript programming language.

View the full example and code.

![](multicloud.png)

Cluster Provisioning

Provisioning Kubernetes clusters and their IaaS resources is made simple through Pulumi’s various SDKs for the cloud providers:

Crosswalk for AWS further allows us to leverage the Pulumi libraries of common infrastructure for AWS to simplify cloud resource instantiation and management while gaining best-practices as defaults. Check out the pulumi/awsx SDK to get started.

For local clusters such as those that are self-managed, or provisioned by a tool like Docker Desktop, Pulumi can still deploy workloads to these these systems given that the pulumi/kubernetes workload SDK only requires a valid kubeconfig file. For more information on Pulumi’s Kubernetes support, check out the Kubernetes reference page.

We will use the cloud SDKs to provision the managed Kubernetes clusters. Given that we’re working with real code, we are afforded developer benefits such as: code linting, type checking, IDE hints and completion, abstractions and inheritance.

Leveraging these development features creates the opportunity to encapsulate the finer-grained details and settings, and expose the capability to create clusters as simple as the following code:

Workload Deployment

Once the clusters are provisioned, we can leverage the pulumi/kubernetes SDK to manage the Kubernetes workloads that will be deployed into the clusters.

The pulumi/kubernetes SDK uses the official Kubernetes client-go library to interact with Kubernetes. Therefore, Pulumi can work pretty much anywhere kubectl works, even if Pulumi was not used to create the cluster.

Summary

As shown in the code samples, it becomes relatively easy to provision and manage Kubernetes clusters across multiple clouds, as well as deploy workloads to the cluster regardless if they are managed by a cloud provider, or self-managed.

The various SDKS allow you to leverage industry standard best-practices and defaults, in addition to allowing you to further configure and customize how your clusters and apps are managed.

Testing apps across various providers in this form allows you to abstract away the details of provider specific implementations, and focus on how your app operates in the various contexts.

Learn More

If you’d like to learn about Pulumi and how to manage your infrastructure and Kubernetes multi-cloud capabilities through code, get started today. Pulumi is open source and free to use.

For further examples on how to use Pulumi to create Kubernetes clusters, or deploy workloads to a cluster, check out the rest of the Kubernetes tutorials.

As always, you can check out our code on GitHub, follow us on Twitter, subscribe to our YouTube channel, or join our Community Slack channel if you have any questions, need support, or just want to say hello.

If you’d like to chat with our team, or get hands-on assistance with migrating your existing configuration code to Pulumi, please don’t hesitate to drop us a line.

We also encourage you to watch Pulumi team member Levi Blackstone demo this post in an episode of the Kubernetes Community Meeting.

Migrate EKS Node Groups with Zero Downtime Using Pulumi

Mike Metral — Tue, 23 Jul 2019 00:00:00 +0000

Managed Kubernetes offerings greatly reduce the overhead required in administering Kubernetes. However, the cluster is only one of the components under management, as app lifecycles are self-driven tasks that vary by workloads.

In Kubernetes, node groups are a useful mechanism for creating pools of resources that can enforce scheduling requirements. They also provide a utility for shifting workloads around during cluster management and updates.

In this post, we’ll see how to use Pulumi for Day 2 Kubernetes administration. We’ll spin up a new EKS cluster with two node groups and a given workload. Then we’ll add one more node group with an updated configuration, and migrate the workload over to it with zero downtime using code and kubectl.

View the full how-to and code.

Create an EKS cluster and Deploy the Workload

For our initial update, we will configure and launch an EKS cluster using v1.13 of Kubernetes, with the cluster’s infrastructure dependencies (such as VPC and IAM) defined using Crosswalk for AWS. Crosswalk allows us to leverage the Pulumi libraries of common infrastructure for AWS to simplify cloud resource instantiation and management while gaining best-practices as defaults.

We’ll also create and attach the following node groups to the cluster:

A standard t2.medium worker node group using the recent v1.13.7 worker AMI, for general purpose workloads such as the EchoServer — a simple app that echo’s client request headers.
A 2xlarge t3.2xlarge worker node group using an older v1.12.7 worker AMI, for use by larger, intensive workloads such as the NGINX Ingress Controller.

With the cluster successfully created and the node groups available, Pulumi will deploy the workload: the EchoServer, and the NGINX Ingress Controller that will manage its ingress. The echoserver will land on the Standard node group, and NGINX is set to specifically target the 2xlarge node group.

Once the workload is deployed, we can validate it is up and running by accessing the echoserver behind the NGINX endpoint using curl:

Note: You can open the GIF for a maximized view.

The Great Migration

After our initial deployment, let’s say we decide to update the node group used by NGINX.

To do so, we’ll move NGINX from the 2xlarge node group over to a new, 4xlarge worker node group with an updated AMI, a different instance type, and a different desired instance count.

As we migrate NGINX over to the 4xlarge and decommission the 2xlarge node group in the next steps, we’ll also actively load test the endpoint of the echoserver to ensure that we are not losing requests throughout the migration.

Step 1: Create the new `4xlarge` node group.

Create the new, 4xlarge node group by defining it and running an update.

Step 2: Migrate NGINX to the `4xlarge` node group.

With the 4xlarge node group created, we’ll migrate the NGINX service away from the 2xlarge node group over to the 4xlarge node group by changing its node selector scheduling terms.

This change updates the NGINX Deployment spec to require the use of c5.4xlarge nodes during scheduling, and forces a rolling update over to the 4xlarge node group.

NGINX is able to successfully migrate across node groups because it is configured with HA settings, spread-type scheduling predicates, and can gracefully terminate within the Kubernetes Pod lifecycle.

Step 3: Decommission the `2xlarge` node group.

Once NGINX has been completely migrated to the 4xlarge node group, we can begin decommissioning the original 2xlarge node group which is no longer in use.

Decommissioning the node group involves:

Draining the Kubernetes nodes.
Deleting the Kubernetes nodes from the APIServer.
Scaling down the Auto Scaling Group to 0.
Deleting the node group.

Set up kubectl by using the kubeconfig from the stack output.

$ pulumi stack output kubeconfig > kubeconfig.json
$ export KUBECONFIG=$PWD/kubeconfig.json

To drain the 2xlarge node group, we’ll use kubectl drain:

for node in $(kubectl get nodes -l beta.kubernetes.io/instance-type=t3.2xlarge -o=name); do
 kubectl drain --force --ignore-daemonsets --delete-local-data --grace-period=10 "$node";
done

After draining has completed, use kubectl delete node to delete the nodes from the APIServer.

for node in $(kubectl get nodes -l beta.kubernetes.io/instance-type=t3.2xlarge -o=name); do
 kubectl delete "$node";
done

Scale down the node group Auto Scaling Group completely by setting the desiredCapacity: 0 and running an update:

Once the Auto Scaling Group has scaled down, we can delete the node group from AWS and the Pulumi program:

Summary

In this post, we stood up an EKS cluster with a couple of node groups, and an echoserver and NGINX workload. We then created a new, updated node group and migrated NGINX over to it.

We achieved this node group migration with zero downtime to our apps during load testing and decommissioning of the original node group.

Note: You can open the GIF for a maximized view.

Learn More

If you’d like to learn about Pulumi and how to manage your infrastructure and Kubernetes through code, get started today. Pulumi is open source and free to use.

For further examples on how to use Pulumi to create Kubernetes clusters, or deploy workloads to a cluster, check out the rest of the Kubernetes tutorials.

If you’d like to chat with our team, or get hands-on assistance with migrating your existing configuration code to Pulumi, please don’t hesitate to drop us a line.

If you liked ksonnet, you'll love Pulumi!

Mike Metral — Wed, 13 Feb 2019 00:00:00 +0000

This article references ksonnet, which was deprecated in 2019. For the most up-to-date information on using Pulumi with Kubernetes, see our Kubernetes documentation.

The Kubernetes landscape is constantly evolving as end users and developers search for the right tools, approaches, and abstractions to help them manage Cloud Native infrastructure in production.

On Feb 5, Heptio (now part of VMWare) announced that work on ksonnet, a project launched by Heptio, Box, Microsoft, and Bitnami, will stop. We’re sad to see ksonnet winding down, but are thankful for the collaborative exchange of ideas between projects, and are excited to see continued investment in VMWare/Heptio’s other projects. The good news is that, if you liked ksonnet, we’re confident that you’ll love Pulumi. In this post, we’ll tell you why.

Recap of ksonnet’s Approach

In the original ksonnet launch post, Joe Beda describes the “wall of YAML” that we face when creating and configuring Kubernetes systems. This intimidates new users, making for a steep learning curve, while also leading to thankless boilerplate and copy-and-paste for the most experienced users. Systems like Helm have attempted to solve the problem by adding templates to YAML.

ksonnet, instead, introduced a new DSL – based on work from Google on the Jsonnet language (itself influenced by experience internally on Borg) – with more “language like” functionality, that could generate configuration. The result was a feature set that included:

Local variables
Functions and lambdas
Modules and imports
Mixins
Configuration concatenation and merging
Template parameterization

The idea is that you can author richer configuration in ksonnet, and from that, simply generate and apply stock Kubernetes configuration, without Kubernetes needing to know about ksonnet.

Recap of Pulumi’s Approach

Pulumi was inspired by many of the same challenges as ksonnet, but from a slightly broader perspective. As we surveyed the landscape of configuration languages, we found a bewildering array of them: plenty of stock YAML, but also JSON and YAML with “built-in” templating mechanisms (AWS CloudFormation and Azure Resource Manager Templates), templating systems bolted on top of JSON and YAML (Helm Charts), and new DSLs (Terraform), to name just a few. To add to the complexity, many engineers have to stitch multiple of them together or regularly switch between different dialects in an increasing multi-cloud environment.

Instead of creating another DSL, we wondered: Why not use languages we’re already familiar with? Is there some inherent reason that’s incompatible with configuration languages? The answer was no, it seemed to us. While Infrastructure-as-Code is based on the idea of a declarative goal state, we realized that it’s possible to use imperative languages to create that state.

By using languages like JavaScript, TypeScript, Python, and Go, we would immediately see many benefits:

All of the powerful language capabilities they already have (including the ksonnet list)
Instant access to entire package ecosystems (NPM, PyPI, etc)
Great IDE support, including integrated API documentation, syntax highlighting, statement completion, inline errors, and refactoring
Ability to test, lint, and statically analyze code using existing tools

But we didn’t want to just generate configuration files. There are systems out there that generate a configuration from languages like Python. In our experience, the underlying configuration system and its limitations always leaks through, quickly ruining the ease-of-use benefits. So, we built an engine that has a full cloud object model across providers, can diff resources, applies updates using robust Infrastructure-as-Code techniques, and provides rich status reporting.

A short tour of Pulumi for ksonnet users

Just like in ksonnet, Pulumi provides access to the complete raw Kubernetes API, so you can do everything that Kubernetes supports - but using programming languages that provide completion lists, error checking, inline documentation, and basic features like variables and control flow. Here’s an example of part of a direct port of the Guestbook example to Pulumi:

Also like in ksonnet, in Pulumi we can introduce additional features like modules/imports, components, functions, and more to structure our code and increase the level of abstraction that we work at as we author our Kubernetes configuration. This example shows the Guestbook app re-written using higher-level components in Pulumi, and you can see similar approaches to those used in the similar example with ksonnet… and just using TypeScript instead of bespoke DSL:

Unlike with ksonnet though, Pulumi programs are deployed using the pulumi CLI, which provides rich feedback on the progress a deployment is making toward being complete (as well as feedback when things go wrong!).

And finally, a major area where Pulumi goes beyond ksonnet in scope, it works with public and private cloud resources as well as Kubernetes, enabling deployment of Kubernetes resources and AWS/Azure/GCP resources using a single deployment workflow. For example, you can stand up a managed Kubernetes cluster and deploying a Helm chart to it.

[

Another fun example is provisioning and consuming an Azure CosmosDB from within your Kubernetes app.

Learn More

If you’d like to learn about Pulumi and how to manage your infrastructure and Kubernetes through code, click here to get started today. Pulumi is open source and free to use.

If you’d like to go deeper on certain topics, here are some additional resources to check out:

Overview of Pulumi Kubernetes Scenarios
Tutorial: Create a Kubernetes cluster on a cloud provider Amazon EKS, Google GKE, or Azure AKS
Tutorial: Operate and deploy to a Kubernetes cluster
Docs: Pulumi docs, including an overview of the programming model
Video: In February, we were honored to have Joe Beda show where Pulumi fits in on his TGIK livestream, highlighting how to use your favorite language and consistent workflows to create, deploy, and manage cloud resources across multiple clouds and Kubernetes. Watch Joe Beda take Pulumi for a spin here.

If you’d like to chat with our team, or get hands-on assistance with migrating your existing configuration code (including ksonnet programs) to Pulumi, please don’t hesitate to drop us a line.

Pulumi Blog: Mike Metral

Introducing the Pulumi Kubernetes Operator

Overview

How it Works

Creating a Pulumi Stack in Kubernetes

Deploying an NGINX Stack on Kubernetes

Deploying an Amazon EKS Stack

CI/CD Scenarios

Wrap-Up

Next Steps

What's new in Pulumi 2.0 for Kubernetes

Overview

Managing Workloads

Managing Clusters

Wrap-Up

Access Control for Pods on Amazon EKS

Overview

Pod Access Control

Create an OIDC provider

Create IAM for a S3 app

Deploy a S3 app

Wrap-Up

Next Steps

Multicloud Kubernetes: Running Apps Across EKS, AKS, and GKE

Cluster Provisioning

Workload Deployment

Summary

Learn More

Migrate EKS Node Groups with Zero Downtime Using Pulumi

Create an EKS cluster and Deploy the Workload

The Great Migration

Step 1: Create the new 4xlarge node group.

Step 2: Migrate NGINX to the 4xlarge node group.

Step 3: Decommission the 2xlarge node group.

Summary

Learn More

If you liked ksonnet, you'll love Pulumi!

Recap of ksonnet’s Approach

Recap of Pulumi’s Approach

A short tour of Pulumi for ksonnet users

Learn More

Step 1: Create the new `4xlarge` node group.

Step 2: Migrate NGINX to the `4xlarge` node group.

Step 3: Decommission the `2xlarge` node group.