Pulumi Blog: Eks

Getting Started with Amazon EKS Auto Mode in Pulumi

Florian Stadler — Mon, 16 Dec 2024 23:26:21 +0100

AWS recently announced Amazon EKS Auto Mode, a significant enhancement to Amazon EKS that streamlines cluster management by automating infrastructure decisions and operations. Today, we are excited to announce that Pulumi fully supports EKS Auto Mode across our AWS provider ecosystem, enabling you to leverage this powerful feature through infrastructure as code.

What is EKS Auto Mode?

EKS Auto Mode extends AWS management beyond the Kubernetes control plane to the underlying infrastructure that powers your workloads. While traditional EKS requires you to make numerous infrastructure decisions and maintain various components, Auto Mode simplifies this by automating these choices based on AWS’s operational expertise.

Instead of manually configuring and managing cluster add-ons, node groups, and infrastructure components, Auto Mode handles:

Compute Management: Automatically provisions and scales EC2 instances based on your workload demands while selecting optimal instance types and sizes. It also handles node lifecycle management, including OS updates and security patches.
Storage Automation: Eliminates the need to manually set up and configure the EBS CSI driver for block storage. Auto Mode manages volume provisioning, attachment, and cleanup for your persistent workloads.
Networking Simplification: Automatically configures and manages core networking components like Load Balancers.

This significantly reduces the operational overhead of running Kubernetes clusters. Instead of spending time on infrastructure decisions and maintenance, teams can now focus on their applications. For example, deploying a stateful application with load balancing no longer requires setting up and maintaining multiple add-ons and configurations - Auto Mode handles this automatically while following AWS best practices.

Common tasks that previously required significant expertise and time are now managed by AWS, making it simpler to run production-grade Kubernetes workloads. This includes:

Setting up and maintaining cluster autoscaling
Configuring and updating networking plugins
Managing storage drivers and volume provisioning
Handling node updates and security patches
Optimizing instance selection and placement

Pulumi Support for EKS Auto Mode

We have added EKS Auto Mode support across our AWS provider ecosystem:

Pulumi AWS Provider (v6.63.0+)
Pulumi AWS Cloud Control Provider (previously known as Pulumi AWS Native) (v1.13.0+)
Pulumi EKS Provider (v3.5.0+)

Creating an EKS Auto Mode Cluster

Let’s explore how to leverage EKS Auto Mode with Pulumi by deploying an example. You can enable EKS Auto Mode by setting autoMode.enabled to true.

First, we’ll create a new EKS cluster with Auto Mode enabled. This requires properly configured networking and authentication settings:

VPC subnets are tagged appropriately for EKS Auto Mode load balancer discovery
Authentication Mode supports Access Entries (Api or ApiAndConfigMap mode) as required by EKS Auto Mode
Default node groups and security groups are skipped as Auto Mode manages these

import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import { SubnetType } from "@pulumi/awsx/ec2";

const config = new pulumi.Config();
const clusterName = config.require("clusterName");

const eksVpc = new awsx.ec2.Vpc("eks-auto-mode", {
 enableDnsHostnames: true,
 cidrBlock: "10.0.0.0/16",
 subnetSpecs: [
 // Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 // See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 {type: SubnetType.Public, tags: {[`kubernetes.io/cluster/${clusterName}`]: "shared", "kubernetes.io/role/elb": "1"}},
 {type: SubnetType.Private, tags: {[`kubernetes.io/cluster/${clusterName}`]: "shared", "kubernetes.io/role/internal-elb": "1"}},
 ],
 subnetStrategy: "Auto"
});

const cluster = new eks.Cluster("eks-auto-mode", {
 name: clusterName,
 // EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 authenticationMode: eks.AuthenticationMode.Api,
 vpcId: eksVpc.vpcId,
 publicSubnetIds: eksVpc.publicSubnetIds,
 privateSubnetIds: eksVpc.privateSubnetIds,
 // Enables compute, storage and load balancing for the cluster.
 autoMode: {
 enabled: true,
 }
});

import pulumi
import pulumi_awsx as awsx
import pulumi_eks as eks
import pulumi_kubernetes as kubernetes

eks_vpc = awsx.ec2.Vpc("eksVpc",
 enable_dns_hostnames=True,
 cidr_block="10.0.0.0/16",
 # Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 # See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 subnet_specs=[
 {
 "type": awsx.ec2.SubnetType.PUBLIC,
 "tags": {
 "kubernetes_io_cluster_eks_auto_mode_demo": "shared",
 "kubernetes_io_role_elb": "1",
 },
 },
 {
 "type": awsx.ec2.SubnetType.PRIVATE,
 "tags": {
 "kubernetes_io_cluster_eks_auto_mode_demo": "shared",
 "kubernetes_io_role_internal_elb": "1",
 },
 },
 ],
 subnet_strategy=awsx.ec2.SubnetAllocationStrategy.AUTO)
cluster = eks.Cluster("cluster",
 name="eks-auto-mode-demo",
 # EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 authentication_mode=eks.AuthenticationMode.API,
 vpc_id=eks_vpc.vpc_id,
 public_subnet_ids=eks_vpc.public_subnet_ids,
 private_subnet_ids=eks_vpc.private_subnet_ids,
 # Enables compute, storage and load balancing for the cluster.
 auto_mode={
 "enabled": True,
 })

package main

import (
 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi-eks/sdk/v3/go/eks"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 networkingv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/networking/v1"
 storagev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/storage/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 eksVpc, err := ec2.NewVpc(ctx, "eksVpc", &ec2.VpcArgs{
 EnableDnsHostnames: pulumi.Bool(true),
 CidrBlock: "10.0.0.0/16",
 // Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 // See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 SubnetSpecs: []ec2.SubnetSpecArgs{
 {
 Type: ec2.SubnetTypePublic,
 Tags: {
 "kubernetes.io/cluster/eks-auto-mode-demo": pulumi.String("shared"),
 "kubernetes.io/role/elb": pulumi.String("1"),
 },
 },
 {
 Type: ec2.SubnetTypePrivate,
 Tags: {
 "kubernetes.io/cluster/eks-auto-mode-demo": pulumi.String("shared"),
 "kubernetes.io/role/internal-elb": pulumi.String("1"),
 },
 },
 },
 SubnetStrategy: ec2.SubnetAllocationStrategyAuto,
 })
 if err != nil {
 return err
 }
 cluster, err := eks.NewCluster(ctx, "cluster", &eks.ClusterArgs{
 Name: pulumi.String("eks-auto-mode-demo"),
 // EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 AuthenticationMode: eks.AuthenticationModeApi,
 VpcId: eksVpc.VpcId,
 PublicSubnetIds: eksVpc.PublicSubnetIds,
 PrivateSubnetIds: eksVpc.PrivateSubnetIds,
 // Enables compute, storage and load balancing for the cluster.
 AutoMode: &eks.AutoModeOptionsArgs{
 Enabled: true,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Awsx = Pulumi.Awsx;
using Eks = Pulumi.Eks;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 var eksVpc = new Awsx.Ec2.Vpc("eksVpc", new()
 {
 EnableDnsHostnames = true,
 CidrBlock = "10.0.0.0/16",
 // Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 // See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 SubnetSpecs = new()
 {
 new Awsx.Ec2.Inputs.SubnetSpecArgs
 {
 Type = Awsx.Ec2.SubnetType.Public,
 Tags =
 {
 { "kubernetes.io/cluster/eks-auto-mode-demo", "shared" },
 { "kubernetes.io/role/elb", "1" },
 },
 },
 new Awsx.Ec2.Inputs.SubnetSpecArgs
 {
 Type = Awsx.Ec2.SubnetType.Private,
 Tags =
 {
 { "kubernetes.io/cluster/eks-auto-mode-demo", "shared" },
 { "kubernetes.io/role/internal-elb", "1" },
 },
 },
 },
 SubnetStrategy = Awsx.Ec2.SubnetAllocationStrategy.Auto,
 });

 var cluster = new Eks.Cluster("cluster", new()
 {
 Name = "eks-auto-mode-demo",
 // EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 AuthenticationMode = Eks.AuthenticationMode.Api,
 VpcId = eksVpc.VpcId,
 PublicSubnetIds = eksVpc.PublicSubnetIds,
 PrivateSubnetIds = eksVpc.PrivateSubnetIds,
 // Enables compute, storage and load balancing for the cluster.
 AutoMode = new Eks.Inputs.AutoModeOptionsArgs
 {
 Enabled = true,
 },
 });
});

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.awsx.ec2.Vpc;
import com.pulumi.awsx.ec2.VpcArgs;
import com.pulumi.awsx.ec2.inputs.SubnetSpecArgs;
import com.pulumi.eks.Cluster;
import com.pulumi.eks.ClusterArgs;
import com.pulumi.eks.inputs.AutoModeOptionsArgs;
import com.pulumi.resources.CustomResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var eksVpc = new Vpc("eksVpc", VpcArgs.builder()
 .enableDnsHostnames(true)
 .cidrBlock("10.0.0.0/16")
 // Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 // See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 .subnetSpecs(
 SubnetSpecArgs.builder()
 .type("Public")
 .tags(Map.ofEntries(
 Map.entry("kubernetes.io/cluster/eks-auto-mode-demo", "shared"),
 Map.entry("kubernetes.io/role/elb", "1")
 ))
 .build(),
 SubnetSpecArgs.builder()
 .type("Private")
 .tags(Map.ofEntries(
 Map.entry("kubernetes.io/cluster/eks-auto-mode-demo", "shared"),
 Map.entry("kubernetes.io/role/internal-elb", "1")
 ))
 .build())
 .subnetStrategy("Auto")
 .build());

 var cluster = new Cluster("cluster", ClusterArgs.builder()
 .name("eks-auto-mode-demo")
 // EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 .authenticationMode("API")
 .vpcId(eksVpc.vpcId())
 .publicSubnetIds(eksVpc.publicSubnetIds())
 .privateSubnetIds(eksVpc.privateSubnetIds())
 // Enables compute, storage and load balancing for the cluster.
 .autoMode(AutoModeOptionsArgs.builder()
 .enabled(true)
 .build())
 .build());
 }
}

name: eks-auto-mode-demo
description: A demo of EKS Auto Mode
runtime: yaml
resources:
 eksVpc:
 type: awsx:ec2:Vpc
 properties:
 enableDnsHostnames: true
 cidrBlock: "10.0.0.0/16"
 # Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 # See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 subnetSpecs:
 - type: "Public"
 tags:
 kubernetes.io/cluster/eks-auto-mode-demo: shared
 kubernetes.io/role/elb: "1"
 - type: "Private"
 tags:
 kubernetes.io/cluster/eks-auto-mode-demo: shared
 kubernetes.io/role/internal-elb: "1"
 subnetStrategy: "Auto"
 cluster:
 type: eks:Cluster
 properties:
 name: eks-auto-mode-demo
 # EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 authenticationMode: API
 vpcId: ${eksVpc.vpcId}
 publicSubnetIds: ${eksVpc.publicSubnetIds}
 privateSubnetIds: ${eksVpc.privateSubnetIds}
 # Enables compute, storage and load balancing for the cluster.
 autoMode:
 enabled: true

Deploying a Load-Balanced Application

Now that we have provisioned an EKS cluster with Auto Mode enabled, let’s deploy a web application fronted by an Application Load Balancer. EKS Auto Mode will automatically provision appropriately sized nodes for your workload, set up the Application Load Balancer according to the Ingress configuration, and manage ongoing maintenance and updates.

//...
const appName = "nginx";
const ns = new k8s.core.v1.Namespace(appName, {
 metadata: { name: appName },
}, { provider: cluster.provider });

const configMap = new k8s.core.v1.ConfigMap(appName, {
 metadata: {
 namespace: ns.metadata.name,
 },
 data: {
 "index.html": "<html><body><h1>Hello, Pulumi!</h1></body></html>",
 },
}, { provider: cluster.provider });

const deployment = new k8s.apps.v1.Deployment(appName, {
 metadata: {
 namespace: ns.metadata.name
 },
 spec: {
 selector: { matchLabels: { app: appName } },
 replicas: 3,
 template: {
 metadata: { labels: { app: appName } },
 spec: {
 containers: [{
 name: appName,
 image: appName,
 ports: [{ containerPort: 80 }],
 volumeMounts: [{ name: "nginx-index", mountPath: "/usr/share/nginx/html" }],
 }],
 volumes: [{
 name: "nginx-index",
 configMap: { name: configMap.metadata.name },
 }],
 },
 },
 },
}, { provider: cluster.provider });

const service = new k8s.core.v1.Service(appName, {
 metadata: {
 name: appName,
 namespace: ns.metadata.name
 },
 spec: {
 selector: { app: appName },
 ports: [{ port: 80, targetPort: 80 }],
 },
}, { provider: cluster.provider, dependsOn: [deployment] });

const ingressClass = new k8s.networking.v1.IngressClass("alb", {
 metadata: {
 namespace: ns.metadata.name,
 labels: {
 "app.kubernetes.io/name": "LoadBalancerController",
 },
 name: "alb",
 },
 spec: {
 controller: "eks.amazonaws.com/alb",
 }
}, { provider: cluster.provider });

const ingress = new k8s.networking.v1.Ingress(appName, {
 metadata: {
 namespace: ns.metadata.name,
 // Annotations for EKS Auto Mode to identify the Ingress as internet-facing and target-type as IP.
 annotations: {
 "alb.ingress.kubernetes.io/scheme": "internet-facing",
 "alb.ingress.kubernetes.io/target-type": "ip",
 }
 },
 spec: {
 ingressClassName: ingressClass.metadata.name,
 rules: [{
 http: {
 paths: [{
 path: "/",
 pathType: "Prefix",
 backend: {
 service: {
 name: service.metadata.name,
 port: {
 number: 80,
 },
 },
 },
 }],
 },
 }],
 }
}, { provider: cluster.provider });

export const url = ingress.status.apply(status => status?.loadBalancer?.ingress?.[0]?.hostname);

# ...
k8s_provider = kubernetes.Provider("k8sProvider", kubeconfig=cluster.kubeconfig)
ns = kubernetes.core.v1.Namespace("ns", metadata={
 "name": "nginx",
},
opts = pulumi.ResourceOptions(provider=k8s_provider))
index_content = kubernetes.core.v1.ConfigMap("index-content",
 metadata={
 "namespace": ns.metadata.name,
 },
 data={
 "index.html": "<html><body><h1>Hello, Pulumi!</h1></body></html>",
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider))
deployment = kubernetes.apps.v1.Deployment("deployment",
 metadata={
 "namespace": ns.metadata.name,
 },
 spec={
 "selector": {
 "match_labels": {
 "app": "nginx",
 },
 },
 "replicas": 3,
 "template": {
 "metadata": {
 "labels": {
 "app": "nginx",
 },
 },
 "spec": {
 "containers": [{
 "name": "nginx",
 "image": "nginx",
 "ports": [{
 "container_port": 80,
 }],
 "volume_mounts": [{
 "name": "nginx-index",
 "mount_path": "/usr/share/nginx/html",
 }],
 }],
 "volumes": [{
 "name": "nginx-index",
 "config_map": {
 "name": index_content.metadata.name,
 },
 }],
 },
 },
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider))
service = kubernetes.core.v1.Service("service",
 metadata={
 "name": "nginx",
 "namespace": ns.metadata.name,
 },
 spec={
 "selector": {
 "app": "nginx",
 },
 "ports": [{
 "port": 80,
 "target_port": 80,
 }],
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider,
 depends_on=[deployment]))
ingress_class = kubernetes.networking.v1.IngressClass("ingressClass",
 metadata={
 "namespace": ns.metadata.name,
 "labels": {
 "app_kubernetes_io_name": "LoadBalancerController",
 },
 "name": "alb",
 },
 spec={
 "controller": "eks.amazonaws.com/alb",
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider))
ingress = kubernetes.networking.v1.Ingress("ingress",
 metadata={
 "namespace": ns.metadata.name,
 "annotations": {
 "alb_ingress_kubernetes_io_scheme": "internet-facing",
 "alb_ingress_kubernetes_io_target_type": "ip",
 },
 },
 spec={
 "ingress_class_name": ingress_class.metadata.name,
 "rules": [{
 "http": {
 "paths": [{
 "path": "/",
 "path_type": "Prefix",
 "backend": {
 "service": {
 "name": service.metadata.name,
 "port": {
 "number": 80,
 },
 },
 },
 }],
 },
 }],
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider))

pulumi.export("url", ingress.status.load_balancer.ingress[0].hostname)

package main

import (
 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi-eks/sdk/v3/go/eks"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 networkingv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/networking/v1"
 storagev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/storage/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // ...
 k8sProvider, err := kubernetes.NewProvider(ctx, "k8sProvider", &kubernetes.ProviderArgs{
 Kubeconfig: cluster.Kubeconfig,
 })
 if err != nil {
 return err
 }
 ns, err := corev1.NewNamespace(ctx, "ns", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx"),
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }
 indexContent, err := corev1.NewConfigMap(ctx, "index-content", &corev1.ConfigMapArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Data: pulumi.StringMap{
 "index.html": pulumi.String("<html><body><h1>Hello, Pulumi!</h1></body></html>"),
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }
 deployment, err := appsv1.NewDeployment(ctx, "deployment", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Spec: &appsv1.DeploymentSpecArgs{
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Replicas: pulumi.Int(3),
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx"),
 Ports: corev1.ContainerPortArray{
 &corev1.ContainerPortArgs{
 ContainerPort: pulumi.Int(80),
 },
 },
 VolumeMounts: corev1.VolumeMountArray{
 &corev1.VolumeMountArgs{
 Name: pulumi.String("nginx-index"),
 MountPath: pulumi.String("/usr/share/nginx/html"),
 },
 },
 },
 },
 Volumes: corev1.VolumeArray{
 &corev1.VolumeArgs{
 Name: pulumi.String("nginx-index"),
 ConfigMap: &corev1.ConfigMapVolumeSourceArgs{
 Name: indexContent.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 },
 },
 },
 },
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }
 service, err := corev1.NewService(ctx, "service", &corev1.ServiceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx"),
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Spec: &corev1.ServiceSpecArgs{
 Selector: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 Ports: corev1.ServicePortArray{
 &corev1.ServicePortArgs{
 Port: pulumi.Int(80),
 TargetPort: pulumi.Any(80),
 },
 },
 },
 }, pulumi.Provider(k8sProvider), pulumi.DependsOn([]pulumi.Resource{
 deployment,
 }))
 if err != nil {
 return err
 }
 ingressClass, err := networkingv1.NewIngressClass(ctx, "ingressClass", &networkingv1.IngressClassArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Labels: pulumi.StringMap{
 "app.kubernetes.io/name": pulumi.String("LoadBalancerController"),
 },
 Name: pulumi.String("alb"),
 },
 Spec: &networkingv1.IngressClassSpecArgs{
 Controller: pulumi.String("eks.amazonaws.com/alb"),
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }
 ingress, err := networkingv1.NewIngress(ctx, "ingress", &networkingv1.IngressArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Annotations: pulumi.StringMap{
 "alb.ingress.kubernetes.io/scheme": pulumi.String("internet-facing"),
 "alb.ingress.kubernetes.io/target-type": pulumi.String("ip"),
 },
 },
 Spec: &networkingv1.IngressSpecArgs{
 IngressClassName: ingressClass.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Rules: networkingv1.IngressRuleArray{
 &networkingv1.IngressRuleArgs{
 Http: &networkingv1.HTTPIngressRuleValueArgs{
 Paths: networkingv1.HTTPIngressPathArray{
 &networkingv1.HTTPIngressPathArgs{
 Path: pulumi.String("/"),
 PathType: pulumi.String("Prefix"),
 Backend: &networkingv1.IngressBackendArgs{
 Service: &networkingv1.IngressServiceBackendArgs{
 Name: service.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Port: &networkingv1.ServiceBackendPortArgs{
 Number: pulumi.Int(80),
 },
 },
 },
 },
 },
 },
 },
 },
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }

 ctx.Export("url", ingress.Status.ApplyT(func(status networkingv1.IngressStatus) (*string, error) {
 return &status.LoadBalancer.Ingress[0].Hostname, nil
 }).(pulumi.StringPtrOutput))
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Awsx = Pulumi.Awsx;
using Eks = Pulumi.Eks;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 // ...
 var k8sProvider = new Kubernetes.Provider.Provider("k8sProvider", new()
 {
 KubeConfig = cluster.Kubeconfig,
 });

 var ns = new Kubernetes.Core.V1.Namespace("ns", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "nginx",
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 var indexContent = new Kubernetes.Core.V1.ConfigMap("index-content", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Data =
 {
 { "index.html", "<html><body><h1>Hello, Pulumi!</h1></body></html>" },
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 var deployment = new Kubernetes.Apps.V1.Deployment("deployment", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Spec = new Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Selector = new Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "nginx" },
 },
 },
 Replicas = 3,
 Template = new Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Labels =
 {
 { "app", "nginx" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 Containers = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "nginx",
 Image = "nginx",
 Ports = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerPortArgs
 {
 ContainerPortValue = 80,
 },
 },
 VolumeMounts = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.VolumeMountArgs
 {
 Name = "nginx-index",
 MountPath = "/usr/share/nginx/html",
 },
 },
 },
 },
 Volumes = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.VolumeArgs
 {
 Name = "nginx-index",
 ConfigMap = new Kubernetes.Types.Inputs.Core.V1.ConfigMapVolumeSourceArgs
 {
 Name = indexContent.Metadata.Apply(metadata => metadata.Name),
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 var service = new Kubernetes.Core.V1.Service("service", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "nginx",
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.ServiceSpecArgs
 {
 Selector =
 {
 { "app", "nginx" },
 },
 Ports = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ServicePortArgs
 {
 Port = 80,
 TargetPort = 80,
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 DependsOn =
 {
 deployment,
 },
 });

 var ingressClass = new Kubernetes.Networking.V1.IngressClass("ingressClass", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 Labels =
 {
 { "app.kubernetes.io/name", "LoadBalancerController" },
 },
 Name = "alb",
 },
 Spec = new Kubernetes.Types.Inputs.Networking.V1.IngressClassSpecArgs
 {
 Controller = "eks.amazonaws.com/alb",
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 var ingress = new Kubernetes.Networking.V1.Ingress("ingress", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 Annotations =
 {
 { "alb.ingress.kubernetes.io/scheme", "internet-facing" },
 { "alb.ingress.kubernetes.io/target-type", "ip" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Networking.V1.IngressSpecArgs
 {
 IngressClassName = ingressClass.Metadata.Apply(metadata => metadata.Name),
 Rules = new[]
 {
 new Kubernetes.Types.Inputs.Networking.V1.IngressRuleArgs
 {
 Http = new Kubernetes.Types.Inputs.Networking.V1.HTTPIngressRuleValueArgs
 {
 Paths = new[]
 {
 new Kubernetes.Types.Inputs.Networking.V1.HTTPIngressPathArgs
 {
 Path = "/",
 PathType = "Prefix",
 Backend = new Kubernetes.Types.Inputs.Networking.V1.IngressBackendArgs
 {
 Service = new Kubernetes.Types.Inputs.Networking.V1.IngressServiceBackendArgs
 {
 Name = service.Metadata.Apply(metadata => metadata.Name),
 Port = new Kubernetes.Types.Inputs.Networking.V1.ServiceBackendPortArgs
 {
 Number = 80,
 },
 },
 },
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 return new Dictionary<string, object?>
 {
 ["url"] = ingress.Status.Apply(status => status?.LoadBalancer?.Ingress[0]?.Hostname),
 };
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.awsx.ec2.Vpc;
import com.pulumi.awsx.ec2.VpcArgs;
import com.pulumi.awsx.ec2.inputs.SubnetSpecArgs;
import com.pulumi.eks.Cluster;
import com.pulumi.eks.ClusterArgs;
import com.pulumi.eks.inputs.AutoModeOptionsArgs;
import com.pulumi.kubernetes.Provider;
import com.pulumi.kubernetes.ProviderArgs;
import com.pulumi.kubernetes.core_v1.Namespace;
import com.pulumi.kubernetes.core_v1.NamespaceArgs;
import com.pulumi.kubernetes.meta_v1.inputs.ObjectMetaArgs;
import com.pulumi.kubernetes.core_v1.ConfigMap;
import com.pulumi.kubernetes.core_v1.ConfigMapArgs;
import com.pulumi.kubernetes.apps_v1.Deployment;
import com.pulumi.kubernetes.apps_v1.DeploymentArgs;
import com.pulumi.kubernetes.apps_v1.inputs.DeploymentSpecArgs;
import com.pulumi.kubernetes.meta_v1.inputs.LabelSelectorArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodTemplateSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodSpecArgs;
import com.pulumi.kubernetes.core_v1.Service;
import com.pulumi.kubernetes.core_v1.ServiceArgs;
import com.pulumi.kubernetes.core_v1.inputs.ServiceSpecArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressClass;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressClassArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.inputs.IngressClassSpecArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.Ingress;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.inputs.IngressSpecArgs;
import com.pulumi.kubernetes.storage.k8s.io_v1.StorageClass;
import com.pulumi.kubernetes.storage.k8s.io_v1.StorageClassArgs;
import com.pulumi.kubernetes.core_v1.PersistentVolumeClaim;
import com.pulumi.kubernetes.core_v1.PersistentVolumeClaimArgs;
import com.pulumi.kubernetes.core_v1.inputs.PersistentVolumeClaimSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.VolumeResourceRequirementsArgs;
import com.pulumi.resources.CustomResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 // ...
 var k8sProvider = new Provider("k8sProvider", ProviderArgs.builder()
 .kubeconfig(cluster.kubeconfig())
 .build());

 var ns = new Namespace("ns", NamespaceArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx")
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 var indexContent = new ConfigMap("indexContent", ConfigMapArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .data(Map.of("index.html", "<html><body><h1>Hello, Pulumi!</h1></body></html>"))
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 var deployment = new Deployment("deployment", DeploymentArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .spec(DeploymentSpecArgs.builder()
 .selector(LabelSelectorArgs.builder()
 .matchLabels(Map.of("app", "nginx"))
 .build())
 .replicas(3)
 .template(PodTemplateSpecArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .labels(Map.of("app", "nginx"))
 .build())
 .spec(PodSpecArgs.builder()
 .containers(ContainerArgs.builder()
 .name("nginx")
 .image("nginx")
 .ports(ContainerPortArgs.builder()
 .containerPort(80)
 .build())
 .volumeMounts(VolumeMountArgs.builder()
 .name("nginx-index")
 .mountPath("/usr/share/nginx/html")
 .build())
 .build())
 .volumes(VolumeArgs.builder()
 .name("nginx-index")
 .configMap(ConfigMapVolumeSourceArgs.builder()
 .name(indexContent.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .build())
 .build())
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 var service = new Service("service", ServiceArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx")
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .spec(ServiceSpecArgs.builder()
 .selector(Map.of("app", "nginx"))
 .ports(ServicePortArgs.builder()
 .port(80)
 .targetPort(80)
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .dependsOn(deployment)
 .build());

 var ingressClass = new IngressClass("ingressClass", IngressClassArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .labels(Map.of("app.kubernetes.io/name", "LoadBalancerController"))
 .name("alb")
 .build())
 .spec(IngressClassSpecArgs.builder()
 .controller("eks.amazonaws.com/alb")
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 var ingress = new Ingress("ingress", IngressArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .annotations(Map.ofEntries(
 Map.entry("alb.ingress.kubernetes.io/scheme", "internet-facing"),
 Map.entry("alb.ingress.kubernetes.io/target-type", "ip")
 ))
 .build())
 .spec(IngressSpecArgs.builder()
 .ingressClassName(ingressClass.metadata().applyValue(metadata -> metadata.name()))
 .rules(IngressRuleArgs.builder()
 .http(HTTPIngressRuleValueArgs.builder()
 .paths(HTTPIngressPathArgs.builder()
 .path("/")
 .pathType("Prefix")
 .backend(IngressBackendArgs.builder()
 .service(IngressServiceBackendArgs.builder()
 .name(service.metadata().applyValue(metadata -> metadata.name()))
 .port(ServiceBackendPortArgs.builder()
 .number(80)
 .build())
 .build())
 .build())
 .build())
 .build())
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 ctx.export("url", ingress.status().applyValue(status -> status.loadBalancer().ingress()[0].hostname()));
 }
}

name: eks-auto-mode-yaml
description: A Pulumi YAML program to deploy a Kubernetes cluster on AWS
runtime: yaml
resources:
 # ...
 k8sProvider:
 type: pulumi:providers:kubernetes
 properties:
 kubeconfig: ${cluster.kubeconfig}
 ns:
 type: kubernetes:core/v1:Namespace
 properties:
 metadata:
 name: nginx
 options:
 provider: ${k8sProvider}
 index-content:
 type: kubernetes:core/v1:ConfigMap
 properties:
 metadata:
 namespace: ${ns.metadata.name}
 data:
 "index.html": "<html><body><h1>Hello, Pulumi!</h1></body></html>"
 options:
 provider: ${k8sProvider}
 deployment:
 type: kubernetes:apps/v1:Deployment
 properties:
 metadata:
 namespace: ${ns.metadata.name}
 spec:
 selector:
 matchLabels:
 app: nginx
 replicas: 3
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: nginx
 image: nginx
 ports:
 - containerPort: 80
 volumeMounts:
 - name: nginx-index
 mountPath: /usr/share/nginx/html
 volumes:
 - name: nginx-index
 configMap:
 name: ${index-content.metadata.name}
 options:
 provider: ${k8sProvider}
 service:
 type: kubernetes:core/v1:Service
 properties:
 metadata:
 name: nginx
 namespace: ${ns.metadata.name}
 spec:
 selector:
 app: nginx
 ports:
 - port: 80
 targetPort: 80
 options:
 provider: ${k8sProvider}
 dependsOn:
 - ${deployment}
 ingressClass:
 type: kubernetes:networking.k8s.io/v1:IngressClass
 properties:
 metadata:
 namespace: ${ns.metadata.name}
 labels:
 app.kubernetes.io/name: LoadBalancerController
 name: alb
 spec:
 controller: eks.amazonaws.com/alb
 options:
 provider: ${k8sProvider}
 ingress:
 type: kubernetes:networking.k8s.io/v1:Ingress
 properties:
 metadata:
 namespace: ${ns.metadata.name}
 annotations:
 alb.ingress.kubernetes.io/scheme: internet-facing
 alb.ingress.kubernetes.io/target-type: ip
 spec:
 ingressClassName: ${ingressClass.metadata.name}
 rules:
 - http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: ${service.metadata.name}
 port:
 number: 80
 options:
 provider: ${k8sProvider}
outputs:
 url: ${ingress.status.loadBalancer.ingress[0].hostname}

After deploying this with pulumi up, you can access your application with curl once the Load Balancer has been provisioned:

~ curl $(pulumi stack output url)
Hello, Pulumi!

Deploying a Stateful Workload

EKS Auto Mode simplifies persistent storage management by automatically handling EBS volume provisioning for your stateful workloads. To leverage this feature, you can create a StorageClass that uses the EBS CSI driver (ebs.csi.eks.amazonaws.com) as its provisioner. EKS Auto Mode will then automatically manage the lifecycle of your persistent volumes, including provisioning, attaching, and cleanup.

This is how you’d configure persistent storage for a sample application:

// ...
const storageClass = new k8s.storage.v1.StorageClass("storage-class", {
 metadata: {
 name: "auto-ebs-sc",
 annotations: {
 "storageclass.kubernetes.io/is-default-class": "true",
 },
 },
 provisioner: "ebs.csi.eks.amazonaws.com",
 volumeBindingMode: "WaitForFirstConsumer",
 parameters: {
 type: "gp3",
 encrypted: "true",
 },
}, {
 provider: cluster.provider,
});
const pvc = new k8s.core.v1.PersistentVolumeClaim("pvc", {
 metadata: {
 name: "auto-ebs-claim",
 namespace: ns.metadata.apply(metadata => metadata.name),
 },
 spec: {
 accessModes: ["ReadWriteOnce"],
 storageClassName: storageClass.metadata.apply(metadata => metadata.name),
 resources: {
 requests: {
 storage: "8Gi",
 },
 },
 },
}, {
 provider: cluster.provider,
});
const statefulWorkload = new k8s.apps.v1.Deployment("stateful-workload", {
 metadata: {
 name: "inflate-stateful",
 namespace: ns.metadata.apply(metadata => metadata.name),
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: {
 app: "inflate-stateful",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "inflate-stateful",
 },
 },
 spec: {
 terminationGracePeriodSeconds: 0,
 nodeSelector: {
 "eks.amazonaws.com/compute-type": "auto",
 },
 containers: [{
 name: "bash",
 image: "public.ecr.aws/docker/library/bash:4.4",
 command: ["/usr/local/bin/bash"],
 args: [
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done",
 ],
 resources: {
 requests: {
 cpu: "1",
 },
 },
 volumeMounts: [{
 name: "persistent-storage",
 mountPath: "/data",
 }],
 }],
 volumes: [{
 name: "persistent-storage",
 persistentVolumeClaim: {
 claimName: pvc.metadata.name,
 },
 }],
 },
 },
 },
}, {
 provider: cluster.provider,
});

export const url = ingress.status.loadBalancer.ingress[0].hostname.apply(h => `http://${h}:80`);

# ...
storage_class = kubernetes.storage.v1.StorageClass("storage-class",
 metadata={
 "name": "auto-ebs-sc",
 "annotations": {
 "storageclass_kubernetes_io_is_default_class": "true",
 },
 },
 provisioner="ebs.csi.eks.amazonaws.com",
 volume_binding_mode="WaitForFirstConsumer",
 parameters={
 "type": "gp3",
 "encrypted": "true",
 },
 opts = pulumi.ResourceOptions(provider=k8_s_provider))
pvc = kubernetes.core.v1.PersistentVolumeClaim("pvc",
 metadata={
 "name": "auto-ebs-claim",
 "namespace": ns.metadata.name,
 },
 spec={
 "access_modes": ["ReadWriteOnce"],
 "storage_class_name": storage_class.metadata.name,
 "resources": {
 "requests": {
 "storage": "8Gi",
 },
 },
 },
 opts = pulumi.ResourceOptions(provider=k8_s_provider))
stateful_workload = kubernetes.apps.v1.Deployment("stateful-workload",
 metadata={
 "name": "inflate-stateful",
 "namespace": ns.metadata.name,
 },
 spec={
 "replicas": 1,
 "selector": {
 "match_labels": {
 "app": "inflate-stateful",
 },
 },
 "template": {
 "metadata": {
 "labels": {
 "app": "inflate-stateful",
 },
 },
 "spec": {
 "termination_grace_period_seconds": 0,
 "node_selector": {
 "eks_amazonaws_com_compute_type": "auto",
 },
 "containers": [{
 "name": "bash",
 "image": "public.ecr.aws/docker/library/bash:4.4",
 "command": ["/usr/local/bin/bash"],
 "args": [
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done",
 ],
 "resources": {
 "requests": {
 "cpu": "1",
 },
 },
 "volume_mounts": [{
 "name": "persistent-storage",
 "mount_path": "/data",
 }],
 }],
 "volumes": [{
 "name": "persistent-storage",
 "persistent_volume_claim": {
 "claim_name": pvc.metadata.name,
 },
 }],
 },
 },
 },
 opts = pulumi.ResourceOptions(provider=k8_s_provider))

package main

import (
 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi-eks/sdk/v3/go/eks"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 networkingv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/networking/v1"
 storagev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/storage/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // ...
 storageClass, err := storagev1.NewStorageClass(ctx, "storage-class", &storagev1.StorageClassArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("auto-ebs-sc"),
 Annotations: pulumi.StringMap{
 "storageclass.kubernetes.io/is-default-class": pulumi.String("true"),
 },
 },
 Provisioner: pulumi.String("ebs.csi.eks.amazonaws.com"),
 VolumeBindingMode: pulumi.String("WaitForFirstConsumer"),
 Parameters: pulumi.StringMap{
 "type": pulumi.String("gp3"),
 "encrypted": pulumi.String("true"),
 },
 }, pulumi.Provider(k8SProvider))
 if err != nil {
 return err
 }
 pvc, err := corev1.NewPersistentVolumeClaim(ctx, "pvc", &corev1.PersistentVolumeClaimArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("auto-ebs-claim"),
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Spec: &corev1.PersistentVolumeClaimSpecArgs{
 AccessModes: pulumi.StringArray{
 pulumi.String("ReadWriteOnce"),
 },
 StorageClassName: storageClass.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Resources: &corev1.VolumeResourceRequirementsArgs{
 Requests: pulumi.StringMap{
 "storage": pulumi.String("8Gi"),
 },
 },
 },
 }, pulumi.Provider(k8SProvider))
 if err != nil {
 return err
 }
 _, err = appsv1.NewDeployment(ctx, "stateful-workload", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("inflate-stateful"),
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Spec: &appsv1.DeploymentSpecArgs{
 Replicas: pulumi.Int(1),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("inflate-stateful"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("inflate-stateful"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 TerminationGracePeriodSeconds: pulumi.Int(0),
 NodeSelector: pulumi.StringMap{
 "eks.amazonaws.com/compute-type": pulumi.String("auto"),
 },
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("bash"),
 Image: pulumi.String("public.ecr.aws/docker/library/bash:4.4"),
 Command: pulumi.StringArray{
 pulumi.String("/usr/local/bin/bash"),
 },
 Args: pulumi.StringArray{
 pulumi.String("-c"),
 pulumi.String("while true; do echo $(date -u) >> /data/out.txt; sleep 60; done"),
 },
 Resources: &corev1.ResourceRequirementsArgs{
 Requests: pulumi.StringMap{
 "cpu": pulumi.String("1"),
 },
 },
 VolumeMounts: corev1.VolumeMountArray{
 &corev1.VolumeMountArgs{
 Name: pulumi.String("persistent-storage"),
 MountPath: pulumi.String("/data"),
 },
 },
 },
 },
 Volumes: corev1.VolumeArray{
 &corev1.VolumeArgs{
 Name: pulumi.String("persistent-storage"),
 PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSourceArgs{
 ClaimName: pvc.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 },
 },
 },
 },
 },
 }, pulumi.Provider(k8SProvider))
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Awsx = Pulumi.Awsx;
using Eks = Pulumi.Eks;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 // ...
 var storageClass = new Kubernetes.Storage.V1.StorageClass("storage-class", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "auto-ebs-sc",
 Annotations =
 {
 { "storageclass.kubernetes.io/is-default-class", "true" },
 },
 },
 Provisioner = "ebs.csi.eks.amazonaws.com",
 VolumeBindingMode = "WaitForFirstConsumer",
 Parameters =
 {
 { "type", "gp3" },
 { "encrypted", "true" },
 },
 }, new CustomResourceOptions
 {
 Provider = k8SProvider,
 });

 var pvc = new Kubernetes.Core.V1.PersistentVolumeClaim("pvc", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "auto-ebs-claim",
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.PersistentVolumeClaimSpecArgs
 {
 AccessModes = new[]
 {
 "ReadWriteOnce",
 },
 StorageClassName = storageClass.Metadata.Apply(metadata => metadata.Name),
 Resources = new Kubernetes.Types.Inputs.Core.V1.VolumeResourceRequirementsArgs
 {
 Requests =
 {
 { "storage", "8Gi" },
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8SProvider,
 });

 var statefulWorkload = new Kubernetes.Apps.V1.Deployment("stateful-workload", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "inflate-stateful",
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Spec = new Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Replicas = 1,
 Selector = new Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "inflate-stateful" },
 },
 },
 Template = new Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Labels =
 {
 { "app", "inflate-stateful" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 TerminationGracePeriodSeconds = 0,
 NodeSelector =
 {
 { "eks.amazonaws.com/compute-type", "auto" },
 },
 Containers = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "bash",
 Image = "public.ecr.aws/docker/library/bash:4.4",
 Command = new[]
 {
 "/usr/local/bin/bash",
 },
 Args = new[]
 {
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done",
 },
 Resources = new Kubernetes.Types.Inputs.Core.V1.ResourceRequirementsArgs
 {
 Requests =
 {
 { "cpu", "1" },
 },
 },
 VolumeMounts = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.VolumeMountArgs
 {
 Name = "persistent-storage",
 MountPath = "/data",
 },
 },
 },
 },
 Volumes = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.VolumeArgs
 {
 Name = "persistent-storage",
 PersistentVolumeClaim = new Kubernetes.Types.Inputs.Core.V1.PersistentVolumeClaimVolumeSourceArgs
 {
 ClaimName = pvc.Metadata.Apply(metadata => metadata.Name),
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8SProvider,
 });
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.awsx.ec2.Vpc;
import com.pulumi.awsx.ec2.VpcArgs;
import com.pulumi.awsx.ec2.inputs.SubnetSpecArgs;
import com.pulumi.eks.Cluster;
import com.pulumi.eks.ClusterArgs;
import com.pulumi.eks.inputs.AutoModeOptionsArgs;
import com.pulumi.kubernetes.Provider;
import com.pulumi.kubernetes.ProviderArgs;
import com.pulumi.kubernetes.core_v1.Namespace;
import com.pulumi.kubernetes.core_v1.NamespaceArgs;
import com.pulumi.kubernetes.meta_v1.inputs.ObjectMetaArgs;
import com.pulumi.kubernetes.core_v1.ConfigMap;
import com.pulumi.kubernetes.core_v1.ConfigMapArgs;
import com.pulumi.kubernetes.apps_v1.Deployment;
import com.pulumi.kubernetes.apps_v1.DeploymentArgs;
import com.pulumi.kubernetes.apps_v1.inputs.DeploymentSpecArgs;
import com.pulumi.kubernetes.meta_v1.inputs.LabelSelectorArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodTemplateSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodSpecArgs;
import com.pulumi.kubernetes.core_v1.Service;
import com.pulumi.kubernetes.core_v1.ServiceArgs;
import com.pulumi.kubernetes.core_v1.inputs.ServiceSpecArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressClass;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressClassArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.inputs.IngressClassSpecArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.Ingress;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.inputs.IngressSpecArgs;
import com.pulumi.kubernetes.storage.k8s.io_v1.StorageClass;
import com.pulumi.kubernetes.storage.k8s.io_v1.StorageClassArgs;
import com.pulumi.kubernetes.core_v1.PersistentVolumeClaim;
import com.pulumi.kubernetes.core_v1.PersistentVolumeClaimArgs;
import com.pulumi.kubernetes.core_v1.inputs.PersistentVolumeClaimSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.VolumeResourceRequirementsArgs;
import com.pulumi.resources.CustomResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 // ...
 var storageClass = new StorageClass("storageClass", StorageClassArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("auto-ebs-sc")
 .annotations(Map.of("storageclass.kubernetes.io/is-default-class", "true"))
 .build())
 .provisioner("ebs.csi.eks.amazonaws.com")
 .volumeBindingMode("WaitForFirstConsumer")
 .parameters(Map.ofEntries(
 Map.entry("type", "gp3"),
 Map.entry("encrypted", "true")
 ))
 .build(), CustomResourceOptions.builder()
 .provider(k8SProvider)
 .build());

 var pvc = new PersistentVolumeClaim("pvc", PersistentVolumeClaimArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("auto-ebs-claim")
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .spec(PersistentVolumeClaimSpecArgs.builder()
 .accessModes("ReadWriteOnce")
 .storageClassName(storageClass.metadata().applyValue(metadata -> metadata.name()))
 .resources(VolumeResourceRequirementsArgs.builder()
 .requests(Map.of("storage", "8Gi"))
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8SProvider)
 .build());

 var statefulWorkload = new Deployment("statefulWorkload", DeploymentArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("inflate-stateful")
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .spec(DeploymentSpecArgs.builder()
 .replicas(1)
 .selector(LabelSelectorArgs.builder()
 .matchLabels(Map.of("app", "inflate-stateful"))
 .build())
 .template(PodTemplateSpecArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .labels(Map.of("app", "inflate-stateful"))
 .build())
 .spec(PodSpecArgs.builder()
 .terminationGracePeriodSeconds(0)
 .nodeSelector(Map.of("eks.amazonaws.com/compute-type", "auto"))
 .containers(ContainerArgs.builder()
 .name("bash")
 .image("public.ecr.aws/docker/library/bash:4.4")
 .command("/usr/local/bin/bash")
 .args(
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done")
 .resources(ResourceRequirementsArgs.builder()
 .requests(Map.of("cpu", "1"))
 .build())
 .volumeMounts(VolumeMountArgs.builder()
 .name("persistent-storage")
 .mountPath("/data")
 .build())
 .build())
 .volumes(VolumeArgs.builder()
 .name("persistent-storage")
 .persistentVolumeClaim(PersistentVolumeClaimVolumeSourceArgs.builder()
 .claimName(pvc.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .build())
 .build())
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8SProvider)
 .build());
 }
}

name: eks-auto-mode-yaml
description: A Pulumi YAML program to deploy a Kubernetes cluster on AWS
runtime: yaml
resources:
 # ...
 storage-class:
 type: kubernetes:storage.k8s.io/v1:StorageClass
 properties:
 metadata:
 name: auto-ebs-sc
 annotations:
 storageclass.kubernetes.io/is-default-class: "true"
 provisioner: ebs.csi.eks.amazonaws.com
 volumeBindingMode: WaitForFirstConsumer
 parameters:
 type: gp3
 encrypted: "true"
 options:
 provider: ${k8sProvider}
 pvc:
 type: kubernetes:core/v1:PersistentVolumeClaim
 properties:
 metadata:
 name: auto-ebs-claim
 namespace: ${ns.metadata.name}
 spec:
 accessModes:
 - ReadWriteOnce
 storageClassName: ${storage-class.metadata.name}
 resources:
 requests:
 storage: 8Gi
 options:
 provider: ${k8sProvider}
 stateful-workload:
 type: kubernetes:apps/v1:Deployment
 properties:
 metadata:
 name: inflate-stateful
 namespace: ${ns.metadata.name}
 spec:
 replicas: 1
 selector:
 matchLabels:
 app: inflate-stateful
 template:
 metadata:
 labels:
 app: inflate-stateful
 spec:
 terminationGracePeriodSeconds: 0
 nodeSelector:
 eks.amazonaws.com/compute-type: auto
 containers:
 - name: bash
 image: public.ecr.aws/docker/library/bash:4.4
 command: ["/usr/local/bin/bash"]
 args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done"]
 resources:
 requests:
 cpu: "1"
 volumeMounts:
 - name: persistent-storage
 mountPath: /data
 volumes:
 - name: persistent-storage
 persistentVolumeClaim:
 claimName: ${pvc.metadata.name}
 options:
 provider: ${k8sProvider}
outputs:
 url: ${ingress.status.loadBalancer.ingress[0].hostname}

Summary

EKS Auto Mode represents a significant step forward in simplifying Kubernetes operations on AWS. Pulumi’s support for this feature allows you to define EKS Auto Mode clusters using your favorite programming language and integrate them with your existing infrastructure code.

To get started with EKS Auto Mode in Pulumi:

Update your AWS providers to the latest versions
Check out our EKS documentation
Join our Community Slack for support and discussions

We are excited to see what you build with EKS Auto Mode and Pulumi!

Fargate vs EC2

Adam Gordon Bell — Mon, 18 Nov 2024 08:42:34 -0500

Building an EKS cluster requires choosing how your containers will actually run - either on EC2 instances you manage or through AWS Fargate’s pod-by-pod approach. The differences can be pretty dramatic in practice. I’m setting up a demo cluster right now using Pulumi, so let me show you what I mean.

Here is my Fargate cluster:

import pulumi
import pulumi_awsx as awsx
import pulumi_eks as eks
import pulumi_kubernetes as k8s

# Create VPC with default CIDR
eks_vpc = awsx.ec2.Vpc("eks-vpc",
 enable_dns_hostnames=True)

# Create minimal EKS cluster
eks_cluster = eks.Cluster("eks-cluster",
 vpc_id=eks_vpc.vpc_id,
 private_subnet_ids=eks_vpc.private_subnet_ids,
 fargate=True,
)

pulumi.export("kubeconfig", eks_cluster.kubeconfig)

And when I look in k9s I see this:

You can see that each pod is on its own node. Even the core-dns pods in the system namespace are each on their own nodes. That is a lot of nodes, and it’s going to keep growing with my cluster.

If I change my cluster to be backed by EC2 nodes, I get something different.

eks_cluster = eks.Cluster("eks-cluster",
 vpc_id=eks_vpc.vpc_id,
 private_subnet_ids=eks_vpc.private_subnet_ids,
- fargate=True,
+ instance_type="t3.medium",
+ desired_capacity=2,
+ min_size=1,
+ max_size=2,
)

Now, there are only two nodes, with the pods distributed among them. Those are my two t3.mediums.

So when should you use Fargate, and when EC2 for EKS? Well, it really all relates to what we are seeing here with the number and size of nodes.

Let me explain.

Bin Packing

Normally, the Kubernetes scheduler has to solve a bin-packing problem—fitting M pods across N nodes based on available resources. This is a straight-up resource optimization problem of balancing CPU, memory, and other resource demands across the cluster.

This can be surprisingly complex in practice. Teams must balance selecting the right instance types for different workload needs while managing multiple node groups. Adding to this complexity are scheduling rules and resource quotas that need careful configuration. Perhaps most challenging is striking the right balance between high resource utilization and maintaining enough headroom for spikes in demand.

With Fargate, AWS sidesteps this challenge by providing a correctly sized bin for each pod. Each Fargate pod runs on its own dynamically provisioned, right-sized mini-environment, where the “bin” (the Fargate instance) is sized to match the pod’s requested resources. This means there’s no need for Kubernetes to optimize resource usage across a pool of shared nodes, as each pod effectively has its own “container” provided by Fargate that fits it’s needs precisely.

In other words, AWS effectively pushes this bin-packing responsibility to itself, so you don’t have to worry ( as much ) about it.

While Fargate abstracts away node management, it’s important to understand that Fargate pods still run on EC2 instances behind the scenes. And each Fargate pod gets its own ENI (Elastic Network Interface) that can sometimes limit your scaling because pods aren’t sharing a network namespace.

Pros and Cons

So yeah, Fargate abstracts away the complexity of managing nodes and scaling clusters by shifting bin-packing to AWS. With its extensive infrastructure, AWS is arguably better positioned to optimize placements at scale. They can monitor their entire cloud landscape, placing Fargate instances dynamically across available resources. This allows them to potentially leverage underutilized resources in ways that would be challenging or impractical at an individual account’s scale.

Responsibility	EC2	Fargate
Instance Types	You pick size and provision nodes	AWS provisions per pod request
Pod Isolation	You prevent noisy neighbors	AWS isolates each pod
Resource Allocation	You find space on nodes	AWS creates dedicated space

Customer vs AWS Responsibility EC2 vs Fargate

Ideally, this frees you to focus on the services rather than on the underlying infrastructure, but in practice, it’s a bit more complex.

Since each pod is treated as a standalone instance, you lose out on the cost efficiencies of co-locating multiple pods on a single node. Even though AWS is “solving” the bin-packing problem globally, to do so, they need to introduce hard barriers, and while noisy neighbors, therefore, aren’t a problem, sharing resources is now impossible. In a way, you are throwing away the run time benefit of containers, the shared kernel virtualization, for something more like VMs. And just like VMs, Fargate pods take longer to start than pods on existing EC2 nodes (typically 30-60 seconds vs near-immediate starts).

A Note on Scaling

Both EC2 and Fargate work with Kubernetes’ Horizontal Pod Autoscaling (HPA), which adds or removes pods based on metrics like CPU usage. The difference? With EC2, you need enough nodes available to handle new pods. With Fargate, AWS automatically provisions resources for each new pod - though you’ll face that 30-60 second startup delay and potential VPC limits since each pod needs its own network interface.

Feature	Fargate	EC2
Cost	Higher per pod	Lower cost per pod
Scalability	Scales on-demand, isolated resources	Requires manual node scaling
Resource Sharing	Not possible	Efficient resource sharing among pods
Startup Time	30-60 seconds	Near instant on existing nodes
Use Cases	Burst workloads, isolated pods	Cost-sensitive workloads, Resource sharing workloads

Maybe this will make more sense with an example.

Workload Example: Static Analysis

Have you ever had to run CPU-intensive batch jobs alongside lighter web services?

In a previous role, when my team and I were tasked with putting our static container image analyzer into production, we went the typical Kubernetes route, throwing it into an EC2-backed K8s cluster. The analyzer was built in Scala, running on the JVM, and was designed to take in work from a Kafka topic. Each analysis ultimately led to the results being sent to a database. We also had scheduled daily rescans of existing images whenever vulnerability lists were updated.

Then, a web app connected to this database and showed all your results. It was stateless and did far less work per request, mainly just answering an HTTP request by talking to a database.

The hard part though is that each analysis job involved uncompressing and analyzing Docker layers from S3 and could use a lot of CPU. Memory needs were around 2 GB per instance, which kept the JVM happy, but long sustained bursts in CPU usage were common, especially during intensive parts of the analysis.

In our K8s setup, the biggest pain point was handling this analysis workload’s burstiness and keeping a bunch of stateless web apps up and answering requests. We ran multiple instances of the static analyzer. Still, the analyzer was the worst type of noisy neighbor to other services in the cluster, so we’d end up over-provisioning to ensure we had enough headroom, which led to wasted resources and extra cost. The setup worked, but was not great.

Thinking back, I can see how Fargate could’ve streamlined things here. With Fargate, each static analyzer instance would run in its own isolated environment, so there’d be no resource contention or need for pre-provisioned node capacity. Each pod would be fully provisioned with the exact CPU and memory needed, and we could do some sort of on-demand scaling to deal with backlogs of analysis work in the Kafka topic.

Fargate on EKS could be a great solution for this type of chunky, resource-heavy macro-services. There are probably other ways to handle this situation, and we worked on decomposing the analysis into smaller chunks, but Fargate might have been a good solution.

But there are lots of situations where Fargate is less of a fit.

You might also like:

Example: Go Services for E-commerce

Ok, another workload that comes to mind is this one from a friend - He worked on an e-commerce backend with many lightweight Go services, each handling specific tasks—user profiles, catalog browsing, order processing, and notifications. These services were network-heavy, handling gRPC calls and database requests, but they didn’t need much CPU or memory per instance.

Fargate wouldn’t have made sense here—it would have isolated each pod, preventing resource sharing and driving up costs without real benefit. For lightweight services like these, EC2-backed Kubernetes is perfect, keeping costs low and maximizing resources by sharing nodes across small, efficient services.

So, for lightweight micro-services that can share resources, EC2 nodes with many pods each are probably the way to go.

Other Fargate Cases

Fargate can excel with long-running event-driven and bursty workloads, where you need compute resources briefly or unpredictably. Think file processing jobs that trigger on uploads, or batch processing that runs sporadically. When jobs are intermittent, Fargate’s ability to scale from zero and its isolated resources can outweigh its higher per-pod cost and startup delay. Just ensure your workload can tolerate the 30-60 second cold start.

Fargate has some hard limitations that might rule it out for your use case:

No DaemonSets
Maximum of 4 vCPU and 30GB memory per pod
Storage limitations: Only ephemeral storage up to 20GB per pod, no persistent volumes
No GPU support

Fargate vs EC2 Pricing

Ok, EC2 costs less than Fargate on a pure cost-of-compute basis. If my t3.medium ($0.0416/hr) can typically run 6-8 pods reliably, and this works out to under $0.01/hr per pod. Comparable Fargate pods (0.5 vCPU/1GB) cost about $0.025/hr each, so 2.5x.

These calculations can change based on how much headroom you need on EC2 and how many containers you can run on a given EC2 instance. Still, Fargate is just more expensive unless all your containers fit the exact computing sizes offered by Fargate. And don’t forget, AWS is likely incurring costs maintaining the warm compute capacity for launching Fargate pods, so it shouldn’t be surprising that Fargate is more expensive.

But really, pure compute cost calculations are an insufficient metric for making this decision - you need to factor in your team’s operational capacity, scaling patterns, and isolation requirements.

( Also, if Fargate makes sense because of your workload but the cost is a concern, then look into Fargate Spot instances, which can be up to 70% cheaper, but be aware that AWS can terminate them at any point to reclaim capacity. )

Misconceptions About Fargate

Fargate saves you from managing servers With containerized workloads and immutable infrastructure, your EC2 nodes are minimal, standardized, and automatically replaced. You’re not logging in, installing packages, or managing configuration - everything runs in containers. When a node needs updates, you replace it entirely.
Fargate handles all scaling challenges While it handles node scaling, you still face ENI limits, longer pod startup times (30-60s), and potential networking bottlenecks.
It’s simpler and more reliable It may be, but it may not be. The complexity shifts to networking, pod configuration, and working within Fargate’s constraints. You trade node management for new operational challenges like slower startups and possible networking complexity.

Managing Container Orchestration with Pulumi

Whether you choose EC2 or Fargate, Pulumi makes it easy to manage in your favorite programming languages. With its rich AWS support and infrastructure as code capabilities, you can define, deploy, and manage your entire Kubernetes infrastructure using familiar tools and practices. Pulumi’s AWS provider helps you implement the right solution for your needs.

With Pulumi, you get:

Complete control over EKS cluster configuration, including node types, scaling policies, and networking setup
Built-in support for both EC2 node groups and Fargate profiles within the same cluster
Infrastructure testing capabilities to validate your cluster configuration before deployment
Automation APIs for programmatic infrastructure management and custom tooling

Ready to streamline your EKS infrastructure management?

Try Pulumi for Free

Why Not Both

I hope my examples point to a pragmatic approach: use EC2 as your foundation for efficient microservices that can share resources, like our Go e-commerce setup. Add Fargate when you need isolation or flexible scaling - like for the static analysis workload. Better yet, combine both approaches in the same cluster, for cost-efficiency and isolation where needed.

If you’re interested in using Pulumi to manage EKS infra, here’s a practical EKS setup guide guide to get your started.

Now, back to getting this demo cluster up and working!

Pulumi EKS Provider Version 3.0.0

Florian Stadler — Thu, 17 Oct 2024 17:32:03 +0200

The latest major release of the Pulumi EKS Provider is available now!

This release delivers significant improvements in flexibility, security and introduces new features to enhance your Kubernetes experience on AWS. AWS recently announced the deprecation of two features used by default in Pulumi EKS: the aws-auth ConfigMap and the AL2 operating system. Pulumi EKS v3 addresses these deprecations, enhances the maintainability of the provider, and aligns it with EKS best practices.

Here are a few links to help you get started if you are new to Pulumi:

EKS Setup & Installation - Instructions on installing and configuring the Pulumi EKS provider
EKS How-to Guides - Learn how to use the EKS provider to provision and manage Kubernetes clusters on AWS
Pulumi Neo - Ask Pulumi Neo to help you create a new EKS project

Key Highlights of EKS V3

Support for Amazon Linux 2023 (AL2023) and Bottlerocket Operating Systems: Enhanced operating system options for node groups, allowing you to choose the OS that best fits your workloads and compliance needs. This addresses the upcoming deprecation of Amazon Linux 2 (AL2).
Access Entries for IAM Integration: Enables replacement of the deprecated aws-auth ConfigMap with Access Entries for managing Kubernetes authentication.
EKS Managed Addons: Simplified management of vpc-cni, coredns, and kube-proxy as EKS managed addons.
EKS Security Groups for Pods and Network Policies: Enhanced network security and control within EKS clusters.

New Features and Improvements

Support for Amazon Linux 2023 and Bottlerocket

We have expanded the operating system options for node groups in EKS v3 to address the upcoming deprecation of Amazon Linux 2 (AL2). You can now choose between Amazon Linux 2 (deprecated), Amazon Linux 2023 and Bottlerocket for your EKS nodes. This flexibility allows you to select the OS that best fits your workloads, security requirements, and compliance needs, while ensuring you are using a supported and actively maintained operating system. We’ve introduced a new operatingSystem property for node groups to facilitate this choice.

Access Entries for IAM Integration

AWS has introduced Access Entries as a new method for granting IAM principals access to Kubernetes resources. This approach relies solely on AWS resources for managing Kubernetes auth, replacing the deprecated aws-auth ConfigMap. You can now leverage Access Entries by setting the authenticationMode to API in your cluster configuration.

EKS Managed Addons

The EKS cluster components vpc-cni, coredns, and kube-proxy are now configured as EKS managed addons. This change simplifies management, especially for clusters with private API endpoints, and ensures that these critical components stay up to date automatically. Additionally it removes the dependency on kubectl, allowing pulumi-native management of clusters.

Cluster Autoscaler Integration

Pulumi EKS v3 introduces better support for the Kubernetes Cluster Autoscaler. A new ignoreScalingChanges parameter for node groups allows Pulumi to ignore external scaling changes, facilitating seamless integration with dynamic scaling solutions.

EKS Security Groups for Pods and Network Policies

We’ve added support for EKS security groups for pods (example) and EKS Network Policies (example), providing more granular control over pod-to-pod and pod-to-external network communication within your EKS clusters.

Migration Guide

To help you transition smoothly, we’ve prepared a migration guide with these key steps:

Update node groups to use AL2023 or explicitly configure AL2 if needed.
Replace the deprecated NodeGroup component with NodeGroupV2.
Update your code to handle new output types for certain properties.
Review and update your use of default security groups, which can now be disabled.

Please refer to our EKS v3 Migration Documentation for a detailed guide.

Conclusion

Pulumi EKS v3 represents a significant step forward in managing Kubernetes clusters on AWS, empowering you to build and manage more robust and efficient EKS clusters.

We encourage all users to upgrade to this latest version to take advantage of these improvements and ensure your EKS deployments remain secure and up-to-date.

For more information on getting started with Pulumi EKS v3, check out our documentation or join our community Slack for support and discussions.

Happy clustering with Pulumi EKS v3!

Deploy Low-Code LLM Apps on AWS with Flowise and LocalAI

Engin Diri — Mon, 26 Feb 2024 06:00:00 -0800

Building low-code LLM applications just got easier! With Flowise, LocalAI, and Pulumi, you can quickly create, deploy, and manage AI-powered workflows on AWS EKS—without writing complex code.

In this guide, you’ll learn how to:

Set up a low-code LLM application using Flowise & LocalAI
Deploy it on AWS EKS with Pulumi & TypeScript
Create a chatbot workflow and expose it via API

Flowise simplifies LLM orchestration with a drag-and-drop interface, while LocalAI enables local inference without requiring a GPU. Pulumi streamlines cloud infrastructure deployment with just a few lines of code.

Whether you’re a developer looking to experiment with LLMs on AWS or an AI enthusiast exploring serverless AI applications, this step-by-step tutorial will guide you through the entire process. Let’s dive in!

Flowise: The Low-Code Tool for LLM Orchestration

Flowise is an open-source low-code/node tool for developers to build customized LLM orchestration workflows and AI agents. The low-code approach helps developers to cut down on the time and effort required to build LLM applications by delivering quick iterations to from testing to production. In Flowise nodes are referred to as integrations, and you can build a customized LLM orchestration flow, chatbot, or AI agent by dragging and dropping these integrations. There are plenty of ready-to-use integrations available in Flowise, like LangChain or LlamaIndex.

LocalAI: Open-Source LLM Inference for AWS & On-Prem

LocalAI is a free and Open Source alternative to OpenAI. It acts as a drop-in replacement for OpenAI’s REST API that is fully compatible with the OpenAI API specifications for local inference. It allows you to run LLMs, generate images, or audio locally or on-prem with consumer grade hardware supporting multiple model families and architectures. And the clue: It does not require a GPU, it runs on CPU only (optional GPU acceleration is still available).

Once LocalAI loaded the model the first time, it keeps it in memory for faster inference. It does not shell-out, but use bindings to the model for faster inference and better performance. It is a great way to test and benchmark different models directly on your local machine or on-prem. Additionally, you are not sending your data to a third-party service, which may contain sensitive information. This is a big advantage, especially in large enterprises or in the public sector where data privacy is a big concern.

Deploying Flowise & LocalAI on AWS with Pulumi

Covered a lot of the basics, but let’s get started with the real thing. To quickly get started, you can use the example project from the Pulumi examples repository.

The AWS architecture of the example project is as follows:

The Pulumi AWS architecture consists of the following resources:

An EKS cluster with a managed node group.
EBS volume support for the EKS cluster to store the downloaded LocalAI models.
Deployment of the Helm chart for Flowise and LocalAI.
A VPC, two subnets in different availability zones, and an internet gateway, and security groups for the EKS cluster.

As programming language, I will use TypeScript, but you can use any other language supported by Pulumi. Rewriting the example code should be straightforward. Feel free to reach out to us if you need help.

Create a Flowise Chatbot

Our first Flowise workflow will be a simple chatbot. This simple LLM chain will be stateless (no memory) with a prompt template and our LocalAI LLM model attached.

Deploy the Infrastructure with Pulumi

First clone the example repository, and head to the aws-ts-localai-flowise directory:

git clone https://github.com/pulumi/examples.git
cd examples/aws-ts-localai-flowise

To deploy the infrastructure with Pulumi, you can use the following command:

pulumi login # if you haven't logged in yet
pulumi up

Next to the definition of the AWS EKS cluster, the Pulumi program uses the pulumi-kubernetes provider to deploy the Helms charts for Flowise and LocalAI.

// imports omitted for brevity
import * as k8s from "@pulumi/kubernetes";

// code omitted for brevity

// Manually creation of the Kubernetes provider, with the kubeconfig from the EKS cluster and the enableServerSideApply set to true
const provider = new k8s.Provider("k8s", {
 kubeconfig: cluster.kubeconfigJson,
 enableServerSideApply: true,
});

// Deploy the Helm charts for LocalAI
const localai = new k8s.helm.v3.Release("local-ai", {
 chart: "local-ai",
 version: "3.2.0",
 repositoryOpts: {
 repo: "https://go-skynet.github.io/helm-charts",
 },
 forceUpdate: true,
 namespace: "local-ai",
 createNamespace: true,
 values: {
 deployment: {
 image: {
 repository: "quay.io/go-skynet/local-ai",
 tag: "latest",
 },
 env: {
 debug: "true",
 context_size: 512,
 modelsPath: "/models",
 },
 }, resources: {
 requests: {
 cpu: "8",
 memory: "32Gi",
 },
 },
 models: {
 list: [
 {
 url: "https://gpt4all.io/models/ggml-gpt4all-j.bin",
 name: "ggml-gpt4all-j",
 },
 ],
 },
 persistence: {
 models: {
 size: "50Gi",
 storageClass: "gp2",
 accessModes: "ReadWriteOnce",
 },
 output: {
 size: "10Gi",
 storageClass: "gp2",
 accessModes: "ReadWriteOnce",

 },
 },
 },
}, {
 provider: provider,
});

// Deploy the Helm charts for Flowise
const flowise = new k8s.helm.v3.Release("flowise", {
 chart: "flowise",
 version: "2.5.0",
 repositoryOpts: {
 repo: "https://cowboysysop.github.io/charts/",
 },
 namespace: "flowise",
 createNamespace: true,
}, {
 provider: provider,
});

Depending on your AWS region, the deployment can take a few minutes. Once the deployment is complete, you will have access to kubeconfig file to access the EKS cluster for the next steps.

To retrieve the kubeconfig file, you can use the following command:

pulumi stack output kubeconfig --show-secrets > kubeconfig.yaml

Now you can use the kubeconfig.yaml file either with kubectl or with the KUBECONFIG environment variable to access the EKS cluster.

As we are not using a load balancer, we will use the kubectl port-forward command to access the Flowise UI. The command is:

kubectl port-forward svc/$(kubectl get svc -n flowise | grep flowise | awk '{print $1}') 3000:3000 -n flowise

Start modeling the Chatbot with Flowise

Open the Flowise UI in your browser with the following URL: http://localhost:3000 and click the Chartflows menu item in the left sidebar. And then click the blue button + Add New

This should open a new window where you can drag and drop the nodes to the main window. To display the available nodes, click the + button on the left.

First we add our LocalAI node to the chartflow. Look under Chat Models for the ChatLocalAI node and drag it to the main canvas.

Fill out the form with the following values:

Base Path: This should be name ouf your LocalAI Kubernetes service. You can find it with the following command:
```
kubectl get svc -n local-ai
```
And should be in the form of http://<svc-name>.local-ai.svc.cluster.local:80
Model Name: The name of the model you want to use. You can find the available models with the following command:
```
kubectl exec -it $(kubectl get pods -n local-ai | grep local-ai | awk '{print $1}') -n local-ai -- curl http://localhost:8080/models | jq '.data[0].id'
```
This will return the name of the model you can use. For example ggml-gpt4all-j_f5d8f27287d3
Temperature: The temperature of the model. This is a float value between 0 and 1. The higher the value, the more creative the model will be. The lower the value, the more conservative the model will be. Keep the default value of 0.9 for now.

Now we can add the next node to our workflow. Search for Prompt Template and drag this node to the main canvas as well. You can fill out the Template field with the following value:

Question: {input}.

As a last step we need to add the LLM Chain node to the workflow. Head over to the Chains section of our nodes menu and drag the LLM Chain node to the main canvas. Give the chain a name chatbot.

This done, we can start to connect different nodes to each other. Start with the Prompt and Language Model nodes by dragging the arrow from the Language Model dot to the ChatLocalAI node. Same for the Prompt dot to the Prompt Template node.

Now we can save the chartflow and test it. Click the Save icon in the top right corner and then click the Dialog button to display the chatbot. You can now test the chatbot by typing a question in the input field and see the response. In my case, I asked the chatbot How does an airplane fly in the air? and got the following response:

There are several methods by which an airplane flies in the air, including:

Lift and Weight: An airplane flies by exploiting its weight in the vertical direction. Therefore, it is necessary for an aircraft to have a strong and heavy enough structure that can hold its weight in the air. The more an airplane weighs, the greater is its lift and climb capability.
Air Density: An airplane exploits the difference in air density below and above it. Airborne objects experience the same buoyant force that causes them to rise. By doing this, an aircraft can maintain a stable position and ascend or descend.
Airfoil Design: The shape of an aircraft’s airfoil is designed such that it allows for the efficient and effective generation of lift force. The shape, thickness and angle control the air flow over the wings of an aircraft.
Control Arm: The control bar controls the angle of attack, pitch and yaw control. By moving the bar, pilots can change and adjust these features of the aircraft.
Propulsion Systems: An airplane requires propulsion systems to provide the thrust needed for takeoff, climb and movement through the air. Engines such as piston, turbojet, diesel engine or rocket engines are commonly used. The combustion process in these systems produces heat, which provides thrust.
Radio Communication: Aircraft can communicate with air traffic control in real-time, allowing pilots and air traffic controllers to coordinate their actions. This communication is essential for safe and efficient air travel.

Go ahead and test your chatbot with different questions! It is a great way to test different LLM models and see how they behave.

Expose the Chatbot via API

If you want to expose the chatbot workflow via API or SDK, you can use Flowise for this as well. Flowise provides a REST API for this purpose. Head in your workflow view to the top right corner and click the </> button. This will open a new dialog with all the different methods you can use. This range from embedding the chatbot in your website to using to directly call the REST API.

There is even the option to enable authorization for the REST API. This is a great way to secure your chatbot and only allow authorized users to access it.

Housekeeping

If you want to destroy the stack, you can use the following command:

pulumi destroy

The Pulumi CLI will ask you to confirm the changes, and if you are happy with the changes, you can confirm them by typing yes and all the resources will be deleted.

In Conclusion

In this blog post, we have seen how easy it is to build a LLM application with Flowise and LocalAI without the need to use any code. This is a great way to quickly prototype and test different LLM models and workflows. We have also seen how easy it is to deploy the infrastructure with Pulumi and how easy it is to destroy the stack if we don’t need it anymore.

Want to get hands on with LangServe and walk through it live? We have two upcoming dates for a workshop focused on LangChain and LangServe!

Deploying LangChain Applications on AWS with Pulumi on March 14, 2024
Deploying LangChain Applications on AWS with Pulumi again on April 17, 2024

As always, we welcome your feedback and contributions in the Pulumi Community Slack, GitHub repository, and Pulumi Community Discussions.

New to Pulumi? Signing up is easy and free. Get started today!

Happy AI building!

IaC Best Practices: Enabling Developer Stacks & Git Branches

Christian Nunciato — Fri, 10 Mar 2023 00:00:00 +0000

In the first post about code organization and stacks, we introduced Zephyr, a fictional company that uses Pulumi to manage its online retail store. Following on from that post, which discusses code organization and stacks, this post explores two more questions users frequently ask when working with Pulumi in teams — namely, How can I best enable multiple developers to collaborate on a Pulumi project? And how can I use Git and Git branching to support this kind of collaboration? In this post, we’ll provide some guidance and best practices around these topics, using Zephyr and its online store as the use case.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series — not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and its retail application changes to accommodate its growth.

Existing IaC Workflow

When we last met up with the Zephyr team, they were off and running, managing their newly refactored online store, Zephyr Archaeotech Emporium, with a single Pulumi project and two Pulumi stacks — one for development (dev) and another for production (prod). The team had chosen to use one Git repository (a monorepo) to manage the code for the online store and its infrastructure after refactoring the store into a set of containerized microservices deployed with Kubernetes on Amazon EKS.

For a snapshot of the code as it was at the end of the first post in the series, see the blog/original branch of the pulumi/zepyhr-app repository on GitHub.

Up to now, the two-stack approach has worked well for the team. It’s allowed them to focus on building new features, integrating those features into the shared dev environment by developing and testing locally (which we’ll cover in a future post) and merging their feature branches regularly into the base branch, main, using standard GitHub pull-request workflows. Around once a day, as the team’s changes are merged into main, the team lead (and original author of the Pulumi program) deploys them into the dev environment with Pulumi with the following commands:

git pull origin main
cd infra
pulumi stack select zephyr/dev
pulumi up

This builds new container images for any services whose code has changed since the last stack update, pushes those images to Amazon ECR, and updates the EKS cluster to deploy the services automatically.

Then, around once a week, as new features are ready for customers, the team lead deploys the main branch of the repository into production similarly:

git pull origin main
cd infra
pulumi stack select zephyr/prod
pulumi up

The deployment process is still a bit more manual than the team would like — but since updates typically take only a few minutes, the team lead is generally satisfied with the status quo.

A few more details worth noting out about Team Zephyr’s current setup:

Both of the project’s stack-configuration files, Pulumi.dev.yaml and Pulumi.prod.yaml, are checked into version control. This is safe and considered good practice (even with secrets) because it allows the team to review or change the configuration of a given stack easily by viewing or editing the code in each file.
The main branch is the only long-lived branch in the repository; all others (e.g., developers’ feature branches) are deleted once they’re merged into main.
Thanks to this workflow, the application deployed into the dev environment tracks closely with the latest code in the main branch. As such, the dev stack serves as a kind of pre-prod or integration environment that not only allows the team to dogfood new features as they’re added, but also to catch bugs or other regressions before they escape into production. When things do go wrong, the team can review the list of recent stack updates to identify the source of the defect and get it fixed.
Zephyr is using the Pulumi Service backend with a single organization (that everyone on the team belongs to) and separate AWS accounts for the dev and prod environments. Everyone on the team has read/write access to the AWS account used for development, but only the team lead has access to the AWS account used for production.

Growing Pains

Life for Team Zephyr was good — for a while.

Then one day during a particularly active sprint, a developer who’d started working on an improvement to the checkout process realized it might benefit from an updated version of Redis, which appeared to be presently pinned at 6.0 in the Pulumi program:

// ...
const checkoutRedisDeployment = new k8s.apps.v1.Deployment("checkout-redis-deployment", {
 spec: {
 template: {
 spec: {
 containers: [{
 image: "redis:6.0-alpine",
 // ...

A quick glance at the dev and prod stacks in the Pulumi Service confirmed this was the case.

After a quick spike to validate the enhancement locally, the developer felt confident that a Redis upgrade was the way to go — but since the deployed version of Redis would ultimately dictate the code that’d have to be written to finish the feature for real, they couldn’t really be sure without testing the theory with a full deployment. What they really wanted was to be able to deploy a Redis upgrade, along with the associated code changes, into an AWS environment as much like production as possible — the dev environment, say — without having to go through the process of making a pull request, getting it merged, and waiting for the team lead to deploy the dev stack only to realize later that the change didn’t actually work as expected, and be forced to revert.

In situations like these, it can be tempting to suggest that the developer simply pick up and use the dev stack — especially on small, fast-moving teams like Zephyr’s where everyone already has access to the Pulumi stack and the shared cloud environment. Indeed, Pulumi makes this sort of thing easy: to deploy their local changes into the dev environment, our developer need only check out their feature branch, select the dev stack, run pulumi up, and then wait a few moments for the changes to be deployed:

cd infra
pulumi stack select zephyr/dev
pulumi up

This approach definitely works — but it presents at least a few problems:

The team would lose all of the practical benefits of having the dev stack track closely with the latest content in main.
The team lead’s deployment of main into dev later that day would overwrite the developer’s own feature-branch deployment. There’s a good chance the lead would notice the diff of the deployment preview and deduce that someone else on the team had possibly deployed into dev out of band:
```
 kubernetes:apps/v1:Deployment (create):
 spec: {
 template: {
 spec: {
 containers: [
 [0]: {
- image: "redis:6.0-alpine"
+ image: "redis:7.0-alpine"
 ...
 }
 ]
 }
 }
 }
 ...
```
But if they didn’t, the developer’s ad-hoc deployment would be lost. At best, it’d force both team members to drop what they were doing and get in touch to figure out what to do next. At worst, it could lead to confusion and the potential for shipping untested code.
In addition to raising the risk of developers overwriting each other’s work, active collaboration the same stack increases the likelihood of deployment conflicts. On the one hand, this is a good thing in that Pulumi prevents same-stack updates from running concurrently:
```
$ pulumi up
error: [409] Conflict: Another update is currently in progress.
```
But on the other, it’s a not-so-good thing in that it’s a reduction in development velocity: The update that can’t run because another update is in progress is an update that will have to wait until sometime later. The more often this happens, the longer it ultimately takes for the team to get the validation it needs to keep moving forward.

So facing these issues, what should the Zephyr team do? The dev stack had originally been created for rapid development — and that worked well, when there was only one developer working on infrastructure. With several, however, things clearly get complicated. Should the team be using a dev stack at all? If so, how? And if not, what should they be doing instead?

Stepping Back to Reassess

With what we’ve learned so far, we can return to the questions we raised at the beginning of this post and begin to sketch out a few recommendations for Team Zephyr. Those questions, again, were:

How can I best enable multiple developers to collaborate on a Pulumi project? For Zephyr, an ideal answer to this question would give every member of the team the ability to iterate on both application and infrastructure code concurrently, deploying as often as needed without having to worry about disrupting the work or the velocity of anyone else on the team.
How can I use Git and Git branching to support this collaboration? So far, the feature-branch/pull-request workflow has been serving the Zephyr team well. Should they continue on the same path, merging frequently to main and deploying to dev and prod from there? Or is there some alternative workflow that might serve the team better, either now or at some time in the future?

Let’s answer these questions now by taking a look at how the Zephyr team handled them.

Using Developer Stacks

We’ve seen a few ways that having multiple developers share a single stack for ad-hoc deployment can lead to various problems. But we’ve also seen at least one situation in which being able to run ad-hoc deployments — without the risk of colliding with existing environments — can be quite valuable. In fact there are many reasons you might want to do this — and fortunately Pulumi makes it quite easy to do so.

The best practice is to allow developers to create and manage their own Pulumi stacks. In this workflow, developers working on feature branches are able to iterate freely on both application and infrastructure, deploying as often as necessary into a shared cloud environment (typically a sandbox environment, isolated from production) and destroying cloud infrastructure when it’s no longer needed.

For the developer, the typical workflow looks something like this:

Check out a new feature branch
Make changes to application and/or infrastructure code
Select your personal developer stack with pulumi stack select
Deploy your stack into your team’s shared cloud environment with pulumi up
Commit your code and submit a pull request, optionally including a link either to the Pulumi update (e.g., when using the Pulumi Service) or to certain key resources belonging to the stack — for example, an application’s primary HTTP endpoint

The benefits of a workflow like this one are many:

No more resource-naming collisions. Because Pulumi names cloud resources uniquely by default, no two stacks will ever produce cloud resources with the same names.
No more deployment conflicts. When every developer has a stack of their own, they can deploy to the sandboxed environment as often as necessary, without having to wait for another deployment to finish first.
Everyone on the team has their own production-like environment to use for end-to-end testing.

For Zephyr, the quickest path to adopting this workflow was to use the dev stack as a foundation for new developer stacks. To do that, team members simply ran the following Pulumi commands:

pulumi stack init zephyr/developer-name --copy-config-from zephyr/dev
pulumi up

Developer stacks are considered ephemeral, so they’re torn down with pulumi destroy when not in use in order to cut down on cost. Stack configuration files (e.g., Pulumi.developer-name.yaml) are checked into version control and stored alongside other stacks like Pulumi.dev.yaml and Pulumi.prod.yaml.

Keeping Feature-Branch Workflows

We’ve also seen that the Zephyr team’s existing Git workflow (short-lived feature branches merged frequently into a long-lived base branch) has worked well for them — and it turns out this workflow also works well when combined with the use of developer stacks.

For example, after creating and deploying a new stack to validate and test the proposed Redis upgrade, our Zephyr developer simply included a link to the application (deployed with their dev stack) in the associated GitHub pull request. Reviewers were then able to follow the link to the online store to see the effects of the shopping-cart change firsthand, which meant they were able to approve and get the change merged quickly and confidently knowing it’d work when deployed into production.

Naming Stacks Clearly

Lastly, having adopted the use of developer stacks, the Zephyr team realized the name “dev” probably didn’t work quite as well as it once had for the dev stack. Since the stack and its long-running deployment had essentially been serving (and continued to serve) as an environment for testing changes before releasing them into production, the team decided to rename the dev stack test instead. To do that, they simply ran:

pulumi stack select zephyr/dev
pulumi stack rename zephyr/test

Viewing the Next Iteration of Code

You can view the second iteration of Zephyr’s Pulumi and application code — the iteration that corresponds to the decisions described in this blog post — by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to the blog/dev-stacks branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Developer Stacks and Git Branches

In this post, we covered several new guidelines for working collaboratively with Pulumi, all aimed at optimizing for high team and developer velocity:

Use short-lived feature branches and merge them frequently into a single base branch.
Deploy the base branch regularly into a long-running pre-prod environment to continuously test the integration of the components of the system.
Enable developers to create and use dev stacks of their own as sandboxes for development, testing, and collaboration.
Name your stacks clearly and in alignment with the environments they deploy into — test, staging, production, etc.

The next Zephyr blog post will dive into how the Zephyr team structures their Pulumi projects. Stay tuned!

IaC Best Practices: Understanding Code Organization & Stacks

Aaron Kao — Mon, 20 Feb 2023 00:00:00 +0000

This is the first in a series of blog posts that explores how a fictional company—Zephyr Archaeotech Emporium—uses Pulumi to manage their online retail store. This post explores a couple of common questions that users ask when working with Pulumi; specifically, where should I store my Pulumi code? And how do I support multiple environments with Pulumi? This post will provide some guidance and Infrastructure as Code best practices around these topics, using Zephyr and their online store as the use case.

The ultimate goal of this series is to discuss recommended practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series—not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.

Setting Up the Scenario

Zephyr is short for Zephyr Archaeotech Emporium, the fictional company in our scenario. Zephyr is an online retailer that specializes in the sale of “rare arcane artifacts and replicas.” Over the past few years, the company has experienced a significant increase in its online presence, making it easier for customers to purchase unique and mysterious items. Zephyr’s collection includes a variety of rare and unusual objects that are difficult to find elsewhere, making it a popular destination for collectors, enthusiasts, and adventurers.

Moving to a Containerized Architecture

Zephyr’s application is the second generation of their online store. Their first generation was a monolith that they manually deployed. It was sufficient in Zephyr’s early days, but as word about Zephyr spread and the volume of traffic to their online store increased, Zephyr found it increasingly difficult to deploy small, incremental changes to the store’s code. As a result, the velocity of their development efforts slowed. In preparation for the next phase of their growth—driven by robust sales of arcane artifacts—Zephyr’s team evaluated a number of different architectures that would enable them to increase development velocity and give them greater flexibility in scaling different aspects of their online store. In the end, they settled on a containerized architecture deployed to Kubernetes because some of the existing team was already familiar with these technologies.

Here’s a diagram of their second-generation application architecture:

As you can see in the diagram, the application has been broken down into microservices. The UI service communicates with other backend services via HTTP APIs; the various backend services communicate with various databases like MySQL and Redis for data persistence and/or caching. The backend services are responsible for implementing the functionality of a typical e-commerce site like orders, checking out, and managing the inventory catalog. While the choice of Kubernetes does afford Zephyr some much-needed functionality—like the ability to easily scale different services within the application—it also brings with it an added level of complexity that did not exist with their previous monolithic architecture. The Kubernetes manifest to deploy their application tops out at just over 1,000 lines of YAML!

Astute readers may note that Zephyr’s application looks very similar to this application; in truth, this series uses a fork of the AWS Containers retail store example as the application in this fictional scenario. (Many thanks to the AWS Containers DA team for their hard work here!) This is an admittedly over-engineered application, but it does afford the opportunity to address a range of customer scenarios in this blog series.

Managing Infrastructure and Application Deployments

As part of the switch to their new architecture, Zephyr decided they wanted to use Pulumi to manage both the infrastructure and the application deployment. Why Pulumi? Zephyr’s team recognized that moving to microservices on Kubernetes was going to make it more difficult to create reproducible, consistent implementations of their application. Zephyr’s team also knew that adopting infrastructure as code would help them with fast and repeatable deployments with little additional complexity. Another bonus was being able to use programming languages they already knew.

As they prepared to embark on using Pulumi, a couple of questions came up for the Zephyr team:

Where should they store their Pulumi code? The ability for Zephyr’s developers to use a full programming language to manage the infrastructure and deployment of their online store is a huge plus, but should this code go in the same repository as their application code? Or a different repository?
How do they address the need for multiple environments? It’s clear to the Zephyr development team that they’ll need multiple instances of the online store (for things like testing, QA, or development). What’s the best way to handle this when using Pulumi?

Let’s examine each of these issues in a bit more detail, and see how Zephyr chose to proceed.

Storing Pulumi Code

When it comes to answering the question of where to store Pulumi code relative to the application(s) it supports, there are two basic options:

In the same repository as your application code: This approach has the advantage of being simpler, but may not offer the necessary flexibility when dealing with multiple teams or varying levels of access control.
In a separate repository from your application code: Using a separate repository adds some complexity, but does provide greater flexibility to more easily address a variety of organizational requirements and varying levels of access control.

So which approach is best? That will depend on a number of different factors, many of which are outlined in the organizing projects & stacks documentation. Some other factors that users need to take into consideration include:

Who is responsible for maintaining the code? If the application code and the Pulumi code are managed by different teams, then using separate repositories may be the best approach.
What sort of access controls are needed? If the organization needs different access controls for the application code and the Pulumi code, then separate repositories are generally needed.
What is the relationship between the infrastructure resources managed by Pulumi and the applications? If the infrastructure is “shared,” meaning it is used by multiple applications, then keeping the Pulumi code in a separate repository may be preferred. If the infrastructure is dedicated to a specific application, then keeping the application code and the Pulumi code together may be preferred.
What about CI/CD? If an organization will use a CI/CD pipeline, that may affect how you organize your code. (This is something we’ll delve into more deeply later in this series.)

It’s worth stressing that the answers to these questions may change over time as organizations grow and their applications evolve. Addressing that change and its effects on an organization’s Pulumi projects and stacks is something we’ll be discussing later in this series.

The discussion above is working from the assumption that Pulumi users will store their Pulumi code in a version control system, like Git. Strictly speaking, using a version control system isn’t required for Pulumi to function (there is one exception we’ll touch on later in this series when we discuss Pulumi Deployments), but we very strongly recommend that all users use Pulumi in conjunction with version control.

In the case of Zephyr, their team felt like a monorepo approach (storing Pulumi code and application code together) was most appropriate for their specific requirements. Zephyr is a relatively small organization with a single team of developers who are responsible for managing both the cloud infrastructure as well as the development and deployment of the online store application. Zephyr’s team also decided to use a single Pulumi project—for now. (Be sure to follow this blog series to see how that evolves, and why!)

Addressing the Need for Multiple Environments

The second question Zephyr encountered is how to handle the need for multiple instances of their application. In addition to a production instance—which is the instance behind Zephyr’s public-facing online store—Zephyr also felt they needed an environment for the developers to use in testing changes to the online store.

This use case—needing to have multiple, separate instances of the infrastructure and applications created by a single Pulumi program—is exactly what Pulumi stacks were designed to address. Each stack is a separate instance of the resources created by a Pulumi program within a project. Further, each stack has its own independent state, and each stack has its own configuration values. Stacks can be short-lived (meaning the associated resources are also short-lived), or stacks can be long-lived. Aside from the cloud resources created by the Pulumi program, stacks are lightweight and simple to create or delete.

Zephyr decided to initially start with two stacks: a production stack (named “prod”) and a development stack (named “dev”). As you’ll observe throughout this series, this is a decision that is easily adjusted over time as your organization’s requirements change.

Organize your infrastructure with stacks

Model separate development, staging, and production environments as Pulumi stacks in a single project, each with its own state and configuration.

Get started

Viewing the First Iteration of Code

You can view the first iteration of Zephyr’s Pulumi and application code—the iteration that corresponds to the decisions described in this blog post—by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to viewing the blog/original branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Organizing Code and Stacks

This article introduced you to Zephyr, the fictional company for this blog series, and reviewed the application architecture for Zephyr’s online store (a containerized application running on Kubernetes). Throughout the blog series, this application serves as the use case around which various recommended practices are discussed.

In this post the following guidelines were covered:

Use version control for your Pulumi code. Although it isn’t strictly required in many cases, a number of the benefits of adopting infrastructure as code are lost without the use of version control.
Place your Pulumi code into version control repositories in a way that aligns with organizational requirements. There is no “right” or “wrong” approach; choose the approach that enables you to satisfy your organization’s requirements around access control, code/repo ownership, and other organizational requirements.
Leverage Pulumi stacks to create multiple instances of cloud resources with a single Pulumi program. Pulumi stacks were created to address the need for organizations to create multiple instances of cloud resources, each with its own state and its own set of configuration data, from a single Pulumi program.

For now, Zephyr has decided to go with a monorepo approach—a single repository that contains both their application code and the Pulumi code to manage the infrastructure resources. All of the resources are defined in a single Pulumi project, with multiple stacks that correspond to development and production environments. Over the course of this series, we’ll see how Zephyr’s use of Pulumi changes as Zephyr grows and their application evolves.

The next Zephyr blog post will examine how the Zephyr team adjusts their use of stacks and Git branches to support developer productivity. Stay tuned!

Deploying Amazon EKS Anywhere on Bare Metal

David Flanagan — Fri, 01 Jul 2022 13:03:41 -0700

Check out version 3.0 of the Pulumi EKS Provider.

Some of the largest and most complex deployments that teams manage are hybrid and multi-cloud deployments. Kubernetes is a common component in these deployments because it enables platform teams to provide a common set of services across cloud and on-premises infrastructure and simplifies the process of migrating and scaling workloads as demand fluctuates. Pulumi simplifies these deployment scenarios but teams often need to manage different flavors of Kubernetes for on-premises deployments versus cloud deployments.

We were excited to support the launch of the Amazon EKS Distro in 2020 as it promised to help customers standardize their Kubernetes version and its dependencies. When EKS Anywhere launched last year, the pieces were starting to fall into place to enable consistent tooling for managing Kubernetes across your datacenter and in any cloud.

How it works (the short version)

AWS has now announced the latest innovation for EKS: Amazon EKS Anywhere Bare Metal and allows EKS Anywhere to be run without the need for VMWare vSphere. EKS Anywhere Bare Metal now enables an automated process to bootstrap provisioning and create the EKS-A control plane and worker nodes. Bare metal servers then PXE boot and run the OS provisioning process by communicating with a Tinkerbell Infrastructure Services Stack. At launch, bare metal servers are provisioned with AWS Bottlerocket with support for Ubuntu OS 20.04 also available.

Here are the current system requirements to run EKS Anywhere Bare Metal:

Cluster Hardware Requirements
- 4 GB of RAM for OSIE/Hook boot and operation
- Capability to netboot using iPXE
Tinkerbell Infrastructure Services Stack Requirements
- CPU - 2vCPUs, RAM - 4 GB, Disk - 20 GB (includes the OS)
- Network - L2 networking is required for the ability to run a DHCP server
- Operating System: Ubuntu 16.04+ and CentOS 7
Operating Systems Supported
- Ubuntu 20.04
- AWS Bottlerocket

Setting up your bare metal servers

As a “born in the cloud” company, Pulumi does not have a data center lying around for testing, so we’ll walk through the process of setting up EKS Anywhere Bare Metal using servers hosted by Equinix Metal. This really shows off the power of running EKS Anywhere and the great lengths the AWS team has gone to in enabling EKS Anywhere for a variety of scenarios.

The process to setup bare metal clusters involves the following steps:

Equinix Metal configuration
eks-a generate hardware config
eks-a generate cluster config
Create a bootstrap cluster on the admin machine
Provision and create control plane bare metal servers
Provision and create bare metal worker nodes
Move cluster management to the workload cluster
Delete the bootstrap cluster

Fortunately, Pulumi components make it easy to automate most of these steps. For example:

import * as eksa from "@pulumiverse/aws-eksa";

const page = new eksa.metal.Cluster("cluster", {
 clusterName: "rawkode",
 metro: "am",
 projectId: "someProjectID",
});

To use the component, take the following steps:

Make sure you’ve followed the steps to setup Pulumi and AWS
Grab the EKS-A component and example code on Pulumiverse.

Set your Equinix Metal Auth token as a Pulumi secret:

pulumi config set --secret equinix-metal:authToken

Run pulumi up.

Your output should look similar to the example below:

Give it a try!

These steps will vary depending on the actual hardware and/or bare metal service you decide to use. Check out our example repo and give it a try for yourself. If you run into any issues, you can reach out to us on Community Slack with your questions on the #aws channel.

EKS Blueprints for Pulumi

Isaac Harris — Wed, 20 Apr 2022 18:00:00 +0000

With the launch of Amazon Elastic Kubernetes Service (EKS) in 2017, it is now easier than ever to build, secure, operate and maintain Kubernetes clusters in the cloud. Notably, EKS removed the need to manage and configure underlying compute resources and scaling for clusters. Further, EKS Anywhere brings many benefits to hybrid and on-premises deployments.

These developments have proved to be a huge leap forward in productivity for teams that manage cloud infrastructure, enabling them to focus their efforts on deploying applications to meet the needs of customers and stakeholders.

As companies have matured in their use of Kubernetes, the day 0 through day 2 activities surrounding deployments have started to look similar regardless of where the workload is running. What has emerged is something we call a Shared Services Platform (SSP) – an internal company service that allows application developers to self-service infrastructure environments. SSPs are highly useful for platform teams when they want to share common infrastructure patterns and standardize/automate the provisioning of resources for their development teams. This allows the platform team to be more responsive to development team requests for new environments.

Common activities for teams managing SSPs include:

Setting up and managing multiple environments like DEV, TEST, STAGE, and PROD
Simplifying platform administration with addons
Onboarding Internal Teams via GitOps

Pulumi makes all of these activities easy to automate with built-in secrets management, reusable components, support for end-to-end infrastructure testing, and centralized policy enforcement. We also see new opportunities for simplification in all these activities. We’re excited to be part of the Amazon EKS Blueprints program to help Pulumi users deliver their platforms and applications faster than ever.

With today’s launch, we’re announcing a preview of a few of these scenarios that highlight the power of multi-language components that work across JavaScript/TypeScript, Python, .NET, and Golang in Pulumi. The goal is to get community feedback on these concepts and build more valuable components that meet the production needs of our users.

Here are some examples of how to simplify Kubernetes deployments using EKS Blueprint patterns built as Pulumi multi-language components:

Setting up and managing multiple environments

Pulumi has a concept called Stacks that allows you to deploy the same Pulumi program multiple times, with minor variations to account for environmental differences. Leveraging this with our new Pulumi SSP SDK, you can deploy a cluster to multiple environments with very little code. Here’s an example with one of Pulumi’s supported languages, Python.

from pulumi_ssp import SharedServicesPlatform, ClusterArgsArgs

ssp = SharedServicesPlatform("python-platform", cluster_args=ClusterArgsArgs(kubernetes_version="1.21.0", region="us-east-1"))

That’s it! With a single line of code, you can deploy an EKS cluster in your region of choice and the version of Kubernetes you want. You’ll get the default EKS node group with this configuration, but what if you want more control? Let’s take a look at that with another supported language, TypeScript.

import * as ssp from "@pulumi/ssp";

const sharedServicesPlatform = new ssp.SharedServicesPlatform("my-platform", {
 clusterArgs: {
 kubernetesVersion: "1.21.0",
 region: "us-east-1",
 },
});

sharedServicesPlatform.addManagedNodeGroup({
 name: "my-managed-group",
 desiredSize: 1,
 minSize: 1,
 kubernetesVersion: "1.21.0",
 maxSize: 2,
 instanceTypes: ["t3.medium"],
});

Our multi-language SDK provides some convenience functions that allow you to add NodeGroups and ManagedNodeGroups with just a few lines of configuration.

Simplifying Platform Administration with Addons

Of course, a Kubernetes cluster with nothing on it isn’t going to give you the platform your organization needs to onboard teams and provide the level of redundancy and resiliency you need. So how can you add common Kubernetes deployments to handle observability, monitoring, and other “day 1 + day 2” tasks? Addons.

Our Pulumi SSP also provides a clusterAddon convenience function that allows you to deploy things with a single line.

A must-have for all Kubernetes clusters is the metrics-server and node-problem-detector, so let’s add these. We’ll stick with TypeScript for this one.

sharedServicesPlatform.clusterAddon({
 name: "metrics-server",
});

sharedServicesPlatform.clusterAddon({
 name: "node-problem-detector",
 version: "0.8.10"
});

Of course, you can easily leverage Pulumi’s integration with a variety of observability tools like Datadog and SumoLogic as well. We will be building more addons over the next few iterations, and we encourage you to reach out on our Community Slack and let us know what you’d like to see.

Onboarding an Internal Team via GitOps

Lastly, Pulumi SSP allows platform teams to onboard teams to the platform, also with a convenience function call.

This time we’ll go wild and show what this looks like with CSharp.

var ssp = new SharedServicesPlatform("csharp", new SharedServicesPlatformArgs
 {
 ClusterArgs = new Pulumi.Ssp.Inputs.ClusterArgsArgs
 {
 KubernetesVersion = "1.21.0",
 Region = "us-east-1",
 }
 });

 ssp.OnboardTeam(new SharedServicesPlatformOnboardTeamArgs
 {
 Name = "payments",
 Controller = "pulumi",
 Repository = "github.com/awesome-org/payments-team/infra"
 });

Onboarding a team to the platform is a wrapper for creating a namespace. Soon, this will add resource quotas and a GitOps delivery pipeline. We’ll initially support ArgoCD and the Pulumi Operator, but support for FluxCD will be added shortly after.

Conclusion

The EKS Blueprints program is helping to simplify the process of managing a shared services platform on Kubernetes by defining standard patterns that every engineering team will find useful. We’d love to get your feedback on this approach and areas where you want to see more high-level components that simplify everyday infrastructure tasks. Check out these code examples and give us your feedback in our EKS Blueprints repo.

Create Amazon EKS clusters in your favorite language

Levi Blackstone — Thu, 04 Feb 2021 00:00:00 +0000

Check out version 3.0 of the Pulumi EKS Provider.

Pulumi’s infrastructure as code tooling combines the programming languages and tools you already know with the full power of cloud infrastructure. But until now, some Pulumi components for cloud infrastructure, like our popular EKS package for Amazon’s Elastic Kubernetes Service, were only available in a subset of the languages supported by Pulumi.

Now, you can use the EKS package–previously only available for TypeScript–in all four Pulumi languages: TypeScript, Python, .NET, and Go. Regardless of the language you choose, you can manage EKS clusters with Pulumi, starting with the v0.22.0 release. Check out our Modern Infrastructure Wednesday video to see it in action:

Making components available in every language

Pulumi’s EKS package is made up of Pulumi components. A component is a collection of Pulumi resources that provides an easy way to package best practices for using cloud resources, even if those best pratices span individual resources or even entire cloud providers. Until now though, components could only be used in Pulumi programs written in the same language as the component.

To change that, we’re creating “multi-language components”. With multi-language components, anyone will be able to author a component in one language and automatically compile it to all other languages supported by Pulumi. This means anyone in the worldwide Pulumi community will be able to take advantage of any component, even if it’s written in a different language. Right now, unless a component author is willing to write code in all four Pulumi languages, their component is only able to reach Pulumi users using the same language SDK.

In addition to expanding the library of publicly available components, this work will also enable organizations to write Pulumi components in the right programming language, or languages, for each team and each product. Infrastructure or platform teams will be able to write components in Go and share them with a webapp team using TypeScript, write components in .NET and share them with a machine learning team using Python, and any other combination that suits their business.

The next step in the story is a seamless authoring experience that enables component creators to write, build, and publish multi-language components. That will be in public preview soon, and we’re excited for you to try it out. If you’d like, subscribe to the multi-language components issue for updates.

Next steps

If you haven’t yet, try out the new pulumi-eks package with this walkthrough.

Easily Create and Manage AWS EKS Kubernetes Clusters

Luke Hoban — Thu, 28 Jan 2021 00:00:00 +0000

Provisioning, managing, and monitoring a Kubernetes cluster is not easy. AWS now offers EKS to reduce that burden – but it’s still difficult to get up and running. Pulumi’s infrastructure as code SDKs can help! We can provision an entire EKS cluster with a single CLI command, thanks to the @pulumi/eks package. Let’s see how.

Updated January 2021 to include Python, .NET, and Go support.

Updated October 2024 to Version 3.0

To get started, download the free and open source Pulumi SDK, which includes the pulumi CLI we’ll use below. If you don’t have the AWS CLI installed and configured, set that up too. You’ll need AWS credentials to access your AWS account and provision resources.

From here, there are two ways to proceed:

Provision a new cluster with a single CLI command
Use infrastructure as code to manage clusters … including Kubernetes resources themselves!

Installing Amazon EKS with Pulumi

With the Pulumi CLI installed and configured, a single command will provision your cluster:

$ pulumi new https://github.com/pulumi/apps/eks

This template will walk you through configuring a few settings such as the region to deploy, the instance type, and number of cluster nodes.

The CLI will first show you a preview of the AWS and Kubernetes resources that will be deployed to create the cluster, before actually performing any operation. Once you have reviewed and selected “Yes,” the cluster will begin deploying. The terminal output will look like this:

Following these steps will provide a complete managed Amazon EKS cluster, configured similarly to the steps outlined in the Amazon EKS Getting Started guide, including optional support for Amazon EBS-backed StorageClasses and access to the Kubernetes Dashboard. Pulumi replaces all of these tedious, manual steps with a straightforward description of your desired state, and then makes it so!

Once the deployment is complete – it will take 10-20 minutes for AWS to provision these resources – you can grab the kubeconfig.json needed to work with this cluster and connect to it with kubectl:

$ pulumi stack output kubeconfig > kubeconfig.json
$ KUBECONFIG=./kubeconfig.json kubectl get nodes

Reusable AWS EKS component

The Amazon EKS installer we just saw is built on top of a reusable eks.Cluster component that is available in Pulumi’s infrastructure as code solution. In fact, the command above created an instance of this component with reasonable defaults. If you prefer, you can do it by hand and programmatically. Here are examples in each of Pulumi’s supported languages:

import * as eks from "@pulumi/eks";

// Create an EKS cluster.
const cluster = new eks.Cluster("cluster", {
 instanceType: "t2.medium",
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

import pulumi
import pulumi_eks as eks

# Create an EKS cluster.
cluster = eks.Cluster(
 "cluster",
 instance_type="t2.medium",
 desired_capacity=2,
 min_size=1,
 max_size=2,
)

# Export the cluster's kubeconfig.
pulumi.export("kubeconfig", cluster.kubeconfig)

using System;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Eks.Cluster;

class EksStack : Stack
{
 public EksStack()
 {
 // Create an EKS cluster.
 var cluster = new Cluster("cluster", new ClusterArgs
 {
 InstanceType = "t2.medium",
 DesiredCapacity = 2,
 MinSize = 1,
 MaxSize = 2,
 });

 // Export the cluster's kubeconfig.
 this.Kubeconfig = cluster.Kubeconfig;
 }

 [Output("kubeconfig")]
 public Output<string> Kubeconfig { get; set; }
}

class Program
{
 static Task<int> Main(string[] args) => Deployment.RunAsync<EksStack>();
}

package main

import (
 "github.com/pulumi/pulumi-eks/sdk/go/eks/cluster"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an EKS cluster.
 cluster, err := cluster.NewCluster(ctx, "cluster",
 cluster.ClusterArgs{
 InstanceType: pulumi.String("t2.medium"),
 DesiredCapacity: pulumi.Int(2),
 MinSize: pulumi.Int(1),
 MaxSize: pulumi.Int(2),
 },
 )
 if err != nil {
 return err
 }

 // Export the cluster's kubeconfig.
 ctx.Export("kubeconfig", cluster.Kubeconfig)

 return nil
 })
}

Adding resources, and deploying Helm charts

You can extend or modify this program to change the details of the Amazon EKS cluster, or to install additional AWS or Kubernetes resources to associate with the cluster. For example, if applications running in the cluster need access to an Amazon S3 bucket, create a new Bucket resource:

const bucket = new aws.s3.Bucket("assets");

bucket = aws.s3.Bucket("assets")

var bucket = new Aws.S3.Bucket("bucket");

bucket, err := s3.NewBucket(ctx, "bucket", &s3.BucketArgs{})

Or, to install Kubernetes applications automatically into the Amazon EKS cluster (such as WordPress), add them using the Pulumi Kubernetes provider pointing at the newly provisioned EKS cluster:

const wordpress = new k8s.helm.v3.Chart("wpdev", {
 version: "9.6.0",
 chart: "wordpress",
 fetchOpts: {
 repo: "https://charts.bitnami.com/bitnami",
 },
}, { providers: { kubernetes: cluster.provider }});

wordpress = Chart(
 "wpdev",
 ChartOpts(
 chart="wordpress",
 version="9.6.0",
 fetch_opts=FetchOpts(
 repo="https://charts.bitnami.com/bitnami",
 ),
 ),
 opts=ResourceOptions(provider=cluster.provider),
)

var wordpress = new Chart("wpdev", new ChartArgs
{
 Chart = "wordpress",
 Version = "9.6.0",
 FetchOptions = new ChartFetchArgs
 {
 Repo = "https://charts.bitnami.com/bitnami"
 }
}, new CustomResourceOptions
{
 Provider = cluster.Provider,
});

wordpress, err := helm.NewChart(ctx, "wpdev", helm.ChartArgs{
 Chart: pulumi.String("wordpress"),
 Version: pulumi.String("9.6.0"),
 FetchArgs: helm.FetchArgs{
 repo: pulumi.String("https://charts.bitnami.com/bitnami"),
 },
}, pulumi.Provider(cluster.Provider))

Pulumi makes it possible to define the cluster, AWS resources, and Kubernetes objects needed to bootstrap your entire Kubernetes environment, all in one place. They can then be versioned together and exact replicas of the environment can be created independently for operations such as testing or disaster recovery.

As usual, by expressing these resources in code instead of in YAML, you get many software engineering benefits: improved tooling, the ability to refactor using your favorite IDE, the ability to create and share reusable components, and strong typing to validate correctness up front.

What’s happening under the hood?

To make this whole process buttery smooth, the Pulumi EKS component takes care of setting up all the pieces needed for an Amazon EKS cluster:

Creating an EKS Service Role
Optionally creating a new Virtual Private Cloud (VPC)
Creating an Amazon EKS cluster
Configuring a Kubernetes provider with access to the Amazon EKS cluster
Launching worker nodes in an Auto Scaling group to join the cluster
Installing a ConfigMap into the cluster to allow new Amazon EC2 worker nodes to join
Installing a StorageClass into the cluster to allow provisioning Amazon EBS-backed PersistentVolumes
Constructing the kubeconfig needed to access the cluster
Optionally installing additional Kubernetes YAML and/or Helm charts

This involves steps that provision resources in AWS, steps that provision resources in Kubernetes, and other custom computation to coordinate these steps. In addition, during the deployment Pulumi also provides complex coordination between these different steps.

Managing Pulumi Deployments

Once you have deployed your Amazon EKS cluster and Kubernetes resources to AWS, you can manage the deployment on app.pulumi.com, including deep links into the AWS console and Kubernetes dashboard to get real-time insights on your cluster.

All of the provisioned resources across AWS and Kubernetes are available from this single view, along with an audit history of deployments to this stack. The stack exports are also published in this same dashboard, so that key outputs, like your kubeconfig, are easily available and can be shared with other users in your organization.

Additional management features, including RBAC, CI/CD integrations, and rich resource graph visualization, are also available via the Pulumi app, including Team and Enterprise Editions for production teams of all sizes.

Get Started

We’ve seen how Pulumi takes care of the heavy lifting with AWS EKS so that you don’t have to. You can use Pulumi to easily deploy new clusters, managed AWS resources, and Kubernetes apps, all with the same powerful toolchain. Support is also available for Azure AKS, Google GKE, DigitalOcean’s Managed Kubernetes, and custom clusters (including Minikube).

For more information:

Getting Started with Amazon EKS Distro & Pulumi

Luke Hoban — Tue, 01 Dec 2020 11:30:00 -0800

As Kubernetes grows in popularity, the number of options for Kubernetes users continues to increase. Providers of managed Kubernetes offerings will often learn lessons about operating large numbers of clusters at scale; it’s increasingly common that they will contribute this knowledge back to the ecosystem, allowing those organizations who need more control and flexibility to reap the benefits.

With the announcement of the Amazon EKS Distro during AWS re:Invent, the Amazon EKS team has contributed back to the cloud-native community in a big way. In this post, we’ll take a brief look at what the Amazon EKS Distro is, explore why you might choose this over current managed service offerings and finally, explore how you can get started with the Amazon EKS Distro on day 1 using Pulumi.

Kubernetes is a complex, distributed system that consists of many components that operate together to create the Kubernetes experience. If you’re using a managed Kubernetes service, often these components are hidden away from you as a user to reduce the operational complexity and ease the usage. The managed service provider will take care of tricky problems like upgrades and taking care of patching and compliance.

The Amazon EKS Distro is the packaging of many of the components needed to run a Kubernetes cluster distributed in an opinionated way by the Amazon EKS team. Parts of a working Kubernetes cluster like the scheduler, API server and the backing database (etcd) have been built into Docker images based on Amazon Linux. The images have backported security fixes for each Kubernetes version and are drop-in replacements for the upstream Kubernetes distribution, meaning you get all the benefits of Kubernetes, with the added support of knowing you’re running a verified and supported version from the Amazon EKS team.

Why use the Amazon EKS Distro?

If you’re already a user of Amazon EKS, you’re already getting all of the benefits provided by the Amazon EKS Distro. However, many users aren’t utilizing the managed Kubernetes cluster provided by the Amazon EKS team, either through choice or because of external requirements. The release of the Amazon EKS Distro means that those organizations can get the benefits provided by EKS, such as secure Docker images, backported security fixes and a single upstream vendor, without having to operate with the constraints a managed service in AWS might provide.

Provisioning an Amazon EKS Distro Cluster with Pulumi

Installing an Amazon EKS Distro cluster involves bootstrapping a Kubernetes cluster but with the images provided by the Amazon EKS team. The simplest way to provision a Kubernetes cluster is using kops and the Amazon EKS Distro team provides instructions on how to replace the default Kubernetes components with the Amazon EKS Distro built parts.

It’s possible to get all of the benefits of provisioning a Kubernetes cluster with kops while also taking part in the Pulumi resource lifecycle. Pulumi offers support for running arbitrary commands using dynamic providers (available in the JavaScript, Typescript and Python SDKs) and you can register the kops commands used to create and destroy clusters as dynamic provider callbacks. Once your Kubernetes cluster has been created, you can provision Kubernetes resources using Pulumi’s Kubernetes provider, using the Pulumi resource model to set dependencies on the created cluster.

Creating an Amazon EKS Distro cluster with Pulumi

Using a Pulumi dynamic provider, you can define an Amazon EKS Distro cluster using just a few lines of code:

import * as eksdistro from "./eksdistro";

const store = new aws.s3.Bucket("kops-state-store");
const cluster = new eksdistro.Cluster("cluster", {
 name: "luke.cluster.pulumi-demos.net",
 state: pulumi.interpolate`s3://${store.id}`,

Behind the scenes, we’ve defined a Typescript dynamic provider, which renders a kops configuration file and renders it to disk using mustache. This kops configuration file references the Amazon EKS Distro images.

This dynamic provider registers callbacks for creating, updating and deleting the cluster and also registers the created cluster’s kubeconfig as a Pulumi output, which can be passed to Kubernetes resources in Pulumi.

import * as pulumi from "@pulumi/pulumi";
import * as cp from "child_process";
import * as path from "path";
import * as fs from "fs";
import * as mustache from "mustache";
import * as tmp from "tmp";

interface ClusterProviderArgs {
 // KOPS_STATE_STORE
 state: string;
 // KOPS_CLUSTER_NAME
 name: string;
}

const clusterYamlTemplate = fs.readFileSync(path.join(__dirname, "cluster.yaml")).toString();
const authenticatorYamlTemplate = fs.readFileSync(path.join(__dirname, "aws-iam-authenticator.yaml")).toString();

const clusterprovider: pulumi.dynamic.ResourceProvider = {
 async create(inputs: ClusterProviderArgs): Promise<pulumi.dynamic.CreateResult> {
 const clusterYaml = mustache.render(clusterYamlTemplate, {
 CLUSTER_NAME: inputs.name,
 STATE_STORE: inputs.state,
 });
 console.log("kops create")
 const createOut = cp.execSync(`kops create --name ${inputs.name} --state ${inputs.state} -f -`, { input: clusterYaml });
 console.log(createOut.toString());
 console.log("kops create secret")
 const createSecretOut = cp.execSync(`kops create secret --name ${inputs.name} --state ${inputs.state} sshpublickey admin -i ~/.ssh/id_rsa.pub`);
 console.log(createSecretOut.toString());

 let outs = { ...inputs };
 try {
 const updateRes = await this.update!(inputs.name, {}, inputs);
 outs = { ...outs, ...updateRes.outs};
 } catch (err) {
 console.log(err);
 }

 return {
 id: inputs.name,
 outs,
 };
 },
 async update(id: pulumi.ID, olds: any, inputs: any): Promise<pulumi.dynamic.UpdateResult> {
 const clusterYaml = mustache.render(clusterYamlTemplate, {
 CLUSTER_NAME: inputs.name,
 STATE_STORE: inputs.state,
 });
 const authenticatorYaml = mustache.render(authenticatorYamlTemplate, {
 CLUSTER_NAME: inputs.name,
 });

 console.log("kops update cluster")
 const updateOut = cp.execSync(`kops update cluster --name ${inputs.name} --state ${inputs.state} --yes`);
 console.log(updateOut.toString());

 console.log("kops export kubecfg")
 const kubeConfigName = tmp.tmpNameSync();
 const exportKubeconfigOut = cp.execSync(`kops export kubecfg --name ${inputs.name} --state ${inputs.state} --kubeconfig ${kubeConfigName}`);
 const kubeconfig = fs.readFileSync(kubeConfigName).toString();
 console.log(exportKubeconfigOut.toString());

 // Needed to allow cluster to come available and DNS to propagate
 try {
 console.log("kops validate cluster")
 const validateOut = cp.execSync(`kops validate cluster --wait 2m --name ${inputs.name} --state ${inputs.state}`);
 console.log(validateOut.toString());

 console.log("kubectl apply -f aws-iam-authenticator.yaml");
 const authApplyOut = cp.execSync(`kubectl apply -f -`, { input: authenticatorYaml });
 console.log(authApplyOut.toString());
 } catch (err) {
 console.log(err);
 }

 return {
 outs: {
 ...inputs,
 kubeconfig,
 },
 }
 },
 async delete(id: pulumi.ID, inputs: any) {
 const clusterYaml = mustache.render(clusterYamlTemplate, {
 CLUSTER_NAME: inputs.name,
 STATE_STORE: inputs.state,
 });
 console.log("kops delete")
 const deleteOut = cp.execSync(`kops delete --name ${inputs.name} --state ${inputs.state} --yes -f -`, { input: clusterYaml });
 console.log(deleteOut.toString());
 },
}

export interface ClusterArgs {
 // KOPS_STATE_STORE
 state: pulumi.Input<string>;
 // KOPS_CLUSTER_NAME
 name: pulumi.Input<string>;
 keeper?: pulumi.Input<number>;
}

export class Cluster extends pulumi.dynamic.Resource {
 kubeconfig!: pulumi.Output<string>;
 constructor(name: string, args: ClusterArgs, opts?: pulumi.CustomResourceOptions) {
 super(clusterprovider, name, {
 ... args,
 kubeconfig: undefined,
 }, opts);
 }
}

Once the cluster is created, we can build a Kubernetes provider and register resources like any other Kubernetes cluster:


const k8sProvider = new k8s.Provider("provider", { kubeconfig: cluster.kubeconfig });
const pod = new k8s.core.v1.Pod("mypod", {
 spec: {
 containers: [{ name: "echo", image: "k8s.gcr.io/echoserver:1.4" }],
 }
}, { provider: k8sProvider });
export const kubeconfig = cluster.kubeconfig;

The full code for defining an Amazon EKS Distro cluster using Pulumi can be found in our examples repo

Wrap up

The Amazon EKS Distro opens lots of avenues for users needing more flexibility and control over their Kubernetes clusters while still providing the security, compliance and reliability required to feel confident in the clusters you’re building. With Pulumi, you can bootstrap Amazon EKS Distro clusters and leverage Pulumi’s best-in-class Kubernetes resource management while still benefiting from the power of Pulumi’s modern programming languages. Give it a try!

Access Control for Pods on Amazon EKS

Mike Metral — Tue, 02 Jun 2020 00:00:00 +0000

Amazon EKS clusters can use IAM roles and policies for Pods to assign fine-grained access control of AWS services. The AWS IAM entities map into Kubernetes RBAC to configure the permissions of Pods that work with AWS services.

Together, AWS IAM and Kubernetes RBAC enable least-privileged access for your apps, scoped to the appropriate policies and user requirements.

Overview

In <100 lines of code we’ll demonstrate how EKS Pods can use AWS IAM to create fine-grained permissions for apps that integrate with other AWS services.

Pod Access Control
Create an OIDC provider
Create IAM for a S3 app
Deploy a S3 app
Wrap-Up
Next Steps

Pod Access Control

AWS EKS supports using IAM entities in a Pod Service Account by leveraging an OIDC provider connected to the Kubernetes cluster.

Continuing with the example from the AWS blog post, when an S3 app written in Go pushes an object to a bucket with the AWS SDK, it will need write access to S3.

When a Pod is launched with a particular Service Account, the OIDC provider works with Kubernetes to verify the Pod’s identity, and in turn collaborates with the AWS Secure Token Service (STS) to grant the Pod temporary credentials to use with the IAM role.

Create an OIDC provider

Creating an OIDC provider is as simple as toggling the createOidcProvider option in the definition of your EKS cluster.

When enabled, the OIDC provider will be created and associated with the cluster’s OIDC provider URL.

import * as eks from '@pulumi/eks'

// Create an EKS cluster with default settings.
// Create and attach an OIDC provider to the cluster.
const cluster = new eks.Cluster('myCluster', {
 createOidcProvider: true,
});

Create IAM for a S3 app

We’ll use the OIDC provider URL and Amazon Resource Name (ARN) to compose the AssumeRoleWithWebIdentity, and S3 IAM policies that will be attached to a new S3 IAM role.

After the role is configured, a Service Account for the S3 Pod will be created, and annotated with the ARN of the S3 role to bind the two together.

import * as aws from '@pulumi/aws'
import * as k8s from '@pulumi/kubernetes'
import * as pulumi from '@pulumi/pulumi'

// Create a pulumi Kubernetes provider using the cluster's kubeconfig.
const k8sProvider = new k8s.Provider('k8s', {
 kubeconfig: cluster.kubeconfig.apply(JSON.stringify),
});

// Create a k8s namespace in the cluster.
const namespace = new k8s.core.v1.Namespace('apps', undefined, { k8sProvider });

// Get the OIDC provider's URL for the cluster.
const clusterOidcProvider = cluster.core.oidcProvider.url;

// Create the new IAM policy for the Service Account using the AssumeRoleWebWebIdentity action.
const saName = 's3'
const saAssumeRolePolicy = pulumi
 .all([clusterOidcProviderUrl, clusterOidcProvider.arn, appsNamespaceName])
 .apply(([url, arn, namespace]) =>
 aws.iam.getPolicyDocument({
 statements: [
 {
 actions: ['sts:AssumeRoleWithWebIdentity'],
 conditions: [
 {
 test: 'StringEquals',
 values: [`system:serviceaccount:${namespace.metadata.name}:${saName}`],
 variable: `${url.replace('https://', '')}:sub`,
 },
 ],
 effect: 'Allow',
 principals: [{identifiers: [arn], type: 'Federated'}],
 },
 ],
 })
 );

// Create a new IAM role that assumes the AssumeRoleWebWebIdentity policy.
const saRole = new aws.iam.Role(saName, {
 assumeRolePolicy: saAssumeRolePolicy.json,
});

// Attach the IAM role to an AWS S3 access policy.
const saS3Rpa = new aws.iam.RolePolicyAttachment(saName, {
 policyArn: 'arn:aws:iam::aws:policy/AmazonS3FullAccess',
 role: saRole,
});

// Create a Service Account with the IAM role annotated to use with the Pod.
const sa = new k8s.core.v1.ServiceAccount(
 saName,
 {
 metadata: {
 namespace: namespace.metadata.name,
 name: saName,
 annotations: {
 'eks.amazonaws.com/role-arn': saRole.arn,
 },
 },
 },
 { k8sProvider });

Deploy a S3 app

We’ll deploy the S3 app to use the new IAM-backed Service Account.

Once the Pod is running, the Service Account annotation will be automatically managed by a Kubernetes dynamic admission controller run by EKS on your behalf.

The AWS EKS webhook manages Pod identity, and injects STS credentials into the Pod to use with the S3 role.

With the credentials, the app can successfully upload data to S3.

import * as aws from "@pulumi/aws";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const bucket = new aws.s3.Bucket("pod-irsa-job-bucket");
const bucketName = bucket.id;
const regionName = pulumi.output(aws.getRegion({}, {async: true})).name;
const s3Pod = new k8s.core.v1.Pod(podName,
 {
 metadata: {labels: labels, namespace: appsNamespaceName},
 spec: {
 serviceAccountName: sa.metadata.name,
 containers: [
 {
 name: podName,
 image: "amazonlinux:2018.03",
 command: ["sh", "-c",
 pulumi.interpolate`curl -sL -o /s3-echoer https://git.io/JfnGX && chmod +x /s3-echoer && echo This is an in-cluster test | /s3-echoer ${bucketName} && sleep 3600`,
 ],
 env: [
 {name: "AWS_DEFAULT_REGION", value: regionName},
 {name: "ENABLE_IRP", value: "true"},
 ],
 },
 ],
 },
 }, { provider: provider },
);

Wrap-Up

Leveraging AWS IAM for Pod workloads is a secure and effective means of limiting privileged execution, and provides a native experience for users.

Pod IAM can be extended further by also using the Kubernetes RBAC system. This allows configuring permissions for Kubernetes API resources, and handle scenarios such as limiting the namespace an IAM role can use, and what resources can be managed in the namespace.

Next Steps

Learn more about how Pulumi works with Kubernetes, and get started if you’re new.

Check out code examples for the S3 app referenced in this post, along with other access control scenarios for EKS.

Watch the video below for more details on how OIDC and Kubernetes RBAC works in EKS. We demonstrate how to deploy fluentd-cloudwatch with IAM to forward Pod logs to AWS CloudWatch.

You can also follow us on Twitter, subscribe to PulumiTV on YouTube, or join our Community Slack channel if you have any questions.

Managing AWS Containers with Pulumi

Sophia Parafina — Mon, 06 Jan 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

The Amazon Web Services (AWS) Cloud ecosystem is large and vibrant, so vast and vibrant that at times, it can be challenging to know where best to start! In the case of containers, Abby Fuller tweeted a descriptive summary about using AWS container services.

It looks straightforward, but it’s easy to get lost in the details when configuring multiple services to get your application running on AWS. Although AWS has extensive documentation and a point and click interface to these services, replicating then can be challenging, and it isn’t conducive to understanding the architecture. However, using infrastructure as code lets you see the details and results in a reproducible deployment that you can expand on. Let’s implement the diagram using Pulumi to deploy your container infrastructure with code in AWS.

Store your images

The AWS Elastic Container Registry (ECR) is a container registry that supports private container registries. ECR makes it easy to store, build, manage, and deploy container images and eliminates the need to operate your registry or use public registries, all in a highly available and scalable architecture.

In this example, we use Pulumi’s Crosswalk for AWS. Crosswalk for AWS is a collection of frequent tasks and best practices that simplify deploying infrastructure on AWS. We first declare a repository; this creates the AWS Container Repository resource. The last line of the code exports the URL so that we can access the repository after we have updated our Pulumi stack. After we’ve run pulumi up, the repository is ready to go, and you can use the Docker CLI to push, pull, and manage images.

import * as awsx from "@pulumi/awsx";

// Create a repository, as before.
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// And publish its URL, so we can push to it if we'd like.
export const url = repo.repository.repositoryUrl;

Choose your scheduler

There are two services for running containers in the AWS Cloud: Elastic Cloud Service (ECS), or Elastic Kubernetes Service (EKS). ECS is a proprietary service for running containers in the AWS Cloud. In contrast, EKS is a managed Kubernetes service for containers using a Kubernetes control plane. It has three master nodes distributed across three availability zones to provide high availability.

ECS is designed to work with other AWS services and provides more straightforward configuration and integration with them while providing high availability. In contrast, EKS is a managed Kubernetes service, and because Kubernetes is open-source, your infrastructure is portable to other cloud providers. Also, Kubernetes can provide fine-grain control over deployed services. The choice of which scheduler to use depends on your requirements.

You can create either an ECS or EKS cluster using Pulumi. If you wish to use ECS, @pulumi/aws provides all the primitives needed to build infrastructure on AWS.

import * as awsx from "@pulumi/aws";

// Create an ECS cluster explicitly, and give it a name tag.
const cluster = new aws.ecs.Cluster("custom", {
 tags: {
 "Name": "my-custom-ecs-cluster",
 },
});

If you wish to use EKS, use the Pulumi EKS library (@pulumi/eks) for Kubernetes primitives on AWS.

import * as eks from "@pulumi/eks";

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster");

Choose where/how to run, Fargate or EC2

Fargate is an AWS service that runs containers. It is suited to running small workloads or batch workloads with occasional bursts that require scaling quickly. AWS EC2 is ideal for large workloads requiring many CPU cores and gigabytes of memory.

Fargate

In this example, we create a load balancer open on port 80, spin up two instances of our container, and publish the endpoint URL.

import * as pulumi from "@pulumi/pulumi";

const lb = new awsx.lb.ApplicationLoadBalancer("nginx");

const service = new awsx.ecs.FargateService("service", {
 cluster: cluster.arn,
 assignPublicIp: true,
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: image.imageUri,
 essential: true,
 portMappings: [{
 containerPort: 80,
 targetGroup: lb.defaultTargetGroup,
 }],
 }
 }
 },
 desiredCount: 2,
});

// Export the load balancer's address so that it's easy to access.
export const appURL = pulumi.interpolate`http://${loadbalancer.loadBalancer.dnsName}`;

EC2 with Kubernetes

With Kubernetes, we create a deployment for the application and a service to make the application accessible via port 80. We publish both application endpoint to make it accessible and cluster config for use with kubectl CLI.

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: "nginx",
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the URL for the load balanced service.
export const url = service.status.loadBalancer.ingress[0].hostname;

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Build!

Let’s put it all together for both ECS and EKS. To get started, follow the Pulumi AWS documentation to install Pulumi, install AWS Client set your AWS environment variables, and create a new project in Typescript.

$ mkdir get-started && cd get-started
$ pulumi new aws-typescript

In this example, we build the application in code using the awsx.ecr.Image resource. This resource uses Docker to build the image locally and push it to our repository. Make sure that you have Docker installed and running.

The container application for this example is an HTML page in NGINX. Make a ./app directory in your Pulumi project and add the Dockerfile below.

Dockerfile:

FROM nginx
COPY content /usr/share/nginx/html

Create a ./content director in the ./app directory and add the HTML file below.

<html>
 <head><meta charset="UTF-8">
 <title>Hello, Pulumi!</title>
 </head>
 <body>
 <p>Hello, containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

Now that setup is complete you project directory should look similar to this:

get-started/
├── Pulumi.dev.yaml
├── Pulumi.yaml
├── app
│   ├── Dockerfile
│   └── content
│   └── index.html
├── index.ts
├── node_modules
├── package-lock.json
├── package.json
└── tsconfig.json

ECS

Now we’re ready to start building our infrastructure. Replace the generated index.ts file with the example below. The example shows how to deploy containers on ECS using Fargate.

import * as awsx from "@pulumi/awsx";

// Create a repository
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// Build an image from the "./app" directory
// and publish it to our ECR repository.
export const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 path: "./app",
});

// Create an ECS cluster explicitly, and give it a name tag.
const cluster = new aws.ecs.Cluster("custom", {
 tags: {
 "Name": "my-custom-ecs-cluster",
 },
});

// load balancer on port 80
const lb = new awsx.lb.ApplicationListener("nginx", { port: 80 });
const service = new awsx.ecs.FargateService("nginx", {
 cluster,
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: image.imageUri,
 memory: 512,
 portMappings: [ lb ],
 },
 },
 },
 desiredCount: 2,
});

// Publish the load balancer address so it's accessible.
export const appURL = lb.endpoint.hostname;

// Publish the repository URL, so we can push to it.
export const repoURL = repo.repository.repositoryUrl;

To deploy the application, run pulumi up. You’ll see a preview of the resources to be deployed. Accept the deployment by selecting yes, and when finished, your program lists the URLs for the application and the repository.

Outputs:
 appURL : "nginx-0296a2a-996897041.us-east-1.elb.amazonaws.com"
 image : "153052954103.dkr.ecr.us-east-1.amazonaws.com/my-repo-41f69f4:37c8add0fbd01474806c9c2319e62a857737de9e1460b09f935133860a069f4e"
 repoURL: "153052954103.dkr.ecr.us-east-1.amazonaws.com/my-repo-41f69f4"

Test to see if your application is working using either a browser or curl to see the web page.

$ curl nginx-0296a2a-996897041.us-east-1.elb.amazonaws.com
<html>
 <head><meta charset="UTF-8">
 <title>Hello, Pulumi!</title>
 </head>
 <body>
 <p>Hello, containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

EKS

The EKS example uses the @pulumi/eks and @pulumi/kubernetes node modules. Add the eks and kubernetes modules to the project.

$ node install “@plulumi/eks”
$ node install “@plulumi/kubernetes”

Like the ECS example, the EKS example builds the application image locally and pushes it into our custom repository. The application deployment has two replicas in the configuration. The service uses the EKS cluster, my-cluster, we declared to create a load balancer with port 80 open.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

// Create a repository, as before.
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// Build an image from the "./app" directory. and publish it to our ECR repository.
export const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 path: "./app",
});

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster")

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: image.imageUri,
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

// Publish the repository URL.
export const repoURL = repo.repository.repositoryUrl;

// Publish the URL for the load balanced service.
export const appURL = service.status.loadBalancer.ingress[0].hostname;

As with the ECS example, run pulumi up to deploy the application. To check the deployment, use either curl or a browser to see the page.

Want to know more?

Although these examples are simple, they demonstrate the basic building blocks for building, storing, and managing AWS Containers. They also show how to create ECS or EKS clusters for deploying apps. To get started with AWS and Pulumi check out the AWS Guide for core services and the Crosswalk for AWS guide for convenience APIs that simplify deploying infrastructure as code. For a deeper dive into managing containers on AWS, check out our blogs on ECS vs Fargate vs EKS: The Lowdown on Containers in AWS and How to Scale Your Amazon EKS Cluster: EC2, Managed Node Groups, and Fargate.

AWS EKS - How to Scale Your Cluster

Joe Duffy — Thu, 05 Dec 2019 00:00:00 +0000

Looking for information on how to deploy an EKS cluster? Checkout our AWS Elastic Kubernetes Service documentation.

AWS Elastic Kubernetes Service (EKS) provides a range of performance and control for dynamically scaling your Kubernetes clusters, including Managed Node Groups, Fargate, and Manually-Managed Node Groups in EC2. In this post, we’ll see how to use each of these compute options, and when to prefer one over the other in order to maximize productivity, flexibility, and control, based on your needs.

Managing EKS Workers

You now have three primary options for powering your EKS clusters:

Fargate: Let AWS Fargate intelligently manage and scale nodes automatically
Managed Node Groups: Let EKS manage and scale nodes based on declarative specifications
EC2: Manage nodes by hand using explicit groups, EC2 instances, and Auto Scaling Groups

Each option controls how your cluster’s worker nodes are provisioned, managed, and scaled. Worker nodes are what physically run the compute and host the storage used by your Kubernetes pods, and impact the cost and your cluster’s ability to meet workload demands. The spectrum of options available provides a range of flexibility, from least control to most control, respectively, over the configuration and provisioning of these worker nodes.

Abby Fuller from AWS explained these options visually and succinctly on Twitter.

Let’s now see how each of these work. To fully appreciate the new features, we will go in the reverse order — from most control and most difficult, to the least control and easiest, option.

Provisioning EKS Clusters

Running a Kubernetes cluster isn’t easy, but EKS makes the task of doing so much simpler. It offers out-of-the-box integrations with essential AWS services like IAM, EBS, Route53, and CloudWatch, so that your EKS clusters fit with your existing AWS security, storage, and monitoring practices.

It’s great to have all of the building blocks at your fingertips, and the @pulumi/aws package exposes these raw capabilities of EKS to you. We also created an EKS package to simplify common tasks, however, including creating the Kubernetes data plane, configuring VPC/CNI and subnet networking, and managing node groups. This package has been enlightened with the new features, and is what we’ll use in our examples.

Provisioning a new EKS cluster today is already as simple as a dozen lines of infrastructure as code:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");

// Create the EKS cluster itself, including a "gp2"-backed StorageClass.
const cluster = new eks.Cluster("my-cluster", {
 vpcId: vpc.vpcId,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 instanceType: "t2.medium",
 desiredCapacity: 5,
 minSize: 3,
 maxSize: 7,
 storageClasses: "gp2",
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

This program, when deployed with pulumi up, will provision an entire EKS cluster, all of its related infrastructure and internal Kubernetes resources, and then print out the kubeconfig that can be used to access the cluster afterwards:

$ pulumi up
Updating (dev):

 Type Name Status
 + pulumi:pulumi:Stack aws-ts-eks-dev created
 + ├─ awsx:network:Network vpc created
 + │ ├─ aws:ec2:Vpc vpc created
 + │ ├─ aws:ec2:InternetGateway vpc created
 + │ ├─ aws:ec2:Subnet vpc-0 created
 + │ ├─ aws:ec2:Subnet vpc-1 created
 + │ ├─ aws:ec2:RouteTable vpc created
 + │ ├─ aws:ec2:RouteTableAssociation vpc-0 created
 + │ └─ aws:ec2:RouteTableAssociation vpc-1 created
 + └─ eks:index:Cluster cluster created
 + ├─ eks:index:ServiceRole cluster-instanceRole created
 + │ ├─ aws:iam:Role cluster-instanceRole-role created
 + │ ├─ aws:iam:RolePolicyAttachment cluster-instanceRole-e1b295bd created
 + │ ├─ aws:iam:RolePolicyAttachment cluster-instanceRole-3eb088f2 created
 + │ └─ aws:iam:RolePolicyAttachment cluster-instanceRole-03516f97 created
 + ├─ eks:index:ServiceRole cluster-eksRole created
 + │ ├─ aws:iam:Role cluster-eksRole-role created
 + │ ├─ aws:iam:RolePolicyAttachment cluster-eksRole-90eb1c99 created
 + │ └─ aws:iam:RolePolicyAttachment cluster-eksRole-4b490823 created
 + ├─ pulumi-nodejs:dynamic:Resource cluster-cfnStackName created
 + ├─ aws:ec2:SecurityGroup cluster-eksClusterSecurityGroup created
 + ├─ aws:iam:InstanceProfile cluster-instanceProfile created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksClusterInternetEgressRule created
 + ├─ aws:eks:Cluster cluster-eksCluster created
 + ├─ aws:ec2:SecurityGroup cluster-nodeSecurityGroup created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksNodeInternetEgressRule created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksExtApiServerClusterIngressRule created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksClusterIngressRule created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksNodeClusterIngressRule created
 + ├─ aws:ec2:SecurityGroupRule cluster-eksNodeIngressRule created
 + ├─ aws:ec2:LaunchConfiguration cluster-nodeLaunchConfiguration created
 + ├─ pulumi-nodejs:dynamic:Resource cluster-vpc-cni created
 + ├─ pulumi:providers:kubernetes cluster-eks-k8s created
 + ├─ kubernetes:core:ConfigMap cluster-nodeAccess created
 + ├─ kubernetes:storage.k8s.io:StorageClass cluster-gp2 created
 + ├─ aws:cloudformation:Stack cluster-nodes created
 + └─ pulumi:providers:kubernetes cluster-provider created

Outputs:
 kubeconfig: {
 apiVersion : "v1"
 clusters : [
 [0]: {
 cluster: {
 certificate-authority-data: "..."
 server : "https://312E10705C16E8095B2A79E8E76EA00D.gr7.us-west-2.eks.amazonaws.com"
 }
 name : "kubernetes"
 }
 ]
 contexts : [
 [0]: {
 context: {
 cluster: "kubernetes"
 user : "aws"
 }
 name : "aws"
 }
 ]
 current-context: "aws"
 kind : "Config"
 users : [
 [0]: {
 name: "aws"
 user: {
 exec: {
 apiVersion: "client.authentication.k8s.io/v1alpha1"
 args : [
 [0]: "token"
 [1]: "-i"
 [2]: "cluster-eksCluster-560236b"
 ]
 command : "aws-iam-authenticator"
 }
 }
 }
 ]
 }

Resources:
 + 37 created

Duration: 12m12s

As soon as this cluster is up and running, we can interact with it with kubectl, or deploy with Pulumi:

$ pulumi stack output kubeconfig > kubeconfig.yml
$ KUBECONFIG=kubeconfig.yml kubectl get nodes
NAME STATUS ROLES AGE VERSION
aws-ip-10-0-164-133.us-east-2.compute.internal Ready <none> 4m v1.14.8-eks
aws-ip-10-0-233-6.us-east-2.compute.internal Ready <none> 4m v1.14.8-eks

Manually Managing EC2 Node Groups

A fully functioning production cluster usually requires many other considerations. For those, we’ve put together a set of playbooks as part of Pulumi Crosswalk for Kubernetes that walk through how to go to production with EKS specifically, in addition to other managed Kubernetes offerings.

Notice, for instance, that we didn’t need to even provision any worker nodes. This is thanks to the eks.Cluster abstraction creating a default node pool for us, which has an auto-scaling policy that attempts to maintain the desiredCapacity while remaining within the bounds of minSize and maxSize. Often you need more explicit control over the worker nodes, however, for reasons such precise capacity, specializing compute or storage for different workload needs, and so on. This ultimately devolves into managing EC2 instances by hand, which the eks.NodeGroup class supports.

For example, this code creates a new EKS cluster, disabling the default node group — using skipDefaultNodeGroup: true, since we will create our own — and creates a single node group with more specific configuration settings: IAM role, desired instance type, scaling parameters, labels, and so on:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");

// IAM roles for the node group.
const role = new aws.iam.Role("my-cluster-ng-role", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
 Service: "ec2.amazonaws.com",
 }),
});
let counter = 0;
for (const policyArn of [
 "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
 "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
 "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
]) {
 new aws.iam.RolePolicyAttachment(`my-cluster-ng-role-policy-${counter++}`,
 { policyArn, role },
 );
}
const instanceProfile = new aws.iam.InstanceProfile("my-cluster-ng-ip", { role });

// Create an EKS cluster.
const cluster = new eks.Cluster("my-cluster", {
 skipDefaultNodeGroup: true,
 vpcId: vpc.vpcId,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 instanceRoles: [ role ],
});

// Create an AWS node group using a cluster as input.
const nodeGroup = cluster.createNodeGroup("my-cluster-ng", {
 instanceType: "t2.medium",
 desiredCapacity: 1,
 minSize: 1,
 maxSize: 2,
 labels: { "ondemand": "true" },
 instanceProfile: instanceProfile,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

There is even more you can do here, including digging deep into the individual EC2 instances and Auto Scaling Groups powering the node group. Needing to manage workers at this level, and connect them to the right IAM and networking services, can be tedious and difficult to get right. That’s where the new AWS features come into play.

Automatically Managed Node Groups

The new EKS feature Managed Node Groups simplifies the task of managing explicit pools of worker nodes, at the cost of some amount of control. The @pulumi/eks package already had many of these conveniences built-in but this is now an official feature of the AWS platform.

Managed Node Groups will automatically scale the EC2 instances powering your cluster using an Auto Scaling Group managed by EKS. This ASG also runs the latest Amazon EKS-optimized Amazon Linux 2 AMI. This is great on one hand — because updates will be applied automatically for you — but if you want control over this you will want to manage your own node groups. Finally, security groups, IAM roles, and connecting them together is handled for you.

To opt-in to using Managed Node Groups, the raw aws.eks.NodeGroup building block is available. However, we’ve enlightened the EKS package with the eks.Cluster.createManagedNodeGroup function to make it easier and to integrate with cluster provisioning.

The following example creates an EKS cluster with a single Managed Node Group. It looks similar to our explicitly managed node group earlier, but a bit simpler because we can lean on EKS to configure and scale it:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");

// IAM roles for the node group.
const role = new aws.iam.Role("my-cluster-ng-role", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
 Service: "ec2.amazonaws.com",
 }),
});
let counter = 0;
for (const policyArn of [
 "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
 "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
 "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
]) {
 new aws.iam.RolePolicyAttachment(`my-cluster-ng-role-policy-${counter++}`,
 { policyArn, role },
 );
}

// Create an EKS cluster.
const cluster = new eks.Cluster("my-cluster", {
 skipDefaultNodeGroup: true,
 vpcId: vpc.vpcId,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 instanceRoles: [ role ],
});

// Create a simple AWS managed node group using a cluster as input.
const managedNodeGroup = eks.createManagedNodeGroup("my-cluster-ng", {
 cluster: cluster,
 nodeGroupName: "aws-managed-ng1",
 nodeRoleArn: role.arn,
 labels: { "ondemand": "true" },
 tags: { "org": "pulumi" },
}, cluster);

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

This can be provisioned with a single pulumi up, just like before, but instead of the automatically created default node group, our cluster will use a Managed Node Group instead.

Note that it’s possible to mix node groups. So, if you need precise control over some groups, but not others, feel free to call createNodeGroup and createManagedNodeGroup interspersed with one another. The EKS package knows what to do.

For a full list of properties you can configure on your Managed Node Groups, please refer to the createManagedNodeGroup API docs. There are fewer options available than manually managed node groups, such as inability to supply kubelet arguments, for instance. We are giving up some control in exchange for simplicity. For complete information about EKS Managed Node Groups, see AWS’s own product documentation.

Let Fargate Manage It All

Using EKS Managed Node Groups is a great step forward to simplifying how we run our Kubernetes workloads. It still means, however, that we need to declare our compute requirements and think at the level of worker nodes. That’s where AWS Fargate support for EKS offers a huge step function in simplifying our task of managing and scaling our EKS clusters.

AWS Fargate technology is essentially “serverless compute for containers.” Instead of thinking at the level of nodes, we can let Fargate handle provisioning, scaling, and scheduling of our cluster’s worker nodes. This means we can just operate at the level of Kubernetes pods abstractions and let AWS do all of the other hard work for us!

Not only is it simpler, but as Clare Liguori explained at re:Invent, Fargate can react very quickly to load changes. Amazon has put a ton of great effort into projects like AWS Nitro and Firecracker, ultimately leading to lightning fast startup times and impressive auto-scaling capabilities. Fargate for EKS taps into all of this goodness:

Now let’s see how to use it. The key building block that enables Fargate support is something called a Fargate profile. Like eks.NodeGroups above, one of these can be allocated explicitly, if you prefer to program at the level of the raw underlying building blocks.

The EKS package, however, has been enlightened to make allocating a Fargate-powered EKS cluster as simple as saying fargate: true. All we need to do is change our original cluster definition to the following:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");

// Create the EKS cluster itself with Fargate enabled.
const cluster = new eks.Cluster("my-cluster", {
 fargate: true,
 vpcId: vpc.id,
 privateSubnetIds: vpc.privateSubnetIds,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

If we provision this with pulumi up, we’ve got a fully functioning EKS cluster running on Fargate:

$ pulumi up
...
$ pulumi stack output kubeconfig > kubeconfig.yml
$ KUBECONFIG=kubeconfig.yml kubectl get nodes
NAME STATUS ROLES AGE VERSION
fargate-ip-10-0-174-4.us-east-2.compute.internal Ready <none> 3m v1.14.8-eks
fargate-ip-10-0-266-133.us-east-2.compute.internal Ready <none> 2m v1.14.8-eks

All of the hard work around managing and scaling our cluster is left to AWS Fargate, letting us focus on our workload!

In Conclusion

In this article, we’ve seen the full range of EKS cluster management options:

Fargate, for easy auto-scaling of your cluster’s worker compute
Managed Node Groups, enabling more control, while still leveraging EKS to auto-scale
EC2, for full control over your cluster’s compute pools and nodes

We’re excited to offer support for this full range of options the same week of AWS re:Invent, including not just the building block support, but the simpler interface provided by our open source EKS package.

To get started with Pulumi and kick the tires with EKS today, check out our Getting Started guide. There are both AWS and Kubernetes versions available.

Happy Fargating!

Multicloud Kubernetes: Running Apps Across EKS, AKS, and GKE

Mike Metral — Wed, 14 Aug 2019 00:00:00 +0000

Kubernetes clusters from the managed platforms of AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and GCP Google Kubernetes Engine (GKE) all vary in configuration, management, and resource properties. This variance creates unnecessary complexity in cluster provisioning and application deployments, as well as for CI/CD and testing.

Additionally, if you wanted to deploy the same app across multiple clusters for specific use cases or test scenarios across providers, subtleties such as LoadBalancer outputs and cluster connection settings can be a nuisance to manage.

In this post, we’ll see how to use Pulumi to deploy the kuard app across EKS, AKS, GKE and a local Kubernetes cluster, such as Docker Desktop or a self-managed cluster. We’ll spin up the clusters in each provider, launch the app, and manage both cluster and app using the TypeScript programming language.

View the full example and code.

![](multicloud.png)

Cluster Provisioning

Provisioning Kubernetes clusters and their IaaS resources is made simple through Pulumi’s various SDKs for the cloud providers:

Crosswalk for AWS further allows us to leverage the Pulumi libraries of common infrastructure for AWS to simplify cloud resource instantiation and management while gaining best-practices as defaults. Check out the pulumi/awsx SDK to get started.

For local clusters such as those that are self-managed, or provisioned by a tool like Docker Desktop, Pulumi can still deploy workloads to these these systems given that the pulumi/kubernetes workload SDK only requires a valid kubeconfig file. For more information on Pulumi’s Kubernetes support, check out the Kubernetes reference page.

We will use the cloud SDKs to provision the managed Kubernetes clusters. Given that we’re working with real code, we are afforded developer benefits such as: code linting, type checking, IDE hints and completion, abstractions and inheritance.

Leveraging these development features creates the opportunity to encapsulate the finer-grained details and settings, and expose the capability to create clusters as simple as the following code:

Workload Deployment

Once the clusters are provisioned, we can leverage the pulumi/kubernetes SDK to manage the Kubernetes workloads that will be deployed into the clusters.

The pulumi/kubernetes SDK uses the official Kubernetes client-go library to interact with Kubernetes. Therefore, Pulumi can work pretty much anywhere kubectl works, even if Pulumi was not used to create the cluster.

Summary

As shown in the code samples, it becomes relatively easy to provision and manage Kubernetes clusters across multiple clouds, as well as deploy workloads to the cluster regardless if they are managed by a cloud provider, or self-managed.

The various SDKS allow you to leverage industry standard best-practices and defaults, in addition to allowing you to further configure and customize how your clusters and apps are managed.

Testing apps across various providers in this form allows you to abstract away the details of provider specific implementations, and focus on how your app operates in the various contexts.

Learn More

If you’d like to learn about Pulumi and how to manage your infrastructure and Kubernetes multi-cloud capabilities through code, get started today. Pulumi is open source and free to use.

For further examples on how to use Pulumi to create Kubernetes clusters, or deploy workloads to a cluster, check out the rest of the Kubernetes tutorials.

As always, you can check out our code on GitHub, follow us on Twitter, subscribe to our YouTube channel, or join our Community Slack channel if you have any questions, need support, or just want to say hello.

If you’d like to chat with our team, or get hands-on assistance with migrating your existing configuration code to Pulumi, please don’t hesitate to drop us a line.

We also encourage you to watch Pulumi team member Levi Blackstone demo this post in an episode of the Kubernetes Community Meeting.

Migrate EKS Node Groups with Zero Downtime Using Pulumi

Mike Metral — Tue, 23 Jul 2019 00:00:00 +0000

Managed Kubernetes offerings greatly reduce the overhead required in administering Kubernetes. However, the cluster is only one of the components under management, as app lifecycles are self-driven tasks that vary by workloads.

In Kubernetes, node groups are a useful mechanism for creating pools of resources that can enforce scheduling requirements. They also provide a utility for shifting workloads around during cluster management and updates.

In this post, we’ll see how to use Pulumi for Day 2 Kubernetes administration. We’ll spin up a new EKS cluster with two node groups and a given workload. Then we’ll add one more node group with an updated configuration, and migrate the workload over to it with zero downtime using code and kubectl.

View the full how-to and code.

Create an EKS cluster and Deploy the Workload

For our initial update, we will configure and launch an EKS cluster using v1.13 of Kubernetes, with the cluster’s infrastructure dependencies (such as VPC and IAM) defined using Crosswalk for AWS. Crosswalk allows us to leverage the Pulumi libraries of common infrastructure for AWS to simplify cloud resource instantiation and management while gaining best-practices as defaults.

We’ll also create and attach the following node groups to the cluster:

A standard t2.medium worker node group using the recent v1.13.7 worker AMI, for general purpose workloads such as the EchoServer — a simple app that echo’s client request headers.
A 2xlarge t3.2xlarge worker node group using an older v1.12.7 worker AMI, for use by larger, intensive workloads such as the NGINX Ingress Controller.

With the cluster successfully created and the node groups available, Pulumi will deploy the workload: the EchoServer, and the NGINX Ingress Controller that will manage its ingress. The echoserver will land on the Standard node group, and NGINX is set to specifically target the 2xlarge node group.

Once the workload is deployed, we can validate it is up and running by accessing the echoserver behind the NGINX endpoint using curl:

Note: You can open the GIF for a maximized view.

The Great Migration

After our initial deployment, let’s say we decide to update the node group used by NGINX.

To do so, we’ll move NGINX from the 2xlarge node group over to a new, 4xlarge worker node group with an updated AMI, a different instance type, and a different desired instance count.

As we migrate NGINX over to the 4xlarge and decommission the 2xlarge node group in the next steps, we’ll also actively load test the endpoint of the echoserver to ensure that we are not losing requests throughout the migration.

Step 1: Create the new `4xlarge` node group.

Create the new, 4xlarge node group by defining it and running an update.

Step 2: Migrate NGINX to the `4xlarge` node group.

With the 4xlarge node group created, we’ll migrate the NGINX service away from the 2xlarge node group over to the 4xlarge node group by changing its node selector scheduling terms.

This change updates the NGINX Deployment spec to require the use of c5.4xlarge nodes during scheduling, and forces a rolling update over to the 4xlarge node group.

NGINX is able to successfully migrate across node groups because it is configured with HA settings, spread-type scheduling predicates, and can gracefully terminate within the Kubernetes Pod lifecycle.

Step 3: Decommission the `2xlarge` node group.

Once NGINX has been completely migrated to the 4xlarge node group, we can begin decommissioning the original 2xlarge node group which is no longer in use.

Decommissioning the node group involves:

Draining the Kubernetes nodes.
Deleting the Kubernetes nodes from the APIServer.
Scaling down the Auto Scaling Group to 0.
Deleting the node group.

Set up kubectl by using the kubeconfig from the stack output.

$ pulumi stack output kubeconfig > kubeconfig.json
$ export KUBECONFIG=$PWD/kubeconfig.json

To drain the 2xlarge node group, we’ll use kubectl drain:

for node in $(kubectl get nodes -l beta.kubernetes.io/instance-type=t3.2xlarge -o=name); do
 kubectl drain --force --ignore-daemonsets --delete-local-data --grace-period=10 "$node";
done

After draining has completed, use kubectl delete node to delete the nodes from the APIServer.

for node in $(kubectl get nodes -l beta.kubernetes.io/instance-type=t3.2xlarge -o=name); do
 kubectl delete "$node";
done

Scale down the node group Auto Scaling Group completely by setting the desiredCapacity: 0 and running an update:

Once the Auto Scaling Group has scaled down, we can delete the node group from AWS and the Pulumi program:

Summary

In this post, we stood up an EKS cluster with a couple of node groups, and an echoserver and NGINX workload. We then created a new, updated node group and migrated NGINX over to it.

We achieved this node group migration with zero downtime to our apps during load testing and decommissioning of the original node group.

Note: You can open the GIF for a maximized view.

Learn More

If you’d like to learn about Pulumi and how to manage your infrastructure and Kubernetes through code, get started today. Pulumi is open source and free to use.

For further examples on how to use Pulumi to create Kubernetes clusters, or deploy workloads to a cluster, check out the rest of the Kubernetes tutorials.

If you’d like to chat with our team, or get hands-on assistance with migrating your existing configuration code to Pulumi, please don’t hesitate to drop us a line.

Persisting Kubernetes workloads with Amazon EFS CSI volumes

Nishi Davidson — Mon, 15 Jul 2019 00:00:00 +0000

Some parts of this blog post are out-of-date. As an alternative, please refer to the EFS CSI Helm Chart and Pulumi’s support for deploying helm charts

The Amazon Elastic File System Container Storage Interface (CSI) Driver implements the CSI specification for container orchestrators to manage the lifecycle of Amazon EFS filesystems. The CSI specification defines an interface along with the minimum operational and packaging recommendations for a storage provider to implement a CSI compatible plugin. The interface declares the RPCs that a plugin must expose. The CSI drivers are the right mechanism to work with, when using a cloud storage component with Kubernetes workloads. Amazon Elastic File System (Amazon EFS) provides parallel shared access through a standard file system interface to Amazon EC2 instances and Linux-based workloads without disrupting your applications. EFS is a regional service storing data across multiple Availability Zones (AZs) for high availability and durability.

The AWS EFS CSI Driver is a Kubernetes SIG-AWS subproject. The AWS EFS CSI Driver triggers the creation of out-of-tree CSI volume plugins in any Kubernetes cluster. Thereafter Kubernetes storage classes, persistent volumes and claims can be used by a Kubernetes workload to access the EFS CSI volume plugin as before.

In this blog, we will work through an example that shows how to use AWS EFS CSI storage components with Kubernetes workloads running on Amazon EKS worker nodes using Pulumi libraries (EKS, AWS, and AWSX). These example steps are running on OS X.

Step 1: Initialize Pulumi Project and Stack for your organization:

Install Pulumi CLI and set up your AWS credentials. Initialize a new Pulumi project from available templates. We use aws-typescript template here to install all library dependencies.

We will work with two Pulumi stacks in this example, one for the Amazon EKS cluster and AWS EFS CSI components called k8sinfra. The other for the application and its storage class, persistent volume and persistent volume claim called app. The AWS EFS CSI (Container Storage Interface) is based on the initial AWS EFS CSI Driver work done by Kubernetes SIG-AWS.

$ brew install pulumi/tap/pulumi # download Pulumi CLI

$ mkdir k8sinfra && cd k8sinfra

$ pulumi new aws-typescript

$ npm install --save @pulumi/kubernetes @pulumi/eks

$ ls -la
drwxr-xr-x 10 nishidavidson staff 320 Jun 18 18:22 .
drwxr-xr-x+ 102 nishidavidson staff 3264 Jun 18 18:13 ..
-rw------- 1 nishidavidson staff 21 Jun 18 18:22 .gitignore
-rw-r--r-- 1 nishidavidson staff 32 Jun 18 18:22 Pulumi.dev.yaml
-rw------- 1 nishidavidson staff 91 Jun 18 18:22 Pulumi.yaml
-rw------- 1 nishidavidson staff 273 Jun 18 18:22 index.ts
drwxr-xr-x 95 nishidavidson staff 3040 Jun 18 18:22 node_modules
-rw-r--r-- 1 nishidavidson staff 50650 Jun 18 18:22 package-lock.json
-rw------- 1 nishidavidson staff 228 Jun 18 18:22 package.json
-rw------- 1 nishidavidson staff 522 Jun 18 18:22 tsconfig.json

Step 2: Create the EKS cluster, EFS endpoint and MountTargets:

The code below first declares an EKS cluster attached to two public subnets in a new VPC. We then declare EFS endpoint and mount the same to both subnets so that the EKS worker nodes can access the filesystem. This code should be pasted into index.ts.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

//* STEP 1: Create an EKS cluster, an EFS endpoint and mount the EFS endpoint in each public subnet of the EKS cluster

// Create an EKS Cluster
const vpc = new awsx.ec2.Vpc("vpc", {});
export const cluster = new eks.Cluster("eks-cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.publicSubnetIds,
 instanceType: "t2.medium",
 version: "1.12",
 nodeRootVolumeSize: 200,
 desiredCapacity: 2,
 maxSize: 3,
 minSize: 2,
 deployDashboard: true,
 vpcCniOptions: {
 warmIpTarget: 4
 }
});

const nodeSecurityGroup = cluster.nodeSecurityGroup;
export const nodesubnetIds = cluster.core.subnetIds;
export const clusterName = cluster.eksCluster.name;
export const kubeconfig = cluster.kubeconfig.apply(JSON.stringify);

// Create an EFS endpoint
export const efsFilesystem = new aws.efs.FileSystem("MyEFS", {
 tags: { Name: "myEFS" }
});

export const efsFilesystemId = efsFilesystem.id;

// Create a mounttarget in each of the EKS cluster VPC's AZs so that EC2 instances across the VPC can access the filesystem
new aws.efs.MountTarget("MyEFS-MountTarget1", {
 fileSystemId: efsFilesystemId,
 securityGroups: [nodeSecurityGroup.id],
 subnetId: nodesubnetIds[0]
});

new aws.efs.MountTarget("MyEFS-MountTarget2", {
 fileSystemId: efsFilesystemId,
 securityGroups: [nodeSecurityGroup.id],
 subnetId: nodesubnetIds[1]
});

Step 3: Create the CSI Driver Node and Controller Components

Currently, only static provisioning for AWS EFS CSI drivers is supported by SIG-AWS, which implies the Amazon EFS filesystem needs to be manually created first. After that it can be mounted inside a container as a volume using the driver. The code below will allow you to deploy allow the CSI Driver components on the Amazon EKS cluster.

//* STEP 3: Install EFS CSI Driver Node and Controller components

// Install EFS CSI Driver node components
const svcacntcsiNode = new k8s.core.v1.ServiceAccount(
 "csi-node-sa",
 {
 metadata: { name: "csi-node-sa", namespace: "kube-system" }
 },
 { provider: cluster.provider }
);

const clusterolecsiNode = new k8s.rbac.v1.ClusterRole(
 "csi-node",
 {
 metadata: { name: "csi-node", namespace: "default" },
 rules: [
 {
 apiGroups: [""],
 resources: ["secrets"],
 verbs: ["get", "list"]
 },
 {
 apiGroups: [""],
 resources: ["nodes"],
 verbs: ["get", "list", "update"]
 },
 {
 apiGroups: [""],
 resources: ["namespaces"],
 verbs: ["get", "list"]
 },
 {
 apiGroups: [""],
 resources: ["persistentvolumes"],
 verbs: ["get", "list", "watch", "update"]
 },
 {
 apiGroups: ["storage.k8s.io"],
 resources: ["volumeattachments"],
 verbs: ["get", "list", "watch", "update"]
 },
 {
 apiGroups: ["csi.storage.k8s.io"],
 resources: ["csinodeinfos"],
 verbs: ["get", "list", "watch", "update"]
 }
 ]
 },
 { provider: cluster.provider }
);

const clusterolebindingcsiNode = new k8s.rbac.v1.ClusterRoleBinding(
 "csi-node",
 {
 metadata: { name: "csi-node", namespace: "default" },
 subjects: [
 {
 kind: "ServiceAccount",
 name: "csi-node-sa",
 namespace: "kube-system"
 }
 ],
 roleRef: {
 kind: "ClusterRole",
 name: "csi-node",
 apiGroup: "rbac.authorization.k8s.io"
 }
 },
 { provider: cluster.provider }
);

const daemonsetcsiNode = new k8s.apps.v1beta2.DaemonSet(
 "efs-csi-node",
 {
 metadata: { name: "efs-csi-node", namespace: "kube-system" },
 spec: {
 selector: { matchLabels: { app: "efs-csi-node" } },
 template: {
 metadata: { labels: { app: "efs-csi-node" } },
 spec: {
 serviceAccount: "csi-node-sa",
 hostNetwork: true,
 containers: [
 {
 name: "efs-plugin",
 securityContext: {
 privileged: true
 },
 image: "amazon/aws-efs-csi-driver:latest",
 imagePullPolicy: "Always",
 args: ["--endpoint=$(CSI_ENDPOINT)", "--logtostderr", "--v=5"],
 env: [{ name: "CSI_ENDPOINT", value: "unix:/csi/csi.sock" }],
 volumeMounts: [
 {
 name: "kubelet-dir",
 mountPath: "/var/lib/kubelet",
 mountPropagation: "Bidirectional"
 },
 { name: "plugin-dir", mountPath: "/csi" },
 { name: "device-dir", mountPath: "/dev" }
 ]
 },
 {
 name: "csi-driver-registrar",
 image: "quay.io/k8scsi/driver-registrar:v0.4.2",
 imagePullPolicy: "Always",
 args: [
 "--csi-address=$(ADDRESS)",
 "--mode=node-register",
 "--driver-requires-attachment=true",
 "--pod-info-mount-version=v1",
 "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)",
 "--v=5"
 ],
 env: [
 { name: "ADDRESS", value: "/csi/csi.sock" },
 {
 name: "DRIVER_REG_SOCK_PATH",
 value: "/var/lib/kubelet/plugins/efs.csi.aws.com/csi.sock"
 },
 {
 name: "KUBE_NODE_NAME",
 valueFrom: { fieldRef: { fieldPath: "spec.nodeName" } }
 }
 ],
 volumeMounts: [
 { name: "plugin-dir", mountPath: "/csi" },
 { name: "registration-dir", mountPath: "/registration" }
 ]
 }
 ],
 volumes: [
 {
 name: "kubelet-dir",
 hostPath: { path: "/var/lib/kubelet", type: "Directory" }
 },
 {
 name: "plugin-dir",
 hostPath: {
 path: "/var/lib/kubelet/plugins/efs.csi.aws.com/",
 type: "DirectoryOrCreate"
 }
 },
 {
 name: "registration-dir",
 hostPath: { path: "/var/lib/kubelet/plugins/", type: "Directory" }
 },
 {
 name: "device-dir",
 hostPath: { path: "/dev", type: "Directory" }
 }
 ]
 }
 }
 }
 },
 { provider: cluster.provider }
);

// Install EFS CSI Driver Controller components
const svcacctcsiController = new k8s.core.v1.ServiceAccount(
 "csi-controller-sa",
 {
 metadata: { name: "csi-controller-sa", namespace: "kube-system" }
 },
 { provider: cluster.provider }
);

const clusterrolecsiController = new k8s.rbac.v1.ClusterRole(
 "external-attacher-role",
 {
 metadata: { name: "external-attacher-role", namespace: "default" },
 rules: [
 {
 apiGroups: [""],
 resources: ["persistentvolumes"],
 verbs: ["get", "list", "watch", "update"]
 },
 {
 apiGroups: [""],
 resources: ["nodes"],
 verbs: ["get", "list", "watch"]
 },
 {
 apiGroups: ["storage.k8s.io"],
 resources: ["volumeattachments"],
 verbs: ["get", "list", "watch", "update"]
 }
 ]
 },
 { provider: cluster.provider }
);

const clusterrolebindingcsiController = new k8s.rbac.v1.ClusterRoleBinding(
 "csi-attacher-role",
 {
 metadata: { name: "csi-attacher-role", namespace: "default" },
 subjects: [
 {
 kind: "ServiceAccount",
 name: "csi-controller-sa",
 namespace: "kube-system"
 }
 ],
 roleRef: {
 kind: "ClusterRole",
 name: "external-attacher-role",
 apiGroup: "rbac.authorization.k8s.io"
 }
 },
 { provider: cluster.provider }
);

const statefulsetcsiController = new k8s.apps.v1beta1.StatefulSet(
 "efs-csi-controller",
 {
 metadata: { name: "efs-csi-controller", namespace: "kube-system" },
 spec: {
 serviceName: "efs-csi-controller",
 replicas: 1,
 template: {
 metadata: { labels: { app: "efs-csi-controller" } },
 spec: {
 serviceAccount: "csi-controller-sa",
 priorityClassName: "system-cluster-critical",
 tolerations: [{ key: "CriticalAddonsOnly", operator: "Exists" }],
 containers: [
 {
 name: "efs-plugin",
 image: "amazon/aws-efs-csi-driver:latest",
 imagePullPolicy: "Always",
 args: ["--endpoint=$(CSI_ENDPOINT)", "--logtostderr", "--v=5"],
 env: [
 {
 name: "CSI_ENDPOINT",
 value: "unix:///var/lib/csi/sockets/pluginproxy/csi.sock"
 }
 ],
 volumeMounts: [
 {
 name: "socket-dir",
 mountPath: "/var/lib/csi/sockets/pluginproxy/"
 }
 ]
 },
 {
 name: "csi-attacher",
 image: "quay.io/k8scsi/csi-attacher:v0.4.2",
 imagePullPolicy: "Always",
 args: ["--csi-address=$(ADDRESS)", "--v=5"],
 env: [
 {
 name: "ADDRESS",
 value: "/var/lib/csi/sockets/pluginproxy/csi.sock"
 }
 ],
 volumeMounts: [
 {
 name: "socket-dir",
 mountPath: "/var/lib/csi/sockets/pluginproxy/"
 }
 ]
 }
 ],
 volumes: [{ name: "socket-dir", emptyDir: {} }]
 }
 }
 }
 },
 { provider: cluster.provider }
);

Step 4: Deploy a sample app with the EFS volume mounts:

Once the step above is complete, you will be ready to deploy your k8s sample application, storage class, persistent volume and persistent volume claim. Lets create a new stack for this called k8sapp as follows:

$ mkdir k8sapp && cd k8sapp

$ pulumi new typescript

$ npm install --save @pulumi/eks @pulumi/kubernetes

Let’s now declare the storage class, pv, pvc and application pod to complete our k8s application set-up for this example. We will update index.ts file again and run pulumi up:

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

const env = pulumi.getStack();
const cluster = new pulumi.StackReference(`d-nishi/k8sinfra/${env}`);
const kubeconfig = cluster.getOutput("kubeconfig");
const efsFilesystemId = cluster.getOutput("efsFilesystemId");

const k8sProvider = new k8s.Provider("cluster", {
 kubeconfig: kubeconfig
});

//* STEP 5: Create a storage class, persistent volume and persistent volume claim

export const storageclassEFS = new k8s.storage.v1.StorageClass(
 "efs-sc",
 {
 metadata: { name: "efs-sc" },
 provisioner: "efs.csi.aws.com"
 },
 { provider: k8sProvider }
);

const pvEFS = new k8s.core.v1.PersistentVolume(
 "efs-pv",
 {
 metadata: { name: "efs-pv" },
 spec: {
 capacity: { storage: "5Gi" },
 volumeMode: "Filesystem",
 accessModes: ["ReadWriteOnce"],
 persistentVolumeReclaimPolicy: "Recycle",
 storageClassName: "efs-sc",
 csi: {
 driver: "efs.csi.aws.com",
 volumeHandle: efsFilesystemId
 }
 }
 },
 { provider: k8sProvider }
);

const pvcEFS = new k8s.core.v1.PersistentVolumeClaim(
 "efs-claim",
 {
 metadata: { name: "efs-claim" },
 spec: {
 accessModes: ["ReadWriteOnce"],
 storageClassName: "efs-sc",
 resources: { requests: { storage: "5Gi" } }
 }
 },
 { provider: k8sProvider }
);

//* STEP 6: Mount the endpoint to pod in EKS cluster

export const newPod = new k8s.core.v1.Pod(
 "efs-app",
 {
 metadata: { name: "efs-app" },
 spec: {
 containers: [
 {
 name: "app",
 image: "centos",
 command: ["/bin/sh"],
 args: [
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"
 ],
 volumeMounts: [
 {
 name: "persistent-storage",
 mountPath: "/data"
 }
 ]
 }
 ],
 volumes: [
 {
 name: "persistent-storage",
 persistentVolumeClaim: { claimName: "efs-claim" }
 }
 ]
 }
 },
 { provider: k8sProvider, dependsOn: pvcEFS }
);

Verify the pod is running and that data is being written into the EFS filesystem using:

$ kubectl exec -ti efs-app -- tail -f /data/out.txt
Mon Jul 8 06:12:00 UTC 2019
Mon Jul 8 06:12:05 UTC 2019
Mon Jul 8 06:12:10 UTC 2019
Mon Jul 8 06:12:15 UTC 2019

This brings us to the end of our solution with Pulumi and AWS EFS on Amazon EKS. For more examples, refer to Pulumi’s open source repository here or refer to my other Kubernetes articles.

Deploy Kubernetes Ingress with AWS ALB & Pulumi

Nishi Davidson — Tue, 09 Jul 2019 00:00:00 +0000

Kubernetes Ingress is an API object that allows you manage external (or) internal HTTP[s] access to Kubernetes services running in a cluster. Amazon Elastic Load Balancing Application Load Balancer (ALB) is a popular AWS service that load balances incoming traffic at the application layer across multiple targets, such as Amazon EC2 instances, in a region. ALB supports multiple features including host or path based routing, TLS (Transport layer security) termination, WebSockets, HTTP/2, AWS WAF (web application firewall) integration, integrated access logs, and health checks.

The AWS ALB Ingress controller is a Kubernetes SIG-AWS subproject - it was the second sub-project added to SIG-AWS after the aws-authenticator subproject. The ALB Ingress controller triggers the creation of an ALB and the necessary supporting AWS resources whenever a Kubernetes user declares an Ingress resource on the cluster. TargetGroups are created for each backend specified in the Ingress resource. Listeners are created for every port specified as Ingress resource annotation. When no port is specified, sensible defaults (80 or 443) are used. Rules are created for each path specified in your ingress resource. This ensures that traffic to a specific path is routed to the correct TargetGroup.

In this post, we will work through a simple example of running ALB based Kubernetes Ingresses with Pulumi EKS, AWS, and AWSX packages.

Step 1: Initialize Pulumi project and stack

Install pulumi CLI and set up your AWS credentials. Initialize a new Pulumi project and Pulumi stack from available programming language templates. We will use the aws-typescript template here and install all library dependencies.

$ brew install pulumi/tap/pulumi # download pulumi CLI
$ mkdir eks-alb-ingress && cd eks-alb-ingress
$ pulumi new aws-typescript
$ npm install --save @pulumi/kubernetes @pulumi/eks
$ ls -la
drwxr-xr-x 10 nishidavidson staff 320 Jun 18 18:22 .
drwxr-xr-x+ 102 nishidavidson staff 3264 Jun 18 18:13 ..
-rw------- 1 nishidavidson staff 21 Jun 18 18:22 .gitignore
-rw-r--r-- 1 nishidavidson staff 32 Jun 18 18:22 Pulumi.dev.yaml
-rw------- 1 nishidavidson staff 91 Jun 18 18:22 Pulumi.yaml
-rw------- 1 nishidavidson staff 273 Jun 18 18:22 index.ts
drwxr-xr-x 95 nishidavidson staff 3040 Jun 18 18:22 node_modules
-rw-r--r-- 1 nishidavidson staff 50650 Jun 18 18:22 package-lock.json
-rw------- 1 nishidavidson staff 228 Jun 18 18:22 package.json
-rw------- 1 nishidavidson staff 522 Jun 18 18:22 tsconfig.json

Step 2: Create an EKS cluster

Once the steps above are complete, we update the typescript code in index.ts file to create an EKS cluster and run pulumi up from the command line:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

const vpc = new awsx.ec2.Vpc("vpc-alb-ingress-eks", {});
const cluster = new eks.Cluster("eks-cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.publicSubnetIds,
 instanceType: "t2.medium",
 version: "1.12",
 nodeRootVolumeSize: 200,
 desiredCapacity: 3,
 maxSize: 4,
 minSize: 3,
 deployDashboard: false,
 vpcCniOptions: {
 warmIpTarget: 4
 }
});

export const clusterName = cluster.eksCluster.name;
export const kubeconfig = cluster.kubeconfig;
export const clusterNodeInstanceRoleName = cluster.instanceRoles.apply(
 roles => roles[0].name
);
export const nodesubnetId = cluster.core.subnetIds;

Configure the Public subnets in the console as defined in this guide.

Step 3: Deploy AWS ALB Ingress Controller

Lets confirm that the EKS cluster is up using the following commands:

$ pulumi stack export kubeconfig > kubeconfig.yaml
$ export KUBECONFIG=kubeconfig.yaml
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-10-0-58.ec2.internal Ready <none> 7h8m v1.12.7
ip-10-10-1-167.ec2.internal Ready <none> 7h8m v1.12.7
ip-10-10-1-84.ec2.internal Ready <none> 7h8m v1.12.7

Adequate roles and policies must be configured in AWS and available to the node(s) running the controller. How access is granted is up to you. Some will attach the needed rights to node’s role in AWS. Others will use projects like kube2iam. We attach a minimal IAM policy to the EKS worker nodes and then declare this on the EKS cluster as shown in the code below.

When declaring the ALB Ingress controller we simply re-use the Helm chart as part of the code. There is no need to rewrite all the logic or install Tiller in the EKS cluster. This frees you from thinking about RBAC for Helm, Tiller and the k8s cluster per se'.

With the default “instance mode” Ingress traffic starts from the ALB and reaches the NodePort opened for the service. Traffic is then routed to the container Pods within cluster. This is all encoded using Pulumi libraries below. If you wish to use “ip-mode” with your Ingress such that traffic directly reaches your pods, you will need to modify the alb.ingress.kubernetes.io/target-type annotation when using the helm chart.

Append index.ts file from Step 2 with the code below and run pulumi up:

// STEP 3: Declare the AWS ALB Ingress Controller

// Create IAM Policy for the IngressController called "ingressController-iam-policy” and read the policy ARN.
const ingressControllerPolicy = new aws.iam.Policy(
 "ingressController-iam-policy",
 {
 policy: {
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Action: [
 "acm:DescribeCertificate",
 "acm:ListCertificates",
 "acm:GetCertificate"
 ],
 Resource: "*"
 },
 {
 Effect: "Allow",
 Action: [
 "ec2:AuthorizeSecurityGroupIngress",
 "ec2:CreateSecurityGroup",
 "ec2:CreateTags",
 "ec2:DeleteTags",
 "ec2:DeleteSecurityGroup",
 "ec2:DescribeInstances",
 "ec2:DescribeInstanceStatus",
 "ec2:DescribeSecurityGroups",
 "ec2:DescribeSubnets",
 "ec2:DescribeTags",
 "ec2:DescribeVpcs",
 "ec2:ModifyInstanceAttribute",
 "ec2:ModifyNetworkInterfaceAttribute",
 "ec2:RevokeSecurityGroupIngress"
 ],
 Resource: "*"
 },
 {
 Effect: "Allow",
 Action: [
 "elasticloadbalancing:AddTags",
 "elasticloadbalancing:CreateListener",
 "elasticloadbalancing:CreateLoadBalancer",
 "elasticloadbalancing:CreateRule",
 "elasticloadbalancing:CreateTargetGroup",
 "elasticloadbalancing:DeleteListener",
 "elasticloadbalancing:DeleteLoadBalancer",
 "elasticloadbalancing:DeleteRule",
 "elasticloadbalancing:DeleteTargetGroup",
 "elasticloadbalancing:DeregisterTargets",
 "elasticloadbalancing:DescribeListeners",
 "elasticloadbalancing:DescribeLoadBalancers",
 "elasticloadbalancing:DescribeLoadBalancerAttributes",
 "elasticloadbalancing:DescribeRules",
 "elasticloadbalancing:DescribeSSLPolicies",
 "elasticloadbalancing:DescribeTags",
 "elasticloadbalancing:DescribeTargetGroups",
 "elasticloadbalancing:DescribeTargetGroupAttributes",
 "elasticloadbalancing:DescribeTargetHealth",
 "elasticloadbalancing:ModifyListener",
 "elasticloadbalancing:ModifyLoadBalancerAttributes",
 "elasticloadbalancing:ModifyRule",
 "elasticloadbalancing:ModifyTargetGroup",
 "elasticloadbalancing:ModifyTargetGroupAttributes",
 "elasticloadbalancing:RegisterTargets",
 "elasticloadbalancing:RemoveTags",
 "elasticloadbalancing:SetIpAddressType",
 "elasticloadbalancing:SetSecurityGroups",
 "elasticloadbalancing:SetSubnets",
 "elasticloadbalancing:SetWebACL"
 ],
 Resource: "*"
 },
 {
 Effect: "Allow",
 Action: ["iam:GetServerCertificate", "iam:ListServerCertificates"],
 Resource: "*"
 },
 {
 Effect: "Allow",
 Action: [
 "waf-regional:GetWebACLForResource",
 "waf-regional:GetWebACL",
 "waf-regional:AssociateWebACL",
 "waf-regional:DisassociateWebACL"
 ],
 Resource: "*"
 },
 {
 Effect: "Allow",
 Action: ["tag:GetResources", "tag:TagResources"],
 Resource: "*"
 },
 {
 Effect: "Allow",
 Action: ["waf:GetWebACL"],
 Resource: "*"
 }
 ]
 }
 }
);

// Attach this policy to the NodeInstanceRole of the worker nodes.
export const nodeinstanceRole = new aws.iam.RolePolicyAttachment(
 "eks-NodeInstanceRole-policy-attach",
 {
 policyArn: ingressControllerPolicy.arn,
 role: clusterNodeInstanceRoleName
 }
);

// Declare the ALBIngressController in 1 step with the Helm Chart.
const albingresscntlr = new k8s.helm.v2.Chart(
 "alb",
 {
 chart:
 "http://storage.googleapis.com/kubernetes-charts-incubator/aws-alb-ingress-controller-0.1.9.tgz",
 values: {
 clusterName: clusterName,
 autoDiscoverAwsRegion: "true",
 autoDiscoverAwsVpcID: "true"
 }
 },
 { provider: cluster.provider }
);

Confirm the alb-ingress-controller was created as follows:

$ kubectl get pods -n default | grep alb
alb-aws-alb-ingress-controller-58f44d4bb8lxs6w
$ kubectl logs alb-ingress-controller-58f44d4bb8lxs6w
-------------------------------------------------------------------------------
AWS ALB Ingress controller
Release: v1.1.2
Build: git-cc1c5971
Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git
-------------------------------------------------------------------------------

Make sure the ingress-controller logs do not show errors about missing subnet tags or missing cluster name before proceeding to Step 4.

Step 4: Deploy Sample Application

The Ingress controller should now be running on the EKS worker nodes. Let’s now create a sample “2048-game” and expose it as an Ingress on our EKS cluster. The code below will let you do so. Append this piece of code into index.ts file from Step 3 and run pulumi up:

function createNewNamespace(name: string): k8s.core.v1.Namespace {
 //Create new namespace
 return new k8s.core.v1.Namespace(
 name,
 { metadata: { name: name } },
 { provider: cluster.provider }
 );
}

// Define the 2048 namespace, deployment, and service.
const nsgame = createNewNamespace("2048-game");

const deploymentgame = new k8s.extensions.v1beta1.Deployment(
 "deployment-game",
 {
 metadata: { name: "deployment-game", namespace: "2048-game" },
 spec: {
 replicas: 5,
 template: {
 metadata: { labels: { app: "2048" } },
 spec: {
 containers: [
 {
 image: "alexwhen/docker-2048",
 imagePullPolicy: "Always",
 name: "2048",
 ports: [{ containerPort: 80 }]
 }
 ]
 }
 }
 }
 },
 { provider: cluster.provider }
);

const servicegame = new k8s.core.v1.Service(
 "service-game",
 {
 metadata: { name: "service-2048", namespace: "2048-game" },
 spec: {
 ports: [{ port: 80, targetPort: 80, protocol: "TCP" }],
 type: "NodePort",
 selector: { app: "2048" }
 }
 },
 { provider: cluster.provider }
);

//declare 2048 ingress
const ingressgame = new k8s.extensions.v1beta1.Ingress(
 "ingress-game",
 {
 metadata: {
 name: "2048-ingress",
 namespace: "2048-game",
 annotations: {
 "kubernetes.io/ingress.class": "alb",
 "alb.ingress.kubernetes.io/scheme": "internet-facing"
 },
 labels: { app: "2048-ingress" }
 },
 spec: {
 rules: [
 {
 http: {
 paths: [
 {
 path: "/*",
 backend: { serviceName: "service-2048", servicePort: 80 }
 }
 ]
 }
 }
 ]
 }
 },
 { provider: cluster.provider }
);

After few seconds, verify the Ingress resource as follows:

$ kubectl get ingress/2048-ingress -n 2048-game
NAME HOSTS ADDRESS PORTS AGE
2048-ingress * DNS-Name-Of-Your-ALB 80 3m

Open a browser. Copy and paste your “DNS-Name-Of-Your-ALB”. You should be to access your newly deployed 2048 game – have fun!

Pulumi is open source and free to use. For more information on our product platform, check out the following resources:

ECS vs Fargate vs EKS: The Lowdown on Containers in AWS

Joe Duffy — Thu, 20 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

An updated breakdown of when to prefer Fargate and when EC2 can be found in Fargate vs EC2

Amazon offers multiple solutions for running containers in AWS, through its managed Elastic Container Service (ECS). This includes three major approaches: ECS managed automatically with Fargate, ECS backed by EC2 instances, and Elastic Kubernetes Service (EKS), delivering the full power of Kubernetes. It's not always easy to choose between these, so in this article we provide some basic guidance on the tradeoffs you'll encounter when choosing.

The Options: ECS vs Fargate vs EKS

At Pulumi, we work with customers to deploy AWS applications all the time, from early development to scaling production environments worldwide, using infrastructure as code and often continuous delivery. During this, we’ve run across a breadth of container workloads and requirements, and developed best practices, which this article distills into some basic advice. As with any advice, the right answer depends on your scenario’s specific requirements.

Here are the three major options, roughly speaking from easiest to hardest:

Option 1 - ECS Fargate

Amazon ECS with Fargate launch type allows you to deploy and scale your containerized workloads, automatically, without needing to manage clusters of EC2 instances by hand. You focus on your Dockerized applications, with minimal extra declarative metadata about the requirements of those applications, and not the details of the underlying compute itself. One could say this is “serverless containers.” Of course, there are servers running your containers, however the ECS control plane makes informed decisions on your behalf about when to add capacity, where, and by how much.

To run a workload in ECS, you use a standard Dockerfile to author, build, and publish your container image, and then run that image inside your ECS cluster. Tasks and services are authored using ECS definitions, which specify the container image to run, instance count, and any CPU, memory, networking, and disk requirements your workload has.

Using Pulumi, these definitions can be easily expressed declaratively in code, like so:

import * as awsx from "@pulumi/awsx";

// Create a load balancer on port 80 and spin up two instances of Nginx.
const lb = new awsx.lb.ApplicationListener("nginx", { port: 80 });
const nginx = new awsx.ecs.FargateService("nginx", {
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: "nginx",
 memory: 128,
 portMappings: [ lb ],
 },
 },
 },
 desiredCount: 2,
});

// Export the load balancer's address so that it's easy to access.
export const url = lb.endpoint.hostname;

This example runs 2 load balanced instances of an NGINX web server using ECS Fargate.

ECS Fargate is often a great choice when getting started, when you don’t have stringent requirements around cost, and/or if you’re a small to medium sized organization who prefers simpler solutions. Many teams are successful running it scale too, however Fargate doesn’t give you as much control over persistent storage, privileged containers, custom scheduling, or special VM types like GPUs or FPGAs. For those, EC2 backed ECS is a better choice.

Option 2 - ECS Backed by EC2 Instances

Amazon ECS with EC2 launch type gives you similar capabilities to ECS managed by Fargate – in that you can deploy containerized workloads, declaratively specify their requirements, and let ECS do the placement and overall scheduling of them – but with full control over the compute environment those containers run in. That includes control over the precise numbers of servers, AMIs they are running, and their auto-scaling policies.

To use an EC2 backed cluster, you will need to create a cluster and configure the EC2 launch and autoscaling configurations, and then simply use the awsx.ecs.EC2Service class to create your service instead. Other than these differences, the resulting code would look very similar to Option 1 - ECS Fargate above.

For sophisticated AWS organizations who already know how to fine tune and run compute at scale efficiently and cost effectively, managing your ECS cluster’s EC2 instances explicitly is often a good choice. The good news is that you can easily start with Fargate, and then over time, shift to managing the EC2 compute by hand if you prefer. Managing your cluster amounts to managing fleets of EC2 instances, CloudWatch logging, and standard AWS services.

Option 3 - Elastic Kubernetes Service (EKS)

Amazon Elastic Kubernetes Service (EKS) offers a very different approach to running containerized workloads than ECS Fargate or EC2. EKS is a fully managed offering that runs Kubernetes inside of your AWS account, making it easier to operate Kubernetes, in addition to integrating with many AWS services like ELB for load balancing, IAM for cluster authentication, and CloudWatch for logging and diagnostics. Using EKS, you don’t need to explicitly provision, manage, and upgrade the Kubernetes control plane’s configuration, compute, and storage.

Kubernetes is an open standard for running containers across a variety of public and private cloud environments. Most public cloud vendors now offer an EKS-like managed offering, including Azure (AKS), Google Cloud (GKE), and Digital Ocean, and the availability of on-premises private cloud configurations enables hybrid cloud for large organizations. Kubernetes in inherently more complicated to operate than the equivalent ECS solutions mentioned earlier, because it requires becoming an expert in Kubernetes, which requires more specialization in its tools and management practices. But the potential payoff is larger for organizations wanting to go “all in” on cloud native technologies to facilitate either multi-cloud or hybrid computing – challenges most large enterprises are facing today.

In the EKS case, as with ECS, we continue using standard Dockerfiles, however we use the Kubernetes object model to configure and run workloads. Instead of task definitions, you use Kubernetes objects like Namespaces, Pods, Deployments, and Services. This full API is available to specify in code using Pulumi, like so:

import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster");

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: "nginx",
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the URL for the load balanced service.
export const url = service.status.loadBalancer.ingress[0].hostname;

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Like the ECS example earlier, this spins up 2 load balanced instances of the NGINX web server.

You will find that the entirety of Kubernetes is more feature rich and offers more control over scheduling and scaling your workloads, but with that richness also comes added complexity that will increase training, and time to ramp up, in addition to ongoing maintenance costs. To learn more about Pulumi’s support for Kubernetes, go here.

Private ECR Container Registries

In all of these cases, Amazon Elastic Container Registry (ECR) lets you publish Docker container images to a privately hosted repository inside your AWS account. The benefit of this is that images are stored within your AWS account, where access is governed by IAM and, where they are stored closer to your workloads (maximizing push/pull efficiency). This is an alternative approach to, say, publishing images to the Docker Hub which entails a different authentication and pushing and pulling images over the Internet, where performance can be an issue.

Next Steps

Pulumi Crosswalk for AWS supports three main options for running containers in AWS – ECS Fargate, ECS with EC2 instances, and EKS – each of which integrates deeply with other AWS services like IAM, ELB, and CloudWatch. It’s possible to use any of these services without Pulumi, but there are many benefits to Pulumi’s infrastructure as code.

In short, to get your containers up on AWS, in an easy yet production-ready way, get started for free with Pulumi for AWS today!

After getting started, here are some additional resources to go deeper on specific topics:

Introducing Pulumi Crosswalk for AWS: The Easiest Way to AWS

Luke Hoban — Mon, 10 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Amazon Web Services provides an incredible platform for developers to build cloud-native applications, and is used by millions of customers of all sizes. The building block services that AWS offers enable teams to offload undifferentiated heavy-lifting to AWS. To maximally benefit from these services though, cloud engineering teams must learn how to compose all of these building blocks together to build and deliver their own applications. Today, this is still too hard. Getting from your laptop to a production-ready AWS deployment frequently takes days or weeks instead of minutes or hours. And AWS building block services frequently leave you to re-implement (and re-discover) best-practices instead of providing these as smart defaults.

Pulumi Crosswalk for AWS is a new open source library of infrastructure-as-code components that make it easier to get from zero to production on AWS, easier to adopt AWS best practices by default, and easier to evolve your AWS infrastructure as your application needs mature.

Crosswalk for AWS offers higher-level components for a wide range of the AWS platform - from serverless (Lambda, API Gateway and more) to containers (ECS, Fargate) to Kubernetes (EKS) and across networking (VPCs, SecurityGroups), monitoring (Cloudwatch Dashboards and Alarms) and load balancing (ALB and NLB). Because Crosswalk builds on the primitive building blocks of the AWS platform, you can use 100% of what AWS offers, and can seamlessly move between high-level components and low-level platform primitives, mixing-and-matching as needed.

In some of these domains, there have been successful tools that have helped developers more easily work with one specific aspect of AWS - Serverless Framework for serverless, Docker Compose for ECS, eksctl for EKS. But these tools have traditionally been limited to one specific kind of compute, limiting what users can accomplish, fragmenting development of complete cloud solutions, and leading to sharp cliffs when application needs evolve. Crosswalk for AWS offers the same kind of simpler and more productive developer interface as these tools, but applied uniformly across the entire AWS platform. Simple things are simple, and arbitrarily complex things are possible.

Whether you are starting a new SaaS project at a startup, migrating an enterprise application to cloud-native infrastructure, or delivering a new service as part of an established cloud application, Crosswalk for AWS provides the easiest way to get from zero to production, and to then evolve with your projects’ needs. You can get started with Crosswalk for AWS today!

Note: Crosswalk for AWS projects are authored using the Pulumi infrastructure-as-code tools. Pulumi allows you to define your infrastructure using code. As your infrastructure code evolves, Pulumi makes the minimal required changes to your cloud resources, ensuring that your infrastructure is updated without downtime (if possible).

From Zero to Production And Beyond with Crosswalk for AWS

Serverless (Lambda, API Gateway)

When getting started with a new cloud project, the starting point is often just serving a simple API on the internet. With just a few lines of code and tens of seconds to deploy, Crosswalk for AWS gives us our API.

import * as awsx from "@pulumi/awsx";

const api = new awsx.apigateway.API("api", {
 routes: [{
 method: "GET",
 path: "/",
 eventHandler: async (ev, ctx) => {
 return {
 statusCode: 200,
 body: JSON.stringify(ev),
 };
 }
 }],
})

export const endpoint = api.url;

By building on top of great AWS building blocks like Lambda, API Gateway, IAM and more, we avoid needing to worry about infrastructure, pay nearly zero fixed costs, and gain the ability to iterate quickly. With just a few more lines of code, we can wire up to our own domain, add authorization, provision a database, or chain together a more complex event-driven application.

Containers (ECS, Fargate)

As our needs grow, we might find we need to deploy containers - either custom applications or existing Docker images. Again, just a few lines of code gets our container running in production in AWS.

import * as awsx from "@pulumi/awsx";

// Create the cluster
const cluster = new awsx.ecs.Cluster("cluster");

// Configure a Load Balancer
const alb = new awsx.lb.ApplicationLoadBalancer("lb", {
 securityGroups: cluster.securityGroups,
});
const web = alb.createListener("web", { port: 80, external: true });

// Define the scaled-out Fargate service
const appService = new awsx.ecs.FargateService("app-svc", {
 cluster,
 taskDefinitionArgs: {
 container: {
 image: awsx.ecs.Image.fromPath("app", "./app"),
 portMappings: [ web ],
 },
 },
 desiredCount: 5,
});

export const url = web.endpoint.hostname;

Using ECS, Fargate, ECR and ELB, we get a robust production-ready container deployment - horizontally scaled out, load-balanced, and integrated with a private image repository. With just a few more lines of code, we can add autoscaling, customize our ECS cluster, or wire through advanced container configuration (volumes, environment variables, and more).

Networking (VPC)

To adopt best practices security posture, we may want to move our cluster into a custom VPC, and run our containers in private subnets. We can define and configure our VPC, and then redeploy our containers into this new VPC with ease:

import * as awsx from "@pulumi/awsx";

const vpc = new awsx.ec2.Vpc("vpc", {
 numberOfAvailabilityZones: 3,
});

const cluster = new awsx.ecs.Cluster("cluster", {
 vpc: vpc,
});

// ... same as before ...

The Vpc component builds on best-practices Virtual Private Cloud design patterns based on AWS guidance and documentation. By building in simple defaults for routing, subnet structure, NATs and multi-AZ you can get started quickly without having to re-discover these best practices. And as your needs grow, you can deeply customize and evolve your VPC structure - defining custom CIDR blocks, customizing Internet and NAT Gateways or additional additional private subnets.

Kubernetes (EKS)

As our project continues to evolve, we may want to move our container workloads into Kubernetes using EKS. We can stand up an EKS cluster with just a few lines of code, then customize our cluster and deploy Kubernetes workloads into our cluster.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("vpc");

// Create the EKS cluster itself.
const cluster = new eks.Cluster("cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.privateSubnetIds,
 instanceType: "m5.large",
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 deployDashboard: true,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Building on EKS we get all the benefits of managed Kubernetes paired with the platform capabilities of AWS. With Crosswalk for AWS, we can get started quickly, and then evolve to take advantage of all of these platform features, like customized node groups, private networking, and even deploying Kubernetes YAML and Helm charts to our cluster.

Monitoring (CloudWatch)

Getting our infrastructure up and running in AWS is great, but we also want to be able to monitor it. We can add a few lines to any of the solutions above to define metrics, dashboards, alerting and more to monitor our AWS infrastructure.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create a bucket and a function to log all new object created events
const bucket = new aws.s3.Bucket("b");
const subscription = bucket.onObjectCreated("newObject", async (ev) => {
 console.log(JSON.stringify(ev));
});

// Create a CloudWatch Dashobard for our functions invocations
const dashboardName = "funcDashboard";
const dashboard = new awsx.cloudwatch.Dashboard(dashboardName, {
 widgets: [
 new awsx.cloudwatch.LineGraphMetricWidget({
 title: "Lambda invocations",
 width: 14,
 metrics: awsx.lambda.metrics.invocations({
 function: subscription.func, statistic: "Sum", period: 60, }), }), ], });

// Export the URL of the dashboard in the AWS console
export const dashboardUrl =
 `https://${aws.config.region}.console.aws.amazon.com/cloudwatch/home?` +
 `region=${aws.config.region}#dashboards:name=${dashboardName}`;

With CloudWatch being deeply integrated into all AWS services, we can easily build up robust logging, alerting and dashboarding solutions across our AWS infrastructure. With just a few more lines of code we can get logs for our functions and containers, create alarms when we cross critical thresholds, and create rich dashboard layouts all of which will version along with our infrastructure!

Early Feedback

We’ve been working on pieces of Crosswalk for AWS since before we launched Pulumi, and we’re excited to be launching Crosswalk for AWS today to bring these together for all AWS users. Over the last few months, users have built on Crosswalk for AWS for all of the scenarios above and more.

Tableau has used Crosswalk for AWS to accelerate their Kubernetes migration on AWS, building on the networking and EKS capabilities of Crosswalk for AWS.

“We’ve been happily using Pulumi’s EKS support for more than three months now. Our team was looking for an end-to-end solution to tame the complexity of Kubernetes on AWS and ensure we adhere to AWS best practices. Pulumi’s Crosswalk effort has equipped our team to scale far better, our delivery is now automated, and we can now deliver new application and infrastructure features with much faster turn-around, sometimes in just a few hours. Pulumi is a key contributor to our team’s improvement in productivity.” - Pankaj Dhingra, Sr Director, Cloud Engineering, Tableau

Compute Software has used Crosswalk for AWS to go from zero to production, and then to evolve their infrastructure in AWS, all without needing a dedicated cloud infrastructure engineering team.

“We use Crosswalk for AWS with Fargate. It’s been instrumental in getting us into production. Deploying to the cloud is necessarily complex. Crosswalk streamlines the process by exposing the controls you care about, always allowing you to dive deeper when needed.” - Kenny Williams, Co-Founder, Compute Software

We’re excited by the success these teams and many more have already had with Crosswalk for AWS, but even more excited about the potential to truly democratize access to the AWS platform and unlock highly productive cloud development for all kinds of engineering organizations.

What’s Next?

We’re still just scratching the surface of what’s possible with Crosswalk. We’re continuing to evolve the open source projects to simplify additional features of the AWS platform. And we’re also working to bring the same approach to other cloud platforms, including Azure, GCP and Kubernetes.

We believe that making it easier for cloud engineers to develop and deploy natively in the cloud will unlock creativity and empower developers to take advantage of more of the rich capabilities of AWS. We can’t wait to see what you build with Crosswalk!

Crosswalk for AWS is free and open source, and you can get started today.

For more on Crosswalk for AWS see:

Kubernetes RBAC in AWS EKS with open source Pulumi packages

Nishi Davidson — Wed, 24 Apr 2019 00:00:00 +0000

Check out version 3.0 of the Pulumi EKS Provider.

One of the most common areas Kubernetes operators struggle with in production involves creating and managing role-based access control (RBAC). This is so daunting that RBAC is often not implemented, or implemented halfway, or the configuration becomes impossible to maintain.

Fortunately, Pulumi makes RBAC on Kubernetes so easy that you’ll never create an insecure cluster again. In this post, we will contrast the traditional way of working with RBAC on EKS with using Pulumi.

Here are a few highlights:

NO MORE YAMLs! Configuring YAMLs, operators or custom resources is now a thing in the past! You use TypeScript or JavaScript to program directly with our cloud SDK and connect all cloud services to your Kubernetes services with a simple reference to the object in your program.
INCREASED DEVELOPMENT VELOCITY: You intuitively program Kubernetes objects with our SDK abstractions using minimal amount of code within hours instead of months. You “autocomplete” AWS, EKS, Kubernetes specifications within your IDE without understanding the entire API.
EASY UPDATES: Changing a roleRef in a RoleBinding, on one or multiple clusters involves updating your TypeScript file index.ts and running pulumi up. The Pulumi Service allows you to share your stack with your team in your GitHub, GitLab, or Atlassian-based organization.
WORKFLOW AUTOMATION FOR RBAC AT SCALE: You can delete or update multiple RoleBindings or Roles from your Pulumi stack source code. As you commit these changes to your repository, you can plan automated triggers that validate such changes as part of your CI/CD flow, whether you use Travis, CircleCI, AzureDevOps and more. Pulumi even has a GitHub Application for surfacing results within pull requests.

Prerequisites to work with Pulumi

Install pulumi CLI and set up your AWS credentials. Initialize a new Pulumi project from available templates. We use aws-typescript template here to install all dependencies and save the configuration.

$ brew install pulumi/tap/pulumi # download pulumi CLI
$ mkdir eks-rbac && cd eks-rbac
$ pulumi new aws-typescript
$ ls -l
-rw-r--r-- 1 nishidavidson staff 32 Apr 18 14:49 Pulumi.dev.yaml
-rw------- 1 nishidavidson staff 84 Apr 18 14:48 Pulumi.yaml
-rw------- 1 nishidavidson staff 273 Apr 18 14:48 index.ts
drwxr-xr-x 92 nishidavidson staff 2944 Apr 18 14:49 node_modules
-rw-r--r-- 1 nishidavidson staff 48352 Apr 18 14:49 package-lock.json
-rw------- 1 nishidavidson staff 228 Apr 18 14:48 package.json
-rw------- 1 nishidavidson staff 522 Apr 18 14:48 tsconfig.json

With Pulumi, you will modify and update the default index.ts file with AWS and EKS resource variable declarations. We show you how to add this code as we contrast Pulumi’s approach with the sequential traditional approach in the steps below. In the end, you will do a one-time run of pulumi up and watch all the steps in the Traditional Way come alive simultaneously.

Step 1: Create three IAM Roles with a Trust-policy to map to Amazon EKS RBAC.

The Traditional-approach:

You sequentially create three IAM roles (clusterAdminRole; AutomationRole; EnvProdRole) with aws command line tool as shown below:

$ aws iam create-role --role-name clusterAdminRole --assume-role-policy-document file://Role-Trust-Policy.json
 
{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "sts:AssumeRole",
                    "Principal": {
                        "AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
                    },
                    "Effect": "Allow",
                    "Sid": ""
                }
            ]
        },
        "RoleId": "AROASHIVKXX3SFFMUUEU6",
        "CreateDate": "2019-04-17T17:43:03Z",
        "RoleName": "clusterAdminRole",
        "Path": "/",
        "Arn": "arn:aws:iam::xxxxxxxxxxxx:role/clusterAdminRole"
    }
}
 
$ cat Role-Trust-Policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "*"
        }
    ]
}

The Pulumi-approach:

You update the default index.ts file in your source code editor such as VSCode as follows:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

/*
 * 1) Single step deployment of three IAM Roles
 */

function createIAMRole(name: string): aws.iam.Role {
    // Create an IAM Role...
    return new aws.iam.Role(`${name}`, {
        assumeRolePolicy: `{
            "Version": "2012-10-17",
            "Statement":[
              {
                "Sid": "",
                "Effect": "Allow",
                "Principal": {
                  "AWS": "arn:aws:iam::153052954103:root"
                },
                "Action": "sts:AssumeRole"
              }
            ]
           }
        `,
          tags: {
              "clusterAccess": `${name}-usr`,
          },
        });
    };
}

// Administrator AWS IAM clusterAdminRole with full access to all AWS resources
const clusterAdminRole = createIAMRole("clusterAdminRole");

// Administer Automation role for use in pipelines, e.g. gitlab CI, Teamcity, etc.
const AutomationRole = createIAMRole("AutomationRole");

// Administer Prod role for use in Prod environment
const EnvProdRole = createIAMRole("EnvProdRole");

Step 2: Create one EKS cluster. Validate cluster creation. Add the namespaces you need.

The Traditional-approach:

You go through the steps below to validate the cluster and k8s resource deployment based on your tool chain and your understanding of Kubernetes:

$ eksctl create cluster eks-nd-test
 
$ kubectl get no
NAME STATUS ROLES AGE VERSION
ip-192-168-41-125.us-east-2.compute.internal Ready <none> 11h v1.11.5
ip-192-168-5-250.us-east-2.compute.internal Ready <none> 11h v1.11.5
 
$ kubectl create -f automation-ns.yaml && kubectl create -f prod-ns.yaml
 
$ cat automation-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: automation
  labels:
    name: automation

The Pulumi-approach:

You use our API docs and your source-code editor to autocomplete the default index.ts file.

/*
 * 2) Single step deployment of EKS cluster with the most important variables and a Simple Function to create namespaces
 * automation and prod
 */

const vpc = new awsx.ec2.Vpc("vpc", {});
const cluster = new eks.Cluster("eks-cluster", {
  vpcId : vpc.id,
  subnetIds : vpc.publicSubnetIds,
  instanceType : "t2.medium",
  nodeRootVolumeSize: 200,
  desiredCapacity : 1,
  maxSize : 2,
  minSize : 1,
  deployDashboard : false,
  vpcCniOptions : {
    warmIpTarget : 4,
  },
  roleMappings : [
    // Provides full administrator cluster access to the k8s cluster
    {
      groups : ["system:masters"],
      roleArn : clusterAdminRole.arn,
      username : "pulumi:admin-usr",
    },
    // Map IAM role arn "AutomationRoleArn" to the k8s user with name "automation-usr", e.g. gitlab CI
    {
      groups : ["pulumi:automation-grp"],
      roleArn : AutomationRole.arn,
      username : "pulumi:automation-usr",
    },
    // Map IAM role arn "EnvProdRoleArn" to the k8s user with name "prod-usr"
    {
      groups : ["pulumi:prod-grp"],
      roleArn : EnvProdRole.arn,
      username : "pulumi:prod-usr",
    },
  ],
});

export const clusterName = cluster.eksCluster.name;

function createNewNamespace(name: string): k8s.core.v1.Namespace {
  // Create new namespace
  return new k8s.core.v1.Namespace(name, { metadata: { name: name } }, { provider: cluster.provider });
}

// Declare namespaces automation and prod.
const automation = createNewNamespace("automation");
const prod = createNewNamespace("prod");

Step 3: Understand Kubernetes RBAC. Declare the Kubernetes objects on the EKS cluster.

The Kubernetes RBAC API declares four top-level types that can be defined as YAMLs syntaxes: a) Role - represents a set of additive rules within a namespace; b) RoleBinding - grants namespace-wide access to k8s subjects and resources; c) ClusterRole - represents a set of additive rules within the cluster; d) ClusterRoleBinding - grants cluster-wide access to k8s subjects and resources.

The Traditional-approach:

You define three k8s users with different privileges in your cluster and test them sequentially: User type1 called pulumi:admin-usr for users have cluster admin rights

 $ cat user1.yaml
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
  name: ClusterAdminRole
  # no namespace needed
 rules:
 - apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
  
 ---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: cluster-admin-binding
 subjects:
 - kind: User
  name: "pulumi:admin-usr"
  apiGroup: rbac.authorization.k8s.io
 roleRef:
  kind: ClusterRole
  name: ClusterAdminRole
  apiGroup: rbac.authorization.k8s.io`

User type2 called pulumi:automation-usr for users that have permissions to all k8s resources in namespace automation. An e.g would be your CI/CD pipeline

$ cat user2.yaml
 
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: AutomationRole
  namespace: automation
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
 
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: automation-binding
  namespace: automation
subjects:
- kind: User
  name: "pulumi:automation-usr"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: AutomationRole
  apiGroup: rbac.authorization.k8s.io

User type 3 called prod-usr for users that have read access to all k8s resources in the namespace prod

$ cat user3.yaml
 
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: EnvProdRole
  namespace: prod
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
 
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: env-prod-binding
  namespace: prod
subjects:
- kind: User
  name: "pulumi:prod-usr"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: EnvProdRole
  apiGroup: rbac.authorization.k8s.io`
$ kubectl apply -f user1.yaml && kubectl apply -f user2.yaml && kubectl apply -f user3.yaml`
 
clusterrole.rbac.authorization.k8s.io/ClusterAdminRole created
clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created
role.rbac.authorization.k8s.io/AutomationRole created
rolebinding.rbac.authorization.k8s.io/automation created
role.rbac.authorization.k8s.io/EnvProdRole created
rolebinding.rbac.authorization.k8s.io/env-prod-binding created

The Pulumi-approach:

Update your index.ts file with more code as follows:

/*
 * 3) Single Step deployment of k8s RBAC configuration for user1, user2 and user3 per our example
 */

// Grant cluster admin access to all admins with k8s ClusterRole and ClusterRoleBinding
new k8s.rbac.v1.ClusterRole("clusterAdminRole", {
  metadata: {
    name: "clusterAdminRole",
  },
  rules: [{
    apiGroups: ["*"],
    resources: ["*"],
    verbs: ["*"],
  }]
}, {provider: cluster.provider});

new k8s.rbac.v1.ClusterRoleBinding("cluster-admin-binding", {
  metadata: {
    name: "cluster-admin-binding",
  },
  subjects: [{
     kind: "User",
     name: "pulumi:admin-usr",
  }],
  roleRef: {
    kind: "ClusterRole",
    name: "clusterAdminRole",
    apiGroup: "rbac.authorization.k8s.io",
  },
}, {provider: cluster.provider});

// User2 called automation-usr for users that have permissions to all k8s resources in the namespace automation
new k8s.rbac.v1.Role("AutomationRole", {
  metadata: {
    name: "AutomationRole",
    namespace: "automation",
  },
  rules: [{
    apiGroups: ["*"],
    resources: ["*"],
    verbs: ["*"],
  }]
}, {provider: cluster.provider});
  
new k8s.rbac.v1.RoleBinding("automation-binding", {
  metadata: {
    name: "automation-binding",
    namespace: "automation",
  },
  subjects: [{
     kind: "User",
     name: "pulumi:automation-usr",
     apiGroup: "rbac.authorization.k8s.io",
  }],
  roleRef: {
    kind: "Role",
    name: "AutomationRole",
    apiGroup: "rbac.authorization.k8s.io",
  },
}, {provider: cluster.provider});

// User3 called prod-usr for users that have read access to all k8s resources in the namespace env-prod
new k8s.rbac.v1.Role("EnvProdRole", {
  metadata: {
    name: "EnvProdRole",
    namespace: "prod",
  },
  rules: [{
    apiGroups: ["*"],
    resources: ["*"],
    verbs: ["get", "watch", "list"],
  }],
}, {provider: cluster.provider});
  
new k8s.rbac.v1.RoleBinding("env-prod-binding", {
  metadata: {
    name: "env-prod-binding",
    namespace: "prod",
  },
  subjects: [{
     kind: "User",
     name: "pulumi:prod-usr",
     apiGroup: "rbac.authorization.k8s.io",
    }],
  roleRef: {
    kind: "Role",
    name: "EnvProdRole",
    apiGroup: "rbac.authorization.k8s.io",
  },
}, {provider: cluster.provider});

export const kubeconfig = cluster.kubeconfig

Step 4: Update aws-iam-authenticator ConfigMap to map the IAM Roles to the Kubernetes usernames.

The Traditional-approach:

You get the three IAM role arns for clusterAdminRole, AutomationRole, EnvProdRole and update the configmap with k8s usernames pulumi:admin-usr, pulumi:automation-usr and pulumi:prod-usr.

mapRoles:
----
- groups:
  - system:masters
  rolearn: arn:aws:iam::XXXXXXXXXXXX:role/clusterAdminRole
  username: pulumi:admin-usr
- groups:
  rolearn: arn:aws:iam::XXXXXXXXXXXX:role/AutomationRole
  username: pulumi:automation-usr
- groups:
  rolearn: arn:aws:iam::XXXXXXXXXXXX:role/EnvProdRole
  username: pulumi:prod-usr
- groups:
  - system:bootstrappers
  - system:nodes
  rolearn: arn:aws:iam::XXXXXXXXXXXX:role/eksctl-eks-rbac-nd-test-nodegroup-NodeInstanceRole-NP542EG8JX8U
  username: system:node:`

The Pulumi-approach:

This step is not required as you have already updated the EKS ConfigMap at cluster creation time in STEP 2 using “RoleMappings”. Simply run pulumi up with the full index.ts file. Watch all your components come alive simultaneously. Setting up RBAC on one EKS cluster is a long convoluted sequential process that requires multiple validations along the way. Imagine the complexity involved when working with multiple tools for an environment that requires multiple groups with many users, namespaces, and clusters.

Testing the Pulumi approach worked

Make sure you run pulumi up with this index.ts file.

$ pulumi stack output kubeconfig | jq > kubeconfig.yaml
$ export KUBECONFIG = kubeconfig.yaml

Assume the IAM role AutomationRole with access to all Kubernetes resources in namespace automation and test if the permissions work.

 "users": [
    {
      "name": "aws",
      "user": {
        "exec": {
          "apiVersion": "client.authentication.k8s.io/v1alpha1",
          "args": [
            "token",
            "-i",
            "eks-cluster-eksCluster-196b0de",
            "-r",
            "arn:aws:iam::xxxxxxxxxxxx:role/AutomationRole"
          ],
          "command": "aws-iam-authenticator"
        }
      }
    }
  ]
}
 
$ kubectl get po --namespace=automation
No resources found.
 
$ kubectl get po --namespace=prod
Error from server (Forbidden): pods is forbidden: User "pulumi: automation-usr" cannot list resource "pods" in API group "" in the namespace "prod"

Upon assuming the IAM role AutomationRole which is mapped to Kubernernetes username pulumi:automation-usr in the EKS cluster configmap, you are only restricted to the resources and verbs allowed in the namespace “automation” and not in namespace “prod”.

Next Step

In this post, we discussed how setting up Kubernetes RBAC with Pulumi is simple, comprehensive, non-sequential and part of your everyday programming experience. You can find the complete pulumi code for our example and try it out yourself.

Pulumi is open source and free to use. For more examples, visit our GitHub examples page here. To learn more about Pulumi and how to manage Kubernetes through code, have a look at our “Get Started with Kubernetes” guide.