What this guide covers
A production-shaped managed Kubernetes blueprint that consumes the Pulumi landing-zone stack and ships with the controllers most teams install by hand on day one. One Pulumi stack provisions the cluster, the add-ons workloads expect, the workload-identity wiring downstream stacks need, and outputs they can consume by name.
The blueprint covers:
- one Pulumi stack that provisions a managed Amazon EKS cluster inside the landing-zone VPC
- a small system node pool sized for the in-cluster controllers, with Karpenter handling every workload node on demand
- pinned installs of External Secrets Operator and AWS Load Balancer Controller through Helm, plus the cloud-side data-plane resources Pulumi provisions as part of the same stack
- IRSA (IAM Roles for Service Accounts) wired end-to-end for every service account the add-ons use, so pods call cloud APIs with short-lived tokens only
restrictedPod Security Admission labels on the add-on namespaces from the first deploy- a reusable
Clustercomponent so other Pulumi projects can provision the same cluster shape in their own stacks - a Pulumi ESC environment and
StackReferencesnippets every workload stack can import by name
Everything the blueprint creates is additive, so you can bring your own add-ons, node pools, or workloads on top without touching the module.
What gets deployed
In one Pulumi stack on AWS this blueprint provisions:
- Cluster control plane: a managed Amazon EKS cluster at Kubernetes version
1.33with IRSA (IAM Roles for Service Accounts) turned on so pods call cloud APIs with short-lived tokens instead of long-lived credentials. - System node pool: 2
t3.largeinstance(s) sized for the in-cluster controllers (External Secrets Operator, the ingress controller, and the cloud-native autoscaler itself). Every additional workload node is launched by Karpenter on demand. - Add-ons:
- External Secrets Operator chart
v2.3.0installed in theexternal-secretsnamespace with workload-identity-backed access to AWS Secrets Manager. - AWS Load Balancer Controller wired for Layer-7 ingress: the in-cluster controller is installed through Helm and the cloud-side data-plane service is provisioned by Pulumi so workload stacks can drop
Ingress/Gateway/HTTPRouteresources on the first deploy. - Karpenter configured to launch workload nodes on demand with scoped IAM/identity and the landing-zone network.
- External Secrets Operator chart
- Workload Identity: one identity per controller service account, scoped to a single namespace + service-account pair so no other pod can assume it.
- Pod Security Admission: the
restrictedprofile is enforced on theexternal-secretsand ingress-controller namespaces so privileged containers cannot land there by default.
Every resource is annotated with pulumi-stack, landing-zone, solution-family, cloud, and language labels/tags so workload stacks can filter them later. Cluster control-plane logs ship to the cloud-native audit destination the landing-zone stack already wires up (CloudWatch on AWS, Log Analytics on Azure, Cloud Logging on GCP).
On AWS
The blueprint uses Amazon EKS for the control plane, the landing-zone VPC for pod networking through the VPC CNI, and EKS managed node groups running AL2023 for the baseline system node group. Karpenter takes over all workload node launches afterwards.
The first deployment creates:
- one EKS cluster at Kubernetes
1.33with API endpoint private, envelope encryption using the landing-zone KMS key, and OIDC federation enabled - one managed node group of
2t3.largeinstances on the landing-zone private subnets - IRSA roles for External Secrets Operator, AWS Load Balancer Controller, and Karpenter
- EKS add-ons for
vpc-cni,kube-proxy, andcorednspinned to the versions the managed control plane ships with - three Helm releases for the controllers listed above, plus a Karpenter
EC2NodeClass+NodePooltargeting the same private subnets
Quickstart
Deploy the landing-zone stack first, then point this stack at it. The landing-zone stack owns the shared primitives this cluster plugs into: the landing-zone VPC the nodes run on, the customer-managed encryption key the control plane uses, the deployer identity that needs kubectl access, and the AWS Secrets Manager instance External Secrets Operator reads from. Keeping those in a separate stack lets one team own the account foundation while many teams stand up their own EKS clusters against it, and destroying a cluster never tears down the network other stacks depend on. This stack reads those outputs over a StackReference and fails fast if any are missing, so a missing landing-zone stack is the first thing pulumi up complains about.
- Make sure the Pulumi landing-zone stack for this cloud is already up. If not, follow the AWS landing-zone guide before coming back.
- Download the example zip at the top of the page and unzip it.
- Open a terminal in the extracted project root.
- Install the Pulumi dependencies for the language you want to use:
npm install
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
go mod tidy
- For a first local test, keep using whichever AWS credentials already work in your shell. If you want a shared or repeatable setup, use the Pulumi ESC section below before continuing.
- Create the stack, tell it which landing-zone stack to consume, and deploy:
pulumi login
pulumi stack init dev
pulumi config set aws:region us-west-2
pulumi config set landingZoneStack <your-org>/landing-zone/dev
pulumi up
- When the update finishes, export the kubeconfig and verify the cluster:
pulumi stack output kubeconfig --show-secrets > kubeconfig.yaml
KUBECONFIG=./kubeconfig.yaml kubectl get nodes
KUBECONFIG=./kubeconfig.yaml kubectl get pods -A
You should see the system nodes Ready and all three controllers (external-secrets, aws-load-balancer-controller, karpenter) running.
Prerequisites
- a Pulumi account and the Pulumi CLI installed
- the Pulumi landing-zone stack already deployed in this AWS account
- kubectl on your path
- Helm 3.14 or newer (the blueprint uses the Pulumi Helm Release resource; Helm on your machine is only required if you later want to
helminto the cluster by hand) - an AWS account where the Pulumi landing-zone stack is already deployed and you have permission to create EKS, IAM, and related resources
- Node.js 20 or newer and npm
Consume the landing-zone stack
This stack reads the outputs it needs from the landing-zone stack through a StackReference. For AWS:
privateSubnetIds- where the system node pool and Karpenter-launched nodes rundataEncryptionKeyArn- wraps the EKS envelope encryption keydeployerRoleArn- mapped through an EKS access entry bound toAmazonEKSClusterAdminPolicyso your Pulumi deployer identity cankubectlwithout editingaws-auth
Set which landing-zone stack to read:
pulumi config set landingZoneStack <your-org>/landing-zone/dev
The blueprint resolves that config value into a pulumi.StackReference at runtime and fails fast if any output it needs is missing. If you want to run this blueprint against a network you already manage, replace the StackReference block in the entrypoint with the ids you already have - the Cluster component does not care where those values come from.
Set up credentials with Pulumi ESC
Before you run pulumi up, configure Pulumi ESC so your stack receives short-lived AWS credentials through dynamic login credentials.
If you already have working AWS credentials in your shell and only want a quick local test, you can skip this section. The landing-zone family has a longer walkthrough that applies here verbatim; reuse the same ESC environment between landing-zone and Amazon EKS stacks so cluster upgrades run with the same deployer identity that created the network.
Step 1: Create or update an ESC environment
imports:
- <your-org>/base
values:
aws:
login:
fn::open::aws-login:
oidc:
roleArn: arn:aws:iam::123456789012:role/pulumi-esc
sessionName: pulumi-esc
environmentVariables:
AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
AWS_SESSION_TOKEN: ${aws.login.sessionToken}
pulumiConfig:
aws:region: us-east-1
Step 2: Attach the environment to your stack
In Pulumi.dev.yaml, add:
environment:
- <your-org>/<your-environment>
Pulumi picks up the environment automatically on pulumi preview, pulumi up, and pulumi destroy. You do not need to run esc open <your-org>/<your-environment> first.
What you get in the download
The downloadable example zip includes:
index.tsas the Pulumi entrypointcomponents/cluster.tsas the reusableClustermodulepackage.jsonandtsconfig.jsonfor the root Pulumi projectREADME.mdwith the same commands you will see on this page
index.tsas the Pulumi entrypointcomponents/cluster.tsas the reusableClustermodulepackage.jsonandtsconfig.jsonfor the root Pulumi project
__main__.pyas the Pulumi entrypointcomponents/cluster.pyas the reusableClustermodulerequirements.txtfor the root Pulumi project
main.goas the Pulumi entrypointcluster/cluster.goas the reusableClustermodulego.modfor the root Pulumi project
The entrypoint stays small: it loads the landing-zone outputs, reads a handful of config values, and instantiates the reusable Cluster component. The component file is where the cluster shape, add-on installs, and IRSA bindings live.
Deploy with Pulumi
Run these from the extracted project root.
Step 1: Install the root Pulumi dependencies for the language you want to use
npm install
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
go mod tidy
Step 2: Create a Pulumi stack and point it at your landing-zone stack
pulumi login
pulumi stack init dev
pulumi config set aws:region us-west-2
pulumi config set landingZoneStack <your-org>/landing-zone/dev
If you already created the stack, pulumi stack select dev instead.
Step 3: Deploy
pulumi up
Approve the preview when Pulumi asks.
The first run creates the EKS control plane, the system managed node group, IRSA roles for each controller, and the three Helm releases (External Secrets Operator, AWS Load Balancer Controller, Karpenter). Expect 12-18 minutes on a cold account; most of that time is EKS waiting for the control plane to become active.
Pulumi imports the ESC environment automatically through the environment: reference in your stack config. You do not need esc open <your-org>/<your-environment> before pulumi up.
Step 4: Verify the cluster
pulumi stack output kubeconfig --show-secrets > kubeconfig.yaml
KUBECONFIG=./kubeconfig.yaml kubectl get nodes
KUBECONFIG=./kubeconfig.yaml kubectl -n external-secrets rollout status deploy/external-secrets
KUBECONFIG=./kubeconfig.yaml kubectl -n kube-system rollout status deploy/aws-load-balancer-controller
KUBECONFIG=./kubeconfig.yaml kubectl -n karpenter rollout status deploy/karpenter
Controllers should report successfully rolled out (or Established for the GKE gateway class) once healthy.
Stack outputs
Every variant exports the same top-level shape so downstream Pulumi projects can consume the cluster the same way regardless of cloud. Run pulumi stack output --show-secrets after pulumi up to see values.
Common across AWS, Azure, and GCP:
kubeconfig(Pulumi secret) - authenticated kubeconfig you can feed intonew pulumi.Provider("kubernetes", { kubeconfig })clusterName- the provider-assigned cluster nameclusterEndpoint- the control-plane API endpointclusterCertificateAuthority- base64 CA cert, useful when the downstream stack builds its own kubeconfigescEnvironment- the Pulumi ESC environment name workload stacks import by reference
Amazon EKS-specific:
clusterSecurityGroupId- the cluster security group, useful for workload ingress rulesclusterOidcProviderArn- the OIDC provider ARN for IRSA bindings in other stackskarpenterNodeRoleArn- the instance role Karpenter uses for launched nodeskarpenterInterruptionQueueName- the SQS queue Karpenter uses for spot-interruption events
Add-ons
What is installed
Every variant installs the same three things, with cloud-appropriate wiring:
- External Secrets Operator (chart
v2.3.0, namespaceexternal-secrets) syncs secrets from AWS Secrets Manager into KubernetesSecretobjects. Its service account uses IRSA (IAM Roles for Service Accounts) so the operator authenticates with short-lived tokens. - AWS Load Balancer Controller is the Layer-7 entry point this cluster will answer
Ingress/Gateway/HTTPRouteresources on. - Karpenter handles workload-node launches. The system pool stays small; every additional node is launched by the autoscaler when a pending pod cannot fit.
The AWS variant additionally installs Karpenter from the oci://public.ecr.aws/karpenter/karpenter OCI chart, a scoped controller IAM role, a shared node instance profile, and an SQS interruption queue wired to EventBridge rules for spot, rebalance, instance state, and AWS health events. A blueprint EC2NodeClass (AL2023, al2023@latest alias) and NodePool (on-demand + spot, c/m/r families, generation >2, amd64) target the landing-zone private subnets via karpenter.sh/discovery tags.
Pod Security Admission
The add-on namespaces (external-secrets, plus the ingress-controller namespace for this cloud) are labelled with pod-security.kubernetes.io/enforce: restricted from the first deploy, matching the Kubernetes project’s recommended baseline for platform add-ons. Drop application workloads in new namespaces with your own PSA labels so the cluster never starts with a “default is permissive” story.
Add-on controls
Each add-on has a config flag. Disable any of them at pulumi up time:
pulumi config set enableExternalSecrets false
pulumi config set enableLoadBalancerController false
pulumi config set enableKarpenter false
Keeping an add-on disabled skips the Helm release and the identity resources that support it, so nothing orphans in your account. You can re-enable later and pulumi up again.
Add another add-on
The Cluster component exposes the in-cluster Kubernetes provider as an output. From the same program you can drop in additional kubernetes.helm.v3.Release resources against that provider and they will install on the same cluster alongside the blueprint add-ons. Keep workload-identity bindings inside the component if they need access to cloud APIs so the audit story stays consistent.
What the blueprint does NOT install
Intentionally out of scope for the first deploy: a full observability stack (Prometheus / Grafana / Loki) and a GitOps controller (Flux / Argo CD). Both are worth adding early - follow the pattern above or add them as dedicated families later.
Consume the cluster from workload stacks
Once the stack is up, every Pulumi workload project in the same AWS account can deploy into the cluster. Two patterns, pick whichever fits your team.
Pattern 1: Pulumi ESC environment
The stack attaches a Pulumi ESC environment (escEnvironment output). Downstream projects import it with one line in their stack config:
environment:
- your-org/aws-kubernetes-dev
After that, a kubernetes.Provider instantiated from pulumi.Config().requireSecret("kubeconfig") talks directly to this cluster.
Pattern 2: StackReference
If you prefer explicit wiring, use a StackReference:
import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
const cluster = new pulumi.StackReference("your-org/kubernetes/dev");
const kubeconfig = cluster.requireOutput("kubeconfig") as pulumi.Output<string>;
const provider = new k8s.Provider("workload", { kubeconfig });
import pulumi
import pulumi_kubernetes as k8s
cluster = pulumi.StackReference("your-org/kubernetes/dev")
kubeconfig = cluster.require_output("kubeconfig")
provider = k8s.Provider("workload", kubeconfig=kubeconfig)
cluster, err := pulumi.NewStackReference(ctx, "your-org/kubernetes/dev", nil)
if err != nil {
return err
}
kubeconfig := cluster.GetStringOutput(pulumi.String("kubeconfig"))
provider, err := kubernetes.NewProvider(ctx, "workload", &kubernetes.ProviderArgs{
Kubeconfig: kubeconfig,
})
if err != nil {
return err
}
Running workloads on Karpenter
Every variant launches nodes on demand; you do not need to manage node pools manually for application workloads.
Karpenter reads pending pods directly. Use standard nodeSelector / nodeAffinity to influence placement (for example karpenter.sh/capacity-type: spot), or leave them off to let the blueprint NodePool mix on-demand and spot freely. Pin workloads with a topologySpreadConstraint if you need AZ spread.
Using External Secrets
Create SecretStore (or ClusterSecretStore) and ExternalSecret resources that point at AWS Secrets Manager.
Provider: aws. Use the IRSA service account already annotated under external-secrets/external-secrets. Secrets must live under the landing-zone secrets prefix the IAM policy scopes.
Set up CI/CD with Pulumi Deployments
A managed Amazon EKS cluster is something you want updated from a tracked source, not from a laptop. Pulumi Deployments runs pulumi up from your GitHub repository whenever you merge to a branch.
What you will configure in Pulumi Deployments for this project:
- the Git repository and branch holding the unzipped blueprint
- the stack name (for example
your-org/aws-kubernetes/dev) - the root dependency command for the language you picked (
npm install) - the Pulumi ESC environment reference, so Deployments receives the same short-lived credentials as your local run
- the
landingZoneStackconfig value so Deployments knows which landing-zone stack to consume
Once Deployments is wired up, land add-on upgrades, Kubernetes version bumps, and node-pool changes through PRs. Workload stacks that consume this cluster pick up the new outputs automatically on their next pulumi up.
Blueprint Pulumi program
The blueprint keeps the entrypoint tight: it reads landing-zone outputs, configures the cluster, and instantiates the reusable Cluster component.
import * as pulumi from "@pulumi/pulumi";
import { Cluster } from "./components/cluster";
const config = new pulumi.Config();
const landingZoneStackName = config.require("landingZoneStack");
const clusterVersion = config.get("clusterVersion") ?? "1.33";
const systemInstanceType = config.get("systemInstanceType") ?? "t3.large";
const systemDesiredSize = config.getNumber("systemDesiredSize") ?? 2;
const enableExternalSecrets = config.getBoolean("enableExternalSecrets") ?? true;
const enableLoadBalancerController = config.getBoolean("enableLoadBalancerController") ?? true;
const enableKarpenter = config.getBoolean("enableKarpenter") ?? true;
const landingZone = new pulumi.StackReference(landingZoneStackName);
const vpcId = landingZone.requireOutput("networkId") as pulumi.Output<string>;
const publicSubnetIds = landingZone.requireOutput("publicSubnetIds") as pulumi.Output<string[]>;
const privateSubnetIds = landingZone.requireOutput("privateSubnetIds") as pulumi.Output<string[]>;
const encryptionKeyArn = landingZone.requireOutput("dataEncryptionKeyArn") as pulumi.Output<string>;
const deployerRoleArn = landingZone.requireOutput("deployerRoleArn") as pulumi.Output<string>;
const clusterName = `${pulumi.getStack()}-eks`;
const cluster = new Cluster("platform", {
clusterName,
version: clusterVersion,
vpcId,
publicSubnetIds,
privateSubnetIds,
encryptionKeyArn,
deployerRoleArn,
systemInstanceType,
systemDesiredSize,
enableExternalSecrets,
enableLoadBalancerController,
enableKarpenter,
externalSecretsChartVersion: "2.3.0",
albControllerChartVersion: "3.2.1",
karpenterChartVersion: "1.11.1",
tags: {
environment: pulumi.getStack(),
"solution-family": "kubernetes",
cloud: "aws",
language: "typescript",
},
});
export const kubeconfig = cluster.kubeconfig;
export const clusterNameOut = clusterName;
export const clusterEndpoint = cluster.cluster.eksCluster.endpoint;
export const clusterCertificateAuthority = cluster.cluster.eksCluster.certificateAuthority.apply(
(ca) => ca.data,
);
export const clusterSecurityGroupId = cluster.cluster.clusterSecurityGroupId;
export const clusterOidcProviderArn = cluster.oidcProviderArn;
export const karpenterNodeRoleArn = cluster.karpenterNodeRoleArn;
export const escEnvironment = `${pulumi.getStack()}-eks`;
import pulumi
from components.cluster import Cluster, ClusterArgs
config = pulumi.Config()
landing_zone_stack_name = config.require("landingZoneStack")
cluster_version = config.get("clusterVersion") or "1.33"
system_instance_type = config.get("systemInstanceType") or "t3.large"
system_desired_size = config.get_int("systemDesiredSize") or 2
enable_external_secrets = config.get_bool("enableExternalSecrets")
if enable_external_secrets is None:
enable_external_secrets = True
enable_load_balancer_controller = config.get_bool("enableLoadBalancerController")
if enable_load_balancer_controller is None:
enable_load_balancer_controller = True
enable_karpenter = config.get_bool("enableKarpenter")
if enable_karpenter is None:
enable_karpenter = True
landing_zone = pulumi.StackReference(landing_zone_stack_name)
vpc_id = landing_zone.require_output("networkId")
public_subnet_ids = landing_zone.require_output("publicSubnetIds")
private_subnet_ids = landing_zone.require_output("privateSubnetIds")
encryption_key_arn = landing_zone.require_output("dataEncryptionKeyArn")
deployer_role_arn = landing_zone.require_output("deployerRoleArn")
cluster_name = f"{pulumi.get_stack()}-eks"
cluster = Cluster(
"platform",
ClusterArgs(
cluster_name=cluster_name,
version=cluster_version,
vpc_id=vpc_id,
public_subnet_ids=public_subnet_ids,
private_subnet_ids=private_subnet_ids,
encryption_key_arn=encryption_key_arn,
deployer_role_arn=deployer_role_arn,
system_instance_type=system_instance_type,
system_desired_size=system_desired_size,
enable_external_secrets=enable_external_secrets,
enable_load_balancer_controller=enable_load_balancer_controller,
enable_karpenter=enable_karpenter,
external_secrets_chart_version="2.3.0",
alb_controller_chart_version="3.2.1",
karpenter_chart_version="1.11.1",
tags={
"environment": pulumi.get_stack(),
"solution-family": "kubernetes",
"cloud": "aws",
"language": "python",
},
),
)
pulumi.export("kubeconfig", cluster.kubeconfig)
pulumi.export("clusterName", cluster_name)
pulumi.export("clusterEndpoint", cluster.cluster.core.endpoint)
pulumi.export("clusterCertificateAuthority", cluster.cluster.core.cluster.certificate_authorities.apply(lambda cas: cas[0].data))
pulumi.export("clusterSecurityGroupId", cluster.cluster.core.cluster_security_group.id)
pulumi.export("clusterOidcProviderArn", cluster.oidc_provider_arn)
pulumi.export("karpenterNodeRoleArn", cluster.karpenter_node_role_arn)
pulumi.export("escEnvironment", f"{pulumi.get_stack()}-eks")
package main
import (
"fmt"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
"kubernetes-aws/cluster"
)
func main() {
pulumi.Run(Program)
}
func Program(ctx *pulumi.Context) error {
cfg := config.New(ctx, "")
landingZoneStackName := cfg.Require("landingZoneStack")
clusterVersion := cfg.Get("clusterVersion")
if clusterVersion == "" {
clusterVersion = "1.33"
}
systemInstanceType := cfg.Get("systemInstanceType")
if systemInstanceType == "" {
systemInstanceType = "t3.large"
}
systemDesiredSize := cfg.GetInt("systemDesiredSize")
if systemDesiredSize == 0 {
systemDesiredSize = 2
}
enableExternalSecrets := true
if v, err := cfg.TryBool("enableExternalSecrets"); err == nil {
enableExternalSecrets = v
}
enableLoadBalancerController := true
if v, err := cfg.TryBool("enableLoadBalancerController"); err == nil {
enableLoadBalancerController = v
}
enableKarpenter := true
if v, err := cfg.TryBool("enableKarpenter"); err == nil {
enableKarpenter = v
}
landingZone, err := pulumi.NewStackReference(ctx, landingZoneStackName, nil)
if err != nil {
return err
}
vpcId := landingZone.GetStringOutput(pulumi.String("networkId"))
publicSubnetIds := landingZone.GetOutput(pulumi.String("publicSubnetIds")).ApplyT(func(v interface{}) []string {
return castStringSlice(v)
}).(pulumi.StringArrayOutput)
privateSubnetIds := landingZone.GetOutput(pulumi.String("privateSubnetIds")).ApplyT(func(v interface{}) []string {
return castStringSlice(v)
}).(pulumi.StringArrayOutput)
encryptionKeyArn := landingZone.GetStringOutput(pulumi.String("dataEncryptionKeyArn"))
deployerRoleArn := landingZone.GetStringOutput(pulumi.String("deployerRoleArn"))
clusterName := fmt.Sprintf("%s-eks", ctx.Stack())
c, err := cluster.New(ctx, "platform", &cluster.Args{
ClusterName: pulumi.String(clusterName),
Version: pulumi.String(clusterVersion),
VpcId: vpcId,
PublicSubnetIds: publicSubnetIds,
PrivateSubnetIds: privateSubnetIds,
EncryptionKeyArn: encryptionKeyArn,
DeployerRoleArn: deployerRoleArn,
SystemInstanceType: pulumi.String(systemInstanceType),
SystemDesiredSize: pulumi.Int(systemDesiredSize),
EnableExternalSecrets: enableExternalSecrets,
EnableLoadBalancerController: enableLoadBalancerController,
EnableKarpenter: enableKarpenter,
ExternalSecretsChartVersion: "2.3.0",
AlbControllerChartVersion: "3.2.1",
KarpenterChartVersion: "1.11.1",
Tags: pulumi.StringMap{
"environment": pulumi.String(ctx.Stack()),
"solution-family": pulumi.String("kubernetes"),
"cloud": pulumi.String("aws"),
"language": pulumi.String("go"),
},
})
if err != nil {
return err
}
ctx.Export("kubeconfig", c.Kubeconfig)
ctx.Export("clusterName", pulumi.String(clusterName))
ctx.Export("clusterEndpoint", c.ClusterEndpoint)
ctx.Export("clusterCertificateAuthority", c.CertificateAuthority)
ctx.Export("clusterSecurityGroupId", c.ClusterSecurityGroupId)
ctx.Export("clusterOidcProviderArn", c.OidcProviderArn)
ctx.Export("karpenterNodeRoleArn", c.KarpenterNodeRoleArn)
ctx.Export("escEnvironment", pulumi.Sprintf("%s-eks", ctx.Stack()))
return nil
}
func castStringSlice(v interface{}) []string {
if v == nil {
return nil
}
raw, ok := v.([]interface{})
if !ok {
return nil
}
result := make([]string, 0, len(raw))
for _, item := range raw {
if s, ok := item.(string); ok {
result = append(result, s)
}
}
return result
}
Reusable components
The cluster wiring and add-on installs live in a reusable module so you can import it from other Pulumi projects or adapt it per team.
components/cluster.ts
Provisions the EKS cluster, a system node pool sized for the controllers, workload-identity wiring (IRSA (IAM Roles for Service Accounts)), and the Helm releases for External Secrets Operator and the ingress controller for this cloud.
import * as aws from "@pulumi/aws";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import { albControllerIamPolicy } from "./alb-controller-policy";
import { karpenterControllerIamPolicy } from "./karpenter-controller-policy";
export interface ClusterArgs {
clusterName: pulumi.Input<string>;
version: pulumi.Input<string>;
vpcId: pulumi.Input<string>;
publicSubnetIds: pulumi.Input<pulumi.Input<string>[]>;
privateSubnetIds: pulumi.Input<pulumi.Input<string>[]>;
encryptionKeyArn: pulumi.Input<string>;
deployerRoleArn: pulumi.Input<string>;
systemInstanceType: pulumi.Input<string>;
systemDesiredSize: pulumi.Input<number>;
enableExternalSecrets: boolean;
enableLoadBalancerController: boolean;
enableKarpenter: boolean;
externalSecretsChartVersion: string;
albControllerChartVersion: string;
karpenterChartVersion: string;
tags: Record<string, string>;
}
export class Cluster extends pulumi.ComponentResource {
public readonly cluster: eks.Cluster;
public readonly provider: k8s.Provider;
public readonly kubeconfig: pulumi.Output<string>;
public readonly oidcProviderArn: pulumi.Output<string>;
public readonly karpenterNodeRoleArn: pulumi.Output<string>;
public readonly karpenterInterruptionQueueName: pulumi.Output<string>;
constructor(name: string, args: ClusterArgs, opts?: pulumi.ComponentResourceOptions) {
super("pulumi:solutions:EksCluster", name, {}, opts);
const parent = { parent: this };
const clusterNameInput = pulumi.output(args.clusterName);
const cluster = new eks.Cluster(name, {
name: args.clusterName,
version: args.version,
vpcId: args.vpcId,
publicSubnetIds: args.publicSubnetIds,
privateSubnetIds: args.privateSubnetIds,
nodeAssociatePublicIpAddress: false,
endpointPrivateAccess: true,
endpointPublicAccess: true,
createOidcProvider: true,
skipDefaultNodeGroup: true,
authenticationMode: eks.AuthenticationMode.ApiAndConfigMap,
accessEntries: {
deployer: {
principalArn: args.deployerRoleArn,
accessPolicies: {
admin: {
policyArn: "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy",
accessScope: { type: "cluster" },
},
},
},
},
encryptionConfigKeyArn: args.encryptionKeyArn,
tags: args.tags,
}, parent);
const oidcProviderArn = cluster.oidcProviderArn;
const oidcProviderUrl = cluster.oidcProviderUrl;
const oidcProviderUrlNoScheme = oidcProviderUrl.apply(u => u.replace(/^https:\/\//, ""));
const systemRole = new aws.iam.Role(`${name}-system-node`, {
assumeRolePolicy: JSON.stringify({
Version: "2012-10-17",
Statement: [{
Effect: "Allow",
Principal: { Service: "ec2.amazonaws.com" },
Action: "sts:AssumeRole",
}],
}),
tags: args.tags,
}, parent);
for (const policy of [
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
]) {
new aws.iam.RolePolicyAttachment(`${name}-system-node-${policy.split("/").pop()}`, {
role: systemRole.name,
policyArn: policy,
}, parent);
}
new eks.ManagedNodeGroup(`${name}-system`, {
cluster: cluster,
nodeRole: systemRole,
instanceTypes: [args.systemInstanceType as pulumi.Input<string>],
subnetIds: args.privateSubnetIds,
scalingConfig: {
desiredSize: args.systemDesiredSize,
minSize: args.systemDesiredSize,
maxSize: args.systemDesiredSize,
},
labels: { "karpenter.sh/controller": "true" },
tags: args.tags,
}, parent);
pulumi.output(args.privateSubnetIds).apply(subnets => {
subnets.forEach((subnetId, idx) => {
new aws.ec2.Tag(`${name}-karpenter-subnet-${idx}`, {
resourceId: subnetId,
key: "karpenter.sh/discovery",
value: clusterNameInput,
}, parent);
});
});
new aws.ec2.Tag(`${name}-karpenter-sg`, {
resourceId: cluster.clusterSecurityGroupId,
key: "karpenter.sh/discovery",
value: clusterNameInput,
}, parent);
const helmProvider = new k8s.Provider(`${name}-k8s`, {
kubeconfig: cluster.kubeconfigJson,
}, parent);
const serviceAccountAssume = (namespace: string, serviceAccount: string): pulumi.Output<string> =>
pulumi.all([oidcProviderArn, oidcProviderUrlNoScheme]).apply(([arn, url]) => JSON.stringify({
Version: "2012-10-17",
Statement: [{
Effect: "Allow",
Principal: { Federated: arn },
Action: "sts:AssumeRoleWithWebIdentity",
Condition: {
StringEquals: {
[`${url}:aud`]: "sts.amazonaws.com",
[`${url}:sub`]: `system:serviceaccount:${namespace}:${serviceAccount}`,
},
},
}],
}));
if (args.enableExternalSecrets) {
const role = new aws.iam.Role(`${name}-eso`, {
assumeRolePolicy: serviceAccountAssume("external-secrets", "external-secrets"),
tags: args.tags,
}, parent);
new aws.iam.RolePolicy(`${name}-eso-secrets`, {
role: role.id,
policy: JSON.stringify({
Version: "2012-10-17",
Statement: [
{
Effect: "Allow",
Action: [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecrets",
],
Resource: "*",
},
{
Effect: "Allow",
Action: [
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:GetParametersByPath",
],
Resource: "*",
},
],
}),
}, parent);
new k8s.helm.v3.Release(`${name}-eso`, {
chart: "external-secrets",
version: args.externalSecretsChartVersion,
namespace: "external-secrets",
createNamespace: true,
repositoryOpts: { repo: "https://charts.external-secrets.io" },
values: {
serviceAccount: {
annotations: { "eks.amazonaws.com/role-arn": role.arn },
},
installCRDs: true,
},
}, { ...parent, provider: helmProvider });
}
if (args.enableLoadBalancerController) {
const role = new aws.iam.Role(`${name}-alb`, {
assumeRolePolicy: serviceAccountAssume("kube-system", "aws-load-balancer-controller"),
tags: args.tags,
}, parent);
new aws.iam.RolePolicy(`${name}-alb-policy`, {
role: role.id,
policy: albControllerIamPolicy,
}, parent);
new k8s.helm.v3.Release(`${name}-alb`, {
chart: "aws-load-balancer-controller",
version: args.albControllerChartVersion,
namespace: "kube-system",
repositoryOpts: { repo: "https://aws.github.io/eks-charts" },
values: {
clusterName: args.clusterName,
region: aws.getRegionOutput().name,
vpcId: args.vpcId,
serviceAccount: {
create: true,
name: "aws-load-balancer-controller",
annotations: { "eks.amazonaws.com/role-arn": role.arn },
},
},
}, { ...parent, provider: helmProvider });
}
const karpenterNodeRole = new aws.iam.Role(`${name}-karpenter-node`, {
assumeRolePolicy: JSON.stringify({
Version: "2012-10-17",
Statement: [{
Effect: "Allow",
Principal: { Service: "ec2.amazonaws.com" },
Action: "sts:AssumeRole",
}],
}),
tags: args.tags,
}, parent);
for (const policy of [
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
]) {
new aws.iam.RolePolicyAttachment(`${name}-karpenter-node-${policy.split("/").pop()}`, {
role: karpenterNodeRole.name,
policyArn: policy,
}, parent);
}
const karpenterInstanceProfile = new aws.iam.InstanceProfile(`${name}-karpenter-node`, {
role: karpenterNodeRole.name,
tags: args.tags,
}, parent);
const karpenterAccessEntry = new aws.eks.AccessEntry(`${name}-karpenter-node`, {
clusterName: cluster.eksCluster.name,
principalArn: karpenterNodeRole.arn,
type: "EC2_LINUX",
}, parent);
const interruptionQueue = new aws.sqs.Queue(`${name}-karpenter`, {
messageRetentionSeconds: 300,
sqsManagedSseEnabled: true,
tags: args.tags,
}, parent);
new aws.sqs.QueuePolicy(`${name}-karpenter`, {
queueUrl: interruptionQueue.url,
policy: interruptionQueue.arn.apply(arn => JSON.stringify({
Version: "2012-10-17",
Id: "KarpenterInterruptionQueue",
Statement: [{
Effect: "Allow",
Principal: { Service: ["events.amazonaws.com", "sqs.amazonaws.com"] },
Action: "sqs:SendMessage",
Resource: arn,
}],
})),
}, parent);
const eventRules: Array<[string, Record<string, any>]> = [
["health", { source: ["aws.health"], "detail-type": ["AWS Health Event"] }],
["spot", { source: ["aws.ec2"], "detail-type": ["EC2 Spot Instance Interruption Warning"] }],
["rebalance", { source: ["aws.ec2"], "detail-type": ["EC2 Instance Rebalance Recommendation"] }],
["state-change", { source: ["aws.ec2"], "detail-type": ["EC2 Instance State-change Notification"] }],
];
for (const [key, pattern] of eventRules) {
const rule = new aws.cloudwatch.EventRule(`${name}-karpenter-${key}`, {
eventPattern: JSON.stringify(pattern),
}, parent);
new aws.cloudwatch.EventTarget(`${name}-karpenter-${key}`, {
rule: rule.name,
arn: interruptionQueue.arn,
}, parent);
}
if (args.enableKarpenter) {
const controllerRole = new aws.iam.Role(`${name}-karpenter`, {
assumeRolePolicy: serviceAccountAssume("karpenter", "karpenter"),
tags: args.tags,
}, parent);
new aws.iam.RolePolicy(`${name}-karpenter-policy`, {
role: controllerRole.id,
policy: pulumi.all([
clusterNameInput,
karpenterNodeRole.arn,
interruptionQueue.arn,
aws.getRegionOutput().name,
aws.getCallerIdentityOutput().accountId,
]).apply(([clusterNameResolved, nodeRoleArn, queueArn, region, accountId]) =>
karpenterControllerIamPolicy({
clusterName: clusterNameResolved,
nodeRoleArn,
queueArn,
region,
accountId,
})),
}, parent);
const karpenterRelease = new k8s.helm.v3.Release(`${name}-karpenter`, {
chart: "oci://public.ecr.aws/karpenter/karpenter",
version: args.karpenterChartVersion,
namespace: "karpenter",
createNamespace: true,
values: {
settings: {
clusterName: args.clusterName,
interruptionQueue: interruptionQueue.name,
},
serviceAccount: {
name: "karpenter",
annotations: { "eks.amazonaws.com/role-arn": controllerRole.arn },
},
controller: {
resources: {
requests: { cpu: "1", memory: "1Gi" },
limits: { cpu: "1", memory: "1Gi" },
},
},
},
}, { ...parent, provider: helmProvider });
new k8s.apiextensions.CustomResource(`${name}-karpenter-nodeclass`, {
apiVersion: "karpenter.k8s.aws/v1",
kind: "EC2NodeClass",
metadata: { name: "default" },
spec: {
amiFamily: "AL2023",
role: karpenterNodeRole.name,
subnetSelectorTerms: [{ tags: { "karpenter.sh/discovery": clusterNameInput } }],
securityGroupSelectorTerms: [{ tags: { "karpenter.sh/discovery": clusterNameInput } }],
amiSelectorTerms: [
{ alias: pulumi.interpolate`al2023@latest` },
],
tags: { ...args.tags, "karpenter.sh/nodepool": "default" },
},
}, { ...parent, provider: helmProvider, dependsOn: [karpenterRelease, karpenterAccessEntry] });
new k8s.apiextensions.CustomResource(`${name}-karpenter-nodepool`, {
apiVersion: "karpenter.sh/v1",
kind: "NodePool",
metadata: { name: "default" },
spec: {
template: {
spec: {
nodeClassRef: {
group: "karpenter.k8s.aws",
kind: "EC2NodeClass",
name: "default",
},
requirements: [
{ key: "kubernetes.io/arch", operator: "In", values: ["amd64"] },
{ key: "karpenter.sh/capacity-type", operator: "In", values: ["on-demand", "spot"] },
{ key: "karpenter.k8s.aws/instance-category", operator: "In", values: ["c", "m", "r"] },
{ key: "karpenter.k8s.aws/instance-generation", operator: "Gt", values: ["2"] },
],
expireAfter: "720h",
},
},
limits: { cpu: "1000", memory: "1000Gi" },
disruption: {
consolidationPolicy: "WhenEmptyOrUnderutilized",
consolidateAfter: "30s",
},
},
}, { ...parent, provider: helmProvider, dependsOn: [karpenterRelease] });
}
this.cluster = cluster;
this.provider = helmProvider;
this.kubeconfig = cluster.kubeconfigJson;
this.oidcProviderArn = oidcProviderArn;
this.karpenterNodeRoleArn = karpenterNodeRole.arn;
this.karpenterInterruptionQueueName = interruptionQueue.name;
this.registerOutputs({
cluster: this.cluster,
kubeconfig: this.kubeconfig,
oidcProviderArn: this.oidcProviderArn,
karpenterNodeRoleArn: this.karpenterNodeRoleArn,
karpenterInterruptionQueueName: this.karpenterInterruptionQueueName,
});
}
}
components/alb-controller-policy.ts
Builds the IAM policy the AWS Load Balancer Controller attaches through IRSA. Copied verbatim from the upstream install/iam_policy.json at the ALB controller version this family pins.
export const albControllerIamPolicy = JSON.stringify({
Version: "2012-10-17",
Statement: [
{
Effect: "Allow",
Action: ["iam:CreateServiceLinkedRole"],
Resource: "*",
Condition: { StringEquals: { "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com" } },
},
{
Effect: "Allow",
Action: [
"ec2:DescribeAccountAttributes",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInternetGateways",
"ec2:DescribeVpcs",
"ec2:DescribeVpcPeeringConnections",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeInstances",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeTags",
"ec2:GetCoipPoolUsage",
"ec2:DescribeCoipPools",
"ec2:GetSecurityGroupsForVpc",
"ec2:DescribeIpamPools",
"ec2:DescribeRouteTables",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeListenerCertificates",
"elasticloadbalancing:DescribeSSLPolicies",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTrustStores",
"elasticloadbalancing:DescribeListenerAttributes",
"elasticloadbalancing:DescribeCapacityReservation",
],
Resource: "*",
},
{
Effect: "Allow",
Action: [
"cognito-idp:DescribeUserPoolClient",
"acm:ListCertificates",
"acm:DescribeCertificate",
"iam:ListServerCertificates",
"iam:GetServerCertificate",
"waf-regional:GetWebACL",
"waf-regional:GetWebACLForResource",
"waf-regional:AssociateWebACL",
"waf-regional:DisassociateWebACL",
"wafv2:GetWebACL",
"wafv2:GetWebACLForResource",
"wafv2:AssociateWebACL",
"wafv2:DisassociateWebACL",
"shield:GetSubscriptionState",
"shield:DescribeProtection",
"shield:CreateProtection",
"shield:DeleteProtection",
],
Resource: "*",
},
{
Effect: "Allow",
Action: ["ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress"],
Resource: "*",
},
{ Effect: "Allow", Action: ["ec2:CreateSecurityGroup"], Resource: "*" },
{
Effect: "Allow",
Action: ["ec2:CreateTags"],
Resource: "arn:aws:ec2:*:*:security-group/*",
Condition: {
StringEquals: { "ec2:CreateAction": "CreateSecurityGroup" },
Null: { "aws:RequestTag/elbv2.k8s.aws/cluster": "false" },
},
},
{
Effect: "Allow",
Action: ["ec2:CreateTags", "ec2:DeleteTags"],
Resource: "arn:aws:ec2:*:*:security-group/*",
Condition: {
Null: {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false",
},
},
},
{
Effect: "Allow",
Action: [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"ec2:DeleteSecurityGroup",
],
Resource: "*",
Condition: { Null: { "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" } },
},
{
Effect: "Allow",
Action: [
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateTargetGroup",
],
Resource: "*",
Condition: { Null: { "aws:RequestTag/elbv2.k8s.aws/cluster": "false" } },
},
{
Effect: "Allow",
Action: [
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:DeleteRule",
],
Resource: "*",
},
{
Effect: "Allow",
Action: ["elasticloadbalancing:AddTags", "elasticloadbalancing:RemoveTags"],
Resource: [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*",
],
Condition: {
Null: {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false",
},
},
},
{
Effect: "Allow",
Action: ["elasticloadbalancing:AddTags", "elasticloadbalancing:RemoveTags"],
Resource: [
"arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*",
],
},
{
Effect: "Allow",
Action: [
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetSubnets",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:ModifyListenerAttributes",
"elasticloadbalancing:ModifyCapacityReservation",
"elasticloadbalancing:ModifyIpPools",
],
Resource: "*",
Condition: { Null: { "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" } },
},
{
Effect: "Allow",
Action: ["elasticloadbalancing:AddTags"],
Resource: [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*",
],
Condition: {
StringEquals: {
"elasticloadbalancing:CreateAction": ["CreateTargetGroup", "CreateLoadBalancer"],
},
Null: { "aws:RequestTag/elbv2.k8s.aws/cluster": "false" },
},
},
{
Effect: "Allow",
Action: ["elasticloadbalancing:RegisterTargets", "elasticloadbalancing:DeregisterTargets"],
Resource: "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
},
{
Effect: "Allow",
Action: [
"elasticloadbalancing:SetWebAcl",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:RemoveListenerCertificates",
"elasticloadbalancing:ModifyRule",
"elasticloadbalancing:SetRulePriorities",
],
Resource: "*",
},
],
});
components/karpenter-controller-policy.ts
Builds the Karpenter controller IAM policy. Mirrors the scoped statements from the Karpenter v1.x CloudFormation template, narrowed with aws:ResourceTag conditions on the cluster name.
export interface KarpenterPolicyArgs {
clusterName: string;
nodeRoleArn: string;
queueArn: string;
region: string;
accountId: string;
}
export function karpenterControllerIamPolicy(args: KarpenterPolicyArgs): string {
const clusterTag = `aws:ResourceTag/kubernetes.io/cluster/${args.clusterName}`;
const clusterArn = `arn:aws:eks:${args.region}:${args.accountId}:cluster/${args.clusterName}`;
return JSON.stringify({
Version: "2012-10-17",
Statement: [
{
Sid: "AllowScopedEC2InstanceAccessActions",
Effect: "Allow",
Action: ["ec2:RunInstances", "ec2:CreateFleet"],
Resource: [
`arn:aws:ec2:${args.region}::image/*`,
`arn:aws:ec2:${args.region}::snapshot/*`,
`arn:aws:ec2:${args.region}:*:security-group/*`,
`arn:aws:ec2:${args.region}:*:subnet/*`,
`arn:aws:ec2:${args.region}:*:capacity-reservation/*`,
],
},
{
Sid: "AllowScopedEC2LaunchTemplateAccessActions",
Effect: "Allow",
Action: ["ec2:RunInstances", "ec2:CreateFleet"],
Resource: `arn:aws:ec2:${args.region}:*:launch-template/*`,
Condition: {
StringEquals: { [clusterTag]: "owned" },
StringLike: { "aws:ResourceTag/karpenter.sh/nodepool": "*" },
},
},
{
Sid: "AllowScopedEC2InstanceActionsWithTags",
Effect: "Allow",
Action: [
"ec2:RunInstances",
"ec2:CreateFleet",
"ec2:CreateLaunchTemplate",
],
Resource: [
`arn:aws:ec2:${args.region}:*:fleet/*`,
`arn:aws:ec2:${args.region}:*:instance/*`,
`arn:aws:ec2:${args.region}:*:volume/*`,
`arn:aws:ec2:${args.region}:*:network-interface/*`,
`arn:aws:ec2:${args.region}:*:launch-template/*`,
`arn:aws:ec2:${args.region}:*:spot-instances-request/*`,
],
Condition: {
StringEquals: {
[`aws:RequestTag/kubernetes.io/cluster/${args.clusterName}`]: "owned",
[`aws:RequestTag/eks:eks-cluster-name`]: args.clusterName,
},
StringLike: { "aws:RequestTag/karpenter.sh/nodepool": "*" },
},
},
{
Sid: "AllowScopedResourceCreationTagging",
Effect: "Allow",
Action: "ec2:CreateTags",
Resource: [
`arn:aws:ec2:${args.region}:*:fleet/*`,
`arn:aws:ec2:${args.region}:*:instance/*`,
`arn:aws:ec2:${args.region}:*:volume/*`,
`arn:aws:ec2:${args.region}:*:network-interface/*`,
`arn:aws:ec2:${args.region}:*:launch-template/*`,
`arn:aws:ec2:${args.region}:*:spot-instances-request/*`,
],
Condition: {
StringEquals: {
[`aws:RequestTag/kubernetes.io/cluster/${args.clusterName}`]: "owned",
"ec2:CreateAction": [
"RunInstances",
"CreateFleet",
"CreateLaunchTemplate",
],
},
StringLike: { "aws:RequestTag/karpenter.sh/nodepool": "*" },
},
},
{
Sid: "AllowScopedResourceTagging",
Effect: "Allow",
Action: "ec2:CreateTags",
Resource: `arn:aws:ec2:${args.region}:*:instance/*`,
Condition: {
StringEquals: { [clusterTag]: "owned" },
StringLike: { "aws:ResourceTag/karpenter.sh/nodepool": "*" },
StringEqualsIfExists: {
"aws:RequestTag/eks:eks-cluster-name": args.clusterName,
},
"ForAllValues:StringEquals": {
"aws:TagKeys": [
"eks:eks-cluster-name",
"karpenter.sh/nodeclaim",
"Name",
],
},
},
},
{
Sid: "AllowScopedDeletion",
Effect: "Allow",
Action: ["ec2:TerminateInstances", "ec2:DeleteLaunchTemplate"],
Resource: [
`arn:aws:ec2:${args.region}:*:instance/*`,
`arn:aws:ec2:${args.region}:*:launch-template/*`,
],
Condition: {
StringEquals: { [clusterTag]: "owned" },
StringLike: { "aws:ResourceTag/karpenter.sh/nodepool": "*" },
},
},
{
Sid: "AllowRegionalReadActions",
Effect: "Allow",
Action: [
"ec2:DescribeCapacityReservations",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeInstanceTypes",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSpotPriceHistory",
"ec2:DescribeSubnets",
"ec2:DescribeAvailabilityZones",
],
Resource: "*",
Condition: {
StringEquals: { "aws:RequestedRegion": args.region },
},
},
{
Sid: "AllowSSMReadActions",
Effect: "Allow",
Action: "ssm:GetParameter",
Resource: `arn:aws:ssm:${args.region}::parameter/aws/service/*`,
},
{
Sid: "AllowPricingReadActions",
Effect: "Allow",
Action: "pricing:GetProducts",
Resource: "*",
},
{
Sid: "AllowInterruptionQueueActions",
Effect: "Allow",
Action: [
"sqs:DeleteMessage",
"sqs:GetQueueUrl",
"sqs:ReceiveMessage",
],
Resource: args.queueArn,
},
{
Sid: "AllowPassingInstanceRole",
Effect: "Allow",
Action: "iam:PassRole",
Resource: args.nodeRoleArn,
Condition: {
StringEquals: { "iam:PassedToService": "ec2.amazonaws.com" },
},
},
{
Sid: "AllowScopedInstanceProfileCreationActions",
Effect: "Allow",
Action: ["iam:CreateInstanceProfile"],
Resource: "*",
Condition: {
StringEquals: {
[`aws:RequestTag/kubernetes.io/cluster/${args.clusterName}`]: "owned",
"aws:RequestTag/eks:eks-cluster-name": args.clusterName,
"aws:RequestTag/topology.kubernetes.io/region": args.region,
},
StringLike: { "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" },
},
},
{
Sid: "AllowScopedInstanceProfileTagActions",
Effect: "Allow",
Action: ["iam:TagInstanceProfile"],
Resource: "*",
Condition: {
StringEquals: {
[`aws:ResourceTag/kubernetes.io/cluster/${args.clusterName}`]: "owned",
"aws:ResourceTag/topology.kubernetes.io/region": args.region,
[`aws:RequestTag/kubernetes.io/cluster/${args.clusterName}`]: "owned",
"aws:RequestTag/eks:eks-cluster-name": args.clusterName,
"aws:RequestTag/topology.kubernetes.io/region": args.region,
},
StringLike: {
"aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
"aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*",
},
},
},
{
Sid: "AllowScopedInstanceProfileActions",
Effect: "Allow",
Action: [
"iam:AddRoleToInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:DeleteInstanceProfile",
],
Resource: "*",
Condition: {
StringEquals: {
[`aws:ResourceTag/kubernetes.io/cluster/${args.clusterName}`]: "owned",
"aws:ResourceTag/topology.kubernetes.io/region": args.region,
},
StringLike: { "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*" },
},
},
{
Sid: "AllowInstanceProfileReadActions",
Effect: "Allow",
Action: "iam:GetInstanceProfile",
Resource: "*",
},
{
Sid: "AllowAPIServerEndpointDiscovery",
Effect: "Allow",
Action: "eks:DescribeCluster",
Resource: clusterArn,
},
],
});
}
components/cluster.py
Provisions the EKS cluster, a system node pool sized for the controllers, workload-identity wiring (IRSA (IAM Roles for Service Accounts)), and the Helm releases for External Secrets Operator and the ingress controller for this cloud.
import json
from dataclasses import dataclass, field
from typing import Mapping, Sequence
import pulumi
import pulumi_aws as aws
import pulumi_eks as eks
import pulumi_kubernetes as k8s
from .alb_controller_policy import ALB_CONTROLLER_IAM_POLICY
from .karpenter_controller_policy import KarpenterPolicyArgs, karpenter_controller_iam_policy
@dataclass
class ClusterArgs:
cluster_name: pulumi.Input[str]
version: pulumi.Input[str]
vpc_id: pulumi.Input[str]
public_subnet_ids: pulumi.Input[Sequence[pulumi.Input[str]]]
private_subnet_ids: pulumi.Input[Sequence[pulumi.Input[str]]]
encryption_key_arn: pulumi.Input[str]
deployer_role_arn: pulumi.Input[str]
system_instance_type: pulumi.Input[str]
system_desired_size: pulumi.Input[int]
enable_external_secrets: bool = True
enable_load_balancer_controller: bool = True
enable_karpenter: bool = True
external_secrets_chart_version: str = ""
alb_controller_chart_version: str = ""
karpenter_chart_version: str = ""
tags: Mapping[str, str] = field(default_factory=dict)
class Cluster(pulumi.ComponentResource):
def __init__(self, name: str, args: ClusterArgs, opts: pulumi.ResourceOptions | None = None):
super().__init__("pulumi:solutions:EksCluster", name, None, opts)
parent = pulumi.ResourceOptions(parent=self)
cluster_name_input = pulumi.Output.from_input(args.cluster_name)
self.cluster = eks.Cluster(
name,
name=args.cluster_name,
version=args.version,
vpc_id=args.vpc_id,
public_subnet_ids=args.public_subnet_ids,
private_subnet_ids=args.private_subnet_ids,
node_associate_public_ip_address=False,
endpoint_private_access=True,
endpoint_public_access=True,
create_oidc_provider=True,
skip_default_node_group=True,
authentication_mode=eks.AuthenticationMode.API_AND_CONFIG_MAP,
access_entries={
"deployer": eks.AccessEntryArgs(
principal_arn=args.deployer_role_arn,
access_policies={
"admin": eks.AccessPolicyAssociationArgs(
policy_arn="arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy",
access_scope=aws.eks.AccessPolicyAssociationAccessScopeArgs(type="cluster"),
),
},
),
},
encryption_config_key_arn=args.encryption_key_arn,
tags=dict(args.tags),
opts=parent,
)
oidc_provider_arn = self.cluster.oidc_provider_arn
oidc_provider_url_no_scheme = self.cluster.oidc_provider_url.apply(
lambda u: u.replace("https://", "") if u.startswith("https://") else u
)
ec2_assume_role_policy = json.dumps({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "ec2.amazonaws.com"},
"Action": "sts:AssumeRole",
}],
})
system_role = aws.iam.Role(
f"{name}-system-node",
assume_role_policy=ec2_assume_role_policy,
tags=dict(args.tags),
opts=parent,
)
for policy in [
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
]:
aws.iam.RolePolicyAttachment(
f"{name}-system-node-{policy.split('/')[-1]}",
role=system_role.name,
policy_arn=policy,
opts=parent,
)
eks.ManagedNodeGroup(
f"{name}-system",
cluster=self.cluster,
node_role=system_role,
instance_types=[args.system_instance_type],
subnet_ids=args.private_subnet_ids,
scaling_config=aws.eks.NodeGroupScalingConfigArgs(
desired_size=args.system_desired_size,
min_size=args.system_desired_size,
max_size=args.system_desired_size,
),
labels={"karpenter.sh/controller": "true"},
tags=dict(args.tags),
opts=parent,
)
def _tag_private_subnets(subnets: Sequence[str]) -> None:
for idx, subnet_id in enumerate(subnets):
aws.ec2.Tag(
f"{name}-karpenter-subnet-{idx}",
resource_id=subnet_id,
key="karpenter.sh/discovery",
value=cluster_name_input,
opts=parent,
)
pulumi.Output.from_input(args.private_subnet_ids).apply(_tag_private_subnets)
aws.ec2.Tag(
f"{name}-karpenter-sg",
resource_id=self.cluster.cluster_security_group_id,
key="karpenter.sh/discovery",
value=cluster_name_input,
opts=parent,
)
self.provider = k8s.Provider(
f"{name}-k8s",
kubeconfig=self.cluster.kubeconfig_json,
opts=parent,
)
def _service_account_assume_policy(namespace: str, service_account: str):
def build(pair):
arn, url = pair
return json.dumps({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated": arn},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
f"{url}:aud": "sts.amazonaws.com",
f"{url}:sub": f"system:serviceaccount:{namespace}:{service_account}",
},
},
}],
})
return pulumi.Output.all(oidc_provider_arn, oidc_provider_url_no_scheme).apply(build)
if args.enable_external_secrets:
eso_role = aws.iam.Role(
f"{name}-eso",
assume_role_policy=_service_account_assume_policy("external-secrets", "external-secrets"),
tags=dict(args.tags),
opts=parent,
)
aws.iam.RolePolicy(
f"{name}-eso-secrets",
role=eso_role.id,
policy=json.dumps({
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecrets",
],
"Resource": "*",
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:GetParametersByPath",
],
"Resource": "*",
},
],
}),
opts=parent,
)
k8s.helm.v3.Release(
f"{name}-eso",
chart="external-secrets",
version=args.external_secrets_chart_version,
namespace="external-secrets",
create_namespace=True,
repository_opts=k8s.helm.v3.RepositoryOptsArgs(repo="https://charts.external-secrets.io"),
values={
"serviceAccount": {
"annotations": {"eks.amazonaws.com/role-arn": eso_role.arn},
},
"installCRDs": True,
},
opts=pulumi.ResourceOptions(parent=self, provider=self.provider),
)
if args.enable_load_balancer_controller:
alb_role = aws.iam.Role(
f"{name}-alb",
assume_role_policy=_service_account_assume_policy("kube-system", "aws-load-balancer-controller"),
tags=dict(args.tags),
opts=parent,
)
aws.iam.RolePolicy(
f"{name}-alb-policy",
role=alb_role.id,
policy=ALB_CONTROLLER_IAM_POLICY,
opts=parent,
)
k8s.helm.v3.Release(
f"{name}-alb",
chart="aws-load-balancer-controller",
version=args.alb_controller_chart_version,
namespace="kube-system",
repository_opts=k8s.helm.v3.RepositoryOptsArgs(repo="https://aws.github.io/eks-charts"),
values={
"clusterName": args.cluster_name,
"region": aws.get_region_output().name,
"vpcId": args.vpc_id,
"serviceAccount": {
"create": True,
"name": "aws-load-balancer-controller",
"annotations": {"eks.amazonaws.com/role-arn": alb_role.arn},
},
},
opts=pulumi.ResourceOptions(parent=self, provider=self.provider),
)
karpenter_node_role = aws.iam.Role(
f"{name}-karpenter-node",
assume_role_policy=ec2_assume_role_policy,
tags=dict(args.tags),
opts=parent,
)
for policy in [
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
]:
aws.iam.RolePolicyAttachment(
f"{name}-karpenter-node-{policy.split('/')[-1]}",
role=karpenter_node_role.name,
policy_arn=policy,
opts=parent,
)
aws.iam.InstanceProfile(
f"{name}-karpenter-node",
role=karpenter_node_role.name,
tags=dict(args.tags),
opts=parent,
)
karpenter_access_entry = aws.eks.AccessEntry(
f"{name}-karpenter-node",
cluster_name=self.cluster.eks_cluster.name,
principal_arn=karpenter_node_role.arn,
type="EC2_LINUX",
opts=parent,
)
interruption_queue = aws.sqs.Queue(
f"{name}-karpenter",
message_retention_seconds=300,
sqs_managed_sse_enabled=True,
tags=dict(args.tags),
opts=parent,
)
aws.sqs.QueuePolicy(
f"{name}-karpenter",
queue_url=interruption_queue.url,
policy=interruption_queue.arn.apply(lambda arn: json.dumps({
"Version": "2012-10-17",
"Id": "KarpenterInterruptionQueue",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": ["events.amazonaws.com", "sqs.amazonaws.com"]},
"Action": "sqs:SendMessage",
"Resource": arn,
}],
})),
opts=parent,
)
event_rules = [
("health", {"source": ["aws.health"], "detail-type": ["AWS Health Event"]}),
("spot", {"source": ["aws.ec2"], "detail-type": ["EC2 Spot Instance Interruption Warning"]}),
("rebalance", {"source": ["aws.ec2"], "detail-type": ["EC2 Instance Rebalance Recommendation"]}),
("state-change", {"source": ["aws.ec2"], "detail-type": ["EC2 Instance State-change Notification"]}),
]
for key, pattern in event_rules:
rule = aws.cloudwatch.EventRule(
f"{name}-karpenter-{key}",
event_pattern=json.dumps(pattern),
opts=parent,
)
aws.cloudwatch.EventTarget(
f"{name}-karpenter-{key}",
rule=rule.name,
arn=interruption_queue.arn,
opts=parent,
)
if args.enable_karpenter:
controller_role = aws.iam.Role(
f"{name}-karpenter",
assume_role_policy=_service_account_assume_policy("karpenter", "karpenter"),
tags=dict(args.tags),
opts=parent,
)
caller = aws.get_caller_identity_output()
region_name = aws.get_region_output().name
aws.iam.RolePolicy(
f"{name}-karpenter-policy",
role=controller_role.id,
policy=pulumi.Output.all(
cluster_name_input,
karpenter_node_role.arn,
interruption_queue.arn,
region_name,
caller.account_id,
).apply(lambda vals: karpenter_controller_iam_policy(KarpenterPolicyArgs(
cluster_name=vals[0],
node_role_arn=vals[1],
queue_arn=vals[2],
region=vals[3],
account_id=vals[4],
))),
opts=parent,
)
karpenter_release = k8s.helm.v3.Release(
f"{name}-karpenter",
chart="oci://public.ecr.aws/karpenter/karpenter",
version=args.karpenter_chart_version,
namespace="karpenter",
create_namespace=True,
values={
"settings": {
"clusterName": args.cluster_name,
"interruptionQueue": interruption_queue.name,
},
"serviceAccount": {
"name": "karpenter",
"annotations": {"eks.amazonaws.com/role-arn": controller_role.arn},
},
"controller": {
"resources": {
"requests": {"cpu": "1", "memory": "1Gi"},
"limits": {"cpu": "1", "memory": "1Gi"},
},
},
},
opts=pulumi.ResourceOptions(parent=self, provider=self.provider),
)
k8s.apiextensions.CustomResource(
f"{name}-karpenter-nodeclass",
api_version="karpenter.k8s.aws/v1",
kind="EC2NodeClass",
metadata={"name": "default"},
spec={
"amiFamily": "AL2023",
"role": karpenter_node_role.name,
"subnetSelectorTerms": [{"tags": {"karpenter.sh/discovery": cluster_name_input}}],
"securityGroupSelectorTerms": [{"tags": {"karpenter.sh/discovery": cluster_name_input}}],
"amiSelectorTerms": [{"alias": "al2023@latest"}],
"tags": {**dict(args.tags), "karpenter.sh/nodepool": "default"},
},
opts=pulumi.ResourceOptions(
parent=self,
provider=self.provider,
depends_on=[karpenter_release, karpenter_access_entry],
),
)
k8s.apiextensions.CustomResource(
f"{name}-karpenter-nodepool",
api_version="karpenter.sh/v1",
kind="NodePool",
metadata={"name": "default"},
spec={
"template": {
"spec": {
"nodeClassRef": {
"group": "karpenter.k8s.aws",
"kind": "EC2NodeClass",
"name": "default",
},
"requirements": [
{"key": "kubernetes.io/arch", "operator": "In", "values": ["amd64"]},
{"key": "karpenter.sh/capacity-type", "operator": "In", "values": ["on-demand", "spot"]},
{"key": "karpenter.k8s.aws/instance-category", "operator": "In", "values": ["c", "m", "r"]},
{"key": "karpenter.k8s.aws/instance-generation", "operator": "Gt", "values": ["2"]},
],
"expireAfter": "720h",
},
},
"limits": {"cpu": "1000", "memory": "1000Gi"},
"disruption": {
"consolidationPolicy": "WhenEmptyOrUnderutilized",
"consolidateAfter": "30s",
},
},
opts=pulumi.ResourceOptions(
parent=self,
provider=self.provider,
depends_on=[karpenter_release],
),
)
self.kubeconfig = self.cluster.kubeconfig_json
self.oidc_provider_arn = oidc_provider_arn
self.karpenter_node_role_arn = karpenter_node_role.arn
self.karpenter_interruption_queue_name = interruption_queue.name
self.register_outputs({
"cluster": self.cluster,
"kubeconfig": self.kubeconfig,
"oidc_provider_arn": self.oidc_provider_arn,
"karpenter_node_role_arn": self.karpenter_node_role_arn,
"karpenter_interruption_queue_name": self.karpenter_interruption_queue_name,
})
components/alb_controller_policy.py
Builds the IAM policy the AWS Load Balancer Controller attaches through IRSA. Copied verbatim from the upstream install/iam_policy.json at the ALB controller version this family pins.
"""Official AWS Load Balancer Controller IAM policy (v2.14.x)."""
import json
ALB_CONTROLLER_IAM_POLICY = json.dumps({
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["iam:CreateServiceLinkedRole"],
"Resource": "*",
"Condition": {"StringEquals": {"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"}},
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeAccountAttributes",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInternetGateways",
"ec2:DescribeVpcs",
"ec2:DescribeVpcPeeringConnections",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeInstances",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeTags",
"ec2:GetCoipPoolUsage",
"ec2:DescribeCoipPools",
"ec2:GetSecurityGroupsForVpc",
"ec2:DescribeIpamPools",
"ec2:DescribeRouteTables",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeListenerCertificates",
"elasticloadbalancing:DescribeSSLPolicies",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTrustStores",
"elasticloadbalancing:DescribeListenerAttributes",
"elasticloadbalancing:DescribeCapacityReservation",
],
"Resource": "*",
},
{
"Effect": "Allow",
"Action": [
"cognito-idp:DescribeUserPoolClient",
"acm:ListCertificates",
"acm:DescribeCertificate",
"iam:ListServerCertificates",
"iam:GetServerCertificate",
"waf-regional:GetWebACL",
"waf-regional:GetWebACLForResource",
"waf-regional:AssociateWebACL",
"waf-regional:DisassociateWebACL",
"wafv2:GetWebACL",
"wafv2:GetWebACLForResource",
"wafv2:AssociateWebACL",
"wafv2:DisassociateWebACL",
"shield:GetSubscriptionState",
"shield:DescribeProtection",
"shield:CreateProtection",
"shield:DeleteProtection",
],
"Resource": "*",
},
{
"Effect": "Allow",
"Action": ["ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress"],
"Resource": "*",
},
{"Effect": "Allow", "Action": ["ec2:CreateSecurityGroup"], "Resource": "*"},
{
"Effect": "Allow",
"Action": ["ec2:CreateTags"],
"Resource": "arn:aws:ec2:*:*:security-group/*",
"Condition": {
"StringEquals": {"ec2:CreateAction": "CreateSecurityGroup"},
"Null": {"aws:RequestTag/elbv2.k8s.aws/cluster": "false"},
},
},
{
"Effect": "Allow",
"Action": ["ec2:CreateTags", "ec2:DeleteTags"],
"Resource": "arn:aws:ec2:*:*:security-group/*",
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false",
}
},
},
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"ec2:DeleteSecurityGroup",
],
"Resource": "*",
"Condition": {"Null": {"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"}},
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateTargetGroup",
],
"Resource": "*",
"Condition": {"Null": {"aws:RequestTag/elbv2.k8s.aws/cluster": "false"}},
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:DeleteRule",
],
"Resource": "*",
},
{
"Effect": "Allow",
"Action": ["elasticloadbalancing:AddTags", "elasticloadbalancing:RemoveTags"],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*",
],
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false",
}
},
},
{
"Effect": "Allow",
"Action": ["elasticloadbalancing:AddTags", "elasticloadbalancing:RemoveTags"],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*",
],
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetSubnets",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:ModifyListenerAttributes",
"elasticloadbalancing:ModifyCapacityReservation",
"elasticloadbalancing:ModifyIpPools",
],
"Resource": "*",
"Condition": {"Null": {"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"}},
},
{
"Effect": "Allow",
"Action": ["elasticloadbalancing:AddTags"],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*",
],
"Condition": {
"StringEquals": {
"elasticloadbalancing:CreateAction": ["CreateTargetGroup", "CreateLoadBalancer"],
},
"Null": {"aws:RequestTag/elbv2.k8s.aws/cluster": "false"},
},
},
{
"Effect": "Allow",
"Action": ["elasticloadbalancing:RegisterTargets", "elasticloadbalancing:DeregisterTargets"],
"Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:SetWebAcl",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:RemoveListenerCertificates",
"elasticloadbalancing:ModifyRule",
"elasticloadbalancing:SetRulePriorities",
],
"Resource": "*",
},
],
})
components/karpenter_controller_policy.py
Builds the Karpenter controller IAM policy. Mirrors the scoped statements from the Karpenter v1.x CloudFormation template, narrowed with aws:ResourceTag conditions on the cluster name.
"""Scoped Karpenter controller IAM policy aligned with the v1.x CloudFormation template."""
import json
from dataclasses import dataclass
@dataclass
class KarpenterPolicyArgs:
cluster_name: str
node_role_arn: str
queue_arn: str
region: str
account_id: str
def karpenter_controller_iam_policy(args: KarpenterPolicyArgs) -> str:
cluster_tag = f"aws:ResourceTag/kubernetes.io/cluster/{args.cluster_name}"
request_cluster_tag = f"aws:RequestTag/kubernetes.io/cluster/{args.cluster_name}"
cluster_arn = f"arn:aws:eks:{args.region}:{args.account_id}:cluster/{args.cluster_name}"
return json.dumps({
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowScopedEC2InstanceAccessActions",
"Effect": "Allow",
"Action": ["ec2:RunInstances", "ec2:CreateFleet"],
"Resource": [
f"arn:aws:ec2:{args.region}::image/*",
f"arn:aws:ec2:{args.region}::snapshot/*",
f"arn:aws:ec2:{args.region}:*:security-group/*",
f"arn:aws:ec2:{args.region}:*:subnet/*",
f"arn:aws:ec2:{args.region}:*:capacity-reservation/*",
],
},
{
"Sid": "AllowScopedEC2LaunchTemplateAccessActions",
"Effect": "Allow",
"Action": ["ec2:RunInstances", "ec2:CreateFleet"],
"Resource": f"arn:aws:ec2:{args.region}:*:launch-template/*",
"Condition": {
"StringEquals": {cluster_tag: "owned"},
"StringLike": {"aws:ResourceTag/karpenter.sh/nodepool": "*"},
},
},
{
"Sid": "AllowScopedEC2InstanceActionsWithTags",
"Effect": "Allow",
"Action": ["ec2:RunInstances", "ec2:CreateFleet", "ec2:CreateLaunchTemplate"],
"Resource": [
f"arn:aws:ec2:{args.region}:*:fleet/*",
f"arn:aws:ec2:{args.region}:*:instance/*",
f"arn:aws:ec2:{args.region}:*:volume/*",
f"arn:aws:ec2:{args.region}:*:network-interface/*",
f"arn:aws:ec2:{args.region}:*:launch-template/*",
f"arn:aws:ec2:{args.region}:*:spot-instances-request/*",
],
"Condition": {
"StringEquals": {
request_cluster_tag: "owned",
"aws:RequestTag/eks:eks-cluster-name": args.cluster_name,
},
"StringLike": {"aws:RequestTag/karpenter.sh/nodepool": "*"},
},
},
{
"Sid": "AllowScopedResourceCreationTagging",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": [
f"arn:aws:ec2:{args.region}:*:fleet/*",
f"arn:aws:ec2:{args.region}:*:instance/*",
f"arn:aws:ec2:{args.region}:*:volume/*",
f"arn:aws:ec2:{args.region}:*:network-interface/*",
f"arn:aws:ec2:{args.region}:*:launch-template/*",
f"arn:aws:ec2:{args.region}:*:spot-instances-request/*",
],
"Condition": {
"StringEquals": {
request_cluster_tag: "owned",
"ec2:CreateAction": ["RunInstances", "CreateFleet", "CreateLaunchTemplate"],
},
"StringLike": {"aws:RequestTag/karpenter.sh/nodepool": "*"},
},
},
{
"Sid": "AllowScopedResourceTagging",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": f"arn:aws:ec2:{args.region}:*:instance/*",
"Condition": {
"StringEquals": {cluster_tag: "owned"},
"StringLike": {"aws:ResourceTag/karpenter.sh/nodepool": "*"},
"StringEqualsIfExists": {
"aws:RequestTag/eks:eks-cluster-name": args.cluster_name,
},
"ForAllValues:StringEquals": {
"aws:TagKeys": [
"eks:eks-cluster-name",
"karpenter.sh/nodeclaim",
"Name",
],
},
},
},
{
"Sid": "AllowScopedDeletion",
"Effect": "Allow",
"Action": ["ec2:TerminateInstances", "ec2:DeleteLaunchTemplate"],
"Resource": [
f"arn:aws:ec2:{args.region}:*:instance/*",
f"arn:aws:ec2:{args.region}:*:launch-template/*",
],
"Condition": {
"StringEquals": {cluster_tag: "owned"},
"StringLike": {"aws:ResourceTag/karpenter.sh/nodepool": "*"},
},
},
{
"Sid": "AllowRegionalReadActions",
"Effect": "Allow",
"Action": [
"ec2:DescribeCapacityReservations",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeInstanceTypes",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSpotPriceHistory",
"ec2:DescribeSubnets",
"ec2:DescribeAvailabilityZones",
],
"Resource": "*",
"Condition": {"StringEquals": {"aws:RequestedRegion": args.region}},
},
{
"Sid": "AllowSSMReadActions",
"Effect": "Allow",
"Action": "ssm:GetParameter",
"Resource": f"arn:aws:ssm:{args.region}::parameter/aws/service/*",
},
{"Sid": "AllowPricingReadActions", "Effect": "Allow", "Action": "pricing:GetProducts", "Resource": "*"},
{
"Sid": "AllowInterruptionQueueActions",
"Effect": "Allow",
"Action": ["sqs:DeleteMessage", "sqs:GetQueueUrl", "sqs:ReceiveMessage"],
"Resource": args.queue_arn,
},
{
"Sid": "AllowPassingInstanceRole",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": args.node_role_arn,
"Condition": {"StringEquals": {"iam:PassedToService": "ec2.amazonaws.com"}},
},
{
"Sid": "AllowScopedInstanceProfileCreationActions",
"Effect": "Allow",
"Action": ["iam:CreateInstanceProfile"],
"Resource": "*",
"Condition": {
"StringEquals": {
request_cluster_tag: "owned",
"aws:RequestTag/eks:eks-cluster-name": args.cluster_name,
"aws:RequestTag/topology.kubernetes.io/region": args.region,
},
"StringLike": {"aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"},
},
},
{
"Sid": "AllowScopedInstanceProfileTagActions",
"Effect": "Allow",
"Action": ["iam:TagInstanceProfile"],
"Resource": "*",
"Condition": {
"StringEquals": {
cluster_tag: "owned",
"aws:ResourceTag/topology.kubernetes.io/region": args.region,
request_cluster_tag: "owned",
"aws:RequestTag/eks:eks-cluster-name": args.cluster_name,
"aws:RequestTag/topology.kubernetes.io/region": args.region,
},
"StringLike": {
"aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
"aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*",
},
},
},
{
"Sid": "AllowScopedInstanceProfileActions",
"Effect": "Allow",
"Action": [
"iam:AddRoleToInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:DeleteInstanceProfile",
],
"Resource": "*",
"Condition": {
"StringEquals": {
cluster_tag: "owned",
"aws:ResourceTag/topology.kubernetes.io/region": args.region,
},
"StringLike": {"aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*"},
},
},
{
"Sid": "AllowInstanceProfileReadActions",
"Effect": "Allow",
"Action": "iam:GetInstanceProfile",
"Resource": "*",
},
{
"Sid": "AllowAPIServerEndpointDiscovery",
"Effect": "Allow",
"Action": "eks:DescribeCluster",
"Resource": cluster_arn,
},
],
})
cluster/cluster.go
Provisions the EKS cluster, a system node pool sized for the controllers, workload-identity wiring (IRSA (IAM Roles for Service Accounts)), and the Helm releases for External Secrets Operator and the ingress controller for this cloud.
package cluster
import (
"encoding/json"
"fmt"
"strings"
"github.com/pulumi/pulumi-aws/sdk/v7/go/aws"
"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/cloudwatch"
"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/ec2"
awseks "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/eks"
"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/iam"
"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/sqs"
"github.com/pulumi/pulumi-eks/sdk/v4/go/eks"
k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
helm "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
type Args struct {
ClusterName pulumi.StringInput
Version pulumi.StringInput
VpcId pulumi.StringInput
PublicSubnetIds pulumi.StringArrayInput
PrivateSubnetIds pulumi.StringArrayInput
EncryptionKeyArn pulumi.StringInput
DeployerRoleArn pulumi.StringInput
SystemInstanceType pulumi.StringInput
SystemDesiredSize pulumi.IntInput
EnableExternalSecrets bool
EnableLoadBalancerController bool
EnableKarpenter bool
ExternalSecretsChartVersion string
AlbControllerChartVersion string
KarpenterChartVersion string
Tags pulumi.StringMapInput
}
type Cluster struct {
pulumi.ResourceState
Cluster *eks.Cluster
Provider *k8s.Provider
Kubeconfig pulumi.StringOutput
OidcProviderArn pulumi.StringOutput
KarpenterNodeRoleArn pulumi.StringOutput
KarpenterInterruptionQueueName pulumi.StringOutput
ClusterEndpoint pulumi.StringOutput
ClusterSecurityGroupId pulumi.StringOutput
CertificateAuthority pulumi.StringOutput
}
const ec2AssumeRolePolicy = `{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}`
func New(ctx *pulumi.Context, name string, args *Args, opts ...pulumi.ResourceOption) (*Cluster, error) {
component := &Cluster{}
if err := ctx.RegisterComponentResource("pulumi:solutions:EksCluster", name, component, opts...); err != nil {
return nil, err
}
parent := pulumi.Parent(component)
clusterNameOutput := args.ClusterName.ToStringOutput()
authMode := eks.AuthenticationModeApiAndConfigMap
cluster, err := eks.NewCluster(ctx, name, &eks.ClusterArgs{
Name: args.ClusterName,
Version: args.Version,
VpcId: args.VpcId,
PublicSubnetIds: args.PublicSubnetIds,
PrivateSubnetIds: args.PrivateSubnetIds,
NodeAssociatePublicIpAddress: pulumi.BoolRef(false),
EndpointPrivateAccess: pulumi.BoolPtr(true),
EndpointPublicAccess: pulumi.BoolPtr(true),
CreateOidcProvider: pulumi.BoolPtr(true),
SkipDefaultNodeGroup: pulumi.BoolRef(true),
AuthenticationMode: &authMode,
AccessEntries: map[string]eks.AccessEntryArgs{
"deployer": {
PrincipalArn: args.DeployerRoleArn,
AccessPolicies: map[string]eks.AccessPolicyAssociationInput{
"admin": eks.AccessPolicyAssociationArgs{
PolicyArn: pulumi.String("arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"),
AccessScope: &awseks.AccessPolicyAssociationAccessScopeArgs{Type: pulumi.String("cluster")},
},
},
},
},
EncryptionConfigKeyArn: args.EncryptionKeyArn.ToStringOutput().ToStringPtrOutput(),
Tags: args.Tags,
}, parent)
if err != nil {
return nil, err
}
oidcArn := cluster.OidcProviderArn
oidcUrlNoScheme := cluster.OidcProviderUrl.ApplyT(func(u string) string {
return strings.TrimPrefix(u, "https://")
}).(pulumi.StringOutput)
systemRole, err := iam.NewRole(ctx, fmt.Sprintf("%s-system-node", name), &iam.RoleArgs{
AssumeRolePolicy: pulumi.String(ec2AssumeRolePolicy),
Tags: args.Tags,
}, parent)
if err != nil {
return nil, err
}
nodePolicies := []string{
"arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
"arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
"arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
"arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
}
for _, policy := range nodePolicies {
if _, err := iam.NewRolePolicyAttachment(ctx, fmt.Sprintf("%s-system-node-%s", name, shortName(policy)), &iam.RolePolicyAttachmentArgs{
Role: systemRole.Name,
PolicyArn: pulumi.String(policy),
}, parent); err != nil {
return nil, err
}
}
if _, err := eks.NewManagedNodeGroup(ctx, fmt.Sprintf("%s-system", name), &eks.ManagedNodeGroupArgs{
Cluster: cluster,
NodeRole: systemRole,
InstanceTypes: pulumi.StringArray{args.SystemInstanceType},
SubnetIds: args.PrivateSubnetIds,
ScalingConfig: &awseks.NodeGroupScalingConfigArgs{
DesiredSize: args.SystemDesiredSize,
MinSize: args.SystemDesiredSize,
MaxSize: args.SystemDesiredSize,
},
Labels: pulumi.StringMap{"karpenter.sh/controller": pulumi.String("true")},
Tags: args.Tags,
}, parent); err != nil {
return nil, err
}
// Tag the landing-zone private subnets for Karpenter discovery. Accept
// the input array as an Output so each element can be tagged individually.
args.PrivateSubnetIds.ToStringArrayOutput().ApplyT(func(subnets []string) []string {
for idx, subnetId := range subnets {
_, err := ec2.NewTag(ctx, fmt.Sprintf("%s-karpenter-subnet-%d", name, idx), &ec2.TagArgs{
ResourceId: pulumi.String(subnetId),
Key: pulumi.String("karpenter.sh/discovery"),
Value: clusterNameOutput,
}, parent)
if err != nil {
ctx.Log.Warn(fmt.Sprintf("failed to tag subnet %s: %v", subnetId, err), nil)
}
}
return subnets
})
if _, err := ec2.NewTag(ctx, fmt.Sprintf("%s-karpenter-sg", name), &ec2.TagArgs{
ResourceId: cluster.ClusterSecurityGroupId,
Key: pulumi.String("karpenter.sh/discovery"),
Value: clusterNameOutput,
}, parent); err != nil {
return nil, err
}
kubeconfig := cluster.KubeconfigJson
helmProvider, err := k8s.NewProvider(ctx, fmt.Sprintf("%s-k8s", name), &k8s.ProviderArgs{
Kubeconfig: kubeconfig,
}, parent)
if err != nil {
return nil, err
}
serviceAccountAssume := func(namespace, serviceAccount string) pulumi.StringOutput {
return pulumi.All(oidcArn, oidcUrlNoScheme).ApplyT(func(vs []interface{}) (string, error) {
arn := vs[0].(string)
url := vs[1].(string)
doc := map[string]any{
"Version": "2012-10-17",
"Statement": []map[string]any{{
"Effect": "Allow",
"Principal": map[string]string{"Federated": arn},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": map[string]any{
"StringEquals": map[string]string{
fmt.Sprintf("%s:aud", url): "sts.amazonaws.com",
fmt.Sprintf("%s:sub", url): fmt.Sprintf("system:serviceaccount:%s:%s", namespace, serviceAccount),
},
},
}},
}
data, err := json.Marshal(doc)
if err != nil {
return "", err
}
return string(data), nil
}).(pulumi.StringOutput)
}
if args.EnableExternalSecrets {
role, err := iam.NewRole(ctx, fmt.Sprintf("%s-eso", name), &iam.RoleArgs{
AssumeRolePolicy: serviceAccountAssume("external-secrets", "external-secrets"),
Tags: args.Tags,
}, parent)
if err != nil {
return nil, err
}
if _, err := iam.NewRolePolicy(ctx, fmt.Sprintf("%s-eso-secrets", name), &iam.RolePolicyArgs{
Role: role.ID(),
Policy: pulumi.String(`{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["secretsmanager:GetSecretValue","secretsmanager:DescribeSecret","secretsmanager:ListSecrets"],"Resource":"*"},{"Effect":"Allow","Action":["ssm:GetParameter","ssm:GetParameters","ssm:GetParametersByPath"],"Resource":"*"}]}`),
}, parent); err != nil {
return nil, err
}
if _, err := helm.NewRelease(ctx, fmt.Sprintf("%s-eso", name), &helm.ReleaseArgs{
Chart: pulumi.String("external-secrets"),
Version: pulumi.String(args.ExternalSecretsChartVersion),
Namespace: pulumi.String("external-secrets"),
CreateNamespace: pulumi.Bool(true),
RepositoryOpts: &helm.RepositoryOptsArgs{Repo: pulumi.String("https://charts.external-secrets.io")},
Values: pulumi.Map{
"serviceAccount": pulumi.Map{
"annotations": pulumi.Map{"eks.amazonaws.com/role-arn": role.Arn},
},
"installCRDs": pulumi.Bool(true),
},
}, pulumi.Parent(component), pulumi.Provider(helmProvider)); err != nil {
return nil, err
}
}
regionName := aws.GetRegionOutput(ctx, aws.GetRegionOutputArgs{}).Name()
if args.EnableLoadBalancerController {
role, err := iam.NewRole(ctx, fmt.Sprintf("%s-alb", name), &iam.RoleArgs{
AssumeRolePolicy: serviceAccountAssume("kube-system", "aws-load-balancer-controller"),
Tags: args.Tags,
}, parent)
if err != nil {
return nil, err
}
if _, err := iam.NewRolePolicy(ctx, fmt.Sprintf("%s-alb-policy", name), &iam.RolePolicyArgs{
Role: role.ID(),
Policy: pulumi.String(AlbControllerIAMPolicy),
}, parent); err != nil {
return nil, err
}
if _, err := helm.NewRelease(ctx, fmt.Sprintf("%s-alb", name), &helm.ReleaseArgs{
Chart: pulumi.String("aws-load-balancer-controller"),
Version: pulumi.String(args.AlbControllerChartVersion),
Namespace: pulumi.String("kube-system"),
RepositoryOpts: &helm.RepositoryOptsArgs{Repo: pulumi.String("https://aws.github.io/eks-charts")},
Values: pulumi.Map{
"clusterName": clusterNameOutput,
"region": regionName,
"vpcId": args.VpcId.ToStringOutput(),
"serviceAccount": pulumi.Map{
"create": pulumi.Bool(true),
"name": pulumi.String("aws-load-balancer-controller"),
"annotations": pulumi.Map{"eks.amazonaws.com/role-arn": role.Arn},
},
},
}, pulumi.Parent(component), pulumi.Provider(helmProvider)); err != nil {
return nil, err
}
}
karpenterNodeRole, err := iam.NewRole(ctx, fmt.Sprintf("%s-karpenter-node", name), &iam.RoleArgs{
AssumeRolePolicy: pulumi.String(ec2AssumeRolePolicy),
Tags: args.Tags,
}, parent)
if err != nil {
return nil, err
}
for _, policy := range nodePolicies {
if _, err := iam.NewRolePolicyAttachment(ctx, fmt.Sprintf("%s-karpenter-node-%s", name, shortName(policy)), &iam.RolePolicyAttachmentArgs{
Role: karpenterNodeRole.Name,
PolicyArn: pulumi.String(policy),
}, parent); err != nil {
return nil, err
}
}
if _, err := iam.NewInstanceProfile(ctx, fmt.Sprintf("%s-karpenter-node", name), &iam.InstanceProfileArgs{
Role: karpenterNodeRole.Name,
Tags: args.Tags,
}, parent); err != nil {
return nil, err
}
karpenterAccessEntry, err := awseks.NewAccessEntry(ctx, fmt.Sprintf("%s-karpenter-node", name), &awseks.AccessEntryArgs{
ClusterName: clusterNameOutput,
PrincipalArn: karpenterNodeRole.Arn,
Type: pulumi.String("EC2_LINUX"),
}, parent)
if err != nil {
return nil, err
}
interruptionQueue, err := sqs.NewQueue(ctx, fmt.Sprintf("%s-karpenter", name), &sqs.QueueArgs{
MessageRetentionSeconds: pulumi.Int(300),
SqsManagedSseEnabled: pulumi.Bool(true),
Tags: args.Tags,
}, parent)
if err != nil {
return nil, err
}
if _, err := sqs.NewQueuePolicy(ctx, fmt.Sprintf("%s-karpenter", name), &sqs.QueuePolicyArgs{
QueueUrl: interruptionQueue.Url,
Policy: interruptionQueue.Arn.ApplyT(func(arn string) (string, error) {
doc := map[string]any{
"Version": "2012-10-17",
"Id": "KarpenterInterruptionQueue",
"Statement": []map[string]any{{
"Effect": "Allow",
"Principal": map[string][]string{"Service": {"events.amazonaws.com", "sqs.amazonaws.com"}},
"Action": "sqs:SendMessage",
"Resource": arn,
}},
}
data, err := json.Marshal(doc)
if err != nil {
return "", err
}
return string(data), nil
}).(pulumi.StringOutput),
}, parent); err != nil {
return nil, err
}
eventRules := [][2]string{
{"health", `{"source":["aws.health"],"detail-type":["AWS Health Event"]}`},
{"spot", `{"source":["aws.ec2"],"detail-type":["EC2 Spot Instance Interruption Warning"]}`},
{"rebalance", `{"source":["aws.ec2"],"detail-type":["EC2 Instance Rebalance Recommendation"]}`},
{"state-change", `{"source":["aws.ec2"],"detail-type":["EC2 Instance State-change Notification"]}`},
}
for _, r := range eventRules {
rule, err := cloudwatch.NewEventRule(ctx, fmt.Sprintf("%s-karpenter-%s", name, r[0]), &cloudwatch.EventRuleArgs{
EventPattern: pulumi.String(r[1]),
}, parent)
if err != nil {
return nil, err
}
if _, err := cloudwatch.NewEventTarget(ctx, fmt.Sprintf("%s-karpenter-%s", name, r[0]), &cloudwatch.EventTargetArgs{
Rule: rule.Name,
Arn: interruptionQueue.Arn,
}, parent); err != nil {
return nil, err
}
}
if args.EnableKarpenter {
controllerRole, err := iam.NewRole(ctx, fmt.Sprintf("%s-karpenter", name), &iam.RoleArgs{
AssumeRolePolicy: serviceAccountAssume("karpenter", "karpenter"),
Tags: args.Tags,
}, parent)
if err != nil {
return nil, err
}
caller := aws.GetCallerIdentityOutput(ctx, aws.GetCallerIdentityOutputArgs{})
policyDoc := pulumi.All(
clusterNameOutput,
karpenterNodeRole.Arn,
interruptionQueue.Arn,
regionName,
caller.AccountId(),
).ApplyT(func(vs []interface{}) (string, error) {
return KarpenterControllerIAMPolicy(KarpenterPolicyArgs{
ClusterName: vs[0].(string),
NodeRoleArn: vs[1].(string),
QueueArn: vs[2].(string),
Region: vs[3].(string),
AccountID: vs[4].(string),
})
}).(pulumi.StringOutput)
if _, err := iam.NewRolePolicy(ctx, fmt.Sprintf("%s-karpenter-policy", name), &iam.RolePolicyArgs{
Role: controllerRole.ID(),
Policy: policyDoc,
}, parent); err != nil {
return nil, err
}
karpenterRelease, err := helm.NewRelease(ctx, fmt.Sprintf("%s-karpenter", name), &helm.ReleaseArgs{
Chart: pulumi.String("oci://public.ecr.aws/karpenter/karpenter"),
Version: pulumi.String(args.KarpenterChartVersion),
Namespace: pulumi.String("karpenter"),
CreateNamespace: pulumi.Bool(true),
Values: pulumi.Map{
"settings": pulumi.Map{
"clusterName": clusterNameOutput,
"interruptionQueue": interruptionQueue.Name,
},
"serviceAccount": pulumi.Map{
"name": pulumi.String("karpenter"),
"annotations": pulumi.Map{"eks.amazonaws.com/role-arn": controllerRole.Arn},
},
"controller": pulumi.Map{
"resources": pulumi.Map{
"requests": pulumi.Map{"cpu": pulumi.String("1"), "memory": pulumi.String("1Gi")},
"limits": pulumi.Map{"cpu": pulumi.String("1"), "memory": pulumi.String("1Gi")},
},
},
},
}, pulumi.Parent(component), pulumi.Provider(helmProvider))
if err != nil {
return nil, err
}
// Subnet discovery tag value for Karpenter selectors.
discoveryTags := clusterNameOutput.ApplyT(func(clusterName string) map[string]string {
return map[string]string{"karpenter.sh/discovery": clusterName}
}).(pulumi.StringMapOutput)
nodeClassTags := pulumi.All(clusterNameOutput, args.Tags.ToStringMapOutput()).ApplyT(func(vs []interface{}) map[string]string {
tags := map[string]string{"karpenter.sh/nodepool": "default"}
if clusterName, ok := vs[0].(string); ok {
_ = clusterName
}
if extra, ok := vs[1].(map[string]string); ok {
for k, v := range extra {
tags[k] = v
}
}
return tags
}).(pulumi.StringMapOutput)
if _, err := apiextensions.NewCustomResource(ctx, fmt.Sprintf("%s-karpenter-nodeclass", name), &apiextensions.CustomResourceArgs{
ApiVersion: pulumi.String("karpenter.k8s.aws/v1"),
Kind: pulumi.String("EC2NodeClass"),
Metadata: &metav1.ObjectMetaArgs{Name: pulumi.String("default")},
OtherFields: map[string]interface{}{
"spec": pulumi.Map{
"amiFamily": pulumi.String("AL2023"),
"role": karpenterNodeRole.Name,
"subnetSelectorTerms": pulumi.Array{
pulumi.Map{"tags": discoveryTags},
},
"securityGroupSelectorTerms": pulumi.Array{
pulumi.Map{"tags": discoveryTags},
},
"amiSelectorTerms": pulumi.Array{
pulumi.Map{"alias": pulumi.String("al2023@latest")},
},
"tags": nodeClassTags,
},
},
}, pulumi.Parent(component), pulumi.Provider(helmProvider), pulumi.DependsOn([]pulumi.Resource{karpenterRelease, karpenterAccessEntry})); err != nil {
return nil, err
}
if _, err := apiextensions.NewCustomResource(ctx, fmt.Sprintf("%s-karpenter-nodepool", name), &apiextensions.CustomResourceArgs{
ApiVersion: pulumi.String("karpenter.sh/v1"),
Kind: pulumi.String("NodePool"),
Metadata: &metav1.ObjectMetaArgs{Name: pulumi.String("default")},
OtherFields: map[string]interface{}{
"spec": pulumi.Map{
"template": pulumi.Map{
"spec": pulumi.Map{
"nodeClassRef": pulumi.Map{
"group": pulumi.String("karpenter.k8s.aws"),
"kind": pulumi.String("EC2NodeClass"),
"name": pulumi.String("default"),
},
"requirements": pulumi.Array{
pulumi.Map{"key": pulumi.String("kubernetes.io/arch"), "operator": pulumi.String("In"), "values": pulumi.StringArray{pulumi.String("amd64")}},
pulumi.Map{"key": pulumi.String("karpenter.sh/capacity-type"), "operator": pulumi.String("In"), "values": pulumi.StringArray{pulumi.String("on-demand"), pulumi.String("spot")}},
pulumi.Map{"key": pulumi.String("karpenter.k8s.aws/instance-category"), "operator": pulumi.String("In"), "values": pulumi.StringArray{pulumi.String("c"), pulumi.String("m"), pulumi.String("r")}},
pulumi.Map{"key": pulumi.String("karpenter.k8s.aws/instance-generation"), "operator": pulumi.String("Gt"), "values": pulumi.StringArray{pulumi.String("2")}},
},
"expireAfter": pulumi.String("720h"),
},
},
"limits": pulumi.Map{
"cpu": pulumi.String("1000"),
"memory": pulumi.String("1000Gi"),
},
"disruption": pulumi.Map{
"consolidationPolicy": pulumi.String("WhenEmptyOrUnderutilized"),
"consolidateAfter": pulumi.String("30s"),
},
},
},
}, pulumi.Parent(component), pulumi.Provider(helmProvider), pulumi.DependsOn([]pulumi.Resource{karpenterRelease})); err != nil {
return nil, err
}
}
component.Cluster = cluster
component.Provider = helmProvider
component.Kubeconfig = kubeconfig
component.OidcProviderArn = oidcArn
component.KarpenterNodeRoleArn = karpenterNodeRole.Arn
component.KarpenterInterruptionQueueName = interruptionQueue.Name
component.ClusterEndpoint = cluster.Core.Endpoint()
component.ClusterSecurityGroupId = cluster.ClusterSecurityGroupId
component.CertificateAuthority = kubeconfig.ApplyT(func(raw string) (string, error) {
var doc struct {
Clusters []struct {
Cluster struct {
CertificateAuthorityData string `json:"certificate-authority-data"`
} `json:"cluster"`
} `json:"clusters"`
}
if err := json.Unmarshal([]byte(raw), &doc); err != nil {
return "", err
}
if len(doc.Clusters) == 0 {
return "", nil
}
return doc.Clusters[0].Cluster.CertificateAuthorityData, nil
}).(pulumi.StringOutput)
if err := ctx.RegisterResourceOutputs(component, pulumi.Map{
"cluster": cluster,
"kubeconfig": kubeconfig,
"oidcProviderArn": oidcArn,
"karpenterNodeRoleArn": karpenterNodeRole.Arn,
"karpenterInterruptionQueueName": interruptionQueue.Name,
}); err != nil {
return nil, err
}
return component, nil
}
func shortName(arn string) string {
for i := len(arn) - 1; i >= 0; i-- {
if arn[i] == '/' {
return arn[i+1:]
}
}
return arn
}
cluster/alb_controller_policy.go
Builds the IAM policy the AWS Load Balancer Controller attaches through IRSA. Copied verbatim from the upstream install/iam_policy.json at the ALB controller version this family pins.
package cluster
// AlbControllerIAMPolicy is the verbatim upstream policy for the AWS Load
// Balancer Controller v2.14.x release. Sourced from the chart documentation at
// github.com/kubernetes-sigs/aws-load-balancer-controller/docs/install/iam_policy.json.
const AlbControllerIAMPolicy = `{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": ["iam:CreateServiceLinkedRole"], "Resource": "*", "Condition": {"StringEquals": {"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"}}},
{"Effect": "Allow", "Action": ["ec2:DescribeAccountAttributes", "ec2:DescribeAddresses", "ec2:DescribeAvailabilityZones", "ec2:DescribeInternetGateways", "ec2:DescribeVpcs", "ec2:DescribeVpcPeeringConnections", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "ec2:DescribeInstances", "ec2:DescribeNetworkInterfaces", "ec2:DescribeTags", "ec2:GetCoipPoolUsage", "ec2:DescribeCoipPools", "ec2:GetSecurityGroupsForVpc", "ec2:DescribeIpamPools", "ec2:DescribeRouteTables", "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DescribeLoadBalancerAttributes", "elasticloadbalancing:DescribeListeners", "elasticloadbalancing:DescribeListenerCertificates", "elasticloadbalancing:DescribeSSLPolicies", "elasticloadbalancing:DescribeRules", "elasticloadbalancing:DescribeTargetGroups", "elasticloadbalancing:DescribeTargetGroupAttributes", "elasticloadbalancing:DescribeTargetHealth", "elasticloadbalancing:DescribeTags", "elasticloadbalancing:DescribeTrustStores", "elasticloadbalancing:DescribeListenerAttributes", "elasticloadbalancing:DescribeCapacityReservation"], "Resource": "*"},
{"Effect": "Allow", "Action": ["cognito-idp:DescribeUserPoolClient", "acm:ListCertificates", "acm:DescribeCertificate", "iam:ListServerCertificates", "iam:GetServerCertificate", "waf-regional:GetWebACL", "waf-regional:GetWebACLForResource", "waf-regional:AssociateWebACL", "waf-regional:DisassociateWebACL", "wafv2:GetWebACL", "wafv2:GetWebACLForResource", "wafv2:AssociateWebACL", "wafv2:DisassociateWebACL", "shield:GetSubscriptionState", "shield:DescribeProtection", "shield:CreateProtection", "shield:DeleteProtection"], "Resource": "*"},
{"Effect": "Allow", "Action": ["ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress"], "Resource": "*"},
{"Effect": "Allow", "Action": ["ec2:CreateSecurityGroup"], "Resource": "*"},
{"Effect": "Allow", "Action": ["ec2:CreateTags"], "Resource": "arn:aws:ec2:*:*:security-group/*", "Condition": {"StringEquals": {"ec2:CreateAction": "CreateSecurityGroup"}, "Null": {"aws:RequestTag/elbv2.k8s.aws/cluster": "false"}}},
{"Effect": "Allow", "Action": ["ec2:CreateTags", "ec2:DeleteTags"], "Resource": "arn:aws:ec2:*:*:security-group/*", "Condition": {"Null": {"aws:RequestTag/elbv2.k8s.aws/cluster": "true", "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"}}},
{"Effect": "Allow", "Action": ["ec2:AuthorizeSecurityGroupIngress", "ec2:RevokeSecurityGroupIngress", "ec2:DeleteSecurityGroup"], "Resource": "*", "Condition": {"Null": {"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"}}},
{"Effect": "Allow", "Action": ["elasticloadbalancing:CreateLoadBalancer", "elasticloadbalancing:CreateTargetGroup"], "Resource": "*", "Condition": {"Null": {"aws:RequestTag/elbv2.k8s.aws/cluster": "false"}}},
{"Effect": "Allow", "Action": ["elasticloadbalancing:CreateListener", "elasticloadbalancing:DeleteListener", "elasticloadbalancing:CreateRule", "elasticloadbalancing:DeleteRule"], "Resource": "*"},
{"Effect": "Allow", "Action": ["elasticloadbalancing:AddTags", "elasticloadbalancing:RemoveTags"], "Resource": ["arn:aws:elasticloadbalancing:*:*:targetgroup/*/*", "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*", "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"], "Condition": {"Null": {"aws:RequestTag/elbv2.k8s.aws/cluster": "true", "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"}}},
{"Effect": "Allow", "Action": ["elasticloadbalancing:AddTags", "elasticloadbalancing:RemoveTags"], "Resource": ["arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*", "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*", "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*", "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"]},
{"Effect": "Allow", "Action": ["elasticloadbalancing:ModifyLoadBalancerAttributes", "elasticloadbalancing:SetIpAddressType", "elasticloadbalancing:SetSecurityGroups", "elasticloadbalancing:SetSubnets", "elasticloadbalancing:DeleteLoadBalancer", "elasticloadbalancing:ModifyTargetGroup", "elasticloadbalancing:ModifyTargetGroupAttributes", "elasticloadbalancing:DeleteTargetGroup", "elasticloadbalancing:ModifyListenerAttributes", "elasticloadbalancing:ModifyCapacityReservation", "elasticloadbalancing:ModifyIpPools"], "Resource": "*", "Condition": {"Null": {"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"}}},
{"Effect": "Allow", "Action": ["elasticloadbalancing:AddTags"], "Resource": ["arn:aws:elasticloadbalancing:*:*:targetgroup/*/*", "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*", "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"], "Condition": {"StringEquals": {"elasticloadbalancing:CreateAction": ["CreateTargetGroup", "CreateLoadBalancer"]}, "Null": {"aws:RequestTag/elbv2.k8s.aws/cluster": "false"}}},
{"Effect": "Allow", "Action": ["elasticloadbalancing:RegisterTargets", "elasticloadbalancing:DeregisterTargets"], "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"},
{"Effect": "Allow", "Action": ["elasticloadbalancing:SetWebAcl", "elasticloadbalancing:ModifyListener", "elasticloadbalancing:AddListenerCertificates", "elasticloadbalancing:RemoveListenerCertificates", "elasticloadbalancing:ModifyRule", "elasticloadbalancing:SetRulePriorities"], "Resource": "*"}
]
}`
cluster/karpenter_controller_policy.go
Builds the Karpenter controller IAM policy. Mirrors the scoped statements from the Karpenter v1.x CloudFormation template, narrowed with aws:ResourceTag conditions on the cluster name.
package cluster
import (
"encoding/json"
"fmt"
)
// KarpenterPolicyArgs gathers the values the Karpenter controller IAM policy
// renders into. All five values are scoped by the cluster name in CloudFormation
// upstream, so we do the same here.
type KarpenterPolicyArgs struct {
ClusterName string
NodeRoleArn string
QueueArn string
Region string
AccountID string
}
// KarpenterControllerIAMPolicy renders the scoped Karpenter controller policy
// matching the upstream v1.x CloudFormation template from
// github.com/aws/karpenter-provider-aws/website/content/en/docs/reference/cloudformation.md.
func KarpenterControllerIAMPolicy(args KarpenterPolicyArgs) (string, error) {
clusterTag := fmt.Sprintf("aws:ResourceTag/kubernetes.io/cluster/%s", args.ClusterName)
requestClusterTag := fmt.Sprintf("aws:RequestTag/kubernetes.io/cluster/%s", args.ClusterName)
clusterArn := fmt.Sprintf("arn:aws:eks:%s:%s:cluster/%s", args.Region, args.AccountID, args.ClusterName)
doc := map[string]any{
"Version": "2012-10-17",
"Statement": []map[string]any{
{
"Sid": "AllowScopedEC2InstanceAccessActions",
"Effect": "Allow",
"Action": []string{"ec2:RunInstances", "ec2:CreateFleet"},
"Resource": []string{
fmt.Sprintf("arn:aws:ec2:%s::image/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s::snapshot/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:security-group/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:subnet/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:capacity-reservation/*", args.Region),
},
},
{
"Sid": "AllowScopedEC2LaunchTemplateAccessActions",
"Effect": "Allow",
"Action": []string{"ec2:RunInstances", "ec2:CreateFleet"},
"Resource": fmt.Sprintf("arn:aws:ec2:%s:*:launch-template/*", args.Region),
"Condition": map[string]any{
"StringEquals": map[string]string{clusterTag: "owned"},
"StringLike": map[string]string{"aws:ResourceTag/karpenter.sh/nodepool": "*"},
},
},
{
"Sid": "AllowScopedEC2InstanceActionsWithTags",
"Effect": "Allow",
"Action": []string{"ec2:RunInstances", "ec2:CreateFleet", "ec2:CreateLaunchTemplate"},
"Resource": []string{
fmt.Sprintf("arn:aws:ec2:%s:*:fleet/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:instance/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:volume/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:network-interface/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:launch-template/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:spot-instances-request/*", args.Region),
},
"Condition": map[string]any{
"StringEquals": map[string]string{
requestClusterTag: "owned",
"aws:RequestTag/eks:eks-cluster-name": args.ClusterName,
},
"StringLike": map[string]string{"aws:RequestTag/karpenter.sh/nodepool": "*"},
},
},
{
"Sid": "AllowScopedResourceCreationTagging",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": []string{
fmt.Sprintf("arn:aws:ec2:%s:*:fleet/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:instance/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:volume/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:network-interface/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:launch-template/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:spot-instances-request/*", args.Region),
},
"Condition": map[string]any{
"StringEquals": map[string]any{
requestClusterTag: "owned",
"ec2:CreateAction": []string{"RunInstances", "CreateFleet", "CreateLaunchTemplate"},
},
"StringLike": map[string]string{"aws:RequestTag/karpenter.sh/nodepool": "*"},
},
},
{
"Sid": "AllowScopedResourceTagging",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": fmt.Sprintf("arn:aws:ec2:%s:*:instance/*", args.Region),
"Condition": map[string]any{
"StringEquals": map[string]string{clusterTag: "owned"},
"StringLike": map[string]string{"aws:ResourceTag/karpenter.sh/nodepool": "*"},
"StringEqualsIfExists": map[string]string{"aws:RequestTag/eks:eks-cluster-name": args.ClusterName},
"ForAllValues:StringEquals": map[string][]string{
"aws:TagKeys": {"eks:eks-cluster-name", "karpenter.sh/nodeclaim", "Name"},
},
},
},
{
"Sid": "AllowScopedDeletion",
"Effect": "Allow",
"Action": []string{"ec2:TerminateInstances", "ec2:DeleteLaunchTemplate"},
"Resource": []string{
fmt.Sprintf("arn:aws:ec2:%s:*:instance/*", args.Region),
fmt.Sprintf("arn:aws:ec2:%s:*:launch-template/*", args.Region),
},
"Condition": map[string]any{
"StringEquals": map[string]string{clusterTag: "owned"},
"StringLike": map[string]string{"aws:ResourceTag/karpenter.sh/nodepool": "*"},
},
},
{
"Sid": "AllowRegionalReadActions",
"Effect": "Allow",
"Action": []string{
"ec2:DescribeCapacityReservations",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeInstanceTypes",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSpotPriceHistory",
"ec2:DescribeSubnets",
"ec2:DescribeAvailabilityZones",
},
"Resource": "*",
"Condition": map[string]any{
"StringEquals": map[string]string{"aws:RequestedRegion": args.Region},
},
},
{
"Sid": "AllowSSMReadActions",
"Effect": "Allow",
"Action": "ssm:GetParameter",
"Resource": fmt.Sprintf("arn:aws:ssm:%s::parameter/aws/service/*", args.Region),
},
{"Sid": "AllowPricingReadActions", "Effect": "Allow", "Action": "pricing:GetProducts", "Resource": "*"},
{
"Sid": "AllowInterruptionQueueActions",
"Effect": "Allow",
"Action": []string{"sqs:DeleteMessage", "sqs:GetQueueUrl", "sqs:ReceiveMessage"},
"Resource": args.QueueArn,
},
{
"Sid": "AllowPassingInstanceRole",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": args.NodeRoleArn,
"Condition": map[string]any{"StringEquals": map[string]string{"iam:PassedToService": "ec2.amazonaws.com"}},
},
{
"Sid": "AllowScopedInstanceProfileCreationActions",
"Effect": "Allow",
"Action": []string{"iam:CreateInstanceProfile"},
"Resource": "*",
"Condition": map[string]any{
"StringEquals": map[string]string{
requestClusterTag: "owned",
"aws:RequestTag/eks:eks-cluster-name": args.ClusterName,
"aws:RequestTag/topology.kubernetes.io/region": args.Region,
},
"StringLike": map[string]string{"aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"},
},
},
{
"Sid": "AllowScopedInstanceProfileTagActions",
"Effect": "Allow",
"Action": []string{"iam:TagInstanceProfile"},
"Resource": "*",
"Condition": map[string]any{
"StringEquals": map[string]string{
clusterTag: "owned",
"aws:ResourceTag/topology.kubernetes.io/region": args.Region,
requestClusterTag: "owned",
"aws:RequestTag/eks:eks-cluster-name": args.ClusterName,
"aws:RequestTag/topology.kubernetes.io/region": args.Region,
},
"StringLike": map[string]string{
"aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
"aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*",
},
},
},
{
"Sid": "AllowScopedInstanceProfileActions",
"Effect": "Allow",
"Action": []string{"iam:AddRoleToInstanceProfile", "iam:RemoveRoleFromInstanceProfile", "iam:DeleteInstanceProfile"},
"Resource": "*",
"Condition": map[string]any{
"StringEquals": map[string]string{
clusterTag: "owned",
"aws:ResourceTag/topology.kubernetes.io/region": args.Region,
},
"StringLike": map[string]string{"aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*"},
},
},
{"Sid": "AllowInstanceProfileReadActions", "Effect": "Allow", "Action": "iam:GetInstanceProfile", "Resource": "*"},
{"Sid": "AllowAPIServerEndpointDiscovery", "Effect": "Allow", "Action": "eks:DescribeCluster", "Resource": clusterArn},
},
}
data, err := json.Marshal(doc)
if err != nil {
return "", err
}
return string(data), nil
}