Pulumi Blog: Kubernetes

Most Effective Infrastructure as Code (IaC) Tools

Asaf Ashirov — Sun, 05 Jul 2026 00:00:00 +0000

Infrastructure as Code (IaC) has evolved beyond simple automation into a fundamental shift toward applying software engineering practices to infrastructure management. In 2026, leading organizations aren’t just provisioning infrastructure—they’re treating it as software, complete with testing, version control, code reviews, and continuous integration.

As infrastructure complexity grows, teams increasingly seek approaches that provide the same developer productivity tools they use for application development. While template-based and domain-specific language approaches serve many use cases effectively, teams with complex requirements or programming backgrounds often find that general-purpose programming languages offer advantages in testing, abstraction, and collaboration.

This comprehensive guide examines the most effective infrastructure as code tools available today, providing detailed analysis of core IaC platforms, complementary tools, and related technologies through the lens of software engineering best practices. Whether you’re starting fresh with IaC or evaluating alternatives to overcome limitations in your current toolchain, we’ll help you navigate this complex landscape and choose solutions that truly bring software engineering to infrastructure.

Ready to try one of these approaches yourself? Get started with Pulumi for free and provision your first resource in minutes, or see Pulumi Neo in action to explore how an AI agent can manage infrastructure changes alongside your team.

What is Infrastructure as Code?

Infrastructure as Code (IaC) is an approach to automating the provisioning and management of infrastructure using software engineering principles, approaches, and tools. Rather than manually configuring servers, networks, and cloud resources through user interfaces or command-line tools, IaC enables you to define your entire infrastructure declaratively through code.

This approach brings the same benefits that have revolutionized software development—version control, automated testing, code reviews, and CI/CD pipelines—to infrastructure management.

What Are Infrastructure as Code Tools?

Infrastructure as Code tools are platforms and frameworks that enable you to define, provision, and manage infrastructure resources through code rather than manual processes. These tools translate your infrastructure definitions into API calls that create, modify, or destroy cloud resources across various providers.

The most effective IaC tools share several key characteristics:

Goal-state focus: Define your desired infrastructure outcome, whether through declarative syntax (like YAML/JSON templates) or imperative languages that express declarative intent
Multi-cloud support: Work across different cloud providers and services
State management: Track the current state of your infrastructure
Preview capabilities: Show what changes will be made before applying them
Idempotency: Safe to run multiple times with consistent results

Why Infrastructure as Code Tools Are Essential

The shift to IaC tools addresses fundamental challenges that manual infrastructure management cannot solve at scale:

Accelerates Deployment Velocity: Teams can provision complex multi-cloud architectures in minutes instead of weeks. This speed enables faster time-to-market and more frequent, reliable deployments.

Enables True Collaboration: Infrastructure becomes code that teams can review, test, and approve together. This collaborative approach reduces errors and ensures knowledge sharing across the organization.

Eliminates Configuration Drift: Manual changes lead to inconsistencies between environments. IaC ensures your production, staging, and development environments remain identical, eliminating the notorious “works on my machine” syndrome for infrastructure.

Provides Cost Control: Automated provisioning and deprovisioning prevents resource sprawl. Teams can easily track infrastructure costs, set budget alerts, and optimize resource usage across environments.

Ensures Compliance and Security: Codified security policies and compliance requirements are automatically enforced across all deployments. Audit trails become automatic, and policy violations are caught before deployment.

Guarantees Business Continuity: Complete infrastructure definitions stored in version control enable rapid disaster recovery. Organizations can reconstruct entire environments from code, minimizing downtime and data loss.

Infrastructure as Code Tools Overview

This guide covers the following infrastructure as code tools and platforms:

10 Most Used IaC Tools in 2026

Pulumi IaC - Modern IaC with general-purpose programming languages
Terraform - BSL-licensed IaC from HashiCorp that uses the HCL domain-specific language
AWS CDK - Cloud Development Kit for AWS
AWS CloudFormation - Native AWS integration
Azure Resource Manager (ARM) - Native Azure JSON templates
Azure Bicep - Azure-native domain-specific language that compiles to ARM
Google Cloud Infrastructure Manager - Terraform-based solution for Google Cloud
Kubernetes YAML - Native Kubernetes resource definitions
Crossplane - Kubernetes as universal control plane
OpenTofu - Open-source Terraform alternative

Configuration Management Tools

Chef - Configuration management and compliance automation
Puppet - Configuration management and compliance automation
Salt - Configuration management and remote execution

Application and Platform Management Tools

Kubernetes - Container orchestration platform

Security and Compliance Tools

Pulumi ESC - Configuration and secrets management platform
Pulumi Insights - Cloud resource search, analytics, and compliance platform
Snyk - Developer security platform with IaC scanning
Wiz - Comprehensive cloud security platform
HashiCorp Sentinel - Policy-as-code framework for Terraform
Checkov - Static analysis for IaC security
TFLint - Terraform linting and validation

Infrastructure Automation and Management Platforms

Pulumi Cloud - Managed service for Pulumi IaC with enterprise features
HashiCorp Cloud Platform - Enterprise SaaS platform for Terraform management and automation
Spacelift - Automation platform for IaC workflows (not an IaC tool itself)
Env0 - Governance and automation platform for existing IaC tools

Core Infrastructure as Code Tools

1. Pulumi

License: Apache 2.0
Best For: Teams who want flexible, language-agnostic IaC for infrastructure and operations

Pulumi IaC represents a modern approach to infrastructure as code, fundamentally changing how teams approach infrastructure by enabling the use of general-purpose programming languages like Python, TypeScript, Go, C#, and Java, plus YAML for simpler configurations. Unlike tools that force teams to learn proprietary domain-specific languages (DSLs), Pulumi leverages familiar languages and software engineering practices, providing unprecedented flexibility, powerful abstractions, and seamless integration with existing development workflows.

Pulumi’s approach combines the best of both imperative and declarative paradigms: you use imperative programming languages to define your desired infrastructure state, but the Pulumi engine processes this declaratively to determine what changes are needed to achieve your intended outcome.

Key Features:

Universal language support: Use Python, TypeScript, Go, C#, Java, or YAML configurations—no new DSL to learn
Any cloud, any architecture: Deploy to AWS, Azure, Google Cloud, Kubernetes, and 100+ other providers
Real programming constructs: Leverage loops, conditionals, functions, classes, packages, and third-party libraries
Superior developer experience: Full IDE support with IntelliSense, debugging, and refactoring
Built-in testing: Unit and integration testing for infrastructure code
Policy as Code: Enforce compliance and security policies with CrossGuard
Component ecosystem: Rich library of reusable infrastructure components

Universal Language Code Examples:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create a VPC with automatic subnets
const vpc = new awsx.ec2.Vpc("main-vpc", {
 cidrBlock: "10.0.0.0/16",
 numberOfAvailabilityZones: 2,
});

// Create an ECS cluster
const cluster = new aws.ecs.Cluster("app-cluster");

// Create an Application Load Balancer
const alb = new awsx.elasticloadbalancingv2.ApplicationLoadBalancer("app-alb", {
 vpcId: vpc.vpcId,
 subnetIds: vpc.publicSubnetIds,
});

// Deploy a containerized application
const service = new awsx.ecs.FargateService("app-service", {
 cluster: cluster.arn,
 taskDefinitionArgs: {
 container: {
 image: "nginx:latest",
 memory: 128,
 ports: [{
 containerPort: 80,
 targetGroup: alb.defaultTargetGroup,
 }],
 },
 },
});

export const vpcId = vpc.vpcId;
export const serviceUrl = alb.loadBalancer.dnsName;

import pulumi
import pulumi_aws as aws
import pulumi_awsx as awsx

# Create a VPC with automatic subnets
vpc = awsx.ec2.Vpc("main-vpc",
 cidr_block="10.0.0.0/16",
 number_of_availability_zones=2)

# Create an ECS cluster
cluster = aws.ecs.Cluster("app-cluster")

# Create an Application Load Balancer
alb = awsx.elasticloadbalancingv2.ApplicationLoadBalancer("app-alb",
 vpc_id=vpc.vpc_id,
 subnet_ids=vpc.public_subnet_ids)

# Deploy a containerized application
service = awsx.ecs.FargateService("app-service",
 cluster=cluster.arn,
 task_definition_args=awsx.ecs.FargateServiceTaskDefinitionArgs(
 container=awsx.ecs.TaskDefinitionContainerDefinitionArgs(
 image="nginx:latest",
 memory=128,
 ports=[awsx.ecs.TaskDefinitionPortMappingArgs(
 container_port=80,
 target_group=alb.default_target_group
 )]
 )
 ))

pulumi.export("vpc_id", vpc.vpc_id)
pulumi.export("service_url", alb.load_balancer.dns_name)

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ecs"
 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/elasticloadbalancingv2"
 awsxecs "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ecs"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create a VPC with automatic subnets
 vpc, err := ec2.NewVpc(ctx, "main-vpc", &ec2.VpcArgs{
 CidrBlock: pulumi.String("10.0.0.0/16"),
 NumberOfAvailabilityZones: pulumi.Int(2),
 })
 if err != nil {
 return err
 }

 // Create an ECS cluster
 cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
 if err != nil {
 return err
 }

 // Create an Application Load Balancer
 alb, err := elasticloadbalancingv2.NewApplicationLoadBalancer(ctx, "app-alb", &elasticloadbalancingv2.ApplicationLoadBalancerArgs{
 VpcId: vpc.VpcId,
 SubnetIds: vpc.PublicSubnetIds,
 })
 if err != nil {
 return err
 }

 // Deploy a containerized application
 _, err = awsxecs.NewFargateService(ctx, "app-service", &awsxecs.FargateServiceArgs{
 Cluster: cluster.Arn,
 TaskDefinitionArgs: &awsxecs.FargateServiceTaskDefinitionArgs{
 Container: &awsxecs.TaskDefinitionContainerDefinitionArgs{
 Image: pulumi.String("nginx:latest"),
 Memory: pulumi.Int(128),
 Ports: awsxecs.TaskDefinitionPortMappingArray{
 &awsxecs.TaskDefinitionPortMappingArgs{
 ContainerPort: pulumi.Int(80),
 TargetGroup: alb.DefaultTargetGroup,
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 ctx.Export("vpcId", vpc.VpcId)
 ctx.Export("serviceUrl", alb.LoadBalancer.DnsName())
 return nil
 })
}

using System.Collections.Generic;
using Pulumi;
using Aws = Pulumi.Aws;
using Awsx = Pulumi.Awsx;

return await Deployment.RunAsync(() =>
{
 // Create a VPC with automatic subnets
 var vpc = new Awsx.Ec2.Vpc("main-vpc", new()
 {
 CidrBlock = "10.0.0.0/16",
 NumberOfAvailabilityZones = 2,
 });

 // Create an ECS cluster
 var cluster = new Aws.Ecs.Cluster("app-cluster");

 // Create an Application Load Balancer
 var alb = new Awsx.ElasticLoadBalancingV2.ApplicationLoadBalancer("app-alb", new()
 {
 VpcId = vpc.VpcId,
 SubnetIds = vpc.PublicSubnetIds,
 });

 // Deploy a containerized application
 var service = new Awsx.Ecs.FargateService("app-service", new()
 {
 Cluster = cluster.Arn,
 TaskDefinitionArgs = new Awsx.Ecs.Inputs.FargateServiceTaskDefinitionArgs
 {
 Container = new Awsx.Ecs.Inputs.TaskDefinitionContainerDefinitionArgs
 {
 Image = "nginx:latest",
 Memory = 128,
 Ports = new[]
 {
 new Awsx.Ecs.Inputs.TaskDefinitionPortMappingArgs
 {
 ContainerPort = 80,
 TargetGroup = alb.DefaultTargetGroup,
 },
 },
 },
 },
 });

 return new Dictionary<string, object?>
 {
 ["vpcId"] = vpc.VpcId,
 ["serviceUrl"] = alb.LoadBalancer.Apply(lb => lb.DnsName),
 };
});

name: aws-ecs-example
runtime: yaml
description: An example that deploys an ECS Fargate service with load balancer

resources:
 # Create a VPC with automatic subnets
 main-vpc:
 type: awsx:ec2:Vpc
 properties:
 cidrBlock: "10.0.0.0/16"
 numberOfAvailabilityZones: 2

 # Create an ECS cluster
 app-cluster:
 type: aws:ecs:Cluster

 # Create an Application Load Balancer
 app-alb:
 type: awsx:elasticloadbalancingv2:ApplicationLoadBalancer
 properties:
 vpcId: ${main-vpc.vpcId}
 subnetIds: ${main-vpc.publicSubnetIds}

 # Deploy a containerized application
 app-service:
 type: awsx:ecs:FargateService
 properties:
 cluster: ${app-cluster.arn}
 taskDefinitionArgs:
 container:
 image: "nginx:latest"
 memory: 128
 ports:
 - containerPort: 80
 targetGroup: ${app-alb.defaultTargetGroup}

outputs:
 vpcId: ${main-vpc.vpcId}
 serviceUrl: ${app-alb.loadBalancer.dnsName}

Key Features:

General-purpose language support: Use Python, TypeScript, Go, C#, Java, or YAML without learning new DSLs
Software engineering practices: Full IDE support, comprehensive testing frameworks, debugging capabilities
Multi-cloud flexibility: Native cloud provider SDKs with same-day feature access across 150+ providers
Incremental adoption: Migration tools and state integration for gradual transitions
Open source licensing: Apache 2.0 ensures long-term freedom and flexibility

Considerations:

Learning curve: Teams new to programming may prefer template-based approaches initially
Ecosystem maturity: Smaller community compared to more established tools like Terraform
Tool complexity: Advanced features may require more setup than simpler template systems

Organizations like Unity, Snowflake, and Starburst have reported significant productivity improvements (80-90% deployment time reductions) when adopting programming language-based approaches. These improvements typically occur when transitioning from manual processes or basic template systems to automated approaches with comprehensive testing, IDE integration, and code reusability. Results vary based on starting point, team expertise, infrastructure complexity, and specific use cases.

Ready to get started? Experience Pulumi’s programming language approach and see how familiar languages can transform your infrastructure management with comprehensive testing, powerful abstractions, and seamless multi-cloud support.

2. Terraform

License: Business Source License (BSL) 1.1 (Not Open Source)
Best For: Teams with existing Terraform expertise and established workflows

Terraform uses HashiCorp Configuration Language (HCL) to define infrastructure across multiple cloud providers. However, its 2023 licensing change to BSL (no longer open source) and inherent limitations with domain-specific languages create challenges for teams requiring advanced software engineering practices.

Key Features:

Extensive provider ecosystem: Largest collection of community-maintained providers covering virtually every cloud service
Established workflows: Mature plan-and-apply process with extensive tooling and CI/CD integrations
Community and resources: Vast ecosystem of modules, extensive documentation, training materials, and community support
Enterprise adoption: Widely adopted in enterprises with established processes and expertise
Module ecosystem: Rich collection of reusable Terraform modules for common patterns

When Terraform Works Well:

Teams with existing HCL expertise and Terraform investments
Infrastructure patterns that fit well within HCL’s declarative model
Teams comfortable with template-based approaches

Considerations for Complex Scenarios:

Language constraints: HCL’s domain-specific nature can require workarounds for complex logic compared to general-purpose programming languages
Testing approach: Primarily supports integration testing; teams needing comprehensive unit testing may require additional tooling
IDE experience: While improving, HCL tooling provides less comprehensive support than mature programming language ecosystems
State management: Requires manual backend configuration and locking setup for team collaboration

Code Example:

data "aws_availability_zones" "available" {}

resource "aws_vpc" "main" {
 cidr_block = "10.0.0.0/16"

 tags = {
 Name = "main-vpc"
 }
}

resource "aws_subnet" "public" {
 count = 2
 vpc_id = aws_vpc.main.id
 cidr_block = "10.0.${count.index + 1}.0/24"
 availability_zone = data.aws_availability_zones.available.names[count.index]

 tags = {
 Name = "public-subnet-${count.index + 1}"
 }
}

3. AWS Cloud Development Kit (CDK)

License: Apache 2.0
Best For: AWS-focused teams who prefer programming languages over templates

AWS CDK allows you to define AWS infrastructure using familiar programming languages, synthesizing CloudFormation templates for deployment while providing higher-level abstractions. CDK addresses many limitations of traditional template-based approaches by enabling general-purpose programming languages.

Key Features:

General-purpose programming languages: TypeScript, Python, Java, C#, JavaScript support with full IDE integration (Go available in Developer Preview)
AWS-optimized constructs: High-level components encapsulating AWS best practices
Type safety: Compile-time checking and IntelliSense support
CloudFormation reliability: Built on AWS’s proven deployment engine

Notable Limitations:

AWS-only ecosystem: Locked into single cloud provider, limiting multi-cloud strategies
CloudFormation constraints: Inherits template size limits and deployment restrictions
Vendor lock-in: Deep AWS integration makes migration to other clouds challenging
Limited cross-cloud consistency: Teams need different tools and approaches for multi-cloud deployments

CDK represents a significant improvement over CloudFormation templates but constrains organizations to AWS-only infrastructure strategies.

Code Example:

import * as cdk from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import { Construct } from 'constructs';

export class MyStack extends cdk.Stack {
 constructor(scope: Construct, id: string, props?: cdk.StackProps) {
 super(scope, id, props);

 const vpc = new ec2.Vpc(this, 'MyVpc', {
 maxAzs: 2
 });

 const cluster = new ecs.Cluster(this, 'MyCluster', {
 vpc: vpc
 });

 const taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDef', {
 memoryLimitMiB: 512,
 cpu: 256
 });

 taskDefinition.addContainer('web', {
 image: ecs.ContainerImage.fromRegistry('nginx:latest'),
 portMappings: [{ containerPort: 80 }]
 });

 new ecs.FargateService(this, 'MyService', {
 cluster: cluster,
 taskDefinition: taskDefinition
 });
 }
}

4. AWS CloudFormation

License: Proprietary (AWS Service)
Best For: AWS-only deployments requiring deep service integration

AWS CloudFormation provides the foundation for infrastructure as code on AWS, offering native integration with all AWS services and deep platform-specific features.

Pulumi Integration: Pulumi provides native AWS providers that offer the same comprehensive AWS service coverage as CloudFormation, with the added benefit of using general-purpose programming languages. You can also import existing CloudFormation stacks into Pulumi for gradual migration or hybrid management approaches.

Key Features:

AWS-native: First-party support for all AWS services
JSON/YAML templates: Declarative resource definitions
Stack management: Organized resource grouping and lifecycle management
Change sets: Preview infrastructure changes before deployment
Service integration: Deep integration with other AWS services

Code Example:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
 MyVPC:
 Type: AWS::EC2::VPC
 Properties:
 CidrBlock: 10.0.0.0/16
 EnableDnsHostnames: true
 Tags:
 - Key: Name
 Value: MyVPC

 MySubnet:
 Type: AWS::EC2::Subnet
 Properties:
 VpcId: !Ref MyVPC
 CidrBlock: 10.0.1.0/24
 AvailabilityZone: !Select [0, !GetAZs '']

5. Azure Resource Manager (ARM)

License: Proprietary (Microsoft Service)
Best For: Azure-native deployments requiring comprehensive platform integration

Azure Resource Manager provides the foundational infrastructure as code solution for Microsoft Azure, offering complete support for Azure services through JSON-based ARM templates. As Azure’s native IaC solution, ARM templates provide the most comprehensive coverage of Azure services and features.

Pulumi Integration: Pulumi’s native Azure providers offer equivalent comprehensive Azure service coverage with general-purpose programming languages. ARM templates can be imported into Pulumi, and you can reference ARM deployments from Pulumi programs for hybrid scenarios.

Key Features:

Azure-native: Complete Azure service coverage with first-party support
JSON templates: Declarative resource definitions in JSON format
Resource groups: Logical organization of related resources
Deployment modes: Complete or incremental deployment options
Comprehensive integration: Deep integration with Azure services and features

Code Example:

{
 "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
 "contentVersion": "1.0.0.0",
 "resources": [
 {
 "type": "Microsoft.Storage/storageAccounts",
 "apiVersion": "2023-01-01",
 "name": "mystorageaccount",
 "location": "[resourceGroup().location]",
 "sku": {
 "name": "Standard_LRS"
 },
 "kind": "StorageV2"
 }
 ]
}

6. Azure Bicep

License: MIT
Best For: Azure deployments requiring improved readability and developer experience

Azure Bicep is a domain-specific language (DSL) that simplifies Azure Resource Manager template authoring. Bicep files compile transparently to ARM templates, providing all the capabilities of ARM with significantly improved syntax and developer experience.

Key Features:

Simplified syntax: Clean, readable syntax compared to JSON ARM templates
ARM compilation: Compiles to ARM templates for deployment, ensuring full compatibility
Type safety: Strong typing and IntelliSense support in development environments
Modular design: Support for modules and code reuse across projects
Azure-native: Complete Azure service coverage through ARM template compilation

Code Example:

@description('Storage Account type')
@allowed([
 'Standard_LRS'
 'Standard_GRS'
 'Standard_ZRS'
 'Premium_LRS'
])
param storageAccountType string = 'Standard_LRS'

@description('Location for the storage account.')
param location string = resourceGroup().location

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
 name: 'mystorageaccount'
 location: location
 sku: {
 name: storageAccountType
 }
 kind: 'StorageV2'
 properties: {}
}

7. Google Cloud Infrastructure Manager

License: Proprietary (Google Service)
Best For: Google Cloud Platform deployments using Terraform

Google Cloud Infrastructure Manager automates the deployment and management of Google Cloud infrastructure resources using Terraform configurations, representing Google’s modern approach to infrastructure as code. Infrastructure Manager replaces Google Cloud Deployment Manager, which reached end of support on December 31, 2025.

Key Features:

Terraform-based: Uses standard Terraform configurations declaratively
Automated workflows: Handles Terraform init, validate, and apply operations
Version control integration: Supports Git repositories and Cloud Storage
Deployment tracking: Comprehensive metadata storage and logging
Multiple Terraform versions: Flexibility in Terraform version selection
Cloud Build integration: Leverages Google Cloud Build for execution environment
Migration path: Provides upgrade path from legacy Cloud Deployment Manager

Code Example:

# main.tf - Terraform configuration for Infrastructure Manager
resource "google_compute_instance" "vm_instance" {
 name = "my-vm"
 machine_type = "e2-medium"
 zone = "us-central1-a"

 boot_disk {
 initialize_params {
 image = "debian-cloud/debian-11"
 }
 }

 network_interface {
 network = "default"
 access_config {
 // Ephemeral public IP
 }
 }

 metadata = {
 startup-script = "echo Hello from Infrastructure Manager!"
 }
}

output "instance_ip" {
 value = google_compute_instance.vm_instance.network_interface[0].access_config[0].nat_ip
}

8. Kubernetes YAML

License: Apache 2.0
Best For: Teams managing container-native applications and cloud-native infrastructure

Kubernetes YAML manifests represent one of the most widely adopted forms of infrastructure as code, enabling teams to define, version, and manage containerized applications and their supporting infrastructure through declarative configuration files.

Key Features:

Declarative configuration: Define desired state through human-readable YAML files
Native Kubernetes integration: Direct integration with Kubernetes API without additional tools
GitOps compatibility: Version control YAML files for automated deployment workflows
Resource relationships: Define dependencies and relationships between Kubernetes resources
Extensive ecosystem: Rich ecosystem of operators and custom resources extending functionality

Code Example:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: web-app
 labels:
 app: web-app
spec:
 replicas: 3
 selector:
 matchLabels:
 app: web-app
 template:
 metadata:
 labels:
 app: web-app
 spec:
 containers:
 - name: web-app
 image: nginx:1.21
 ports:
 - containerPort: 80
 resources:
 requests:
 memory: "64Mi"
 cpu: "250m"
 limits:
 memory: "128Mi"
 cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
 name: web-app-service
spec:
 selector:
 app: web-app
 ports:
 - port: 80
 targetPort: 80
 type: LoadBalancer

9. Crossplane

License: Apache 2.0
Best For: Kubernetes-first organizations managing multi-cloud infrastructure

Crossplane is a Cloud-Native Framework for Platform Engineering that extends Kubernetes to help organizations build custom infrastructure management platforms, allowing teams to provision and manage cloud resources using Kubernetes APIs and patterns.

Pulumi Integration: Pulumi offers the Pulumi Kubernetes Operator (PKO) that provides similar Kubernetes-native infrastructure management capabilities, plus support for YAML-based definitions. Teams can also use Pulumi programs to provision the underlying infrastructure that Crossplane manages, creating layered infrastructure management approaches.

Key Features:

Kubernetes-native: Uses CRDs and standard Kubernetes patterns
Composite resources: Create higher-level infrastructure abstractions
GitOps compatibility: Seamless integration with GitOps workflows
Multi-cloud support: Provision resources across cloud providers
Policy integration: Leverage Kubernetes RBAC and admission controllers

Code Example:

apiVersion: ec2.aws.crossplane.io/v1alpha1
kind: VPC
metadata:
 name: sample-vpc
spec:
 cidrBlock: 10.0.0.0/16
 region: us-east-1
 tags:
 Name: sample-vpc
 providerConfigRef:
 name: aws-provider-config

10. OpenTofu

License: Mozilla Public License 2.0
Best For: Teams seeking an open-source Terraform alternative with community governance

OpenTofu emerged as a fork of Terraform v1.5.x following HashiCorp’s license change, maintained by the Linux Foundation. It provides full compatibility with Terraform while ensuring long-term open-source availability under MPL 2.0 licensing.

Key Features:

True open source: MPL 2.0 license with community governance via Linux Foundation ensuring long-term accessibility
Terraform compatibility: Drop-in replacement maintaining existing workflows, modules, and provider ecosystem
Community-driven development: Transparent roadmap, open contribution process, and vendor-neutral governance
License certainty: Removes concerns about future licensing restrictions for commercial use

When OpenTofu Makes Sense:

Organizations requiring guaranteed open-source licensing for compliance or philosophical reasons
Teams with significant Terraform investments wanting to avoid vendor lock-in
Environments where community governance and transparency are priorities
Projects needing long-term license stability

Architectural Considerations: As a Terraform fork, OpenTofu inherits the same architectural approach, which means teams evaluating it should consider whether HCL-based infrastructure definition meets their long-term needs for testing, IDE integration, and developer productivity, or whether a programming language-based approach might better serve complex scenarios.

Code Example:

resource "aws_instance" "web" {
 ami = "ami-0c55b159cbfafe1d0"
 instance_type = "t3.micro"

 tags = {
 Name = "HelloWorld"
 }
}

output "instance_ip" {
 value = aws_instance.web.public_ip
}

Ansible

License: GPL v3
Best For: Configuration management with some infrastructure provisioning capabilities

Ansible is primarily a configuration management tool, not a pure Infrastructure as Code tool. While Ansible can provision some cloud resources, its core strength lies in configuring and managing software on existing systems rather than comprehensive infrastructure provisioning.

Ansible provides configuration management and limited infrastructure provisioning through its agentless architecture and simple YAML-based playbooks.

Pulumi Integration: Rather than competing with Ansible, Pulumi complements it perfectly. Use Pulumi for infrastructure provisioning and Ansible for configuration management. Pulumi’s Command provider can execute Ansible playbooks as part of your infrastructure deployment, and many Pulumi customers use both tools together for comprehensive infrastructure automation. See example: Deploy WordPress to AWS using Pulumi and Ansible.

Key Features:

Agentless architecture: No software installation required on target systems
YAML playbooks: Human-readable automation definitions
Idempotent operations: Safe to run multiple times
Large module library: Extensive built-in functionality for various systems
Push-based execution: Centralized control and execution

Code Example:

---
- name: Provision AWS infrastructure
 hosts: localhost
 tasks:
 - name: Create VPC
 amazon.aws.ec2_vpc_net:
 name: ansible-vpc
 cidr_block: 10.0.0.0/16
 region: us-east-1
 tags:
 Environment: production
 register: vpc

 - name: Create subnet
 amazon.aws.ec2_vpc_subnet:
 vpc_id: "{{ vpc.vpc.id }}"
 cidr: 10.0.1.0/24
 region: us-east-1
 tags:
 Name: ansible-subnet

Chef

License: Apache 2.0
Best For: Complex configuration management scenarios requiring programmable logic

Chef is a configuration management tool, not an Infrastructure as Code tool. Chef focuses on configuring and maintaining software, services, and system settings on existing infrastructure rather than provisioning cloud resources.

Chef provides configuration management and system automation using Ruby-based recipes and cookbooks, offering powerful programmability for complex configuration scenarios.

Key Features:

Ruby DSL: Full programming language for configuration logic
Agent-based architecture: Continuous compliance and drift detection
Cookbook ecosystem: Reusable configuration patterns and community recipes
Test Kitchen: Infrastructure testing and validation framework
Enterprise features: Advanced reporting and compliance capabilities

Code Example:

# cookbook/recipes/default.rb
package 'nginx' do
 action :install
end

service 'nginx' do
 action [:enable, :start]
end

template '/etc/nginx/sites-available/default' do
 source 'default.erb'
 owner 'root'
 group 'root'
 mode '0644'
 notifies :restart, 'service[nginx]'
end

Puppet

License: Apache 2.0
Best For: Enterprise environments requiring strong governance and compliance

Puppet is a configuration management tool, not an Infrastructure as Code tool. Puppet specializes in maintaining desired configuration state on existing systems and ensuring compliance, rather than provisioning cloud infrastructure.

Puppet offers enterprise-grade configuration management with a focus on compliance, governance, and declarative system state management.

Key Features:

Declarative language: Puppet DSL for describing desired system state
Compliance reporting: Built-in governance and audit capabilities
Forge marketplace: Community modules and enterprise content
Enterprise support: Professional services and enterprise features
Continuous enforcement: Ongoing configuration compliance monitoring

Code Example:

# manifests/webserver.pp
class webserver {
 package { 'nginx':
 ensure => installed,
 }

 service { 'nginx':
 ensure => running,
 enable => true,
 require => Package['nginx'],
 }

 file { '/var/www/html/index.html':
 ensure => file,
 content => '<h1>Hello from Puppet!</h1>',
 owner => 'www-data',
 group => 'www-data',
 mode => '0644',
 }
}

Salt

License: Apache 2.0
Best For: Python-oriented teams requiring high-performance configuration management

Salt is primarily a configuration management and remote execution tool, not a pure Infrastructure as Code tool. While Salt can manage some infrastructure components, its primary focus is on configuring systems and executing commands across large infrastructures.

Salt provides fast, scalable configuration management and remote execution using Python, designed for high-performance system automation at scale.

Key Features:

Python-based: Leverage the Python ecosystem and libraries
High performance: Fast execution across large infrastructures
Event-driven automation: Reactive automation and orchestration
Pillar data system: Secure, hierarchical data management
Flexible communication: Support for various communication patterns

Code Example:

# /srv/salt/webserver.sls
nginx:
 pkg.installed: []
 service.running:
 - enable: True
 - require:
 - pkg: nginx

/var/www/html/index.html:
 file.managed:
 - source: salt://files/index.html
 - user: www-data
 - group: www-data
 - mode: 644

Infrastructure as Code Tools

Kubernetes - Container Orchestration Platform

License: Apache 2.0
Best For: Container-native infrastructure and application management

While primarily a container orchestration platform, Kubernetes itself serves as an infrastructure as code tool through its declarative YAML manifests and API-driven resource management.

Key Features:

Declarative configuration: YAML-based resource definitions
API-driven: RESTful API for all resource operations
Self-healing: Automatic recovery and reconciliation
Extensible: Custom resources and controllers
GitOps compatible: Works seamlessly with Git-based workflows

Code Example:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx-deployment
spec:
 replicas: 3
 selector:
 matchLabels:
 app: nginx
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: nginx
 image: nginx:1.21
 ports:
 - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: nginx-service
spec:
 selector:
 app: nginx
 ports:
 - port: 80
 targetPort: 80
 type: LoadBalancer

Infrastructure as Code Security and Compliance Tools

While the tools above focus on provisioning and managing infrastructure, a complete IaC ecosystem includes security scanning and compliance tools. These tools complement your primary IaC tool by providing security analysis, policy enforcement, and compliance checking:

Security Scanning Tools

Snyk - License: Proprietary
Leading developer security platform that includes comprehensive infrastructure as code scanning alongside container and application security. Provides real-time vulnerability detection, compliance checking, and automated remediation guidance. Integrates with popular development tools and CI/CD pipelines with extensive enterprise adoption.

Wiz - License: Proprietary
Comprehensive cloud security platform that includes infrastructure as code scanning capabilities. Provides vulnerability management, compliance monitoring, and security posture assessment across cloud environments. Offers integration with development workflows and supports multiple IaC formats with strong enterprise presence.

Checkov - License: Apache 2.0
Popular open-source static analysis tool for infrastructure as code that scans cloud infrastructure configurations for security and compliance issues. Supports Terraform, CloudFormation, Kubernetes, Helm, ARM templates, and more. Integrates with CI/CD pipelines and provides over 1000+ built-in policies covering CIS benchmarks, PCI DSS, and GDPR compliance.

Linting and Validation Tools

TFLint - License: MPL 2.0
Terraform linter focused on possible errors, best practices, and security issues in Terraform configurations. Provides pluggable rule sets for cloud providers (AWS, Azure, GCP) and helps enforce coding standards, detect deprecated syntax, and prevent common configuration errors.

These security tools integrate into CI/CD pipelines alongside your chosen IaC tool to provide comprehensive security coverage throughout the infrastructure lifecycle.

Pulumi ESC

License: Apache 2.0 (Open Source) / Proprietary (SaaS)
Best For: Teams needing centralized configuration and secrets management across environments and tools

Pulumi ESC (Environments, Secrets, and Configuration) is a comprehensive platform for managing configuration data, secrets, and environment variables across your entire infrastructure and application stack. ESC provides a single source of truth for configuration that works with any infrastructure tool, not just Pulumi.

Key Features:

Universal configuration management: Works with any infrastructure tool, CI/CD system, or application
Hierarchical environments: Compose configuration from multiple sources with inheritance and overrides
Dynamic secrets: Integration with cloud providers for short-lived credentials and just-in-time access
Policy-based access control: Fine-grained permissions and audit logging for configuration access
Multiple consumption methods: CLI, REST API, SDK integration, and direct cloud provider integration
GitOps workflows: Version control integration with pull request-based configuration management

Code Example:

# Production environment configuration
values:
 app:
 name: myapp
 version: ${fn.fromJSON(aws.ecr.getAuthorizationToken).password}
 database:
 connectionString: ${pulumi.database.connectionString}
imports:
 - shared/common
 - aws/production

Pulumi Insights

License: Proprietary (SaaS)
Best For: Organizations needing comprehensive cloud resource visibility, search, and compliance monitoring

Pulumi Insights provides cloud resource search, analytics, and compliance capabilities across your entire multi-cloud infrastructure, regardless of how resources were provisioned. It offers a unified view of your cloud resources with powerful search, cost analysis, and policy enforcement.

Key Features:

Universal cloud inventory: Discover and catalog resources across AWS, Azure, Google Cloud, and Kubernetes
Advanced search and analytics: Query resources by tags, properties, relationships, and compliance status
Cost optimization insights: Resource cost analysis, trends, and optimization recommendations
Compliance monitoring: Continuous policy evaluation and drift detection across all cloud resources
Resource relationships: Understand dependencies and relationships between cloud resources
Integration with any IaC tool: Works with resources created by Pulumi, Terraform, CloudFormation, or manually

Use Cases:

Cloud governance: Ensure compliance with organizational policies and industry standards
Cost management: Identify unused resources and optimization opportunities
Security auditing: Track resource configurations and detect security misconfigurations
Multi-cloud visibility: Single pane of glass for resources across different cloud providers

Infrastructure Automation and Management Platforms

While the above tools focus on defining and provisioning infrastructure, several platforms provide automation, orchestration, and management capabilities that work with Infrastructure as Code tools. Important: These are not IaC tools themselves, but rather automation platforms that rely on underlying IaC tools.

IaC Automation Platforms

Pulumi Cloud Best For: Teams using Pulumi IaC who need enterprise-grade collaboration, security, and governance

Pulumi Cloud is the smartest and easiest way to automate, secure, and manage everything you run in the cloud. It serves as the managed service companion to Pulumi’s open-source infrastructure as code tool, providing enterprise capabilities that extend beyond basic IaC functionality.

The relationship between Pulumi IaC and Pulumi Cloud follows a Git/GitHub model: Pulumi IaC is like Git (fully open source and functional on its own), while Pulumi Cloud is like GitHub (a managed service that makes the open-source tool much easier to use securely at scale with teams).

Key Features:

Managed state backend: Secure, scalable state management with transactional protocols
Team collaboration: Full audit logs, RBAC, and identity provider integration
Centralized secrets management: Pulumi ESC for configuration and secrets across environments
Cloud inventory and insights: Pulumi Insights for search, compliance, and drift detection
Remote deployments: CI/CD integrations and automated workflows
Policy enforcement: Built-in security and compliance policy as code
Multi-product platform: Integrated IaC, secrets management, and cloud insights

Pulumi IaC works with or without Pulumi Cloud - you can use DIY backends (S3, Azure Blob, etc.) or the managed service. However, Pulumi Cloud becomes the default experience when you install the Pulumi CLI, providing instant collaboration capabilities without the overhead of building and maintaining your own infrastructure management platform.

HashiCorp Cloud Platform (HCP) - License: Proprietary (SaaS)
Best For: Organizations standardizing on HashiCorp tools across infrastructure and security lifecycle management

HashiCorp Cloud Platform (HCP) is an enterprise-grade SaaS platform that provides unified lifecycle management for infrastructure and security operations. HCP Terraform (formerly Terraform Cloud) serves as the managed service for Terraform/OpenTofu workflows, while the broader platform integrates multiple HashiCorp tools.

Key Features:

HCP Terraform management: Workspace organization, remote execution, and state management for Terraform
Infrastructure lifecycle automation: Continuous validation, drift detection, and module lifecycle management
Private VCS access: Secure integration with private version control repositories
Policy enforcement: Sentinel policy-as-code framework for compliance and governance
Integrated security services: HCP Vault for secrets management, HCP Consul for service networking
Enterprise access controls: RBAC, SAML SSO integration, and fine-grained permissions
Module and image management: HCP Packer for automated machine/container image creation

HCP focuses specifically on HashiCorp tool integration and provides a comprehensive platform for organizations that have standardized on the HashiCorp ecosystem for infrastructure and security operations.

Spacelift - Spacelift is not an Infrastructure as Code tool—it’s an automation and workflow platform that relies on other IaC tools like Terraform, OpenTofu, Pulumi, CloudFormation, and Kubernetes. Spacelift provides CI/CD pipelines, policy enforcement, and collaboration features for teams using these underlying IaC tools.

Env0 - Env0 is not an Infrastructure as Code tool—it’s an automation platform that provides workflow management, governance, and collaboration features for existing IaC tools like Terraform, OpenTofu, and Terragrunt. It adds CI/CD pipelines, cost management, and policy enforcement on top of these tools.

Atlantis - An open-source tool that provides GitOps-style workflows for Terraform and OpenTofu by automatically running terraform plan and apply operations via pull request automation.

Development Environment Tools

Vagrant - Vagrant is not an Infrastructure as Code tool—it’s a development environment management tool that creates and configures lightweight, reproducible virtual development environments. While it can provision VMs, its focus is on local development environments rather than cloud infrastructure provisioning.

Docker Compose - While not an IaC tool, Docker Compose defines multi-container applications and can be used alongside IaC tools for application deployment after infrastructure provisioning.

These platforms and tools serve important roles in the infrastructure automation ecosystem but should not be confused with Infrastructure as Code tools themselves. They enhance and orchestrate the work of actual IaC tools rather than replacing them.

The Future of Infrastructure as Code

The infrastructure as code landscape is rapidly evolving toward software engineering maturity, with several transformative trends reshaping how organizations approach infrastructure:

Software Engineering Convergence: The most significant trend is the convergence of infrastructure and software engineering practices. Organizations are moving away from limited DSLs toward full programming languages that enable testing, debugging, refactoring, and other software engineering best practices. This shift enables infrastructure teams to leverage the same tools, skills, and methodologies that have proven successful in application development.

Real-Time Cloud Integration: Native cloud provider SDK integration is becoming the standard, replacing community-maintained providers that lag behind new cloud features. Organizations expect same-day access to new cloud services without waiting weeks or months for provider updates.

Comprehensive Testing Paradigms: Infrastructure testing is evolving beyond basic integration tests to include unit testing, property-based testing, and continuous validation. Teams are applying test-driven development principles to infrastructure, catching issues before deployment rather than discovering them in production.

Internal Developer Platform Evolution: Organizations are building sophisticated Internal Developer Platforms that provide self-service infrastructure capabilities while maintaining governance and compliance. These platforms leverage infrastructure as code tools to create standardized, reusable components that accelerate development velocity.

AI-Enhanced Development: Integration of AI tools to help generate, optimize, and troubleshoot infrastructure code, with particular strength in environments that use familiar programming languages where AI assistance is most mature. Emerging technologies like Model Context Protocol (MCP) and AI prompt templates are beginning to enable more sophisticated AI-infrastructure interactions.

These trends favor tools that embrace software engineering principles from the ground up, rather than attempting to retrofit programming capabilities onto template-based or DSL-limited approaches.

Bring software engineering to IaC

Define infrastructure across AWS, Azure, Google Cloud, and Kubernetes in the language your team already knows, with testing, IDE support, and reusable components.

Get started

Migration and Adoption Strategies

Organizations don’t need to choose between maintaining existing infrastructure and adopting modern IaC approaches. Proven migration strategies enable gradual adoption while preserving operational stability.

Incremental Adoption Approaches

Team-by-Team Migration: Start with new projects or specific teams, allowing gradual skill development and process refinement. Teams can maintain existing infrastructure while building new capabilities with modern tools.

Project-by-Project Transition: Migrate individual applications or services incrementally, enabling teams to learn and optimize approaches before expanding scope.

Hybrid Operations: Use state integration and import tools to reference existing infrastructure while building new components with modern IaC tools.

Proven Migration Timelines

Real-world migrations demonstrate that adoption can be remarkably fast with proper tooling:

Atlassian Bitbucket: Complete Terraform to Pulumi migration in 2 days using automated conversion tools. This rapid migration was enabled by existing Python expertise on the team, well-structured Terraform code, and straightforward infrastructure patterns that converted cleanly.
Enterprise migrations: Typical team migrations complete in weeks, not months, depending on infrastructure complexity and team preparation
Learning curve: Teams with programming experience adapt to language-based IaC approaches within days, while those new to programming may require additional training time

Migration Tools and Resources

Automated Conversion: Tools like tf2pulumi and pulumi convert automatically translate existing infrastructure definitions, preserving logic and structure while upgrading to modern approaches.

State Integration: Import existing cloud resources and reference Terraform state during transition periods, enabling zero-downtime migrations.

Training and Support: Comprehensive documentation, tutorials, and community resources accelerate team onboarding and reduce migration risks.

Best Practices for Successful Adoption

Start with pilot projects to build confidence and establish patterns
Leverage automated migration tools to reduce manual conversion effort
Maintain state integration during transition periods for operational continuity
Invest in team training to maximize productivity gains from modern approaches
Establish testing practices early to realize reliability and quality benefits

The key insight: migration to modern IaC approaches is a technical upgrade, not a complete rebuild. Organizations can preserve existing investments while gaining access to superior tooling and practices.

Frequently Asked Questions

Which IaC tool should I choose for AWS?

For teams wanting programming languages: Pulumi IaC and AWS CDK both offer excellent developer experiences with general-purpose programming languages. Pulumi provides multi-cloud flexibility, while CDK offers deep AWS-native integration.

For AWS-only deployments: CloudFormation provides the deepest native AWS service support and direct integration with AWS services.

For Terraform ecosystem users: OpenTofu provides open-source licensing with Terraform compatibility, while Terraform offers extensive community resources and established workflows.

Is Terraform still worth learning in 2026?

Terraform remains a valuable skill and viable choice for many scenarios, though teams should consider their specific needs and long-term goals:

When Terraform Makes Sense:

Teams building on existing Terraform infrastructure and expertise
Organizations with successful Terraform workflows and established processes
Projects that benefit from Terraform’s extensive module ecosystem
Teams comfortable with HCL and template-based approaches

When to Consider Alternatives:

Teams wanting comprehensive testing capabilities (unit, property, integration testing)
Organizations requiring full IDE support with debugging and refactoring
Projects needing complex programmatic logic or extensive code reuse
Teams with strong programming backgrounds who prefer familiar languages

Key Considerations:

Licensing: Terraform’s Business Source License may create restrictions for some commercial scenarios
Ecosystem: Terraform has the largest provider ecosystem and community resources
Learning investment: Consider whether learning HCL aligns with your team’s skill development goals
Migration options: Tools exist for migrating between different IaC platforms when needs change

The choice often depends on team background, infrastructure complexity, and development workflow preferences rather than one approach being universally superior.

How do I get started with infrastructure as code?

Choose Based on Your Team Profile:

Programming-oriented teams: Consider Pulumi IaC for familiar languages with full IDE support, testing frameworks, and multi-cloud flexibility
Template-oriented teams: OpenTofu or Terraform offer extensive community resources, established workflows, and broad ecosystem support
Cloud-specific teams: Native tools (AWS CDK/CloudFormation, Azure ARM/Bicep, Google Cloud Infrastructure Manager) provide deep platform integration
Kubernetes-focused teams: Consider Crossplane or Pulumi Kubernetes Operator for container-native approaches

Getting Started Steps:

Start small: Begin with simple projects like deploying a single application or basic infrastructure
Use examples: Leverage existing templates, tutorials, and community examples to accelerate learning
Plan for growth: Consider how your tool choice will scale with team size and infrastructure complexity
Experiment: Most tools offer free tiers or trials - try multiple approaches with simple projects to find what fits your team best

What’s the difference between configuration management and infrastructure provisioning?

Infrastructure provisioning (Pulumi IaC, Terraform, CloudFormation) creates and manages cloud resources like VMs, networks, and databases
Configuration management (Ansible, Chef, Puppet) configures and maintains software on existing systems
Many modern tools do both, with Pulumi IaC and Ansible offering comprehensive capabilities across both domains

Can I use multiple IaC tools together?

Absolutely! Many organizations use complementary tools for different aspects of infrastructure management:

Pulumi + Configuration Management:

Pulumi + Ansible: Infrastructure provisioning + server configuration (example)
Pulumi + Chef/Puppet: Cloud resources + complex configuration management

Pulumi + Native Cloud Tools:

Reference existing CloudFormation/ARM deployments from Pulumi programs
Import CloudFormation/ARM resources into Pulumi for gradual migration
Use Pulumi alongside CDK for different parts of AWS infrastructure

Pulumi + Kubernetes:

Pulumi Kubernetes Operator (PKO) for GitOps workflows
Pulumi + Crossplane for layered infrastructure management
Pulumi + security scanners like Checkov or Terrascan for compliance

Terraform + Pulumi Coexistence:

Use Terraform for base infrastructure and Pulumi for Internal Developer Platforms (IDPs)
Reference existing Terraform state from Pulumi programs during gradual migration
Manage different infrastructure layers with different tools based on team expertise

Pulumi IaC is designed for heterogeneous environments where multiple tools may be in use. For example, you can manage existing Terraform or CloudFormation resources with Pulumi, either using both tools in tandem or for temporary management while migrating to native Pulumi IaC code. Other Pulumi tools, like Pulumi IDP, also enable you to manage IaC self-service workflows like other tools on this list. See what’s possible when migrating to Pulumi.

Which tool has the best learning resources?

Pulumi: Excellent documentation with examples in multiple languages, comprehensive tutorials
Terraform/OpenTofu: Extensive community content, many courses and books available
AWS CDK: Outstanding official documentation with workshops and examples
Kubernetes: Vast ecosystem but steeper learning curve

How important is open-source licensing for IaC tools?

Very important for long-term strategy:

Apache 2.0 (Pulumi, CDK) offers maximum flexibility
MPL 2.0 (OpenTofu) ensures open-source availability
BSL (Terraform) restricts commercial competitors
Proprietary (CloudFormation, ARM) ties you to specific vendors

Choose open-source tools like Pulumi or OpenTofu to avoid vendor lock-in.

What about security and compliance?

All major IaC tools support security best practices:

Built-in policy engines: Pulumi CrossGuard, AWS Config, Azure Policy
Third-party scanners: Checkov, KICS, Terrascan work with all tools
Compliance frameworks: Most tools support SOC 2, PCI DSS, CIS benchmarks
Secret management: Integration with Vault, AWS Secrets Manager, Azure Key Vault

Pulumi offers the most comprehensive built-in security features with CrossGuard policy as code.

Can I migrate from one IaC tool to another?

Yes, and many organizations are successfully migrating to overcome limitations in their current tools. Migration tools and proven approaches exist:

Proven Migration Success Stories:

Atlassian Bitbucket: Completed migration from Terraform to Pulumi’s Python-based IaC in just 2 days using migration tools
Starburst: Achieved 112x faster deployments after incrementally migrating from Terraform
Multiple organizations: Report 80-90% deployment time reductions after moving to programming language-based approaches

Available Migration Tools:

Pulumi: Offers pulumi convert for importing from Terraform, ARM, and CloudFormation with state integration
Terraformer: Can import existing cloud resources into Terraform/OpenTofu
CDK Migrate: Helps move from CloudFormation to CDK
Manual migration: Always possible by recreating resources in the new tool

Migration Strategies:

Incremental adoption: Migrate team-by-team or project-by-project without disruption
State integration: Reference existing infrastructure during transition periods
Zero downtime: Tools like Pulumi enable seamless migration without service interruption

The key is choosing tools that provide comprehensive migration support and incremental adoption paths rather than requiring “rip and replace” approaches.

Conclusion: The Evolution of Infrastructure as Code

The infrastructure as code landscape in 2026 reflects a maturing field where different approaches serve different organizational needs and team preferences. The evolution from manual processes to automated infrastructure has branched into multiple viable paths, each with distinct advantages.

The Spectrum of Approaches:

Template-based tools like Terraform and OpenTofu continue to serve teams effectively, particularly those comfortable with declarative configuration and established HCL workflows. Cloud-native solutions provide deep platform integration for single-cloud strategies. Programming language-based approaches offer familiar development experiences for teams seeking comprehensive testing, IDE integration, and code reusability.

Evidence of Success Across Approaches:

Organizations achieve significant improvements regardless of their chosen path—the key is selecting tools that align with team expertise and infrastructure requirements. Understanding what DevOps is and platform engineering concepts helps inform these decisions. Success stories span the entire ecosystem, from Terraform’s widespread enterprise adoption to programming language approaches enabling dramatic productivity gains at companies like Unity, Snowflake, and Starburst.

Choosing Your Path:

The most successful organizations focus on key decision criteria:

Team background: Match tools to existing skills and preferred development approaches
Infrastructure complexity: Consider testing, abstraction, and maintainability needs
Organizational constraints: Factor in licensing, governance, and compliance requirements
Growth trajectory: Plan for how needs may evolve with scale and team expansion
Integration requirements: Ensure compatibility with existing workflows and toolchains

The Future of Infrastructure:

The industry continues evolving toward treating infrastructure as software, but this transformation takes many forms. Organizations exploring serverless architectures and container strategies particularly benefit from programmable infrastructure approaches. Whether through enhanced DSLs, visual design tools, programming languages, or hybrid approaches, the goal remains consistent: enabling teams to manage infrastructure with the same reliability, collaboration, and velocity they expect from modern software development.

For teams ready to embrace programming language-based infrastructure as code, get started with Pulumi for free to experience how familiar languages and software engineering practices can transform infrastructure management with comprehensive testing, powerful abstractions, and seamless multi-cloud support. Want to see where infrastructure is headed next? Explore Pulumi Neo, the AI agent that proposes changes, runs previews, and opens PRs alongside your team.

Six Live Kubernetes Recommendations: AKS, Cilium, Rate Limiting, and More

Adam Gordon Bell — Mon, 15 Jun 2026 12:00:00 +0000

On June 10th, Engin and I ran a live workshop building an AKS cluster, an Azure Container Registry, and a random-cat web app from scratch in C#. This is the writeup, including the parts we didn’t get to live.

Live demos keep you honest. On June 10th my AKS workshop went a little sideways. Partway through, Docker Hub rate-limited my image pull and we had to adapt the content on the fly. The original plan was to stand up an AKS cluster with Cilium, an Azure Container Registry with the cluster’s pull permission wired in code, and a random-cat web app, then split the infrastructure from the workload into separate Pulumi stacks. Live, we didn’t make it through all of that, but we had some fun tangents and it turned out to be a great session. So here are my six recommendations for working with Kubernetes on Azure from this recent workshop.

1. Pick the language your team already uses

I opened the session by asking the room what language they prefer to work in, with one caveat from me: “Hopefully, it’s not PowerShell because this is not in PowerShell.” We got some C# answers and zero requests for PowerShell, which works out, because the whole workshop is in ~~PowerShell~~ C#.

I chose C# because, first, I genuinely love the language. I spent the early part of my career as a C# developer, and on Azure it just makes sense. Second, an Azure workshop tends to draw a big turnout of C# devs, and if they aren’t C# devs themselves, they often work somewhere the backend team uses C#. Pulumi lets you bring that language to your infrastructure rather than learn a new DSL.

AI coding assistants also came up. LLMs have substantially more C# and Go in their training data than HCL or Bicep. As I put it live: “If you’re working in HCL, well, there’s not that much out there in the training set, but there’s lots of C#. There’s lots of Go.” I co-authored parts of the workshop code with Claude.

Pick the language your team already writes.

Pulumi runs on the language your backend engineers know, and that language has more AI assistant training data behind it too.

Aside: everyone’s path ran through Docker Swarm

We also had some fun historical conversations about orchestrators. I asked Engin: “Did you use anything before Kubernetes became popular? Like Mesosphere or any of those other interesting solutions of the heyday?”

Engin had walked the same road most of us did.

Engin: “We used the classical Ansible powered provisioning of VMs… And then we switched to Consul to have a little bit more control… And then we had a short intermezzo with Docker Swarm… So Docker Swarm with Portainer was really goated. And Kubernetes then took over.”

My own history rhymed with his.

Adam: “I worked at a place where we used a bunch of shell scripts to have like JVM services, like a red green deploy… And then it was Docker Swarm for a little bit. And then it was Kubernetes.”

No single tool won. Teams picked what fit their existing skills and shifted gradually as the ecosystem matured.

2. Use Cilium on AKS if you can

By default, Kubernetes on AKS doesn’t turn on Cilium. My strong recommendation is to turn it on. The option is called Azure CNI Powered by Cilium, and it takes one config block:

NetworkProfile = new ACI.ContainerServiceNetworkProfileArgs
{
 NetworkDataplane = "cilium",
 NetworkPlugin = "azure",
 NetworkPluginMode = "overlay",
 NetworkPolicy = "cilium",
 PodCidr = "192.168.0.0/16",
},

This buys you eBPF networking instead of kube-proxy and iptables. eBPF lets you run sandboxed programs inside the Linux kernel itself, so packet handling and NetworkPolicy enforcement happen down in the kernel instead of up in userspace. You also get Hubble flow visibility built in, the same technology as GKE’s Dataplane V2. As I said live, “this is very easy to set up on Azure. You just network data plane, set it to Cilium. Network policy, set it to Cilium.”

Self-hosted, Cilium is helm install cilium with a dozen flags and a lifecycle you own forever. On AKS, it’s these five lines and Azure runs it. On a getting-started cluster, I’ll take the managed version every time. Owning that lifecycle yourself buys you nothing here.

Full code: 01-cluster/Program.cs

On managed Kubernetes, complexity you’d self-host becomes a field you set.

AKS gives you the managed version of eBPF networking. Pulumi makes the configuration code instead of portal clicks.

3. Debugging Kubernetes is easier when your tools have the full context

The rate limit wasn’t even the first failure of the hour. That honor went to Azure, which refused to provision the cluster on the first pulumi up. I was running in Canada Central, and Pulumi Neo read the error and diagnosed it:

Neo: “Capacity constraints in the selected region. We might need to change our location.”

Adam: “Maybe this Canadian Azure instance is not all it’s cracked up to be.”

Engin: “East US is always a good bet.”

We switched the region to East US, and the cluster came up. That is the kind of diagnosis a stack-aware assistant is good at, reading the actual provisioning error instead of leaving me to guess.

Then the image step hit. The workshop’s original flow pulled the cat app image into ACR with az acr import --source docker.io/agbell/my-random-cat, and Docker Hub rate-limited me, even though I was the only person in the room running the code. So how do you blow through an anonymous pull limit all by yourself? The trick is that az acr import runs server-side. Azure does the pull from Docker Hub, not my laptop, and it goes out over an egress IP shared by who-knows-how-many other Azure tenants. Docker Hub counts anonymous pulls per IP address, so all of that shared traffic lands in one bucket. It flagged me at the worst possible moment: “I just got flagged by Docker Hub. That is awesome. That will be a challenge for this demo.”

Nothing focuses a workshop like watching your own demo get rate-limited in front of a live audience.

Live, I asked Neo to route around it: “we’re using Neo. And Neo is going to switch us out to a nice Hello World project.” Neo could see the whole stack, down to the failing image reference, and it swapped the workload over to a standard hello-world image hosted on Azure, so the deploy no longer depended on Docker Hub. A few minutes later: “we should have a Hello World container running here. It’s way less fun than my image… just a bunch of random cats. But here we can see that this is working.” I won’t oversell it. The cat was gone, and we didn’t get through every planned stage. But the cluster was up with a workload deployed, and the session kept moving.

That’s the case for a stack-aware assistant over a chatbot. A general chatbot can tell you that Docker Hub has rate limits and that you should host images closer to home. Neo was working with the actual program state, so the workaround it proposed was already wired to my resources. Engin put it this way: “for me, Pulumi Neo, the subcommand is the new Pulumi app.” When something breaks mid-demo, the tool that can read your stack is the one that helps.

The long-term fix was switching to az acr build, building the image in ACR from source, so the app image never has to come from Docker Hub at all. Every stage in the repo now builds the cat image this way, from source shipped in each stage’s app/ folder:

az acr build --registry <your-acr> --image my-random-cat:latest app

No local Docker, and the app image never touches Docker Hub. The only Docker Hub dependency left is the base image (python:3.9-slim), pulled server-side by ACR’s build instead of by an import you re-run every time.

Full code: 02-app/Program.cs

An assistant that can read your stack beats one that can’t.

A general chatbot knows Docker Hub has rate limits. Neo could see the deployment and the failing image reference, and that context is what turned advice into a working fix mid-session.

4. You don’t have to abandon Terraform to start

One attendee, Martin, had been using Terraform for eight-plus years. His question, put directly: “I’m using TF for eight plus years. Why should I consider Pulumi? Where is the strength?… It seems unnecessary to TF for me. At least from what I’m seeing now.”

Fair question, and the honest answer is you don’t have to abandon your modules. Pulumi has first-class Terraform module support, so you can reference existing modules from a Pulumi program. Python ML teams can import infrastructure modules, and platform teams can wrap them in typed abstractions. The portfolio doesn’t change. The programs that use it just get more capable.

The sharper reason is about edges. HCL and Bicep are purpose-built DSLs, and when you hit what they can’t express, there’s no escape hatch inside the language. With a general-purpose language you can always drop to the SDK or shell out. The edges still exist. They’re just further out, and that’s the whole difference once a project gets past the basics.

You don’t have to rip and replace.

Pulumi can reference existing Terraform modules. Move what makes sense now. The DSL/language tradeoff shows up at the edges, not the basics.

5. Split slow infrastructure from fast workloads

Once we had the cluster up and the app running, the single program had grown: cluster, ACR, AcrPull, image build, deployment, service. Every new workload meant editing the file that also declares the cluster. We had planned this turn in the session in advance. I posed the problem and Engin had the recommendation ready:

Adam: “What do you think the problem is with my code here, Engin? … The problem is like every time I want to add a new service to my Kubernetes cluster, I have to go into this file that declares the cluster and all its infrastructure. I think I would prefer to split it.”

Engin: “Create other stacks which are then referencing each other to get the values out. Actually, I highly recommend.”

Adam: “Yes, I totally agree with you, Engin. We should split these out into two separate stacks.”

So that’s what we did. The infrastructure stack exports the kubeconfig and registry URL. The workload stack references those outputs via StackReference. The slow-moving infra and the fast-moving app become independent deployment units. The infra team and the app team can ship on their own cadences, without fighting over one file.

// workload/Program.cs — pulls cluster outputs, never touches cluster resources
var clusterStack = new StackReference(cfg.Require("clusterStack"));
var kubeconfig = clusterStack.GetOutput("kubeconfig");
var acrLoginServer = clusterStack.GetOutput("acrLoginServer");

The tricky part is one we didn’t cover live, which is how you split a running workload without taking down the running services. The real answer is migrating the stack state, moving resources between stacks instead of destroying and recreating them. As I noted in the session: “now we are just removing the service from our project. If this was actual production stuff, I don’t think I’d be just shutting down the service… You can actually just make changes to the state rather than tearing them down.”

For the workshop, a simpler move kept the Kubernetes cluster up and the demo moving: keep the project name. Pulumi identifies stacks as org/project/stack, so if you move the cluster code into aks-cluster/ while keeping the project name kube-kitties, Pulumi still sees org/kube-kitties/dev. Same stack, nothing recreated. Rename the project and you get a second cluster.

In stage four of the demo, I showed how to extend this pattern to work with existing YAML manifests: ConfigGroup drives raw Kubernetes YAML through the Pulumi provider, giving you previews and dependency ordering without rewriting files. One transformation swaps the Docker Hub image reference for the ACR copy, so manifests stay portable and pods pull from your private registry.

Code: 03-split/ and 04-split-yaml/

Keep the project name when you split, and the cluster doesn’t move.

Pulumi identifies stacks as org/project/stack. Same project name means the same stack. Rename it and you spin up a new cluster.

Run Kubernetes on AKS with code

Provision your AKS cluster, container registry, and workloads in C#, then split infrastructure from apps into separate stacks so each ships on its own cadence.

Get started

6. GitOps is the way

The final stage of the workshop is left as homework, and it’s a great one to pick up. GitOps is the standard way to run Kubernetes at this point, the One True Way™ depending on who you ask. My position from the call: “I think I’m on team GitOps.”

The take-home is the 05-gitops folder. We didn’t demo it live, but the code is real and verified: Pulumi stands up the cluster, installs Argo CD, and registers the cat as an Argo Application. From there, changes are git push.

To see Kubernetes and GitOps worked through end to end, watch the video above. Engin also recently ran a GCP workshop with great examples of GitOps with Pulumi, using Flux.

Let Pulumi stand up the cluster, then let GitOps run the apps.

Pulumi installs Argo CD or Flux as part of the infrastructure. From there, workload changes are a git push. Stage 5 of the workshop repo is a complete Argo CD example.

Try it yourself

That’s the whole workshop, six recommendations and two live failures that ended with a cat app running on AKS. To run it yourself, clone the repo. There’s a DEMO.md with every step of this session, and the stage folders mirror the arc above. 01-cluster is the infrastructure checkpoint, 02-app adds the cat app built into your own registry, 03-split and 04-split-yaml separate infrastructure from workload, and 05-gitops is the homework.

Pick a stage and bring it up:

git clone https://github.com/pulumi/workshops
cd workshops/az-getting-started-aks/02-app
pulumi up
curl http://$(pulumi stack output catServiceIp)/

Expect 5 to 8 minutes waiting on cluster create. That’s just how long AKS takes. And since each folder is a complete, runnable checkpoint with its own stack, run pulumi destroy before moving to the next stage, or you’ll have several clusters billing at once.

You’ll need an Azure subscription, the Pulumi CLI, and the .NET SDK. Basic Kubernetes familiarity helps. For more on the building blocks, see the Azure Native provider and the Kubernetes provider in the Pulumi Registry.

The full workshop code lives at github.com/pulumi/workshops under az-getting-started-aks. Check out our upcoming sessions on the workshop page, or subscribe to the newsletter to hear what’s coming up.

If there’s a seventh recommendation hiding in here, it’s the one I learned the hard way: never pull from Docker Hub live again. I’ll be keeping my images on a paid cloud registry from now on.

Engin and I will be doing more of these, and we’d love to see you at the next one.

Parts of the workshop code were authored by Claude and Pulumi Neo, and parts of this writeup got help from both as well.

Use Your Mac for AI Agents: Self-Host Gemma 4 12 B with Pulumi and Tailscale

Pablo Seibelt — Thu, 04 Jun 2026 00:00:00 +0000

If you run AI tools and agents, you’ve probably accepted three tradeoffs: your data leaves your network, you can’t work offline, and your bill scales with usage.

Open-weight models now run well on consumer hardware. Once the model is on your machine, your data stays local, inference works offline, and tokens cost nothing. If you own a modern Mac, you can run a high-quality model yourself.

Gemma 4 is an open-weights model family from Google. This post focuses on Gemma 4 12 B, released in June 2026, using Unsloth’s Q8_0 GGUF. The 12 B model fits comfortably on a modern Mac while leaving enough headroom for local llama.cpp and a chat UI.

We’ll use llama.cpp for host-native inference, k3d for a local Kubernetes cluster, Pulumi for infrastructure as code, and Tailscale for secure access.

Prerequisites

This setup was validated on the following hardware:

macOS 26 Tahoe, version 26.5
MacBook Pro with Apple M3 Max
36 GB RAM

On this machine, llama.cpp reported about 20 output tokens per second for a 160-token validation response with unsloth/gemma-4-12b-it-GGUF, gemma-4-12b-it-Q8_0.gguf, and a 131,072-token context. Sustained throughput varies by prompt length, thermal state, and llama.cpp settings.

You’ll need brew, docker, pulumi, and tailscale installed. We’ll also install k3d during the process.

Run Gemma 4 with host-native llama.cpp

We use llama.cpp directly on macOS to leverage Apple Metal acceleration. Running the LLM on the host is more efficient than trying to pass GPU access into a local Kubernetes VM.

Install the build tools:

brew install cmake git

Then build llama.cpp from source and download the multimodal projector. In validation, Homebrew llama.cpp 9430 could run text inference, but it could not load the new Gemma 4 12 B projector and failed with unknown projector type: gemma4uv. Building current llama.cpp from source fixed that.

llm_home="$HOME/pulumi-gemma4-llm"
mkdir -p "$llm_home/models" "$llm_home/logs"

if [ ! -d "$llm_home/llama.cpp/.git" ]; then
 git clone --depth 1 https://github.com/ggml-org/llama.cpp.git "$llm_home/llama.cpp"
fi

cmake -S "$llm_home/llama.cpp" \
 -B "$llm_home/llama.cpp/build" \
 -DGGML_METAL=ON \
 -DGGML_BLAS=ON \
 -DCMAKE_BUILD_TYPE=Release

cmake --build "$llm_home/llama.cpp/build" --target llama-server -j 10

curl -L --fail \
 --output "$llm_home/models/mmproj-F16.gguf" \
 https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/resolve/main/mmproj-F16.gguf

Then download and run the model with this command:

"$HOME/pulumi-gemma4-llm/llama.cpp/build/bin/llama-server" \
 --hf-repo unsloth/gemma-4-12b-it-GGUF \
 --hf-file gemma-4-12b-it-Q8_0.gguf \
 --mmproj "$HOME/pulumi-gemma4-llm/models/mmproj-F16.gguf" \
 --host 127.0.0.1 \
 --port 18080 \
 --ctx-size 131072 \
 --parallel 1 \
 --jinja \
 --reasoning off

We use port 18080 because 8080 is commonly used and is likely to conflict with another service you may already have running locally. If your port 8080 is free, you can use it and adjust the Pulumi config later.

The model file is about 12.65 GB, and the projector is about 116 MB. Gemma 4 12 B advertises a 131,072-token context, and this Mac loaded that full context with --parallel 1. llama.cpp projected about 15.1 GiB of Apple Metal device memory for the text model and about 258 MiB worst-case memory for the projector, leaving enough headroom for Open WebUI and the rest of the local stack. The --reasoning off flag keeps OpenAI-compatible chat responses visible in clients that do not read separate reasoning fields.

With --mmproj, /v1/models advertised capabilities: ["completion","multimodal"]. In local validation, Open WebUI accepted an uploaded Pulumi logo image and Gemma 4 described it correctly. A small WAV file also worked through the OpenAI-compatible input_audio request shape, though llama.cpp logs still mark audio input as experimental.

Verify the LLM API

Open a new terminal and check if llama.cpp is responding:

curl http://127.0.0.1:18080/v1/models

The /v1/models endpoint should return the model ID unsloth/gemma-4-12b-it-GGUF. Now try a chat completion:

curl http://127.0.0.1:18080/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "unsloth/gemma-4-12b-it-GGUF",
 "messages": [{"role": "user", "content": "Reply with exactly: OK"}],
 "temperature": 0,
 "max_tokens": 32
 }'

The chat prompt Reply with exactly: OK should return content OK. In validation, llama.cpp reported an output token velocity of about 20 tokens per second for a longer 160-token response.

Keep llama.cpp running after reboot

For a permanent setup, put the llama.cpp startup script and logs under a folder in your home directory and let launchd restart it when you sign in:

llm_home="$HOME/pulumi-gemma4-llm"
mkdir -p "$llm_home/logs" "$HOME/Library/LaunchAgents"

cat > "$llm_home/start-llama-server.sh" <<'EOF'
#!/bin/zsh
set -euo pipefail

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"

exec "$HOME/pulumi-gemma4-llm/llama.cpp/build/bin/llama-server" \
 --hf-repo unsloth/gemma-4-12b-it-GGUF \
 --hf-file gemma-4-12b-it-Q8_0.gguf \
 --mmproj "$HOME/pulumi-gemma4-llm/models/mmproj-F16.gguf" \
 --host 127.0.0.1 \
 --port 18080 \
 --ctx-size 131072 \
 --parallel 1 \
 --jinja \
 --reasoning off
EOF

chmod +x "$llm_home/start-llama-server.sh"

cat > "$HOME/Library/LaunchAgents/com.pulumi.gemma4.llama-server.plist" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>com.pulumi.gemma4.llama-server</string>
 <key>ProgramArguments</key>
 <array>
 <string>$llm_home/start-llama-server.sh</string>
 </array>
 <key>WorkingDirectory</key>
 <string>$llm_home</string>
 <key>RunAtLoad</key>
 <true/>
 <key>KeepAlive</key>
 <true/>
 <key>StandardOutPath</key>
 <string>$llm_home/logs/llama-server.out.log</string>
 <key>StandardErrorPath</key>
 <string>$llm_home/logs/llama-server.err.log</string>
</dict>
</plist>
EOF

launchctl bootout gui/$(id -u)/com.pulumi.gemma4.llama-server 2>/dev/null || true
launchctl bootstrap gui/$(id -u) "$HOME/Library/LaunchAgents/com.pulumi.gemma4.llama-server.plist"
launchctl kickstart -k gui/$(id -u)/com.pulumi.gemma4.llama-server

Check the launchd service and llama.cpp logs:

launchctl print gui/$(id -u)/com.pulumi.gemma4.llama-server
tail -f "$HOME/pulumi-gemma4-llm/logs/llama-server.err.log"

If you want to stop llama.cpp later, unload the launchd service:

launchctl bootout gui/$(id -u)/com.pulumi.gemma4.llama-server

Deploy Open WebUI with Pulumi and k3d

Now we’ll deploy Open WebUI into a local Kubernetes cluster. This provides a polished chat interface that connects to our host-native LLM.

First, install k3d if you haven’t already:

brew install k3d

Create a new cluster for this project:

k3d cluster create pulumi-gemma4-blog-qa

We’ll use the Pulumi program in pulumi/examples. This program defaults to runtimeMode=host, which creates a Kubernetes ExternalName service pointing to your host machine.

Why not run the LLM inside Kubernetes on this Mac? Pulumi can do that, and the example supports it with runtimeMode=cluster, but that path is meant for Linux hosts with NVIDIA or AMD GPU device plugins.

On macOS, llama.cpp enables Metal by default, and Metal acceleration is available to native macOS processes. k3d runs Linux containers through Docker Desktop, so those pods do not get direct access to the Mac’s Metal device. Docker’s own vLLM Metal announcement calls out the same boundary: Metal-backed inference runs natively on the host because there is no Metal GPU passthrough for containers. That is why this setup keeps inference host-native and lets Pulumi manage the Kubernetes UI, service wiring, and optional Tailscale access around it.

Clone the examples repo, navigate to the program directory, and initialize a new stack:

git clone https://github.com/pulumi/examples.git
cd examples/kubernetes-py-self-host-gemma4-llm
pulumi stack init gemma4-local

Configure the program to match your local setup:

pulumi config set hostLlmPort 18080
pulumi config set llmBaseUrl http://llm-server:18080/v1

The Kubernetes service named llm-server maps to host.k3d.internal. In our validation, we confirmed that a disposable k3d pod could reach the Mac’s llama.cpp API at http://llm-server:18080/v1/models after a CoreDNS restart.

kubectl rollout restart deployment coredns -n kube-system

Run pulumi up to deploy Open WebUI and connect it to host-native llama.cpp:

pulumi up

In our validation environment, this command successfully reached the resource synthesis phase without requiring Tailscale credentials because Tailscale exposure is opt-in.

Access Open WebUI through Tailscale

Tailscale allows you to access your private Open WebUI instance from any device on your tailnet. Note that we only expose the web interface, not the raw LLM API, to keep the system secure.

The base Open WebUI deployment works without Tailscale credentials. To expose the web UI on your tailnet, enable Tailscale resources and provide an explicit api_key or OAuth/identity token:

pulumi config set enableTailscale true
pulumi config set tailscale:apiKey YOUR_API_KEY --secret

Once configured, Pulumi will create a Tailscale device or proxy that routes traffic to your Open WebUI service.

Use the model with Pi

Open WebUI gives you a browser-based chat interface, but local models are also useful from coding agents. Pi is the local coding agent used for this validation; if you do not use Pi, treat this section as an example of how any OpenAI-compatible client can point at the same local endpoint. Pi can point at the same OpenAI-compatible llama.cpp endpoint and use the model running on your Mac.

For a fresh Pi config, create ~/.pi/agent/models.json with a local provider that points at the llama.cpp API:

{
 "providers": {
 "local-llama": {
 "baseUrl": "http://127.0.0.1:18080/v1",
 "api": "openai-completions",
 "apiKey": "local",
 "compat": {
 "supportsDeveloperRole": false,
 "supportsReasoningEffort": false
 },
 "models": [
 {
 "id": "unsloth/gemma-4-12b-it-GGUF",
 "name": "Gemma 4 12B Q8 (local llama.cpp)",
 "reasoning": false,
 "input": ["text"],
 "contextWindow": 131072,
 "maxTokens": 1024,
 "cost": {
 "input": 0,
 "output": 0,
 "cacheRead": 0,
 "cacheWrite": 0
 }
 }
 ]
 }
 }
}

Then set Pi to use that provider and model by default in ~/.pi/agent/settings.json:

{
 "defaultProvider": "local-llama",
 "defaultModel": "unsloth/gemma-4-12b-it-GGUF",
 "defaultThinkingLevel": "off",
 "hideThinkingBlock": true
}

If you already have Pi configuration files, merge the local-llama provider and defaults into your existing JSON instead of replacing the files.

Manage your local AI stack as code

Let Pulumi provision the Kubernetes cluster, service wiring, and Tailscale access around your self-hosted model, so you can rebuild the whole setup with pulumi up.

Get started

Advanced: Linux GPU in-cluster serving

If you’re running on a Linux host with an NVIDIA or AMD GPU, you can run the LLM directly inside the Kubernetes cluster. This requires the NVIDIA or ROCm device plugins.

The Pulumi program supports this through runtimeMode=cluster. In this mode, it deploys a LlmServer pod that manages the llama.cpp process within the cluster, using GPU resource requests to ensure hardware acceleration.

Cleanup

When you’re done, you can tear down the resources:

pulumi destroy
k3d cluster delete pulumi-gemma4-blog-qa
# Stop llama.cpp using the PID from your terminal
kill <PID>

Build an EKS Environment Factory with Pulumi and vCluster

Pablo Seibelt — Thu, 04 Jun 2026 00:00:00 +0000

AWS reports in an AWS Architecture Blog case study that Deloitte’s move to a virtual cluster model on Amazon EKS resulted in 89% faster testing environment provisioning. By consolidating dozens of disparate clusters into a single host cluster with over 50 vCluster instances, the case study says Deloitte saved about 500 QA hours per year. This “Environment Factory” pattern allows platform teams to provide isolated, ephemeral Kubernetes environments on demand without the cost or lag of full cluster provisioning.

This post adapts that general architecture with Pulumi to orchestrate Amazon EKS Auto Mode and vCluster.

The problem: environment sprawl and provisioning lag

Traditional development workflows often rely on one full EKS cluster per developer or feature branch. While this provides strong isolation, it introduces major pain points. Provisioning a full cluster can take 15 minutes or more, which slows down CI/CD pipelines. Managing dozens of clusters also leads to high costs and significant operational overhead.

Platform teams need a “soft multi-tenancy” model. This model should feel like a dedicated cluster to the developer but run on shared infrastructure to keep costs low and startup times fast.

Architecture overview: the host and the tenants

The environment factory architecture consists of two main layers.

Host cluster: A single, reliable EKS cluster managed with EKS Auto Mode. This cluster provides the underlying compute, networking, and storage.
Tenant environments: Virtual clusters (vCluster) running as pods within host namespaces.

According to the vCluster architecture, the virtual control plane handles API requests while a syncer maps virtual resources to the host cluster. This separation allows tenants to manage their own CRDs, namespaces, and RBAC while platform teams use quotas, NetworkPolicies, pod security, IAM boundaries, and node isolation controls to protect the host and other tenants.

Implementation: the EKS Auto Mode host

EKS Auto Mode simplifies the host cluster by automating infrastructure management. It handles node provisioning, scaling, and updates based on pod requirements.

The following snippet shows how to define an EKS cluster with Auto Mode enabled using Pulumi.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import { SubnetType } from "@pulumi/awsx/ec2";

const clusterName = "environment-factory";

const vpc = new awsx.ec2.Vpc("environment-factory", {
 enableDnsHostnames: true,
 cidrBlock: "10.0.0.0/16",
 subnetSpecs: [
 {
 type: SubnetType.Public,
 tags: {
 [`kubernetes.io/cluster/${clusterName}`]: "shared",
 "kubernetes.io/role/elb": "1",
 },
 },
 {
 type: SubnetType.Private,
 tags: {
 [`kubernetes.io/cluster/${clusterName}`]: "shared",
 "kubernetes.io/role/internal-elb": "1",
 },
 },
 ],
 subnetStrategy: "Auto",
});

// Create an EKS cluster with Auto Mode enabled.
const hostCluster = new eks.Cluster("host-cluster", {
 name: clusterName,
 authenticationMode: eks.AuthenticationMode.Api, // Use API authentication mode for EKS access entries.
 vpcId: vpc.vpcId,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 autoMode: {
 enabled: true,
 },
});

const hostProvider = new k8s.Provider("host-provider", {
 kubeconfig: hostCluster.kubeconfig,
});

Implementation: the environment factory

Once the host cluster is ready, we can build the factory that stamps out tenant environments. Each tenant needs a dedicated namespace, resource quotas, and the vCluster itself.

Tenant guardrails

Before installing vCluster, we set up a namespace and resource quotas to ensure one tenant cannot consume all host resources.

import * as k8s from "@pulumi/kubernetes";

// Define a tenant namespace on the host cluster.
const tenantNamespace = new k8s.core.v1.Namespace("tenant-alpha", {
 metadata: { name: "tenant-alpha" },
}, { provider: hostProvider });

// Apply resource quotas to the tenant namespace.
const quota = new k8s.core.v1.ResourceQuota("tenant-quota", {
 metadata: { namespace: tenantNamespace.metadata.name },
 spec: {
 hard: {
 pods: "20",
 "requests.cpu": "4",
 "requests.memory": "8Gi",
 "limits.cpu": "8",
 "limits.memory": "16Gi",
 },
 },
}, { provider: hostProvider });

// Define a Role for the tenant within their namespace.
const tenantRole = new k8s.rbac.v1.Role("tenant-role", {
 metadata: { namespace: tenantNamespace.metadata.name },
 rules: [{
 apiGroups: [""],
 resources: ["pods", "services", "configmaps", "secrets"],
 verbs: ["get", "list", "watch", "create", "update", "patch", "delete"],
 }],
}, { provider: hostProvider });

// Bind the Role to a tenant user or group.
const tenantRoleBinding = new k8s.rbac.v1.RoleBinding("tenant-role-binding", {
 metadata: { namespace: tenantNamespace.metadata.name },
 subjects: [{
 kind: "User",
 // Replace "tenant-user" with the IAM-mapped user or group for this tenant.
 name: "tenant-user",
 apiGroup: "rbac.authorization.k8s.io",
 }],
 roleRef: {
 kind: "Role",
 name: tenantRole.metadata.name,
 apiGroup: "rbac.authorization.k8s.io",
 },
}, { provider: hostProvider });

For production use, map these Kubernetes identities to IAM principals using EKS Access Entries, with the legacy aws-auth ConfigMap still appearing in older clusters.

Deploying vCluster with Helm

We use the kubernetes.helm.v3.Release resource to install vCluster. This resource provides controlled Helm lifecycle management for the vCluster release. The values block should be adjusted for each tenant profile to control resource synchronization and control plane behavior. Review the vCluster release notes when changing chart versions because values schema and generated secret names can change across releases.

import * as k8s from "@pulumi/kubernetes";

// Install vCluster using the Helm Release resource.
const vcluster = new k8s.helm.v3.Release("vcluster-alpha", {
 name: "vcluster-alpha",
 chart: "vcluster",
 version: "0.20.0", // Tested with vCluster 0.20.x; review release notes before changing versions.
 repositoryOpts: {
 repo: "https://charts.loft.sh",
 },
 namespace: tenantNamespace.metadata.name,
 values: {
 // Explicit sync configuration; adjust per tenant profile.
 sync: {
 toHost: {
 pods: { enabled: true },
 },
 },
 },
}, { provider: hostProvider });

Accessing the virtual cluster

The vCluster generates a kubeconfig that allows developers to interact with the virtual API server. We must treat this kubeconfig as a secret, and the endpoint in that kubeconfig must be reachable from the Pulumi runner before using it to create resources inside the virtual cluster.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Retrieve the vCluster kubeconfig from the generated secret.
// The vCluster-generated secret can lag behind Helm release readiness on first creation,
// so teams may choose an explicit readiness check or rerun after the virtual control plane initializes.
const vclusterKubeconfig = k8s.core.v1.Secret.get("vcluster-secret",
 pulumi.interpolate`${tenantNamespace.metadata.name}/vc-vcluster-alpha`,
 {
 provider: hostProvider,
 dependsOn: [vcluster],
 }
).data.apply(data => Buffer.from(data["config"], "base64").toString());

// Export the kubeconfig as a secret.
export const tenantKubeconfig = pulumi.secret(vclusterKubeconfig);

// Create a provider for the virtual cluster using the secret kubeconfig.
const vclusterProvider = new k8s.Provider("vcluster-provider", {
 kubeconfig: tenantKubeconfig,
});

Provision environments on demand

Model your host cluster, vCluster tenants, and guardrails in Pulumi so your platform team can hand developers isolated test environments and clean them up as code.

Get started

Operational caveats

RBAC and permissions: vCluster generates default RBAC rules that work for most scenarios. However, if your host cluster is heavily locked down, you may need to provide additional permissions to the vCluster service account.
Helm release previews: When using kubernetes.helm.v3.Release, Pulumi previews may not show every detail of the rendered Kubernetes resources. It primarily tracks the state of the Helm release itself.
EKS Auto Mode node lifetime: EKS Auto Mode uses immutable AMIs and has a 21-day node lifetime. Kubernetes reschedules vCluster pods and tenant workloads when nodes are replaced, so configure replicas, PodDisruptionBudgets, requests, and persistent storage for disruption tolerance.

Conclusion: ephemeral environments at scale

By combining Pulumi with EKS Auto Mode and vCluster, you can build a scalable environment factory. This approach provides the isolation developers need while maintaining the speed and cost-efficiency required by platform teams.

The snippets provided here are adapted for illustration. In a production environment, you would likely wrap these resources into a Pulumi ComponentResource to provide a clean, reusable API for your internal developers. When a feature branch is merged, deleting the Pulumi stack removes the resources managed by that stack, but validate namespace finalizers, persistent volume reclaim policies, and external cloud artifacts as part of cleanup.

For more on managing EKS with Pulumi, see the EKS guide.

KubeCon EU 2026 Recap: The Year AI Moved Into Production on Kubernetes

Engin Diri — Wed, 01 Apr 2026 00:00:00 -0700

Amsterdam in late March still has that sharp North Sea wind, but inside the RAI Convention Centre, 13,350 people generated enough energy to heat the building twice over. KubeCon + CloudNativeCon EU 2026 was the biggest European edition yet, and the shift from previous years was impossible to miss. AI dominated the conference.

I spent most of the conference at the Pulumi booth, and that turned out to be the best vantage point. Hundreds of visitors stopped by over four days, and I kept asking the same question: what are you actually running in production with AI on Kubernetes? The answers shaped this post more than any keynote did. Almost everyone had a proof of concept. Almost nobody had a production story they were happy with.

Here is the stat that framed the entire conference for me: 66% of organizations use Kubernetes to host generative AI workloads, but only 7% deploy to production daily. That gap between experimentation and actual production use matched what I was hearing at the booth. The CNCF’s own survey now counts 19.9 million cloud native developers worldwide, 7.3 million of them building AI workloads. The tooling and the infrastructure need to catch up.

My takeaway after four days on the ground: lots of working demos, very few production setups people trust. Teams are trying to scale inference, put guardrails around agents, and make GPU infrastructure behave like anything else they run.

Here is what I saw.

From training to inference: the big pivot

About 67% of AI compute now goes to inference, not training. The inference market is projected to hit $255 billion by 2030. It’s also where most of the operational complexity lives.

NVIDIA leaned into this hard. Their open-source stack around NeMo and Dynamo got significant stage time, but the bigger move was donating three projects to the CNCF: the DRA driver for fractional GPU allocation, the KAI Scheduler for GPU-aware scheduling, and Grove. Moving these to community governance signals that GPU infra is becoming part of the standard Kubernetes toolkit.

The CNCF donations that will reshape AI on Kubernetes

Every KubeCon has its crop of new CNCF projects, but this year’s batch felt different. We are starting to see the building blocks of an AI runtime for Kubernetes.

llm-d was the headline donation. Created by IBM Research, Red Hat, and Google Cloud, it splits inference workloads by separating prefill and decode phases across different pods. The collaborator list reads like an industry consortium: NVIDIA, CoreWeave, AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley, and UChicago. When that many organizations agree on a single approach to distributed inference, pay attention.

NVIDIA’s DRA driver enables fractional GPU allocation and multi-node NVLink support. GPU multi-tenancy is one of the hardest unsolved problems in Kubernetes right now. Scheduling, isolation, cost attribution — all of it breaks down when multiple workloads share a GPU. The DRA driver does not solve everything, but it gives the community a real starting point.

KAI Scheduler entered the CNCF Sandbox for GPU-aware scheduling. If llm-d handles the inference runtime and the DRA driver handles allocation, KAI Scheduler handles placement. Together, these three projects form the skeleton of a GPU-native Kubernetes stack.

Velero, donated by Broadcom, moved into CNCF Sandbox for backup and restore. AI workloads are stateful now (model weights, checkpoints, fine-tuning data), and backup is no longer optional. Good timing.

Microsoft AI Runway is an open-source Kubernetes API for inference that plugs in Hugging Face model discovery, GPU memory fit calculations, and cost estimates. Think of it as a model-aware control plane. HolmesGPT and Dalec, also from Microsoft, entered CNCF Sandbox for AI-powered troubleshooting and dependency analysis.

The Kubernetes AI Conformance Program is growing fast, with certifications nearly doubled and three new requirements proposed for Kubernetes 1.36. Conformance programs are boring until they are not. This one will determine which distributions can credibly claim AI readiness.

Agentic AI gets an identity layer

If inference was this year’s production story, agentic AI was the architecture story. Agents are proliferating, and nobody has quite figured out how to manage and secure them inside Kubernetes yet.

kagent, donated to CNCF Sandbox by Solo.io, defines agents as Kubernetes CRDs. It ships with pre-built MCP (Model Context Protocol) servers for Kubernetes, Istio, Helm, Argo, Prometheus, Grafana, and Cilium. An agent becomes a first-class Kubernetes resource, schedulable and observable and subject to RBAC, instead of a rogue process running in someone’s notebook.

kagenti from IBM goes after the identity problem directly. Using SPIFFE/SPIRE, it gives agents cryptographic identities. When an agent calls an API, you can verify exactly which agent made the call, what trust domain it belongs to, and whether it is authorized. This kind of security work needs to happen before agents proliferate across production clusters. Retrofitting identity later is ugly.

Dapr Agents took a different angle with the actor model and durable execution. Each agent gets reliable state management and exactly-once messaging semantics. If your workflows cannot tolerate lost messages or duplicate actions, this matters.

agentregistry showed up as a centralized discovery service for MCP servers and agents. As agents and tool servers multiply, you need a registry to find and manage them, the same way container registries became necessary for images.

David Soria Parra from Anthropic gave a talk on MCP evolving beyond simple tool-calling into richer interaction patterns (sched). Google announced the Kubernetes Agent Sandbox for running agentic AI workloads in secure, isolated environments.

AI gateways and inference routing

Gateway infrastructure had its own mini-conference within KubeCon. The Gateway API Inference Extension from the Kubernetes SIG introduces model-aware routing and load balancing at the gateway level. Instead of routing by URL path, your gateway routes by model name, version, and capacity. That changes how inference traffic flows through a cluster in a fundamental way.

Envoy AI Gateway builds on Envoy’s existing proxy capabilities with token-aware rate limiting and provider failover. If your primary inference provider is saturated, traffic shifts to a secondary automatically. Rate limiting by token count rather than request count makes much more sense for LLM workloads, where a single request can consume vastly different amounts of compute.

I want to call out Agentgateway specifically. Written in Rust, it proxies LLM traffic, MCP connections, and agent-to-agent communication, with Cedar and CEL policy engines for fine-grained access control. Rust’s performance characteristics matter here because inference gateway latency adds directly to user-perceived response time.

Kuadrant, now in CNCF Sandbox, layers policy on top of gateway infrastructure and includes MCP server aggregation. Gateways are evolving from dumb traffic proxies into intelligent control planes for AI workloads, and these four projects are driving that shift.

Platform engineering absorbs LLMOps

The observability and platform engineering vendors showed up in force. The message was consistent: LLMOps is just platform engineering with new requirements.

Chronosphere demonstrated parallel AI investigation, with multiple agents analyzing different aspects of an incident simultaneously and combining their findings. SUSE Liz takes a domain-specialized approach, deploying different AI agents for different operational domains rather than one general-purpose assistant. groundcover combines eBPF with OpenTelemetry to give coding agents rich runtime context about the systems they are modifying. That last one is subtle but important: if an AI agent is writing code that touches a service, it should understand that service’s actual runtime behavior, not just its source code.

Dynatrace and DevCycle partnered to make feature flags observable primitives via OpenFeature. Rolling out AI features behind feature flags is table stakes, but having those flags show up in your observability pipeline as first-class signals closes a real gap.

Shadow AI governance emerged as its own theme. CAST AI’s Kimchi can route requests across 50+ models while providing centralized visibility into what models are being used, by whom, and at what cost. Every large organization I talked to had some version of the same problem: teams spinning up model endpoints without central oversight, burning through GPU budgets, creating compliance blind spots they did not even know about.

GPU multi-tenancy remains genuinely unsolved. Scheduling, workload isolation, cost attribution across shared GPUs — all of it breaks down at scale. Multiple talks addressed pieces of this, but nobody had a complete answer.

Sovereignty shapes infrastructure architecture

Regulation came up in almost every conversation. The EU Cyber Resilience Act is driving compliance requirements deep into software supply chains, and every European organization I spoke with is feeling the pressure. Teams are already changing how they build and deploy software.

Sovereign Kubernetes is a platform architecture requirement now, not something you can defer to next quarter. Organizations need Kubernetes distributions and cloud regions that guarantee data residency, and they need the tooling to enforce those guarantees programmatically. Self-hosted models are proliferating partly because of capability and cost, but data sovereignty is the accelerant. If your data cannot leave a jurisdiction, neither can your model.

Runtime isolation is expanding beyond containers. Several talks covered KVM-based isolation for AI workloads, which is heavier than containers but necessary when the threat model includes side-channel attacks on shared GPU memory. The sandboxing conversation has gotten more sophisticated since last year.

These constraints are not uniquely European. Any organization operating across jurisdictions faces similar pressures, and the regulatory direction globally is toward more data sovereignty requirements, not fewer.

What this means for your team

Four days in Amsterdam distilled into five things I would act on now:

Treat inference workloads like production services. If you are still deploying models with scripts and hope, stop. Inference infrastructure needs the same IaC discipline as any other production system: version-controlled, tested, policy-enforced.
Evaluate the Gateway API Inference Extension and llm-d. These are not speculative projects. They have broad industry backing and solve real problems around inference routing and distributed serving. Get them into your test environments.
Plan agent identity before agents proliferate. SPIFFE/SPIRE for agent identity is not optional if you are running agents in production. Retrofitting identity onto an existing agent fleet is painful. Start with kagenti now.
Platform teams should own AI infrastructure. Shadow AI is already happening in your organization. The platform engineering team needs to provide self-service AI infrastructure with guardrails before ungoverned model endpoints become a security and cost problem.
Sovereignty and GPU multi-tenancy are universal. Even if you are not subject to the EU Cyber Resilience Act today, data residency requirements are spreading globally. GPU multi-tenancy will affect every organization running inference at scale.

Kubernetes spent the past decade proving it could orchestrate containers. The next decade will test whether it can orchestrate intelligence. Based on what I saw in Amsterdam, the community is building the right pieces, but the gap between what exists and what production demands is still wide. That gap is where the interesting work happens.

From Kubernetes Gatekeeper to Full-Stack Governance with OPA

Levi Blackstone — Thu, 19 Mar 2026 00:00:00 +0000

Pulumi’s OPA (Open Policy Agent) support is now stable. The v1.1.0 release of pulumi-policy-opa makes OPA/Rego a first-class policy language for Pulumi with full feature parity alongside the native TypeScript and Python policy SDKs. Write Rego policies that validate any resource Pulumi manages, across AWS, Azure, GCP, Kubernetes, and the rest of the provider ecosystem. If you already have Kubernetes Gatekeeper constraint templates, a new compatibility mode lets you drop those .rego files directly into a Pulumi policy pack and enforce them against your Kubernetes resources without modification.

What’s in the stable release

OPA/Rego is now fully supported as a policy language for Pulumi Insights, with the same capabilities as the TypeScript and Python SDKs:

Resource and stack-level policies: Validate individual resources with deny and warn rules, or evaluate your entire stack at once with stack_deny and stack_warn for cross-resource checks like relationship validation and resource count limits.
Enforcement levels: Control how violations are handled. mandatory blocks deployments, advisory surfaces warnings, and disabled turns rules off without removing them. Enforcement levels can be overridden per policy without modifying Rego source.
Policy configuration: Pass custom parameters to policies via configuration files, with optional JSON schema validation. Configuration values are accessible in Rego as data.config.<policy_name>.<key>.
OPA metadata annotations: Use standard OPA # METADATA comments to provide titles, descriptions, and messages for your policies. These populate the policy metadata displayed in Pulumi Cloud.
Preventative and audit evaluation: OPA policies work with both preventative enforcement during pulumi up and audit policy scans for continuous compliance monitoring.

You can choose whichever language best fits your team. Organizations already using OPA across their toolchain can standardize on Rego for Pulumi policies, while teams preferring TypeScript or Python can continue to use those. All three languages work side by side in the same policy groups.

Kubernetes Gatekeeper compatibility

The headline feature of this release is native support for Kubernetes Gatekeeper constraint template rules. If you’re running Gatekeeper as an admission controller in your clusters, you likely have a library of .rego policies that enforce security and operational standards at admission time. With v1.1.0, those same rules can now run as Pulumi policies, catching violations during pulumi preview before resources ever reach the cluster.

To enable Gatekeeper compatibility, set inputFormat: kubernetes-admission in your PulumiPolicy.yaml:

description: Kubernetes Gatekeeper Policy Pack
runtime: opa
inputFormat: kubernetes-admission

With this setting, Pulumi automatically wraps Kubernetes resources in the Gatekeeper AdmissionReview structure (input.review.object, input.review.kind, etc.), so your existing rules work without modification. Non-Kubernetes resources are silently skipped.

Here’s an example that reuses standard Gatekeeper-style rules, requiring an app label and prohibiting the latest image tag:

package gatekeeper

import rego.v1

# METADATA
# title: Require App Label
# description: All Kubernetes resources must have an "app" label.
violation contains {"msg": msg} if {
 not input.review.object.metadata.labels["app"]
 msg := sprintf("%s '%s' is missing required label: app",
 [input.review.kind.kind, input.review.name])
}

# METADATA
# title: Disallow Latest Tag
# description: Container images must not use the "latest" tag.
deny contains msg if {
 container := input.review.object.spec.template.spec.containers[_]
 endswith(container.image, ":latest")
 msg := sprintf("container '%s' uses the 'latest' tag -- pin to a specific version",
 [container.name])
}

These rules are identical to what you’d write for Gatekeeper. Both rule head formats are supported and can coexist: the violation[{"msg": msg}] map format and the deny[msg] string format. Per-policy configuration via input.parameters also works as expected. You can take a .rego file from your Gatekeeper constraint templates, drop it into a Pulumi policy pack, and publish it to Pulumi Cloud to enforce automatically across your stacks.

This shifts policy enforcement left. Instead of waiting for the Kubernetes API server to reject a resource at admission time, you catch the violation during pulumi preview, before anything is deployed.

Walkthrough: Reusing policies from the gatekeeper-library

The OPA Gatekeeper Library is a community-maintained collection of constraint templates covering common Kubernetes guardrails like pod security, image provenance, and resource limits. You can use these policies directly with Pulumi. Here’s an end-to-end example using the allowedrepos policy to restrict which container image registries your deployments can use.

Create a new Kubernetes OPA policy pack:
```
pulumi policy new kubernetes-opa
```

Copy the Rego source from gatekeeper-library into your policy pack as allowedrepos.rego. No modifications are needed:

package k8sallowedrepos

violation[{"msg": msg}] {
 container := input.review.object.spec.containers[_]
 not strings.any_prefix_match(container.image, input.parameters.repos)
 msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v",
 [container.name, container.image, input.parameters.repos])
}

violation[{"msg": msg}] {
 container := input.review.object.spec.initContainers[_]
 not strings.any_prefix_match(container.image, input.parameters.repos)
 msg := sprintf("initContainer <%v> has an invalid image repo <%v>, allowed repos are %v",
 [container.name, container.image, input.parameters.repos])
}

violation[{"msg": msg}] {
 container := input.review.object.spec.ephemeralContainers[_]
 not strings.any_prefix_match(container.image, input.parameters.repos)
 msg := sprintf("ephemeralContainer <%v> has an invalid image repo <%v>, allowed repos are %v",
 [container.name, container.image, input.parameters.repos])
}

Verify that your PulumiPolicy.yaml has Gatekeeper compatibility enabled:

description: Kubernetes Gatekeeper Policy Pack
runtime: opa
inputFormat: kubernetes-admission

Configure the allowed registries. Create a policy-config.json file to pass the repos parameter:

{
 "k8sallowedrepos": {
 "repos": ["gcr.io/my-company/", "docker.io/library/"]
 }
}

Test the policy locally against a stack:
```
pulumi preview --policy-pack . --policy-pack-config policy-config.json
```
Any Kubernetes deployment using an image outside the allowed registries will produce a violation at preview time, before it reaches the cluster.
Publish the pack and add it to a policy group to enforce it across your organization:
```
pulumi policy publish
```

The same approach works for any policy in the gatekeeper-library: containerlimits, requiredlabels, disallowedtags, and others. Copy the Rego, configure parameters, and publish.

Part of the Pulumi Insights governance story

OPA policy support is part of the broader Pulumi Insights governance platform. Insights gives you visibility and compliance across your entire cloud footprint, and OPA policies plug directly into that:

Audit policy scans continuously evaluate OPA policies against your Pulumi stacks and discovered cloud resources, providing a compliance baseline without redeploying anything.
Self-hosted execution lets you run policy evaluations on your own infrastructure using customer-managed workflow runners, keeping credentials and data within your network.
Pre-built compliance packs for CIS, NIST, PCI DSS, and other frameworks are available alongside your custom OPA policies in the same policy groups.

Whether you’re enforcing policy at deployment time, scanning existing infrastructure for drift, or running continuous compliance checks, OPA policies are a native participant.

Frequently asked questions

Do I need to modify my existing Gatekeeper `.rego` files?

No. Set inputFormat: kubernetes-admission in your PulumiPolicy.yaml and your existing Gatekeeper constraint template rules work as-is. Pulumi handles the AdmissionReview wrapping automatically.

What happens with non-Kubernetes resources?

When using inputFormat: kubernetes-admission, non-Kubernetes resources are silently skipped during evaluation. Your Gatekeeper rules only run against Kubernetes resources.

Do I need OPA installed locally?

No. The pulumi-policy-opa analyzer plugin embeds the OPA evaluation engine and is installed automatically by the Pulumi CLI (v3.227.0+). The standalone OPA CLI is only needed if you want to run opa test against your policies independently.

When should I use OPA vs. TypeScript or Python for policies?

If your team already writes Rego for other tools like Gatekeeper, writing Pulumi policies in Rego keeps your policy language consistent. If your team is more comfortable with general-purpose languages or needs auto-remediation, use the TypeScript or Python SDKs.

Gatekeeper constraint templates can be reused directly via the kubernetes-admission input format, but other OPA integrations use different input structures, so those policies would need to be adapted to Pulumi’s resource model. All three languages work together in the same policy groups.

Get started

Templates are available for kubernetes-opa, aws-opa, azure-opa, and gcp-opa via pulumi policy new. For more details, see the policy authoring guide and the Policy as Code overview.

Get started with OPA policies

github.com/pulumi/pulumi-policy-opa

GitOps Best Practices I Wish I Had Known Before

Engin Diri — Thu, 19 Feb 2026 00:00:00 +0000

Getting started with GitOps can feel like trying to herd cats through a YAML factory while the factory is on fire. It’s one of those things that seems like it ought to be simple (just use Git!), but in practice is much more complex — and you may not realize how much more complex until you’re weeks or more into a project. After years of running GitOps workflows in production across dozens of clusters, I’ve collected a list of best practices that I’m hoping can save you from having to make many of the mistakes I’ve made. Think of it as the GitOps cheat sheet I wish I’d had from Day 1.

If you’re not familiar with the formal definition, the OpenGitOps project distills it into four principles:

Declarative desired state
Versioned and immutable storage
Automatic pulling
Continuous reconciliation

But those principles only define the what. This post is also about the how — the practical lessons that can make or break a GitOps implementation.

In this post, I’ll walk you through the GitOps best practices I’ve picked up from production experience, community talks, and more than a few late-night incident calls. Whether you’re just getting started with GitOps or looking to level up, these tips should help you avoid the potholes.

The GitOps mindset in a nutshell.

1. Git is your single source of truth (no, really)

This is the bedrock that the principles of GitOps are built on: every piece of your environment’s desired state lives in a Git repository. No exceptions. No “I’ll just fix it real quick with kubectl edit.” No “let me patch this configmap by hand because the PR process takes too long.”

The moment you make a manual change to your cluster, you’ve created drift between what Git says the world should look like and what it actually looks like. Drift will quietly wreck your GitOps workflows if you let it.

Put everything in Git: Kubernetes manifests, Helm values, Kustomize overlays, policy rules, even your GitOps tool configuration itself.
No manual kubectl edits. If it’s not in Git, it doesn’t exist. Period. Train your team to treat direct cluster changes like touching a hot stove.
You get an audit trail for free. Git gives you a complete history of who changed what, when, and why. That’s your compliance audit trail baked right in.

Pro Tip: Enable branch protection rules on your GitOps repos from Day 1. This prevents anyone from pushing directly to main and bypassing the review process. Future you will thank present you during the next audit.

2. Declarative over imperative, always

If you’re still running sequences of kubectl create, kubectl patch, and kubectl delete commands to manage your cluster, you’re not really doing GitOps yet. Declarative means you define what you want the end state to look like, not the step-by-step instructions to get there.

Think of it like ordering at a restaurant. Imperative: “Go to the kitchen, grab flour, knead dough, preheat the oven to 200 degrees, shape the pizza, add sauce…” Declarative: “One margherita pizza, please.” Let the system figure out how to make it happen.

Your manifests describe the end result. The GitOps operator reconciles reality to match.
It’s idempotent by design. Apply the same manifest ten times, get the same result. No side effects, no surprises.
Rollbacks are easier: just revert a commit. The operator sees the previous desired state and reconciles. A caveat though: this only works for stateless resources; database schema migrations, CRD version changes, persistent volume modifications, and rotated secrets don’t always revert cleanly by rolling back to a previous commit. Be sure to plan rollbacks involving stateful resources carefully.

Pro Tip: If you find yourself writing shell scripts that run a sequence of kubectl commands, stop and ask yourself: “Can I express this as a declarative manifest instead?” Nine times out of ten, the answer is yes.

3. Pull-based deployments are the way

Traditional CI/CD is push-based: your pipeline builds an artifact and then pushes it to the cluster. GitOps flips this. An agent running inside your cluster continuously polls a Git repository and pulls changes when it detects them.

Why does this matter? With push-based deployments, your CI system needs credentials to access your cluster. That’s a wide attack surface. With pull-based, the agent already lives inside the cluster and only needs read access to your Git repo.

Tools like ArgoCD and FluxCD run controllers inside your cluster that watch your Git repos.
Your CI pipeline never needs kubeconfig access. The agent handles deployment.
The agent doesn’t just deploy once; it continuously ensures the cluster matches the desired state in Git.

Pro Tip: You may be tempted to set your reconciliation interval to something aggressive like 1 minute so you always know exactly which version is deployed. That works for a while, but at scale (200+ applications polling every minute) you can blow through GitHub’s API rate limits (5,000 requests/hour for authenticated users) and put real pressure on the Kubernetes API server.

A better approach: set up webhook receivers. Both ArgoCD and FluxCD support incoming webhooks from GitHub, GitLab, and Bitbucket. Your Git provider pings the GitOps controller on every push, so reconciliation kicks off in seconds instead of waiting for the next poll. That alone kills most of the API rate limit pain at scale. You can then relax polling to a 5 or 10 minute fallback for the rare case where a webhook doesn’t fire.

4. Separate app code from deployment config (when it hurts not to)

Let me be upfront: if you’re a small team with one or two services, keeping your Kubernetes manifests next to your application source code in the same repo is fine. A monorepo is simpler to manage and one less thing to automate. Don’t split repos just because a blog post told you to.

That said, ArgoCD’s official best practices call separate repos “highly recommended,” and there are real reasons for that once you grow. Application code and deployment configuration have different lifecycles. You might bump resource limits in a Helm values file without touching a single line of app code. Or you might refactor your entire codebase without changing any deployment parameters. In a shared repo, every config tweak triggers your full CI pipeline, and ArgoCD invalidates the manifest cache for all applications in the repo on every commit, not just the ones that changed. That cache invalidation alone can become a performance problem with dozens of apps in one repo.

When config tweaks start blocking app code from getting through CI, it may be time to split.
When different teams need different access levels (not everyone who pushes app code should modify production deployment config), it’s time to split.
When you’re running microservices with independent release cadences that step on each other, it’s time to split.

If none of those apply yet, don’t split. You’ll know when the pain arrives.

Pro Tip: When you do split, a common pattern is two repos: one for application source code, one for deployment configuration (Helm charts, Kustomize overlays, environment-specific values). Some larger orgs go further with a third repo for environment overrides, following the fleet repo model from the FluxCD maintainers. Either way, pair it with image update automation (Flux Image Automation Controller or ArgoCD Image Updater) so image tag bumps across repos don’t turn into manual toil.

5. Use directories, not branches, for environments

This is the number one anti-pattern I encounter in the wild. Teams create a dev branch, a staging branch, and a prod branch, then cherry-pick or merge between them for promotions. It sounds logical, but it’s a trap. Branches diverge, cherry-picks get missed, configs meant for dev sneak into staging or prod, and before you know it, your environments have drifted, and you can’t tell what’s actually different between them.

A folder structure like environments/dev/, environments/staging/, environments/prod/ makes differences visible in a single diff command.
Moving a change from dev to prod is just copying or updating files across directories, reviewed via a standard pull request.
You’ll never accidentally skip a commit or introduce an environment-specific change where it doesn’t belong. No more cherry-pick roulette.

One thing to note: the commonly used rendered manifests pattern actually does use per-environment branches, but not in a human-managed way. In that pattern, config changes are still committed to main (using named directories as above), and an automated CI process pushes environment YAML to read-only environment branches, which ArgoCD reads from. If you want to try it, ArgoCD’s built-in source hydrator and Kargo Render can automate the rendering for you.

To keep things DRY, you can look to a tool like Kustomize, which lets you use a shared base environment and then apply per-environment patches. Your base directory holds the common manifests, and each environment directory contains only the differences. For example:

gitops-repo/
├── base/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
├── environments/
│ ├── dev/
│ │ ├── kustomization.yaml # replicas: 1, limits: 256Mi
│ │ └── patches/
│ ├── staging/
│ │ ├── kustomization.yaml # replicas: 2, limits: 512Mi
│ │ └── patches/
│ └── prod/
│ ├── kustomization.yaml # replicas: 3, limits: 1Gi
│ └── patches/

This minimizes duplication while keeping environment boundaries clear.

Pro Tip: If you’re working at the IaC layer, Pulumi stack configuration solves a similar problem: each stack (dev, staging, prod) has its own config file with environment-specific values, while the program logic stays shared. It’s the same principle of “one source, per-environment overrides” applied above the Kubernetes manifest level.

6. Validate before you merge

Catching errors after they’ve been applied to your cluster is expensive. Catching them in CI before the merge is cheap. Shift left as hard as you can.

Your GitOps CI pipeline should validate everything it can before the change reaches your cluster: YAML syntax, Kubernetes schema validation, policy compliance, dry-run rendering.

Tools like yamllint and kubeconform catch syntax errors and schema violations before they become runtime failures. (Note: kubeval, which you’ll see in older guides, is no longer maintained — its own README points to kubeconform as the replacement.)
Run helm template or kustomize build in CI to verify that your templates render without errors.
Use OPA/Conftest or Kyverno to enforce organizational policies (no privileged containers, required labels, resource limits set) before merge.

Here’s a minimal GitHub Actions workflow that validates your GitOps manifests on every PR:

name: Validate GitOps Manifests
on:
 pull_request:
 paths: ["environments/**"]

jobs:
 validate:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - name: Validate YAML schemas
 run: kubeconform --strict environments/**/*.yaml
 - name: Build and verify Kustomize output
 run: |
 for env in environments/*/; do
 kustomize build "$env" > /dev/null
 done
 - name: Diff against live cluster
 run: |
 kustomize build environments/staging/ | \
 kubectl diff -f - || true

Pro Tip: Add a kubectl diff step in your CI pipeline that shows exactly what would change in the cluster if the PR were merged. This gives reviewers a concrete view of the impact, not just the YAML diff.

7. Tag with commit SHAs, not “latest”

Using latest as your image tag in a GitOps workflow is playing fast and loose with your deployments. You have no idea what version is actually running, you can’t reliably roll back, and your Git history becomes meaningless because the same manifest could deploy completely different code depending on when it was synced.

Tag your container images with the Git commit SHA that produced them. This creates a direct link between your source code, your build, and your deployment.
Once a tag is pushed, it should never be overwritten. No re-tagging, no “oops let me push a fix to the same tag.”
Need to roll back? Just revert the commit that updated the image tag. The previous SHA still points to the exact same image it always did.

Pro Tip: Set up an admission controller or CI policy that rejects any manifest using latest or any other mutable tag. Make it impossible to deploy without a pinned version.

8. Automate drift detection and reconciliation

Drift is when your cluster’s actual state doesn’t match the desired state in Git. It happens when someone runs a manual kubectl command, when an autoscaler changes a replica count, or when a CRD controller modifies a resource behind your back.

The whole point of GitOps is continuous reconciliation. Your operator should constantly compare the live state against Git and bring things back in line.

Configure your GitOps tool to automatically revert manual changes, but build your exclusion lists (the set of fields and resources you intentionally skip during reconciliation) before you turn on auto-sync, not after. Controllers like Istio, cert-manager, and Crossplane legitimately modify resources, and auto-remediating those changes creates reconciliation loops that can destabilize a cluster. Start with a conservative exclusion list and tighten it over time.
Even with auto-remediation, you want to know when drift happens. Set up alerts. It might indicate a process problem or a team member who needs coaching.

Pro Tip: Be precise about what you reconcile. HPAs, VPAs, cert-manager annotations, and external-dns records all legitimately modify resources. Use ignore rules to exclude fields that are intentionally dynamic. In ArgoCD, that looks like this:

spec:
 ignoreDifferences:
 - group: apps
 kind: Deployment
 jsonPointers:
 - /spec/replicas
 - group: admissionregistration.k8s.io
 kind: MutatingWebhookConfiguration
 jqPathExpressions:
 - .webhooks[]?.clientConfig.caBundle

Also, don’t overlook resource ordering. ArgoCD sync waves and Flux’s dependsOn let you enforce that CRDs are applied before custom resources and namespaces before workloads. Getting ordering wrong is one of the most common causes of failed syncs in production.

9. Practice progressive delivery

Progressive delivery lets you gradually roll out changes, verify they’re working, and automatically roll back if something goes wrong.

GitOps and progressive delivery fit well together. Your Git repo describes the desired rollout strategy, and controllers in the cluster execute it.

Canary deployments: send a small percentage of traffic to the new version. If error rates spike, automatically roll back.
Blue-green deployments: run two identical environments, switch traffic once the new one is verified healthy.
Argo Rollouts extends Kubernetes with canary, blue-green, and analysis-driven rollout strategies that integrate with ArgoCD.
Flagger is the Flux ecosystem equivalent, with automated canary analysis, A/B testing, and blue-green deployments using service mesh or ingress controllers.
Kargo takes it a step further by orchestrating promotions across multiple stages and environments. Instead of manually wiring up pipelines to move changes from dev to staging to prod, Kargo automates the entire promotion workflow on top of ArgoCD. I did a deep dive on GitOps promotion tools and why they belong in your toolkit:

Pro Tip: Combine progressive delivery with automated analysis. Tools like Argo Rollouts and Kargo can query Prometheus metrics during a canary and automatically promote or roll back based on error rates, latency percentiles, or custom metrics.

10. Policy-as-code: your automated guardrails

Reviews are great, but humans make mistakes. Policy-as-code adds an automated layer that enforces organizational rules before a change can land in your cluster. Think of it as guardrails on a mountain road: they don’t slow you down, they just keep you from going over the edge.

OPA/Gatekeeper lets you define policies in Rego that the admission controller enforces at deploy time (e.g., no containers running as root, all deployments must have resource limits).
Kyverno is a Kubernetes-native policy engine that uses familiar YAML to define and enforce policies, including mutation and generation.
Starting with Kubernetes 1.26+, you can write CEL-based ValidatingAdmissionPolicies natively without any external tooling. A lightweight option if you don’t need the full feature set of OPA or Kyverno.
Pulumi CrossGuard lets you write policy packs in TypeScript or Python that validate infrastructure before it’s deployed. If your IaC and GitOps layers are bridged (see the next section), CrossGuard can enforce rules across both.
Run policy checks in your CI pipeline so violations are caught during the PR, not after deployment.

Pro Tip: Start with a small set of high-impact policies (required labels, no privileged containers, resource limits) and expand over time. Trying to enforce 50 policies on Day 1 will create so much friction that teams will revolt.

Try infrastructure as code with Pulumi

Define your cloud resources in TypeScript, Python, or Go, then pass stack outputs straight into your GitOps manifests so both layers stay in sync.

Get started

11. Bridge your IaC and GitOps (don’t choose one)

I hear this all the time: “We use Terraform for infrastructure and ArgoCD for deployments, and they live in completely separate worlds.” Sound familiar? Most teams treat IaC and GitOps as independent workflows, but they’re really two halves of the same story.

IaC handles “Day 0,” creating the cloud resources your cluster depends on: VPCs, the Kubernetes cluster itself, IAM roles, databases, DNS zones. GitOps handles “Day 2,” managing what runs inside the cluster: applications, addons, configurations. The gap between them is where things get messy. Your Helm charts need IAM role ARNs. Your GitOps-deployed addons need to know the cluster’s OIDC provider endpoint. That metadata has to flow from the IaC layer to the GitOps layer somehow.

The gitops-bridge pattern solves this cleanly: IaC provisions cloud resources and writes metadata (role ARNs, account IDs, endpoints) into Kubernetes resources (like ConfigMaps or Secrets) that your GitOps tool can consume directly.

Keep each tool in its lane. Don’t force Terraform’s Helm provider to fight with ArgoCD over resource ownership.
Pulumi lets you define your cloud infrastructure in real programming languages (TypeScript, Python, Go), which makes it natural to compute and pass metadata to your GitOps layer. You can use stack outputs to expose values like role ARNs and cluster endpoints, then feed them into your GitOps manifests. The Pulumi Kubernetes Operator even lets ArgoCD reconcile Pulumi stacks via GitOps.
Statsig went from “1-2 devs clicking around cloud consoles with SEVs left and right” to fully self-service by using Pulumi to generate manifests and ArgoCD to deploy them.
As you grow beyond a handful of clusters, think multi-cluster from the start. Patterns like ArgoCD ApplicationSets with cluster generators and Flux’s Kustomization targeting become important for fleet management. The gitops-bridge pattern works well here because your IaC layer can register new clusters and write their metadata into Kubernetes resources, which ApplicationSets automatically pick up.

Here’s how the gitops-bridge looks in practice. Pulumi provisions cloud resources and writes metadata into a ConfigMap that ArgoCD consumes:

import * as k8s from "@pulumi/kubernetes";

const clusterMetadata = new k8s.core.v1.ConfigMap("cluster-metadata", {
 metadata: {
 name: "cluster-metadata",
 namespace: "argocd",
 labels: {
 "gitops-bridge": "true",
 },
 },
 data: {
 aws_account_id: "123456789012",
 cluster_name: "prod-us-east-1",
 iam_role_arn: "arn:aws:iam::123456789012:role/app-role",
 oidc_provider: "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE",
 },
});

import pulumi_kubernetes as k8s

cluster_metadata = k8s.core.v1.ConfigMap(
 "cluster-metadata",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="cluster-metadata",
 namespace="argocd",
 labels={
 "gitops-bridge": "true",
 },
 ),
 data={
 "aws_account_id": "123456789012",
 "cluster_name": "prod-us-east-1",
 "iam_role_arn": "arn:aws:iam::123456789012:role/app-role",
 "oidc_provider": "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE",
 },
)

Your ArgoCD Applications or Helm values can then reference this ConfigMap, closing the loop between IaC and GitOps without hardcoding cloud-specific values.

Pro Tip: If you’re using Terraform today, the gitops-bridge pattern works there too. But if you’re tired of wrangling HCL and want type-safe infrastructure code with real programming constructs, give Pulumi a look. The bridge from IaC to GitOps becomes much more natural when both sides speak a real programming language.

12. Be pragmatic, not dogmatic

Michael Crenshaw from Intuit gave a talk titled “How GitOps Should I Be?” at a CNCF conference. Intuit runs 45 ArgoCD instances managing 20,000 applications across 200 clusters. The approach they took: “Be strategic, be pragmatic. GitOps most of the time.”

One of the largest GitOps adopters in the world explicitly deviates from pure GitOps when the situation calls for it. Dogmatic adherence to principles that don’t serve you is just another form of technical debt.

Intuit found that GitHub’s SLA of 20 minutes meant they couldn’t rely on GitOps for region evacuation, so they wrote a kubectl cron job that bypasses Git for failover. Pragmatic.
Massive Jenkins ecosystems don’t get rewritten overnight. Intuit’s compromise: the last step in the Jenkins pipeline calls argocd app sync. It’s push-based, yes, but the source of truth stays in Git. The Jenkins pipeline doesn’t own the desired state; it just tells ArgoCD “go reconcile now” instead of waiting for the next poll. It’s a stepping stone while teams migrate off Jenkins, not a permanent architecture.
Continuous reconciliation with self-heal (auto-reverting manual changes) is where most teams deviate. In practice, many teams start with automated drift detection and alerting (so you know when someone runs a manual kubectl command) but leave auto-remediation off until they’ve built confidence in their exclusion lists. That’s a reasonable middle ground.

Pro Tip: Document your intentional deviations. Create an ADR (Architecture Decision Record) for each case where you’ve chosen not to follow pure GitOps, explaining why and what the plan is to close the gap (if ever). This turns exceptions into conscious decisions rather than accidental shortcuts.

Final thoughts

GitOps isn’t a silver bullet, but it’s one of the best approaches we have for managing Kubernetes at scale. The practices in this list aren’t theoretical. They’re what I’ve learned from running GitOps in production, watching talks from teams at Intuit and Statsig, and making every mistake in the book at least once.

If I had to boil it all down to one sentence: treat your GitOps repo like production code, because it is.

Start with the basics (Git as source of truth, declarative config, pull-based delivery) and layer on the advanced practices as your team matures. Don’t try to implement all 12 on Day 1. Pick the three that address your biggest pain points and iterate from there.

And remember, even the largest GitOps adopters in the world aren’t doing it perfectly. They’re being pragmatic. You should be too.

How Pulumi fits into your GitOps workflow

Several of the practices in this post touch on problems Pulumi is built to solve. If you’re bridging IaC and GitOps, managing environment-specific configuration, or enforcing policy across both layers, here’s where to start:

Pulumi IaC: Define cloud infrastructure in TypeScript, Python, or Go and use stack outputs to feed metadata into your GitOps manifests.
Pulumi ESC: Manage secrets and configuration across environments with fine-grained access controls, integrated with Kubernetes via the External Secrets Operator or the Secret Store CSI Driver.
Pulumi Kubernetes Operator: Let ArgoCD reconcile Pulumi stacks via GitOps, closing the loop between your IaC and Day 2 workflows.

Kubernetes ConfigMap Revisions with Pulumi

Matan Baruch — Tue, 13 Jan 2026 00:00:00 +0000

ConfigMaps in Kubernetes don’t have built-in revision support, which can create challenges when deploying applications with canary strategies.

When using Argo Rollouts with AWS Spot instances, ConfigMap deletions during canary deployments can cause older pods to fail when they try to reload configuration. We solved this by implementing a custom ConfigMap revision system using Pulumi’s ConfigMapPatch and Kubernetes owner references.

The Problem

When deploying applications to Kubernetes using canary strategies with Argo Rollouts, we encountered a specific challenge:

Pulumi ConfigMap replacement behavior: By default, when a ConfigMap’s data changes, Pulumi may replace it rather than update it in place, which for auto-named ConfigMaps results in a new generated name (suffix).
Canary deployment issues: During canary deployments, the old ConfigMap gets deleted, but older pods (especially on AWS Spot instances that can be replaced during canary) may fail to reload
No native revision support: Neither Kubernetes nor Pulumi natively supports ConfigMap revisions like they do for deployments

Going to KubeCon Europe 2026? Tame Kubernetes complexity with code.

Pulumi demos, platform engineering talks, AI-powered Neo in action, and yes, free plushies. Booth 784.

The solution: ConfigMap revisions with owner references

Our solution leverages Kubernetes’ garbage collection mechanism by using owner references to tie ConfigMaps to ReplicaSets created during canary deployments.

Pulumi makes this approach practical by letting us express patching logic, owner references, and rollout coordination directly in code, instead of encoding complex behavior in static YAML.

Key components

Pulumi’s ConfigMapPatch: Patches existing ConfigMaps with owner references
ReplicaSet Owner References: Links ConfigMaps to ReplicaSets for automatic cleanup
Kubernetes Garbage Collection: Automatically cleans up ConfigMaps when ReplicaSets are deleted
Retain on Delete: Protects ConfigMaps from immediate deletion during Pulumi updates

Implementation

Here’s how we implemented this solution in our rollout component:

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as k8sClient from "@kubernetes/client-node";

interface RolloutComponentArgs {
 namespace: string;
 configMapPatch?: boolean;
 kubeconfig: pulumi.Output<any>;
 configMapName: pulumi.Output<string>;
 rolloutSpec: k8s.types.input.apiextensions.CustomResourceArgs["spec"];
}

export class ConfigMapRevisionRollout extends pulumi.ComponentResource {
 public readonly rollout: k8s.apiextensions.CustomResource;

 constructor(
 name: string,
 args: RolloutComponentArgs,
 opts?: pulumi.ComponentResourceOptions
 ) {
 super("pulumi:component:ConfigMapRevisionRollout", name, {}, opts);

 // Create the Argo Rollout using CustomResource
 this.rollout = new k8s.apiextensions.CustomResource(
 `${name}-rollout`,
 {
 apiVersion: "argoproj.io/v1alpha1",
 kind: "Rollout",
 metadata: {
 name: name,
 namespace: args.namespace,
 },
 spec: args.rolloutSpec,
 },
 { parent: this, ...opts }
 );

 // Apply ConfigMap revision patching if enabled
 if (args.configMapPatch) {
 this.setupConfigMapRevisions(name, args);
 }

 this.registerOutputs({
 rollout: this.rollout,
 });
 }

 private setupConfigMapRevisions(name: string, args: RolloutComponentArgs): void {
 pulumi
 .all([args.kubeconfig, args.configMapName])
 .apply(async ([kubeconfig, configMapName]) => {
 try {
 // Create Server-Side Apply enabled provider
 const ssaProvider = new k8s.Provider(`${name}-ssa-provider`, {
 kubeconfig: JSON.stringify(kubeconfig),
 enableServerSideApply: true,
 });

 // Wait for rollout to stabilize and create ReplicaSets
 await this.waitForRolloutStabilization();

 // Get ReplicaSets associated with this rollout
 const replicaSets = await this.getAssociatedReplicaSets(
 args.namespace,
 configMapName,
 kubeconfig
 );

 if (replicaSets.length === 0) {
 pulumi.log.warn("No ReplicaSets found for ConfigMap patching");
 return;
 }

 // Create owner references for the ConfigMap
 const ownerReferences = replicaSets.map(rs => ({
 apiVersion: "apps/v1",
 kind: "ReplicaSet",
 name: rs.metadata?.name!,
 uid: rs.metadata?.uid!,
 controller: false,
 blockOwnerDeletion: false,
 }));

 // Patch the ConfigMap with owner references
 new k8s.core.v1.ConfigMapPatch(
 `${configMapName}-revision-patch`,
 {
 metadata: {
 name: configMapName,
 namespace: args.namespace,
 ownerReferences: ownerReferences,
 annotations: {
 "pulumi.com/patchForce": "true",
 "configmap.kubernetes.io/revision-managed": "true",
 },
 },
 },
 {
 provider: ssaProvider,
 retainOnDelete: true,
 parent: this,
 }
 );

 pulumi.log.info(
 `Successfully patched ConfigMap ${configMapName} with ${ownerReferences.length} owner references`
 );
 } catch (error) {
 pulumi.log.error(`Failed to setup ConfigMap revisions: ${error}`);
 throw error;
 }
 });
 }

 private async waitForRolloutStabilization(): Promise<void> {
 // Wait for rollout to create and stabilize ReplicaSets
 // In production, consider using a more sophisticated polling mechanism
 await new Promise(resolve => setTimeout(resolve, 10000));
 }

 private async getAssociatedReplicaSets(
 namespace: string,
 configMapName: string,
 kubeconfig: any
 ): Promise<k8sClient.V1ReplicaSet[]> {
 const kc = new k8sClient.KubeConfig();
 kc.loadFromString(JSON.stringify(kubeconfig));

 const appsV1Api = kc.makeApiClient(k8sClient.AppsV1Api);

 try {
 const response = await appsV1Api.listNamespacedReplicaSet(
 namespace,
 undefined, // pretty
 false, // allowWatchBookmarks
 undefined, // continue
 undefined, // fieldSelector
 `configMap=${configMapName}` // labelSelector
 );

 return response.body.items;
 } catch (error) {
 pulumi.log.error(`Failed to list ReplicaSets: ${error}`);
 return [];
 }
 }
}

How do ConfigMap revisions work with Argo Rollouts?

Rollout Creation: When a new rollout is created, Argo Rollouts generates new ReplicaSets for the canary deployment
ConfigMap Patching: Our code waits for the ReplicaSet creation, then patches the ConfigMap with owner references pointing to these ReplicaSets
Garbage Collection: Kubernetes automatically tracks the relationship between ConfigMaps and ReplicaSets
Automatic Cleanup: When ReplicaSets are cleaned up (based on the default 10 revision history), their associated ConfigMaps are also garbage collected

Benefits

Revision Control: ConfigMaps now have revision-like behavior tied to ReplicaSet history
Automatic Cleanup: No manual intervention needed for ConfigMap cleanup
Canary Safety: Old ConfigMaps remain available during canary deployments until ReplicaSets are cleaned up
Spot Instance Resilience: Pods that get replaced during canary deployments can still access their original ConfigMaps

Configuration options

interface RolloutComponentArgs {
 namespace: string;
 configMapPatch?: boolean;
 kubeconfig: pulumi.Output<any>;
 configMapName: pulumi.Output<string>;
 rolloutSpec: k8s.types.input.apiextensions.CustomResourceArgs["spec"];
}

To enable this feature in your rollout:

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Create your EKS cluster
const cluster = new k8s.Provider("k8s-provider", {
 kubeconfig: clusterKubeconfig,
});

// Create ConfigMap
const appConfig = new k8s.core.v1.ConfigMap("app-config", {
 metadata: {
 name: "my-app-config",
 namespace: "default",
 labels: {
 app: "my-app",
 configMap: "my-app-config", // Important for ReplicaSet selection
 },
 },
 data: {
 "app.properties": "key=value\nother=setting",
 },
}, { provider: cluster });

// Create rollout with ConfigMap revision management
const rollout = new ConfigMapRevisionRollout("my-app", {
 namespace: "default",
 configMapPatch: true,
 kubeconfig: clusterKubeconfig,
 configMapName: appConfig.metadata.name,
 rolloutSpec: {
 replicas: 3,
 selector: {
 matchLabels: { app: "my-app" },
 },
 template: {
 metadata: {
 labels: { app: "my-app" },
 },
 spec: {
 containers: [{
 name: "app",
 image: "nginx:latest",
 volumeMounts: [{
 name: "config",
 mountPath: "/etc/config",
 }],
 }],
 volumes: [{
 name: "config",
 configMap: {
 name: appConfig.metadata.name,
 },
 }],
 },
 },
 strategy: {
 canary: {
 maxSurge: 1,
 maxUnavailable: 0,
 steps: [
 { setWeight: 20 },
 { pause: { duration: "1m" } },
 { setWeight: 50 },
 { pause: { duration: "2m" } },
 ],
 },
 },
 },
});

Key dependencies

The solution uses several key packages:

@pulumi/kubernetes: For Kubernetes resources and ConfigMapPatch
@kubernetes/client-node: For direct Kubernetes API access
Argo Rollouts CRDs installed in your cluster

Manage Kubernetes config as code

You can express patching logic, owner references, and rollout coordination in TypeScript instead of static YAML, and version every change to your cluster. Start a project to build your own revision workflow.

Get started

When should you use this approach?

This pattern is a good fit if you:

Use Argo Rollouts with canary deployments
Rely on ConfigMaps that must remain available across rollout revisions
Run workloads on Spot or preemptible instances
Need automatic cleanup without custom controllers

Conclusion

This approach gives us ConfigMap revision functionality that doesn’t exist natively in Kubernetes or Pulumi. By leveraging Kubernetes’ garbage collection mechanism and Pulumi’s patching capabilities, we created a robust solution for managing ConfigMap lifecycles during canary deployments.

The solution is particularly valuable when:

Running canary deployments with Argo Rollouts
Using AWS Spot instances that can be replaced during deployments
Needing automatic cleanup of old ConfigMaps without manual intervention
Wanting to maintain configuration availability for older pods during deployment transitions

This pattern can be extended to other scenarios where you need revision control for Kubernetes resources that don’t natively support it.

Want to learn how to put these practices into action? Meet us at KubeCon Europe 2026 (Booth 784) or register for our upcoming Zero to Production in Kubernetes workshop.

Try Pulumi for Free

How to Move to the Gateway API: post ingress-nginx Retirement

Engin Diri — Fri, 02 Jan 2026 11:35:51 +0100

The upcoming retirement of ingress-nginx in early 2026 gives infrastructure teams both a deadline and an opportunity to rethink traffic management. Configuring the Ingress API often meant relying on controller-specific annotations that varied between implementations. The Gateway API offers a cleaner, standardized alternative. This post investigates the practical reality of this migration and explores why kgateway emerges as a robust solution for the future.

With ingress-nginx entering its sunset phase in early 2026, the Kubernetes community faces a decision point. While the controller served as the default standard for a decade, its architecture now struggles to meet modern requirements. The transition to the Gateway API offers a chance to adopt a standard designed for contemporary traffic patterns.

The Gateway API addresses the portability issues of its predecessor by establishing a standardized, expressive approach that behaves consistently across implementations. A technical evaluation of the available options points to kgateway as a particularly strong candidate for production workloads.

Why ingress-nginx is retiring

The Kubernetes SIG Network and Security Response Committee was direct in its announcement regarding the project’s future. After March 2026, ingress-nginx will cease to receive releases, bug fixes, or security patches. The repositories will become read-only, leaving existing deployments functional but unmaintained.

This decision stems from both resource constraints and technical debt. The project relied on a very small group of maintainers working primarily in their spare time. Furthermore, features that once provided flexibility, such as “snippets” for arbitrary NGINX configuration injection, now pose significant security liabilities. With no path to modernize the codebase, retirement was inevitable.

It is worth noting that the Ingress API itself remains supported. The retirement affects only the ingress-nginx controller. NGINX as a web server continues unchanged. However, clusters relying on this specific controller must prepare for a transition.

To identify affected resources, checking the cluster for specific pods can reveal the scope of the dependency:

kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx

Understanding the Gateway API

The Gateway API represents a fundamental shift in traffic management concepts. Some elements will look familiar to Ingress users, but the underlying philosophy addresses the “baggage” that the previous standard accumulated.

The core improvement lies in expressiveness. Advanced routing requirements (header matching, weighted traffic splitting, and traffic policies) are now native parts of the specification. This eliminates the need for the non-portable annotations that plagued the Ingress ecosystem.

The API also introduces a role-oriented structure that mirrors actual organizational workflows.

GatewayClass resources are managed by infrastructure teams to define the underlying controller.
Gateway resources are created by platform teams to specify entry points and listeners.
HTTPRoute resources are owned by application teams to define service traffic rules.

This separation allows for genuine self-service. Application teams can manage their routing logic without requiring broad cluster privileges.

The relationship between these resources defines the traffic flow:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
 name: kgateway
spec:
 controllerName: kgateway.dev/kgateway

A Gateway references this class to establish the entry point:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: my-gateway
 namespace: default
spec:
 gatewayClassName: kgateway
 listeners:
 - name: http
 port: 8080
 protocol: HTTP
 allowedRoutes:
 namespaces:
 from: All

An HTTPRoute then attaches to the gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: my-route
 namespace: default
spec:
 parentRefs:
 - name: my-gateway
 hostnames:
 - "example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /api
 backendRefs:
 - name: api-service
 port: 8080

Beyond standard HTTP, the API supports GRPCRoute, TCPRoute, UDPRoute, and TLSRoute, offering a protocol diversity that the original Ingress spec lacked.

Cross-namespace routing and ReferenceGrant

The Gateway API enables a security model where routes can cross namespace boundaries, but only with explicit permission. The ReferenceGrant resource controls these connections. Without a grant, an HTTPRoute in one namespace cannot reference a Service in another.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
 name: allow-routes-from-apps
 namespace: gateway-system
spec:
 from:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 namespace: my-app
 to:
 - group: ""
 kind: Service

This mechanism allows platform teams to strictly control which namespaces can attach to shared gateways.

Evaluating Gateway API implementations

Several implementations have emerged to support the new standard, each with a distinct philosophy.

NGINX Gateway Fabric leverages NGINX as the data plane, offering continuity for teams in the F5 ecosystem. It provides impressive throughput and integrates with enterprise security tools, making it a logical choice for existing NGINX shops.

Traefik focuses on simplicity. As a single binary with no external dependencies, it aligns well with environments that prioritize developer experience and rapid iteration. Its declarative configuration is particularly friendly to GitOps workflows.

Envoy Gateway is built on the Envoy Proxy and operates under a community-driven governance model. It appeals to those seeking strong conformance to the Gateway API spec without vendor lock-in. Its companion project, Envoy AI Gateway, adds support for LLM routing.

kgateway, formerly Gloo Gateway, brings seven years of production history to the table. Donated to the CNCF in early 2025, it combines an Envoy data plane with a control plane optimized for scale. Its internal architecture uses the “krt” framework to handle massive route tables efficiently, avoiding the performance bottlenecks often seen in snapshot-based systems.

Aspect	NGINX Gateway Fabric	Traefik	Envoy Gateway	kgateway
Base Technology	NGINX (C)	Custom Go	Envoy Proxy (C++)	Envoy Proxy (C++)
Maturity	Production-ready	Mature	GA (v1.0+)	Battle-tested (7+ years as Gloo)
Setup Complexity	Medium	Low	Low	Low
Commercial Support	F5 Enterprise	Traefik Labs	Multiple vendors	Solo.io
AI/LLM Support	No	No	Via Envoy AI Gateway	Native (Agentgateway)
Learning Curve	Familiar for NGINX users	Gentle	Moderate	Moderate
Community Size	Growing	Large	Large	Established
Best For	Teams with NGINX expertise, high-throughput requirements	Teams prioritizing simplicity and rapid deployment	Standards-focused teams, multi-vendor environments	Enterprise scale, AI workloads, Istio integration

Decision framework

NGINX Gateway Fabric suits teams needing F5 support or deep NGINX tuning.
Traefik fits best where simplicity and operational ease are paramount.
Envoy Gateway works for those valuing community governance and strict standards adherence.
kgateway is the choice for enterprise scale, native AI gateway needs, and Istio integration.

This exploration uses kgateway for its proven track record and native handling of both traditional microservices and AI traffic.

Migrating from Ingress to Gateway API

The official migration guide outlines the transition process. It’s less translation, more restructuring.

The process typically involves:

Defining a Gateway resource: Unlike Ingress, listeners must be explicitly defined. This provides granular control over ports and protocols.
Creating HTTPRoute resources: These replace Ingress routing rules. You can split complex Ingress resources into multiple HTTPRoutes for better management.
Configuring filters: Annotations for headers or redirects are replaced by standardized filters within the HTTPRoute spec.

The ingress2gateway tool can automate much of the initial conversion, providing a baseline of resources to review and refine.

Migration planning

Scope the migration effort first:

1. Assessment Identify the volume and complexity of existing resources.

# Count your Ingress resources across all namespaces
kubectl get ingress -A | wc -l

# List all Ingress resources with their hosts
kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.spec.rules[*].host}{"\n"}{end}'

2. Parallel Operations Running kgateway alongside ingress-nginx allows for incremental migration. Services can be moved one by one, validating the new configuration without risking the entire cluster’s traffic.

3. Rollback Strategy Keeping the original Ingress manifests ensures that traffic can be quickly reverted to the old controller if issues arise during the transition.

Why kgateway stands out

kgateway distinguishes itself through its maturity and architectural decisions. Having started as Gloo Gateway in 2018, it has hardened through years of production use.

Its control plane uses the krt framework, originally developed for Istio. This allows it to track dependencies precisely and recalculate only the parts of the configuration that have changed. In large clusters where pods churn constantly, this incremental approach prevents the control plane from becoming a bottleneck, enabling it to handle over 10,000 routes efficiently.

For teams managing AI workloads, kgateway includes the Agentgateway component. This Rust-based data plane is built for LLM traffic, supporting the Model Context Protocol (MCP) and providing token-based rate limiting. It unifies the management of AI and traditional traffic under a single control plane.

Additional features include:

Route Delegation: Allows large routing tables to be split across teams with clear inheritance.
Security Integration: Seamless mTLS with Istio ambient mesh and external authorization support.
Traffic Management: Advanced capabilities like traffic mirroring and session affinity are available without complex Envoy configuration.

Putting it all together with Pulumi

Seeing these concepts in code clarifies the architecture. The following Pulumi program demonstrates a complete setup: a DigitalOcean Kubernetes cluster, the kgateway installation via Helm, Gateway API resources, and an httpbin application to validate the configuration.

The program provisions infrastructure in a specific sequence. The cluster comes first, followed by the Gateway API CRDs, then kgateway itself. Once the control plane is running, the program creates a GatewayClass, a Gateway listener, and an HTTPRoute that directs traffic to the sample application. A ReferenceGrant permits the cross-namespace reference between the HTTPRoute and the backend Service.

import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";
import * as k8s from "@pulumi/kubernetes";

// Configuration
const config = new pulumi.Config();
const clusterName = config.get("clusterName") || "kgateway-demo-cluster";
const region = config.get("region") || "fra1";
const nodeSize = config.get("nodeSize") || "s-2vcpu-4gb";
const nodeCount = config.getNumber("nodeCount") || 2;
const k8sVersion = config.get("k8sVersion") || "1.32.10-do.2";

// Create a DigitalOcean Kubernetes cluster
const cluster = new digitalocean.KubernetesCluster(clusterName, {
 name: clusterName,
 region: region,
 version: k8sVersion,
 nodePool: {
 name: "default-pool",
 size: nodeSize,
 nodeCount: nodeCount,
 },
});

// Create a Kubernetes provider using the cluster's kubeconfig
const k8sProvider = new k8s.Provider("k8s-provider", {
 kubeconfig: cluster.kubeConfigs[0].rawConfig,
});

// Install Gateway API CRDs (standard channel)
const gatewayApiCrds = new k8s.yaml.ConfigFile("gateway-api-crds", {
 file: "https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml",
}, { provider: k8sProvider });

// Create kgateway-system namespace
const kgatewayNamespace = new k8s.core.v1.Namespace("kgateway-system", {
 metadata: {
 name: "kgateway-system",
 },
}, { provider: k8sProvider });

// Install kgateway CRDs via Helm
const kgatewayCrds = new k8s.helm.v3.Release("kgateway-crds", {
 chart: "oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds",
 version: "v2.1.2",
 namespace: kgatewayNamespace.metadata.name,
 createNamespace: false,
}, { provider: k8sProvider, dependsOn: [gatewayApiCrds, kgatewayNamespace] });

// Install kgateway control plane via Helm
const kgateway = new k8s.helm.v3.Release("kgateway", {
 chart: "oci://cr.kgateway.dev/kgateway-dev/charts/kgateway",
 version: "v2.1.2",
 namespace: kgatewayNamespace.metadata.name,
 createNamespace: false,
}, { provider: k8sProvider, dependsOn: [kgatewayCrds] });

// Create hello-world namespace
const helloWorldNamespace = new k8s.core.v1.Namespace("hello-world", {
 metadata: {
 name: "hello-world",
 },
}, { provider: k8sProvider });

// Deploy hello world application (using httpbin as shown in kgateway docs)
const helloWorldLabels = { app: "httpbin" };

const helloWorldDeployment = new k8s.apps.v1.Deployment("httpbin", {
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: helloWorldLabels,
 },
 template: {
 metadata: {
 labels: helloWorldLabels,
 },
 spec: {
 containers: [{
 name: "httpbin",
 image: "mccutchen/go-httpbin:v2.15.0",
 ports: [{
 containerPort: 8080,
 name: "http",
 }],
 }],
 },
 },
 },
}, { provider: k8sProvider, dependsOn: [helloWorldNamespace] });

const helloWorldService = new k8s.core.v1.Service("httpbin", {
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 selector: helloWorldLabels,
 ports: [{
 port: 8000,
 targetPort: 8080,
 name: "http",
 }],
 },
}, { provider: k8sProvider, dependsOn: [helloWorldDeployment] });

// Create GatewayClass for kgateway
const gatewayClass = new k8s.apiextensions.CustomResource("kgateway-gateway-class", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "GatewayClass",
 metadata: {
 name: "kgateway",
 },
 spec: {
 controllerName: "kgateway.dev/kgateway",
 },
}, { provider: k8sProvider, dependsOn: [kgateway] });

// Create Gateway with LoadBalancer service (matching kgateway docs pattern)
const gateway = new k8s.apiextensions.CustomResource("http-gateway", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "Gateway",
 metadata: {
 name: "http",
 namespace: kgatewayNamespace.metadata.name,
 },
 spec: {
 gatewayClassName: "kgateway",
 listeners: [{
 name: "http",
 port: 8080,
 protocol: "HTTP",
 allowedRoutes: {
 namespaces: {
 from: "All",
 },
 },
 }],
 },
}, { provider: k8sProvider, dependsOn: [gatewayClass] });

// Create HTTPRoute for httpbin application
const httpRoute = new k8s.apiextensions.CustomResource("httpbin-route", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "HTTPRoute",
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 parentRefs: [{
 name: "http",
 namespace: kgatewayNamespace.metadata.name,
 }],
 hostnames: [
 "*.nip.io",
 ],
 rules: [{
 backendRefs: [{
 name: "httpbin",
 port: 8000,
 }],
 }],
 },
}, { provider: k8sProvider, dependsOn: [gateway, helloWorldService] });

// Create a reference grant to allow the HTTPRoute to reference the service cross-namespace
const referenceGrant = new k8s.apiextensions.CustomResource("httpbin-reference-grant", {
 apiVersion: "gateway.networking.k8s.io/v1beta1",
 kind: "ReferenceGrant",
 metadata: {
 name: "allow-gateway-to-httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 from: [{
 group: "gateway.networking.k8s.io",
 kind: "HTTPRoute",
 namespace: helloWorldNamespace.metadata.name,
 }],
 to: [{
 group: "",
 kind: "Service",
 }],
 },
}, { provider: k8sProvider, dependsOn: [gatewayApiCrds] });

// Create a LoadBalancer Service for the Gateway proxy
const gatewayProxyService = new k8s.core.v1.Service("gateway-proxy", {
 metadata: {
 name: "gateway-proxy",
 namespace: kgatewayNamespace.metadata.name,
 },
 spec: {
 type: "LoadBalancer",
 selector: {
 "gateway.networking.k8s.io/gateway-name": "http",
 },
 ports: [{
 name: "http",
 port: 8080,
 targetPort: 8080,
 protocol: "TCP",
 }],
 },
}, { provider: k8sProvider, dependsOn: [gateway] });

// Extract the LoadBalancer IP from our managed Service
const gatewayIP = gatewayProxyService.status.apply(status => {
 const ingress = status?.loadBalancer?.ingress;
 if (ingress && ingress.length > 0) {
 return ingress[0].ip || ingress[0].hostname || "pending";
 }
 return "pending";
});

// Exports
export const clusterNameOutput = cluster.name;
export const clusterEndpoint = cluster.endpoint;
export const kubeconfig = pulumi.secret(cluster.kubeConfigs[0].rawConfig);

// Export the LoadBalancer IP and nip.io URL
export const gatewayLoadBalancerIP = gatewayIP;
export const httpbinUrl = gatewayIP.apply(ip =>
 ip !== "pending" ? `http://httpbin.${ip}.nip.io:8080/get` : "Waiting for LoadBalancer IP..."
);

The demo uses nip.io, a wildcard DNS service that resolves *.IP.nip.io hostnames to the specified IP address. This eliminates the need to configure separate DNS records during testing. Once deployed, the httpbinUrl output provides a ready-to-use endpoint for validating the gateway configuration.

Migrate to the Gateway API

You can provision your cluster, install kgateway, and define your Gateway API resources in one Pulumi program, then migrate services incrementally and review each change. Start a project to plan your migration.

Get started

The short-term option: Chainguard’s ingress-nginx fork

For organizations that cannot immediately migrate to the Gateway API, there is another path forward. Chainguard has forked ingress-nginx as part of their EmeritOSS program, providing continued maintenance after the upstream project’s retirement.

This fork is not about adding new features. Chainguard is explicit that they are maintaining stability, not continuing development. Their commitment includes keeping dependencies updated, addressing CVEs on a best-efforts basis, and providing commercial container images with low vulnerability counts and SLAs. FIPS-compliant versions are also available for regulated environments.

The chainguard-forks/ingress-nginx repository on GitHub provides the maintained codebase. For teams running ingress-nginx in production, this fork offers a viable bridge while evaluating the Gateway API or other alternatives. However, this buys time rather than solving the problem permanently. The architectural limitations of the Ingress API remain, and the eventual move to a more expressive standard like the Gateway API is still the recommended path forward.

Looking ahead

The retirement of ingress-nginx marks the end of an era and the beginning of a more structured approach to Kubernetes networking. The Gateway API offers a robust framework that reflects how teams actually build and secure applications today.

While March 2026 provides a comfortable runway, early planning prevents rushed decisions. Setting up a test environment with kgateway to validate the new API constructs is a prudent next step.

Moving to the Gateway API means building a networking foundation ready for the next decade of infrastructure challenges.

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

Resources

Gateway API Documentation – Comprehensive guides and examples.
Migration Guide – Official steps for transitioning from Ingress.
ingress2gateway – Tooling to automate resource conversion.
kgateway Documentation – Installation and configuration details.
Designing kgateway for Scalability – Deep dive into the krt framework.
Kubernetes Blog: ingress-nginx Retirement – Full context on the deprecation.

Future of the Cloud: 10 Trends Shaping 2026 and Beyond

Sara Huddleston — Thu, 04 Dec 2025 07:56:40 +0000

In 2026, several trends will dominate cloud computing, driving innovation, efficiency, and scalability. From Infrastructure as Code (IaC) to AI/ML, platform engineering to multi-cloud and hybrid strategies, and security practices, let’s explore the 10 biggest emerging trends.

On This Article

1. Cloud Will Become a Business Necessity by 2028
2. Hyperscalers Accelerate AI-Driven Cloud Expansion
3. Hybrid and Multi-Cloud to Drive Innovation
4. Enterprises Rebuild Their Cloud Foundations to Operationalize AI
5. IaC Drives Scalable Cloud, Multi-Cloud, and AI Operations
6. DevSecOps Evolves Into AI-Integrated Security
7. Platform Engineering and Internal Developer Platforms
8. AIOps Matures Into a Cloud Operations Standard
9. Kubernetes Dominance and Increased Complexity
10. AI Code Assistants in the Enterprise
The Future of Cloud: Reinvented for an AI-First Decade

1. Cloud Will Become a Business Necessity by 2028

According to Gartner, by 2028 the cloud will be the key driver for business innovation, and estimates that over 95% of new digital workloads will be deployed on cloud-native platforms.

The future of cloud computing. Credit: Gartner

According to McKinsey & Company’s “In search of cloud value” report:

Cloud value is driven by innovation, worth 5x more than cost savings.
EBITDA uplift of 20–30% by 2030 for high-performing organizations.
Asia leads in projected cloud value, followed by the US and Europe.
High-ROI organizations excel by aligning cloud strategy with business priorities, building strong cloud foundations, and using modern operating models.

Teams succeeding in this transition increasingly use Infrastructure as Code, automation, and unified governance frameworks like Pulumi Insights + Policies to operationalize this value.

2. Hyperscalers Accelerate AI-Driven Cloud Expansion

Hyperscalers are making the largest infrastructure investments in cloud history — nearly all centered on AI workloads, inference, and high-performance compute.

AWS has integrated Anthropic’s Claude 3 and Claude 4 models into Amazon Bedrock for enterprise LLM workflows. “Claude Opus 4 and Claude Sonnet 4 are available today in Amazon Bedrock, enabling customers to build agents with stronger reasoning, memory, and tool use.” — AWS, May 2025
Microsoft Azure revenue rose 33% year-over-year in Q3 (ended March 31), outperforming estimates of ~29.7%. AI contributed 16 percentage points to this growth, up from 13 points in the prior quarter. “Microsoft is on track to invest approximately $80 billion to build out AI-enabled datacenters to train AI models and deploy AI and cloud-based applications around the world,” said Brad Smith, the Microsoft Vice Chair and President.
Google Cloud is committing $25 billion over two years for data center and AI infrastructure expansion across the PJM grid, with total capital expenditure for 2025 ranging from $75–85 billion. “As our CEO has said, in these early days of a very transformative technology, the risks of under-investing are dramatically higher than the risks of over-investing,” said Eunice Huang, Head of AI and Emerging Tech Policy.
Oracle anticipates 15–20% cloud revenue growth in FY 2026–2027 attributable to AI infrastructure demand, tied to its partnership in the Stargate initiative.

As hyperscalers integrate AI deeper into their service layers, engineering teams must adapt with IaC-driven automation, reusable patterns, and policy controls to deploy cloud and AI infrastructure consistently. See how organizations deploy AWS infrastructure at the speed of AI with Pulumi and Pulumi Policies.

3. Hybrid and Multi-Cloud to Drive Innovation

Hybrid and multi-cloud strategies are now mainstream:

Hybrid cloud will grow from $130B to $310–330B by 2030 (ResearchAndMarkets).
87% of enterprises run workloads across multiple clouds (Mordor Intelligence).
Gartner predicts that 40% of enterprises will adopt hybrid compute architectures in mission-critical workflows by 2028 (up from 8%).

Credit: Cloud Worldwide Service, Forbes

As AI and regulatory requirements grow, organizations must deploy workloads across AWS, Azure, Google Cloud, on-prem, and edge — while maintaining consistent security, compliance, and configuration.

Modern cloud teams use:

Infrastructure as Code for consistent multi-cloud provisioning and environment standardization, forming the backbone of AI infrastructure orchestration
Reusable components and internal platforms to define scalable architecture patterns and accelerate delivery across Kubernetes, AI/ML pipelines, and hybrid environments
Policy-driven guardrails to maintain cost, security, and compliance across environments, supporting cloud governance automation and modern cloud cost governance

Pulumi enables all three through its multi-cloud IaC model, Pulumi Policies, and internal developer platform capabilities.

4. Enterprises Rebuild Their Cloud Foundations to Operationalize AI

While hyperscalers are transforming the global cloud platform, enterprises face a different challenge: adapting their own cloud foundations to support AI at scale. Organizations are moving beyond prototypes and integrating AI into core products, internal workflows, and customer-facing systems, requiring new levels of automation, governance, and AI infrastructure orchestration.

According to Gartner, global AI infrastructure spending is expected to surpass $2 trillion in 2026. IDC predicts that by 2027, more than 50% of enterprises will use AI agents to drive core workflows, which requires scalable, secure, and automated cloud architectures to support model execution and orchestration.

To enable this transition, enterprises are investing in:

GPU provisioning and orchestration, data pipelines, vector databases, feature stores, and LLM infrastructure needed for real-time AI workloads.
Data pipelines, vector databases, and feature stores needed for real-time AI workloads
Model-serving infrastructure, including gateways, inference routers, and autoscaling layers
Strong identity, secrets, and access controls as AI systems increase security exposure
Automation through Infrastructure as Code to ensure reproducibility and reduce drift
Policy-driven governance to secure cost, compliance, and architectural consistency

As AI becomes deeply embedded across engineering organizations, teams are increasingly using software engineering approaches such as Infrastructure as Code, reusable components, platform engineering, and policy automation to standardize how AI infrastructure is deployed, scaled, and secured across clouds.

To support this shift, Pulumi’s perspective on Superintelligence Infrastructure explains why AI workloads, from pre-training to inference at massive scale, require dynamic infrastructure orchestration rather than static configuration.

Pulumi users increasingly rely on:

Pulumi IaC for standardized AI infrastructure
Pulumi ESC to manage all secrets and configuration at scale
Pulumi Insights for visibility and misconfiguration analysis
Pulumi Policies for AI-specific guardrails in code, cost detection, and to provide automated compliance protections

5. IaC Drives Scalable Cloud, Multi-Cloud, and AI Operations

As cloud environments expand and AI workloads demand highly dynamic infrastructure, Infrastructure as Code (IaC) is becoming the foundation for scaling reliably across all environments. Organizations are increasingly adopting IaC in general-purpose languages to unify development and infrastructure workflows, reduce configuration drift, and deliver cloud resources at speed.

Modern Infrastructure as Code is advancing far beyond simple provisioning:

Standardizing multi-cloud and hybrid patterns so teams can deploy consistently across AWS, Azure, Google Cloud, on-prem, and edge environments.
Integrating seamlessly with cloud providers and third-party services, including data platforms and messaging systems like CockroachDB, Confluent Cloud, and Kafka.
Providing deeper validation and type-safety, ensuring parameters, dependencies, and security controls are correct before deployment.
Improving cloud resource efficiency and visibility with tools like Pulumi Insights Discovery.
Embedding security and compliance through Policy as Code, enforcing guardrails, cost controls, and regulatory requirements automatically, enabling truly policy-driven cloud management.
Enabling intelligent automation, from unit and integration tests to auto-remediation policies and policy-driven approvals.
Incorporating AI-driven optimization and insights, helping teams detect misconfigurations, analyze usage patterns, and generate infrastructure updates with tools like Pulumi Neo and Pulumi Policies.

As organizations scale both traditional cloud workloads and AI-driven systems, IaC has become critical for achieving secure, repeatable, and high-velocity operations across every environment.

6. DevSecOps Evolves Into AI-Integrated Security

As AI becomes embedded across cloud-native systems, DevSecOps is entering a new era. Gartner predicts that by 2028, over 50% of enterprises will use AI security platforms to protect their AI investments. Below are the 3 key predictions for the future of DevSecOps:

AI-driven security automation: Teams will increasingly rely on AI to detect threats, enforce policies, and generate secure infrastructure patches. See Pulumi’s capabilities in AI-powered remediation.
More focus on secrets management: With AI systems accessing more sensitive data, secure secret storage will be essential. Pulumi ESC helps teams centralize and govern credentials, keys, and tokens safely.
Greater cross-team collaboration: Dev, Sec, and Ops workflows will converge under shared frameworks: IaC, policy automation, runtime scanning, and GitOps.

As organizations increase their use of AI across cloud-native systems, the need for tightly aligned security, governance, and cloud governance automation becomes even more urgent. At the Gartner Data & Analytics Summit in Sydney, Carlie Idoine, VP Analyst at Gartner, emphasized this growing dependency:

"[AI]… it doesn’t deliver value on its own – AI needs to be tightly aligned with data, analytics, and governance to enable intelligent, adaptive decisions and actions across the organization."

This perspective mirrors what we’re seeing across modern DevSecOps practices: AI can amplify security, but only when paired with strong foundations in secrets management, governance, and cross-team collaboration.

Build for an AI-first cloud

Pulumi gives teams infrastructure as code, reusable components, and policy guardrails to deliver consistently across every cloud and AI workload.

Get started

7. Platform Engineering & Internal Developer Platforms (IDPs)

According to Gartner, by 2026, 80% of large software engineering organizations will establish platform engineering teams as internal providers of reusable services, components, and tools for application delivery. Platform engineering will ultimately solve the central problem of cooperation between software developers and operators.

Mid-size to large companies will begin or continue to invest in implementing platform engineering practices, with large tech companies as first adopters. They will provide Internal Developer Platforms (IDP) to elevate the Developer Experience (DX, sometimes referred to as DE or DevEx), helping them work faster, like abstracting the complexities of configuring, testing, and validation, deploying infrastructure, and scanning their code for security.

Internal developer platform-in-a-box. Credit: Pulumi

IDPs are reshaping how developers interact with cloud infrastructure, bringing together platform engineering, automation, and emerging AI platform engineering practices.

8. AIOps Matures into a Cloud Operations Standard

AIOps is becoming mainstream, helping teams predict failures, auto-scale infrastructure, and resolve incidents with minimal manual effort. As AI and automation continue to evolve, the fusion of these technologies will enable organizations to achieve unprecedented levels of efficiency and scalability.

Proactive Operations: AI-powered tools will assist teams in foreseeing issues with greater accuracy, minimizing downtime, and reducing the firefighting nature of incident management. These tools will automatically detect anomalies, optimize performance, and trigger remediation actions.
Intelligent Automation: Routine operational tasks like patching, monitoring, and resource scaling will be fully automated. AI-driven decision-making will allow for smarter resource allocation and optimization, dynamically adjusting infrastructure and workloads in response to real-time demands and predictions.
Data-Driven Insights: AIOps will analyze vast amounts of operational data and provide actionable insights, enabling teams to focus on high-impact tasks such as improving system architecture and user experience. The AI-powered insights will also inform better strategic decisions, helping teams to continuously evolve their DevOps practices.
Collaboration Across Teams: AIOps will bridge the gap between DevOps, SecOps, and IT operations by bridging monitoring and automation. Cross-team collaboration will improve as AI systems consolidate and interpret data from various departments, allowing for a more cohesive approach to system management.

AIOps features include observability, automation, and real-time analytics to bridge DevOps, SRE, and IT operations.

9. Kubernetes Dominance and Increased Complexity

Kubernetes will continue its ascent in 2026. According to Research & Markets, the global Kubernetes market was valued at USD 2.3 billion in 2024 and is projected to reach USD 8.2 billion by 2030, with a CAGR of 23.8% over the forecast period.

The CNCF Annual Survey shows AI/ML workloads rapidly moving onto Kubernetes — including batch pipelines, model experimentation, real-time inference, and data preprocessing — even as only 41% of ML/AI developers are cloud-native today. This shift is accelerating as teams need flexible GPU scheduling, distributed pipelines, and portable execution environments.

Kubernetes is also evolving in response to AI demands. Inference workloads, powered by LLMs and GPUs, now require low-latency execution closer to the user. This shift is pushing organizations to build intelligent orchestration layers that schedule AI pipelines across edge and core clusters, often leveraging Kubernetes as the common control plane for AI cluster orchestration.

As we move into 2026, three patterns are becoming clear:

Kubernetes is evolving to support AI through GPU-aware scheduling, Kubernetes GPU scheduling optimizations, and more advanced workload orchestration.
Governance and consistency matter more than ever, as teams struggle to secure and manage multi-cluster, multi-cloud environments.
Platform engineering is essential, providing curated patterns and automation rather than raw YAML to reduce cognitive load.

Kubernetes will remain a strategic foundation — but operating it effectively now depends on robust automation, strong security controls, and standardized delivery models that scale across clouds, clusters, and AI pipelines.

10. AI Code Assistants in the Enterprise

AI-powered coding assistants like GitHub Copilot, Claude Code, Cursor, and others are rapidly becoming part of modern software development.

By 2027, the use of AI assistants will dramatically increase developer velocity to meet functional business requirements for 70% of new digital solutions in production (source: IDC).

The value of AI code assistants. Credit: Gartner

According to Gartner, by 2028, 75% of enterprise software engineers will use dedicated AI code assistants, and 63% of organizations are currently piloting, deploying or beginning to use AI code assistants.

As enterprise adoption of AI assistants increases, expectations are rising: they must not only generate code but also understand the state of infrastructure, configurations, and security posture. That means being able to answer questions about environments, surface misconfigurations, or act directly on infrastructure.

One of the newest developments is the release of Pulumi Agent Skills, a collection of infrastructure expertise packaged for use in AI coding assistants. These skills teach tools such as Claude Code, Cursor, or Gemini CLI to reason about Pulumi projects, reducing hallucination and improving outputs based on real infrastructure conventions and practices.

Combined with infrastructure access via tools like Pulumi’s Remote MCP Server, teams can build secure, AI-driven workflows where assistants provide insights and Pulumi Neo safely executes actions with previews, policies, and orchestration.

AI code assistants are no longer experimental; they’re fast becoming a competitive advantage in cloud software development.

The Future of Cloud: Reinvented for an AI-First Decade

Cloud infrastructure is entering its most transformative era since the rise of Kubernetes. The trends shaping 2026 reveal a clear pattern: AI is no longer a workload — it’s becoming the organizing principle of cloud strategy.

AI-native cloud architectures that require elastic compute, GPU orchestration, fast data access, and governance built into every layer
Infrastructure as Code as the operational backbone, standardizing deployments across AI, cloud, and hybrid environments
Platform engineering and IDPs to enable self-service, gold-standard patterns, and automated guardrails
Security integrated into every pipeline, with AI-assisted threat detection, strong secrets management, and policy-driven compliance
AIOps and intelligent automation are becoming standard for scaling modern cloud systems
Kubernetes evolving for AI, driving new orchestration patterns across edge, core, and inference clusters
Multi-cloud and hybrid ecosystems accelerating to support interoperability, resilience, and global workload placement

Taken together, these shifts point to a new model of cloud operations that is intelligent, automated, policy-aware, and built on software engineering principles rather than manual configuration.

Organizations that invest now in modern IaC, unified governance, reusable components, and policy frameworks — all core capabilities of the Pulumi Cloud platform — will be positioned to lead in an AI-first world. The gap between teams that modernize and those that do not will widen rapidly in 2026 and beyond.

Try Pulumi for Free

Pulumi Kubernetes Operator v2.3.0: Preview Mode and Structured Configuration

Eron Wright — Tue, 25 Nov 2025 00:00:00 +0000

We’re excited to announce the release of Pulumi Kubernetes Operator v2.3.0, introducing two powerful capabilities that enhance GitOps workflows: preview mode for validating infrastructure changes before deployment, and structured configuration support for managing complex data types. Building on the success of the v2.0 GA release, this update addresses long-standing community requests while maintaining full backwards compatibility. These features enable safer, more sophisticated infrastructure management patterns for platform engineering teams.

Preview mode: Validate infrastructure changes before deployment

Preview mode enables you to run Pulumi stacks in dry-run fashion, allowing you to visualize what infrastructure changes would occur without actually applying them. This capability is essential for GitOps workflows that require validation gates and incremental rollouts.

By adding preview: true to your Stack specification, the operator runs pulumi preview instead of pulumi up. The Stack’s Ready condition indicates preview success, and you get full status including preview links, standard output, and program outputs—all without making actual infrastructure changes.

This unlocks sophisticated workflow patterns:

Multiple Stacks for what-if analysis: Create several Stack resources pointing to the same Pulumi stack, with all-but-one operating in preview mode to compare different configurations
Branch validation: Preview changes from feature branches before merging to production
Tick-tock rollout patterns: Toggle the preview flag on and off with external verification steps between each deployment phase

Here’s a simple example showing a production stack alongside its preview counterpart:

# Production stack
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-infrastructure
spec:
 stack: org/project/prod
 projectRepo: https://github.com/example/infra
 branch: main
---
# Preview stack - same Pulumi stack, preview mode
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-infrastructure-preview
spec:
 stack: org/project/prod
 projectRepo: https://github.com/example/infra
 branch: feature-branch
 preview: true # Only runs pulumi preview

Preview mode closes issue #16, one of our longest-standing feature requests.

Structured configuration: Native support for complex data types

Configuration management takes a significant step forward with native support for complex data types. Previously limited to string values, Stack configuration now supports objects, arrays, numbers, and booleans as first-class citizens.

This enhancement addresses the reality that complex environments require rich configuration. You can now express sophisticated configuration structures inline in your Stack manifests or load them from ConfigMaps with automatic JSON parsing—all using Kubernetes-native patterns.

The implementation leverages Pulumi CLI’s JSON configuration support (available in v3.202.0+) with automatic version detection. If your workspace uses an earlier CLI version, you’ll receive clear guidance to upgrade. Existing string-only configurations continue to work without modification, ensuring full backwards compatibility.

Here’s an example of inline structured configuration:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-app
spec:
 stack: org/app/prod
 projectRepo: https://github.com/example/app
 branch: main
 config:
 # String values (existing behavior)
 environment: "production"

 # Objects (NEW)
 database:
 host: "db.example.com"
 port: 5432
 ssl: true

 # Arrays (NEW)
 regions: ["us-west-2", "us-east-1", "eu-west-1"]

 # Numbers and booleans (NEW)
 maxConnections: 100
 enableCaching: true

You can also reference ConfigMaps for more complex scenarios:

apiVersion: v1
kind: ConfigMap
metadata:
 name: app-settings
data:
 database.json: |
 {
 "host": "db.example.com",
 "port": 5432,
 "maxConnections": 100
 }
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-app
spec:
 stack: org/app/prod
 projectRepo: https://github.com/example/app
 branch: main
 configRefs:
 database:
 name: app-settings
 key: database.json
 json: true # Parse as JSON

Note that Secrets are not a supported source of structured configuration values.

Structured configuration closes issue #258 and issue #872, addressing long-standing configuration management needs from the community.

Bug fixes and reliability improvements

This release includes several fixes that improve operational reliability:

Stack name validation: Added validation to limit Stack names to 42 characters, preventing resource naming conflicts (#899)
secretsProvider fix: The secretsProvider parameter now properly applies when creating new stacks (#935)
Helm chart fix: Resolved YAML parsing errors for podLabels containing special characters like colons (#1014)
Stack deletion: Stack deletion is no longer blocked when prerequisite stacks are missing, enabling proper cleanup workflows (#751)
Update TTL: Completed Update objects now respect the ttlAfterCompleted setting for automatic garbage collection (#960)

Upgrading to v2.3.0

This release includes updates to the Custom Resource Definitions (CRDs) to support the new features. If you’re using Helm, you’ll need to manually apply the updated CRDs before upgrading, as Helm v3 does not automatically upgrade CRDs:

# Apply updated CRDs
kubectl apply --server-side -k 'github.com/pulumi/pulumi-kubernetes-operator//operator/config/crd?ref=v2.3.0'

# Upgrade via Helm
helm upgrade pulumi-kubernetes-operator \
 oci://ghcr.io/pulumi/helm-charts/pulumi-kubernetes-operator \
 --version 2.3.0 \
 -n pulumi-kubernetes-operator

If you’re using the quickstart YAML installation method, the CRDs will update automatically when you apply the new manifest.

All changes are backwards compatible—existing Stack resources work without modification. For structured configuration support, ensure your workspace pods use Pulumi CLI v3.202.0 or later; the operator provides automatic version detection with clear upgrade guidance when needed.

Get started today

The v2.3.0 release enhances the Pulumi Kubernetes Operator with safer deployment workflows and more flexible configuration management. Preview mode enables validation-first GitOps patterns, while structured configuration simplifies complex multi-environment setups.

Get started with the Pulumi Kubernetes Operator in our documentation, or view the complete v2.3.0 release notes on GitHub. Join the conversation in the Pulumi Community Slack #kubernetes channel—we’d love to hear how these features impact your workflows.

Change Management with the Pulumi Kubernetes Operator and Kargo

Elisabeth Lichtie — Tue, 25 Nov 2025 00:00:00 +0000

The Pulumi Kubernetes Operator (PKO) enables you to manage Pulumi stacks as Kubernetes resources, but it doesn’t provide much guidance on change management. Kargo fills this gap by providing controlled, staged promotions with verification steps. Together, they let you keep your infrastructure defined in Pulumi while managing multi-environment rollouts in a systematic way.

This pattern is especially useful for platform teams that need to balance delivery speed with governance as environments and expectations scale.

Going to KubeCon Europe 2026?

Visit Pulumi at Booth 784 for live demos of Pulumi Kubernetes Operator and Kargo workflows for managing infrastructure change at scale.

What you can do with Argo CD, PKO, and Kargo

When you combine these tools, you gain several change management capabilities:

Control release timing: Determine when updates to your Pulumi program are released to different environments, rather than deploying everywhere at once.
Visualize deployments: See which environments are running which versions of your infrastructure code at a glance.
Automatic verification: Automatically verify deployment success before promoting changes to the next environment.
Approval gates: Set up discrete approval requirements before promotion, ensuring human review when needed.
Change tracking: Maintain a clear audit trail of what changed, when, and how it moved through your environments.

About Kargo

Kargo is a continuous promotion platform that manages how changes move through application lifecycle stages. While it’s commonly used for application deployments, it also works for infrastructure code managed by PKO.

The key concepts you’ll work with:

Freight: A packaged set of artifacts that move together through your pipeline. When integrating with PKO, this will typically be the pulumi program that defines your infrastructure and the git commit with the latest version.

Stages: Promotion targets that represent different lifecycle phases (like dev, staging, production). Each stage corresponds to a different PKO Stack resource managing that environment’s infrastructure.

Promotions: The process of moving freight from one stage to the next. Kargo updates the Stack resources to point to the promoted freight.

Warehouses: Monitors that watch your Git repository for infrastructure code changes and package new commits or releases as freight.

Projects: Organizational units that group related stages and warehouses, mapping to Kubernetes namespaces for access control.

When Kargo promotes freight to a stage, it updates the corresponding PKO Stack resource with the new Git reference. PKO then reconciles the stack, and your infrastructure changes are applied.

Example architecture

Workflow

To demonstrate how these tools work together, let’s look at an example setup that manages infrastructure code through a controlled promotion pipeline:

This architecture includes six main components working together:

Security Scanner repository: Contains your Pulumi infrastructure code. Kargo watches this repository for new commits or releases.
Kargo: Controls the release pipeline. When it detects new code in the security scanner repository, it triggers an update to promote that code to the next stage.
Kargo manifests repository: Stores the Kubernetes manifests that define PKO Stack resources. Kargo updates this repository with the new Git references when promoting freight.
Argo CD: Watches the Kargo manifests repository and creates or updates Stack objects in Kubernetes when it detects changes.
Pulumi Kubernetes Operator: Watches Stack objects and deploys them to Pulumi, triggering stack updates or previews.
Pulumi: Performs the actual infrastructure deployments and previews based on PKO’s instructions.

The workflow follows this pattern: Kargo detects new infrastructure code, updates the manifests repository with the new Git reference, Argo CD syncs the Stack objects to Kubernetes, PKO triggers Pulumi to deploy the changes, and Kargo verifies the successful deployment. This creates a closed loop where infrastructure changes are systematically promoted and verified at each stage.

Dashboard and Management

The Kargo dashboard brings operational visibility to this entire workflow. The timeline view at the top shows every freight version moving through your pipeline, making it immediately clear which infrastructure code version is running in each environment and how long it has been there.

The flow diagram above maps out your complete promotion path with color-coded stages. You can see at a glance where changes are flowing smoothly and where they are waiting at approval gates or blocked by failed verifications.

Status indicators on each stage show real-time health information, eliminating the need to manually check multiple systems or run status commands. This single-pane view reduces context switching, speeds up troubleshooting when issues arise, and gives an understanding of exactly what infrastructure is deployed where.

You can see the complete setup for this example, including configuration files and manifests, in the pulumi-operator-with-kargo-change-management repository.

Setting up the environment

The example repository contains three Pulumi projects that deploy in sequence:

cluster: Creates a basic Amazon EKS cluster where everything will run.
cluster-setup: Installs the required dependencies including PKO, cert-manager, Argo CD, Argo Rollouts, and AWS Cognito for authentication.
kargo: Deploys Kargo via Helm and provides examples to set up the project with stages, approval gates, and analysis templates for the promotion pipeline.

Configuring Kargo

Once Kargo is installed, you need to configure several components to set up your promotion pipeline.

Creating a project

Start by creating a Kargo project, which automatically creates a corresponding Kubernetes namespace.

Git credentials

Kargo needs access to your Git repositories to monitor for changes and update manifests. Create a secret with your Git credentials, labeled so Kargo recognizes it:

apiVersion: v1
kind: Secret
metadata:
 name: github-creds
 namespace: kargo-managed-stack
 labels:
 kargo.akuity.io/cred-type: git
stringData:
 repoURL: https://github.com/YOUR_REPO
 username: YOUR_USERNAME
 password: GITHUB_TOKEN

Setting up approver roles

To control who can approve promotions at approval gates, you can create custom roles with the promote permission. This requires three resources: a ServiceAccount with OIDC claim mapping, a Role with promotion permissions, and a RoleBinding to connect them. The easiest way to set up these resources is through the Kargo UI.

The screenshot shows a role configured with the promote verb on specific stages (like approvalgateqa), which allows users bound to this role to approve promotions at that gate. The role also includes permissions to create promotions and view other resources.

Project configuration

Use a ProjectConfig resource to control promotion policies for your project. This example enables automatic promotion for preview and deployment stages:

apiVersion: kargo.akuity.io/v1alpha1
kind: ProjectConfig
metadata:
 name: kargo-managed-stack
 namespace: kargo-managed-stack
spec:
 promotionPolicies:
 - stageSelector:
 name: devpreview
 autoPromotionEnabled: true
 - stageSelector:
 name: dev2preview
 autoPromotionEnabled: true
 - stageSelector:
 name: dev
 autoPromotionEnabled: true
 - stageSelector:
 name: dev2
 autoPromotionEnabled: true
 - stageSelector:
 name: qapreview
 autoPromotionEnabled: true
 - stageSelector:
 name: qa
 autoPromotionEnabled: true

This configuration automatically promotes freight through preview stages and dev environments, while approval gates remain manual.

Pulumi access token

Kargo needs access to the Pulumi API to verify that infrastructure deployments completed successfully. Create a secret in your project through the Kargo UI:

Navigate to your project in the Kargo UI
Navigate to Settings → Credentials
Select Create a new credential with:
- Name: pulumi-api-token
- Type: Opaque
- Key: PULUMI_ACCESS_TOKEN
- Value: Your Pulumi access token

This token allows Kargo’s analysis templates to query the Pulumi API and confirm that stack updates and previews ran successfully before promoting freight to the next stage.

Defining stages

Stages represent different points in your promotion pipeline. This example uses three types of stages to control how infrastructure changes move from development to production.

Preview stages

Preview stages run pulumi preview operations to show what changes would be applied without actually deploying them. These stages help catch issues early by validating that the infrastructure code is valid and showing the expected changes:

devpreview and dev2preview: Run previews for the dev environments
qapreview: Runs a preview for QA after passing approval gates

Preview stages automatically promote to their corresponding update stages once the preview completes successfully and verification passes.

Update stages

Update stages run pulumi up to actually deploy the infrastructure changes:

dev and dev2: Deploy changes to parallel dev environments for testing
qa: Deploy to the QA environment after all approvals

Update stages are configured to automatically promote once the deployment succeeds and verification confirms the infrastructure is in the expected state.

Approval gates

Approval gates are stages that require manual approval before freight can proceed. They act as checkpoints where designated approvers review changes before they move forward:

approvalgatedev: Manual checkpoint before dev deployments
approvalgateqa: Functional review gate before QA deployment
approvalgateqasec: Security review gate before QA deployment

The QA preview stage uses availabilityStrategy: All, meaning freight must pass through both approvalgateqa and approvalgateqasec before proceeding. This ensures both functional and security teams approve changes before they reach QA.

Custom approver roles

Different approval gates may require different approvers. Using the custom roles described earlier, you can grant specific users or groups the promote permission on individual approval gates. For example, security team members might have the promote permission on approvalgateqasec, while functional testers have it on approvalgateqa. This ensures the right people review changes at each checkpoint.

Manage Kubernetes change with Pulumi

Define your Pulumi stacks as Kubernetes resources, then promote and verify infrastructure changes across dev, staging, and production with controlled, staged rollouts.

Get started

Verification

After each stage completes, Kargo can verify that the operation succeeded before promoting to the next stage. This example uses analysis templates to query the Pulumi API and confirm deployments completed successfully.

Kargo uses Argo Rollouts for verification, so Argo Rollouts must be installed in your cluster.

Analysis templates

Analysis templates define how to verify a stage. They can make HTTP requests, run queries, or execute any checks you need. For Pulumi verification, the templates:

Wait for operations to complete: Include a delay to give Pulumi time to finish the preview or update operation
Query the Pulumi API: Make HTTP requests to the Pulumi API to retrieve the stack’s latest update information
Verify the commit ID: Check that the deployed commit matches the freight being promoted
Confirm success: Ensure the operation completed without errors

The analysis template uses the PULUMI_ACCESS_TOKEN secret created earlier to authenticate with the Pulumi API.

Custom verification standards

Analysis templates are flexible enough to enforce other standards your organization requires as well.

This gives you a way to codify your deployment standards and ensure they’re enforced automatically as infrastructure changes move through environments.

Conclusion

Combining the Pulumi Kubernetes Operator with Kargo gives you systematic control over how infrastructure changes move through your environments. Rather than manually managing deployments or pushing changes directly to production, you get automated promotions with verification steps, approval gates where human review is needed, and a clear audit trail of what changed and when. This approach is particularly valuable for teams managing multiple environments where infrastructure changes need to be tested and approved before reaching production. By defining your promotion pipeline as code with stages, approval gates, and verification templates, you ensure that your infrastructure changes follow the same rigorous process every time.

If you want to put this pattern into practice, you can see it live at KubeCon Europe 2026 (Booth 784) or register for our upcoming Zero to Production in Kubernetes workshop.

Try Pulumi for Free

Beyond YAML in Kubernetes: The 2026 Automation Era

Sara Huddleston — Wed, 12 Nov 2025 07:09:46 +0000

Kubernetes continues to evolve, powering not only applications but entire AI and ML systems across clouds, edges, and enterprises. By 2026, DevOps engineers, SREs, cloud engineers, and platform teams face growing pressure to deliver faster, smarter, and more secure infrastructure at scale.

Kubernetes automation is entering a new era where infrastructure as code, policy enforcement, and AI-driven orchestration work together to manage cloud environments intelligently.

Pulumi’s 2025 advancements, including Pulumi Kubernetes Operator 2.0 GA, new Kubernetes best practices playbooks, Pulumi Neo for AI assisted infrastructure management, and Policy Automation, set the foundation for a new era of Kubernetes automation that extends across every role involved in managing modern infrastructure.

In this Kubernetes article:

Why Kubernetes Needs to Go Beyond YAML

The article Kubernetes Best Practices I Wish I Had Known Before highlights a key challenge: relying solely on YAML as the source of truth for Kubernetes is no longer sustainable. Clusters are dynamic, environments multiply, and static YAML files cannot keep up with the complexity of modern infrastructure.

Teams across disciplines face similar challenges:

Multi-cluster and hybrid-cloud sprawl
Inefficient manual configuration and drift
Lack of policy enforcement and governance at scale
Secrets scattered across systems

Pulumi addresses these challenges by introducing general-purpose programming languages such as TypeScript, Python, Go, C#, and Java into Kubernetes management. This approach enables teams to define, test, and share reusable infrastructure code, bridging the gap between declarative manifests and modern software engineering practices.

Learn more: Pulumi Kubernetes documentation

The 2026 Convergence of AI, Platforms, and Policy in Kubernetes

The CNCF State of Cloud Native Development Q3 2025 report shows that more than half of DevOps professionals and nearly a third of all developers now identify as cloud native. Adoption of hybrid and multi-cloud architectures continues to grow, while AI and ML workloads are becoming first-class citizens in Kubernetes environments. The report notes that 41% of professional ML and AI developers are cloud native, confirming that Kubernetes has become a foundational technology for building intelligent, scalable systems.

This trend highlights how the next generation of Kubernetes operations is evolving. Teams need unified platforms that simplify AI-driven workloads, automatically enforce governance, and reduce operational complexity across environments. Pulumi already delivers this through an integrated platform that brings automation, security, and developer productivity together in one place.

Pulumi’s platform provides a complete foundation for intelligent infrastructure management that extends from development to production.

Pulumi Neo adds AI-assisted infrastructure management that turns natural language into production-ready Kubernetes and cloud code.
Pulumi IDP enables engineering and platform teams to build self-service environments that abstract complexity for developers while maintaining consistency and control.
Pulumi Policies brings continuous compliance and policy enforcement directly into the delivery workflow.
Pulumi ESC secures credentials, API keys, and sensitive configurations across Kubernetes and cloud environments.

Together, these capabilities form a unified automation and governance layer for Kubernetes and cloud native systems.

The result is a model where infrastructure, policy, and developer experience work together to power secure, scalable, and AI-ready platforms that meet the needs of engineering teams of every size and discipline, from application development to security and cloud operations.

The 2026 Shift: AI-Assisted Kubernetes Operations

The next phase of Kubernetes management will be AI-driven, context-aware, and self-healing. Infrastructure will not only follow instructions but also understand intent. This is the vision behind Pulumi Neo, an AI Infrastructure Agent designed to help teams automate complex systems.

Pulumi Neo can interpret natural-language requests such as “deploy a GPU-backed EKS cluster with three node groups,” generate infrastructure code that adheres to organizational policies, and continuously refine that code as environments change.

AI-powered observability and decision-making. According to the CNCF Annual Survey, 93% of organizations already use or plan to adopt AI-driven monitoring and predictive analysis for Kubernetes environments. The goal is to identify performance and reliability issues before they affect users. Pulumi Policies assists to get clean and stay clean, while Neo extends this capability by acting on those insights, transforming detected issues into actionable infrastructure updates that teams can validate or deploy.
ML and GPU workloads on Kubernetes. As organizations expand AI and ML pipelines across clusters, automation and cost efficiency become critical (FinOps Foundation). Pulumi Insights is an intelligent infrastructure management service that helps you discover, understand, manage, and improve your infrastructure. Insights improves security, compliance, and efficiency through AI-powered asset and compliance management.
Unifying DevOps, SRE, and MLOps workflows. The convergence of software and model delivery continues to accelerate (TechRadar). Pulumi’s code-based approach, combined with Neo’s agentic reasoning, creates a unified workflow for infrastructure, applications, and AI systems.

Additional insights and demonstrations:

Kubernetes and Pulumi Neo together represent a future of autonomous infrastructure management, where AI assists teams in deploying, maintaining, and improving their environments intelligently and securely.

Operator-First: Kubernetes Deploys Your Cloud

The Pulumi Kubernetes Operator 2.0 GA introduced a Kubernetes-native approach to infrastructure management. Each Pulumi stack becomes a Kubernetes Custom Resource, allowing Kubernetes itself to execute Pulumi programs written in any supported language. This enables:

Management of AWS, Azure, and GCP infrastructure from within the cluster
Integration with GitOps systems such as Argo CD and Flux
Continuous reconciliation and drift detection through Pulumi’s state and policy engine

With the new Pulumi Kubernetes Operator 2.3, the operator gets even more production-ready features:

Faster and more resilient stack updates through improved workspace lifecycle management
Support for multi-namespace and multi-tenant deployments
Better visibility with enhanced events, status reporting, and error surfacing
Performance improvements for large-scale environments and parallel stack operations

Documentation: Using the Pulumi Kubernetes Operator

Kubernetes beyond YAML

Define clusters, workloads, and policies in TypeScript, Python, or Go, and manage EKS, AKS, and GKE from a single code-first workflow.

Get started

Intelligent Infrastructure Across Every Cloud

Kubernetes has become the control plane for everything from application deployments to AI and ML workloads across clouds and environments. Engineering teams are running clusters on Amazon EKS, Microsoft AKS, and Google GKE while also managing edge, hybrid, and on-premises environments. Pulumi extends Kubernetes automation across all of them, unifying clusters, workloads, and cloud resources under a consistent model of infrastructure as code.

With Pulumi, teams can use familiar programming languages to define Kubernetes resources, cloud infrastructure, and policies together. This approach provides a single workflow for managing compute, networking, storage, and identity across multiple clouds without relying on brittle YAML templates. It enables consistent provisioning, policy enforcement, and automation across every cluster and environment.

By treating Kubernetes as the universal control plane for cloud infrastructure, Pulumi gives teams a scalable foundation that adapts to any workload and environment. The result is an intelligent, multi-cloud Kubernetes infrastructure that combines the flexibility of cloud providers with the reliability of modern automation.

Bring Your YAML and Helm, Then Evolve

Teams do not need to start from scratch. Pulumi supports importing existing YAML manifests and Helm charts to help organizations adopt a code-first approach incrementally. Many teams begin by wrapping existing manifests in Pulumi code, then refactor them into reusable components that enforce best practices and compliance policies.

This hybrid approach enables modernization without disrupting existing CI/CD pipelines or team workflows.

Begin Your Kubernetes Automation Journey

For teams preparing for the next phase of Kubernetes management in 2026:

Get Started with Kubernetes to create your first Pulumi program.
Add the Pulumi Kubernetes Operator to enable infrastructure deployments from within your clusters.
Integrate GitOps workflows with Argo CD, Flux, or Jenkins X.
Apply policy guardrails to enforce security and compliance automatically.
Refactor infrastructure into reusable components for consistent, scalable operations.

Agentic Workflows for Production-ready Kubernetes

Experience Kubernetes automation in practice. See how agentic workflows power production-ready Kubernetes by automating GitOps operations, managing environments, and reducing manual effort in this on-demand workshop.

Watch Agentic Workflows for Production-ready Kubernetes to learn how to:

Provision and manage clusters across clouds using general-purpose programming languages
Multi-cloud Kubernetes infrastructure management
Fleet management with GitOps (Argo CD)
Reduce YAML complexity while maintaining reliability and speed

Watch On Demand.

Final Thoughts

Kubernetes in 2026 and beyond is not just for platform engineers. It is for DevOps professionals, SREs, and cloud teams, big and small, responsible for maintaining infrastructure security, reliability, and performance. Pulumi unifies infrastructure as code, secrets management, policy governance, and AI automation into a single platform, enabling every team to adopt an intelligent, code-driven approach to Kubernetes that scales across workloads, clouds, and the AI-driven future.

Try Pulumi for Free

Platform Engineering Buffet at SEITENBAU

Nico Thomas — Tue, 27 May 2025 00:00:00 +0000

This post is based on my video interview with Nico Thomas, a platform engineer at SEITENBAU. In this article, Nico shares how he and his team built a flexible, reusable platform to support 20+ diverse projects using Pulumi. Watch the video below or read on for Nico’s take on building an infrastructure buffet. - Adam Gordon Bell

Imagine your company isn’t Spotify with one product pipeline, but 20 independent government projects running on Ansible-provisioned VMs—some on Kubernetes, some delivered into other customer data centers, some operated by clients themselves. Each team is building its own CI/CD, secrets management, and integrations. You’re left with a million reinvented wheels.

That was the situation my team and I faced at SEITENBAU, a German software company specializing in custom development for both public and private sectors. Unlike the typical platform engineering stories—where teams build infrastructure for a single product with multiple microservices—we had a fundamentally different challenge: How do you build a platform that serves dozens of completely independent projects, each with their own technology choices, deployment targets, and operational models?

The answer? Stop trying to force everyone onto the same plate. Instead, build a buffet.

The Buffet Philosophy

Think of a traditional platform as a set menu—also known as prix fixe—where everyone gets the same courses, prepared the same way, in the same order. But what if your diners include vegans, carnivores, and people with various dietary restrictions? What if some want a quick snack while others need a seven-course meal?

That was exactly the situation we faced. We had:

Government projects that needed to run on-premises due to data sovereignty requirements.
Modern cloud-native applications leveraging Kubernetes and GitOps.
Legacy systems still operating on virtual machines managed with Ansible.
Projects we ran as managed services and others we simply delivered to customers.
Teams using various CI/CD tools, artifact repositories, and deployment strategies.

A one-size-fits-all platform would have been a disaster. Instead, my team built a platform offering a rich selection of pre-configured, production-ready components. Teams could mix and match according to their specific needs, creating their own unique infrastructure combinations.

The Challenge: Beyond Microservices

Most platform engineering resources assume you’re building infrastructure for a single product – Spotify’s Backstage, for example. But what happens when you’re dealing with over 20 completely independent projects, each with distinct requirements?

Each project is distinct, featuring its own:

Technology stack (Java predominates, but we support various frameworks and languages).
Infrastructure requirements (cloud, on-premises, or hybrid setups).
Security and compliance constraints.
Operational models (managed by us, the client, or jointly).
Release cadences and deployment strategies.

Initially, this diversity seemed chaotic, but we quickly noticed that underneath these variations, common patterns emerged. Every project consistently required source control, CI/CD pipelines, artifact storage, secrets management, and deployment targets. The critical realization was that we didn’t need to standardize what teams chose—we needed to standardize how they connected these building blocks.

That’s where Pulumi became essential. Its flexibility allowed us to construct an infrastructure buffet that genuinely accommodated the diverse appetites and needs of every team.

Building the Kitchen: Architecture with Pulumi

Behind every great buffet is a well-organized kitchen. For our infrastructure, that kitchen is powered by Pulumi and Python, orchestrating a complex web of infrastructure services, configurations, and deployments. Using Pulumi together with Python has proven extremely valuable—it makes our day-to-day tasks easier to maintain and understand.

The Multi-Stack Approach

Rather than creating a single monolithic infrastructure program, we adopted a modular approach with multiple Pulumi stacks. This decision was essential for managing complexity and empowering team autonomy.

Our architecture comprises several specialized Pulumi programs:

GitLab System: Manages installations and configurations of GitLab.
Project Configuration: Sets up team-specific GitLab groups, CI/CD runners, and agents.
Cluster Management: Provisions Kubernetes clusters complete with standard add-ons.
VM Infrastructure: Manages traditional VM-based deployments utilizing Ansible.
Platform Services: Configures shared services such as artifact repositories and identity providers.

Each program can maintain multiple stacks (dev, staging, prod), allowing us to manage various environments independently while leveraging shared infrastructure code.

From Ingredients to Ready-Made Dishes: Reusable Components

The real magic happens in our component library. Just as a buffet offers complete dishes rather than raw ingredients, we’ve built high-level components that encapsulate complex infrastructure patterns.

Example: The Kubernetes Storage Component

Consider our Longhorn storage component. Instead of requiring teams to understand Helm charts, storage classes, and snapshot configurations, they simply declare they need persistent storage.

The component handles all the complexity: installing Longhorn, configuring appropriate replication for test vs. production environments, setting up snapshot classes, and integrating with our internal certificate authority.

Two Years Later: The Buffet is Open

After two years of running this platform in production, we’ve learned what works and what we’d do differently.

Building a platform for diverse, independent projects requires a different mindset than traditional platform engineering. Instead of forcing standardization, we’ve embraced flexibility while maintaining consistency where it matters.

The buffet approach—powered by Pulumi’s flexibility and Python’s expressiveness—has allowed us to serve 20+ different projects efficiently while giving teams the autonomy they need. It’s not about having one perfect dish; it’s about having a kitchen flexible enough to satisfy everyone.

If you’re facing similar challenges with diverse project requirements, consider building a buffet instead of a prix fixe menu. Your teams will thank you for it.

Want to build your own infrastructure buffet? Get started with Pulumi and join our community Slack to connect with engineers solving similar challenges.

AI/ML on Kubernetes: Deploying Models with Pulumi on Google Cloud

Sara Huddleston — Mon, 24 Mar 2025 07:32:19 +0000

Kubernetes has transformed cloud infrastructure by enabling scalable, containerized applications. While it initially gained traction for managing web applications and microservices, its capabilities now extend to AI/ML workloads, making it the go-to platform for data scientists and machine learning engineers.

Running AI/ML workloads on Kubernetes presents unique challenges, including:

Specialized hardware requirements (e.g., GPUs, TPUs)
Scalability for model training and inference
Complex data pipelines that integrate various cloud services
Infrastructure automation for seamless deployment

Google Cloud Kubernetes (GKE) provides a robust foundation for AI/ML workloads, but managing infrastructure manually can be cumbersome. This is where Pulumi comes in—enabling Infrastructure as Code (IaC) to automate and simplify AI/ML infrastructure on Kubernetes.

Pulumi: Automating AI/ML Infrastructure on Google Cloud

Pulumi is a modern Infrastructure as Code (IaC) tool that allows teams to define and manage cloud infrastructure using general-purpose programming languages like Python, TypeScript, and Go. This approach is particularly beneficial for AI/ML teams, as Python is already the dominant language in data science and machine learning.

With Pulumi, you can:

Provision and scale Kubernetes clusters on Google Cloud automatically.
Define AI/ML environments as code, making deployments repeatable and version-controlled.
Integrate infrastructure with machine learning pipelines, reducing operational overhead.

Deploying AI/ML Workloads on Kubernetes

Below, we explore two use cases for running AI/ML workloads on Google Kubernetes Engine (GKE) using Pulumi.

Use Case 1: Deploying a Large Language Model (LLM) with Retrieval Augmented Generation (RAG)

Large Language Models (LLMs) like GPT-3, Whisper, and DALL-E require significant infrastructure for training and inference. Retrieval Augmented Generation (RAG) enhances LLMs by integrating external knowledge sources, improving accuracy and relevance.

Using Pulumi, you can automate the deployment of an open-source LLM with RAG on Kubernetes.

Step 1: Set Up a Kubernetes Cluster on Google Cloud

import pulumi
import pulumi_gcp as gcp

# Create a GKE cluster for AI/ML workloads
cluster = gcp.container.Cluster("ml-cluster",
 location="us-central1",
 initial_node_count=3,
 node_version="1.23",
 min_master_version="1.23")

# Create a node pool optimized for ML workloads
node_pool = gcp.container.NodePool("ml-node-pool",
 cluster=cluster.name,
 node_config=gcp.container.NodePoolNodeConfigArgs(
 machine_type="n1-standard-4",
 oauth_scopes=[
 "https://www.googleapis.com/auth/devstorage.read_only",
 "https://www.googleapis.com/auth/logging.write",
 ],
 labels={"team": "ml"},
 shielded_instance_config=gcp.container.NodePoolNodeConfigShieldedInstanceConfigArgs(
 enable_secure_boot=True,
 enable_integrity_monitoring=True,
 ),
 ),
 initial_node_count=2,
 location="us-central1",
 version="1.23")
...

This Pulumi script:

Provisions a Kubernetes cluster on Google Cloud.
Creates a dedicated node pool optimized for AI/ML workloads.
Ensures security best practices for machine learning environments.

Step 2: Deploy the LLM with RAG Model

Once the cluster is set up, we can deploy the LLM with RAG model using Pulumi’s Kubernetes provider:

Define Kubernetes Deployments for the LLM and RAG model.
Package models as Docker containers and deploy them to the cluster.
Configure Kubernetes Services to expose APIs for model inference.
Use ConfigMaps and Secrets to manage parameters and credentials.

By defining these resources in Pulumi, deployments become fully automated, repeatable, and scalable.

Use Case 2: Training and Serving Custom Machine Learning Models

Beyond pre-trained LLMs, Kubernetes is ideal for training and serving custom AI/ML models. Pulumi can help automate every stage of the ML lifecycle.

Step 1: Set Up the Model Training Environment

Using Pulumi, we define a training environment with:

A Kubernetes Deployment for training jobs (GPU-enabled).
Persistent Volume Claims (PVCs) for storing training data and model artifacts.
Monitoring tools for tracking performance.

Step 2: Deploy and Serve the Trained Model

Once the model is trained, Pulumi can be used to:

Deploy the trained model as a Kubernetes Deployment.
Expose the model via a Kubernetes Service (REST API or gRPC).
Add autoscaling rules for dynamic inference scaling.

Pulumi allows teams to manage the entire AI/ML pipeline in a structured and automated way.

Try Jay’s demo code on Creating an AI Training Platform on GKE with Pulumi.

Why Use Pulumi for AI/ML on Kubernetes?

Pulumi provides several advantages for AI/ML teams running workloads on Kubernetes:

1. Use General-Purpose Languages for Infrastructure as Code

Most AI/ML engineers already work with Python or Go, and Pulumi lets them manage infrastructure using the same language.
No need to learn YAML or Kubernetes manifests—define everything programmatically.

2. Automate AI/ML Workflows

Define infrastructure, training jobs, and model serving in one unified IaC framework.
Ensure consistency across development, staging, and production environments.

3. Improve Scalability and Cost Efficiency

Pulumi integrates with Google Cloud AI services for optimized compute resources.
Automate autoscaling and resource allocation for AI/ML workloads.

4. Increase Security and Compliance

Manage credentials and secrets securely with Pulumi ESC (Secrets Management).
Apply policy-as-code to enforce security best practices.

Get Started with AI/ML on Kubernetes with Pulumi

Pulumi makes it easy to deploy, scale, and manage AI/ML workloads on Kubernetes, leveraging Google Cloud’s AI infrastructure. Whether you’re serving LLMs, training custom models, or automating ML pipelines, Pulumi provides a developer-friendly, scalable, and secure solution.

By combining Kubernetes, Google Cloud, and Pulumi, you can accelerate AI/ML innovation while reducing infrastructure complexity.

Why Every Cloud Engineer Needs Pulumi ESC for Secrets Management

Sara Huddleston — Mon, 17 Mar 2025 07:28:56 +0000

Managing secrets is one of the most critical responsibilities in cloud engineering. Secrets like API keys, database credentials, and encryption tokens are the backbone of secure and seamless cloud operations. Yet they are so often an afterthought. They get replicated across cloud-specific secrets managers and stuffed in GitHub secrets, compromising security for the sake of simplicity. ¿Por que no los dos? Why can’t secrets management be secure and simple?

Enter Pulumi ESC (Environments, Secrets, and Configuration)—a breakthrough in taming secrets sprawl and streamlining configuration management across infrastructure. Let’s explore why Pulumi ESC is a necessity for cloud engineers, helping make secrets management secure while keeping it simple.

The Challenge of Secrets Management in Modern Cloud Environments

Secrets management has changed significantly over the past decade. Gone are the days when secrets could be hardcoded, manually maintained in static configuration files, or left floating around in plaintext on dev machines. Nowadays, cloud environments demand a higher level of sophistication:

Distributed Systems: Even in multi-cloud and hybrid setups, secrets must be accessible across varied platforms without exposing vulnerabilities.
Dynamic Infrastructure: Kubernetes, serverless architectures, and ephemeral environments must have secrets that adapt dynamically.
Security Risks: Hardcoded or poorly managed secrets can lead to catastrophic breaches, costing companies millions in data recovery and compliance penalties.
Operational Burdens: Manual secrets management is error-prone, inefficient, unscalable, and needs to align with DevOps and DevSecOps best practices.

Pulumi ESC addresses these issues head-on, redefining how cloud engineers manage secrets efficiently, securely, and at scale.

Jk Jensen, Team Lead at Mysten Labs, said:

“Pulumi ESC has been a lifesaver for us. It’s nice to throw everything behind an ESC environment and eliminate one-off granting/IAM permissions and other issues related to static credentials. It gives us peace of mind knowing that we can grant permissions quickly and revoke easily limiting blast radius for any access.”

What is Pulumi ESC?

Pulumi ESC simplifies how organizations manage secrets and configurations. It is designed to secure sensitive configurations across modern cloud environments and supports seamless integration, enabling engineers to:

Access, share, and manage secrets, passwords, API keys, and configuration settings like network and deployment options.
Synchronize secrets and configuration from any secrets manager to any app, tool, or CI/CD platform.
Access secrets securely through CLI, API, Kubernetes operator, the Pulumi Cloud UI, and in-code with Typescript/Javascript, Python, and Go SDKs.
Connect to cloud providers and secrets stores via OIDC to generate dynamic, short-lived, auto-expiring credentials.
Set role-based access controls (RBAC), making securing secrets and configurations easy by assigning permissions to users based on their roles.

Whether integrated with Pulumi’s Infrastructure as Code (IaC) platform or used as a standalone service, Pulumi ESC enables cloud engineers to streamline secrets management with centralized control.

Key Features of Pulumi ESC

1. Seamless Integration with External Platforms

Pulumi ESC integrates with popular secrets providers, including AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, 1Password, and HashiCorp Vault, making it adaptable for multi-cloud and hybrid cloud architectures.

2. Dynamic Secrets Synchronization with ESO

Partnering with the External Secrets Operator, Pulumi ESC synchronizes secrets securely into Kubernetes clusters. This eliminates hardcoding secrets into manifests or relying on unsecured manual processes.

3. Safely Roll Back to a Previous Version

Pulumi ESC Versioning gives you unprecedented control over your secrets and configuration. Every change is captured in an immutable revision history, allowing you to audit modifications, compare versions, and safely roll back.

4. Secure by Design

Pulumi ESC follows a “secure by default” model, employing encryption, access control, and detailed audit trails. Engineers can meet compliance regulations effortlessly while gaining full visibility into secret access patterns.

5. Automated Rotation and Expiry

Pulumi ESC minimizes security risks by automating the rotation of secrets. This feature aligns secrets management with CI/CD processes for cloud engineers focused on DevOps, ensuring credentials remain valid only when needed.

6. Configuration-as-Code, Automation, and Integration Everywhere

Pulumi ESC embraces an “as-code” approach, enabling configuration and secrets management using TypeScript, JavaScript, Go, Python, or YAML. The ’esc’ CLI and API support automation in CI/CD environments, reducing credential duplication and ensuring a single source of truth.

7. Dev Tools Integrations

Pulumi ESC’s metadata and support for popular configuration formats enable seamless integration with tools like Direnv, Docker, and GitHub, allowing easy management of environment variables, secrets, and configurations.

8. Infrastructure Tools Integrations

Pulumi ESC extends its capabilities beyond Pulumi IaC by integrating with other infrastructure tools such as Cloudflare, Terraform, and OpenTofu. These integrations enable seamless provisioning of cloud credentials and input variables directly from ESC environments.

Why Cloud Engineers Need Pulumi ESC

Enhanced Security and Compliance

Cloud engineers often have to balance security and operations. Pulumi ESC centralizes secrets in a secure vault, ensuring encrypted storage, access control, and audit visibility. It also complies with industry standards like SOC 2, PCI-DSS, and MTCS.

Streamlined Multi-Cloud Management

Managing secrets across AWS, Azure, and GCP can quickly become chaotic. Pulumi ESC’s cross-platform integration simplifies this process, ensuring engineers can manage and sync secrets from a single interface.

Simplified Kubernetes Secrets

Kubernetes’ default secrets offer limited security and scalability. Pulumi ESC with ESO and Secrets Store CSI Driver overcomes these limitations by securely synchronizing and managing secrets within Kubernetes clusters. This prevents unauthorized access and provides automated updates.

Zero Downtime Through Automation

Manual secrets management often leads to errors such as expired credentials or outdated tokens. Pulumi ESC automates the entire lifecycle of secrets—creation, rotation, replication, and expiry—guaranteeing uninterrupted services.

Developer-Friendly Workflows

Tools should make engineers’ lives easier, not harder. Pulumi ESC’s CLI, SDKs, and API provide intuitive ways to integrate into existing workflows. For cloud engineers leveraging Infrastructure as Code with Pulumi, managing secrets alongside the stack becomes effortless.

Thomas Meckel, Senior Cloud Architect at Ophios GmbH, explains:

“Pulumi’s built-in encryption of secrets sets it apart as a leader in secure infrastructure management. Unlike other tools that leave sensitive data exposed or require complex configurations, Pulumi ensures secrets are encrypted by default, leveraging flexible options like Azure Key Vault or AWS KMS. This eliminates the risk of plaintext exposure and simplifies secure deployments. For any organization serious about cloud security, Pulumi’s approach to secrets management is a critical differentiator, combining ease of use with robust protection against breaches.”

Kubernetes Secrets Management

For secrets management in Kubernetes environments, Pulumi ESC becomes even more powerful when paired with External Secrets Operator (ESO) or the Secrets Store CSI Driver. Here’s how it works:

Using Pulumi ESC with External Secrets Operator

External Secrets Operator is an open-source Kubernetes operator that syncs secrets from external providers (like Pulumi ESC) into Kubernetes as native secrets.

Centralized Storage: Store secrets securely in Pulumi ESC and synchronize them with Kubernetes across multiple clusters.
Dynamic Updates: Whenever a secret is updated in ESC, ESO automatically replicates the changes into Kubernetes, eliminating manual intervention.
Granular Management: Define which secrets are synchronized and to which namespaces, ensuring tight access controls and minimizing risk.
Automated Secret Rotation: By leveraging ESC’s rotation capabilities, ESO ensures Kubernetes receives refreshed credentials without requiring downtime.

Learn how to manage Kubernetes secrets with Pulumi ESC and External Secrets Operator: The Perfect Solution for Today’s Cloud-Native Secret Management.

Using Pulumi ESC with Secrets Store CSI Driver

The Secret Store CSI Driver mounts secrets, certificates, and keys from external stores into Kubernetes pods as volumes.

Automated Secrets Injection: Secrets Store CSI Driver enables Pulumi ESC to automatically inject secrets into Kubernetes pods as mounted volumes or environment variables. This reduces the manual overhead of managing secrets directly in Kubernetes.
Secure and Dynamic Secret Access: By leveraging Pulumi ESC, Kubernetes applications can securely fetch secrets from external providers, ensuring dynamic access without exposing credentials in pod specifications.
Streamlined Operations: This integration simplifies the process of synchronizing secrets from Pulumi ESC with Kubernetes-native constructs. The automation reduces errors and boosts efficiency.

Learn how to Master Kubernetes Secrets with Pulumi ESC + Secrets Store CSI Driver.

The Future of Secrets Management with Pulumi ESC

Pulumi ESC is more than just a secrets management tool — it’s the foundation of secure, scalable, and agile cloud operations. For cloud engineers grappling with multi-cloud complexity, Kubernetes deployments, and compliance concerns, it provides a streamlined, automated approach that enhances efficiency and security.

With rising cyber threats and stricter data regulations, adopting tools like Pulumi ESC is no longer optional. It’s a competitive advantage for cloud engineers and organizations aiming to lead in today’s cloud-native world.

Next Steps to Secure Your Secrets

Secrets management doesn’t have to be a headache! With Pulumi ESC, you can safeguard your infrastructure, ensure compliance, and eliminate manual errors—all while enhancing security and agility.

Get started with Pulumi ESC today and experience the future of secrets management. Take control of your cloud environment—securely and effortlessly.

Pulumi Kubernetes Operator 2.0 is Now Generally Available!

Eron Wright — Thu, 20 Feb 2025 00:00:00 +0000

Today marks an exciting milestone as we announce the General Availability (GA) release of the Pulumi Kubernetes Operator 2.0! This release builds upon the foundation we laid during the beta phase, delivering a production-ready solution that transforms how teams manage their cloud infrastructure.

Learn more about the Pulumi Kubernetes Operator in our documentation

What Is the Pulumi Kubernetes Operator?

The Pulumi Kubernetes Operator brings infrastructure automation directly to your Kubernetes cluster. By treating your infrastructure as native Kubernetes resources, it enables teams to manage their entire cloud environment—from databases to DNS records—using familiar Kubernetes workflows and tools.

Imagine having a dedicated infrastructure automation engine running within your cluster, continuously ensuring your cloud resources stay in sync with your desired state. That’s exactly what the Pulumi Kubernetes Operator delivers, complete with the observability and control you need for production environments.

The Evolution to 2.0

Pulumi Kubernetes Operator 2.0 introduces a completely new architecture for running Pulumi programs. The key changes include:

Each Stack now runs in its own dedicated Workspace pod, rather than within the Operator pod
A new Workspace custom resource manages the Workspace pod lifecycle
Stack operations are coordinated through a new Update custom resource
The Stack custom resource API remains the primary interface, maintaining backward compatibility

This architecture delivers improved resource isolation, better secrets management, and flexible Workspace customization options. The pod-per-stack model ensures reliable resource allocation and clear operational boundaries for each deployment.

What’s New in GA?

The GA release introduces three powerful capabilities that give you more control over your deployments:

Enhanced Logging Control

You can now set the log verbosity level of the Pulumi CLI for any given stack or Workspace. This feature offers granular control over logging output, making debugging and fine-tuning your deployment process more efficient.

New Controller Events for Better Instrumentation

Additional controller events have been introduced to provide richer instrumentation and observability. These events offer improved insights into the operator’s behavior, enabling more effective monitoring of your infrastructure.

Workspace Reclaim Policy

Managing Workspace lifecycles is now even more flexible with the new workspaceReclaimPolicy field in the Stack specification. This enhancement allows you to define how workspace resources should be handled after stack deployment. The supported policies are:

Retain: Keeps the Workspace and its resources after deployment (default)
Delete: Automatically removes the Workspace once the stack is fully deployed

This gives you fine-grained control over resource cleanup and helps prevent workspace sprawl in your cluster.

See It in Action

Here’s an example that demonstrates how to deploy a Pulumi stack using the operator. This configuration creates a service account with appropriate permissions and defines a Stack resource that deploys a sample project from the Pulumi examples repository:

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as pulumiservice from "@pulumi/pulumiservice";

// Create a Kubernetes ServiceAccount for the Pulumi Workspace pod
const sa = new k8s.core.v1.ServiceAccount("random-yaml", {});

// Grant system:auth-delegator to the ServiceAccount
const crb = new k8s.rbac.v1.ClusterRoleBinding("random-yaml:system:auth-delegator", {
 roleRef: {
 apiGroup: "rbac.authorization.k8s.io",
 kind: "ClusterRole",
 name: "system:auth-delegator",
 },
 subjects: [{
 kind: "ServiceAccount",
 name: sa.metadata.name,
 namespace: sa.metadata.namespace,
 }],
});

// Provision a Pulumi Cloud access token and store it in a Kubernetes Secret
const accessToken = new pulumiservice.AccessToken("random-yaml", {
 description: `For stack "${pulumi.runtime.getOrganization()}/${pulumi.runtime.getProject()}/${pulumi.runtime.getStack()}"`,
});
const apiSecret = new k8s.core.v1.Secret("random-yaml", {
 stringData: {
 "accessToken": accessToken.value,
 }
});

// Deploy the "random-yaml" program from the github.com/pulumi/examples repository.
const stack = new k8s.apiextensions.CustomResource("random-yaml", {
 apiVersion: 'pulumi.com/v1',
 kind: 'Stack',
 spec: {
 serviceAccountName: sa.metadata.name,
 projectRepo: "https://github.com/pulumi/examples",
 repoDir: "random-yaml/",
 branch: "master",
 shallow: true,
 stack: "pulumi-ts",
 refresh: true,
 destroyOnFinalize: true,
 envRefs: {
 PULUMI_ACCESS_TOKEN: {
 type: "Secret",
 secret: {
 name: apiSecret.metadata.name,
 key: "accessToken",
 },
 },
 },
 workspaceTemplate: {
 spec: {
 image: "pulumi/pulumi:3.147.0-nonroot",
 },
 },
 },
}, {dependsOn: [sa, crb, apiSecret]});

Here’s an example of the enhanced event stream that shows the lifecycle of a stack deployment, from workspace creation to successful completion:

88s Normal SuccessfulCreate statefulset/random-yaml-09aa1aad-workspace create Pod random-yaml-09aa1aad-workspace-0 in StatefulSet random-yaml-09aa1aad-workspace successful
88s Normal StackUpdateDetected stack/random-yaml-09aa1aad Initial stack update
0s Normal Pulled pod/random-yaml-09aa1aad-workspace-0 Container image "pulumi/pulumi:3.147.0-nonroot" already present on machine
0s Normal Created pod/random-yaml-09aa1aad-workspace-0 Created container pulumi
0s Normal Started pod/random-yaml-09aa1aad-workspace-0 Started container pulumi
0s Normal Initialized workspace/random-yaml-09aa1aad Initialized workspace pod "random-yaml-09aa1aad-workspace-0"
0s Normal StackUpdateDetected stack/random-yaml-09aa1aad Initial stack update
0s Normal UpdateSucceeded update/random-yaml-09aa1aad-195200641a9 Updated stack "pulumi-ts"
0s Normal StackCreated stack/random-yaml-09aa1aad Successfully updated stack

Additional Resources

To help you get the most out of Pulumi Kubernetes Operator 2.0, we’ve prepared comprehensive documentation:

Migration Guide: Step-by-step instructions for upgrading from v1.x to v2.0
Troubleshooting Guide: Common issues and their solutions
Release Notes: Full details of what’s new in version 2.0

Start Using It Today

We are excited to see how you will use these new features to transform your cloud operations. Share your experiences, questions, and insights with us through our community forums or Slack channel.

Get started with the Pulumi Kubernetes Operator in our documentation

Happy deploying!

Enforcing Policy as Code on Discovered Resources with Pulumi

Tyler D — Tue, 04 Feb 2025 00:00:00 +0000

In this post, we’re introducing a powerful new capability in Pulumi Insights that extends policy as code (PaC) beyond infrastructure as code to automatically govern all cloud resources in your environment. By unifying policy enforcement across both IaC and discovered resources, you can now write policies once and apply them universally - dramatically simplifying how organizations maintain security and compliance standards at scale.

What is Policy as Code (PaC)?

Policy as Code (PaC) is a way to define and enforce infrastructure policies using code. Instead of relying on manual processes or static configuration files, PaC enables automated policy enforcement in a consistent, versioned way. Pulumi’s CrossGuard feature allows you to write policies in general purpose programming languages, applying them across various cloud platforms such as AWS, Azure, GCP, and Kubernetes.

You can read more about CrossGuard in Pulumi’s CrossGuard documentation.

Accounts and Insights

Before diving into how you can configure PaC for discovered resources, it’s important to understand how Pulumi discovers resources within your cloud environment regardless of how they were created. Pulumi Insights provides visibility into the state of your cloud resources by using read only access to your cloud environment in order to catalog the resources it contains. This is done through the Accounts page in Pulumi, which shows all the accounts for discovering resources you have added to your Pulumi organization. On this page, you can add new accounts which you can then add to policy groups to begin evaluating all of your cloud resources using your organization’s policy as code.

After you have added an account, you can connect it with one or multiple Policy Groups to enforce the policies within those groups across your stacks and Insights accounts. Once linked to a Policy Group, your resources will be automatically evaluated against the policies you define whenever those policies or resources are updated.

Setting Up Policy Groups for Discovered Resources

To apply policies to your discovered resources, you first need to set up a Policy Group. A Policy Group in Pulumi enforces policies across a group of stacks and accounts in your organization. Each Policy Group can contain multiple stacks and accounts, and you can assign multiple Policy Packs to these groups.

Here’s an example of the Policy Group configuration page, where you can create and manage policy groups:

Once your Policy Group is set up, you can add Insights Accounts to the group, which will ensure that the resources in those accounts are evaluated against the policies in the group.

Running Policies on Discovered Resources

Now that your Policy Group is set up, you can start running policies against your discovered resources. Currently, Pulumi Insights supports running resource policies against all resource types supported by Pulumi Insights with support for stack policies coming soon. Pulumi Insights will evaluate each resource against the policies defined in your Policy Pack. If the resources violate any of the defined policies, those violations will show up on the Policy Violations page, which gives you a detailed view of any non-compliant resources.

Each violation includes details about the resource and the reason for the violation, helping you quickly identify and address issues.

Key Takeaways

With Pulumi’s new PaC over discovered resources feature, you can now apply the same policies to both IaC and discovered resources across your cloud environment. This enables you to write a policy once and have it apply across all of your cloud resources. This results in consistent, compliant infrastructure no matter how resources are provisioned.

To begin securing your cloud infrastructure, here are the steps to take:

Set up Policy Groups to enforce policies across stacks and accounts.
Link Insights Accounts to your Policy Groups.
View violations on the Policy Violations page and take action to resolve them.

This new capability is available for AWS, Azure, OCI, and Kubernetes. Stay tuned for more updates on policy enforcement and discover how you can leverage Pulumi for cloud security and compliance.

Try it yourself: Start enforcing policies across your discovered resources in minutes with Pulumi Insights. Check out our getting started guide to begin securing your entire cloud environment with policy as code.

For more information on getting started with Policy as Code, check out the following resources:

Kubernetes Best Practices I Wish I Had Known Before

Engin Diri — Mon, 20 Jan 2025 00:00:00 +0000

Kubernetes best practices are the production-tested rules that keep clusters reliable, secure, and affordable: set resource requests and limits on every container, isolate workloads with namespaces and NetworkPolicies, enforce least-privilege RBAC, automate health checks, ship via GitOps, validate every change with policy-as-code, generate an SBOM for every image, and manage Kubernetes itself with infrastructure as code instead of hand-rolled YAML. The 20 practices below cover what production teams actually do in 2026 — not what tutorials suggest.

TL;DR: 20 Kubernetes best practices for 2026

Set resource requests and limits on every container.
Use namespaces plus ResourceQuota and LimitRange for isolation.
Run one container per Pod unless you genuinely need a sidecar.
Manage manifests with Helm, Kustomize, or Pulumi — never raw copies.
Use the Gateway API (ingress-nginx is retired) for north-south traffic.
Configure liveness, readiness, and startup probes for every workload.
Enforce RBAC with the principle of least privilege from day one.
Apply Pod Security Admission at the restricted level by default.
Lock down east-west traffic with NetworkPolicy (default-deny).
Store secrets in an external secret manager (Pulumi ESC, Vault, AWS Secrets Manager) — not as plain Kubernetes Secrets.
Autoscale with HPA, VPA, and Cluster Autoscaler / Karpenter.
Monitor with Prometheus, Grafana, and OpenTelemetry; alert before users do.
Deploy with GitOps (Argo CD or Flux) — Git is the only source of truth.
Gate every change with policy-as-code (Pulumi CrossGuard, OPA/Gatekeeper, Kyverno).
Generate and verify an SBOM for every image; sign with Sigstore/cosign.
Manage cost with FinOps practices: right-size, schedule, and chargeback by namespace.
Keep clusters and add-ons patched on a documented cadence.
Use labels and annotations consistently for ownership, cost, and SLO tracking.
Separate dev, staging, and prod clusters — or virtualize with vCluster.
Treat clusters as cattle, not pets: rebuild from code, never edit live.

Last updated April 30, 2026 — added GitOps, policy-as-code, SBOM/supply-chain, autoscaling, NetworkPolicy, and FinOps sections; added an anti-pattern table; restructured headings around the questions teams actually ask.

Going to KubeCon Europe 2026? Tame Kubernetes complexity with code.

Pulumi demos, platform engineering talks, AI-powered Neo in action, and yes, free plushies. Booth 784.

The ‘Kubernetes is easy’-iceberg meme is a classic example of how Kubernetes can be deceivingly complex

What are Kubernetes best practices?

Kubernetes best practices are the operational patterns that keep workloads available, secure, observable, and cost-efficient on a Kubernetes cluster. They span four areas: workload configuration (resources, probes, images), security (RBAC, NetworkPolicy, Pod Security, secrets, supply chain), delivery (GitOps, policy-as-code, IaC), and platform operations (autoscaling, monitoring, upgrades, FinOps). The rest of this post explains each one with a copy-pasteable rule, a definition of the underlying Kubernetes object, and a common anti-pattern to avoid.

Common Kubernetes anti-patterns vs. the correct approach

Common mistake	What goes wrong	Correct approach
No resource requests or limits	Noisy-neighbor evictions, OOMKills, unschedulable Pods	Set requests for every container; use VPA recommendations to tune
Everything in the `default` namespace	No isolation, no quota, no RBAC boundary	One namespace per team or environment + ResourceQuota + RBAC
`cluster-admin` for service accounts	Full-cluster blast radius on compromise	Per-namespace Role + RoleBinding (least privilege)
Plain Kubernetes `Secret` objects	Base64 ≠ encryption; secrets leak via etcd, kubectl, audit logs	External Secrets Operator + Pulumi ESC, Vault, or AWS Secrets Manager
Open east-west traffic	One compromised Pod can reach every service	Default-deny NetworkPolicy; allow only required flows
`kubectl apply -f` from a laptop	No history, no review, no rollback	GitOps with Argo CD or Flux; Git is the source of truth
Manual `latest` image tags	Non-reproducible deploys, supply-chain risk	Pinned digests (`@sha256:...`) with cosign verification
Ingress-nginx in 2026	Project retired March 2026 — no security fixes	Migrate to the Gateway API (Envoy Gateway, Istio, Kgateway)
Hand-rolled YAML in dozens of repos	Drift, copy-paste bugs, no testing	Pulumi, Helm, or Kustomize with reviews
One shared `prod` cluster for everything	Blast radius spans every team	Separate clusters or vCluster virtual clusters

1. How do you set Kubernetes resource requests and limits?

What are resource requests and limits? A resource request is the minimum CPU and memory the kube-scheduler guarantees a container; a resource limit is the maximum the kubelet allows it to consume before throttling CPU or OOMKilling memory. Pods without requests are scheduled blind, so a noisy neighbor can evict your workload at any time.

The rule: every container in production sets a request for both CPU and memory, and a memory limit. CPU limits are optional and frequently counter-productive — they cause throttling spikes that look like outages.

Start at 100–200m CPU and 128–512Mi memory for a typical web service.
Tune from real data using Vertical Pod Autoscaler with updateMode: Off (recommendation-only) or Prometheus histograms.
Use a per-namespace LimitRange so teams cannot ship a container with no requests at all.

2. How should you structure Kubernetes namespaces?

What is a Kubernetes namespace? A namespace is a logical partition of cluster resources. It scopes object names, RBAC bindings, NetworkPolicies, and quotas — but does not provide hard isolation by itself.

The rule: never deploy production workloads to default. Map namespaces to ownership boundaries — usually one per team, environment, or tenant — and pair every namespace with three things:

A ResourceQuota to cap aggregate CPU, memory, and object counts.
A LimitRange to enforce per-container defaults.
A RoleBinding that grants only the permissions that team needs.

This is the smallest unit that gives you predictable cost, blast-radius containment, and an RBAC perimeter.

3. Should you run multiple containers in one Pod?

What is a Pod? A Pod is the smallest deployable unit in Kubernetes — one or more containers that share a network namespace, IPC, and storage volumes, and that are scheduled and scaled as a single unit.

The rule: one container per Pod by default. Add a second container only when it must share the Pod’s network or volumes — for example, a service-mesh sidecar (Istio/Linkerd), a logging shipper, or an init container that prepares state before the main process starts. Multi-container Pods couple lifecycles and scaling, so reach for them deliberately, not by default.

You might also like:

4. Should you use Helm, Kustomize, or Pulumi for Kubernetes manifests?

The rule: never copy-paste raw YAML across services. Pick one of these:

Helm — Go-templated charts; the de-facto distribution format for off-the-shelf software (Prometheus, cert-manager, Argo CD).
Kustomize — overlay-based, no templating; ships in kubectl. Best for “same app, different environment” diffs.
Pulumi — real programming languages (TypeScript, Python, Go, Java, .NET) for Kubernetes plus the cloud resources around it (EKS, GKE, AKS, DNS, IAM). Type checking, tests, and the Pulumi Kubernetes Operator for GitOps-style reconciliation.

If you are already managing cloud infrastructure with code, extending the same Pulumi program to Kubernetes is the smallest cognitive jump — the cluster, its IAM, its DNS, and its workloads live in one stack and one review.

5. How should you handle ingress and networking on Kubernetes in 2026?

What is the Gateway API? The Gateway API is the successor to the legacy Ingress resource. It splits responsibility between infrastructure (GatewayClass, Gateway) and application (HTTPRoute, GRPCRoute, TLSRoute) teams, and it is the SIG-Network direction for north-south traffic.

The rule: new clusters should standardize on the Gateway API. The widely deployed ingress-nginx controller retired in March 2026, and remaining deployments should migrate to Envoy Gateway, Istio, Linkerd Gateway, or Kgateway.

Terminate TLS at the gateway and automate certificates with cert-manager.
Use path-based and host-based routing instead of one gateway per service.
Protect every gateway with a WAF (managed cloud WAF or Coraza on Envoy).

ingress-nginx is retiring in March 2026.

Here is a guide on How to Move to the Gateway API: post ingress-nginx Retirement.

6. What’s the difference between liveness, readiness, and startup probes?

Probes tell the kubelet whether to restart a container, route traffic to it, or wait for it. Configure them per workload:

Liveness probe — restarts a hung container. Use sparingly; a bad liveness probe is a self-inflicted DoS.
Readiness probe — gates traffic from Services and Gateway routes. Use for every network-facing workload.
Startup probe — gives slow-booting apps (JVM, ML models) time to come up before liveness checks begin. Use whenever cold-start exceeds 10 seconds.

See the upstream guide for HTTP, TCP, and exec probe syntax. Tune failureThreshold, periodSeconds, and timeoutSeconds based on real latency, not defaults.

7. How do you secure a Kubernetes cluster?

Cluster security is layered: identity, workload posture, network, and secrets. Get all four right.

What is RBAC in Kubernetes?

RBAC (Role-Based Access Control) is the Kubernetes authorization model that maps subjects (users, groups, ServiceAccounts) to verbs (get, list, create, delete) on resources, scoped to a namespace (Role) or the whole cluster (ClusterRole).

Grant least privilege — never bind humans or workloads to cluster-admin.
Prefer per-namespace Role + RoleBinding over ClusterRoleBinding.
Audit with kubectl auth can-i --as=... and tools like rbac-lookup.

What is Pod Security Admission?

Pod Security Admission is the built-in admission controller that enforces the three Pod Security Standards (privileged, baseline, restricted) at the namespace level.

Label every namespace pod-security.kubernetes.io/enforce=restricted by default.
Drop NET_RAW and all capabilities; run as non-root with a read-only root filesystem.
Use seccompProfile: RuntimeDefault and a tight securityContext for every container.

What is a NetworkPolicy?

A NetworkPolicy is a Kubernetes object that defines allowed ingress and egress traffic for selected Pods at the IP/port level. Without one, every Pod can reach every other Pod by default.

Apply a default-deny ingress and egress policy in every namespace.
Allow only the flows your service actually needs (DNS, the database, the upstream API).
For richer L7 controls (mTLS, JWT auth), layer a service mesh on top.

How should you manage Kubernetes secrets?

Built-in Secret objects are base64-encoded, not encrypted. Treat them as a transport, not as storage.

Store the source of truth in Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault.
Project secrets into Pods with the External Secrets Operator or the Secret Store CSI Driver.
Enable etcd encryption-at-rest with a KMS provider so leaked etcd snapshots stay opaque.
Rotate automatically; never commit secrets to Git, even encrypted.

8. How do you set up Kubernetes observability?

Observability rests on the three pillars: metrics, logs, and traces. The 2026 default stack is OpenTelemetry-first.

Metrics — Prometheus (or a Prometheus-compatible TSDB like Mimir, Thanos, VictoriaMetrics) plus Grafana dashboards.
Logs — Fluent Bit or the OpenTelemetry Collector shipping structured JSON to Loki, Elastic/OpenSearch, or a managed service. Never depend on kubectl logs for incidents — Pods are ephemeral.
Traces — OpenTelemetry SDKs in your apps emitting to Tempo, Jaeger, or a vendor backend. Correlate trace IDs with logs.
Alerts — page on SLO burn rate, not on every crashed Pod. Burn-rate alerting catches real user impact without firing on every restart.

9. How do you automate Kubernetes deployments with GitOps?

What is GitOps? GitOps is a deployment model where the desired cluster state lives in a Git repository and a controller continuously reconciles the live cluster to match. Changes happen through pull requests, not kubectl apply.

The rule: every cluster has a controller — Argo CD, Flux, or the Pulumi Kubernetes Operator — that owns reconciliation. Humans never kubectl apply to production.

Use app-of-apps (Argo) or Kustomization trees (Flux) to bootstrap whole clusters from a single root.
Keep manifests, Helm values, and Pulumi stack references in Git, with environment promotion via PR.
Combine GitOps with Pulumi Deployments so cloud infrastructure (VPCs, EKS clusters, IAM, DNS) and the workloads on top promote through the same review pipeline. See improving GitOps with the Pulumi Operator for a worked example.
Configure automated rollback on health-check failure, and drift detection alerts when someone edits live state.

10. How often should you upgrade Kubernetes?

Kubernetes ships a minor release roughly every four months and supports each one for about 14 months. Treat upgrades as routine maintenance, not a project.

Stay within two minor versions of upstream — older versions miss CVE patches.
Test upgrades in a non-prod cluster, run a conformance test, then promote.
Back up etcd before control-plane changes; managed services (EKS, GKE, AKS) handle most of this for you.
Upgrade add-ons (CNI, CSI, Gateway controller, cert-manager, metrics-server) on the same cadence — pin versions in code so the upgrade is auditable.
Watch the Kubernetes deprecation guide and run pluto or kubent to find deprecated APIs before they break.

11. How should you label and annotate Kubernetes resources?

Labels and annotations are the metadata layer that everything — Services, NetworkPolicies, monitoring, cost tools, GitOps — keys off of. Consistency matters more than cleverness.

Adopt the recommended labels: app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/version, app.kubernetes.io/component, app.kubernetes.io/part-of, app.kubernetes.io/managed-by.
Add organization-specific labels for owner, cost-center, environment, and SLO tier — your FinOps and on-call processes will need them.
Reserve annotations for non-identifying metadata: build SHA, change-ticket, last-deployed timestamp, ingress controller hints.

Manage Kubernetes with code

Provision your clusters, their cloud resources, and the workloads on them from one Pulumi program, and use stacks to promote changes from dev to production.

Get started

12. How should you separate Kubernetes environments?

The rule: production is its own cluster. Dev and staging can share — sometimes. Mixing prod with anything else couples blast radius, upgrade cadence, and quotas.

Separate clusters per environment is the safe default. With managed control planes (EKS, GKE, AKS) the cost is small.
Virtual clusters with vCluster give each team a Kubernetes API of their own without the operational cost of a real cluster.
Namespace-only segregation can work for dev/staging if you enforce strict NetworkPolicy, RBAC, and ResourceQuota boundaries.
Provision all of them from the same Pulumi program with different stacks — dev, staging, prod — so promotion is a config change, not a copy.

13. How do you optimize Kubernetes container images?

Start from a distroless (gcr.io/distroless) or chainguard/static base. Alpine is fine if you accept musl.
Build with multi-stage Dockerfiles so build tools never ship in the final image.
Pin the base image by digest (@sha256:...), not by tag — latest and 1.21 move under you.
Scan every image with Trivy, Grype, or your registry’s built-in scanner. Block builds on Critical/High CVEs.
Smaller images mean faster pulls, faster autoscaling, and a smaller attack surface.

14. What’s a reliable Kubernetes logging strategy?

Centralize to Loki, Elastic/OpenSearch, or a managed service via Fluent Bit or the OpenTelemetry Collector running as a DaemonSet.
Structure logs as JSON with consistent field names (trace_id, span_id, service, level).
Retain by tier — 7 days hot, 30–90 days warm, archive to object storage for compliance.
Redact secrets and PII at the collector, not after the fact.

You might also like:

15. How should you autoscale workloads on Kubernetes?

What is the Horizontal Pod Autoscaler? The HPA scales the replica count of a Deployment or StatefulSet up and down based on CPU, memory, or custom/external metrics.

What is the Vertical Pod Autoscaler? The VPA adjusts a Pod’s CPU and memory requests over time based on observed usage. Run it with updateMode: Off (recommendation-only) in production and apply the recommendations through your IaC tool.

What are Cluster Autoscaler and Karpenter? Cluster Autoscaler adds and removes nodes to match Pod demand. Karpenter is a faster, group-less node provisioner now widely used on EKS and gaining traction on other clouds.

The rule: every production workload has an HPA. Every cluster has a node-level autoscaler. Set sensible min/max replicas, define a PodDisruptionBudget, and gate scale-down behavior so traffic spikes don’t pile up at restart time. Pair with KEDA for event-driven scaling (queues, Kafka lag, scheduled scale).

16. How do you enforce policy-as-code on Kubernetes?

What is policy-as-code? Policy-as-code expresses governance rules — security baselines, tagging, region restrictions, cost guardrails — as code that runs in CI and at admission time, blocking non-compliant changes before they reach the cluster.

The rule: every change goes through at least one policy engine.

Pre-deploy — validate Pulumi or Helm output with Pulumi CrossGuard or Conftest in CI.
Admission-time — install OPA Gatekeeper or Kyverno and require, for example, signed images, a team label, and a non-root securityContext on every Pod.
Continuous — scan running clusters with Pulumi Insights or kubescape for drift from policy.

See the benefits of policy-as-code and enforcing policy-as-code on discovered resources for end-to-end examples.

17. How do you secure the Kubernetes software supply chain?

What is an SBOM? A Software Bill of Materials is a machine-readable inventory of every component in a container image — packages, versions, licenses, hashes — typically in SPDX or CycloneDX format.

The rule: every image you run in production has a verified SBOM and a verified signature.

Generate SBOMs at build time with Syft or docker sbom.
Sign images and SBOMs with Sigstore/cosign keylessly via GitHub OIDC.
Verify signatures at admission with Kyverno or Connaisseur.
Generate provenance attestations (SLSA) so you can prove which build pipeline produced a given image.
Pin base images by digest and rebuild on a cadence so CVE patches actually land.

This is what regulators and customers ask for in 2026 — Executive Order 14028, the EU Cyber Resilience Act, and most enterprise procurement checklists all require it.

18. How do you manage Kubernetes cost (FinOps)?

The rule: Kubernetes cost is a labeling and right-sizing problem before it is a discount problem.

Right-size with VPA recommendations, Goldilocks, or your APM’s recommendation engine. Most workloads request more CPU than they actually use; right-sizing is usually the largest single saving.
Schedule non-prod clusters off overnight and on weekends.
Spot/preemptible nodes for fault-tolerant workloads; use Karpenter’s consolidation to compact bin-packing.
Show back / charge back by namespace and label using OpenCost or Kubecost. Without per-team attribution, no team owns cost.
Provision the underlying cluster with IaC — see Pulumi for Kubernetes — so node types, autoscaling limits, and reserved capacity are reviewable artifacts. Pulumi’s hidden-cost analysis walks through where the money actually goes.

19. Why should you treat Kubernetes clusters as cattle, not pets?

The old “cattle, not pets” adage applies more strictly to Kubernetes than to VMs. Manual edits to a live cluster will be reconciled away, drift between environments, or vanish on the next Pod restart.

Fix problems in code — YAML, Helm chart, or Pulumi program — and let the controller redeploy.
If you can’t redeploy a cluster from Git in under an hour, you have a pet.
Use ephemeral preview environments for PRs; use blue/green or progressive delivery (Argo Rollouts, Flagger) for production cutovers.

20. Why use Pulumi to manage Kubernetes?

Native YAML scales until your team doesn’t. Once you have more than a handful of services, dependencies between cloud resources and Kubernetes objects, or more than one cluster, infrastructure as code in a real programming language wins on every axis.

Real languages — TypeScript, Python, Go, Java, .NET — for type-safe, testable Kubernetes infrastructure. Loops, conditionals, and unit tests instead of templating.
One stack, full topology — manage the cloud (EKS/GKE/AKS, VPC, IAM, DNS) and the workloads in it together. See easily create and manage AWS EKS clusters with Pulumi.
Reusable components — abstract platform patterns into Pulumi packages other teams import instead of copy-pasting.
GitOps reconciliation — the Pulumi Kubernetes Operator reconciles a cluster to a Pulumi stack on every Git push.
Policy-as-code built in — CrossGuard blocks non-compliant changes before pulumi up.
Secrets done right — Pulumi ESC federates secrets and configuration across stacks, environments, and Kubernetes clusters.

For a deeper comparison of hand-written YAML, Terraform, and Pulumi for Kubernetes, see YAML, Terraform, Pulumi: what’s the smart choice for deployment automation with Kubernetes, and the beyond YAML write-up on where Kubernetes automation is heading in 2026.

See how Pulumi helps you manage Kubernetes Services and Deployments with code instead of YAML—bringing structure, reuse, and type safety to your configs.

By adopting Pulumi, you can avoid the complexity of juggling endless YAML files and gain a more streamlined, maintainable workflow for your Kubernetes infrastructure.

Final thoughts

Kubernetes rewards discipline. The 20 practices above — resource hygiene, namespaced isolation, RBAC, NetworkPolicy, Pod Security, external secrets, probes, the Gateway API, observability, GitOps, policy-as-code, signed images and SBOMs, autoscaling, FinOps, multi-cluster strategy, and IaC — are the dial settings that separate clusters that page their owners every week from clusters that don’t.

Pick the three weakest spots in your environment and fix those first. Then promote the fixes through code, not through kubectl.

Want to learn how to put these practices into action? Meet us at KubeCon Europe 2026 (Booth 784) or register for our upcoming Zero to Production in Kubernetes workshop.

Try Pulumi for Free

Master Kubernetes Secrets with Pulumi ESC + Secrets Store CSI Driver

Engin Diri — Wed, 18 Dec 2024 09:58:01 +0100

Welcome to the second blog post of the Pulumi ESC and Kubernetes secrets series. If you haven’t had the chance to read the first blog post, go ahead and read it here.

In the previous blog post, we have learned how to manage secrets with Pulumi ESC and the External Secrets Operator. While the External Secrets Operator is a great tool to manage secrets in a cloud-native way, it still creates Kubernetes secrets in the cluster. Depending on your security requirements, you might want to avoid the use of Kubernetes secrets in your cluster at all. This is the point where you hit the limits of the External Secrets Operator.

But don’t worry, we also have a solution for this problem. In this blog post, we will introduce the Secrets Store CSI Driver.

TL;DR?

Jump straight to the comparison of the External Secrets Operator and the Secrets Store CSI Driver here.

Refresher: Why Should You Avoid Using Kubernetes Secrets?

Before we dive into the Secrets Store CSI Driver, let’s quickly recap why you might want to avoid using native Kubernetes secrets. I wrote in depth about this topic in the previous blog post, but here is a quick summary:

By default, etcd stores them in a base64-encoded format, which is not secure.
Developers create them either manually by using kubectl commands or by using a manifest file, which makes them hard to manage at scale.
Hard to manage and synchronise secrets across different environments and clusters.
There is no default way to rotate secrets automatically.

Point 1 is the most critical one, as there is even a warning in the Kubernetes documentation:

Kubernetes Secrets. Credit: kubernetes.io/docs/concepts/configuration/secret/

What is the Secrets Store CSI Driver?

The first thing you may notice is that in the name of the Secret Store CSI Driver, there is a reference to the Container Storage Interface (CSI) standard.

What is CSI and why it is used in the Secret Store CSI Driver*?*

How to write a Container Storage Interface (CSI) plugin. Credit: arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/

Container Storage Interface (CSI)

The Container Storage Interface (CSI) is a standard to unify the interface between container orchestrators (like Kubernetes) and different storage vendors (like NetApp, Ceph, etc.). This helps to guarantee that implementing a CSI for a storage vendor is going to work with all orchestrators that support CSI.

Before CSI, we needed to create volume plugins for every orchestrator and storage vendor. This meant that developers coupled volume plugin development with the Kubernetes version and made it dependent on it. Bugs in volume plugins could break the Kubernetes components, instead of the volume plugin. Besides, volume plugins had full privileges on the Kubernetes components like the kubelet.

Combining CSI, Kubernetes and Secrets = Secret Store CSI Driver

The Secret Store CSI Driver is a CSI driver that allows you to mount many secrets, certificates, and keys from external secret stores into Kubernetes pods as volumes. After attaching the volume, the system mounts the secrets into the container file system.

The benefits of using the Secret Store CSI Driver are that you manage the lifecycle of the secrets outside of Kubernetes while still providing a Kubernetes-native experience of using the secrets in your pods.

The Architecture of the Secret Store CSI Driver

The Secret Store CSI Driver is a daemonset that lets all kublets communicate with each other and uses gRPC to talk to a provider. A SecretProviderClass custom resource specifies the definition of the external Secret Store. Then the system mounts a volume in the pod as tmpfs, and it injects the secret into the volume. When you delete the pod, the system cleans up the volume and removes the secret from the tmpfs volume.

How to Use the Secrets Store CSI Driver with Pulumi ESC

Before we start, make sure you have the following prerequisites:

A Pulumi Cloud account. If you don’t have one, you can create one for free.
A Kubernetes cluster (I will be using a local KinD cluster, but you can use any Kubernetes cluster)
Pulumi CLI installed. You can use the pulumi env commands to manage secrets and configurations.
kubectl CLI installed for some debugging

Step 1: Deploy the Secrets Store CSI Driver and Pulumi ESC CSI Provider

We will use Pulumi with the pulumi-kubernetes provider to deploy the Secrets Store CSI Driver and the Pulumi ESC CSI Provider to our Kubernetes cluster.

And here we also use the integration of Pulumi ESC with Pulumi IaC to supply the Pulumi access token in a secure way.

First, we need to create a new Pulumi project:

# Choose your favorite Pulumi supported language
pulumi new kubernetes-<your-programming-language> --name

Before we dig into the code, let’s head to the Pulumi Cloud Console and create a new Pulumi ESC project with the name esc-secrets-store-csi-driver-demo.

And create the environment dev:

In the editor add the following yaml into the Environment definition:

values:
 pulumiConfig:
 pulumi-pat:
 fn::secret: <your-pulumi-pat>

Don’t know how to create a PAT? Check out the official Pulumi documentation.

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/esc-secrets-store-csi-driver-demo/dev

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/esc-secrets-store-csi-driver-demo/dev

Now, we need to link the Pulumi ESC project to the Pulumi IaC project. To do this, we need to add the following to your Pulumi.dev.yaml:

environment:
- esc-secrets-store-csi-driver-demo/dev

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

const secretsStoreCSIDriver = new k8s.helm.v4.Chart("secrets-store-csi-driver", {
 chart: "secrets-store-csi-driver",
 namespace: "kube-system",
 repositoryOpts: {
 repo: "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 },
 values: {
 nodeSelector: {
 "kubernetes.io/os": "linux",
 },
 },
});

const secretsStoreCSIPulumiESCProvider = new k8s.helm.v4.Chart(
 "secrets-store-csi-pulumi-esc-provider",
 {
 chart: "oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 namespace: "kube-system",
 values: {
 nodeSelector: {
 "kubernetes.io/os": "linux",
 },
 },
 },
 { dependsOn: secretsStoreCSIDriver },
);

const config = new pulumi.Config();

const mySecret = new k8s.core.v1.Secret(
 "my-secret",
 {
 metadata: {
 namespace: "default",
 name: "pulumi-access-token",
 },
 stringData: {
 "pulumi-access-token": config.require("pulumi-pat"),
 },
 type: "Opaque",
 },
 { dependsOn: secretsStoreCSIPulumiESCProvider },
);

import pulumi
import pulumi_kubernetes as k8s

secrets_store_csi_driver = k8s.helm.v4.Chart(
 "secrets-store-csi-driver",
 k8s.helm.v4.ChartArgs(
 chart="secrets-store-csi-driver",
 namespace="kube-system",
 repository_opts=k8s.helm.v4.RepositoryOptsArgs(
 repo="https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 ),
 values={
 "nodeSelector": {
 "kubernetes.io/os": "linux",
 },
 },
 ),
)

secrets_store_csi_pulumi_esc_provider = k8s.helm.v4.Chart(
 "secrets-store-csi-pulumi-esc-provider",
 k8s.helm.v4.ChartArgs(
 chart="oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 namespace="kube-system",
 values={
 "nodeSelector": {
 "kubernetes.io/os": "linux",
 },
 },
 ),
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_driver]),
)

config = pulumi.Config()

my_secret = k8s.core.v1.Secret(
 "my-secret",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 namespace="default", name="pulumi-access-token"
 ),
 string_data={
 "pulumi-access-token": config.require("pulumi-pat"),
 },
 type="Opaque",
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_pulumi_esc_provider]),
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 appv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 chartv4 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v4"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 secretsStoreCsiDriver, err := chartv4.NewChart(ctx, "secrets-store-csi-driver", &chartv4.ChartArgs{
 Chart: pulumi.String("secrets-store-csi-driver"),
 Namespace: pulumi.String("kube-system"),
 RepositoryOpts: chartv4.RepositoryOptsArgs{
 Repo: pulumi.String("https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"),
 },
 Values: pulumi.Map{
 "nodeSelector": pulumi.Map{
 "kubernetes.io/os": pulumi.String("linux"),
 },
 },
 })
 if err != nil {
 return err
 }

 secretsStoreCsiPulumiEscProvider, err := chartv4.NewChart(ctx, "secrets-store-csi-pulumi-esc-provider", &chartv4.ChartArgs{
 Chart: pulumi.String("oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider"),
 Namespace: pulumi.String("kube-system"),
 Values: pulumi.Map{
 "nodeSelector": pulumi.Map{
 "kubernetes.io/os": pulumi.String("linux"),
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiDriver}))
 if err != nil {
 return err
 }

 pulumiPAT := config.Get(ctx, "pulumi-pat")

 mySecret, err := corev1.NewSecret(ctx, "my-secret", &corev1.SecretArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: pulumi.String("default"),
 Name: pulumi.String("pulumi-access-token"),
 },
 StringData: pulumi.StringMap{
 "pulumi-access-token": pulumi.String(pulumiPAT),
 },
 Type: pulumi.String("Opaque"),
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiPulumiEscProvider}))
 if err != nil {
 return err
 }


 return nil
 })
}

using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Helm;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using Pulumi.Kubernetes.ApiExtensions;
using System.Collections.Generic;



return await Deployment.RunAsync(() =>
{
 var secretsStoreCsiDriver = new Release("secrets-store-csi-driver", new()
 {
 Chart = "secrets-store-csi-driver",
 Namespace = "kube-system",
 RepositoryOpts = new Pulumi.Kubernetes.Types.Inputs.Helm.V3.RepositoryOptsArgs
 {
 Repo = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 },
 Values = new Dictionary<string, object>
 {
 { "nodeSelector", new Dictionary<string, object>
 {
 { "kubernetes.io/os", "linux" },
 } },
 },
 });

 var secretsStoreCsiPulumiEscProvider = new Release("secrets-store-csi-pulumi-esc-provider", new()
 {
 Chart = "oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 Namespace = "kube-system",
 Values = new Dictionary<string, object>
 {
 { "nodeSelector", new Dictionary<string, object>
 {
 { "kubernetes.io/os", "linux" },
 } },
 },
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiDriver } });

 var config = new Config();
 var pulumiPAT = config.Require("pulumi-pat");

 var mySecret = new Secret("my-secret", new SecretArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Namespace = "default",
 Name = "pulumi-access-token",
 },
 StringData =
 {
 { "pulumi-access-token", pulumiPAT }
 },
 Type = "Opaque",
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiPulumiEscProvider } });

});


class SecretProviderClassArgs : CustomResourceArgs
{
 public SecretProviderClassArgs(): base("secrets-store.csi.x-k8s.io/v1", "SecretProviderClass")
 {
 }

 [Input("spec")]
 public Input<SecretProviderClassSpecArgs>? Spec { get; set; }
}


class SecretProviderClassSpecArgs : ResourceArgs
{
 [Input("provider")]
 public Input<string>? Provider { get; set; }

 [Input("parameters")]
 public Input<InputMap<object>>? Parameters { get; set; }
}

class SecretProviderParametersArgs : ResourceArgs
{
 [Input("apiUrl")]
 public Input<string>? ApiUrl { get; set; }

 [Input("organization")]
 public Input<string>? Organization { get; set; }

 [Input("project")]
 public Input<string>? Project { get; set; }

 [Input("environment")]
 public Input<string>? Environment { get; set; }

 [Input("authSecretName")]
 public Input<string>? AuthSecretName { get; set; }

 [Input("authSecretNamespace")]
 public Input<string>? AuthSecretNamespace { get; set; }

 [Input("secrets")]
 public Input<string>? Secrets { get; set; }
}

Deploy the stack by running:

pulumi up

And you should see that the secret was created in the Kubernetes cluster and the ESO instance was deployed successfully.

kubectl get secret pulumi-access-token -o jsonpath='{.data.PULUMI_ACCESS_TOKEN}' | base64 -d

Step 2: Create a SecretProviderClass

Now, we will create a secret in the Pulumi ESC project and synchronize it into the Kubernetes cluster by creating an SecretProviderClass:

Create a new ESC environment called csi-secrets-store-app in the esc-secrets-store-csi-driver-demo project:

values:
 app:
 hello: world
 hello-secret:
 fn::secret: world

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/esc-secrets-store-csi-driver-demo/csi-secrets-store-app

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/esc-secrets-store-csi-driver-demo/csi-secrets-store-app

Either way, you should see following environment configuration in the Pulumi Cloud Console:

We can now create the SecretProviderClass in the Kubernetes cluster:

const secretProviderClass = new k8s.apiextensions.CustomResource(
 "example-provider-pulumi-esc",
 {
 apiVersion: "secrets-store.csi.x-k8s.io/v1",
 kind: "SecretProviderClass",
 metadata: {
 name: "example-provider-pulumi-esc",
 namespace: "default",
 },
 spec: {
 provider: "pulumi",
 parameters: {
 apiUrl: "https://api.pulumi.com/api/esc",
 organization: "dirien",
 project: "esc-secrets-store-csi-driver-demo",
 environment: "csi-secrets-store-app",
 authSecretName: mySecret.metadata.name,
 authSecretNamespace: mySecret.metadata.namespace,
 secrets: `- secretPath: "/"
 fileName: "hello"
 secretKey: "app.hello"
`,
 },
 },
 },
 { dependsOn: secretsStoreCSIPulumiESCProvider },
);

secret_provider_class = k8s.apiextensions.CustomResource(
 "example-provider-pulumi-esc",
 api_version="secrets-store.csi.x-k8s.io/v1",
 kind="SecretProviderClass",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="example-provider-pulumi-esc", namespace="default"
 ),
 spec={
 "provider": "pulumi",
 "parameters": {
 "apiUrl": "https://api.pulumi.com/api/esc",
 "organization": "dirien",
 "project": "esc-secrets-store-csi-driver-demo",
 "environment": "csi-secrets-store-app",
 "authSecretName": my_secret.metadata["name"],
 "authSecretNamespace": my_secret.metadata["namespace"],
 "secrets": '- secretPath: "/"\n fileName: "hello"\n secretKey: "app.hello"\n',
 },
 },
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_pulumi_esc_provider]),
)

 secretProviderClass, err := apiextensions.NewCustomResource(ctx, "example-provider-pulumi-esc", &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("secrets-store.csi.x-k8s.io/v1"),
 Kind: pulumi.String("SecretProviderClass"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-provider-pulumi-esc"),
 Namespace: pulumi.String("default"),
 },
 OtherFields: k8s.UntypedArgs{
 "provider": pulumi.String("pulumi"),
 "parameters": pulumi.Map{
 "apiUrl": pulumi.String("https://api.pulumi.com/api/esc"),
 "organization": pulumi.String("dirien"),
 "project": pulumi.String("esc-secrets-store-csi-driver-demo"),
 "environment": pulumi.String("csi-secrets-store-app"),
 "authSecretName": mySecret.Metadata.Name().Elem(),
 "authSecretNamespace": mySecret.Metadata.Namespace().Elem(),
 "secrets": pulumi.String(`- secretPath: "/"
 fileName: "hello"
 secretKey: "app.hello"
`),
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiPulumiEscProvider}))
 if err != nil {
 return err
 }

 var secretProviderClass = new Pulumi.Kubernetes.ApiExtensions.CustomResource("example-provider-pulumi-esc", new SecretProviderClassArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "example-provider-pulumi-esc",
 Namespace = "default",
 },
 Spec = new SecretProviderClassSpecArgs
 {
 Provider = "pulumi",
 Parameters = new InputMap<object>
 {
 { "apiUrl", "https://api.pulumi.com/api/esc" },
 { "organization", "dirien" },
 { "project", "esc-secrets-store-csi-driver-demo" },
 { "environment", "csi-secrets-store-app" },
 { "authSecretName", mySecret.Metadata.Apply(metadata => metadata.Name) },
 { "authSecretNamespace", mySecret.Metadata.Apply(metadata => metadata.Namespace) },
 { "secrets", "- secretPath: \"/\"\n fileName: \"hello\"\n secretKey: \"app.hello\"\n" }
 },
 },
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiPulumiEscProvider } });

We can check that the secret was successfully synchronized by running:

kubectl get secretproviderclasses example-provider-pulumi-esc

You might also like:

Step 3: Deploy an Application and Mount the Secret

Now, we can deploy an application that references the secret from the Kubernetes cluster. I am going to use busybox that reads mounted file in.

const deployment = new k8s.apps.v1.Deployment("example-provider-pulumi-esc", {
 metadata: {
 name: "example-provider-pulumi-esc",
 namespace: "default",
 labels: {
 app: "example-provider-pulumi-esc",
 },
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: {
 app: "example-provider-pulumi-esc",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "example-provider-pulumi-esc",
 },
 },
 spec: {
 containers: [
 {
 name: "client",
 image: "busybox:latest",
 command: ["sh", "-c"],
 args: [
 `set -eux
 ls /run/secrets
 find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo "$(cat "{}")"'
 tail -f /dev/null`,
 ],
 volumeMounts: [
 {
 name: "data",
 mountPath: "/run/secrets",
 },
 ],
 },
 ],
 volumes: [
 {
 name: "data",
 csi: {
 driver: "secrets-store.csi.k8s.io",
 readOnly: true,
 volumeAttributes: {
 secretProviderClass: secretProviderClass.metadata.name,
 },
 },
 },
 ],
 },
 },
 },
});

export const deploymentName = deployment.metadata.name;

deployment = k8s.apps.v1.Deployment(
 "example-provider-pulumi-esc",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 namespace="default",
 name="example-provider-pulumi-esc",
 labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 spec=k8s.apps.v1.DeploymentSpecArgs(
 replicas=1,
 selector=k8s.meta.v1.LabelSelectorArgs(
 match_labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 template=k8s.core.v1.PodTemplateSpecArgs(
 metadata=k8s.meta.v1.ObjectMetaArgs(
 labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 spec=k8s.core.v1.PodSpecArgs(
 containers=[
 k8s.core.v1.ContainerArgs(
 name="client",
 image="busybox:latest",
 command=["sh", "-c"],
 args=[
 "set -eux\nls /run/secrets\nfind /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo \"$(cat \"{}\")\"'\ntail -f /dev/null",
 ],
 volume_mounts=[
 k8s.core.v1.VolumeMountArgs(
 name="data",
 mount_path="/run/secrets",
 ),
 ],
 ),
 ],
 volumes=[
 k8s.core.v1.VolumeArgs(
 name="data",
 csi=k8s.core.v1.CSIVolumeSourceArgs(
 driver="secrets-store.csi.k8s.io",
 read_only=True,
 volume_attributes={
 "secretProviderClass": secret_provider_class.metadata[
 "name"
 ],
 },
 ),
 ),
 ],
 ),
 ),
 ),
)

pulumi.export("deploymentName", deployment.metadata["name"])

 deployment, err := appv1.NewDeployment(ctx, "example-provider-pulumi-esc", &appv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-provider-pulumi-esc"),
 Namespace: pulumi.String("default"),
 Labels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Spec: &appv1.DeploymentSpecArgs{
 Replicas: pulumi.Int(1),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("client"),
 Image: pulumi.String("busybox:latest"),
 Command: pulumi.StringArray{
 pulumi.String("sh"),
 pulumi.String("-c"),
 },
 Args: pulumi.StringArray{
 pulumi.String(`set -eux
 ls /run/secrets
 find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo "$(cat "{}")"'
 tail -f /dev/null`),
 },
 VolumeMounts: &corev1.VolumeMountArray{
 &corev1.VolumeMountArgs{
 Name: pulumi.String("data"),
 MountPath: pulumi.String("/run/secrets"),
 },
 },
 },
 },
 Volumes: &corev1.VolumeArray{
 &corev1.VolumeArgs{
 Name: pulumi.String("data"),
 Csi: corev1.CSIVolumeSourceArgs{
 Driver: pulumi.String("secrets-store.csi.k8s.io"),
 ReadOnly: pulumi.Bool(true),
 VolumeAttributes: pulumi.StringMap{
 "secretProviderClass": secretProviderClass.Metadata.Name().Elem(),
 },
 },
 },
 },
 },
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretProviderClass}))
 if err != nil {
 return err
 }

 var deployment = new Pulumi.Kubernetes.Apps.V1.Deployment("example-provider-pulumi-esc", new Pulumi.Kubernetes.Types.Inputs.Apps.V1.DeploymentArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "example-provider-pulumi-esc",
 Namespace = "default",
 Labels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Spec = new Pulumi.Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Replicas = 1,
 Selector = new Pulumi.Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Template = new Pulumi.Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Spec = new Pulumi.Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 Containers =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "client",
 Image = "busybox:latest",
 Command =
 {
 "sh",
 "-c",
 },
 Args =
 {
 "set -eux\nls /run/secrets\nfind /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo \"$(cat \"{}\")\"'\ntail -f /dev/null",
 },
 VolumeMounts =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.VolumeMountArgs
 {
 Name = "data",
 MountPath = "/run/secrets",
 },
 },
 },
 },
 Volumes =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.VolumeArgs
 {
 Name = "data",
 Csi = new Pulumi.Kubernetes.Types.Inputs.Core.V1.CSIVolumeSourceArgs
 {
 Driver = "secrets-store.csi.k8s.io",
 ReadOnly = true,
 VolumeAttributes =
 {
 { "secretProviderClass", secretProviderClass.Metadata.Apply(metadata => metadata.Name) },
 },
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions { DependsOn = { secretProviderClass } });

After deploying the stack, you can get the logs of the busybox pod to see that the secret was successfully mounted:

NAME=$(kubectl get pods -o name | grep example-provider-pulumi-esc | cut -d'/' -f2)
kubectl logs $NAME

You should see the following output:

+ ls /run/secrets
hello
+ find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*'
+ xargs -t -I '{}' sh -c 'echo "$(cat "{}")"'
sh -c echo "$(cat "/run/secrets/hello")"
world
+ tail -f /dev/null

Or you can exec into the pod and check the content of the mounted secret:

kubectl exec -it $NAME -- cat /run/secrets/hello

You should see the following output:

world

Step 4: Clean Up

After you are done with the demo, you can clean up the resources by running:

pulumi destroy

Conclusion

The Secret Store CSI Driver is a great option for managing secrets in a cloud-native way when the use of Kubernetes secrets is not an option due to enhanced security requirements.

Below is a quick comparison of the External Secrets Operator and the Secrets Store CSI Driver to help you decide which one is the best fit for your use case. As always, the best way to find out is to try both solutions and see which one fits your requirements best.

The good part is, whatever you choose, you can use Pulumi ESC to manage your secrets and avoid any secret sprawl in your organization as both solutions are supported by Pulumi ESC.

Comparison of the External Secrets Operator and the Secrets Store CSI Driver

Feature/Aspect	External Secrets Operator	CSI Secrets Driver
Primary Use Case	Synchronizing external secrets into Kubernetes as native secrets.	Mounting secrets directly to pods as files or environment variables.
Integration with External Secret Stores	Supports multiple external secret stores like AWS Secrets Manager, Google Secret Manager, Vault, Azure Key Vault, etc.	Primarily supports integrations defined by the CSI specification (e.g., Vault, AWS, Azure Key Vault, Pulumi ESC).
Mechanism for Secrets Delivery	Secrets are pulled and stored as Kubernetes `Secret` resources.	Secrets are mounted directly into pods as files or injected as environment variables.
Kubernetes Resource Requirements	Requires a custom Kubernetes resource (`ExternalSecret`) and the External Secrets Operator.	Relies on the Kubernetes CSI (Container Storage Interface) standard and requires a specific CSI driver.
Secret Rotation	Supports automatic rotation by polling the external secret store at defined intervals.	Supports rotation, but the application needs to handle reloading mounted secrets.
Security Considerations	Secrets are stored temporarily in Kubernetes as native secrets, which may pose a risk if not properly managed.	Secrets are mounted directly and not stored as native Kubernetes secrets, reducing exposure but relying on file permissions.
Customization and Features	Highly customizable with fine-grained mappings and transformation options.	Limited customization, focused on mounting secrets as-is.
Application Compatibility	Applications access secrets as Kubernetes secrets (e.g., via env variables or volumes).	Applications access secrets directly via mounted files or injected env variables.
Performance Considerations	Potential slight delay due to syncing secrets to Kubernetes secrets.	Faster access as secrets are directly mounted.
Community and Ecosystem	Supported by a growing community; integrates well with diverse secret management solutions.	Part of the Kubernetes ecosystem; supports CSI-compliant secret storage solutions.
Best Use Cases	Ideal for scenarios where secrets need to be accessed as Kubernetes native secrets.	Best for applications requiring direct access to secrets without intermediate storage in Kubernetes secrets.

You can find detailed information about all current available Pulumi ESC Kubernetes integrations in the Pulumi ESC documentation.

Let us know what you think about the Secret Store CSI Driver and how you are managing secrets in your Kubernetes clusters. We would love to hear your feedback and experiences.

Try Pulumi for Free

Getting Started with Amazon EKS Auto Mode in Pulumi

Florian Stadler — Mon, 16 Dec 2024 23:26:21 +0100

AWS recently announced Amazon EKS Auto Mode, a significant enhancement to Amazon EKS that streamlines cluster management by automating infrastructure decisions and operations. Today, we are excited to announce that Pulumi fully supports EKS Auto Mode across our AWS provider ecosystem, enabling you to leverage this powerful feature through infrastructure as code.

What is EKS Auto Mode?

EKS Auto Mode extends AWS management beyond the Kubernetes control plane to the underlying infrastructure that powers your workloads. While traditional EKS requires you to make numerous infrastructure decisions and maintain various components, Auto Mode simplifies this by automating these choices based on AWS’s operational expertise.

Instead of manually configuring and managing cluster add-ons, node groups, and infrastructure components, Auto Mode handles:

Compute Management: Automatically provisions and scales EC2 instances based on your workload demands while selecting optimal instance types and sizes. It also handles node lifecycle management, including OS updates and security patches.
Storage Automation: Eliminates the need to manually set up and configure the EBS CSI driver for block storage. Auto Mode manages volume provisioning, attachment, and cleanup for your persistent workloads.
Networking Simplification: Automatically configures and manages core networking components like Load Balancers.

This significantly reduces the operational overhead of running Kubernetes clusters. Instead of spending time on infrastructure decisions and maintenance, teams can now focus on their applications. For example, deploying a stateful application with load balancing no longer requires setting up and maintaining multiple add-ons and configurations - Auto Mode handles this automatically while following AWS best practices.

Common tasks that previously required significant expertise and time are now managed by AWS, making it simpler to run production-grade Kubernetes workloads. This includes:

Setting up and maintaining cluster autoscaling
Configuring and updating networking plugins
Managing storage drivers and volume provisioning
Handling node updates and security patches
Optimizing instance selection and placement

Pulumi Support for EKS Auto Mode

We have added EKS Auto Mode support across our AWS provider ecosystem:

Pulumi AWS Provider (v6.63.0+)
Pulumi AWS Cloud Control Provider (previously known as Pulumi AWS Native) (v1.13.0+)
Pulumi EKS Provider (v3.5.0+)

Creating an EKS Auto Mode Cluster

Let’s explore how to leverage EKS Auto Mode with Pulumi by deploying an example. You can enable EKS Auto Mode by setting autoMode.enabled to true.

First, we’ll create a new EKS cluster with Auto Mode enabled. This requires properly configured networking and authentication settings:

VPC subnets are tagged appropriately for EKS Auto Mode load balancer discovery
Authentication Mode supports Access Entries (Api or ApiAndConfigMap mode) as required by EKS Auto Mode
Default node groups and security groups are skipped as Auto Mode manages these

import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import { SubnetType } from "@pulumi/awsx/ec2";

const config = new pulumi.Config();
const clusterName = config.require("clusterName");

const eksVpc = new awsx.ec2.Vpc("eks-auto-mode", {
 enableDnsHostnames: true,
 cidrBlock: "10.0.0.0/16",
 subnetSpecs: [
 // Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 // See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 {type: SubnetType.Public, tags: {[`kubernetes.io/cluster/${clusterName}`]: "shared", "kubernetes.io/role/elb": "1"}},
 {type: SubnetType.Private, tags: {[`kubernetes.io/cluster/${clusterName}`]: "shared", "kubernetes.io/role/internal-elb": "1"}},
 ],
 subnetStrategy: "Auto"
});

const cluster = new eks.Cluster("eks-auto-mode", {
 name: clusterName,
 // EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 authenticationMode: eks.AuthenticationMode.Api,
 vpcId: eksVpc.vpcId,
 publicSubnetIds: eksVpc.publicSubnetIds,
 privateSubnetIds: eksVpc.privateSubnetIds,
 // Enables compute, storage and load balancing for the cluster.
 autoMode: {
 enabled: true,
 }
});

import pulumi
import pulumi_awsx as awsx
import pulumi_eks as eks
import pulumi_kubernetes as kubernetes

eks_vpc = awsx.ec2.Vpc("eksVpc",
 enable_dns_hostnames=True,
 cidr_block="10.0.0.0/16",
 # Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 # See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 subnet_specs=[
 {
 "type": awsx.ec2.SubnetType.PUBLIC,
 "tags": {
 "kubernetes_io_cluster_eks_auto_mode_demo": "shared",
 "kubernetes_io_role_elb": "1",
 },
 },
 {
 "type": awsx.ec2.SubnetType.PRIVATE,
 "tags": {
 "kubernetes_io_cluster_eks_auto_mode_demo": "shared",
 "kubernetes_io_role_internal_elb": "1",
 },
 },
 ],
 subnet_strategy=awsx.ec2.SubnetAllocationStrategy.AUTO)
cluster = eks.Cluster("cluster",
 name="eks-auto-mode-demo",
 # EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 authentication_mode=eks.AuthenticationMode.API,
 vpc_id=eks_vpc.vpc_id,
 public_subnet_ids=eks_vpc.public_subnet_ids,
 private_subnet_ids=eks_vpc.private_subnet_ids,
 # Enables compute, storage and load balancing for the cluster.
 auto_mode={
 "enabled": True,
 })

package main

import (
 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi-eks/sdk/v3/go/eks"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 networkingv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/networking/v1"
 storagev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/storage/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 eksVpc, err := ec2.NewVpc(ctx, "eksVpc", &ec2.VpcArgs{
 EnableDnsHostnames: pulumi.Bool(true),
 CidrBlock: "10.0.0.0/16",
 // Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 // See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 SubnetSpecs: []ec2.SubnetSpecArgs{
 {
 Type: ec2.SubnetTypePublic,
 Tags: {
 "kubernetes.io/cluster/eks-auto-mode-demo": pulumi.String("shared"),
 "kubernetes.io/role/elb": pulumi.String("1"),
 },
 },
 {
 Type: ec2.SubnetTypePrivate,
 Tags: {
 "kubernetes.io/cluster/eks-auto-mode-demo": pulumi.String("shared"),
 "kubernetes.io/role/internal-elb": pulumi.String("1"),
 },
 },
 },
 SubnetStrategy: ec2.SubnetAllocationStrategyAuto,
 })
 if err != nil {
 return err
 }
 cluster, err := eks.NewCluster(ctx, "cluster", &eks.ClusterArgs{
 Name: pulumi.String("eks-auto-mode-demo"),
 // EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 AuthenticationMode: eks.AuthenticationModeApi,
 VpcId: eksVpc.VpcId,
 PublicSubnetIds: eksVpc.PublicSubnetIds,
 PrivateSubnetIds: eksVpc.PrivateSubnetIds,
 // Enables compute, storage and load balancing for the cluster.
 AutoMode: &eks.AutoModeOptionsArgs{
 Enabled: true,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Awsx = Pulumi.Awsx;
using Eks = Pulumi.Eks;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 var eksVpc = new Awsx.Ec2.Vpc("eksVpc", new()
 {
 EnableDnsHostnames = true,
 CidrBlock = "10.0.0.0/16",
 // Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 // See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 SubnetSpecs = new()
 {
 new Awsx.Ec2.Inputs.SubnetSpecArgs
 {
 Type = Awsx.Ec2.SubnetType.Public,
 Tags =
 {
 { "kubernetes.io/cluster/eks-auto-mode-demo", "shared" },
 { "kubernetes.io/role/elb", "1" },
 },
 },
 new Awsx.Ec2.Inputs.SubnetSpecArgs
 {
 Type = Awsx.Ec2.SubnetType.Private,
 Tags =
 {
 { "kubernetes.io/cluster/eks-auto-mode-demo", "shared" },
 { "kubernetes.io/role/internal-elb", "1" },
 },
 },
 },
 SubnetStrategy = Awsx.Ec2.SubnetAllocationStrategy.Auto,
 });

 var cluster = new Eks.Cluster("cluster", new()
 {
 Name = "eks-auto-mode-demo",
 // EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 AuthenticationMode = Eks.AuthenticationMode.Api,
 VpcId = eksVpc.VpcId,
 PublicSubnetIds = eksVpc.PublicSubnetIds,
 PrivateSubnetIds = eksVpc.PrivateSubnetIds,
 // Enables compute, storage and load balancing for the cluster.
 AutoMode = new Eks.Inputs.AutoModeOptionsArgs
 {
 Enabled = true,
 },
 });
});

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.awsx.ec2.Vpc;
import com.pulumi.awsx.ec2.VpcArgs;
import com.pulumi.awsx.ec2.inputs.SubnetSpecArgs;
import com.pulumi.eks.Cluster;
import com.pulumi.eks.ClusterArgs;
import com.pulumi.eks.inputs.AutoModeOptionsArgs;
import com.pulumi.resources.CustomResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var eksVpc = new Vpc("eksVpc", VpcArgs.builder()
 .enableDnsHostnames(true)
 .cidrBlock("10.0.0.0/16")
 // Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 // See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 .subnetSpecs(
 SubnetSpecArgs.builder()
 .type("Public")
 .tags(Map.ofEntries(
 Map.entry("kubernetes.io/cluster/eks-auto-mode-demo", "shared"),
 Map.entry("kubernetes.io/role/elb", "1")
 ))
 .build(),
 SubnetSpecArgs.builder()
 .type("Private")
 .tags(Map.ofEntries(
 Map.entry("kubernetes.io/cluster/eks-auto-mode-demo", "shared"),
 Map.entry("kubernetes.io/role/internal-elb", "1")
 ))
 .build())
 .subnetStrategy("Auto")
 .build());

 var cluster = new Cluster("cluster", ClusterArgs.builder()
 .name("eks-auto-mode-demo")
 // EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 .authenticationMode("API")
 .vpcId(eksVpc.vpcId())
 .publicSubnetIds(eksVpc.publicSubnetIds())
 .privateSubnetIds(eksVpc.privateSubnetIds())
 // Enables compute, storage and load balancing for the cluster.
 .autoMode(AutoModeOptionsArgs.builder()
 .enabled(true)
 .build())
 .build());
 }
}

name: eks-auto-mode-demo
description: A demo of EKS Auto Mode
runtime: yaml
resources:
 eksVpc:
 type: awsx:ec2:Vpc
 properties:
 enableDnsHostnames: true
 cidrBlock: "10.0.0.0/16"
 # Necessary tags for EKS Auto Mode to identify the subnets for the load balancers.
 # See: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/subnet_discovery/
 subnetSpecs:
 - type: "Public"
 tags:
 kubernetes.io/cluster/eks-auto-mode-demo: shared
 kubernetes.io/role/elb: "1"
 - type: "Private"
 tags:
 kubernetes.io/cluster/eks-auto-mode-demo: shared
 kubernetes.io/role/internal-elb: "1"
 subnetStrategy: "Auto"
 cluster:
 type: eks:Cluster
 properties:
 name: eks-auto-mode-demo
 # EKS Auto Mode requires Access Entries, use either the `Api` or `ApiAndConfigMap` authentication mode.
 authenticationMode: API
 vpcId: ${eksVpc.vpcId}
 publicSubnetIds: ${eksVpc.publicSubnetIds}
 privateSubnetIds: ${eksVpc.privateSubnetIds}
 # Enables compute, storage and load balancing for the cluster.
 autoMode:
 enabled: true

Deploying a Load-Balanced Application

Now that we have provisioned an EKS cluster with Auto Mode enabled, let’s deploy a web application fronted by an Application Load Balancer. EKS Auto Mode will automatically provision appropriately sized nodes for your workload, set up the Application Load Balancer according to the Ingress configuration, and manage ongoing maintenance and updates.

//...
const appName = "nginx";
const ns = new k8s.core.v1.Namespace(appName, {
 metadata: { name: appName },
}, { provider: cluster.provider });

const configMap = new k8s.core.v1.ConfigMap(appName, {
 metadata: {
 namespace: ns.metadata.name,
 },
 data: {
 "index.html": "<html><body><h1>Hello, Pulumi!</h1></body></html>",
 },
}, { provider: cluster.provider });

const deployment = new k8s.apps.v1.Deployment(appName, {
 metadata: {
 namespace: ns.metadata.name
 },
 spec: {
 selector: { matchLabels: { app: appName } },
 replicas: 3,
 template: {
 metadata: { labels: { app: appName } },
 spec: {
 containers: [{
 name: appName,
 image: appName,
 ports: [{ containerPort: 80 }],
 volumeMounts: [{ name: "nginx-index", mountPath: "/usr/share/nginx/html" }],
 }],
 volumes: [{
 name: "nginx-index",
 configMap: { name: configMap.metadata.name },
 }],
 },
 },
 },
}, { provider: cluster.provider });

const service = new k8s.core.v1.Service(appName, {
 metadata: {
 name: appName,
 namespace: ns.metadata.name
 },
 spec: {
 selector: { app: appName },
 ports: [{ port: 80, targetPort: 80 }],
 },
}, { provider: cluster.provider, dependsOn: [deployment] });

const ingressClass = new k8s.networking.v1.IngressClass("alb", {
 metadata: {
 namespace: ns.metadata.name,
 labels: {
 "app.kubernetes.io/name": "LoadBalancerController",
 },
 name: "alb",
 },
 spec: {
 controller: "eks.amazonaws.com/alb",
 }
}, { provider: cluster.provider });

const ingress = new k8s.networking.v1.Ingress(appName, {
 metadata: {
 namespace: ns.metadata.name,
 // Annotations for EKS Auto Mode to identify the Ingress as internet-facing and target-type as IP.
 annotations: {
 "alb.ingress.kubernetes.io/scheme": "internet-facing",
 "alb.ingress.kubernetes.io/target-type": "ip",
 }
 },
 spec: {
 ingressClassName: ingressClass.metadata.name,
 rules: [{
 http: {
 paths: [{
 path: "/",
 pathType: "Prefix",
 backend: {
 service: {
 name: service.metadata.name,
 port: {
 number: 80,
 },
 },
 },
 }],
 },
 }],
 }
}, { provider: cluster.provider });

export const url = ingress.status.apply(status => status?.loadBalancer?.ingress?.[0]?.hostname);

# ...
k8s_provider = kubernetes.Provider("k8sProvider", kubeconfig=cluster.kubeconfig)
ns = kubernetes.core.v1.Namespace("ns", metadata={
 "name": "nginx",
},
opts = pulumi.ResourceOptions(provider=k8s_provider))
index_content = kubernetes.core.v1.ConfigMap("index-content",
 metadata={
 "namespace": ns.metadata.name,
 },
 data={
 "index.html": "<html><body><h1>Hello, Pulumi!</h1></body></html>",
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider))
deployment = kubernetes.apps.v1.Deployment("deployment",
 metadata={
 "namespace": ns.metadata.name,
 },
 spec={
 "selector": {
 "match_labels": {
 "app": "nginx",
 },
 },
 "replicas": 3,
 "template": {
 "metadata": {
 "labels": {
 "app": "nginx",
 },
 },
 "spec": {
 "containers": [{
 "name": "nginx",
 "image": "nginx",
 "ports": [{
 "container_port": 80,
 }],
 "volume_mounts": [{
 "name": "nginx-index",
 "mount_path": "/usr/share/nginx/html",
 }],
 }],
 "volumes": [{
 "name": "nginx-index",
 "config_map": {
 "name": index_content.metadata.name,
 },
 }],
 },
 },
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider))
service = kubernetes.core.v1.Service("service",
 metadata={
 "name": "nginx",
 "namespace": ns.metadata.name,
 },
 spec={
 "selector": {
 "app": "nginx",
 },
 "ports": [{
 "port": 80,
 "target_port": 80,
 }],
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider,
 depends_on=[deployment]))
ingress_class = kubernetes.networking.v1.IngressClass("ingressClass",
 metadata={
 "namespace": ns.metadata.name,
 "labels": {
 "app_kubernetes_io_name": "LoadBalancerController",
 },
 "name": "alb",
 },
 spec={
 "controller": "eks.amazonaws.com/alb",
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider))
ingress = kubernetes.networking.v1.Ingress("ingress",
 metadata={
 "namespace": ns.metadata.name,
 "annotations": {
 "alb_ingress_kubernetes_io_scheme": "internet-facing",
 "alb_ingress_kubernetes_io_target_type": "ip",
 },
 },
 spec={
 "ingress_class_name": ingress_class.metadata.name,
 "rules": [{
 "http": {
 "paths": [{
 "path": "/",
 "path_type": "Prefix",
 "backend": {
 "service": {
 "name": service.metadata.name,
 "port": {
 "number": 80,
 },
 },
 },
 }],
 },
 }],
 },
 opts = pulumi.ResourceOptions(provider=k8s_provider))

pulumi.export("url", ingress.status.load_balancer.ingress[0].hostname)

package main

import (
 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi-eks/sdk/v3/go/eks"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 networkingv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/networking/v1"
 storagev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/storage/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // ...
 k8sProvider, err := kubernetes.NewProvider(ctx, "k8sProvider", &kubernetes.ProviderArgs{
 Kubeconfig: cluster.Kubeconfig,
 })
 if err != nil {
 return err
 }
 ns, err := corev1.NewNamespace(ctx, "ns", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx"),
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }
 indexContent, err := corev1.NewConfigMap(ctx, "index-content", &corev1.ConfigMapArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Data: pulumi.StringMap{
 "index.html": pulumi.String("<html><body><h1>Hello, Pulumi!</h1></body></html>"),
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }
 deployment, err := appsv1.NewDeployment(ctx, "deployment", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Spec: &appsv1.DeploymentSpecArgs{
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Replicas: pulumi.Int(3),
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx"),
 Ports: corev1.ContainerPortArray{
 &corev1.ContainerPortArgs{
 ContainerPort: pulumi.Int(80),
 },
 },
 VolumeMounts: corev1.VolumeMountArray{
 &corev1.VolumeMountArgs{
 Name: pulumi.String("nginx-index"),
 MountPath: pulumi.String("/usr/share/nginx/html"),
 },
 },
 },
 },
 Volumes: corev1.VolumeArray{
 &corev1.VolumeArgs{
 Name: pulumi.String("nginx-index"),
 ConfigMap: &corev1.ConfigMapVolumeSourceArgs{
 Name: indexContent.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 },
 },
 },
 },
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }
 service, err := corev1.NewService(ctx, "service", &corev1.ServiceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx"),
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Spec: &corev1.ServiceSpecArgs{
 Selector: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 Ports: corev1.ServicePortArray{
 &corev1.ServicePortArgs{
 Port: pulumi.Int(80),
 TargetPort: pulumi.Any(80),
 },
 },
 },
 }, pulumi.Provider(k8sProvider), pulumi.DependsOn([]pulumi.Resource{
 deployment,
 }))
 if err != nil {
 return err
 }
 ingressClass, err := networkingv1.NewIngressClass(ctx, "ingressClass", &networkingv1.IngressClassArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Labels: pulumi.StringMap{
 "app.kubernetes.io/name": pulumi.String("LoadBalancerController"),
 },
 Name: pulumi.String("alb"),
 },
 Spec: &networkingv1.IngressClassSpecArgs{
 Controller: pulumi.String("eks.amazonaws.com/alb"),
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }
 ingress, err := networkingv1.NewIngress(ctx, "ingress", &networkingv1.IngressArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Annotations: pulumi.StringMap{
 "alb.ingress.kubernetes.io/scheme": pulumi.String("internet-facing"),
 "alb.ingress.kubernetes.io/target-type": pulumi.String("ip"),
 },
 },
 Spec: &networkingv1.IngressSpecArgs{
 IngressClassName: ingressClass.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Rules: networkingv1.IngressRuleArray{
 &networkingv1.IngressRuleArgs{
 Http: &networkingv1.HTTPIngressRuleValueArgs{
 Paths: networkingv1.HTTPIngressPathArray{
 &networkingv1.HTTPIngressPathArgs{
 Path: pulumi.String("/"),
 PathType: pulumi.String("Prefix"),
 Backend: &networkingv1.IngressBackendArgs{
 Service: &networkingv1.IngressServiceBackendArgs{
 Name: service.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Port: &networkingv1.ServiceBackendPortArgs{
 Number: pulumi.Int(80),
 },
 },
 },
 },
 },
 },
 },
 },
 },
 }, pulumi.Provider(k8sProvider))
 if err != nil {
 return err
 }

 ctx.Export("url", ingress.Status.ApplyT(func(status networkingv1.IngressStatus) (*string, error) {
 return &status.LoadBalancer.Ingress[0].Hostname, nil
 }).(pulumi.StringPtrOutput))
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Awsx = Pulumi.Awsx;
using Eks = Pulumi.Eks;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 // ...
 var k8sProvider = new Kubernetes.Provider.Provider("k8sProvider", new()
 {
 KubeConfig = cluster.Kubeconfig,
 });

 var ns = new Kubernetes.Core.V1.Namespace("ns", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "nginx",
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 var indexContent = new Kubernetes.Core.V1.ConfigMap("index-content", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Data =
 {
 { "index.html", "<html><body><h1>Hello, Pulumi!</h1></body></html>" },
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 var deployment = new Kubernetes.Apps.V1.Deployment("deployment", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Spec = new Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Selector = new Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "nginx" },
 },
 },
 Replicas = 3,
 Template = new Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Labels =
 {
 { "app", "nginx" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 Containers = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "nginx",
 Image = "nginx",
 Ports = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerPortArgs
 {
 ContainerPortValue = 80,
 },
 },
 VolumeMounts = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.VolumeMountArgs
 {
 Name = "nginx-index",
 MountPath = "/usr/share/nginx/html",
 },
 },
 },
 },
 Volumes = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.VolumeArgs
 {
 Name = "nginx-index",
 ConfigMap = new Kubernetes.Types.Inputs.Core.V1.ConfigMapVolumeSourceArgs
 {
 Name = indexContent.Metadata.Apply(metadata => metadata.Name),
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 var service = new Kubernetes.Core.V1.Service("service", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "nginx",
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.ServiceSpecArgs
 {
 Selector =
 {
 { "app", "nginx" },
 },
 Ports = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ServicePortArgs
 {
 Port = 80,
 TargetPort = 80,
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 DependsOn =
 {
 deployment,
 },
 });

 var ingressClass = new Kubernetes.Networking.V1.IngressClass("ingressClass", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 Labels =
 {
 { "app.kubernetes.io/name", "LoadBalancerController" },
 },
 Name = "alb",
 },
 Spec = new Kubernetes.Types.Inputs.Networking.V1.IngressClassSpecArgs
 {
 Controller = "eks.amazonaws.com/alb",
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 var ingress = new Kubernetes.Networking.V1.Ingress("ingress", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 Annotations =
 {
 { "alb.ingress.kubernetes.io/scheme", "internet-facing" },
 { "alb.ingress.kubernetes.io/target-type", "ip" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Networking.V1.IngressSpecArgs
 {
 IngressClassName = ingressClass.Metadata.Apply(metadata => metadata.Name),
 Rules = new[]
 {
 new Kubernetes.Types.Inputs.Networking.V1.IngressRuleArgs
 {
 Http = new Kubernetes.Types.Inputs.Networking.V1.HTTPIngressRuleValueArgs
 {
 Paths = new[]
 {
 new Kubernetes.Types.Inputs.Networking.V1.HTTPIngressPathArgs
 {
 Path = "/",
 PathType = "Prefix",
 Backend = new Kubernetes.Types.Inputs.Networking.V1.IngressBackendArgs
 {
 Service = new Kubernetes.Types.Inputs.Networking.V1.IngressServiceBackendArgs
 {
 Name = service.Metadata.Apply(metadata => metadata.Name),
 Port = new Kubernetes.Types.Inputs.Networking.V1.ServiceBackendPortArgs
 {
 Number = 80,
 },
 },
 },
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8sProvider,
 });

 return new Dictionary<string, object?>
 {
 ["url"] = ingress.Status.Apply(status => status?.LoadBalancer?.Ingress[0]?.Hostname),
 };
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.awsx.ec2.Vpc;
import com.pulumi.awsx.ec2.VpcArgs;
import com.pulumi.awsx.ec2.inputs.SubnetSpecArgs;
import com.pulumi.eks.Cluster;
import com.pulumi.eks.ClusterArgs;
import com.pulumi.eks.inputs.AutoModeOptionsArgs;
import com.pulumi.kubernetes.Provider;
import com.pulumi.kubernetes.ProviderArgs;
import com.pulumi.kubernetes.core_v1.Namespace;
import com.pulumi.kubernetes.core_v1.NamespaceArgs;
import com.pulumi.kubernetes.meta_v1.inputs.ObjectMetaArgs;
import com.pulumi.kubernetes.core_v1.ConfigMap;
import com.pulumi.kubernetes.core_v1.ConfigMapArgs;
import com.pulumi.kubernetes.apps_v1.Deployment;
import com.pulumi.kubernetes.apps_v1.DeploymentArgs;
import com.pulumi.kubernetes.apps_v1.inputs.DeploymentSpecArgs;
import com.pulumi.kubernetes.meta_v1.inputs.LabelSelectorArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodTemplateSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodSpecArgs;
import com.pulumi.kubernetes.core_v1.Service;
import com.pulumi.kubernetes.core_v1.ServiceArgs;
import com.pulumi.kubernetes.core_v1.inputs.ServiceSpecArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressClass;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressClassArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.inputs.IngressClassSpecArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.Ingress;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.inputs.IngressSpecArgs;
import com.pulumi.kubernetes.storage.k8s.io_v1.StorageClass;
import com.pulumi.kubernetes.storage.k8s.io_v1.StorageClassArgs;
import com.pulumi.kubernetes.core_v1.PersistentVolumeClaim;
import com.pulumi.kubernetes.core_v1.PersistentVolumeClaimArgs;
import com.pulumi.kubernetes.core_v1.inputs.PersistentVolumeClaimSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.VolumeResourceRequirementsArgs;
import com.pulumi.resources.CustomResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 // ...
 var k8sProvider = new Provider("k8sProvider", ProviderArgs.builder()
 .kubeconfig(cluster.kubeconfig())
 .build());

 var ns = new Namespace("ns", NamespaceArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx")
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 var indexContent = new ConfigMap("indexContent", ConfigMapArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .data(Map.of("index.html", "<html><body><h1>Hello, Pulumi!</h1></body></html>"))
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 var deployment = new Deployment("deployment", DeploymentArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .spec(DeploymentSpecArgs.builder()
 .selector(LabelSelectorArgs.builder()
 .matchLabels(Map.of("app", "nginx"))
 .build())
 .replicas(3)
 .template(PodTemplateSpecArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .labels(Map.of("app", "nginx"))
 .build())
 .spec(PodSpecArgs.builder()
 .containers(ContainerArgs.builder()
 .name("nginx")
 .image("nginx")
 .ports(ContainerPortArgs.builder()
 .containerPort(80)
 .build())
 .volumeMounts(VolumeMountArgs.builder()
 .name("nginx-index")
 .mountPath("/usr/share/nginx/html")
 .build())
 .build())
 .volumes(VolumeArgs.builder()
 .name("nginx-index")
 .configMap(ConfigMapVolumeSourceArgs.builder()
 .name(indexContent.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .build())
 .build())
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 var service = new Service("service", ServiceArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx")
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .spec(ServiceSpecArgs.builder()
 .selector(Map.of("app", "nginx"))
 .ports(ServicePortArgs.builder()
 .port(80)
 .targetPort(80)
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .dependsOn(deployment)
 .build());

 var ingressClass = new IngressClass("ingressClass", IngressClassArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .labels(Map.of("app.kubernetes.io/name", "LoadBalancerController"))
 .name("alb")
 .build())
 .spec(IngressClassSpecArgs.builder()
 .controller("eks.amazonaws.com/alb")
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 var ingress = new Ingress("ingress", IngressArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .annotations(Map.ofEntries(
 Map.entry("alb.ingress.kubernetes.io/scheme", "internet-facing"),
 Map.entry("alb.ingress.kubernetes.io/target-type", "ip")
 ))
 .build())
 .spec(IngressSpecArgs.builder()
 .ingressClassName(ingressClass.metadata().applyValue(metadata -> metadata.name()))
 .rules(IngressRuleArgs.builder()
 .http(HTTPIngressRuleValueArgs.builder()
 .paths(HTTPIngressPathArgs.builder()
 .path("/")
 .pathType("Prefix")
 .backend(IngressBackendArgs.builder()
 .service(IngressServiceBackendArgs.builder()
 .name(service.metadata().applyValue(metadata -> metadata.name()))
 .port(ServiceBackendPortArgs.builder()
 .number(80)
 .build())
 .build())
 .build())
 .build())
 .build())
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8sProvider)
 .build());

 ctx.export("url", ingress.status().applyValue(status -> status.loadBalancer().ingress()[0].hostname()));
 }
}

name: eks-auto-mode-yaml
description: A Pulumi YAML program to deploy a Kubernetes cluster on AWS
runtime: yaml
resources:
 # ...
 k8sProvider:
 type: pulumi:providers:kubernetes
 properties:
 kubeconfig: ${cluster.kubeconfig}
 ns:
 type: kubernetes:core/v1:Namespace
 properties:
 metadata:
 name: nginx
 options:
 provider: ${k8sProvider}
 index-content:
 type: kubernetes:core/v1:ConfigMap
 properties:
 metadata:
 namespace: ${ns.metadata.name}
 data:
 "index.html": "<html><body><h1>Hello, Pulumi!</h1></body></html>"
 options:
 provider: ${k8sProvider}
 deployment:
 type: kubernetes:apps/v1:Deployment
 properties:
 metadata:
 namespace: ${ns.metadata.name}
 spec:
 selector:
 matchLabels:
 app: nginx
 replicas: 3
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: nginx
 image: nginx
 ports:
 - containerPort: 80
 volumeMounts:
 - name: nginx-index
 mountPath: /usr/share/nginx/html
 volumes:
 - name: nginx-index
 configMap:
 name: ${index-content.metadata.name}
 options:
 provider: ${k8sProvider}
 service:
 type: kubernetes:core/v1:Service
 properties:
 metadata:
 name: nginx
 namespace: ${ns.metadata.name}
 spec:
 selector:
 app: nginx
 ports:
 - port: 80
 targetPort: 80
 options:
 provider: ${k8sProvider}
 dependsOn:
 - ${deployment}
 ingressClass:
 type: kubernetes:networking.k8s.io/v1:IngressClass
 properties:
 metadata:
 namespace: ${ns.metadata.name}
 labels:
 app.kubernetes.io/name: LoadBalancerController
 name: alb
 spec:
 controller: eks.amazonaws.com/alb
 options:
 provider: ${k8sProvider}
 ingress:
 type: kubernetes:networking.k8s.io/v1:Ingress
 properties:
 metadata:
 namespace: ${ns.metadata.name}
 annotations:
 alb.ingress.kubernetes.io/scheme: internet-facing
 alb.ingress.kubernetes.io/target-type: ip
 spec:
 ingressClassName: ${ingressClass.metadata.name}
 rules:
 - http:
 paths:
 - path: /
 pathType: Prefix
 backend:
 service:
 name: ${service.metadata.name}
 port:
 number: 80
 options:
 provider: ${k8sProvider}
outputs:
 url: ${ingress.status.loadBalancer.ingress[0].hostname}

After deploying this with pulumi up, you can access your application with curl once the Load Balancer has been provisioned:

~ curl $(pulumi stack output url)
Hello, Pulumi!

Deploying a Stateful Workload

EKS Auto Mode simplifies persistent storage management by automatically handling EBS volume provisioning for your stateful workloads. To leverage this feature, you can create a StorageClass that uses the EBS CSI driver (ebs.csi.eks.amazonaws.com) as its provisioner. EKS Auto Mode will then automatically manage the lifecycle of your persistent volumes, including provisioning, attaching, and cleanup.

This is how you’d configure persistent storage for a sample application:

// ...
const storageClass = new k8s.storage.v1.StorageClass("storage-class", {
 metadata: {
 name: "auto-ebs-sc",
 annotations: {
 "storageclass.kubernetes.io/is-default-class": "true",
 },
 },
 provisioner: "ebs.csi.eks.amazonaws.com",
 volumeBindingMode: "WaitForFirstConsumer",
 parameters: {
 type: "gp3",
 encrypted: "true",
 },
}, {
 provider: cluster.provider,
});
const pvc = new k8s.core.v1.PersistentVolumeClaim("pvc", {
 metadata: {
 name: "auto-ebs-claim",
 namespace: ns.metadata.apply(metadata => metadata.name),
 },
 spec: {
 accessModes: ["ReadWriteOnce"],
 storageClassName: storageClass.metadata.apply(metadata => metadata.name),
 resources: {
 requests: {
 storage: "8Gi",
 },
 },
 },
}, {
 provider: cluster.provider,
});
const statefulWorkload = new k8s.apps.v1.Deployment("stateful-workload", {
 metadata: {
 name: "inflate-stateful",
 namespace: ns.metadata.apply(metadata => metadata.name),
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: {
 app: "inflate-stateful",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "inflate-stateful",
 },
 },
 spec: {
 terminationGracePeriodSeconds: 0,
 nodeSelector: {
 "eks.amazonaws.com/compute-type": "auto",
 },
 containers: [{
 name: "bash",
 image: "public.ecr.aws/docker/library/bash:4.4",
 command: ["/usr/local/bin/bash"],
 args: [
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done",
 ],
 resources: {
 requests: {
 cpu: "1",
 },
 },
 volumeMounts: [{
 name: "persistent-storage",
 mountPath: "/data",
 }],
 }],
 volumes: [{
 name: "persistent-storage",
 persistentVolumeClaim: {
 claimName: pvc.metadata.name,
 },
 }],
 },
 },
 },
}, {
 provider: cluster.provider,
});

export const url = ingress.status.loadBalancer.ingress[0].hostname.apply(h => `http://${h}:80`);

# ...
storage_class = kubernetes.storage.v1.StorageClass("storage-class",
 metadata={
 "name": "auto-ebs-sc",
 "annotations": {
 "storageclass_kubernetes_io_is_default_class": "true",
 },
 },
 provisioner="ebs.csi.eks.amazonaws.com",
 volume_binding_mode="WaitForFirstConsumer",
 parameters={
 "type": "gp3",
 "encrypted": "true",
 },
 opts = pulumi.ResourceOptions(provider=k8_s_provider))
pvc = kubernetes.core.v1.PersistentVolumeClaim("pvc",
 metadata={
 "name": "auto-ebs-claim",
 "namespace": ns.metadata.name,
 },
 spec={
 "access_modes": ["ReadWriteOnce"],
 "storage_class_name": storage_class.metadata.name,
 "resources": {
 "requests": {
 "storage": "8Gi",
 },
 },
 },
 opts = pulumi.ResourceOptions(provider=k8_s_provider))
stateful_workload = kubernetes.apps.v1.Deployment("stateful-workload",
 metadata={
 "name": "inflate-stateful",
 "namespace": ns.metadata.name,
 },
 spec={
 "replicas": 1,
 "selector": {
 "match_labels": {
 "app": "inflate-stateful",
 },
 },
 "template": {
 "metadata": {
 "labels": {
 "app": "inflate-stateful",
 },
 },
 "spec": {
 "termination_grace_period_seconds": 0,
 "node_selector": {
 "eks_amazonaws_com_compute_type": "auto",
 },
 "containers": [{
 "name": "bash",
 "image": "public.ecr.aws/docker/library/bash:4.4",
 "command": ["/usr/local/bin/bash"],
 "args": [
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done",
 ],
 "resources": {
 "requests": {
 "cpu": "1",
 },
 },
 "volume_mounts": [{
 "name": "persistent-storage",
 "mount_path": "/data",
 }],
 }],
 "volumes": [{
 "name": "persistent-storage",
 "persistent_volume_claim": {
 "claim_name": pvc.metadata.name,
 },
 }],
 },
 },
 },
 opts = pulumi.ResourceOptions(provider=k8_s_provider))

package main

import (
 "github.com/pulumi/pulumi-awsx/sdk/v2/go/awsx/ec2"
 "github.com/pulumi/pulumi-eks/sdk/v3/go/eks"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 networkingv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/networking/v1"
 storagev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/storage/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // ...
 storageClass, err := storagev1.NewStorageClass(ctx, "storage-class", &storagev1.StorageClassArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("auto-ebs-sc"),
 Annotations: pulumi.StringMap{
 "storageclass.kubernetes.io/is-default-class": pulumi.String("true"),
 },
 },
 Provisioner: pulumi.String("ebs.csi.eks.amazonaws.com"),
 VolumeBindingMode: pulumi.String("WaitForFirstConsumer"),
 Parameters: pulumi.StringMap{
 "type": pulumi.String("gp3"),
 "encrypted": pulumi.String("true"),
 },
 }, pulumi.Provider(k8SProvider))
 if err != nil {
 return err
 }
 pvc, err := corev1.NewPersistentVolumeClaim(ctx, "pvc", &corev1.PersistentVolumeClaimArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("auto-ebs-claim"),
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Spec: &corev1.PersistentVolumeClaimSpecArgs{
 AccessModes: pulumi.StringArray{
 pulumi.String("ReadWriteOnce"),
 },
 StorageClassName: storageClass.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 Resources: &corev1.VolumeResourceRequirementsArgs{
 Requests: pulumi.StringMap{
 "storage": pulumi.String("8Gi"),
 },
 },
 },
 }, pulumi.Provider(k8SProvider))
 if err != nil {
 return err
 }
 _, err = appsv1.NewDeployment(ctx, "stateful-workload", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("inflate-stateful"),
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Spec: &appsv1.DeploymentSpecArgs{
 Replicas: pulumi.Int(1),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("inflate-stateful"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("inflate-stateful"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 TerminationGracePeriodSeconds: pulumi.Int(0),
 NodeSelector: pulumi.StringMap{
 "eks.amazonaws.com/compute-type": pulumi.String("auto"),
 },
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("bash"),
 Image: pulumi.String("public.ecr.aws/docker/library/bash:4.4"),
 Command: pulumi.StringArray{
 pulumi.String("/usr/local/bin/bash"),
 },
 Args: pulumi.StringArray{
 pulumi.String("-c"),
 pulumi.String("while true; do echo $(date -u) >> /data/out.txt; sleep 60; done"),
 },
 Resources: &corev1.ResourceRequirementsArgs{
 Requests: pulumi.StringMap{
 "cpu": pulumi.String("1"),
 },
 },
 VolumeMounts: corev1.VolumeMountArray{
 &corev1.VolumeMountArgs{
 Name: pulumi.String("persistent-storage"),
 MountPath: pulumi.String("/data"),
 },
 },
 },
 },
 Volumes: corev1.VolumeArray{
 &corev1.VolumeArgs{
 Name: pulumi.String("persistent-storage"),
 PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSourceArgs{
 ClaimName: pvc.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return &metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 },
 },
 },
 },
 },
 }, pulumi.Provider(k8SProvider))
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Awsx = Pulumi.Awsx;
using Eks = Pulumi.Eks;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 // ...
 var storageClass = new Kubernetes.Storage.V1.StorageClass("storage-class", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "auto-ebs-sc",
 Annotations =
 {
 { "storageclass.kubernetes.io/is-default-class", "true" },
 },
 },
 Provisioner = "ebs.csi.eks.amazonaws.com",
 VolumeBindingMode = "WaitForFirstConsumer",
 Parameters =
 {
 { "type", "gp3" },
 { "encrypted", "true" },
 },
 }, new CustomResourceOptions
 {
 Provider = k8SProvider,
 });

 var pvc = new Kubernetes.Core.V1.PersistentVolumeClaim("pvc", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "auto-ebs-claim",
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.PersistentVolumeClaimSpecArgs
 {
 AccessModes = new[]
 {
 "ReadWriteOnce",
 },
 StorageClassName = storageClass.Metadata.Apply(metadata => metadata.Name),
 Resources = new Kubernetes.Types.Inputs.Core.V1.VolumeResourceRequirementsArgs
 {
 Requests =
 {
 { "storage", "8Gi" },
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8SProvider,
 });

 var statefulWorkload = new Kubernetes.Apps.V1.Deployment("stateful-workload", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "inflate-stateful",
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Spec = new Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Replicas = 1,
 Selector = new Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "inflate-stateful" },
 },
 },
 Template = new Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Labels =
 {
 { "app", "inflate-stateful" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 TerminationGracePeriodSeconds = 0,
 NodeSelector =
 {
 { "eks.amazonaws.com/compute-type", "auto" },
 },
 Containers = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "bash",
 Image = "public.ecr.aws/docker/library/bash:4.4",
 Command = new[]
 {
 "/usr/local/bin/bash",
 },
 Args = new[]
 {
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done",
 },
 Resources = new Kubernetes.Types.Inputs.Core.V1.ResourceRequirementsArgs
 {
 Requests =
 {
 { "cpu", "1" },
 },
 },
 VolumeMounts = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.VolumeMountArgs
 {
 Name = "persistent-storage",
 MountPath = "/data",
 },
 },
 },
 },
 Volumes = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.VolumeArgs
 {
 Name = "persistent-storage",
 PersistentVolumeClaim = new Kubernetes.Types.Inputs.Core.V1.PersistentVolumeClaimVolumeSourceArgs
 {
 ClaimName = pvc.Metadata.Apply(metadata => metadata.Name),
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions
 {
 Provider = k8SProvider,
 });
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.awsx.ec2.Vpc;
import com.pulumi.awsx.ec2.VpcArgs;
import com.pulumi.awsx.ec2.inputs.SubnetSpecArgs;
import com.pulumi.eks.Cluster;
import com.pulumi.eks.ClusterArgs;
import com.pulumi.eks.inputs.AutoModeOptionsArgs;
import com.pulumi.kubernetes.Provider;
import com.pulumi.kubernetes.ProviderArgs;
import com.pulumi.kubernetes.core_v1.Namespace;
import com.pulumi.kubernetes.core_v1.NamespaceArgs;
import com.pulumi.kubernetes.meta_v1.inputs.ObjectMetaArgs;
import com.pulumi.kubernetes.core_v1.ConfigMap;
import com.pulumi.kubernetes.core_v1.ConfigMapArgs;
import com.pulumi.kubernetes.apps_v1.Deployment;
import com.pulumi.kubernetes.apps_v1.DeploymentArgs;
import com.pulumi.kubernetes.apps_v1.inputs.DeploymentSpecArgs;
import com.pulumi.kubernetes.meta_v1.inputs.LabelSelectorArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodTemplateSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodSpecArgs;
import com.pulumi.kubernetes.core_v1.Service;
import com.pulumi.kubernetes.core_v1.ServiceArgs;
import com.pulumi.kubernetes.core_v1.inputs.ServiceSpecArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressClass;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressClassArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.inputs.IngressClassSpecArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.Ingress;
import com.pulumi.kubernetes.networking.k8s.io_v1.IngressArgs;
import com.pulumi.kubernetes.networking.k8s.io_v1.inputs.IngressSpecArgs;
import com.pulumi.kubernetes.storage.k8s.io_v1.StorageClass;
import com.pulumi.kubernetes.storage.k8s.io_v1.StorageClassArgs;
import com.pulumi.kubernetes.core_v1.PersistentVolumeClaim;
import com.pulumi.kubernetes.core_v1.PersistentVolumeClaimArgs;
import com.pulumi.kubernetes.core_v1.inputs.PersistentVolumeClaimSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.VolumeResourceRequirementsArgs;
import com.pulumi.resources.CustomResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 // ...
 var storageClass = new StorageClass("storageClass", StorageClassArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("auto-ebs-sc")
 .annotations(Map.of("storageclass.kubernetes.io/is-default-class", "true"))
 .build())
 .provisioner("ebs.csi.eks.amazonaws.com")
 .volumeBindingMode("WaitForFirstConsumer")
 .parameters(Map.ofEntries(
 Map.entry("type", "gp3"),
 Map.entry("encrypted", "true")
 ))
 .build(), CustomResourceOptions.builder()
 .provider(k8SProvider)
 .build());

 var pvc = new PersistentVolumeClaim("pvc", PersistentVolumeClaimArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("auto-ebs-claim")
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .spec(PersistentVolumeClaimSpecArgs.builder()
 .accessModes("ReadWriteOnce")
 .storageClassName(storageClass.metadata().applyValue(metadata -> metadata.name()))
 .resources(VolumeResourceRequirementsArgs.builder()
 .requests(Map.of("storage", "8Gi"))
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8SProvider)
 .build());

 var statefulWorkload = new Deployment("statefulWorkload", DeploymentArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("inflate-stateful")
 .namespace(ns.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .spec(DeploymentSpecArgs.builder()
 .replicas(1)
 .selector(LabelSelectorArgs.builder()
 .matchLabels(Map.of("app", "inflate-stateful"))
 .build())
 .template(PodTemplateSpecArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .labels(Map.of("app", "inflate-stateful"))
 .build())
 .spec(PodSpecArgs.builder()
 .terminationGracePeriodSeconds(0)
 .nodeSelector(Map.of("eks.amazonaws.com/compute-type", "auto"))
 .containers(ContainerArgs.builder()
 .name("bash")
 .image("public.ecr.aws/docker/library/bash:4.4")
 .command("/usr/local/bin/bash")
 .args(
 "-c",
 "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done")
 .resources(ResourceRequirementsArgs.builder()
 .requests(Map.of("cpu", "1"))
 .build())
 .volumeMounts(VolumeMountArgs.builder()
 .name("persistent-storage")
 .mountPath("/data")
 .build())
 .build())
 .volumes(VolumeArgs.builder()
 .name("persistent-storage")
 .persistentVolumeClaim(PersistentVolumeClaimVolumeSourceArgs.builder()
 .claimName(pvc.metadata().applyValue(metadata -> metadata.name()))
 .build())
 .build())
 .build())
 .build())
 .build())
 .build(), CustomResourceOptions.builder()
 .provider(k8SProvider)
 .build());
 }
}

name: eks-auto-mode-yaml
description: A Pulumi YAML program to deploy a Kubernetes cluster on AWS
runtime: yaml
resources:
 # ...
 storage-class:
 type: kubernetes:storage.k8s.io/v1:StorageClass
 properties:
 metadata:
 name: auto-ebs-sc
 annotations:
 storageclass.kubernetes.io/is-default-class: "true"
 provisioner: ebs.csi.eks.amazonaws.com
 volumeBindingMode: WaitForFirstConsumer
 parameters:
 type: gp3
 encrypted: "true"
 options:
 provider: ${k8sProvider}
 pvc:
 type: kubernetes:core/v1:PersistentVolumeClaim
 properties:
 metadata:
 name: auto-ebs-claim
 namespace: ${ns.metadata.name}
 spec:
 accessModes:
 - ReadWriteOnce
 storageClassName: ${storage-class.metadata.name}
 resources:
 requests:
 storage: 8Gi
 options:
 provider: ${k8sProvider}
 stateful-workload:
 type: kubernetes:apps/v1:Deployment
 properties:
 metadata:
 name: inflate-stateful
 namespace: ${ns.metadata.name}
 spec:
 replicas: 1
 selector:
 matchLabels:
 app: inflate-stateful
 template:
 metadata:
 labels:
 app: inflate-stateful
 spec:
 terminationGracePeriodSeconds: 0
 nodeSelector:
 eks.amazonaws.com/compute-type: auto
 containers:
 - name: bash
 image: public.ecr.aws/docker/library/bash:4.4
 command: ["/usr/local/bin/bash"]
 args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 60; done"]
 resources:
 requests:
 cpu: "1"
 volumeMounts:
 - name: persistent-storage
 mountPath: /data
 volumes:
 - name: persistent-storage
 persistentVolumeClaim:
 claimName: ${pvc.metadata.name}
 options:
 provider: ${k8sProvider}
outputs:
 url: ${ingress.status.loadBalancer.ingress[0].hostname}

Summary

EKS Auto Mode represents a significant step forward in simplifying Kubernetes operations on AWS. Pulumi’s support for this feature allows you to define EKS Auto Mode clusters using your favorite programming language and integrate them with your existing infrastructure code.

To get started with EKS Auto Mode in Pulumi:

Update your AWS providers to the latest versions
Check out our EKS documentation
Join our Community Slack for support and discussions

We are excited to see what you build with EKS Auto Mode and Pulumi!

Why Every Platform Engineer Should Care About Kubernetes Operators

Engin Diri — Tue, 10 Dec 2024 09:31:38 +0100

In one of my recent talks, I mentioned that the foundation of a successful Kubernetes-powered platform is the use of Kubernetes Operators, as they are a great way to automate operational tasks and the lifecycle of complex applications and services on Kubernetes.

In case you missed the talk, here is the link to the recording:

After the talk, I received feedback from the audience that they would like to have more information about Kubernetes operators and some of the advanced operators I would recommend to use in their Kubernetes-powered platforms. And who am I to reject such a great opportunity to share my knowledge with the community?

As you guess, the base for a Kubernetes Operator is, well, Kubernetes. So, if you are not familiar with Kubernetes, let me give you a brief introduction. Kubernetes, or K8s in short, is by far the most powerful open-source container orchestration platform out there - especially for its capability to maintain a desired state. This means you, as the platform admin, can define how you want your cluster and the workload on top of it to look, and Kubernetes (aka the control plane) executes the necessary steps to ensure the desired state is achieved using a feedback loop.

And how does Kubernetes achieve this complex operational task management? Kubernetes leverages the concept of controllers and operators. Both ensure that the cluster resources are in the desired state. But why is there a distinction between controllers and operators inside the Kubernetes ecosystem? There is also great confusion when folks talk about controllers and operators, as the differences are in the details and operators are a subcategory of controllers.

Kubernetes Controllers vs. Operators

K8s controllers are control loops that watch your resources and reconcile the current state with the desired state in a continuous loop. Think about how a thermostat works in your house. The thermostat watches the current temperature and, if it is below the desired temperature, it turns on the heating. If it is above the desired temperature, it turns off the heating. This is how a controller works in Kubernetes - it watches the resources and conditions defined in the desired state and executes the necessary steps to make sure the current state matches the desired state.

Here are some typical Kubernetes controller key functions as part of the maintain, observe, and enforce loop of the control loop:

Function	Description
Rollouts and Rollbacks	Deployment controller takes care of rolling out new versions of your application and rolling back to the previous version in case of failure.
Cluster maintenance	Node controller watches the nodes in the cluster and makes sure that the nodes are in the desired state.
Event handling	Event controller watches for events and takes action based on the events.
Schedule	Controller to run jobs and cron jobs at a specific time.
Resource enforcement	Resource controller watches the resource usage and enforces resource limits in a namespace.
State enforcement	Different controllers watch the state of the resources and enforce the desired state.

As you can see, controllers automate the routine operational tasks, enforce policies, handle failures and take actions to continuously maintain the desired state of the cluster.

Some of the typical use cases for controllers are:

Limit resource usage - Prevent the overload of the cluster by enforcing resource limits with controllers like ResourceQuota.
Auto-scaling - Automatically scales the number of pods based on the different metrics with HorizontalPodAutoscaler.
Task scheduling - Run jobs and cron jobs at a specific time with Job and CronJob controllers.
Run stateful workloads - Deploy and manage stateful workloads like databases with StatefulSet controller, by ensuring that the pods are created in a specific order and have a stable network identity.
Scaling deployments - Deploy and manage stateless workloads like web applications with the Deployment controller, by ensuring that the desired number of pods is running and handling the rolling updates and rollbacks.

Controllers, as you see, reduce the manual effort for the platform engineering teams and gives the cluster self-managing capabilities.

With all the above points, the benefits of controllers are clear - Let me list some of the main ones:

Automation - Controllers automate the routine operational tasks and enforce policies.
Availability - Controllers ensure that the resources are always available, addressing failures and taking actions to maintain the desired state.
Efficiency - Controllers provide an efficient way to manage the resources and workloads in the cluster with quota management and auto-scaling.
Reliability - Controllers help to make workload management more reliable with features like replica management, pod creation and deletion, and scheduling.
Flexibility - Controllers provide a flexible way to handle a variety of workloads and resources in the cluster, such as DaemonSet, StatefulSet, jobs, cron jobs, deployments, and services.
Observability - Controllers provide a view into the cluster resources and the state of the resources via the Kubernetes API.

With Controllers explained, let’s move on to the next level of automation in Kubernetes and talk about Kubernetes Operators.

Kubernetes Operators

As mentioned before, Kubernetes Operators are a subcategory of controllers that use API extensions (Custom Resource) to complete the automation task. Operators are a set of independent controllers, with each controller responsible for its own task. And while controllers can share similar functions with a controller, it is only focused on one domain and only uses Custom Resources to manage the domain.

Keep in mind that controllers work without the need for custom resources or a link to a specific domain.

Operators are custom-built controllers that focus on the deployment, management, and operation of a specific application or service. Some key functions of operators include:

Function	Description
Security	Operators integrate security best practices and policies, such as encryption, access control, and secret management, to protect the application. You can customize these measures and adjust them to the specific needs of the application.
Workload Lifecycle management	Operators manages the full lifecycle of complex stateful applications through deployment, upgrades, and scaling. It even applies logic to automate operations like backup and restore.
Error Handling	Operators detects and handles errors, such as application crashes, network failures, and resource constraints.
Migration	Operators help to migrate the application to a new version, a new cluster, or a new cloud provider.
Disaster Recovery	Operators help to recover the application from a disaster, such as data loss, network failure, or hardware failure.

Operators are a great way to encode the operational knowledge of the platform engineering team into automated controllers. This way, the platform engineering team can focus on the strategic tasks and let the operators handle the boring routine operational tasks. This reduces the manual effort and increases the reliability of the platform.

Some of the typical use cases for operators are:

Database management - Deploy and manage databases such as MySQL, PostgreSQL, and MongoDB.
Storage management - Deploy and manage storage such as Ceph, GlusterFS, or NFS.
Logging and monitoring - Simplify the deployment and management of logging and monitoring solutions like Prometheus, Grafana, or ELK stack.
CI/CD - Automate the deployment and management of CI/CD pipelines like Jenkins, GitLab, or Tekton. - Backup and restore - Automate the backup and restore of applications and data.
Messaging and event streaming - Simplify the deployment and management of messaging and event streaming solutions like Kafka, RabbitMQ, or NATS.
Machine learning - Automate the deployment and management of machine learning workloads such as TensorFlow, PyTorch, or Kubeflow.

With the applications-centric approach provided by operators, the platform engineering team can now enable fully automated operations while enhancing the reliability, efficiency, and observability of the platform and ensuring adherence to best practices and compliance requirements.

And similar to controllers, operators also have distinct benefits:

Simplification - Operators simplify the deployment and management of complex applications and services by providing a declarative way (via Custom Resources).
Higher Productivity - Operators increase the productivity of the platform engineering team by automating the routine operational tasks and allowing users to focus on delivering business value.
Extensibility - Extending the K8s API with Custom Resource Definitions (CRDs) allows administrators to support new applications and services without changing the core Kubernetes code.
Consistency - Operators ensure that they deploy and run the applications and services consistently across different environments, such as on-premises, cloud, or hybrid.
Modularity - Operators are modular and focused on a specific domain.
Legacy Transformation - Operators help to transform legacy applications into cloud-native applications by automating the deployment and management of complex applications.

Here is a visual representation of a Kubernetes Operator in action, which I found very nice and helpful:

Kubernetes Operators: what are they? Some examples Credit: SparkFabrik

Comparison of Kubernetes Controllers and Operators

Here is a quick summary of the differences between Kubernetes controllers and operators:

Feature	Kubernetes Controllers	Kubernetes Operators
Definition	Core control loops within Kubernetes that reconcile the current state with the desired state of resources.	Custom-built controllers with domain-specific logic, typically extending Kubernetes API with Custom Resource Definitions (CRDs).
Scope	General-purpose automation for cluster-wide resource management and workload operations.	Focused on application or service-specific lifecycle management and operations.
Custom Resources (CRDs)	Not mandatory. Most controllers work with built-in Kubernetes resources like Pods, Deployments, and StatefulSets.	Mandatory. Operators depend on CRDs to define and manage application-specific resources.
Domain-Specific Logic	Limited. Primarily focuses on generic Kubernetes operations like scheduling, scaling, and resource enforcement.	Extensive. Includes application-specific tasks such as backups, upgrades, migrations, and disaster recovery.
Automation	Automates operational tasks like resource scheduling, scaling, and maintaining cluster states.	Automates both operational and domain-specific tasks, often encoding deep operational knowledge of specific applications.
Complexity	Relatively low complexity as they focus on Kubernetes-native workflows.	Higher complexity due to domain-specific implementations and integrations with external systems.
Observability	Provides insight into cluster resource states via Kubernetes API and built-in events.	Enhances observability with application-specific metrics, logs, and health checks tailored to the managed application.
Use Cases	- Pod scaling (e.g., HPA) - Resource enforcement (e.g., ResourceQuota) - Rolling updates and rollbacks - Scheduling jobs (e.g., CronJob).	- Database management (e.g., MySQL Operator) - CI/CD pipelines (e.g., Tekton) - Backup and restore - Monitoring solutions (e.g., Prometheus Operator) - Stateful applications (e.g., Kafka Operator).
Deployment	Part of Kubernetes core components or extensions like kube-controller-manager.	External tools developed by vendors, community, or in-house, deployed using Custom Resources.
Key Benefits	- Cluster-wide automation - Ensures resource availability - Efficient and reliable workload management.	- Simplifies application lifecycle management - Increases productivity with domain-specific automation - Consistent deployments across environments.
Flexibility	Flexible for general Kubernetes operations but limited to built-in resources.	Highly flexible and extensible for custom applications and domain-specific requirements.
Target Audience	Platform administrators and Kubernetes users looking to automate Kubernetes-native workflows.	Developers and platform engineers needing advanced automation and lifecycle management for specific applications or services.
Maintenance	Managed and updated by Kubernetes core or community projects.	Requires domain-specific knowledge and regular updates to accommodate application changes and best practices.

With this detailed explanation of Kubernetes controllers and operators, I would like to share some advanced Kubernetes operators that I recommend for use in your Kubernetes-powered platform.

Advanced Kubernetes Operators

As we have seen, Kubernetes Operators are the go-to way to automate operational tasks and lifecycle management. I should mention that not all operators are created equal. Some operators are more advanced and provide more features than others. That is why, for example, OperatorHub.io categorises operators into five capability levels:

Basic Install - Installs the application on the cluster.
Seamless Upgrades - Manages Application upgrades.
Full Lifecycle Management - Manages the full lifecycle of the application.
Deep Insights - Provides monitoring and metrics.
Auto Pilot - Automatically optimizes and tunes the application.

The more advanced the operator, the more features it provides and the more automation it offers. At its most advanced level, the Auto Pilot, the operator automatically optimizes and tunes the application based on the workload and environment.

Here is a list of advanced Kubernetes operators that I recommend using in your Kubernetes-powered platform:

CloudNativePG Operator

CloudNativePG Operator Credit: cloudnative-pg.io

CloudNativePG Operator is a PostgreSQL operator that covers the full lifecycle of PostgreSQL databases on Kubernetes. It simplifies the deployment, management, and scaling of PostgreSQL databases by providing a declarative way to define and manage PostgreSQL clusters using custom resources.

Some of the key features of CloudNativePG Operator are:

Automated Backups - Schedule and manage backups of PostgreSQL databases.
High Availability - Deploy and manage high-availability PostgreSQL clusters.
Scaling - Scale the number of PostgreSQL instances based on the workload.

Installation

Either use the Helm chart or the Kubernetes manifests to deploy the CloudNativePG Operator:

helm repo add cloudnative-pg https://cloudnative-pg.io/charts/
helm repo update
helm upgrade --install cloudnativepg cloudnative-pg/cloudnative-pg --namespace cnpg-system --create-namespace

Or use Pulumi to deploy the CloudNativePG Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const ns = new k8s.core.v1.Namespace("cnpg-system", {
 metadata: {
 name: "cnpg-system",
 },
});

new k8shelm.Release("cloudnativepg", {
 chart: "oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 namespace: ns.metadata.name,
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

ns = k8s.core.v1.Namespace(
 "cnpg-system",
 metadata=meta.ObjectMetaArgs(
 name="cnpg-system",
 ),
)

release = k8shelm.Release(
 "cloudnativepg",
 chart="oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 namespace=ns.metadata["name"],
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 ns, err := corev1.NewNamespace(ctx, "cnpg-system", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("cnpg-system"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "cloudnativepg", &helmv3.ReleaseArgs{
 Chart: pulumi.String("oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg"),
 Namespace: ns.Metadata.Name().Elem(),
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class MyStack : Stack
{
 public MyStack()
 {
 var ns = new Namespace("cnpg-system", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "cnpg-system"
 }
 });

 var release = new Release("cloudnativepg", new ReleaseArgs
 {
 Chart = "oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 Namespace = ns.Metadata.Apply(m => m.Name),
 CreateNamespace = true
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is a simple example of a PostgreSQL cluster with backup configured on S3 API-compatible storage:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: cluster-example-with-backup
spec:
 instances: 3
 primaryUpdateStrategy: unsupervised

 # Persistent storage configuration
 storage:
 storageClass: standard
 size: 1Gi

 # Backup properties
 # This assumes a local minio setup
 backup:
 barmanObjectStore:
 destinationPath: s3://backups/
 endpointURL: http://minio:9000
 s3Credentials:
 accessKeyId:
 name: minio
 key: ACCESS_KEY_ID
 secretAccessKey:
 name: minio
 key: ACCESS_SECRET_KEY
 wal:
 compression: gzip

Or use Pulumi:

new k8s.apiextensions.CustomResource("cluster-example-with-backup", {
 apiVersion: "postgresql.cnpg.io/v1",
 kind: "Cluster",
 metadata: {
 name: "cluster-example-with-backup",
 },
 spec: {
 instances: 3,
 primaryUpdateStrategy: "unsupervised",

 // Persistent storage configuration
 storage: {
 storageClass: "standard",
 size: "1Gi",
 },

 // Backup properties
 backup: {
 barmanObjectStore: {
 destinationPath: "s3://backups/",
 endpointURL: "http://minio:9000",
 s3Credentials: {
 accessKeyId: {
 name: "minio",
 key: "ACCESS_KEY_ID",
 },
 secretAccessKey: {
 name: "minio",
 key: "ACCESS_SECRET_KEY",
 },
 },
 wal: {
 compression: "gzip",
 },
 },
 },
 },
});

from pulumi_kubernetes import apiextensions

postgresql_cluster = apiextensions.CustomResource(
 "cluster-example-with-backup",
 api_version="postgresql.cnpg.io/v1",
 kind="Cluster",
 metadata=meta.ObjectMetaArgs(
 name="cluster-example-with-backup",
 ),
 spec={
 "instances": 3,
 "primaryUpdateStrategy": "unsupervised",
 # Persistent storage configuration
 "storage": {
 "storageClass": "standard",
 "size": "1Gi",
 },
 # Backup properties
 "backup": {
 "barmanObjectStore": {
 "destinationPath": "s3://backups/",
 "endpointURL": "http://minio:9000",
 "s3Credentials": {
 "accessKeyId": {
 "name": "minio",
 "key": "ACCESS_KEY_ID",
 },
 "secretAccessKey": {
 "name": "minio",
 "key": "ACCESS_SECRET_KEY",
 },
 },
 "wal": {
 "compression": "gzip",
 },
 },
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "cluster-example-with-backup", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("postgresql.cnpg.io/v1"),
 Kind: pulumi.String("Cluster"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("cluster-example-with-backup"),
 },
 OtherFields: k8s.UntypedArgs{
 "instances": pulumi.Int(3),
 "primaryUpdateStrategy": pulumi.String("unsupervised"),
 "storage": pulumi.Map{
 "storageClass": pulumi.String("standard"),
 "size": pulumi.String("1Gi"),
 },
 "backup": pulumi.Map{
 "barmanObjectStore": pulumi.Map{
 "destinationPath": pulumi.String("s3://backups/"),
 "endpointURL": pulumi.String("http://minio:9000"),
 "s3Credentials": pulumi.Map{
 "accessKeyId": pulumi.Map{
 "name": pulumi.String("minio"),
 "key": pulumi.String("ACCESS_KEY_ID"),
 },
 "secretAccessKey": pulumi.Map{
 "name": pulumi.String("minio"),
 "key": pulumi.String("ACCESS_SECRET_KEY"),
 },
 },
 "wal": pulumi.Map{
 "compression": pulumi.String("gzip"),
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class PostGreSQLClusterArgs : CustomResourceArgs
{
 [Input("postgresql.cnpg.io/v1")]
 public Input<string> ApiVersion { get; set; }
 [Input("Cluster")]
 public Input<string> Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Others { get; set; }
 [Input("spec")]
 public InputMap<object> Spec { get; set; }

 public PostGreSQLClusterArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}
class MyStack : Stack
{
 public MyStack()
 {
 var postgresqlCluster = new Pulumi.Kubernetes.ApiExtensions.CustomResource("cluster-example-with-backup", new PostGreSQLClusterArgs("postgresql.cnpg.io/v1", "Cluster")
 {
 ApiVersion = "postgresql.cnpg.io/v1",
 Kind = "Cluster",
 Metadata = new ObjectMetaArgs
 {
 Name = "cluster-example-with-backup",
 },
 Spec = new InputMap<object>
 {
 { "instances", 3 },
 { "primaryUpdateStrategy", "unsupervised" },
 { "storage", new InputMap<object>
 {
 { "storageClass", "standard" },
 { "size", "1Gi" },
 }
 },
 { "backup", new InputMap<object>
 {
 { "barmanObjectStore", new InputMap<object>
 {
 { "destinationPath", "s3://backups/" },
 { "endpointURL", "http://minio:9000" },
 { "s3Credentials", new InputMap<object>
 {
 { "accessKeyId", new InputMap<object>
 {
 { "name", "minio" },
 { "key", "ACCESS_KEY_ID" },
 }
 },
 { "secretAccessKey", new InputMap<object>
 {
 { "name", "minio" },
 { "key", "ACCESS_SECRET_KEY" },
 }
 },
 }
 },
 { "wal", new InputMap<object>
 {
 { "compression", "gzip" },
 }
 },
 }
 },
 }
 },
 }
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

CloudNativePG Operator is a great way to deploy and manage PostgreSQL databases on Kubernetes. It simplifies a wide range of tasks, including backups, high availability, and scaling, making it a must-have for PostgreSQL users.

You might also like:

Flux Operator

Flux Operator Credit: fluxcd.control-plane.io/operator

The Flux Operator is a full autopilot solution that provides an alternative to the traditional Flux Bootstrap procedure, removing the operational burden of managing Flux across multiple clusters by automating the installation, configuration, and upgrade of Flux fully.

Some of the key features of Flux Operator are:

GitOps Deployment - Automate the deployment of applications and services using GitOps.
Lifecycle Management - Manage the full lifecycle of Flux, including upgrades and rollbacks.
Deep Insights - Provide deep insights into the state of Flux and the applications deployed with Flux.

Installation

Use the Helm chart to deploy the Flux Operator:

helm upgrade -i flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator --namespace flux-system --create-namespace

Or use Pulumi to deploy the Flux Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const namespace = new k8s.core.v1.Namespace("flux-system", {
 metadata: {
 name: "flux-system",
 },
});

new k8s.helm.v3.Release("flux-operator", {
 chart: "oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 namespace: namespace.metadata.name,
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

flux_namespace = k8s.core.v1.Namespace(
 "flux-system",
 metadata=meta.ObjectMetaArgs(
 name="flux-system",
 ),
)
flux_operator = k8shelm.Release(
 "flux-operator",
 chart="oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 namespace=flux_namespace.metadata["name"],
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 namespace, err := corev1.NewNamespace(ctx, "flux-system", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("flux-system"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "flux-operator", &helmv3.ReleaseArgs{
 Chart: pulumi.String("oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator"),
 Namespace: namespace.Metadata.Name().Elem(),
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

 var namespaceFluxSystem = new Namespace("flux-system", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "flux-system",
 },
 });

 var fluxOperator = new Release("flux-operator", new ReleaseArgs
 {
 Chart = "oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 Namespace = namespaceFluxSystem.Metadata.Apply(metadata => metadata.Name),
 CreateNamespace = true,
 });
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is an example of a FluxInstance Custom Resource that defines a Flux instance:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
 name: flux
 namespace: flux-system
 annotations:
 fluxcd.controlplane.io/reconcileEvery: "1h"
 fluxcd.controlplane.io/reconcileTimeout: "5m"
spec:
 distribution:
 version: "2.x"
 registry: "ghcr.io/fluxcd"
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"
 components:
 - source-controller
 - kustomize-controller
 - helm-controller
 - notification-controller
 - image-reflector-controller
 - image-automation-controller
 cluster:
 type: kubernetes
 multitenant: false
 networkPolicy: true
 domain: "cluster.local"
 kustomize:
 patches:
 - target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s

Or use Pulumi:

new k8s.apiextensions.CustomResource("flux", {
 apiVersion: "fluxcd.controlplane.io/v1",
 kind: "FluxInstance",
 metadata: {
 name: "flux",
 namespace: "flux-system",
 annotations: {
 "fluxcd.controlplane.io/reconcileEvery": "1h",
 "fluxcd.controlplane.io/reconcileTimeout": "5m",
 },
 },
 spec: {
 distribution: {
 version: "2.x",
 registry: "ghcr.io/fluxcd",
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests",
 },
 components: ["source-controller", "kustomize-controller", "helm-controller", "notification-controller", "image-reflector-controller", "image-automation-controller"],
 cluster: {
 type: "kubernetes",
 multitenant: false,
 networkPolicy: true,
 domain: "cluster.local",
 },
 kustomize: {
 patches: [
 {
 target: {
 kind: "Deployment",
 name: "(kustomize-controller|helm-controller)",
 },
 patch: `
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s
 `,
 },
 ],
 },
 },
});

from pulumi_kubernetes import apiextensions

flux_instance = k8s.apiextensions.CustomResource(
 "flux",
 api_version="fluxcd.controlplane.io/v1",
 kind="FluxInstance",
 metadata={
 "name": "flux",
 "namespace": flux_namespace.metadata["name"],
 "annotations": {
 "fluxcd.controlplane.io/reconcileEvery": "1h",
 "fluxcd.controlplane.io/reconcileTimeout": "5m",
 },
 },
 spec={
 "distribution": {
 "version": "2.x",
 "registry": "ghcr.io/fluxcd",
 "artifact": "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests",
 },
 "components": [
 "source-controller",
 "kustomize-controller",
 "helm-controller",
 "notification-controller",
 "image-reflector-controller",
 "image-automation-controller",
 ],
 "cluster": {
 "type": "kubernetes",
 "multitenant": False,
 "networkPolicy": True,
 "domain": "cluster.local",
 },
 "kustomize": {
 "patches": [
 {
 "target": {
 "kind": "Deployment",
 "name": "(kustomize-controller|helm-controller)",
 },
 "patch": [
 {
 "op": "add",
 "path": "/spec/template/spec/containers/0/args/-",
 "value": "--concurrent=10",
 },
 {
 "op": "add",
 "path": "/spec/template/spec/containers/0/args/-",
 "value": "--requeue-dependency=5s",
 },
 ],
 }
 ]
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "flux", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("fluxcd.controlplane.io/v1"),
 Kind: pulumi.String("FluxInstance"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("flux"),
 Namespace: pulumi.String("flux-system"),
 Annotations: pulumi.StringMap{
 "fluxcd.controlplane.io/reconcileEvery": pulumi.String("1h"),
 "fluxcd.controlplane.io/reconcileTimeout": pulumi.String("5m"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "distribution": pulumi.Map{
 "version": pulumi.String("2.x"),
 "registry": pulumi.String("ghcr.io/fluxcd"),
 "artifact": pulumi.String("oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"),
 },
 "components": pulumi.StringArray{
 pulumi.String("source-controller"),
 pulumi.String("kustomize-controller"),
 pulumi.String("helm-controller"),
 pulumi.String("notification-controller"),
 pulumi.String("image-reflector-controller"),
 pulumi.String("image-automation-controller"),
 },
 "cluster": pulumi.Map{
 "type": pulumi.String("kubernetes"),
 "multitenant": pulumi.Bool(false),
 "networkPolicy": pulumi.Bool(true),
 "domain": pulumi.String("cluster.local"),
 },
 "kustomize": pulumi.Map{
 "patches": pulumi.Array{
 pulumi.Map{
 "target": pulumi.Map{
 "kind": pulumi.String("Deployment"),
 "name": pulumi.String("(kustomize-controller|helm-controller)"),
 },
 "patch": pulumi.Array{
 pulumi.Map{
 "op": pulumi.String("add"),
 "path": pulumi.String("/spec/template/spec/containers/0/args/-"),
 "value": pulumi.String("--concurrent=10"),
 },
 pulumi.Map{
 "op": pulumi.String("add"),
 "path": pulumi.String("/spec/template/spec/containers/0/args/-"),
 "value": pulumi.String("--requeue-dependency=5s"),
 },
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class FluxInstanceArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public FluxInstanceArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}
 var flux = new Pulumi.Kubernetes.ApiExtensions.CustomResource("flux", new FluxInstanceArgs("fluxcd.controlplane.io/v1", "FluxInstance")
 {
 ApiVersion = "fluxcd.controlplane.io/v1",
 Kind = "FluxInstance",
 Metadata = new ObjectMetaArgs
 {
 Name = "flux",
 Annotations =
 {
 { "fluxcd.controlplane.io/reconcileEvery", "1h" },
 { "fluxcd.controlplane.io/reconcileTimeout", "5m" },
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "distribution", new Dictionary<string, object>
 {
 { "version", "2.x" },
 { "registry", "ghcr.io/fluxcd" },
 { "artifact", "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests" }
 }
 },
 { "components", new[]
 {
 "source-controller",
 "kustomize-controller",
 "helm-controller",
 "notification-controller",
 "image-reflector-controller",
 "image-automation-controller"
 }
 },
 { "cluster", new Dictionary<string, object>
 {
 { "type", "kubernetes" },
 { "multitenant", false },
 { "networkPolicy", true },
 { "domain", "cluster.local" }
 }
 },
 { "kustomize", new Dictionary<string, object>
 {
 { "patches", new[]
 {
 new Dictionary<string, object>
 {
 { "target", new Dictionary<string, object>
 {
 { "kind", "Deployment" },
 { "name", "(kustomize-controller|helm-controller)" }
 }
 },
 { "patch", new[]
 {
 new Dictionary<string, object>
 {
 { "op", "add" },
 { "path", "/spec/template/spec/containers/0/args/-" },
 { "value", "--concurrent=10" }
 },
 new Dictionary<string, object>
 {
 { "op", "add" },
 { "path", "/spec/template/spec/containers/0/args/-" },
 { "value", "--requeue-dependency=5s" }
 }
 }
 }
 }
 }
 }
 }
 }
 },
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

The Flux Operator is another excellent example of an advanced Kubernetes operator, providing full lifecycle management with numerous automation features.

Strimzi Operator

Strimzi Operator Credit: strimzi.io

Last but not least, the Strimzi Operator is a full lifecycle management operator for Apache Kafka on Kubernetes. As event-driven architectures are becoming increasingly popular, Apache Kafka has become the de-facto standard for building scalable and reliable event streaming platforms. The Strimzi Operator makes it easier to deploy, manage, and scale Apache Kafka clusters on Kubernetes.

Some of the key features of Strimzi Operator are:

Cluster Management - Deploy and manage Apache Kafka clusters.
Mirror Clusters - Deploy and manage mirror clusters for disaster recovery.
Streaming Capabilities - With Kafka Connect you can stream data between Kafka and other systems.

Installation

Use the Helm chart to deploy the Strimzi Operator:

helm repo add strimzi https://strimzi.io/charts/
helm repo update
helm install strimzi strimzi/strimzi-kafka-operator -n kafka --create-namespace

Or use Pulumi to deploy the Strimzi Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const namespaceKafka = new k8s.core.v1.Namespace("kafka", {
 metadata: {
 name: "kafka",
 },
});

new k8s.helm.v3.Release("strimzi-kafka-operator", {
 chart: "strimzi-kafka-operator",
 namespace: namespaceKafka.metadata.name,
 repositoryOpts: {
 repo: "https://strimzi.io/charts/",
 },
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

kafka_namespace = k8s.core.v1.Namespace(
 "kafka",
 metadata=meta.ObjectMetaArgs(
 name="kafka",
 ),
)

strimzi_kafka_operator = k8shelm.Release(
 "strimzi-kafka-operator",
 chart="strimzi-kafka-operator",
 namespace=kafka_namespace.metadata["name"],
 repository_opts={
 "repo": "https://strimzi.io/charts/",
 },
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 namespaceKafka, err := corev1.NewNamespace(ctx, "kafka", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("kafka"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "strimzi-kafka-operator", &helmv3.ReleaseArgs{
 Chart: pulumi.String("strimzi-kafka-operator"),
 Namespace: namespaceKafka.Metadata.Name(),
 RepositoryOpts: helmv3.RepositoryOptsArgs{
 Repo: pulumi.String("https://strimzi.io/charts/"),
 },
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;
class MyStack : Stack
{
 public MyStack()
 {
 var namespaceKafka = new Namespace("kafka", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "kafka",
 }
 });

 var strimziKafkaOperator = new Release("strimzi-kafka-operator", new ReleaseArgs
 {
 Chart = "strimzi-kafka-operator",
 Namespace = namespaceKafka.Metadata.Apply(ns => ns.Name),
 RepositoryOpts = new RepositoryOptsArgs
 {
 Repo = "https://strimzi.io/charts/",
 },
 CreateNamespace = true,
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is an example of a Kafka Custom Resource that defines a Kafka cluster:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
 name: example-kafka-node-pool
 labels:
 strimzi.io/cluster: example-kafka-cluster
spec:
 replicas: 1
 roles:
 - controller
 - broker
 storage:
 type: ephemeral
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
 name: example-kafka-cluster
 annotations:
 strimzi.io/kraft: enabled
 strimzi.io/node-pools: enabled
spec:
 kafka:
 replicas: 1
 version: 3.8.0
 storage:
 type: ephemeral
 metadataVersion: 3.8-IV0
 listeners:
 - name: plain
 port: 9092
 type: internal
 tls: false
 - name: tls
 port: 9093
 type: internal
 tls: true
 config:
 offsets.topic.replication.factor: 1
 transaction.state.log.replication.factor: 1
 transaction.state.log.min.isr: 1
 default.replication.factor: 1
 min.insync.replicas: 1
 entityOperator:
 topicOperator: {}
 userOperator: {}

This example defines a Kafka cluster with one replica and two listeners, one for plain and one for TLS and uses ephemeral storage.

To create a topic, you can use the following KafkaTopic Custom Resource:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
 name: example-kafka-cluster
 labels:
 strimzi.io/cluster:
spec:
 partitions: 5
 replicas: 1
 config:
 retention.ms: 7200000
 segment.bytes: 1073741824

Or use Pulumi:

new k8s.apiextensions.CustomResource("example-kafka-node-pool", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "KafkaNodePool",
 metadata: {
 name: "example-kafka-node-pool",
 labels: {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 namespace: "kafka",
 },
 spec: {
 replicas: 1,
 roles: ["controller", "broker"],
 storage: {
 type: "ephemeral",
 },
 },
});

new k8s.apiextensions.CustomResource("example-kafka-cluster", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "Kafka",
 metadata: {
 name: "example-kafka-cluster",
 annotations: {
 "strimzi.io/kraft": "enabled",
 "strimzi.io/node-pools": "enabled",
 },
 namespace: "kafka",
 },
 spec: {
 kafka: {
 replicas: 1,
 version: "3.8.0",
 storage: {
 type: "ephemeral",
 },
 metadataVersion: "3.8-IV0",
 listeners: [
 {
 name: "plain",
 port: 9092,
 type: "internal",
 tls: false,
 },
 {
 name: "tls",
 port: 9093,
 type: "internal",
 tls: true,
 },
 ],
 config: {
 "offsets.topic.replication.factor": 1,
 "transaction.state.log.replication.factor": 1,
 "transaction.state.log.min.isr": 1,
 "default.replication.factor": 1,
 "min.insync.replicas": 1,
 },
 },
 entityOperator: {
 topicOperator: {},
 userOperator: {},
 },
 },
});

new k8s.apiextensions.CustomResource("example-kafka-topic", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "KafkaTopic",
 metadata: {
 name: "example-kafka-topic",
 labels: {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 namespace: "kafka",
 },
 spec: {
 partitions: 5,
 replicas: 1,
 config: {
 "retention.ms": 7200000,
 "segment.bytes": 1073741824,
 },
 },
});

from pulumi_kubernetes import apiextensions

kafka_node_pool = k8s.apiextensions.CustomResource(
 "example-kafka-node-pool",
 api_version="kafka.strimzi.io/v1beta2",
 kind="KafkaNodePool",
 metadata={
 "name": "example-kafka-node-pool",
 "labels": {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "replicas": 1,
 "roles": ["controller", "broker"],
 "storage": {
 "type": "ephemeral",
 },
 },
)

kafka_cluster = k8s.apiextensions.CustomResource(
 "example-kafka-cluster",
 api_version="kafka.strimzi.io/v1beta2",
 kind="Kafka",
 metadata={
 "name": "example-kafka-cluster",
 "annotations": {
 "strimzi.io/kraft": "enabled",
 "strimzi.io/node-pools": "enabled",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "kafka": {
 "replicas": 1,
 "version": "3.8.0",
 "storage": {
 "type": "ephemeral",
 },
 "metadataVersion": "3.8-IV0",
 "listeners": [
 {
 "name": "plain",
 "port": 9092,
 "type": "internal",
 "tls": False,
 },
 {
 "name": "tls",
 "port": 9093,
 "type": "internal",
 "tls": True,
 },
 ],
 "config": {
 "offsets.topic.replication.factor": 1,
 "transaction.state.log.replication.factor": 1,
 "transaction.state.log.min.isr": 1,
 "default.replication.factor": 1,
 "min.insync.replicas": 1,
 },
 },
 "entityOperator": {
 "topicOperator": {},
 "userOperator": {},
 },
 },
)

kafka_topic = k8s.apiextensions.CustomResource(
 "example-kafka-topic",
 api_version="kafka.strimzi.io/v1beta2",
 kind="KafkaTopic",
 metadata={
 "name": "example-kafka-topic",
 "labels": {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "partitions": 5,
 "replicas": 1,
 "config": {
 "retention.ms": 7200000,
 "segment.bytes": 1073741824,
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-node-pool", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("KafkaNodePool"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-node-pool"),
 Namespace: pulumi.String("kafka"),
 Labels: pulumi.StringMap{
 "strimzi.io/cluster": pulumi.String("example-kafka-cluster"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "replicas": pulumi.Int(1),
 "roles": pulumi.StringArray{
 pulumi.String("controller"),
 pulumi.String("broker"),
 },
 "storage": pulumi.Map{
 "type": pulumi.String("ephemeral"),
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-cluster", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("Kafka"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-cluster"),
 Namespace: pulumi.String("kafka"),
 Annotations: pulumi.StringMap{
 "strimzi.io/kraft": pulumi.String("enabled"),
 "strimzi.io/node-pools": pulumi.String("enabled"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "kafka": pulumi.Map{
 "replicas": pulumi.Int(1),
 "version": pulumi.String("3.8.0"),
 "storage": pulumi.Map{
 "type": pulumi.String("ephemeral"),
 },
 "metadataVersion": pulumi.String("3.8-IV0"),
 "listeners": pulumi.Array{
 pulumi.Map{
 "name": pulumi.String("plain"),
 "port": pulumi.Int(9092),
 "type": pulumi.String("internal"),
 "tls": pulumi.Bool(false),
 },
 pulumi.Map{
 "name": pulumi.String("tls"),
 "port": pulumi.Int(9093),
 "type": pulumi.String("internal"),
 "tls": pulumi.Bool(true),
 },
 },
 "config": pulumi.Map{
 "offsets.topic.replication.factor": pulumi.Int(1),
 "transaction.state.log.replication.factor": pulumi.Int(1),
 "transaction.state.log.min.isr": pulumi.Int(1),
 "default.replication.factor": pulumi.Int(1),
 "min.insync.replicas": pulumi.Int(1),
 },
 },
 "entityOperator": pulumi.Map{
 "topicOperator": pulumi.Map{},
 "userOperator": pulumi.Map{},
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-topic", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("KafkaTopic"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-topic"),
 Namespace: pulumi.String("kafka"),
 Labels: pulumi.StringMap{
 "strimzi.io/cluster": pulumi.String("example-kafka-cluster"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "partitions": pulumi.Int(5),
 "replicas": pulumi.Int(1),
 "config": pulumi.Map{
 "retention.ms": pulumi.Int(7200000),
 "segment.bytes": pulumi.Int(1073741824),
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class KafkaNodePoolArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaNodePoolArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class KafkaClusterArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaClusterArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class KafkaTopicArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaTopicArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class MyStack : Stack
{
 public MyStack()
 {
 var kafkaNodePool = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaNodePool", new KafkaNodePoolArgs("kafka.strimzi.io/v1beta2", "KafkaNodePool")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "KafkaNodePool",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-node-pool",
 Labels =
 {
 { "strimzi.io/cluster", "example-kafka-cluster" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "replicas", 1 },
 { "roles", new List<string> { "controller", "broker" } },
 { "storage", new Dictionary<string, object> { { "type", "ephemeral" } } }
 }
 });

 var kafkaCluster = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaCluster", new KafkaClusterArgs("kafka.strimzi.io/v1beta2", "Kafka")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "Kafka",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-cluster",
 Annotations =
 {
 { "strimzi.io/kraft", "enabled" },
 { "strimzi.io/node-pools", "enabled" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "kafka", new Dictionary<string, object> {
 { "replicas", 1 },
 { "version", "3.8.0" },
 { "storage", new Dictionary<string, object> { { "type", "ephemeral" } } },
 { "metadataVersion", "3.8-IV0" },
 { "listeners", new List<Dictionary<string, object>> {
 new Dictionary<string, object> {
 { "name", "plain" },
 { "port", 9092 },
 { "type", "internal" },
 { "tls", false }
 },
 new Dictionary<string, object> {
 { "name", "tls" },
 { "port", 9093 },
 { "type", "internal" },
 { "tls", true }
 }
 }},
 { "config", new Dictionary<string, object> {
 { "offsets.topic.replication.factor", 1 },
 { "transaction.state.log.replication.factor", 1 },
 { "transaction.state.log.min.isr", 1 },
 { "default.replication.factor", 1 },
 { "min.insync.replicas", 1 }
 }}
 }},
 { "entityOperator", new Dictionary<string, object> {
 { "topicOperator", new Dictionary<string, object>() },
 { "userOperator", new Dictionary<string, object>() }
 }}
 }
 });

 var kafkaTopic = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaTopic", new KafkaTopicArgs("kafka.strimzi.io/v1beta2", "KafkaTopic")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "KafkaTopic",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-topic",
 Labels =
 {
 { "strimzi.io/cluster", "example-kafka-cluster" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "partitions", 5 },
 { "replicas", 1 },
 { "config", new Dictionary<string, object> {
 { "retention.ms", 7200000 },
 { "segment.bytes", 1073741824 }
 }}
 }
 });
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

The Strimzi Operator demonstrates the power of Kubernetes Operators by providing full lifecycle management for Apache Kafka on Kubernetes. It takes care of the heavy lifting of deploying and managing Kafka clusters, making the Strimzi Operator a valuable tool that should be part of every Kubernetes-powered platform, as it covers a wide range of use cases.

Conclusion

In this blog post, I highlighted the importance of Kubernetes Operators for platform engineers and how they can harness the advanced automation capabilities of operators to simplify deploying, managing, and scaling applications and services on Kubernetes.

I provided some examples of advanced Kubernetes operators that I recommend, but if you want to try using Kubernetes Operators in your Kubernetes-powered platform, here are some great guides to get you started with Pulumi and Kubernetes:

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

105 Ways to Run Containers: The Cloud Container Iceberg

Adam Gordon Bell — Tue, 03 Dec 2024 10:29:54 -0500

If a computer can be connected to the internet, someone has tried to run a container on it. From quantum computers to smart toasters, from phones in AWS racks to CI pipelines, there’s many ways to deploy containers. While most people only know about the mainstream cloud providers at the surface, there’s actually a vast world of increasingly unusual and specialized options beneath.

This guide serves two purposes: to showcase the surprising breadth of container options available today, and to help you understand the full spectrum of choices – from practical to very experimental. Whether you’re looking for production-ready solutions or just curious about what’s out there in the dark depths, you’ll find something interesting here.

Constraints

But first, we need ground rules for our container adventures. All listed items should:

Technically possible: Must work today with existing tech, even if the setup makes you question your life choices.
Runs real containers: OCI containers only. No JavaScript “containers” pretending to be the real thing.
Actually doable: You can set this up right now with a laptop and a credit card. No theoretical schemes allowed.

Above the iceberg – The usual suspects

These are the familiar faces of container orchestration—the ones you’d introduce to your parents. Reliable, well-documented, and about as exciting as watching paint dry.

Amazon Elastic Kubernetes Service (EKS)
Amazon Elastic Container Service (ECS)
AWS Fargate
Azure Kubernetes Service (AKS)
Google Kubernetes Engine (GKE)
Google Cloud Run
Amazon EC2
AWS App Runner
Azure App Service for Containers
Google Compute Engine with Containers

On the iceberg – The slightly less obvious

As we dip just below the surface, we find services like a rare B-side acoustic release from a big band. It’s not as mainstream but still a respectable choice for running your containers.

AWS Elastic Beanstalk – Deploy containers, debug YAML
Amazon Lightsail Containers – AWS with the guard rails on
Azure Container Apps – Serverless, but complicated
Google App Engine Flexible Environment – PaaS for Containerized Apps
Azure Container Instances (ACI) – Kubernetes without the chaos

The kubernetes multiverse – Because one Universe of k8s wasn’t enough

Diving deeper, we discover there’s an entire multiverse of regional and specialty K8s providers, helpfully cataloged by the CNCF. Think parallel universes, but instead of evil twins, you get slightly different container runtime implementations.

The Global Titans – The international tech giants couldn’t resist joining the K8s party: Huawei Cloud Container Engine, Tencent, and Baidu bring their own flavor of cloud-native orchestration to the mix. Add in Samsung, China Mobile KCS, JD Cloud, Inspur, and QingCloud, and you have enough options to start a diplomatic incident.
The European Union of Containers – Europeans take their container privacy seriously: IONOS and plusserver bring German engineering to K8s, OVH adds French flair, Exoscale offers Swiss precision, and STACKIT, SysEleven, WIIT, Gcore, and Previder ensure your containers comply with every possible EU regulation. Conoa Proact throws in some Swedish design principles for good measure.
The Asia-Pacific Pod Party – The APAC region isn’t missing out: NHN brings Korean optimization to your clusters, while ARDC Nectar and Catalyst let you run containers upside down in Australia and New Zealand. TKS Cloud Service ensures your containers can run anywhere from Tokyo to Tasmania.
The Specialists’ Society – For when regular K8s isn’t quirky enough: OKESTRO VIOLA orchestrates your containers with musical flair, while Petasus, De Novo, Aruba, and Taikun provide enough specialty K8s options to ensure every cluster is a unique snowflake.

Below the water – Kubernetes everywhere – Let a thousand K8s bloom

Let’s plunge deeper. Because one Kubernetes cluster is never enough, why not sprinkle them everywhere? On-premises, in the cloud, under your desk—there’s a K8s solution for all that.

Amazon EKS Anywhere – AWS container but in your own data center
Amazon ECS Anywhere – Like EKS Anywhere but ECS
Azure Arc – Hybrid cloud for masochists
Google Anthos – Hybrid cloud something something
AWS Outposts – AWS, now with hardware shipping delays
Azure Red Hat OpenShift – Azure but with a fedora
VMware Tanzu – Because VMware heard you like Kubernetes
Red Hat OpenShift – Fedoras on AWS
IBM Cloud Kubernetes Service – Kubernetes with a touch of enterprise nostalgia
Oracle Container Engine for Kubernetes (OKE) – For the brave souls trusting Oracle
Alibaba Cloud Elastic Container Service (ACK) – K8s with Chinese characteristics
DigitalOcean Kubernetes (DOKS) – K8s in the ocean, hopefully no sharks
Linode Kubernetes Engine (LKE) – Lightweight K8s for the minimalist
Vultr Kubernetes Engine (VKE) – Yet another place to run your pods
Scaleway Kapsule – French elegance meets Kubernetes
Hetzner Cloud Managed Kubernetes – German engineering for your clusters
Nutanix Karbon - Because regular Kubernetes wasn’t converged enough
Metal Stack Cloud Kubernetes - For when virtual machines are too virtual

You might also like:

Middle of the iceberg – Serverless and function containers – Now you see me, now you don’t

As we descend to the depths, we find containers with commitment issues. They appear, do their job, and disappear.

AWS Batch – Containers that only work the night shift
AWS Lambda – Lambda’s containers for when a 250 MB zip file wasn’t enough baggage.
Azure Functions – Roll your own functions in containers and hope for the best.
Google Cloud Functions – Containers pretending to be functions, but now on Google Cloud.
Alibaba Function Compute – You get the idea.

Bottom of the iceberg – Specialized and hybrid platforms – The swiss army knife of container hosting

Down here, containers run in ways that make people question your life choices. These platforms let you host containers in environments many fear to tread.

AWS Proton – Automate your deployments and run a container
Azure Virtual Machines with Docker – When you miss managing servers but love containers
IBM Cloud Code Engine – Run anything, anywhere, if you can figure it out
Alibaba Elastic Container Instance (ECI) – Containers that disappear faster than you can say “Jack Ma”

Below the iceberg – Containers on the edge of reason

In these lightless depths, you want your containers to live life on the edge—literally. Because nothing says “enterprise-ready” like running production workloads on a smart toaster.

AWS IoT Greengrass – Run AWS Lambda and Docker containers on everything from factory equipment to smart fridges. Nothing says “digital transformation” like your smart fridge running machine learning models.
Azure IoT Edge – Like Greengrass, but only when your toaster prefers Windows updates.
AWS Snowball Edge – For when ‘cloud migration’ means paying FedEx to move your containers. Yes, it’s literally a box they mail you that can run containers.
AWS Snowcone – An even smaller box that runs just enough containers to make you appreciate your data center. Perfect for when you need AWS in places where AWS doesn’t want to build a region.
BalenaCloud IoT Device Management – A platform that lets you remotely deploy and manage containers on fleets of Raspberry Pis. Because nothing says “enterprise IoT” like running your production stack on $35 computers.
Fly.io – Global Edge Container Deployment, because apparently just picking one region was too simple.
Edge Node Service (Tencent Cloud) – Edge deployment behind great firewall.
StackPath Edge Computing – More edge containers.
Koyeb – Serverless Edge containers.
Platform.sh – Multi-Cloud Edge containers.
StarlingX - Edge computing platform for when 5G and IoT buzzwords aren’t enough
ZStack Edge - Edge computing with Chinese characteristics, now at the edge of the network, too

The abyss – Where containers fear to tread

And now we reach the darkest depths. For the bravest souls who look at conventional container platforms and think, “Not edgy enough.” Here lie the most unusual ways to run containers in production.

Amazon GameLift – Run your containerized services disguised as game servers. High scores may apply.
AWS Device Farm – Deploy containers on hundreds of actual phones in AWS racks. Perfect for when “mobile-first” goes too far.
AWS AppStream 2.0 – Stream Windows to run Docker to stream containers. It’s virtualization all the way down.
Google Cloud Dataflow – Hide containerized services in pipeline steps named “daily_data_normalize_37”. Hope your SRE enjoys debugging ETL.
Google Cloud Composer – Your containerized microservice masquerades as an Airflow DAG. Because nothing says reliability like cron.
Google Colab & Jupyter Notebooks – Run containers in data_preprocessing_step.ipynb. Watch ML teams debug production.
Azure Quantum – Package quantum containers with az quantum job submit. Now, your uptime is truly uncertain.
Azure Cognitive Services in Containers – Offline AI containers that promise not to become self-aware. Terms and conditions may apply.
Azure DevTest Labs – Run containerized test environments that self-destruct like Mission Impossible tapes. Perfect for when temporary becomes permanent.

Beyond the abyss

There you have it, the container iceberg. And I didn’t even have room to add the extra cursed ways you can run containers. Here is some of what didn’t make it:

Cloud IDEs and Workspaces – Want to turn your development environment into a sneaky container platform? Take your pick: GitHub Codespaces, GitPod for the GitLab enthusiasts, or Eclipse Che for the masochists who miss Eclipse crashing locally. JetBrains Space or Replit? All this lets you run a container until you close your browser tab.
GitHub Actions et al. – Ever wanted to run a production service disguised as a CI job? Just label your production API start-up script as “integration_test_step_42”, add a sleep 21600 and a call to retrigger itself, and hope nobody notices your “test” has been running for months. Apply this same trick to AWS CodeBuild, GitLab CI/CD, CircleCI, Azure Pipelines, Travis CI, Bitbucket Pipelines, Drone CI, Jenkins Cloud, TeamCity Cloud, and Google Cloud Build for Bonus points.

Managing your container zoo with pulumi

Whether you’re deploying containers to mainstream cloud providers or attempting to run them on quantum computers (really, please let us know how that goes), Pulumi helps you manage it all with real code instead of config file soup. Our Kubernetes support means you can wrangle your containers wherever they might roam – from AWS to Azure, GCP to Kubernetes, or even that smart toaster you’re eyeing for production deployment.

With Pulumi, you get:

Cloud-native container orchestration across all major platforms and Kubernetes distributions
Infrastructure testing to ensure your containers deploy correctly (even in the weird places)
Provider templates for when you invent yet another way to run containers
Stack references for managing multiple environments without losing your mind
Automation API for programmatically herding your container fleets

Ready to bring some sanity to your container deployment strategy? Get started with Pulumi:

Try Pulumi for Free

The final word

At this point, 105 options in, you might wonder: “Has cloud container hosting gone too far?” Well, I’ve a solution to propose:

At Pulumi, we’ve got providers for all the usual suspects and, surprisingly, many more in this container circus. And if you’ve figured out how to run containers on a quantum computer inside a data pipeline in New Zealand, you can write a provider for that, too.

So, use Pulumi and we will figure out how RedHat-Tanzu-Greengrass-Icecore-Device-Farm works and provide you a sane interface to it. You’ve got other things to worry about.

Amazon EKS Hybrid Nodes: Bridging Cloud and On-Premises

Josh Kodroff — Sun, 01 Dec 2024 15:06:25 -0500

AWS has introduced a new feature for Amazon Elastic Kubernetes Service (EKS): Amazon EKS Hybrid Nodes. This addition to the Amazon EKS Hybrid/Edge portfolio allows organizations to maintain their Kubernetes control plane in AWS while running workloads on-premises or at the edge. This hybrid approach offers the best of both worlds - AWS’s reliable and scalable control plane management combined with the flexibility to run workloads wherever they make the most sense for your business.

Amazon EKS Hybrid Nodes are particularly valuable for organizations that need to keep certain workloads on-premises due to data sovereignty requirements, latency concerns, or existing infrastructure investments. By extending EKS to support on-premises workloads, AWS is addressing a critical need in the enterprise market, especially for industries like manufacturing, telecommunications, and healthcare that often require local data processing.

Pulumi’s strategic partnership with AWS has always focused on making cloud infrastructure easier to manage through modern infrastructure as code. As an EKS Service Ready Partner, Pulumi provides first-class support for AWS services, including EKS. This partnership ensures that as AWS releases new features like EKS Hybrid Nodes, Pulumi users can quickly adopt and integrate them into their infrastructure management workflows. Managing Hybrid Environments with Pulumi

Pulumi’s infrastructure as code platform is perfectly positioned to help organizations manage their hybrid EKS environments. Using Pulumi, teams can define and manage both their EKS clusters and Kubernetes resources using their favorite programming languages, bringing software engineering practices to infrastructure management. This includes managing Hybrid Node configurations, networking setup, and security policies across both cloud and on-premises environments.

Pulumi’s Environments Secrets and Config (ESC) adds another layer of capability to Hybrid Node deployments. Operators can securely manage OIDC IAM credentials, kubeconfig files and other sensitive credentials through ESC, ensuring secure access to EKS clusters regardless of where the workloads run. Furthermore, pods running on Hybrid Nodes can consume ESC secrets through Pulumi ESC’s integration with the Kubernetes External Secrets Operator, maintaining a consistent secrets management approach across your entire hybrid infrastructure.

Pulumi’s solutions for EKS Anywhere are automatically validated across a variety of EKS deployment models (including cloud, VMWare, Nutanix, and bare metal) via AWS’ Conformitron framework. The framework makes it simple for AWS partners to test their offerings across common customer environments without having to provision these environments themselves.

Automation and Operations

Organizations can leverage Pulumi’s CI/CD capabilities within their Hybrid Node environment. Pulumi Deployments or the Pulumi Kubernetes Operator can run directly on Hybrid Nodes, enabling automated infrastructure management from within your on-premises environment. This approach allows for consistent infrastructure automation while respecting network boundaries and security requirements.

Summary

The combination of EKS Hybrid Nodes and Pulumi’s infrastructure as code platform provides organizations with a powerful solution for managing hybrid Kubernetes environments. As this feature continues to evolve, Pulumi will provide comprehensive support for deploying and managing Hybrid Node configurations, making it easier for teams to adopt and maintain hybrid architectures while modernizing their application workloads.

Stay tuned for upcoming Pulumi examples for managing EKS Hybrid Nodes, which will enable you to manage your hybrid Kubernetes infrastructure using your preferred programming language, complete with type safety, testing capabilities, and modern development practices.

If you’d like to learn more about how you can harness the power of EKS with Pulumi, check out the latest version of our EKS package or join us for our AWS Immersion Day workshop following re:Invent 2024.

Fargate vs EC2

Adam Gordon Bell — Mon, 18 Nov 2024 08:42:34 -0500

Building an EKS cluster requires choosing how your containers will actually run - either on EC2 instances you manage or through AWS Fargate’s pod-by-pod approach. The differences can be pretty dramatic in practice. I’m setting up a demo cluster right now using Pulumi, so let me show you what I mean.

Here is my Fargate cluster:

import pulumi
import pulumi_awsx as awsx
import pulumi_eks as eks
import pulumi_kubernetes as k8s

# Create VPC with default CIDR
eks_vpc = awsx.ec2.Vpc("eks-vpc",
 enable_dns_hostnames=True)

# Create minimal EKS cluster
eks_cluster = eks.Cluster("eks-cluster",
 vpc_id=eks_vpc.vpc_id,
 private_subnet_ids=eks_vpc.private_subnet_ids,
 fargate=True,
)

pulumi.export("kubeconfig", eks_cluster.kubeconfig)

And when I look in k9s I see this:

You can see that each pod is on its own node. Even the core-dns pods in the system namespace are each on their own nodes. That is a lot of nodes, and it’s going to keep growing with my cluster.

If I change my cluster to be backed by EC2 nodes, I get something different.

eks_cluster = eks.Cluster("eks-cluster",
 vpc_id=eks_vpc.vpc_id,
 private_subnet_ids=eks_vpc.private_subnet_ids,
- fargate=True,
+ instance_type="t3.medium",
+ desired_capacity=2,
+ min_size=1,
+ max_size=2,
)

Now, there are only two nodes, with the pods distributed among them. Those are my two t3.mediums.

So when should you use Fargate, and when EC2 for EKS? Well, it really all relates to what we are seeing here with the number and size of nodes.

Let me explain.

Bin Packing

Normally, the Kubernetes scheduler has to solve a bin-packing problem—fitting M pods across N nodes based on available resources. This is a straight-up resource optimization problem of balancing CPU, memory, and other resource demands across the cluster.

This can be surprisingly complex in practice. Teams must balance selecting the right instance types for different workload needs while managing multiple node groups. Adding to this complexity are scheduling rules and resource quotas that need careful configuration. Perhaps most challenging is striking the right balance between high resource utilization and maintaining enough headroom for spikes in demand.

With Fargate, AWS sidesteps this challenge by providing a correctly sized bin for each pod. Each Fargate pod runs on its own dynamically provisioned, right-sized mini-environment, where the “bin” (the Fargate instance) is sized to match the pod’s requested resources. This means there’s no need for Kubernetes to optimize resource usage across a pool of shared nodes, as each pod effectively has its own “container” provided by Fargate that fits it’s needs precisely.

In other words, AWS effectively pushes this bin-packing responsibility to itself, so you don’t have to worry ( as much ) about it.

While Fargate abstracts away node management, it’s important to understand that Fargate pods still run on EC2 instances behind the scenes. And each Fargate pod gets its own ENI (Elastic Network Interface) that can sometimes limit your scaling because pods aren’t sharing a network namespace.

Pros and Cons

So yeah, Fargate abstracts away the complexity of managing nodes and scaling clusters by shifting bin-packing to AWS. With its extensive infrastructure, AWS is arguably better positioned to optimize placements at scale. They can monitor their entire cloud landscape, placing Fargate instances dynamically across available resources. This allows them to potentially leverage underutilized resources in ways that would be challenging or impractical at an individual account’s scale.

Responsibility	EC2	Fargate
Instance Types	You pick size and provision nodes	AWS provisions per pod request
Pod Isolation	You prevent noisy neighbors	AWS isolates each pod
Resource Allocation	You find space on nodes	AWS creates dedicated space

Customer vs AWS Responsibility EC2 vs Fargate

Ideally, this frees you to focus on the services rather than on the underlying infrastructure, but in practice, it’s a bit more complex.

Since each pod is treated as a standalone instance, you lose out on the cost efficiencies of co-locating multiple pods on a single node. Even though AWS is “solving” the bin-packing problem globally, to do so, they need to introduce hard barriers, and while noisy neighbors, therefore, aren’t a problem, sharing resources is now impossible. In a way, you are throwing away the run time benefit of containers, the shared kernel virtualization, for something more like VMs. And just like VMs, Fargate pods take longer to start than pods on existing EC2 nodes (typically 30-60 seconds vs near-immediate starts).

A Note on Scaling

Both EC2 and Fargate work with Kubernetes’ Horizontal Pod Autoscaling (HPA), which adds or removes pods based on metrics like CPU usage. The difference? With EC2, you need enough nodes available to handle new pods. With Fargate, AWS automatically provisions resources for each new pod - though you’ll face that 30-60 second startup delay and potential VPC limits since each pod needs its own network interface.

Feature	Fargate	EC2
Cost	Higher per pod	Lower cost per pod
Scalability	Scales on-demand, isolated resources	Requires manual node scaling
Resource Sharing	Not possible	Efficient resource sharing among pods
Startup Time	30-60 seconds	Near instant on existing nodes
Use Cases	Burst workloads, isolated pods	Cost-sensitive workloads, Resource sharing workloads

Maybe this will make more sense with an example.

Workload Example: Static Analysis

Have you ever had to run CPU-intensive batch jobs alongside lighter web services?

In a previous role, when my team and I were tasked with putting our static container image analyzer into production, we went the typical Kubernetes route, throwing it into an EC2-backed K8s cluster. The analyzer was built in Scala, running on the JVM, and was designed to take in work from a Kafka topic. Each analysis ultimately led to the results being sent to a database. We also had scheduled daily rescans of existing images whenever vulnerability lists were updated.

Then, a web app connected to this database and showed all your results. It was stateless and did far less work per request, mainly just answering an HTTP request by talking to a database.

The hard part though is that each analysis job involved uncompressing and analyzing Docker layers from S3 and could use a lot of CPU. Memory needs were around 2 GB per instance, which kept the JVM happy, but long sustained bursts in CPU usage were common, especially during intensive parts of the analysis.

In our K8s setup, the biggest pain point was handling this analysis workload’s burstiness and keeping a bunch of stateless web apps up and answering requests. We ran multiple instances of the static analyzer. Still, the analyzer was the worst type of noisy neighbor to other services in the cluster, so we’d end up over-provisioning to ensure we had enough headroom, which led to wasted resources and extra cost. The setup worked, but was not great.

Thinking back, I can see how Fargate could’ve streamlined things here. With Fargate, each static analyzer instance would run in its own isolated environment, so there’d be no resource contention or need for pre-provisioned node capacity. Each pod would be fully provisioned with the exact CPU and memory needed, and we could do some sort of on-demand scaling to deal with backlogs of analysis work in the Kafka topic.

Fargate on EKS could be a great solution for this type of chunky, resource-heavy macro-services. There are probably other ways to handle this situation, and we worked on decomposing the analysis into smaller chunks, but Fargate might have been a good solution.

But there are lots of situations where Fargate is less of a fit.

You might also like:

Example: Go Services for E-commerce

Ok, another workload that comes to mind is this one from a friend - He worked on an e-commerce backend with many lightweight Go services, each handling specific tasks—user profiles, catalog browsing, order processing, and notifications. These services were network-heavy, handling gRPC calls and database requests, but they didn’t need much CPU or memory per instance.

Fargate wouldn’t have made sense here—it would have isolated each pod, preventing resource sharing and driving up costs without real benefit. For lightweight services like these, EC2-backed Kubernetes is perfect, keeping costs low and maximizing resources by sharing nodes across small, efficient services.

So, for lightweight micro-services that can share resources, EC2 nodes with many pods each are probably the way to go.

Other Fargate Cases

Fargate can excel with long-running event-driven and bursty workloads, where you need compute resources briefly or unpredictably. Think file processing jobs that trigger on uploads, or batch processing that runs sporadically. When jobs are intermittent, Fargate’s ability to scale from zero and its isolated resources can outweigh its higher per-pod cost and startup delay. Just ensure your workload can tolerate the 30-60 second cold start.

Fargate has some hard limitations that might rule it out for your use case:

No DaemonSets
Maximum of 4 vCPU and 30GB memory per pod
Storage limitations: Only ephemeral storage up to 20GB per pod, no persistent volumes
No GPU support

Fargate vs EC2 Pricing

Ok, EC2 costs less than Fargate on a pure cost-of-compute basis. If my t3.medium ($0.0416/hr) can typically run 6-8 pods reliably, and this works out to under $0.01/hr per pod. Comparable Fargate pods (0.5 vCPU/1GB) cost about $0.025/hr each, so 2.5x.

These calculations can change based on how much headroom you need on EC2 and how many containers you can run on a given EC2 instance. Still, Fargate is just more expensive unless all your containers fit the exact computing sizes offered by Fargate. And don’t forget, AWS is likely incurring costs maintaining the warm compute capacity for launching Fargate pods, so it shouldn’t be surprising that Fargate is more expensive.

But really, pure compute cost calculations are an insufficient metric for making this decision - you need to factor in your team’s operational capacity, scaling patterns, and isolation requirements.

( Also, if Fargate makes sense because of your workload but the cost is a concern, then look into Fargate Spot instances, which can be up to 70% cheaper, but be aware that AWS can terminate them at any point to reclaim capacity. )

Misconceptions About Fargate

Fargate saves you from managing servers With containerized workloads and immutable infrastructure, your EC2 nodes are minimal, standardized, and automatically replaced. You’re not logging in, installing packages, or managing configuration - everything runs in containers. When a node needs updates, you replace it entirely.
Fargate handles all scaling challenges While it handles node scaling, you still face ENI limits, longer pod startup times (30-60s), and potential networking bottlenecks.
It’s simpler and more reliable It may be, but it may not be. The complexity shifts to networking, pod configuration, and working within Fargate’s constraints. You trade node management for new operational challenges like slower startups and possible networking complexity.

Managing Container Orchestration with Pulumi

Whether you choose EC2 or Fargate, Pulumi makes it easy to manage in your favorite programming languages. With its rich AWS support and infrastructure as code capabilities, you can define, deploy, and manage your entire Kubernetes infrastructure using familiar tools and practices. Pulumi’s AWS provider helps you implement the right solution for your needs.

With Pulumi, you get:

Complete control over EKS cluster configuration, including node types, scaling policies, and networking setup
Built-in support for both EC2 node groups and Fargate profiles within the same cluster
Infrastructure testing capabilities to validate your cluster configuration before deployment
Automation APIs for programmatic infrastructure management and custom tooling

Ready to streamline your EKS infrastructure management?

Try Pulumi for Free

Why Not Both

I hope my examples point to a pragmatic approach: use EC2 as your foundation for efficient microservices that can share resources, like our Go e-commerce setup. Add Fargate when you need isolation or flexible scaling - like for the static analysis workload. Better yet, combine both approaches in the same cluster, for cost-efficiency and isolation where needed.

If you’re interested in using Pulumi to manage EKS infra, here’s a practical EKS setup guide guide to get your started.

Now, back to getting this demo cluster up and working!

YAML, Terraform, Pulumi: What’s the Smart Choice for Deployment Automation with Kubernetes?

Engin Diri — Mon, 18 Nov 2024 08:00:00 +0100

YAML and Kubernetes go together like peanut butter and jelly. While Kubernetes objects can be defined in JSON, YAML has emerged as the de facto standard.

It’s often the first tool developers encounter when diving into Kubernetes, and for good reason - its human-readable format makes it the preferred choice in most tutorials, documentation, and even production deployments.

But is it the best way to manage your Kubernetes resources? Let’s compare YAML, Terraform, and Pulumi and see which is best for managing your Kubernetes resources.

I also gave a talk before on No YAML! and what alternatives you have if you would only stick with the Go language:

Kubernetes Components and the Kubernetes API

Before we discuss the ways to manage Kubernetes objects, let’s look at the components of a typical Kubernetes cluster and their interactions.

Kubernetes Components. Credit: kubernetes.io

A Kubernetes cluster has two component categories: control plane and node. The control plane components manage the cluster. They consist of:

kube-apiserver - The API server is the front end for the Kubernetes control plane. It exposes the Kubernetes API.
etcd - A consistent, highly available key-value store. It is Kubernetes’ backing store for all API data.
kube-scheduler - Watches for newly created Pods with no assigned node and selects a node for them to run on.
kube-controller-manager - Runs a controller to implement the control loop.
cloud-controller-manager - Runs controllers that interact with the underlying cloud providers.

The node components run on every node in the cluster. They provide the runtime environment for the containers.

kubelet - An agent that runs on each node in the cluster. It ensures that pods are running.
kube-proxy - Maintains network rules on nodes.
Container Runtime - Is responsible for running containers.

The Kubernetes API is the central component in a Kubernetes cluster. It is the interface to the cluster and the only way to work with Kubernetes objects.

A Kubernetes object is a “record of intent”. It represents the desired state of the workload running on the cluster. The whole Kubernetes system will continuously work to ensure that the object exists.

YAML Manifests

If you are long enough exposed to Kubernetes, you will have seen a lot of YAML files. You can define many Kubernetes objects in a YAML file. These include Deployments, Services, ConfigMaps, Secrets, and more.

Let’s have a look at an example of a Deployment object in YAML for an Nginx web server:

Example

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx-deployment
spec:
 replicas: 1
 selector:
 matchLabels:
 app: nginx
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: "nginx:latest"
 image: nginx

Pros and Cons of YAML

Pros

Simple: Kubernetes YAML files are easy to read and write.
The standard: YAML is the standard way to define Kubernetes objects. It is a first-class citizen in the kubectl command-line tool.

Cons

Multiple environments: If you have multiple environments, like development, staging, and production, you need to maintain multiple YAML files.
Too manual: You must create and update the YAML files by hand. Also, you must track changes across many files.
Not DRY: You cannot use the DRY principle (Don’t Repeat Yourself) in YAML files. If you have the same configuration in many files, you need to update all of them by hand.
No real abstraction: The advantage of a static configuration file is also its disadvantage. You cannot use dynamic values in the YAML files.
Need for tools from the ecosystem: You need to use tools like helm or kustomize to manage the complexity of many YAML files, environments, and templating.

To sum it up, YAML is a good choice for simple Kubernetes objects and small projects. But, if you have a complex project, and to be honest, most are, you need a better way to manage your Kubernetes resources.

Or to rephrase it: Friends don’t let friends manage Kubernetes resources with YAML.

Why should you avoid YAML? Credit: No YAML!

Terraform Kubernetes Provider

Terraform is an Infrastructure as Code tool. It lets you define and manage your infrastructure in a declarative way. It uses a high-level configuration language called HashiCorp Configuration Language (HCL). To manage Kubernetes objects with Terraform, use the Kubernetes provider. It allows you to define Kubernetes objects in Terraform HCL.

The Kubernetes provider allows you to define Kubernetes objects like kubernetes_deployment, kubernetes_service, kubernetes_config_map, kubernetes_secret, and more in code, leveraging the language features of Terraform. You can use, to a certain extent, loops, conditions, and variables in a way that is not possible with plain YAML files.

Let’s have a look at an example of a Deployment object in Terraform HCL for an Nginx web server:

Example

resource "kubernetes_deployment" "nginx" {
 metadata {
 name = "nginx-deployment"
 }

 spec {
 replicas = 1

 selector {
 match_labels = {
 app = "nginx"
 }
 }

 template {
 metadata {
 labels = {
 app = "nginx"
 }
 }

 spec {
 container {
 image = "nginx:latest"
 name = "nginx"
 }
 }
 }
 }
}

This is a very simple example of a Deployment object in Terraform HCL. Let’s add some complexity to the example by adding variables and loops:

Example with Variables and Loops

variable "namespace" {
 description = "The namespace to deploy the resources"
 type = string
 default = "default"
}

variable "nginx_image" {
 description = "The Nginx image to use"
 type = string
 default = "nginx:latest"
}

variable "replicas" {
 description = "The number of replicas to deploy"
 type = number
 default = 1
}

variable "number_of_deployments" {
 description = "The number of deployments to create"
 type = number
 default = 2
}

locals {
 nginx_image = var.nginx_image
 replicas = var.replicas
 namespace = var.namespace
 number_of_deployments = var.number_of_deployments
}

resource "kubernetes_deployment" "nginx" {
 for_each = {for i in range(local.number_of_deployments) : i => i}

 metadata {
 name = "nginx-deployment-${each.key}"
 namespace = local.namespace
 }

 spec {
 replicas = local.replicas
 selector {
 match_labels = {
 app = "nginx-${each.key}"
 }
 }

 template {
 metadata {
 labels = {
 app = "nginx-${each.key}"
 }
 }

 spec {
 container {
 image = local.nginx_image
 name = "nginx"
 }
 }
 }
 }
}

In this example, we can already see the advantage of Terraform over plain YAML files. We can use variables and loops to define multiple kubernetes_deployment objects in a more dynamic way.

Pros and Cons of Terraform

Pros

Dynamic: You can use variables, loops, and conditions in Terraform HCL. This lets you define your Kubernetes objects in a more dynamic way.
Abstraction: You can apply the DRY principle in Terraform HCL by defining and reusing modules across projects.
Environment management: By defining variables, you can manage multiple environments, like development, staging, and production. This is a more structured way to do it, without needing extra tools.

Cons

Complexity: Terraform is more complex than plain YAML files. You must learn Terraform’s HCL to define your Kubernetes objects.
DSL: Terraform is a Domain-Specific Language (DSL). You need to learn it to define your Kubernetes objects.
Testability: Testing Terraform code is harder. You need tools like Terratest and Kitchen-Terraform for it.
Helm Support: Terraform does not support Helm charts out of the box. You need to use the Helm provider to manage them with Terraform. This can add complexity to your Terraform code, and you need to keep track of the Helm provider dependency.
IDE Support: Terraform lacks the same IDE support as YAML files. You need plugins, like the Terraform Extension for Visual Studio Code, for syntax highlighting and autocompletion.

Terraform is a good choice for managing your Kubernetes resources if you have a complex project and you do not mind the complexity of the Terraform language. It gives you more flexibility and abstraction than plain YAML files, and you can manage multiple environments in a more structured way.

But is there a better way to manage your Kubernetes resources? Let’s have a look at Pulumi.

Pulumi and the Power of General Purpose Languages

Last but not least, let’s have a look at Pulumi. Pulumi is an Infrastructure as Code tool. It lets you define and manage your Kubernetes objects using general-purpose programming languages. Pulumi supports multiple languages like Python, Go, TypeScript, JavaScript, .NET, and JVM-based languages. This gives you all the benefits of a full programming language. It includes native IDE support, code completion, and debugging.

You need to install the Pulumi CLI. Then, create a new Pulumi project. Finally, add the Kubernetes library to it. It depends on the language you choose. You could use a package manager, like npm or pip.

Let’s have a look at an example of what a Deployment object would look like in all the supported languages.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

const nginxDeployment = new k8s.apps.v1.Deployment("nginx-deployment", {
 metadata: {
 name: "nginx-deployment",
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: {
 app: "nginx",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "nginx",
 },
 },
 spec: {
 containers: [
 {
 name: "nginx",
 image: "nginx:latest",
 },
 ],
 },
 },
 },
});

import pulumi
import pulumi_kubernetes as k8s

nginx_deployment = k8s.apps.v1.Deployment(
 "nginx-deployment",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="nginx-deployment",
 ),
 spec=k8s.apps.v1.DeploymentSpecArgs(
 replicas=1,
 selector=k8s.meta.v1.LabelSelectorArgs(
 match_labels={
 "app": "nginx",
 },
 ),
 template=k8s.core.v1.PodTemplateSpecArgs(
 metadata=k8s.meta.v1.ObjectMetaArgs(
 labels={
 "app": "nginx",
 },
 ),
 spec=k8s.core.v1.PodSpecArgs(
 containers=[
 k8s.core.v1.ContainerArgs(
 name="nginx",
 image="nginx:latest",
 ),
 ],
 ),
 ),
 ),
)

package main

import (
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := v1.NewDeployment(ctx, "nginxDeployment", &v1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx-deployment"),
 },
 Spec: &v1.DeploymentSpecArgs{
 Replicas: pulumi.Int(1),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx:latest"),
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using Pulumi;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Apps.V1;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Threading.Tasks;
using Deployment = Pulumi.Deployment;

class MyStack : Stack
{
 public MyStack()
 {
 var nginxDeployment = new Pulumi.Kubernetes.Apps.V1.Deployment("nginxDeployment", new DeploymentArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "nginx-deployment",
 },
 Spec = new DeploymentSpecArgs
 {
 Replicas = 1,
 Selector = new LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "nginx" }
 }
 },
 Template = new PodTemplateSpecArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels =
 {
 { "app", "nginx" }
 }
 },
 Spec = new PodSpecArgs
 {
 Containers =
 {
 new ContainerArgs
 {
 Name = "nginx",
 Image = "nginx:latest",
 }
 }
 }
 }
 }
 });
 }
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

package myproject;

import com.pulumi.Pulumi;
import com.pulumi.kubernetes.apps.v1.Deployment;
import com.pulumi.kubernetes.apps.v1.DeploymentArgs;
import com.pulumi.kubernetes.apps.v1.inputs.DeploymentSpecArgs;
import com.pulumi.kubernetes.core.v1.inputs.ContainerArgs;
import com.pulumi.kubernetes.core.v1.inputs.PodSpecArgs;
import com.pulumi.kubernetes.core.v1.inputs.PodTemplateSpecArgs;
import com.pulumi.kubernetes.meta.v1.inputs.LabelSelectorArgs;
import com.pulumi.kubernetes.meta.v1.inputs.ObjectMetaArgs;

import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 var nginxDeployment = new Deployment("nginxDeployment", DeploymentArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx-deployment")
 .build())
 .spec(DeploymentSpecArgs.builder()
 .replicas(1)
 .selector(LabelSelectorArgs.builder()
 .matchLabels(Map.of("app", "nginx"))
 .build())
 .template(PodTemplateSpecArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .labels(Map.of("app", "nginx"))
 .build())
 .spec(PodSpecArgs.builder()
 .containers(ContainerArgs.builder()
 .name("nginx")
 .image("nginx:latest")
 .build())
 .build())
 .build())
 .build())
 .build());
 });
 }

Now have a look into the more complex example with variables and loops in Pulumi. I am using TypeScript as an example, but you can of course use any of the Pulumi supported languages.

Example with Variables and Loops

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

let config = new pulumi.Config();

const namespace = config.get("namespace") || "namespace";
const nginxImage = config.get("nginxImage") || "nginx:latest";
const replicas = config.getNumber("replicas") || 1;
const numberOfDeployments = config.getNumber("numberOfDeployments") || 1;

for (let i = 0; i < numberOfDeployments; i++) {
 new k8s.apps.v1.Deployment(`nginxDeployment-${i}`, {
 metadata: {
 name: `nginx-deployment-${i}`,
 namespace: namespace,
 },
 spec: {
 replicas: replicas,
 selector: {
 matchLabels: {
 app: `nginx-${i}`,
 },
 },
 template: {
 metadata: {
 labels: {
 app: `nginx-${i}`,
 },
 },
 spec: {
 containers: [
 {
 name: "nginx",
 image: nginxImage,
 },
 ],
 },
 },
 },
 });
}

In this example, you can see how easy it is to define multiple Deployment objects with variables and loops syntax provided by TypeScript. This is a big advantage over plain YAML files and Terraform HCL. Blending the full power of a general-purpose programming language with the Kubernetes API is a natural process. Consider the possibilities of creating reusable functions, classes, and modules. You could share them across multiple projects and teams.

You might also like:

Pros and Cons of Pulumi

Pros

You can use general-purpose languages to define your Kubernetes objects. These include Python, Go, TypeScript, JavaScript, .NET, and JVM-based languages.

Full IDE support: You get full IDE support. It includes syntax highlighting, code completion, and debugging.
Testability: You can use the full power of your programming language to test your Pulumi code.
Abstraction: You can use the DRY principle in Pulumi. Define reusable functions, classes, and modules in your chosen programming language.
Environment management: You can manage multiple environments, like development, staging, and production, by defining config variables. This is a more structured way, without needing extra tools.
Kubernetes Helm Support: Pulumi supports Helm charts out of the box. You can manage Helm charts with Pulumi without needing extra tools or providers.
Programmatic execution of Pulumi: You can further blend Pulumi code into your projects using the Pulumi Automation API.

Cons

Runtime dependency: You must have the runtime of your chosen programming language installed on your machine, or use a containerised version of it in your CI/CD pipeline.
Follow best practices and compliance: Set up a code review process. It should ensure the Pulumi code follows the best practices of your chosen programming language and your organization’s additional best practices.
No possibility to convert one Pulumi language to another: If you start with one language, you can’t convert the Pulumi code to another language. You need to rewrite the Pulumi code in the new language.

YAML vs. Terraform vs. Pulumi - Results

Feature	YAML	Terraform	Pulumi
Readability	✅ Simple and human-readable	⚠️ Requires learning HCL	✅ Uses familiar programming languages
Environment Management	❌ Requires manual management of multiple files	✅ Structured management via variables	✅ Structured management via config variables
DRY Principle	❌ No reusability, requires manual updates	✅ Supports modules and reusable code	✅ Full support via functions, classes, modules
Dynamic Configuration	❌ Static configuration only	✅ Supports loops, conditions, variables	✅ Full programming language capabilities
IDE Support	✅ Good support	⚠️ Limited, requires plugins	✅ Full language-native IDE support
Testing	❌ Limited testing options	⚠️ Requires specialized tools (Terratest)	✅ Native language testing frameworks
Helm Support	✅ Native support	⚠️ Requires additional provider	✅ Built-in support
Learning Curve	✅ Low	⚠️ Medium (new DSL)	⚠️ Varies by language knowledge
Dependencies	✅ Minimal	⚠️ Terraform installation	❌ Language runtime required
Ecosystem Tools	❌ Requires additional tools (Helm, Kustomize)	✅ Integrated tooling	✅ Integrated tooling
Abstraction	❌ Limited	✅ Good	✅ Excellent

Legend:

✅ Good/Supported
⚠️ Partial/With limitations
❌ Poor/Not supported

Time to say Goodbye to YAML?

So, what’s the smart choice for deployment automation with Kubernetes? Is YAML still the best choice for Kubernetes? Or, is there a better way to manage your resources? I understand if you are a YAML fan. Despite complaints, it is still the most popular way to define Kubernetes objects. Many folks feel a sense of accomplishment when their YAML file works on the first try.

If you have a complex project, you will soon hit the limits of YAML. You want to manage your Kubernetes resources in a more structured, dynamic way while following best practices and compliance. Then, you keep adding more complexity to your YAML files. You do this by using tools like helm and kustomize to fix YAML’s flaws. Or, you adopt a hybrid approach with jsonnet, cuelang, or ytt. This makes it even more complex.

Giving up on YAML is not easy, but it does not mean that you have to give up on control. You can still have full control by using Terraform or Pulumi.

Conclusion

Now it’s time to make a decision. What’s the smart choice for deployment automation with Kubernetes? Is it YAML, Terraform, or Pulumi? For me, the smart choice is Pulumi. It gives you all the benefits of a general-purpose programming language. You gain full control, a more streamlined workflow, and the power to define all your infrastructure needs as code beyond Kubernetes. Especially on complex projects, use Pulumi. It’s better than dealing with many YAML files and the complexity of Terraform HCL.

Next time you start a new project, try Pulumi. See for yourself that it’s a smart choice. It’s an upgrade to your productivity and happiness.

But that’s my opinion. What’s your opinion? What’s your smart choice for deployment automation with Kubernetes?

If you want to try out Pulumi or play around to prove me wrong, you can start with:

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

Pulumi's Latest Kubernetes Updates from KubeCon North America

Gavin Johnson — Tue, 12 Nov 2024 08:00:00 +0000

Pulumi is excited to be at KubeCon North America this week, the premier event for all things Kubernetes and cloud-native. KubeCon is the gathering place for developers, enterprises, and cloud native experts to meet and further the education and advancement of Kubernetes and cloud native computing. At Pulumi, we are strongly committed to Kubernetes and continue to support the ecosystem with infrastructure management solutions that empower teams to automate, secure, and manage Kubernetes at scale.

Come See Us at KubeCon

Stop by the Pulumi KubeCon booth R1 this week to chat with experts on the Pulumi team. If you can’t make it to KubeCon, join our workshop on November 18, 2024, to see how easy it is to manage Kubernetes with Pulumi.

Recent Pulumi Kubernetes Updates

We’ve launched several new and improved capabilities recently to serve Kubernetes users and to make it easier than ever to automate, secure, manage, and scale clusters and workloads.

Pulumi Kubernetes Operator 2.0

The Pulumi Kubernetes Operator automates the deployment and management of infrastructure by running Pulumi programs directly in Kubernetes clusters, enabling teams to manage cloud resources alongside Kubernetes-native resources. The Pulumi Kubernetes Operator 2.0 is a significant upgrade that introduces dedicated “workspace” pods for each Stack resource, effectively isolating each stack’s compute and memory resources, improving the isolation of secrets, and opening up new customization options. The operator now scales horizontally, enhancing performance and enabling teams to manage complex Kubernetes setups with greater reliability.

Pulumi ESC Secrets in Kubernetes with the External Secrets Operator

Pulumi ESC integrates with the External Secrets Operator (ESO) to securely access secrets from Pulumi ESC in Kubernetes. This enables a secure, automated way to manage and use secrets from Pulumi ESC across Kubernetes clusters, reducing the need for manual secrets maintenance and potential security risks from the local storage of secrets in manifest files.

Pulumi EKS Provider Version 3.0.0

The Pulumi EKS Provider Version 3.0.0 simplifies Amazon Elastic Kubernetes Service (EKS) cluster management with improvements in flexibility and security and the introduction of new features to enhance the Kubernetes experience on AWS. This update allows users to more easily customize networking, scaling, and security configurations for EKS clusters, streamlining deployment and management workflows.

Kubernetes-Native Support for Customer-Managed Pulumi Deployments Agents

Kubernetes-native support for customer-managed Pulumi Deployments agents allows organizations to host Pulumi Deployments agents within their Kubernetes environments enhancing the flexibility and control they have over their infrastructure deployments. Kubernetes-native support offers greater flexibility, scalability, and security for self-hosted Pulumi Deployments agents.

Improved Kubernetes await Logic in the Pulumi Kubernetes Provider

Improved Kubernetes await logic in the Pulumi Kubernetes provider enables more reliable handling of resource dependencies and synchronization when managing Kubernetes resources. This update makes it easier to ensure that resources are created or updated in the correct order, improving the overall predictability and stability of Kubernetes deployments.

Helm Chart v4 Resource in the Pulumi Kubernetes Provider

The Helm Chart v4 resource in the Pulumi Kubernetes provider introduces several new capabilities for users managing Helm charts. It allows for more sophisticated chart management, including enhanced support for dependencies between charts, applying transformations to chart resources, and the ability to handle Kubernetes namespaces more flexibly.

Revamped crd2pulumi

crd2pulumi is a CLI tool that generates strongly-typed Pulumi resources from Kubernetes CustomResourceDefinitions (CRDs), enabling better IDE support, type safety, and autocompletion for managing complex CRDs. It was revamped to enhance its handling of CRDs with multiple versions, refine the structure of its generated code, and offer more flexible output paths for smoother integration into Pulumi programs.

Learn about the crd2pulumi and its recent updates

Pulumi Kubernetes Operator 2.0

Eron Wright — Fri, 18 Oct 2024 00:00:00 +0000

Update: “Pulumi Kubernetes Operator 2.0 is Now Generally Available!”

A few years ago we released the Pulumi Kubernetes Operator, a cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. We’ve heard your feedback about limitations related to scalability and isolation. Today, we’re excited to announce a preview release of version v2.0 of the Pulumi Kubernetes Operator. We’ve put a new, horizontally scalable architecture in place along with a variety of new security features and customization options. Let’s dig in!

What is the Pulumi Kubernetes Operator?

The Pulumi Kubernetes Operator defines a Kubernetes Custom Resource called pulumi.com/v1/Stack, which represents a Pulumi stack. The Pulumi stack can be authored in any supported Pulumi language (TypeScript, Python, Go, .NET, Java, YAML) and can deploy and manage cloud infrastructure in any supported cloud (AWS, Azure, GCP, Kubernetes and 60+ additional cloud and SaaS providers). The Pulumi Kubernetes Operator triggers cloud deployments based on changes to the Stack Custom Resource or the resources it uses.

As a result, the Pulumi Kubernetes Operator enables users to specify the desired state of their cloud infrastructure by using resources managed directly in their Kubernetes cluster. Modifying those Kubernetes resources will trigger creation, updates, and deletion of the underlying cloud infrastructure that they manage.

For example, the Operator can use the Stack resource below to run a Pulumi program, as defined in a GitHub repository, to deploy NGINX. Subsequent commits to the repository, or changes to the stack’s configuration, will cause the Operator to re-deploy the infrastructure.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 commit: 2b0889718d3e63feeb6079ccd5e4488d8601e353

The Pulumi Kubernetes Operator can also be used along with Pulumi’s Kubernetes Provider, which allows Pulumi programs to deploy resources to Kubernetes – either in the same cluster as the Operator, or in another cluster. This can be used to package up complex Kubernetes infrastructure into more abstracted units, to mix Kubernetes and cloud infrastructure into a single unit of deployment, to define how application workloads are deployed into a cluster, or for a wide variety of other use cases. The Pulumi Kubernetes Provider has full support for the Kubernetes API available in every Pulumi language, including support for Helm.

What’s New in Pulumi Kubernetes Operator 2.0?

The 2.0 release is based on a whole new architecture for running Pulumi programs in your Kubernetes cluster. The Operator now allocates a dedicated pod for each Stack to serve as the execution environment for Pulumi stack operations. Previously, all stack operations took place within the Operator’s own pod. This new approach effectively isolates each stack’s compute and memory resources, improves the isolation of secrets, and opens up new customization options.

These pods are referred to as workspace pods. Under the hood, a new Workspace Custom Resource handles the lifecycle of the workspace pod, and is created automatically by the system. A related Update Custom Resource applies a stack operation to a given workspace.

The Stack Custom Resource is still the primary API for using the Pulumi Kubernetes Operator and is largely unchanged. See the Migration section below to understand how to prepare your Stack objects for use with the 2.0 release.

Note: a key limitation of the new system is that cross-namespace references are strictly forbidden. For example, a Stack cannot refer to a Secret in another namespace.

Installation

With the 2.0 release, the supported installation modes are limited to a cluster-wide installation. You’ll need to be a cluster admin to install the Operator. The Operator is able to handle stacks across all the namespaces of your cluster.

We provide three ways to install the Operator:

a Pulumi program (see: deploy/pulumi-operator-yaml/)
a Helm chart (see: deploy/helm/pulumi-operator/)
a simple manifest file for dev environments (see: deploy/quickstart/install.yaml)

Please uninstall any earlier version of the Pulumi Kubernetes Operator, then install the new version using one of the above options. Note that this will remove any existing Stack objects in your cluster, so be sure to export their manifests or have a way to restore them.

Security

To protect your Kubernetes cluster, workspace pods are configured to have minimal permissions as defined by the ‘restricted’ profile of the Pod Security Standards. If necessary, you may configure a stack to use the ‘baseline’ profile.

A Kubernetes ServiceAccount provides an identity for your workspace pod, and we recommend that you create a service account for your stack rather than using the default account. If your Pulumi program uses the Kubernetes provider to manage resources within the cluster, the stack’s service account will need permissions, e.g. a RoleBinding to the ClusterRole named cluster-admin. See “Service Accounts” for more information.

The workspace pod has a RPC endpoint that is used by the Operator to run stack operations. The pod protects the RPC endpoint using Kubernetes RBAC, and for that reason you must bind the ClusterRole named system:auth-delegator to the stack’s service account.

Scalability

Each stack is run in its own “workspace” pod, and so the system scales horizontally to support many stacks. The workspace pod is, by default, configured to use minimal resources when idle, and to ‘burst’ into the unused memory and compute resources of the node during a stack operation. You may customize the resource constraints to fit the needs of your stack by using the workspaceTemplate field. If your Kubernetes cluster has high utilization, we recommend increasing the resource requests and limits. See “Pod Quality of Service Classes” for more information.

Once a stack reaches its desired state, the workspace pod may be retained or deleted based on the workspaceReclaimPolicy field. The default behavior is to retain the workspace for subsequent updates. This feature enables you to make a trade-off decision between performance and efficiency.

Stability

A Kubernetes cluster is a dynamic environment where a given pod may be terminated at any time, e.g. due to maintenance operations, unplanned node events, or resource pressure. This can cause problems for ongoing stack operations, and historically may cause a stack to become “locked”, necessitating manual intervention with the use of pulumi cancel.

The new system has been made more resilient by being responsive to pod termination requests. It responds by proactively sending CTRL-C to the ongoing stack operation, to gracefully wind down the operation.

If your stack has a lot of resources, consider increasing the pod’s termination grace period (default: 10 minutes).

Customization and Extensibility

Let’s discuss how to customize the stack’s environment. The Stack resource continues to provide ways to set environment variables and to set stack configuration values. What’s new is the ability to:

use a custom Docker image for your stack environment
set the stack environment’s compute and storage resources
attach volumes, init containers, and sidecars

This is accomplished using the workspaceTemplate field of the Stack resource, to customize the Workspace object that is automatically created by the system for a given stack.

By default, the system uses the official pulumi/pulumi Docker image. You may wish to create your own Docker image that is based on it, e.g. to pre-install your program’s dependencies (using pulumi install) for performance reasons. Be prepared to update the image over time to incorporate the latest version of the Pulumi CLI.

One use case for an extra init container is to customize how your program’s sources are obtained. The system supports Git repositories and Flux sources out-of-the-box, but you have the option of handling it yourself, e.g. to generate a Pulumi project or to download from another source. You simply inject an init container, mount the share volume to /share, and then place the project files into /share/workspace. Contact us to learn more.

Here’s an example of a stack with some customizations applied:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-stack
spec:
 ...
 workspaceTemplate:
 spec:
 image: example/pulumi-with-dependencies:v3.334
 podTemplate:
 metadata:
 labels:
 example.com/mylabel: bar
 spec:
 terminationGracePeriodSeconds: 3600
 tolerations:
 - key: "example.com/foo"
 operator: "Exists"
 effect: "NoSchedule"
 initContainers:
 - name: extra # add an extra init container
 image: busybox
 command: ["sh", "-c", "echo 'Hello, extra init container!'"]
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 add:
 - NET_BIND_SERVICE
 drop:
 - ALL
 volumeMounts:
 - name: share
 mountPath: /share
 containers:
 - name: pulumi # customize the main container where Pulumi stack operations are executed
 volumeMounts:
 - name: oidc-token
 mountPath: /var/run/secrets/pulumi
 volumes:
 - name: oidc-token # mount an OIDC token
 projected:
 sources:
 - serviceAccountToken:
 audience: urn:pulumi:org:ORG_NAME
 path: token
 expirationSeconds: 3600

Operability

Since each stack has its own long-running pod, you may inspect the console output for a given stack by accessing the pod logs. The pod’s name is based on the stack name; for example, given a stack named my-stack, look for a pod named my-stack-workspace-0. We recommend the stern tool to access pod logs. See “Pod and container logs” for more information.

If you need to run an interactive Pulumi command for your stack, e.g. pulumi import, exec into the workspace pod. Navigate to the /share/workspace directory and you should find your program there.

Use the spec.pulumiLogLevel field to increase the log verbosity level of the Pulumi CLI.

Walkthrough

Here’s a quick demonstration of the new system. Let’s deploy the random-yaml program from the pulumi/examples repository.

We’ll be using Flux for this demonstration; to follow along, install Flux (see “Installation”).

Setup the GitRepository Source

For this demonstration, we’ll use Flux to clone and cache the pulumi/examples repository, since it is a large repository and the Flux cache provides a big performance improvement. The system works well without Flux too.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: pulumi-examples
 namespace: default
spec:
 interval: 24h
 ref:
 branch: master
 url: https://github.com/pulumi/examples

Install a Pulumi Access Token

Create a Secret containing a Pulumi access token to be used by the stack to authenticate to Pulumi Cloud. Follow these instructions to create a personal, organization, or team access token.

Store the access token into a Kubernetes Secret. Here’s an easy way to create a Secret named pulumi-api-secret. Replace TOKEN with the token that you created.

$ kubectl create secret generic pulumi-api-secret -n default --from-literal=accessToken=TOKEN

Create a Service Account

The stack requires a Kubernetes service account with a binding to the system:auth-delegator cluster role.

If you don’t have admin permissions in your Kubernetes cluster, ask your administrator to create the service account for you. The Pulumi Kubernetes Operator installer also pre-creates a usable ServiceAccount named pulumi in the default namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
 name: random-yaml
 namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: random-yaml:system:auth-delegator
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:auth-delegator
subjects:
- kind: ServiceAccount
 name: random-yaml
 namespace: default

Create the Stack

Let’s apply a Stack object to deploy the program. Note that the specification makes reference to the GitRepository (pulumi-examples), the Secret (pulumi-api-secret), and the ServiceAccount (random-yaml) created earlier.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: random-yaml
 namespace: default
spec:
 serviceAccountName: random-yaml
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: pulumi-examples
 dir: random-yaml/
 stack: dev
 refresh: true
 continueResyncOnCommitMatch: true
 resyncFrequencySeconds: 300
 destroyOnFinalize: true
 envRefs:
 PULUMI_ACCESS_TOKEN:
 type: Secret
 secret:
 name: pulumi-api-secret
 key: accessToken

This stack is configured to deploy immediately and to resync every five minutes. The term “resync” means to execute pulumi up, which is useful for stacks that produce resources or outputs that require a periodic update. Without resync, pulumi up runs whenever a new commit is pushed to the repository.

Observe the Workspace

The system automatically creates a Workspace object named random-yaml to provision an execution environment. Let’s watch as the workspace’s status progresses towards readiness. For this demonstration, we’ll use the --watch flag to observe the status updates:

$ kubectl get workspace --watch
NAME IMAGE READY ADDRESS
random-yaml pulumi/pulumi:3.134.1-nonroot
random-yaml pulumi/pulumi:3.134.1-nonroot False
random-yaml pulumi/pulumi:3.134.1-nonroot False random-yaml-workspace.default.svc.cluster.local:50051
random-yaml pulumi/pulumi:3.134.1-nonroot False random-yaml-workspace.default.svc.cluster.local:50051
random-yaml pulumi/pulumi:3.134.1-nonroot True random-yaml-workspace.default.svc.cluster.local:50051

Observe the Updates

Once the workspace is ready, the system proceeds to run a periodic update. Each update is represented by an Update object. Let’s watch the progression across three updates:

$ kubectl get update --watch
NAME WORKSPACE PROGRESSING FAILED COMPLETE URL
random-yaml-5kmn7dcm random-yaml
random-yaml-5kmn7dcm random-yaml True False False
random-yaml-5kmn7dcm random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/1
random-yaml-rftp8gj8 random-yaml
random-yaml-rftp8gj8 random-yaml True False False
random-yaml-rftp8gj8 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/2
random-yaml-sjpw5hm8 random-yaml
random-yaml-sjpw5hm8 random-yaml True False False
random-yaml-sjpw5hm8 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/3

Observe the Stack

The Stack object maintains a status block showing the result of the latest attempt and whether an update is currently running. Taking a look, we see a successful outcome.

$ kubectl get stack random-yaml -oyaml

apiVersion: pulumi.com/v1
kind: Stack
...
status:
 conditions:
 - lastTransitionTime: "2024-10-17T20:41:49Z"
 message: the stack has been processed and is up to date
 reason: ProcessingCompleted
 status: "True"
 type: Ready
 lastUpdate:
 failures: 0
 generation: 2
 lastAttemptedCommit: sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3
 lastResyncTime: "2024-10-17T20:41:49Z"
 lastSuccessfulCommit: sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3
 name: random-yaml-sjpw5hm8
 permalink: https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/3
 state: succeeded
 type: up
 observedGeneration: 2
 outputs:
 password: '[secret]'

Delete the Stack

To complete the demonstration, let’s delete the Stack object. Because spec.destroyOnFinalize is enabled, deleting the object causes the Pulumi stack to be destroyed. That is, the stack resources are deleted.

$ kubectl delete stack random-yaml
stack.pulumi.com "random-yaml" deleted

$ kubectl get workspace
No resources found in default namespace.

$ kubectl get update
No resources found in default namespace.

Run Pulumi inside your cluster

Use the Pulumi Kubernetes Operator to deploy and manage cloud infrastructure with the Stack custom resource, driven by the Git changes your cluster already watches.

Get started

Migration

Ready to leverage these new features? Let’s quickly cover how to migrate your existing setup to Pulumi Kubernetes Operator 2.0.

The Stack custom resource continues to be the primary way to use the Pulumi Kubernetes Operator, and your existing stack definitions should continue to work after you follow a simple migration procedure:

Create a ServiceAccount and associated ClusterRoleBinding as shown in the Security section.
If your stack uses the Pulumi Kubernetes Provider, create a RoleBinding or a ClusterRoleBinding to grant the necessary permissions, e.g. to the well-known ClusterRole named cluster-admin. See “Using RBAC Authorization” for more information.
Edit your Stack manifest(s) to use the service account, by setting the spec.serviceAccountName field.
Double-check that your Stack objects do not make use of cross-namespace references.

If you’re using a private Git repository as your program source, use the Stack’s secretRefs field to reference Git credentials. SSH-based access to the repository may require a custom image with your host’s public key(s) baked in. Contact us to learn more.

If you’re using Flux integration, there’s an additional step to be performed by your cluster administrator. Please create a NetworkPolicy to allow the workspace pods to access the Flux source controller. Here’s the manifest to be applied:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-pulumi-fetch-flux-artifacts
 namespace: flux-system
spec:
 podSelector:
 matchLabels:
 app: source-controller
 ingress:
 - ports:
 - protocol: TCP
 port: http
 from:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: default
 - podSelector:
 matchLabels:
 app.kubernetes.io/managed-by: pulumi-kubernetes-operator
 app.kubernetes.io/name: pulumi
 app.kubernetes.io/component: workspace
 policyTypes:
 - Ingress

Conclusion

The Pulumi Kubernetes Operator is a great way to deploy and manage cloud infrastructure from within your Kubernetes environment. With the 2.0 release, we’ve solved the scalability and isolation limitations that affected early adopters. You can get started with the Pulumi Kubernetes Operator today. Please join us on the #kubernetes channel of the Pulumi Community Slack workspace to give feedback, and enjoy!

Pulumi ESC and External Secrets Operator: The Perfect Solution for Today's Cloud-Native Secret Management

Engin Diri — Fri, 04 Oct 2024 00:00:00 +0000

Managing secrets in a cloud-native environment can be challenging, but it is crucial for ensuring the security and integrity of any application or infrastructure. We encounter a lot of different types of secrets, from API keys, database passwords, and certificates to tokens and passwords. These secrets need to be stored securely and accessed by different services in a secure way without exposing any sensitive information to unauthorized users.

Here is where Pulumi ESC and External Secrets Operator come into play by providing a secure and efficient solution for cloud-native secret management.

Pulumi ESC (Environments, Secrets, and Configuration)

Pulumi ESC is our new secrets management & orchestration service for environments, secrets, and configurations. It can be used as part of your Pulumi IaC (Infrastructure as Code) projects using pulumiConfig to expose the stored values to your Pulumi stacks.

But it can also be used as a standalone service to manage secrets and configurations. Pulumi ESC provides support for applications through its dedicated SDKs built for different programming languages. It can also be managed from the command line with the pulumi env commands in the Pulumi CLI, which can inject secrets and configuration as a context for your CLI tools (e.g. terraform).

For more information on Pulumi ESC, you can check out the official documentation.

External Secrets Operator (ESO)

The Open Source project External Secrets Operator (ESO) is a Kubernetes operator that helps manage secrets in a highly secure and scalable way. ESO synchronizes secrets stored in external secret management systems like Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and more into Kubernetes secrets. This approach allows us to control secret sprawl and centralize the management of secrets in a single place while providing a secure and controlled way to access them.

Why Kubernetes secrets are not enough?

Kubernetes secrets are a great way to store sensitive information in a Kubernetes cluster. Kubernetes secrets are created and stored as a Kubernetes resource object and then used by pods or other resources inside the cluster.

Here is an example of a Kubernetes secret manifest file:

apiVersion: v1
kind: Secret
metadata:
 name: mysecret
type: Opaque
data:
 username: YWRtaW4= # admin
 password: YWRtaW4= # admin

or using kubectl:

kubectl create secret generic mysecret --from-literal=username=admin --from-literal=password=admin

Kubernetes secrets have several limitations. Here are the top ones:

They are stored by default in etcd in base64 encoded format, which is not secure.
They are created either manually by using kubectl commands or by using a manifest file, making them hard to manage at scale.
Hard to manage and synchronize secrets across different environments and clusters.
There is no default way to rotate secrets automatically.

ESO shines by tackling all the challenges listed and providing a way to manage secrets in a more secure and efficient manner. Benefits include:

Mitigate security risks as secrets are stored in a secure external secret management system like Pulumi ESC, which provides more robust security features like encryption, access control, and auditing.
Eliminate manual management of secrets by synchronizing them from external secret management systems into the Kubernetes cluster, reducing the need for manual intervention and minimizing the risk of human error.
ESO provides a unified interface, making it easy to manage secrets across clusters and multi-cloud environments.
ESO integrates with external secret management systems which ensure that access to secrets is tracked and an audit trail is maintained. This is crucial for security compliance and adhering to regulatory standards.
Automate secrets rotation when the external secret management system has support it.

External Secrets Operator architecture

The central part of the ESO architecture is the extension of the Kubernetes API with a new Custom Resource Definition (CRD), which defines where the secrets are stored and how they are synchronized into Kubernetes secrets. The ESO controller then retrieves the secrets from the external secret management system and creates or updates the Kubernetes secrets. Every time a secret is updated in the external secret management system, the ESO controller will synchronize the secret in the Kubernetes cluster and update the secret accordingly.

Core resources of ESO

(Cluster)SecretStore: Defines the external secret management system where the secrets are stored. It contains the necessary information to connect to the external secret management system, such as the endpoint, authentication credentials, and other configuration options. The SecretStore is namespaced while the ClusterSecretStore is cluster wide, which allows you cross-namespaced access to the secrets.
(Cluster)ExternalSecret: Defines the secret that needs to be synchronized into the Kubernetes cluster. It contains the reference to the SecretStore and the secret key and how the data should be transformed and saved as a Kubernetes secret.
PushSecret: With the PushSecret resource, you can push secrets from the Kubernetes cluster to the external secret management system. The Pulumi ESC provider for ESO supports this feature, which allows you to push secrets from the cluster into Pulumi ESC. This feature is useful when you create secrets through other means and want to synchronize them into Pulumi ESC to manage them centrally.

How to Use Pulumi ESC and ESO Together

To demonstrate how to use Pulumi ESC and ESO together, we will store some secrets and configuration values in Pulumi ESC, then synchronize them into a Kubernetes cluster using ESO.

To follow along, you’ll need the following prerequisites:

A Kubernetes cluster (I will be using a local KinD cluster, but you can use any Kubernetes cluster)
Pulumi CLI installed. You can use the pulumi env commands to manage secrets and configurations.
kubectl CLI installed for some debugging

Step 1: Install ESO and Link to Pulumi ESC

We will use Pulumi with the pulumi-kubernetes provider to deploy ESO into the Kubernetes cluster. The power of Pulumi ESC integration comes in two ways: First, to supply the Pulumi access token to ESO via Pulumi IaC, allowing the new ESO instance to access Pulumi ESC later. Second, to provide configuration and secrets from Pulumi ESC, via ESO, to the PodInfo application within the Kubernetes cluster.

pulumi new kubernetes-typescript --name eso-pulumi-esc

You should see the following output:

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

Project description (A minimal Kubernetes TypeScript Pulumi program):
Created project 'eso-pulumi-esc'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
Stack name (dev):
Created stack 'dev'

The package manager to use for installing dependencies npm
Installing dependencies...


added 310 packages, and audited 311 packages in 12s

44 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

Head to Pulumi Cloud Console and create a new Pulumi ESC project with the name eso-esc-demo:

And the environment eso-esc-dev:

In the editor add the following yaml into the Environment definition:

values:
 pulumiConfig:
 pulumi-pat:
 fn::secret: <your-pulumi-pat>

Don’t know how to create a PAT? Check out the official Pulumi documentation.

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/eso-esc-demo/eso-esc-dev

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-esc-demo/eso-esc-dev

Now, we need to link the Pulumi ESC project to the Pulumi IaC project. To do this, we need to add the following to your Pulumi.dev.yaml:

environment:
- eso-esc-demo/eso-esc-dev

Now that all the wiring is done, we can deploy the ESO instance into the Kubernetes cluster. Open the index.ts file and add the following code:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();

// Create a Kubernetes namespace
const ns = new k8s.core.v1.Namespace("external-secrets", {
 metadata: {
 name: "external-secrets",
 },
});

// Deploy a Helm release into the namespace
const externalSecrets = new k8s.helm.v3.Release("external-secrets", {
 chart: "external-secrets",
 version: "0.10.4", // Specify the version of the chart
 namespace: ns.metadata.name,
 repositoryOpts: {
 repo: "https://charts.external-secrets.io",
 },
});

// Deploy a secret into the namespace
const mySecret = new k8s.core.v1.Secret("my-secret", {
 metadata: {
 namespace: ns.metadata.name,
 name: "pulumi-access-token",
 },
 stringData: {
 "PULUMI_ACCESS_TOKEN": config.require("pulumi-pat"),
 },
 type: "Opaque",
});

// Export the namespace name
export const namespaceName = ns.metadata.name;

// Export the Helm release name
export const helmReleaseName = externalSecrets.name;

Deploy the stack by running:

pulumi up

And you should see that the secret was created in the Kubernetes cluster and the ESO instance was deployed successfully.

kubectl get secret -n external-secrets pulumi-access-token -o jsonpath='{.data.PULUMI_ACCESS_TOKEN}' | base64 -d

Step 2: Create an External Secret

Now, we will create a secret in the Pulumi ESC project and synchronize it into the Kubernetes cluster by creating an ESO reference to the external secret.

Create a new ESC environment called esp-esc-app in the eso-esc-demo ESC project.

Add the following YAML into the Environment definition:

values:
 app:
 hello: world
 hello-secret:
 fn::secret: world

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/eso-esc-demo/esp-esc-app

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-esc-demo/esp-esc-app

Either way, you should see following environment configuration in the Pulumi Cloud Console:

We can now create ClusterExternalSecret in the index.ts and reference the ESC secret.

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// cut for brevity

const crd = new k8s.apiextensions.CustomResource("cluster-secret-store", {
 apiVersion: "external-secrets.io/v1beta1",
 kind: "ClusterSecretStore",
 metadata: {
 name: "secret-store",
 },
 spec: {
 provider: {
 pulumi: {
 organization: pulumi.getOrganization(),
 project: "eso-esc-demo",
 environment: "esp-esc-app",
 accessToken: {
 secretRef: {
 name: mySecret.metadata.name,
 key: "PULUMI_ACCESS_TOKEN",
 namespace: mySecret.metadata.namespace,
 },
 },
 },
 },
 },
});

We can check that the secret was successfully synchronized by running:

kubectl get clustersecretstore secret-store -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

Step 3: Deploy an Application and Reference the Secret

Now, we can deploy an application that references the secret from the Kubernetes cluster. I am going to deploy the app called PodInfo, which is an application that exposes the pod information as an API. Add the following code to the index.ts:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// cut for brevity
const podInfo = new k8s.helm.v3.Release("podinfo", {
 chart: "podinfo",
 version: "6.7.0",
 namespace: "podinfo",
 createNamespace: true,
 repositoryOpts: {
 repo: "https://stefanprodan.github.io/podinfo",
 },
 values: {
 extraEnvs: [
 {
 name: "FROM_ESC_VIA_ESO",
 valueFrom: {
 secretKeyRef: {
 name: "esc-secret-store",
 key: "hello-secret",
 }
 }
 },
 {
 name: "FROM_ESC_VIA_ESO_2",
 valueFrom: {
 secretKeyRef: {
 name: "esc-secret-store",
 key: "hello",
 }
 }
 }
 ]
 },
});

const externalSecretPodInfo = new k8s.apiextensions.CustomResource("external-secret-podinfo", {
 apiVersion: "external-secrets.io/v1beta1",
 kind: "ExternalSecret",
 metadata: {
 name: "esc-secret-store",
 namespace: podInfo.namespace,
 },
 spec: {
 dataFrom: [
 {
 extract: {
 conversionStrategy: "Default",
 key: "app",
 }
 }
 ],
 refreshInterval: "1h",
 secretStoreRef: {
 kind: clusterSecretStore.kind,
 name: clusterSecretStore.metadata.name,
 }
 },
});

This will deploy the PodInfo application and create an ExternalSecret resource that references the secret from the Pulumi ESC project. We mount the secret into the pod as an environment variable by using the extraEnvs field in the Helm chart.

In a real-world application, you might use alternate ways to reference the secret, like mounting it as a volume or using fromEnv field in the Helm chart. It depends on your application and how you want to consume the secret.

After deploying the stack, you should see that the PodInfo application is running. To access the application, we can port-forward the service by running:

kubectl port-forward svc/podinfo 9898:9898 -n podinfo

Our PodInfo demo application provides a large selection of different endpoints to interact with. We are interested in the /env endpoint to check if the secret was successfully mounted into the pod. Let’s try that:

curl -s localhost:9898/env | grep FROM
 "FROM_ESC_VIA_ESO=world",
 "FROM_ESC_VIA_ESO_2=world",

And here we see secret values from the ESC project, proxied by ESO, and passed to the PodInfo application as environment variables. That means the secret was successfully mounted into the pod and the application is running as expected.

Step 4: Clean Up

After you are done with the demo, you can clean up the resources by running:

pulumi destroy

Conclusion

The combination of Pulumi ESC and External Secrets Operator provides a secure and efficient way to manage secrets in a cloud-native environment, improving security, efficiency, and compliance when consuming secrets in your Kubernetes cluster. By using Pulumi ESC, you can manage secrets and configurations in a centralized way, and with ESO, you now have a streamlined process to synchronize secrets into your Kubernetes cluster.

Start using Pulumi ESC and ESO today by creating an account on the Pulumi Cloud Console and begin managing your secrets in a secure and efficient way.

Create an Account

Improved Kubernetes Await Logic

Bryce Lampe — Wed, 04 Sep 2024 09:00:00 -0700

Kubernetes’ eventual consistency model is incredibly powerful but can also pose a challenge for workflows that provision complex applications and infrastructure. Sometimes you really just need to know that a step has succeeded before being able to proceed.

One of the advantages of using Pulumi to manage Kubernetes resources is that it natively and intuitively handles this problem of readiness and dependencies, giving you an easy way to express complex rollout relationships in the programming language of your choice.

The latest v4.18.0 release of the Pulumi Kubernetes provider includes a number of bug fixes and enhancements to the way Pulumi Kubernetes handles resource readiness:

More helpful errors are surfaced when a resource doesn’t become ready in time.
A new pulumi.com/deletionPropagationPolicy annotation enables faster deletions.
A new pulumi.com/waitFor annotation allows for custom readiness criteria.
And a new experimental feature enables readiness criteria for all resources.

Better error reporting

Our goal is that Pulumi Kubernetes users should never need to manually query the cluster to figure out why an update failed, so we try to surface as much information as possible when something fails.

To that end we’ve made three improvements to the information Pulumi surfaces when something goes wrong while it’s waiting for a resource to become ready (or be deleted):

Any Kubernetes Events involving the resource with type: Warning are now surfaced by Pulumi as warnings. This can help you see if a Service had trouble provisioning a LoadBalancer, for example.
Pulumi will now warn you if it looks like a resource’s deletion might be blocked by stuck finalizers. This is a common gotcha that is not always obvious.
Error messages from resources that actuate Pods (i.e. Deployments, StatefulSets, etc.) now include status codes and termination messages if their Pods failed to start. A Pod’s termination message can be helpful in CrashLoopBackOff situations and is read from /dev/termination-log by default.

(You can configure this path with terminationMessagePath, or use terminationMessagePolicy: FallbackToLogsOnError to use a Pod’s logs as its termination message.)

Faster deletions with “pulumi.com/deletionPropagationPolicy”

Pulumi uses foreground cascading deletion by default, which deletes a resource and all of its dependents. This is the most robust form of deletion but it is also the slowest, and until now there was no way to safely customize this behavior.

The new pulumi.com/deletionPropagationPolicy annotation allows you to customize delete behavior while still allowing Pulumi to await readiness when creating or updating the resource.

The annotation accepts three propagation policies, corresponding to the options supported by Kubernetes (see “Cascading Deletion”).

“background”: delete the owner object and leave dependent objects to be asynchronously garbage collected. This is faster than “foreground” deletion propagation, but dependent objects can remain temporarily or even indefinitely if they are not finalized.
“orphan”: delete the owner object and leave dependent objects untouched. This can be useful if you want to keep objects around for migration or debugging purposes.
“foreground”: the provider’s default behavior of deleting an object and all of its dependents. This is slower but guarantees all dependents have been cleaned up if it succeeds.

Using skipAwait to speed up deletion is not recommended when server-side apply is enabled because it is not respected by all resources and it can lead to race conditions during replacement. The current behavior is considered buggy, and future releases might change the behavior of the skipAwait annotation to have no effect during deletion.

The pulumi.com/deletionPropagationPolicy annotation is the recommended way to delete something quickly and safely if it has children that are slow to cleanup.

Experimental: Custom readiness with “pulumi.com/waitFor”

The pulumi.com/waitFor annotation is considered experimental and its syntax may change in the future.

There is no one-size-fits-all definition for what it means for a Kubernetes resource to be “ready,” so we have also introduced a new annotation, pulumi.com/waitFor, which allows you to specify custom readiness criteria on a per-resource basis. This is functionally similar to kubectl wait.

The new pulumi.com/waitFor annotation can be applied to any resource with metadata, and it accepts three possible formats:

A kubectl JSONPath expression – a string prefixed with “jsonpath=” followed by a path expression and an optional value.

For example this resource expects its “phase” status field to have a value of “Running”:
```
 "pulumi.com/waitFor": "jsonpath={.status.phase}=Running"
```
If a value isn’t provided, the resource is considered ready when any value exists at the given path. This resource will wait until it has a web hook configured with a CA bundle:
```
 "pulumi.com/waitFor": "jsonpath={.webhooks[*].clientConfig.caBundle}"
```
A string prefixed with “condition=” followed by the type of the condition and an optional status. This matches the behavior of kubectl wait --for=condition=... and will wait until the resource has a matching status condition. The expected status defaults to “True” if not specified. For example this will wait until the resource has a “Synced=True” condition:
```
"pulumi.com/waitFor": "condition=Synced"
```
and this will wait until it has a “Reconciling=False” condition:
```
"pulumi.com/waitFor": "condition=Reconciling=False"
```
Lastly, the pulumi.com/waitFor annotation also accepts a JSON-encoded string containing an array of one or more “jsonpath=” or “condition=” expressions. For example, this resource will be considered ready when it simultaneously has a “Bar=True” condition and a populated “foo” field.
```
"pulumi.com/waitFor": '["jsonpath={.foo}", "condition=Bar"]'
```

Custom readiness example: cert-manager

cert-manager is a popular X.509 certificate management solution with some installation nuances. If you follow the default instructions with a goal of also creating a new Certificate as part of your Pulumi stack, you might end up with something like this:

import * as kubernetes from "@pulumi/kubernetes";

const install = new kubernetes.yaml.v2.ConfigFile("install",
 {
 file: "https://github.com/cert-manager/cert-manager/releases/download/v1.15.2/cert-manager.yaml",
 },
);

const issuer = new kubernetes.apiextensions.CustomResource("issuer",
 {
 apiVersion: "cert-manager.io/v1",
 kind: "ClusterIssuer",
 metadata: {
 name: "selfsigned-issuer",
 },
 spec: {
 selfSigned: {},
 },
 },
 { dependsOn: [install] }
);

new kubernetes.apiextensions.CustomResource("certificate", {
 apiVersion: "cert-manager.io/v1",
 kind: "Certificate",
 metadata: {
 name: "selfsigned-cert",
 },
 spec: {
 isCA: true,
 commonName: "my-ca",
 secretName: "root-secret",
 privateKey: {
 algorithm: "ECDSA",
 size: 256,
 },
 issuerRef: {
 name: issuer.metadata.name,
 kind: issuer.kind,
 group: "cert-manager.io",
 },
 },
});

This uses a v2 ConfigFile to kubectl apply the cert-manager custom resource definitions (CRDs), and then it creates a self-signed Issuer and Certificate using those CRDs.

The dependencies between resources appear reasonable, however this program will likely fail when it is first run:

error: resource “kubernetes:cert-manager.io/v1:ClusterIssuer::issuer” was not successfully created by the Kubernetes API server: Internal error occurred: failed calling webhook “webhook.cert-manager.io”: failed to call webhook: Post “https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s”: tls: failed to verify certificate: x509: certificate signed by unknown authority

We see this “certificate signed by unknown authority” error because a few things need to happen before the cert-manager webhooks are operational:

At least one cainjector pod needs to be running.
The cainjector pod(s) need to have a leader elected.
The cainjector leader needs to inject certificate authority data into the cert-manager webhooks.

Previously, Pulumi had no way to know whether those pre-requisite steps had taken place. However, now we can use the pulumi.com/waitFor annotation to instruct Pulumi to wait until web hooks have CA data injected before proceeding to create Certificates. One way to achieve this is by using the new transforms API to annotate the webhooks appropriately:

const install = new kubernetes.yaml.v2.ConfigFile("install",
 {
 file: "https://github.com/cert-manager/cert-manager/releases/download/v1.15.2/cert-manager.yaml",
 },
 {
 // Add a "waitFor" annotation which waits for the cainjector pod to come
 // online and modify our webhooks with valid CA bundles. Webhooks
 // needed by our Certificate will fail until this happens.
 transforms: [
 (args) => {
 if (
 args.type ===
 "kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration" ||
 args.type ===
 "kubernetes:admissionregistration.k8s.io/v1:MutatingWebhookConfiguration"
 ) {
 args.props.metadata.annotations["pulumi.com/waitFor"] =
 "jsonpath={.webhooks[*].clientConfig.caBundle}";
 return {
 props: args.props,
 opts: args.opts,
 };
 }
 return undefined;
 },
 ],
 },
);

Here we’ve given the pulumi.com/waitFor annotation a value of jsonpath={.webhooks[*].clientConfig.caBundle}, which is a JSONPath expression that will wait until a webhook in the webhooks array has a clientConfig with a caBundle populated – indicating that the webhook is fully operational.

Now the program will deploy successfully with a Missing {.webhooks[*].clientConfig.caBundle} message while Pulumi waits for the field to be configured.

Experimental: Generic readiness

Being able to specify custom readiness is powerful, but a lot of resources follow well-known conventions like the Ready=True condition. Rather than requiring you to specify "pulumi.com/waitFor": "condition=Ready" on every resource that needs it, there’s now an opt-in feature to automatically apply well-known readiness criteria to all resources managed by Pulumi.

Running your program with the environment variable PULUMI_K8S_AWAIT_ALL=true will cause Pulumi to automatically wait for all resources to become ready when creating or updating them – including custom resources.

If a resource appears to follow well-known conventions (like the “Ready” condition, among others) Pulumi will wait for it to become ready. If the resource doesn’t appear to follow well-known conventions, Pulumi will maintain its existing behavior and assume the resource is ready immediately.

(Generic readiness criteria are determined by cli-utils (a collection of Go libraries maintained by the SIG CLI), and behavior is unchanged for any resources that Pulumi already waits for readiness.)

Please don’t hesitate to reach out on GitHub or Slack if you’re excited about this feature; if you experience any issues with it; or if you have any feedback to share!

Manage Kubernetes readiness with Pulumi

Express complex rollout and readiness relationships in the language you already use, and let Pulumi wait for your cluster resources to become ready before it continues.

Get started

Additional improvements

Several other bug fixes related to the Pulumi Kubernetes provider’s await logic are also worth mentioning.

PersistentVolumeClaims with a bind mode of WaitForFirstConsumer are now assumed to be ready as soon as they are created. Previously, these PVCs could cause deadlocks by preventing their consumer from being provisioned.

Services are now considered ready as long as they have no non-ready Endpoints. Previously, Services would block forever if they matched Deployments that had been scaled down to zero replicas, or if they had manually provisioned endpoints.

Pulumi will now wait for DaemonSets to become ready instead of always using skipAwait semantics.

Conclusion

We hope you will enjoy the increased flexibility of the pulumi.com/deletionPropagationPolicy annotation; the extensibility of the pulumi.com/waitFor annotation; and the improved visibility into errors and events Pulumi Kubernetes now provides while managing your cluster resources.

Check out the following links to learn more about Pulumi Kubernetes today!

Pulumi in a Cloud Native World

Engin Diri — Fri, 02 Aug 2024 00:00:00 +0000

In today’s complex digital landscape, organizations are increasingly turning to platform engineering to optimize their software delivery processes and maximize efficiency. The growing complexity of modern applications, coupled with the need for rapid, secure, and scalable deployments, has created a pressing demand for robust Internal Developer Platforms (IDPs).

IDPs are a key component of modern platform engineering strategies. An IDP is a self-service layer that sits on top of an organization’s infrastructure, abstracting away complexity and providing developers with the tools and environments they need to build, test, and deploy applications efficiently.

Key features of an IDP:

Developer Control Plane. Curated experiences that empower developers by meeting them at their level of expertise, whether it’s an abstracted developer portal, custom CLI, or shared IaC templates.
Integration & Delivery. Automations to version control, test, trace, and deploy all infrastructure from resources, configurations, environments, and secrets as well as orchestration automations to manage provisioning workflows.
Monitoring & Logging. Components to log, monitor, and observe all infrastructure for greater operational control as well as optimize against unnecessary costs.
Security & Identity. Security and compliance guardrails that regulate every piece of infrastructure from policies to fine-grained access controls to secrets.
Resources. Providers that support modern cloud architectures such as Kubernetes, containers, serverless, generative AI, machine learning, data lakes, hybrid cloud/on-premises, and more.

By implementing an IDP, organizations can significantly reduce cognitive load on developers, accelerate time-to-market, improve security and compliance, and foster a culture of innovation. IDPs act as a force multiplier, allowing development teams to focus on creating business value while the platform handles the underlying complexity of cloud-native infrastructure.

The Cloud Native Computing Foundation (CNCF) landscape offers a large ecosystem of tools and technologies that can be leveraged to build these platforms from scratch. However, the sheer number of options can be overwhelming, making it difficult for organizations to determine the best combination of tools for their specific needs.

At Pulumi’s Customer Experience Team, we’ve observed that adopting Infrastructure as Code often serves as a catalyst for organizations to reassess their entire software delivery process. This reassessment frequently leads to the implementation of more streamlined, automated, and secure methodologies – the core principles of platform engineering mentioned above.

To provide guidance for our customers, we have created a reference architecture called the KEBAP stack, this stack offers a structured approach to integrate these technologies into an effective platform engineering solution using Pulumi.

The KEBAP Stack Reference Architecture

The KEBAP stack consists of the following (most commonly used) CNCF projects:

Kubernetes (with Kyverno as a bonus)
External Secrets Operator
Backstage
Argo CD
Pulumi

Have a look on the high-level architectural diagram of the KEBAP stack to get a first idea of what we are going to talk about in the following sections:

Breaking Down the KEBAP Stack

Let’s take a closer look at each component of the KEBAP stack and understand what their role. The KEBAP is layered in such a way that each component builds on the capabilities of the previous one. The five layers of the KEBAP stack are designed to address different capabilities of our internal developer platform:

Kubernetes

In modern cloud computing, the hub-and-spoke architecture pattern is widely used to manage multiple Kubernetes clusters. In this pattern, a central control plane (hub) manages a set of clusters (spokes) that run applications and services. The hub provides a unified view of the entire infrastructure, enabling centralized management, monitoring, and security.

The control plane cluster in the KEBAP stack is the hub cluster and contains the control plane logic for the entire IDP.

Pulumi

Pulumi manages the lifecycle of all cloud infrastructure, providing developers with self-service capabilities. It allows you to use familiar programming languages to define your infrastructure as code, offering more flexibility and power than traditional YAML or domain-specific languages.

The Pulumi Kubernetes Operator enables you to manage your infrastructure directly from within Kubernetes, seamlessly integrating with your existing Kubernetes workflows.

Since we use GitOps as our deployment method, we define the infrastructure in a Git repository and then let Argo CD deploy it to the hub cluster. As soon as the infrastructure definition is deployed, the Pulumi Kubernetes Operator will take over and create the defined resources in the selected cloud provider.

Here is an example Pulumi definition of a new Kubernetes cluster using DigitalOcean:

apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: ${{values.name}}
program:
 variables:
 clusterName: "do-cluster"
 clusterRegion: "nyc1"
 nodePoolName: "default"
 nodeCount: 1
 version: 1.30.1-do.0

 resources:
 do_cluster:
 type: digitalocean:KubernetesCluster
 properties:
 name: "${clusterName}"
 region: "${clusterRegion}"
 version: "${version}"
 destroyAllAssociatedResources: true
 nodePool:
 name: "${nodePoolName}"
 size: "s-2vcpu-2gb"
 nodeCount: "${nodeCount}"
 # cut for brevity
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: ${{values.name}}
spec:
 stack: ${{values.stack}}
 envRefs:
 DIGITALOCEAN_TOKEN:
 type: Env
 env:
 name: DIGITALOCEAN_TOKEN
 programRef:
 name: ${{values.name}}
 destroyOnFinalize: false

Argo CD

Argo CD handles the deployment layer, ensuring that the current deployment matches what is declaratively defined in the source. It’s emerging as the de facto standard due to its rich front-end and powerful features. Argo CD implements the GitOps model, continuously monitoring your Git repositories and automatically updating the deployed applications to match the desired state.

In the KEBAP stack, we use Argo CD to deploy the Pulumi infrastructure defined in the Git repository to our hub cluster.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: ${{values.name}}-${{values.stage}}-${{ (values.owner | parseEntityRef).name }}
 namespace: argocd
spec:
 destination:
 namespace: pulumi-operator
 server: https://kubernetes.default.svc
 project: ${{ (values.owner | parseEntityRef).name }}
 source:
 path: "gitops/teams/clusters/${{values.name}}-${{values.stage}}-cluster"
 directory:
 recurse: false
 exclude: '{catalog-info.yaml}'
 targetRevision: main
 repoURL: https://github.com/my-backstage-demo/backstage-infrastructure-provisioning-templates-workshop.git
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - ServerSideApply=true
 - CreateNamespace=true

External Secrets Operator (ESO)

The External Secrets Operator (ESO) addresses the challenge of securely managing secrets in Kubernetes. It integrates with various secret stores (such as Pulumi ESC, AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault). This allows you to configure access to the secret store once for all your required secrets, enhancing security and simplifying management.

We even take it a step further by using ESO to manage the secrets from Pulumi ESC (Environments, Secrets, and Configurations) with the Pulumi Provider for ESO. This way, we use the Pulumi not only for the infrastructure but also for the secrets management reducing the number of tools we need to manage.

To give the Pulumi Kubernetes Operator access to the secrets, we create following ClusterSecretStore object:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
 name: secret-store
spec:
 provider:
 pulumi:
 organization: <org>
 environment: <env>
 accessToken:
 secretRef:
 name: pulumi-access-token
 key: PULUMI_ACCESS_TOKEN
 namespace: argocd

And then we can define the ExternalSecret with all the secrets that we want to manage and pass to our Operator as environment variables extraEnv:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: pulumi-operator-secrets
spec:
 refreshInterval: 1h
 secretStoreRef:
 kind: ClusterSecretStore
 name: secret-store
 data:
 - secretKey: pulumi-access-token
 remoteRef:
 key: "backstage.pulumi-pat"
 - secretKey: do-token
 remoteRef:
 key: "backstage.do"

Backstage

Backstage is an open platform for building developer portals. It provides a centralized place for managing software catalogs, documentation, and tooling. This layer helps in organizing microservices and infrastructure, streamlining the process for developers to create, manage, and explore services.

In our KEBAB stack, we installed the Pulumi plugin for Backstage, which allows us to manage our infrastructure as code directly from the Backstage UI by cookie-cutting the Pulumi project and making the result available in our source control repository. From there, Argo CD takes over and deploys the definition to the hub cluster as described above.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
 name: kubernetes-template
 title: Kubernetes Cluster
 description: |
 A template for creating a new Kubernetes Cluster.
 tags:
 - pulumi
 - kubernetes
spec:
 steps:
 - id: pulumi-new-component
 name: Cookie cut the component Pulumi project
 action: pulumi:new
 input:
 name: "${{ parameters.component_id }}-infrastructure"
 description: ${{ parameters.description | dump }}
 organization: ediri
 stack: ${{ parameters.stack }}
 template: "https://github.com/my-silly-organisation/microservice-civo/tree/main/infrastructure-${{ parameters.cloud }}-${{ parameters.language }}"
 config:
 "node:node_count": "${{ parameters.nodeCount }}"
 folder: .

Bonus: Kyverno

Kyverno can be used to enforce policies and automate security and operational best practices in Kubernetes clusters. Kyverno is a policy engine designed for Kubernetes that allows you to define policies as code and enforce them at runtime. This ensures that your Kubernetes clusters are secure and compliant with your organization’s policies.

In the KEBEP stack, we use Kyverno to create a ClusterPolicy object to automatically add vCluster clusters to the Argo CD as spoke clusters. This will generate a Secret object in the argocd namespace for each vCluster cluster the dev teams create.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: sync-secret-vcluster
 annotations:
 argocd.argoproj.io/sync-wave: "5"
spec:
 generateExistingOnPolicyUpdate: true
 rules:
 - name: sync-secret-vcluster-nonprod
 match:
 any:
 - resources:
 names:
 - "*-dev-*"
 - "*-staging-*"
 - "*-prod-*"
 kinds:
 - Secret
 exclude:
 any:
 - resources:
 namespaces:
 - kube-system
 - default
 - kube-public
 - kyverno
 context:
 - name: namespace
 variable:
 value: "{{ request.object.metadata.namespace }}"
 - name: name
 variable:
 value: "{{ request.object.metadata.name }}"
 - name: ca
 variable:
 value: "{{ request.object.data.\"certificate-authority\" }}"
 - name: cert
 variable:
 value: "{{ request.object.data.\"client-certificate\" }}"
 - name: key
 variable:
 value: "{{ request.object.data.\"client-key\" }}"
 - name: vclusterName
 variable:
 value: "{{ replace_all(name, 'vc-', '') }}"
 jmesPath: 'to_string(@)'
 - name: stage
 variable:
 value: "{{ split(vclusterName, '-')[2] }}"
 jmesPath: 'to_string(@)'
 - name: team
 variable:
 value: "{{ split(vclusterName, '-')[3] }}"
 jmesPath: 'to_string(@)'
 generate:
 kind: Secret
 apiVersion: v1
 name: "{{ vclusterName }}"
 namespace: argocd
 synchronize: true
 data:
 kind: Secret
 metadata:
 labels:
 argocd.argoproj.io/secret-type: cluster
 clusterType: "vcluster"
 env: "{{ stage }}"
 team: "{{ team }}"
 stringData:
 name: "{{ vclusterName }}"
 server: "https://{{ vclusterName }}.{{ namespace }}:443"
 config: |
 {
 "tlsClientConfig": {
 "insecure": false,
 "caData": "{{ ca }}",
 "certData": "{{ cert }}",
 "keyData": "{{ key }}"
 }
 }

Putting all the Pieces Together

As we have seen, each component of the KEBAP stack plays a crucial role in the overall architecture. Let’s how the different workflows look like when we put all the pieces together. We go through the following scenarios:

Ignite the Control Plane Cluster With Pulumi
Ordering a New Development Kubernetes Cluster
Ordering a New Production Kubernetes Cluster
Ordering Other Infrastructure

Ignite the Control Plane Cluster With Pulumi

We use Pulumi code to create the control plane cluster in our cloud provider of choice. After the creation of the control plane, we install the Argo CD in a very bare-bones way:

const argocd = new ArgoCD("argocd", {
 initialObjects: initialObjects
}, {
 providers: {
 kubernetes: k8sProvider,
 },
});

The idea is to let then Argo CD fetch the rest of the configuration from the Git repository and apply it to the cluster. And here comes the power of GitOps: It will also manage the Argo CD itself. This way, we separated the workload running on the cluster from the creation of the cluster itself.

Ordering a New Development Kubernetes Cluster

Starting point is Backstage, where we can order a new development Kubernetes cluster. As we want to keep the costs low for development cluster, we opted here for the vCluster project. The vCluster project allows us to create a Kubernetes cluster on demand, which runs as containers on the control plane cluster. The moment the cluster is created, we add it to the Argo CD as spoke cluster. This way, we can roll out applications to the development cluster in the same way as we would do for production clusters.

See the definition of the ClusterPolicy object in the Kyverno section.

Ordering a New Production Kubernetes Cluster

The Backstage catalog is again the starting point to order a new production Kubernetes cluster. This time, we use the Pulumi code to create the cluster in the cloud provider of choice. After the creation of the cluster, we add it to the Argo CD as a production cluster. This way, we can roll out applications to the production cluster in the same way as we would do for development clusters.

Ordering Other Infrastructure

Any other infrastructure components such as databases, message queues, and storage will be also ordered via Backstage. And similar to the ordering of a new production Kubernetes cluster, we use Pulumi to create the cloud infrastructure.

Conclusion

A lot of organizations are building their internal developer platforms using CNCF projects to fulfill the needs of their internal customers. The KEBAP stack provides a structured approach to integrate these technologies into an effective platform engineering solution. We also saw that Pulumi plays a crucial role in this stack by taking care of the management of the cloud infrastructure.

This KEBAP stack offers several benefits, including:

A secure, automated, and auditable environment
Reproducible and programmable infrastructure-as-code
Unified management of cloud resources, Kubernetes, and application deployments
Faster onboarding for new team members
Easier compliance with regulatory requirements due to increased auditability

But there is also a downside: The KEBAP stack requires a significant investment in time and resources to set up and maintain as you are in full charge of the integration of the components as well as operating them. All the projects under the umbrella of CNCF are not automatically implying that they are compatible with each other out of the box. Any platform engineering team will need to invest continuously time and resources to keep the stack up-to-date and running. This is a not negligible investment to consider for any organization.

For this reason, we have another reference architecture that uses components from all of the Pulumi Platform for an end to end fully-managed solution. This solution takes care of the integration of the different components of an internal developer portal and provides a carefree experience of managing your infrastructure at scale.

Continue to upskill and learn about platform engineering:

Read Pulumi Patterns and Practices (P3): A Pulumi-based reference architecture for large-scale organizations
Learn how to build Developer Portals with Pulumi.
Register for one of our Platform Engineering or DevOps workshops.

New: Helm Chart v4 resource with new features and languages

Eron Wright — Thu, 13 Jun 2024 00:00:00 -0700

Today we’re happy to announce a new “v4” version of the Chart resource, available now in v4.13 of the Pulumi Kubernetes provider. The new kubernetes.helm.sh/v4.Chart resource is provided side-by-side with the existing kubernetes.helm.sh/v3.Chart resource. We expect to deprecate v3 in the future.

When you need to install a third-party application into your Kubernetes cluster, you’re likely to find a Helm chart for that in Artifact Hub or other registry. Pulumi provides two ways to apply a Helm chart, as outlined in Choosing the Right Helm Resource For Your Use Case. The Chart resource offers deeper integration with Pulumi and better drift remediation. v4 brings a host of new features, including enhanced SDK support across all Pulumi SDKs, full OCI registry support, improved handling of chart values, better connectivity for cluster interactions, and improved resource ordering. Let’s dig in.

What’s new

Let’s look at what’s new with Chart v4.

New language support - Java SDK & YAML SDK

The Chart v4 resource is a Pulumi Component that works consistently across all Pulumi SDKs. Earlier versions were implemented separately for each SDK, leading to various inconsistencies and limited SDK support.

Here, for example, is a simple deployment of cert-manager, a well-known Kubernetes add-on:

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";

const certman = new kubernetes.helm.v4.Chart("cert-manager", {
 namespace: "cert-manager",
 chart: "oci://registry-1.docker.io/bitnamicharts/cert-manager",
 version: "1.3.1",
});

import pulumi
import pulumi_kubernetes as kubernetes

certman = kubernetes.helm.v4.Chart("cert-manager",
 namespace="cert-manager",
 chart="oci://registry-1.docker.io/bitnamicharts/cert-manager",
 version="1.3.1")

package main

import (
 helmv4 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v4"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := helmv4.NewChart(ctx, "cert-manager", &helmv4.ChartArgs{
 Namespace: pulumi.String("cert-manager"),
 Chart: pulumi.String("oci://registry-1.docker.io/bitnamicharts/cert-manager"),
 Version: pulumi.String("1.3.1"),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 var certman = new Kubernetes.Helm.V4.Chart("cert-manager", new()
 {
 Namespace = "cert-manager",
 Chart = "oci://registry-1.docker.io/bitnamicharts/cert-manager",
 Version = "1.3.1",
 });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.kubernetes.helm.sh_v4.Chart;
import com.pulumi.kubernetes.helm.sh_v4.ChartArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var certman = new Chart("cert-manager", ChartArgs.builder()
 .namespace("cert-manager")
 .chart("oci://registry-1.docker.io/bitnamicharts/cert-manager")
 .version("1.3.1")
 .build());

 }
}

resources:
 cert-manager:
 type: kubernetes:helm.sh/v4:Chart
 properties:
 namespace: cert-manager
 chart: oci://registry-1.docker.io/bitnamicharts/cert-manager
 version: "1.3.1"

OCI registry support

You can use container registries with OCI support such as Docker Hub to store and share Helm chart packages. The Chart v4 resource now has full support for OCI, bringing it to parity with the Release resource.

To use an authenticated OCI registry, you must first login using helm registry login or docker login.

Chart v4 also supports the use of Helm chart repositories, and adopts the same repositoryOpts API as was introduced in the Release resource.

Lock file support

Helm has support for lock files (Chart.lock) to control a chart’s dependencies. When deploying a chart from a local directory, Pulumi automatically rebuilds the chart’s dependencies if a lock file is present. See Helm Dependency Build for details.

Better handling of chart values

Chart v4 offers new ways to work with Chart values. It is now possible to use multiple values files and to use Pulumi Assets. Of course you can also use output values from other resources as chart values.

It is also possible to set a chart value to the contents of a text file, similarly to using Helm’s --set-file argument. To do that, simply use a Pulumi Asset as a value within the values map. This is useful for injecting large values into a chart, such as additional templates as supported by the extraDeploy parameter of some Bitnami charts.

Here’s an example of using a combination of values from different sources.

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";

const certManager = new kubernetes.helm.v4.Chart("cert-manager", {
 namespace: "cert-manager",
 chart: "oci://registry-1.docker.io/bitnamicharts/cert-manager",
 version: "1.3.1",
 valueYamlFiles: [new pulumi.asset.FileAsset("values.yaml")],
 values: {
 commonLabels: {
 "pulumi.com/stackName": pulumi.getStack(),
 },
 extraDeploy: [new pulumi.asset.FileAsset("manifest.yaml")],
 },
});

import pulumi
import pulumi_kubernetes as kubernetes

cert_manager = kubernetes.helm.v4.Chart("cert-manager",
 namespace="cert-manager",
 chart="oci://registry-1.docker.io/bitnamicharts/cert-manager",
 version="1.3.1",
 value_yaml_files=[pulumi.FileAsset("values.yaml")],
 values={
 "commonLabels": {
 "pulumi.com/stackName": pulumi.get_stack(),
 },
 "extraDeploy": [pulumi.FileAsset("manifest.yaml")],
 })

package main

import (
 helmv4 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v4"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := helmv4.NewChart(ctx, "cert-manager", &helmv4.ChartArgs{
 Namespace: pulumi.String("cert-manager"),
 Chart: pulumi.String("oci://registry-1.docker.io/bitnamicharts/cert-manager"),
 Version: pulumi.String("1.3.1"),
 ValueYamlFiles: pulumi.AssetOrArchiveArray{
 pulumi.NewFileAsset("values.yaml"),
 },
 Values: pulumi.Map{
 "commonLabels": pulumi.Any(map[string]interface{}{
 "pulumi.com/stackName": ctx.Stack(),
 }),
 "extraDeploy": pulumi.Array{
 pulumi.NewFileAsset("manifest.yaml"),
 },
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 var certManager = new Kubernetes.Helm.V4.Chart("cert-manager", new()
 {
 Namespace = "cert-manager",
 Chart = "oci://registry-1.docker.io/bitnamicharts/cert-manager",
 Version = "1.3.1",
 ValueYamlFiles = new[]
 {
 new FileAsset("values.yaml"),
 },
 Values =
 {
 { "commonLabels", new Dictionary<string, object?>
 {
 ["pulumi.com/stackName"] = Deployment.Instance.StackName,
 } },
 { "extraDeploy", new[]
 {
 new FileAsset("manifest.yaml"),
 } },
 },
 });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.deployment.Deployment;
import com.pulumi.kubernetes.helm.v4.Chart;
import com.pulumi.kubernetes.helm.v4.ChartArgs;
import com.pulumi.asset.FileAsset;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var certManager = new Chart("cert-manager", ChartArgs.builder()
 .namespace("cert-manager")
 .chart("oci://registry-1.docker.io/bitnamicharts/cert-manager")
 .version("1.3.1")
 .valueYamlFiles(new FileAsset("values.yaml"))
 .values(Map.ofEntries(
 Map.entry("commonLabels", Map.of("pulumi.com/stackName", Deployment.getInstance().getStackName())),
 Map.entry("extraDeploy", List.of(new FileAsset("manifest.yaml")))
 ))
 .build());

 }
}

resources:
 cert-manager:
 type: kubernetes:helm.sh/v4:Chart
 properties:
 namespace: cert-manager
 chart: oci://registry-1.docker.io/bitnamicharts/cert-manager
 version: "1.3.1"
 valueYamlFiles:
 - fn::fileAsset: values.yaml
 values:
 commonLabels:
 pulumi.com/stackName: ${pulumi.stack}
 extraDeploy:
 - fn::fileAsset: manifest.yaml

Better connectivity

You may now use charts that use chart functions requiring a connection to the cluster, e.g. to:

Check the Kubernetes server version with .Capabilities.KubeVersion
Check if an API version or kind is available with .Capabilities.APIVersions.Has
Use the lookup function in your templates to get existing objects in your live cluster.

Note that the lookup function is executed in both the preview and the non-preview mode, and keep in mind that the expected object may not exist during a preview.

Improved resource ordering

It’s now easy to wait for a chart’s resources to be installed before installing other resources, simply by using the dependsOn option. In earlier versions, we relied on a ready output property.

The Chart resource automatically detects dependencies between resources in the manifest(s). For example, it knows to install namespaces and Custom Resource Definitions (CRDs) first.

Use the config.kubernetes.io/depends-on annotation to declare an explicit resource dependency. The annotation accepts a list of resource references, delimited by commas.

It consists of the group, kind, name, and optionally the namespace, delimited by forward slashes.

Resource Scope	Format
namespace-scoped	`<group>/namespaces/<namespace>/<kind>/<name>`
cluster-scoped	`<group>/<kind>/<name>`

For resources in the “core” group, the empty string is used instead (for example: /namespaces/test/Pod/pod-a).

New-style Pulumi transformations

Pulumi has a new way to transform component resources and their children, the transforms options. The older transformations option doesn’t work with multi-language components like Chart v4. See Resource Option: transforms for more details.

Note: you cannot change an object’s namespace or name using a Pulumi transformation, and you cannot add or discard an object.

Here’s an example of using the transforms option to add the pulumi.com/patchForce annotation to a chart’s resources.

const applyPatchForceAnnotation = async (args: pulumi.ResourceTransformArgs) => {
 switch(args.type) {
 case "kubernetes:helm.sh/v4:Chart":
 break;
 default:
 args.props.metadata.annotations = {
 "pulumi.com/patchForce": "true",
 ...args.props.metadata.annotations
 }
 }
 return {
 props: args.props,
 opts: args.opts,
 };
};

const ingressController = new kubernetes.helm.v4.Chart("ingresscontroller", {
 chart: "nginx-ingress",
 namespace: ingressNs.metadata.name,
 repositoryOpts: {
 repo: "https://helm.nginx.com/stable",
 },
 version: "0.14.1",
}, {transforms: [applyPatchForceAnnotation]});

def apply_patchforce_annotation(args: ResourceTransformArgs):
 if not args.type_ == "kubernetes:helm.sh/v4:Chart":
 if not 'metadata' in args.props:
 args.props['metadata'] = {}
 if not 'annotations' in args.props['metadata']:
 args.props['metadata']['annotations'] = {}
 args.props['metadata']['annotations']['pulumi.com/patchForce'] = 'true'

 return ResourceTransformResult(
 props=args.props,
 opts=args.opts)

ingresscontroller = kubernetes.helm.v4.Chart(
 "ingresscontroller",
 chart="nginx-ingress",
 namespace=ingress_ns.metadata.name,
 repository_opts=kubernetes.helm.v3.RepositoryOptsArgs(
 repo="https://helm.nginx.com/stable",
 ),
 version="0.14.1",
 opts=pulumi.ResourceOptions(transforms=[apply_patchforce_annotation])
)

applyPatchForceAnnotation := func(ctx context.Context, rta *pulumi.ResourceTransformArgs) *pulumi.ResourceTransformResult {
 transform := func(applier interface{}) {
 o := rta.Props.ToMapOutputWithContext(ctx).ApplyT(applier)
 r, err := internals.UnsafeAwaitOutput(ctx, o)
 if err != nil {
 panic(err)
 }
 rta.Props = r.Value.(pulumi.Map)
 }

 switch rta.Type {
 case "kubernetes:helm.sh/v4:Chart":
 // Do nothing for Helm charts
 default:
 transform(func(obj map[string]any) pulumi.Map {
 // note: obj is an ordinary Unstructured object at this point.
 unstructured.SetNestedField(obj, "true", "metadata", "annotations", "pulumi.com/patchForce")
 return pulumi.ToMap(obj)
 })
 }
 return &pulumi.ResourceTransformResult{
 Props: rta.Props,
 Opts: rta.Opts,
 }
}

// Use Helm to install the Nginx ingress controller
_, err = helmv4.NewChart(ctx, "ingresscontroller", &helmv4.ChartArgs{
 Chart: pulumi.String("nginx-ingress"),
 Namespace: ingressNs.Metadata.Name(),
 RepositoryOpts: &helmv4.RepositoryOptsArgs{
 Repo: pulumi.String("https://helm.nginx.com/stable"),
 },
 Version: pulumi.String("0.14.1"),
}, pulumi.Transforms([]pulumi.ResourceTransform{applyPatchForceAnnotation}))
if err != nil {
 return err
}

Post-rendering support

New to v4 is support for a post-rendering command, with optional arguments, to be applied to the rendered manifest. See Advanced Helm Techniques: Post Rendering for details.

“Keep” policy

The Chart v4 resource now understands Helm resource policies, specifically “keep” which instructs Pulumi not to delete a given object when the resource is destroyed. Simply apply the helm.sh/resource-policy: keep annotation to the object. See Tell Helm Not To Uninstall a Resource for details.

Release name

Charts typically generate object names based on the Helm release name. By default, Chart v4 uses its own resource name as the release name. To customize the release name, set the name property on the Chart.

Some charts also accept a fullnameOverride value to control object naming, use that if possible.

Limitations

Not supported: Kubernetes transformations

Chart v4 does not support the transformations argument as seen in Chart v3, that facilitates a Kubernetes-centric transformation and/or discarding of objects from the rendered manifest.

One alternative is to use use Pulumi transformations to transform the object and resource options. Another is to use post-rendering, which we’ll cover next.

Not supported: Helm hooks

Some charts use hooks to perform pre- and post-installation tasks, and we recommend using the Release resource to install such charts. Another option is to emulate the hook by performing the task directly, using other Pulumi resources such as kubernetes.batch/v1.Job and kubernetes.core/v1.Secret.

Note that Chart v3 does not support hooks either but, for historical reasons, does emit the hooks as ordinary child resources.

Evolving: Resource outputs

The child resources created by the chart are presented to your program via the resources output, as an array of Pulumi Resource objects. In some SDKs, you can cast the resource into the appropriate resource type, e.g. corev1.ConfigMap. Not all SDKs support this (yet); see pulumi/pulumi#15788.

Deploy Helm charts with Pulumi

Apply any Helm chart with the Chart v4 resource and manage the cloud resources it depends on, like IAM roles and storage buckets, in the same program.

Get started

Example: Argo CD

Here’s a real-world example of installing Argo CD into a Kubernetes cluster, and of using Argo CD’s Application custom resource to deploy the ‘guestbook’ example.

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";
import * as random from "@pulumi/random";

const ns = new kubernetes.core.v1.Namespace("ns", {metadata: {
 name: "argocd",
}});
const redisPasswordResource = new random.RandomPassword("redis-password", {length: 16});
const redisSecret = new kubernetes.core.v1.Secret("redis-secret", {
 metadata: {
 name: "argocd-redis",
 namespace: ns.metadata.apply(metadata => metadata.name),
 },
 type: "Opaque",
 stringData: {
 auth: redisPasswordResource.result,
 },
});
const argocd = new kubernetes.helm.v4.Chart("argocd", {
 chart: "argo-cd",
 version: "6.11.1",
 namespace: ns.metadata.apply(metadata => metadata.name),
 repositoryOpts: {
 repo: "https://argoproj.github.io/argo-helm",
 },
 values: {
 fullnameOverride: "",
 },
});
const guestbook = new kubernetes.yaml.v2.ConfigGroup("guestbook", {objs: [{
 apiVersion: "argoproj.io/v1alpha1",
 kind: "Application",
 metadata: {
 name: "guestbook",
 namespace: ns.metadata.apply(metadata => metadata.name),
 },
 spec: {
 project: "default",
 source: {
 repoURL: "https://github.com/argoproj/argocd-example-apps.git",
 targetRevision: "HEAD",
 path: "guestbook",
 },
 destination: {
 server: "https://kubernetes.default.svc",
 namespace: "default",
 },
 },
}]}, {
 dependsOn: [argocd],
});
export const redisPassword = redisPasswordResource.result;

import pulumi
import pulumi_kubernetes as kubernetes
import pulumi_random as random

ns = kubernetes.core.v1.Namespace("ns", metadata=kubernetes.meta.v1.ObjectMetaArgs(
 name="argocd",
))
redis_password_resource = random.RandomPassword("redis-password", length=16)
redis_secret = kubernetes.core.v1.Secret("redis-secret",
 metadata=kubernetes.meta.v1.ObjectMetaArgs(
 name="argocd-redis",
 namespace=ns.metadata.name,
 ),
 type="Opaque",
 string_data={
 "auth": redis_password_resource.result,
 })
argocd = kubernetes.helm.v4.Chart("argocd",
 chart="argo-cd",
 version="6.11.1",
 namespace=ns.metadata.name,
 repository_opts=kubernetes.helm.v4.RepositoryOptsArgs(
 repo="https://argoproj.github.io/argo-helm",
 ),
 values={
 "fullnameOverride": "",
 })
guestbook = kubernetes.yaml.v2.ConfigGroup("guestbook", objs=[{
 "apiVersion": "argoproj.io/v1alpha1",
 "kind": "Application",
 "metadata": {
 "name": "guestbook",
 "namespace": ns.metadata.name,
 },
 "spec": {
 "project": "default",
 "source": {
 "repoURL": "https://github.com/argoproj/argocd-example-apps.git",
 "targetRevision": "HEAD",
 "path": "guestbook",
 },
 "destination": {
 "server": "https://kubernetes.default.svc",
 "namespace": "default",
 },
 },
}],
opts=pulumi.ResourceOptions(depends_on=[argocd]))
pulumi.export("redisPassword", redis_password_resource.result)

package main

import (
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv4 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v4"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 yamlv2 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/yaml/v2"
 "github.com/pulumi/pulumi-random/sdk/v4/go/random"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 ns, err := corev1.NewNamespace(ctx, "ns", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("argocd"),
 },
 })
 if err != nil {
 return err
 }
 redisPasswordResource, err := random.NewRandomPassword(ctx, "redis-password", &random.RandomPasswordArgs{
 Length: pulumi.Int(16),
 })
 if err != nil {
 return err
 }
 _, err = corev1.NewSecret(ctx, "redis-secret", &corev1.SecretArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("argocd-redis"),
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 Type: pulumi.String("Opaque"),
 StringData: pulumi.StringMap{
 "auth": redisPasswordResource.Result,
 },
 })
 if err != nil {
 return err
 }
 argocd, err := helmv4.NewChart(ctx, "argocd", &helmv4.ChartArgs{
 Chart: pulumi.String("argo-cd"),
 Version: pulumi.String("6.11.1"),
 Namespace: ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 RepositoryOpts: &helmv4.RepositoryOptsArgs{
 Repo: pulumi.String("https://argoproj.github.io/argo-helm"),
 },
 Values: pulumi.Map{
 "fullnameOverride": pulumi.Any(""),
 },
 })
 if err != nil {
 return err
 }
 _, err = yamlv2.NewConfigGroup(ctx, "guestbook", &yamlv2.ConfigGroupArgs{
 Objs: pulumi.Array{
 pulumi.Any(map[string]interface{}{
 "apiVersion": "argoproj.io/v1alpha1",
 "kind": "Application",
 "metadata": map[string]interface{}{
 "name": "guestbook",
 "namespace": ns.Metadata.ApplyT(func(metadata metav1.ObjectMeta) (*string, error) {
 return metadata.Name, nil
 }).(pulumi.StringPtrOutput),
 },
 "spec": map[string]interface{}{
 "project": "default",
 "source": map[string]interface{}{
 "repoURL": "https://github.com/argoproj/argocd-example-apps.git",
 "targetRevision": "HEAD",
 "path": "guestbook",
 },
 "destination": map[string]interface{}{
 "server": "https://kubernetes.default.svc",
 "namespace": "default",
 },
 },
 }),
 },
 }, pulumi.DependsOn([]pulumi.Resource{
 argocd,
 }))
 if err != nil {
 return err
 }
 ctx.Export("redisPassword", redisPasswordResource.Result)
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Kubernetes = Pulumi.Kubernetes;
using Random = Pulumi.Random;

return await Deployment.RunAsync(() =>
{
 var ns = new Kubernetes.Core.V1.Namespace("ns", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "argocd",
 },
 });

 var redisPasswordResource = new Random.RandomPassword("redis-password", new()
 {
 Length = 16,
 });

 var redisSecret = new Kubernetes.Core.V1.Secret("redis-secret", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "argocd-redis",
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 },
 Type = "Opaque",
 StringData =
 {
 { "auth", redisPasswordResource.Result },
 },
 });

 var argocd = new Kubernetes.Helm.V4.Chart("argocd", new()
 {
 Chart = "argo-cd",
 Version = "6.11.1",
 Namespace = ns.Metadata.Apply(metadata => metadata.Name),
 RepositoryOpts = new Kubernetes.Types.Inputs.Helm.V4.RepositoryOptsArgs
 {
 Repo = "https://argoproj.github.io/argo-helm",
 },
 Values =
 {
 { "fullnameOverride", "" },
 },
 });

 var guestbook = new Kubernetes.Yaml.V2.ConfigGroup("guestbook", new()
 {
 Objs = new[]
 {
 new Dictionary<string, object?>
 {
 ["apiVersion"] = "argoproj.io/v1alpha1",
 ["kind"] = "Application",
 ["metadata"] = new Dictionary<string, object?>
 {
 ["name"] = "guestbook",
 ["namespace"] = ns.Metadata.Apply(metadata => metadata.Name),
 },
 ["spec"] = new Dictionary<string, object?>
 {
 ["project"] = "default",
 ["source"] = new Dictionary<string, object?>
 {
 ["repoURL"] = "https://github.com/argoproj/argocd-example-apps.git",
 ["targetRevision"] = "HEAD",
 ["path"] = "guestbook",
 },
 ["destination"] = new Dictionary<string, object?>
 {
 ["server"] = "https://kubernetes.default.svc",
 ["namespace"] = "default",
 },
 },
 },
 },
 }, new ComponentResourceOptions
 {
 DependsOn =
 {
 argocd,
 },
 });

 return new Dictionary<string, object?>
 {
 ["redisPassword"] = redisPasswordResource.Result,
 };
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.kubernetes.core.v1.Namespace;
import com.pulumi.kubernetes.core.v1.NamespaceArgs;
import com.pulumi.kubernetes.meta.v1.inputs.ObjectMetaArgs;
import com.pulumi.random.RandomPassword;
import com.pulumi.random.RandomPasswordArgs;
import com.pulumi.kubernetes.core.v1.Secret;
import com.pulumi.kubernetes.core.v1.SecretArgs;
import com.pulumi.kubernetes.helm.v4.Chart;
import com.pulumi.kubernetes.helm.v4.ChartArgs;
import com.pulumi.kubernetes.helm.v4.inputs.RepositoryOptsArgs;
import com.pulumi.kubernetes.yaml.v2.ConfigGroup;
import com.pulumi.kubernetes.yaml.v2.ConfigGroupArgs;
import com.pulumi.resources.ComponentResourceOptions;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var ns = new Namespace("ns", NamespaceArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("argocd")
 .build())
 .build());

 var redisPasswordResource = new RandomPassword("redisPasswordResource", RandomPasswordArgs.builder()
 .length(16)
 .build());

 var redisSecret = new Secret("redisSecret", SecretArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("argocd-redis")
 .namespace(ns.metadata().applyValue(metadata -> metadata.name().get()))
 .build())
 .type("Opaque")
 .stringData(redisPasswordResource.result().applyValue(password -> Map.of("auth", password)))
 .build());

 var argocd = new Chart("argocd", ChartArgs.builder()
 .chart("argo-cd")
 .version("6.11.1")
 .namespace(ns.metadata().applyValue(metadata -> metadata.name().get()))
 .repositoryOpts(RepositoryOptsArgs.builder()
 .repo("https://argoproj.github.io/argo-helm")
 .build())
 .values(Map.of("fullnameOverride", ""))
 .build());

 var guestbook = new ConfigGroup("guestbook", ConfigGroupArgs.builder()
 .objs(Map.ofEntries(
 Map.entry("apiVersion", "argoproj.io/v1alpha1"),
 Map.entry("kind", "Application"),
 Map.entry("metadata", Map.ofEntries(
 Map.entry("name", "guestbook"),
 Map.entry("namespace", ns.metadata().applyValue(metadata -> metadata.name().get()))
 )),
 Map.entry("spec", Map.ofEntries(
 Map.entry("project", "default"),
 Map.entry("source", Map.ofEntries(
 Map.entry("repoURL", "https://github.com/argoproj/argocd-example-apps.git"),
 Map.entry("targetRevision", "HEAD"),
 Map.entry("path", "guestbook")
 )),
 Map.entry("destination", Map.ofEntries(
 Map.entry("server", "https://kubernetes.default.svc"),
 Map.entry("namespace", "default")
 ))
 ))
 ))
 .build(), ComponentResourceOptions.builder()
 .dependsOn(argocd)
 .build());

 ctx.export("redisPassword", redisPasswordResource.result());
 }
}

resources:
 ns:
 type: kubernetes:core/v1:Namespace
 properties:
 metadata:
 name: argocd

 redis-password:
 type: random:RandomPassword
 properties:
 length: 16

 redis-secret:
 type: kubernetes:core/v1:Secret
 properties:
 metadata:
 name: argocd-redis
 namespace: ${ns.metadata.name}
 type: Opaque
 stringData:
 auth: ${redis-password.result}
 options:
 retainOnDelete: true

 argocd:
 type: kubernetes:helm.sh/v4:Chart
 properties:
 chart: argo-cd
 version: "6.11.1"
 namespace: ${ns.metadata.name}
 repositoryOpts:
 repo: https://argoproj.github.io/argo-helm
 values:
 fullnameOverride: ""

 guestbook:
 type: kubernetes:yaml/v2:ConfigGroup
 properties:
 objs:
 - apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
 name: guestbook
 namespace: ${ns.metadata.name}
 spec:
 project: default
 source:
 repoURL: https://github.com/argoproj/argocd-example-apps.git
 targetRevision: HEAD
 path: guestbook
 destination:
 server: https://kubernetes.default.svc
 namespace: default
 options:
 dependsOn:
 - ${argocd}

outputs:
 redisPassword: ${redis-password.result}

The program creates the argocd namespace, installs the ArgoCD server, and then creates an ArgoCD Application resource. Observe how the program installs and uses a Custom Resource Definition (CRD) successfully, and uses dependsOn to ensure that the CRD is installed before using it.

The argo-cd chart normally makes use of a Helm hook to initialize a password for the redis server. Since the Chart v4 resource doesn’t support Helm hooks, this program creates the password directly.

Conclusion

Pulumi loves empowering developers to use the best tools for the job, and we recommend using Helm charts to install third-party Kubernetes applications. Pulumi complements Helm by handling the cloud resources that are often required, such as an IAM Role or cloud storage bucket. Such combinations make for great reusable componentry.

Check out the following links to learn more about Pulumi Kubernetes today!

New: ConfigGroup, ConfigFile resources for Java, YAML SDKs

Eron Wright — Fri, 12 Apr 2024 00:00:00 -0700

The Pulumi Kubernetes provider makes it easy to deploy Kubernetes resources to your cluster, giving you options based on how your application or workload is packaged. The options include strongly-typed resources for standard Kubernetes types, Helm charts, Kustomizations, and Kubernetes manifests.

In v4.10, we leveled up the support for working with Kubernetes manifests with the introduction of the yaml/v2 package. The package provides new implementations of the ConfigGroup and ConfigFile resources, expanding support to the Pulumi Java SDK and to Pulumi YAML. The new implementations are also smarter about applying the objects in the correct order.

Please note that these resources are in a preview stage of maturity, as we continue to round out the feature set. These new resources are provided side-by-side with the existing implementations.

Let’s explore how it’s easier than ever to deploy diverse Kubernetes workloads using Pulumi.

kubernetes.yaml/v2.ConfigGroup

The ConfigGroup resource creates a set of Kubernetes objects from Kubernetes manifests and/or object literals. The Kubernetes manifest may be supplied using any combination of the following methods:

a filename or a list of filenames.
a file pattern or a list of file patterns.
a literal string containing YAML or JSON text.
a literal object or a list of objects.

The ConfigGroup resource accepts the following input properties:

Property	Description
`files`	Set of files, patterns and/or URLs to Kubernetes manifest files. Filenames containing any of the characters ``, `?`, and `[` are treated as patterns*.
`objs`	Set of Kubernetes objects.
`yaml`	A Kubernetes manifest as a YAML or JSON string.
`resourcePrefix`	A prefix for auto-generated names of the child resources. Defaults to the `ConfigGroup` name.
`skipAwait`	Skips over the readiness checks on the child resources.

Note that all objects defined within the manifest must have a metadata.name field; Pulumi autonaming is not supported.

See the API Reference documentation for more information.

Manifest Files

Here’s an example of applying a Kubernetes manifest file to your cluster.

name: demo-cg-1
runtime: yaml
resources:
 cg:
 type: kubernetes:yaml/v2:ConfigGroup
 properties:
 files:
 - ./manifest.yaml

YAML Text

Here’s how to apply an inline manifest.

name: demo-cg-2
runtime: yaml
resources:
 cg:
 type: kubernetes:yaml/v2:ConfigGroup
 properties:
 yaml: |
 apiVersion: v1
 kind: ConfigMap
 metadata:
 name: my-server-config
 data:
 altGreeting: "Good Morning!"
 ---
 apiVersion: v1
 kind: Secret
 metadata:
 name: my-dotfile-secret
 data:
 .secret-file: dmFsdWUtMg0KDQo=

Object Literals

With ConfigGroup it’s easy to define a Kubernetes object using Pulumi property values. Let’s create a Kubernetes namespace and deploy a custom resource into that namespace.

name: demo-cg-3
runtime: yaml
resources:
 ns:
 type: kubernetes:core/v1:Namespace
 cg:
 type: kubernetes:yaml/v2:ConfigGroup
 properties:
 objs:
 - apiVersion: stable.example.com/v1
 kind: CronTab
 metadata:
 namespace: ${ns.metadata.name}
 name: my-crontab
 spec:
 cronSpec: "* * * * */5"

This feature is especially useful for Pulumi YAML, since the apiextensions.k8s.io:CustomResource resource isn’t supported yet (see pulumi-kubernetes#2787).

Resource Ordering

The ConfigGroup resource automatically detects dependencies between resources in the manifest(s). For example, it knows to install namespaces and Custom Resource Definitions (CRDs) first.

Use the config.kubernetes.io/depends-on annotation to declare an explicit resource dependency. The annotation accepts a list of resource references, delimited by commas.

It consists of the group, kind, name, and optionally the namespace, delimited by forward slashes.

Resource Scope	Format
namespace-scoped	`<group>/namespaces/<namespace>/<kind>/<name>`
cluster-scoped	`<group>/<kind>/<name>`

For resources in the “core” group, the empty string is used instead (for example: /namespaces/test/Pod/pod-a).

Resource Prefixes

The purpose of the resource prefix is to ensure the uniqueness of child resource names, as described in Pulumi: Resource Names. The best practice is to use the component name as a prefix, and now that’s the default behavior.

Note that the resource prefix is not applied to the Kubernetes object names.

Transformation Support

The Kubernetes provider generally supports resource transformations, e.g. to apply a resource alias or to apply an object name prefix or object label. Transformations aren’t yet possible with the new ConfigGroup resource (see pulumi/pulumi#12996), but stay tuned as we work towards a GA release later this year.

kubernetes.yaml/v2.ConfigFile

The ConfigFile resource creates a set of Kubernetes objects from a Kubernetes manifest. While ConfigGroup offers a superset of functionality, the ConfigFile gives a concise way to apply a specific manifest.

The ConfigFile resource accepts the following input properties:

Property	Description
`file`	A file or URL to a Kubernetes manifest file.
`resourcePrefix`	A prefix for auto-generated names of the child resources. Defaults to the `ConfigFile` name.
`skipAwait`	Skips over the readiness checks on the child resources.

See the API Reference documentation for more information.

Manifest File

Here’s an example of applying a Kubernetes manifest file to your cluster:

name: demo-cg-1
runtime: yaml
resources:
 cf:
 type: kubernetes:yaml/v2:ConfigFile
 properties:
 file: ./manifest.yaml

Example: Certificate Manager

Here’s a complete example of how to install and use cert-manager using ConfigGroup resources.

name: cert-manager
runtime: yaml
description: Installs cert-manager. See https://cert-manager.io/docs/installation/kubectl/ for details.
variables: {}
resources:
 install:
 type: kubernetes:yaml/v2:ConfigGroup
 properties:
 files:
 - https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml
 test:
 type: kubernetes:yaml/v2:ConfigGroup
 options:
 dependsOn:
 - ${install}
 properties:
 yaml: |
 apiVersion: v1
 kind: Namespace
 metadata:
 name: cert-manager-test
 ---
 apiVersion: cert-manager.io/v1
 kind: Issuer
 metadata:
 name: test-selfsigned
 namespace: cert-manager-test
 spec:
 selfSigned: {}
 ---
 apiVersion: cert-manager.io/v1
 kind: Certificate
 metadata:
 name: selfsigned-cert
 namespace: cert-manager-test
 annotations:
 config.kubernetes.io/depends-on: cert-manager.io/namespaces/cert-manager-test/Issuer/test-selfsigned
 spec:
 dnsNames:
 - example.com
 secretName: selfsigned-cert-tls
 issuerRef:
 name: test-selfsigned

Conclusion

With the new resources in yaml/v2, we’re excited to enable new scenarios in Pulumi YAML and the Pulumi Java SDK. Stay tuned for new editions of the Chart and Directory resources, coming soon.

This is part of a broad initiative to use Multi-Language Component (MLC) technology to offer a consistent experience across the Pulumi SDKs. See pulumi-kubernetes#1971 for details.

Check out the following links to learn more about Pulumi Kubernetes today!

Secure your Kubernetes toolchain with Pulumi ESC and OIDC

Levi Blackstone — Tue, 21 Nov 2023 00:00:00 -0700

Keeping long-lived kubeconfig around on disk is insecure and error-prone. You need a secure workflow that removes tedium. With Pulumi and ESC, we provide an automated workflow that generates a kubeconfig on-the-fly for every command using short-term credentials issued via OIDC. This makes it easy for your team to connect to a given Kubernetes environment, and it works well with Kubernetes tools such as kubectl and the Pulumi Kubernetes provider. Let’s take a look.

In this post, we will create a Kubernetes cluster, deploy an application to the created cluster, and then use kubectl to check on the deployed application. Pulumi gives you great flexibility to factor your infrastructure into reusable parts, and we’ll do that here. With ESC, we’ll define an environment that provides access to an AWS account, and use that environment to create the cluster. Then we’ll define an environment that encapsulates the kubeconfig needed to connect to that cluster.

Create a Kubernetes Cluster

First, we create a Kubernetes cluster using the pulumi-eks component.

import * as pulumi from "@pulumi/pulumi";
import * as eks from "@pulumi/eks";

const eksCluster = new eks.Cluster("eks-cluster", {});
export const kubeconfig = eksCluster.kubeconfig;

import pulumi
import pulumi_eks as eks

eks_cluster = eks.Cluster("eks-cluster")
pulumi.export("kubeconfig", eks_cluster.kubeconfig)

import (
 "github.com/pulumi/pulumi-eks/sdk/go/eks"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 eksCluster, err := eks.NewCluster(ctx, "eks-cluster", nil)
 if err != nil {
 return err
 }
 ctx.Export("kubeconfig", eksCluster.Kubeconfig)
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Eks = Pulumi.Eks;

return await Deployment.RunAsync(() =>
{
 var eksCluster = new Eks.Cluster("eks-cluster");

 return new Dictionary<string, object?>
 {
 ["kubeconfig"] = eksCluster.Kubeconfig,
 };
});

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.eks.Cluster;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var eksCluster = new Cluster("eksCluster");

 ctx.export("kubeconfig", eksCluster.kubeconfig());
 }
}

name: eks-cluster
runtime: nodejs
description: EKS Kubernetes Cluster
resources:
 eks-cluster:
 type: eks:Cluster
outputs:
 kubeconfig: ${eks-cluster.kubeconfig}

This component includes an output value for the cluster’s kubeconfig that we can use to connect to the cluster. However, before we create the cluster, we need to configure AWS credentials so that Pulumi can deploy the requested changes. While it is possible to set this configuration directly on the stack, let’s see how it can be done with Pulumi ESC. First, we create the following ESC environment (the aws-sandbox environment):

values:
 aws:
 creds:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::0123456789:role/deploy-oidc
 sessionName: pulumi-environments-session
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.creds.keyId}
 AWS_SECRET_ACCESS_KEY: ${aws.creds.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.creds.sessionToken}
 pulumiConfig:
 aws:region: us-west-2

Notice that this environment uses the aws-login ESC provider to dynamically load short-lived credentials using OIDC. These credentials are then exposed as environment variables for consumers of this environment. We also set the AWS region in the pulumiConfig section to configure the region for pulumi to manage resources with the pulumi-aws provider.

To have Pulumi use this ESC environment when deploying the cluster, we set the following configuration for our stack:

environment:
 - aws-sandbox

The environment key accepts a list of ESC environments to import. When the stack is updated, pulumi automatically sets the configured environment variables and stack configuration based on the ESC environment. We run our update, and Pulumi creates our EKS cluster.

pulumi up --skip-preview
Updating (demo)
Type Name Status
+ pulumi:pulumi:Stack eks-cluster-demo created (577s)
+ └─ eks:index:Cluster eks-cluster created (561s)
+ ├─ eks:index:ServiceRole eks-cluster-eksRole created (2s)
+ │ ├─ aws:iam:Role eks-cluster-eksRole-role created (0.78s)
+ │ └─ aws:iam:RolePolicyAttachment eks-cluster-eksRole-4b490823 created (0.44s)
+ ├─ eks:index:ServiceRole eks-cluster-instanceRole created (3s)
+ │ ├─ aws:iam:Role eks-cluster-instanceRole-role created (0.85s)
+ │ ├─ aws:iam:RolePolicyAttachment eks-cluster-instanceRole-3eb088f2 created (0.46s)
+ │ ├─ aws:iam:RolePolicyAttachment eks-cluster-instanceRole-03516f97 created (0.59s)
+ │ └─ aws:iam:RolePolicyAttachment eks-cluster-instanceRole-e1b295bd created (0.75s)
+ ├─ eks:index:RandomSuffix eks-cluster-cfnStackName created (0.49s)
+ ├─ aws:ec2:SecurityGroup eks-cluster-eksClusterSecurityGroup created (1s)
+ ├─ aws:iam:InstanceProfile eks-cluster-instanceProfile created (0.48s)
+ ├─ aws:eks:Cluster eks-cluster-eksCluster created (441s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksClusterInternetEgressRule created (0.81s)
+ ├─ aws:ec2:SecurityGroup eks-cluster-nodeSecurityGroup created (1s)
+ ├─ pulumi:providers:kubernetes eks-cluster-eks-k8s created (0.14s)
+ ├─ eks:index:VpcCni eks-cluster-vpc-cni created (2s)
+ ├─ kubernetes:core/v1:ConfigMap eks-cluster-nodeAccess created (0.60s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksExtApiServerClusterIngressRule created (1s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksClusterIngressRule created (1s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksNodeInternetEgressRule created (1s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksNodeClusterIngressRule created (2s)
+ ├─ aws:ec2:SecurityGroupRule eks-cluster-eksNodeIngressRule created (2s)
+ ├─ aws:ec2:LaunchConfiguration eks-cluster-nodeLaunchConfiguration created (1s)
+ ├─ aws:cloudformation:Stack eks-cluster-nodes created (108s)
+ └─ pulumi:providers:kubernetes eks-cluster-provider created (0.12s)
Outputs:
kubeconfig: {
apiVersion : "v1"
clusters : [
[0]: {
cluster: {
certificate-authority-data: "<redacted>"
server : "https://FB9978041F7B72B227B7E95F076822E4.gr7.us-west-2.eks.amazonaws.com"
}
name : "kubernetes"
}
]
contexts : [
[0]: {
context: {
cluster: "kubernetes"
user : "aws"
}
name : "aws"
}
]
current-context: "aws"
kind : "Config"
users : [
[0]: {
name: "aws"
user: {
exec: {
apiVersion: "client.authentication.k8s.io/v1beta1"
args : [
[0]: "eks"
[1]: "get-token"
[2]: "--cluster-name"
[3]: "eks-cluster-eksCluster-1268ff9"
]
command : "aws"
env : [
[0]: {
name : "KUBERNETES_EXEC_INFO"
value: (json) {
apiVersion: "client.authentication.k8s.io/v1beta1"
}
}
]
}
}
}
]
}
Resources:
+ 27 created
Duration: 9m38s

Connect to the cluster

Let’s create another Pulumi ESC environment called eks-sandbox to encapsulate the configuration needed to connect to the EKS cluster.

imports:
 - aws-sandbox
values:
 stacks:
 fn::open::pulumi-stacks:
 stacks:
 eks-cluster:
 stack: eks-cluster/demo
 kubeconfig: {'fn::toJSON': "${stacks.eks-cluster.kubeconfig}"}
 pulumiConfig:
 kubernetes:kubeconfig: ${kubeconfig}
 files:
 KUBECONFIG: ${kubeconfig}

This ESC environment uses another ESC provider to reference outputs from Pulumi stacks in your organization. In this case, we specify that we want to read outputs from the eks-cluster/demo stack, and then export the kubeconfig output from that stack as pulumiConfig and files. As we saw on the cluster environment, pulumiConfig sets configuration for Pulumi providers, so this allows us to run our stack and deploy the application to our cluster.

We import the aws-sandbox environment defined earlier, to provide the user with AWS credentials to authenticate to the EKS cluster.

Since the eks-sandbox environment defines a KUBECONFIG variable that kubectl understands, we can run kubectl in the context of this environment like this:

$ pulumi env run eks-sandbox -- kubectl cluster-info
Kubernetes control plane is running at https://2F3303BF9E03F8D61099AAB5ED6F31A4.gr7.us-west-2.eks.amazonaws.com
CoreDNS is running at https://2F3303BF9E03F8D61099AAB5ED6F31A4.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Deploy an application to the cluster

Now, we want to deploy an application to our new EKS cluster. While we could have included this in the same stack as the cluster, it is generally better to follow the separation of concerns principle, and manage applications separately from the underlying infrastructure. To achieve this goal, we create a new application stack with the following program:

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";

const nginxDeployment = new kubernetes.apps.v1.Deployment("nginx-deployment", {
 metadata: {
 name: "nginx",
 labels: {
 app: "nginx",
 },
 },
 spec: {
 replicas: 3,
 selector: {
 matchLabels: {
 app: "nginx",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "nginx",
 },
 },
 spec: {
 containers: [{
 name: "nginx",
 image: "nginx",
 ports: [{
 containerPort: 80,
 }],
 }],
 },
 },
 },
});
const nginxService = new kubernetes.core.v1.Service("nginx-service", {
 metadata: {
 name: "nginx",
 },
 spec: {
 type: "LoadBalancer",
 selector: {
 app: "nginx",
 },
 ports: [{
 protocol: "TCP",
 port: 80,
 }],
 },
});

import pulumi
import pulumi_kubernetes as kubernetes

nginx_deployment = kubernetes.apps.v1.Deployment("nginx-deployment",
 metadata=kubernetes.meta.v1.ObjectMetaArgs(
 name="nginx",
 labels={
 "app": "nginx",
 },
 ),
 spec=kubernetes.apps.v1.DeploymentSpecArgs(
 replicas=3,
 selector=kubernetes.meta.v1.LabelSelectorArgs(
 match_labels={
 "app": "nginx",
 },
 ),
 template=kubernetes.core.v1.PodTemplateSpecArgs(
 metadata=kubernetes.meta.v1.ObjectMetaArgs(
 labels={
 "app": "nginx",
 },
 ),
 spec=kubernetes.core.v1.PodSpecArgs(
 containers=[kubernetes.core.v1.ContainerArgs(
 name="nginx",
 image="nginx",
 ports=[kubernetes.core.v1.ContainerPortArgs(
 container_port=80,
 )],
 )],
 ),
 ),
 ))
nginx_service = kubernetes.core.v1.Service("nginx-service",
 metadata=kubernetes.meta.v1.ObjectMetaArgs(
 name="nginx",
 ),
 spec=kubernetes.core.v1.ServiceSpecArgs(
 type="LoadBalancer",
 selector={
 "app": "nginx",
 },
 ports=[kubernetes.core.v1.ServicePortArgs(
 protocol="TCP",
 port=80,
 )],
 ))

import (
 appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := appsv1.NewDeployment(ctx, "nginx-deployment", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx"),
 Labels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Spec: &appsv1.DeploymentSpecArgs{
 Replicas: pulumi.Int(3),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx"),
 Ports: corev1.ContainerPortArray{
 &corev1.ContainerPortArgs{
 ContainerPort: pulumi.Int(80),
 },
 },
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }
 _, err = corev1.NewService(ctx, "nginx-service", &corev1.ServiceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx"),
 },
 Spec: &corev1.ServiceSpecArgs{
 Type: pulumi.String("LoadBalancer"),
 Selector: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 Ports: corev1.ServicePortArray{
 &corev1.ServicePortArgs{
 Protocol: pulumi.String("TCP"),
 Port: pulumi.Int(80),
 },
 },
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 var nginxDeployment = new Kubernetes.Apps.V1.Deployment("nginx-deployment", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "nginx",
 Labels =
 {
 { "app", "nginx" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Replicas = 3,
 Selector = new Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "nginx" },
 },
 },
 Template = new Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Labels =
 {
 { "app", "nginx" },
 },
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 Containers = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "nginx",
 Image = "nginx",
 Ports = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ContainerPortArgs
 {
 ContainerPortValue = 80,
 },
 },
 },
 },
 },
 },
 },
 });

 var nginxService = new Kubernetes.Core.V1.Service("nginx-service", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "nginx",
 },
 Spec = new Kubernetes.Types.Inputs.Core.V1.ServiceSpecArgs
 {
 Type = "LoadBalancer",
 Selector =
 {
 { "app", "nginx" },
 },
 Ports = new[]
 {
 new Kubernetes.Types.Inputs.Core.V1.ServicePortArgs
 {
 Protocol = "TCP",
 Port = 80,
 },
 },
 },
 });

});

import java.security.Provider.Service;
import java.util.Map;

import javax.naming.Context;

import com.pulumi.Pulumi;
import com.pulumi.kubernetes.apps_v1.Deployment;
import com.pulumi.kubernetes.apps_v1.DeploymentArgs;
import com.pulumi.kubernetes.apps_v1.inputs.DeploymentSpecArgs;
import com.pulumi.kubernetes.core_v1.ServiceArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.PodTemplateSpecArgs;
import com.pulumi.kubernetes.core_v1.inputs.ServiceSpecArgs;
import com.pulumi.kubernetes.meta_v1.inputs.LabelSelectorArgs;
import com.pulumi.kubernetes.meta_v1.inputs.ObjectMetaArgs;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var nginxDeployment = new Deployment("nginxDeployment", DeploymentArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx")
 .labels(Map.of("app", "nginx"))
 .build())
 .spec(DeploymentSpecArgs.builder()
 .replicas(3)
 .selector(LabelSelectorArgs.builder()
 .matchLabels(Map.of("app", "nginx"))
 .build())
 .template(PodTemplateSpecArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .labels(Map.of("app", "nginx"))
 .build())
 .spec(PodSpecArgs.builder()
 .containers(ContainerArgs.builder()
 .name("nginx")
 .image("nginx")
 .ports(ContainerPortArgs.builder()
 .containerPort(80)
 .build())
 .build())
 .build())
 .build())
 .build())
 .build());

 var nginxService = new Service("nginxService", ServiceArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx")
 .build())
 .spec(ServiceSpecArgs.builder()
 .type("LoadBalancer")
 .selector(Map.of("app", "nginx"))
 .ports(ServicePortArgs.builder()
 .protocol("TCP")
 .port(80)
 .build())
 .build())
 .build());

 }
}

name: yaml-example
runtime: yaml
description: Deployment + Service
resources:
 nginx-deployment:
 type: kubernetes:apps/v1:Deployment
 properties:
 metadata:
 name: nginx
 labels:
 app: nginx
 spec:
 replicas: 3
 selector:
 matchLabels:
 app: nginx
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: nginx
 image: "nginx"
 ports:
 - containerPort: 80
 nginx-service:
 type: kubernetes:core/v1:Service
 properties:
 metadata:
 name: nginx
 spec:
 type: LoadBalancer
 selector:
 app: nginx
 ports:
 - protocol: TCP
 port: 80

We need to set some configuration before we deploy this stack. In this case, we need a kubeconfig to allow the pulumi-kubernetes provider to connect to our EKS cluster.

We import the eks-sandbox environment with the following stack configuration:

environment:
 - eks-sandbox

Now we update our stack, and see that our application gets deployed to the cluster, without needing to manually configure any credentials – ESC automatically injects them for us!

pulumi up --skip-preview
Updating (demo)
Type Name Status
+ pulumi:pulumi:Stack yaml-example-demo created (0.31s)
+ ├─ kubernetes:core/v1:Service nginx-service created (10s)
+ └─ kubernetes:apps/v1:Deployment nginx-deployment created (2s)
Resources:
+ 3 created
Duration: 13s

Now that the application is deployed, let’s take a look using kubectl:

$ pulumi env run eks-sandbox -- kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/3 3 3 103s

ESC environments can be used with the pulumi CLI using the pulumi env command.

Export Files

As we see in the ESC environment definition named eks-sandbox, there’s a files section (under values) for defining values that are automatically exported as temporary files when you run the environment. You can use this feature to generate configuration files to support your preferred tools. Pulumi exports each key-value pair as a temporary file containing the actual value, and an associated environment variable pointing to it.

Earlier we exported an environment variable named KUBECONFIG with a dynamic value based on a stack output. Here’s a similar example with the configuration declared statically:

values:
 files:
 KUBECONFIG: |
 apiVersion: v1
 kind: Config
 clusters:
 - cluster:
 server: https://127.0.0.1:6443
 name: docker-desktop

Conclusion

Pulumi ESC makes it easier than ever to tame infrastructure complexity. For this demo, we created two reusable ESC environments to configure AWS and Kubernetes credentials, created an EKS cluster, deployed a load-balanced application to the cluster, and configured our local toolchain to connect to the cluster. Since we factored the shared configuration into ESC environments, it is easy to extend this example to deploy additional AWS or Kubernetes resources. Pulumi ESC supports dynamic credentials using OIDC across AWS, Azure, and Google Cloud, with additional providers on the roadmap. Check out the following links to learn more about Pulumi ESC today!