Pulumi Blog: Best Practices

How to Test Infrastructure as Code

Alex Leventer — Tue, 30 Jun 2026 14:00:00 -0700

IaC testing means validating your infrastructure code the same way you test application software—unit tests with mocked cloud providers that run in milliseconds, integration tests that deploy and inspect real resources, and policy checks that enforce compliance rules on every preview and deploy. Together, these layers catch misconfigurations before they reach production.

Untested infrastructure code is a liability. A missing tag in a security group rule, an S3 bucket with public read access, a misconfigured IAM policy—any of these can slip through code review and land in production. Infrastructure bugs are often harder to debug than application bugs, because the feedback loop is slow (deploy, observe, destroy) and the blast radius is large (an outage, a security incident, a surprise AWS bill).

The good news: Pulumi is built on general-purpose programming languages, which means you can test infrastructure code with the exact same tools and frameworks you already use to test application code. No new language to learn. No separate toolchain. Just pytest, Mocha, or go test—pointing at your infrastructure program.

This guide walks through all three testing layers with complete, runnable examples.

Why should you test infrastructure as code?

The case for testing application code is well-established. The case for testing IaC is just as strong—arguably stronger, because the consequences of an undetected bug are more severe.

Catch misconfigurations before they deploy. A unit test that checks whether a security group allows SSH from 0.0.0.0/0 will catch that misconfiguration in milliseconds, before any cloud API call is made. The same test in CI catches it before the code is merged.

Prevent drift and outages. Integration tests that deploy to a staging environment and validate runtime behavior—does the HTTP endpoint return 200? does the database accept connections?—give you confidence that an infrastructure change doesn’t silently break a dependent service.

Refactor safely. Well-tested infrastructure code can be restructured, modularized, and upgraded without fear. When all your properties, tags, and security rules are asserted in tests, you know immediately if a refactor breaks a constraint.

Enforce security and compliance automatically. Policy as code lets you encode your organization’s security rules as executable tests. Mandatory policies block non-compliant deployments entirely; advisory policies surface warnings for human review. These rules run on every pulumi preview and pulumi up—not just at audit time.

Ship infrastructure at software speed. Teams that test IaC ship changes faster, with less fear. The CI pipeline becomes a confidence engine rather than a deployment gate.

The annual DORA State of DevOps research is consistently associated with a pattern among elite-performing engineering teams—those that deploy frequently and recover quickly from failures: comprehensive automated testing across the stack. Infrastructure is no longer exempt from that standard.

What types of IaC tests are there?

IaC testing mirrors the classic software test pyramid: a wide base of fast, cheap unit tests, a smaller layer of integration tests against real infrastructure, and policy checks that enforce rules throughout.

Layer	What it tests	Speed	Cost	Cloud credentials?
Unit tests	Resource properties, tags, security rules, naming conventions	Milliseconds	Free	No
Integration tests	Full stack lifecycle, runtime behavior, cross-resource dependencies	Minutes	Real cloud spend	Yes
Policy as code	Compliance rules, security baselines, organizational standards	Milliseconds	Free	No (runs at preview time)

graph TD
A["🔒 Policy as Code<br/>Compliance &amp; guardrails<br/>Runs at preview time<br/>Milliseconds · Free"] --> B
B["🧪 Integration Tests<br/>Real cloud deployments<br/>Full lifecycle validation<br/>Minutes · Cloud spend"] --> C
C["⚡ Unit Tests<br/>Mocked providers<br/>Property assertions<br/>Milliseconds · Free"]
style A fill:#8A3FFC,color:#fff,stroke:#6B2FD4
style B fill:#0EA5E9,color:#fff,stroke:#0284C7
style C fill:#10B981,color:#fff,stroke:#059669

Start with unit tests—they’re free, fast, and catch the most common mistakes. Add integration tests for critical paths (the deployment pipeline, data infrastructure, security-sensitive resources). Layer in policy as code for compliance requirements that must never regress.

How do you write unit tests for Pulumi?

Pulumi unit tests replace the communication channel between the Pulumi program and the cloud provider with mocks. When the program registers a resource (e.g., aws.ec2.SecurityGroup(...)), the mock intercepts the call, returns dummy state, and never makes a real cloud API call. Tests run in the same process, in the same language, using the same test runner you already use.

Setting up mocks

The import-order rule: You must set up mocks before importing your Pulumi program. If you import the program first, the Pulumi runtime initializes without mocks and will attempt real cloud calls.

# test_infra.py
import unittest
import pulumi

class MyMocks(pulumi.runtime.Mocks):
 def new_resource(self, args: pulumi.runtime.MockResourceArgs):
 # Return [id, state]. State keys must be camelCase.
 return [args.name + "_id", args.inputs]

 def call(self, args: pulumi.runtime.MockCallArgs):
 return {}

# Set mocks BEFORE importing the program
pulumi.runtime.set_mocks(MyMocks())
import infra # import program AFTER setting mocks

class TestInfra(unittest.TestCase):

 @pulumi.runtime.test
 def test_no_public_ssh(self):
 """Security group must not expose SSH to the public internet."""
 def check_security_group(args):
 ingress_rules = args[0]
 for rule in (ingress_rules or []):
 for cidr in (rule.get("cidrBlocks") or []):
 self.assertNotEqual(
 cidr, "0.0.0.0/0",
 "Security group must not expose port 22 to the internet."
 )
 return pulumi.Output.all(infra.group.ingress).apply(check_security_group)

 @pulumi.runtime.test
 def test_server_has_tags(self):
 """EC2 instance must have required tags."""
 def check_tags(args):
 tags = args[0]
 self.assertIn("Environment", tags, "Missing 'Environment' tag")
 self.assertIn("Name", tags, "Missing 'Name' tag")
 return pulumi.Output.all(infra.server.tags).apply(check_tags)

if __name__ == "__main__":
 unittest.main()

Run with: python -m unittest discover -v

// infra.test.ts
import * as pulumi from "@pulumi/pulumi";
import { expect } from "chai";

// Set mocks BEFORE dynamic import of the program
pulumi.runtime.setMocks(
 {
 newResource: (args: pulumi.runtime.MockResourceArgs): { id: string; state: any } => {
 // State keys must be camelCase
 return { id: `${args.name}-id`, state: args.inputs };
 },
 call: (args: pulumi.runtime.MockCallArgs) => ({ ...args.inputs }),
 },
 "my-project",
 "test",
 false // preview = false
);

describe("Infrastructure", function () {
 let infra: typeof import("./index");

 before(async function () {
 // Dynamic import AFTER setting mocks
 infra = await import("./index");
 });

 it("must not expose SSH to the internet", function (done) {
 pulumi.all([infra.group.ingress]).apply(([ingress]) => {
 for (const rule of ingress ?? []) {
 for (const cidr of rule.cidrBlocks ?? []) {
 expect(cidr).to.not.equal(
 "0.0.0.0/0",
 "Security group must not expose port 22 to the internet"
 );
 }
 }
 done();
 });
 });

 it("must have required tags", function (done) {
 pulumi.all([infra.server.tags]).apply(([tags]) => {
 expect(tags).to.include.keys("Environment", "Name");
 done();
 });
 });
});

Run with: mocha -r ts-node/register infra.test.ts

// infra_test.go
package infra_test

import (
 "testing"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/stretchr/testify/assert"
)

// MockResourceMonitor implements pulumi.ResourceMonitor for unit tests.
func TestNoPublicSSH(t *testing.T) {
 err := pulumi.RunErr(func(ctx *pulumi.Context) error {
 infra, err := NewInfra(ctx)
 assert.NoError(t, err)

 infra.Group.Ingress.ApplyT(func(rules []ec2.SecurityGroupIngress) error {
 for _, rule := range rules {
 for _, cidr := range rule.CidrBlocks {
 assert.NotEqual(t, "0.0.0.0/0", cidr,
 "Security group must not expose port 22 to the internet")
 }
 }
 return nil
 })
 return nil
 }, pulumi.WithMocks("my-project", "test", mocks{}))
 assert.NoError(t, err)
}

type mocks struct{}

func (mocks) NewResource(args pulumi.MockResourceArgs) (string, map[string]interface{}, error) {
 return args.Name + "_id", args.Inputs, nil
}
func (mocks) Call(args pulumi.MockCallArgs) (map[string]interface{}, error) {
 return args.Args, nil
}

Run with: go test ./... -v

// InfraTests.cs
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Testing;
using Xunit;

public class InfraTests
{
 [Fact]
 public async Task NoPublicSSH()
 {
 var resources = await Testing.RunAsync<MyStack>();

 var sg = resources.OfType<Pulumi.Aws.Ec2.SecurityGroup>().First();
 var ingress = await sg.Ingress.GetValueAsync();

 foreach (var rule in ingress)
 {
 foreach (var cidr in rule.CidrBlocks)
 {
 Assert.NotEqual("0.0.0.0/0", cidr);
 }
 }
 }

 [Fact]
 public async Task ServerHasRequiredTags()
 {
 var resources = await Testing.RunAsync<MyStack>();

 var server = resources.OfType<Pulumi.Aws.Ec2.Instance>().First();
 var tags = await server.Tags.GetValueAsync();

 Assert.True(tags.ContainsKey("Environment"), "Missing 'Environment' tag");
 Assert.True(tags.ContainsKey("Name"), "Missing 'Name' tag");
 }
}

Run with: dotnet test

// InfraTest.java
package myproject;

import com.pulumi.aws.ec2.Instance;
import com.pulumi.aws.ec2.SecurityGroup;
import com.pulumi.test.Mocks;
import com.pulumi.test.Mocks.CallArgs;
import com.pulumi.test.Mocks.ResourceArgs;
import com.pulumi.test.Mocks.ResourceResult;
import com.pulumi.test.PulumiTest;
import com.pulumi.test.TestOptions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import static org.junit.jupiter.api.Assertions.*;

class InfraTest {

 // Implement Mocks to intercept resource creation. State keys must be camelCase.
 static class MyMocks implements Mocks {
 @Override
 public CompletableFuture<ResourceResult> newResourceAsync(ResourceArgs args) {
 var state = new HashMap<>(args.inputs);
 return CompletableFuture.completedFuture(
 ResourceResult.of(Optional.of(args.name + "_id"), state));
 }

 @Override
 public CompletableFuture<Map<String, Object>> callAsync(CallArgs args) {
 return CompletableFuture.completedFuture(Map.of());
 }
 }

 // PulumiTest.cleanup() must run after every test to reset the runtime state.
 @AfterEach
 void cleanup() {
 PulumiTest.cleanup();
 }

 @Test
 void serverHasRequiredTags() {
 var result = PulumiTest
 .withMocks(new MyMocks())
 .withOptions(TestOptions.builder()
 .projectName("project").stackName("stack").preview(false)
 .build())
 .runTest(ctx -> new Infra());

 var instance = result.resources().stream()
 .filter(r -> r instanceof Instance)
 .map(r -> (Instance) r)
 .findFirst()
 .orElseThrow(() -> new AssertionError("EC2 instance not found"));

 var tags = PulumiTest.extractValue(instance.tags())
 .orElseThrow(() -> new AssertionError("Tags must not be null"));
 assertTrue(tags.containsKey("Environment"), "Missing 'Environment' tag");
 assertTrue(tags.containsKey("Name"), "Missing 'Name' tag");
 }

 @Test
 void noPublicSshExposed() {
 var result = PulumiTest
 .withMocks(new MyMocks())
 .withOptions(TestOptions.builder()
 .projectName("project").stackName("stack").preview(false)
 .build())
 .runTest(ctx -> new Infra());

 var group = result.resources().stream()
 .filter(r -> r instanceof SecurityGroup)
 .map(r -> (SecurityGroup) r)
 .findFirst()
 .orElseThrow(() -> new AssertionError("Security group not found"));

 var rules = PulumiTest.extractValue(group.ingress());
 for (var rule : rules) {
 for (var cidr : rule.cidrBlocks()) {
 assertNotEquals("0.0.0.0/0", cidr,
 "Security group must not expose SSH to the internet");
 }
 }
 }
}

Run with: mvn test (JUnit 5)

The program under test

Both examples above test this program:

# infra.py
import pulumi
from pulumi_aws import ec2

group = ec2.SecurityGroup("web-secgrp",
 ingress=[{
 "protocol": "tcp",
 "from_port": 80,
 "to_port": 80,
 "cidrBlocks": ["0.0.0.0/0"],
 }],
)

server = ec2.Instance("web-server",
 instance_type="t2.micro",
 ami="ami-0b0ea68c435eb488d",
 vpc_security_group_ids=[group.id],
 tags={"Name": "web-server", "Environment": "dev"},
)

// index.ts
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

export const group = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [{
 protocol: "tcp",
 fromPort: 80,
 toPort: 80,
 cidrBlocks: ["0.0.0.0/0"],
 }],
});

export const server = new aws.ec2.Instance("web-server", {
 instanceType: "t2.micro",
 ami: "ami-0b0ea68c435eb488d",
 vpcSecurityGroupIds: [group.id],
 tags: { Name: "web-server", Environment: "dev" },
});

// main.go
package infra_test

import (
 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

type Infra struct {
 Group *ec2.SecurityGroup
 Server *ec2.Instance
}

func NewInfra(ctx *pulumi.Context) (*Infra, error) {
 group, err := ec2.NewSecurityGroup(ctx, "web-secgrp", &ec2.SecurityGroupArgs{
 Ingress: ec2.SecurityGroupIngressArray{
 ec2.SecurityGroupIngressArgs{
 Protocol: pulumi.String("tcp"),
 FromPort: pulumi.Int(80),
 ToPort: pulumi.Int(80),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 },
 })
 if err != nil {
 return nil, err
 }

 server, err := ec2.NewInstance(ctx, "web-server", &ec2.InstanceArgs{
 InstanceType: pulumi.String("t2.micro"),
 Ami: pulumi.String("ami-0b0ea68c435eb488d"),
 VpcSecurityGroupIds: pulumi.StringArray{group.ID()},
 Tags: pulumi.StringMap{
 "Name": pulumi.String("web-server"),
 "Environment": pulumi.String("dev"),
 },
 })
 if err != nil {
 return nil, err
 }

 return &Infra{Group: group, Server: server}, nil
}

// MyStack.cs
using Pulumi;
using Pulumi.Aws.Ec2;
using Pulumi.Aws.Ec2.Inputs;

public class MyStack : Stack
{
 public MyStack()
 {
 var group = new SecurityGroup("web-secgrp", new SecurityGroupArgs
 {
 Ingress = new[]
 {
 new SecurityGroupIngressArgs
 {
 Protocol = "tcp", FromPort = 80, ToPort = 80,
 CidrBlocks = new[] { "0.0.0.0/0" },
 },
 },
 });

 var server = new Instance("web-server", new InstanceArgs
 {
 InstanceType = "t2.micro",
 Ami = "ami-0b0ea68c435eb488d",
 VpcSecurityGroupIds = new[] { group.Id },
 Tags = new InputMap<string>
 {
 { "Name", "web-server" },
 { "Environment", "dev" },
 },
 });
 }
}

// Infra.java
package myproject;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.aws.ec2.SecurityGroup;
import com.pulumi.aws.ec2.SecurityGroupArgs;
import com.pulumi.aws.ec2.Instance;
import com.pulumi.aws.ec2.InstanceArgs;
import com.pulumi.aws.ec2.inputs.SecurityGroupIngressArgs;
import java.util.List;
import java.util.Map;

public class Infra {
 public final SecurityGroup group;
 public final Instance server;

 public Infra() {
 this.group = new SecurityGroup("web-secgrp",
 SecurityGroupArgs.builder()
 .ingress(SecurityGroupIngressArgs.builder()
 .protocol("tcp")
 .fromPort(80)
 .toPort(80)
 .cidrBlocks("0.0.0.0/0")
 .build())
 .build());

 this.server = new Instance("web-server",
 InstanceArgs.builder()
 .instanceType("t2.micro")
 .ami("ami-0b0ea68c435eb488d")
 .vpcSecurityGroupIds(group.id().applyValue(List::of))
 .tags(Map.of("Name", "web-server", "Environment", "dev"))
 .build());
 }
}

Key detail: State property keys in the mock’s new_resource/newResource return value must be camelCase (cidrBlocks, not cidr_blocks), regardless of the language you’re writing in. This is how Pulumi serializes properties internally.

How do you run integration tests against real cloud resources?

Unit tests tell you whether your resource definitions are correct. Integration tests tell you whether your infrastructure actually works. Integration tests deploy real cloud resources, run assertions against them, and then destroy everything—giving you end-to-end confidence at the cost of actual cloud spend and time.

Using the Automation API

The Pulumi Automation API lets you drive pulumi up, pulumi destroy, and every other Pulumi CLI operation programmatically, from within a test. You define the stack inline (using a function that runs your Pulumi program) or point it at an existing project directory.

# test_integration.py
import pytest
import pulumi
import pulumi_aws as aws
from pulumi import automation as auto

def create_bucket_program():
 """The Pulumi program to deploy during the test."""
 bucket = aws.s3.Bucket("test-bucket",
 tags={"Environment": "test", "ManagedBy": "pulumi"},
 )
 pulumi.export("bucket_name", bucket.id)
 pulumi.export("bucket_arn", bucket.arn)

@pytest.fixture(scope="module")
def deployed_stack():
 """Deploy the stack for the test module and destroy it when done."""
 stack = auto.create_or_select_stack(
 stack_name="integration-test",
 project_name="bucket-test",
 program=create_bucket_program,
 )
 stack.set_config("aws:region", auto.ConfigValue(value="us-west-2"))

 # Install provider plugins
 stack.workspace.install_plugin("aws", "v7.34.0")

 # Deploy
 up_result = stack.up(on_output=print)

 yield up_result.outputs

 # Teardown: destroy everything after the test module completes
 stack.destroy(on_output=print)
 stack.workspace.remove_stack("integration-test")

def test_bucket_was_created(deployed_stack):
 """The S3 bucket should have been created with the expected name prefix."""
 bucket_name = deployed_stack["bucket_name"].value
 assert bucket_name.startswith("test-bucket"), f"Unexpected bucket name: {bucket_name}"

def test_bucket_has_tags(deployed_stack):
 """Verify the bucket was created with required tags (via AWS SDK)."""
 import boto3
 s3 = boto3.client("s3", region_name="us-west-2")
 bucket_name = deployed_stack["bucket_name"].value
 tagging = s3.get_bucket_tagging(Bucket=bucket_name)
 tag_map = {t["Key"]: t["Value"] for t in tagging["TagSet"]}
 assert tag_map.get("Environment") == "test"
 assert tag_map.get("ManagedBy") == "pulumi"

Run with: pytest test_integration.py -v

// integration.test.ts
import { LocalWorkspace, ConfigValue, UpResult } from "@pulumi/pulumi/automation";
import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";
import { expect } from "chai";

describe("S3 Bucket Integration", function () {
 this.timeout(300_000); // 5 min — real deploys take time

 let outputs: UpResult["outputs"];
 let stackName = "integration-test-ts";

 before(async function () {
 const stack = await LocalWorkspace.createOrSelectStack({
 stackName,
 projectName: "bucket-test",
 program: async () => {
 const bucket = new aws.s3.Bucket("test-bucket", {
 tags: { Environment: "test", ManagedBy: "pulumi" },
 });
 return {
 bucketName: bucket.id,
 bucketArn: bucket.arn,
 };
 },
 });

 await stack.setConfig("aws:region", { value: "us-west-2" });
 await stack.workspace.installPlugin("aws", "v7.34.0");

 const upResult = await stack.up({ onOutput: console.log });
 outputs = upResult.outputs;
 });

 after(async function () {
 const stack = await LocalWorkspace.selectStack({ stackName, projectName: "bucket-test", program: async () => {} });
 await stack.destroy({ onOutput: console.log });
 await stack.workspace.removeStack(stackName);
 });

 it("creates a bucket with the right name prefix", function () {
 const name = outputs["bucketName"].value as string;
 expect(name).to.match(/^test-bucket/);
 });
});

Run with: mocha -r ts-node/register integration.test.ts

// integration_test.go
package integration_test

import (
 "context"
 "fmt"
 "os"
 "testing"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/auto"
 "github.com/pulumi/pulumi/sdk/v3/go/auto/optup"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/stretchr/testify/assert"
)

func TestBucketIntegration(t *testing.T) {
 ctx := context.Background()

 stack, err := auto.NewStackInlineSource(ctx, "integration-test", "bucket-test",
 func(ctx *pulumi.Context) error {
 bucket, err := s3.NewBucket(ctx, "test-bucket", &s3.BucketArgs{
 Tags: pulumi.StringMap{
 "Environment": pulumi.String("test"),
 "ManagedBy": pulumi.String("pulumi"),
 },
 })
 if err != nil {
 return err
 }
 ctx.Export("bucketName", bucket.ID())
 return nil
 },
 )
 assert.NoError(t, err)

 _ = stack.SetConfig(ctx, "aws:region", auto.ConfigValue{Value: "us-west-2"})
 upResult, err := stack.Up(ctx, optup.ProgressStreams(os.Stdout))
 assert.NoError(t, err)

 t.Cleanup(func() {
 _, _ = stack.Destroy(ctx)
 _ = stack.Workspace().RemoveStack(ctx, "integration-test")
 })

 name := fmt.Sprintf("%v", upResult.Outputs["bucketName"].Value)
 assert.Contains(t, name, "test-bucket")
}

Run with: go test ./... -v -timeout 10m

// IntegrationTests.cs
using System;
using System.Threading.Tasks;
using Pulumi.Automation;
using Xunit;

public class IntegrationTests : IAsyncLifetime
{
 private WorkspaceStack _stack = null!;
 private UpResult _upResult = null!;

 public async Task InitializeAsync()
 {
 var program = PulumiFn.Create(() =>
 {
 var bucket = new Pulumi.Aws.S3.Bucket("test-bucket", new()
 {
 Tags = new() { ["Environment"] = "test", ["ManagedBy"] = "pulumi" },
 });
 return new System.Collections.Generic.Dictionary<string, object?>
 {
 ["bucketName"] = bucket.Id,
 };
 });

 _stack = await LocalWorkspace.CreateOrSelectStackAsync(new InlineProgramArgs(
 projectName: "bucket-test",
 stackName: "integration-test",
 program: program
 ));

 await _stack.SetConfigAsync("aws:region", new ConfigValue("us-west-2"));
 _upResult = await _stack.UpAsync(new UpOptions { OnStandardOutput = Console.WriteLine });
 }

 public async Task DisposeAsync()
 {
 await _stack.DestroyAsync(new DestroyOptions { OnStandardOutput = Console.WriteLine });
 await _stack.Workspace.RemoveStackAsync("integration-test");
 }

 [Fact]
 public void BucketNameHasExpectedPrefix()
 {
 var name = _upResult.Outputs["bucketName"].Value.ToString();
 Assert.StartsWith("test-bucket", name);
 }
}

Run with: dotnet test

// IntegrationTest.java
package myproject;

import com.pulumi.Context;
import com.pulumi.automation.LocalWorkspace;
import com.pulumi.automation.UpOptions;
import com.pulumi.automation.DestroyOptions;
import com.pulumi.automation.ConfigValue;
import com.pulumi.automation.WorkspaceStack;
import org.junit.jupiter.api.*;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.*;

@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class IntegrationTest {

 private static final String PROJECT_NAME = "bucket_test_project";
 private static final String STACK_NAME = "integration-test";

 private WorkspaceStack stack;
 private Map<String, com.pulumi.automation.OutputValue> outputs;

 @BeforeAll
 void setUp() throws Exception {
 stack = LocalWorkspace.createOrSelectStack(PROJECT_NAME, STACK_NAME, (Context ctx) -> {
 var bucket = new com.pulumi.aws.s3.Bucket("test-bucket",
 com.pulumi.aws.s3.BucketArgs.builder()
 .tags(Map.of("Environment", "test", "ManagedBy", "pulumi"))
 .build());
 ctx.export("bucketName", bucket.id());
 });

 stack.workspace().installPlugin("aws", "v7.34.0");
 stack.setConfig("aws:region", new ConfigValue("us-west-2"));

 var upResult = stack.up(UpOptions.builder().onStandardOutput(System.out::println).build());
 outputs = upResult.outputs();
 }

 @AfterAll
 void tearDown() throws Exception {
 if (stack != null) {
 stack.destroy(DestroyOptions.builder().onStandardOutput(System.out::println).build());
 stack.workspace().removeStack(STACK_NAME);
 }
 }

 @Test
 void bucketNameHasExpectedPrefix() {
 var name = (String) outputs.get("bucketName").value();
 assertTrue(name.startsWith("test-bucket"), "Unexpected bucket name: " + name);
 }
}

Run with: mvn test

Using the Go integration framework

If you prefer Go-based integration tests, Pulumi ships a purpose-built Go test framework at github.com/pulumi/pulumi/pkg/v3/testing/integration. It handles the full create → validate → destroy lifecycle for any Pulumi program (written in any language), with an ExtraRuntimeValidation hook for assertions against deployed state.

func TestS3Website(t *testing.T) {
 integration.ProgramTest(t, &integration.ProgramTestOptions{
 Dir: path.Join("..", "my-s3-website"),
 Config: map[string]string{"aws:region": "us-west-2"},
 ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
 url := stack.Outputs["websiteUrl"].(string)
 resp, _ := http.Get("http://" + url)
 assert.Equal(t, 200, resp.StatusCode)
 },
 })
}

Best practice: Always destroy test stacks in a finally block or test teardown fixture. Leaked integration test stacks are a common cause of unexpected cloud spend.

How does policy as code complement IaC testing?

Unit and integration tests validate what your infrastructure does. Policy as code enforces what your infrastructure must conform to—security standards, tagging conventions, compliance requirements—and it runs automatically on every pulumi preview and pulumi up.

Pulumi Policies are currently supported in Python and TypeScript. You write policies in the same language as the rest of your stack, without learning a DSL. A mandatory policy blocks a deployment entirely when violated; an advisory policy surfaces a warning but allows the deploy to proceed.

# __main__.py in your policy pack directory
from pulumi_policy import (
 EnforcementLevel,
 PolicyPack,
 ReportViolation,
 ResourceValidationArgs,
 ResourceValidationPolicy,
)

REQUIRED_TAGS = {"Name", "Environment", "Owner"}

def require_tags(args: ResourceValidationArgs, report_violation: ReportViolation):
 if args.resource_type.startswith("aws:"):
 tags = args.props.get("tags") or {}
 missing = REQUIRED_TAGS - set(tags.keys())
 if missing:
 report_violation(
 f"Resource is missing required tags: {', '.join(sorted(missing))}"
 )

def no_public_s3(args: ResourceValidationArgs, report_violation: ReportViolation):
 if args.resource_type == "aws:s3/bucket:Bucket":
 if args.props.get("acl") in ("public-read", "public-read-write"):
 report_violation("S3 buckets must not have a public ACL.")

PolicyPack(
 name="aws-security-baseline",
 policies=[
 ResourceValidationPolicy(
 name="require-tags",
 description="All AWS resources must have Name, Environment, and Owner tags.",
 enforcement_level=EnforcementLevel.MANDATORY,
 validate=require_tags,
 ),
 ResourceValidationPolicy(
 name="no-public-s3",
 description="S3 buckets must not be publicly accessible via ACL.",
 enforcement_level=EnforcementLevel.MANDATORY,
 validate=no_public_s3,
 ),
 ],
)

// index.ts in your policy pack directory
import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy";

new PolicyPack("aws-security-baseline", {
 policies: [
 {
 name: "require-tags",
 description: "All EC2 instances must have Name, Environment, and Owner tags.",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.ec2.Instance, (instance, args, reportViolation) => {
 const required = ["Name", "Environment", "Owner"];
 const tags = instance.tags || {};
 for (const tag of required) {
 if (!tags[tag]) {
 reportViolation(`Missing required tag: ${tag}`);
 }
 }
 }),
 },
 {
 name: "no-public-s3",
 description: "S3 buckets must not have a public ACL.",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => {
 if (bucket.acl === "public-read" || bucket.acl === "public-read-write") {
 reportViolation("S3 bucket must not have a public ACL.");
 }
 }),
 },
 ],
});

Running a policy pack

# Scaffold a new policy pack from a template
mkdir my-policies && cd my-policies
pulumi policy new aws-typescript

# Run policy checks against a stack
pulumi preview --policy-pack ./my-policies
pulumi up --policy-pack ./my-policies

# Or publish to Pulumi Cloud and enforce organization-wide
pulumi policy publish

As Joe Duffy, Pulumi CEO, puts it: “The smartest agent in the world still needs guardrails, audit trails, and policy enforcement to be trusted with production systems at scale.”

Policy checks complement unit tests: unit tests verify your intent (“I wrote tags into this resource”), while policies enforce your standards (“all resources must have these tags, regardless of what any one developer intended”). For complex compliance environments, publish policies to Pulumi Cloud and assign them via policy groups to enforce them across the stacks or accounts you choose—including organization-wide.

How is testing IaC in Pulumi different from Terraform?

The fundamental difference is language cohesion. With Pulumi, you write infrastructure, tests, and policies all in the same general-purpose language. With Terraform, your infrastructure is HCL but your tests must be something else—either Go (Terratest) or a constrained HCL DSL (terraform test).

	Pulumi	Terratest	`terraform test` (≥ 1.6)
Language for tests	Same language as infra (Python, TypeScript, Go, C#, Java); YAML programs tested via Automation API from any of those languages	Always Go (regardless of infra language)	HCL DSL (`.tftest.hcl`)
Unit tests with mocks	Yes — `pulumi.runtime.set_mocks()`, no cloud credentials needed	No — no mock layer for cloud providers	Limited — `mock_provider` (v1.7+) is HCL-declarative, no programmatic logic
Integration testing	Automation API (any language) or Go framework	Full-featured but Go-only	Yes, via `command = apply` run blocks
Policy / guardrails	Pulumi Policies: Python or TypeScript, runs at preview time	External tools (checkov, tfsec, OPA/Rego — separate toolchain)	Sentinel (enterprise) or external
Execution speed	Unit: milliseconds; integration: depends on cloud	Minutes (real deployments by default)	Plan-only: fast; apply: minutes
Learning curve	Use existing language skills and test frameworks	Requires Go knowledge and understanding of `go test`	Low if you know HCL; limited expressiveness

Terratest is battle-tested and widely adopted, particularly for Terraform module testing. It offers deep integrations with AWS, GCP, Azure, and Kubernetes. Its limitation is that it has no mock layer for cloud providers, so most test suites deploy real infrastructure, which makes them slower and costlier to run on every commit.

terraform test has improved significantly in Terraform 1.6 and 1.7. The command = plan mode avoids real deployments, and mock_provider enables basic unit-like testing. But expressiveness is bounded by HCL’s declarative model—no general-purpose programming, no third-party assertion libraries, and a fixed set of run/assert test constructs.

With Pulumi, there’s no context switch. The same developer who writes the infrastructure writes the tests in the same language, using the same IDE, the same debugger, and the same CI pipeline. That cohesion is a productivity multiplier.

How to set up IaC testing with Pulumi

Follow these steps to add a complete test suite to a Pulumi project. The examples use Python, but the pattern is identical in TypeScript, Go, C#, or Java.

Step 1: Install Pulumi and choose a language SDK

# Install Pulumi CLI
curl -fsSL https://get.pulumi.com | sh

# Create a new project (or use an existing one)
pulumi new aws-python --name my-project --stack dev

Step 2: Add a test framework

# Python
pip install pytest pytest-asyncio

# TypeScript
npm install --save-dev mocha ts-node @types/mocha chai @types/chai

# Go uses the built-in testing package — no additional dependencies

Step 3: Implement the Mocks class

Create mocks.py (or equivalent) with your Mocks subclass. Keep it simple for unit tests: return [args.name + "_id", args.inputs] from new_resource.

Step 4: Write unit tests

Create test_infra.py in your project root. Import mocks first, then your program. Write test functions that use @pulumi.runtime.test and pulumi.Output.all(...).apply(...) to assert resource properties.

Step 5: Add integration tests

Create test_integration.py with a pytest fixture that creates a stack, calls stack.up(), yields outputs for assertions, and calls stack.destroy() in teardown. Point this at a dedicated test cloud account.

Step 6: Create a policy pack

mkdir policies && cd policies
pulumi policy new aws-python
# Edit __main__.py to add your rules

Step 7: Wire it all into CI

# .github/workflows/test.yml (GitHub Actions excerpt)
jobs:
 unit-tests:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: pulumi/actions@v6
 - run: pip install -r requirements.txt
 - run: python -m pytest tests/unit/ -v

 integration-tests:
 runs-on: ubuntu-latest
 needs: unit-tests
 environment: staging
 steps:
 - uses: actions/checkout@v4
 - uses: pulumi/actions@v6
 - run: pip install -r requirements.txt
 - env:
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 run: python -m pytest tests/integration/ -v --timeout=300

Frequently asked questions

What is infrastructure as code testing?

Infrastructure as code testing is the practice of writing automated checks that validate your IaC programs before they deploy to production. Just like application testing, it includes unit tests (fast, no cloud credentials, mock the provider), integration tests (real cloud deployments in ephemeral environments), and policy checks (rules that enforce compliance standards on every preview and deploy).

Can you unit test IaC without deploying to the cloud?

Yes. Pulumi’s mocking framework intercepts all cloud API calls and replaces them with in-process stubs. Unit tests run entirely in memory—no cloud credentials, no network calls, no cost. A suite of 50 unit tests runs in under a second. This is the primary reason to write unit tests: fast, free, deterministic feedback on every commit.

Do you need real cloud credentials to test Pulumi?

Only for integration tests. Unit tests use Pulumi’s mock runtime and require no cloud credentials at all. Policy checks also run without credentials at pulumi preview time. Integration tests that use the Automation API do need credentials—they deploy real resources—so they’re typically run in a CI stage with access to a dedicated test account, not on every PR.

What’s the difference between unit and integration tests for IaC?

Unit tests validate resource properties, configurations, and relationships using mocked cloud providers—they run in milliseconds with no cloud credentials. Integration tests deploy actual resources to a real cloud environment, validate runtime behavior (does the endpoint respond? does the database accept connections?), and then destroy everything. Unit tests are cheap and frequent; integration tests are slower, cost real cloud spend, and run less often.

How do you mock cloud resources in Pulumi?

Implement pulumi.runtime.Mocks (Python) or call pulumi.runtime.setMocks() (TypeScript/JavaScript) before importing your Pulumi program. The new_resource method receives every resource registration and returns a fake [id, state] pair. The call method handles data-source lookups. Set mocks with pulumi.runtime.set_mocks(MyMocks()) and then import your program—the runtime will route all resource registrations through your mock instead of making real API calls.

What test frameworks work with Pulumi?

Any language-native test framework works. For Python: pytest (recommended) or unittest. For TypeScript/JavaScript: Mocha, Jest, or Vitest. For Go: the standard testing package. For C#: NUnit or xUnit. For Java: JUnit. The @pulumi/pulumi/automation SDK and pulumi.automation module integrate naturally with all of them. YAML is Pulumi’s declarative configuration language; because it has no control flow or executable statements, you cannot write test logic in YAML itself. YAML programs are tested by pointing the Automation API from Python, TypeScript, Go, C#, or Java at the project directory and asserting on the stack outputs.

How do you test Pulumi in CI/CD?

Split your test suite into two CI stages. Stage 1 (unit tests): runs on every PR commit, no cloud credentials required, fast. Stage 2 (integration tests): runs on merge to main or on a schedule, requires cloud credentials for a dedicated test account, uses the Automation API to deploy ephemeral stacks. The Pulumi GitHub Actions integration simplifies credential injection and stack management in both stages.

Is policy as code the same as testing?

Policy as code is complementary to—but distinct from—testing. Tests validate what your program produces (resource A has property B); policies enforce what your resources must conform to (all resources must have tag X). Policies run automatically at preview time and block non-compliant deployments at deploy time. Together, they create defense in depth: tests verify intent, policies enforce standards regardless of intent.

How is Pulumi testing different from Terratest?

Terratest typically deploys real infrastructure—it has no mock layer for cloud providers—so most test runs provision actual cloud resources, incurring real cost and taking 5–30 minutes. Tests must be written in Go regardless of what language your infrastructure uses. Pulumi’s unit testing framework runs in milliseconds with no cloud credentials, uses your existing language and test runner, and provides true mock isolation. For integration testing, Pulumi’s Automation API offers a similar model to Terratest but in any language.

How do you tear down resources after integration tests?

In Python pytest, use a module-scoped fixture that yields after stack.up() and calls stack.destroy() in the teardown. In TypeScript Mocha, call stack.destroy() in an after() hook. In Go, use defer stack.Destroy(ctx, ...). Always use a try/finally pattern (or equivalent in your language) so teardown runs even when a test assertion fails. Pulumi Cloud also has built-in stack TTL and auto-destroy features for extra safety.

Where to go next

Testing IaC is not an all-or-nothing investment. Start by adding a handful of unit tests to your most critical stack—the one that manages your production VPC, your IAM roles, or your data infrastructure. Pick three to five properties that, if wrong, would cause an outage or a security incident. Write tests for those. Run them in CI. Expand from there.

When you’re ready to go deeper:

Testing overview — Pulumi’s full testing documentation
Unit testing guide — detailed mock API reference with Go and C# examples
Automation API integration testing — advanced patterns (multi-stack tests, config injection, parallel test execution)
Policy as code authoring — writing and publishing Pulumi Policies
Pulumi vs. Terraform — a full comparison of the two platforms

If you’re migrating from Terraform, the Pulumi conversion tool translates your existing HCL to Python, TypeScript, Go, C#, and more, including your test infrastructure. Your Terratest or terraform test suites can be ported to Pulumi’s native test runner—using your existing language—as part of the migration.

Get started with Pulumi for free →

Five Stacks Before Lunch: The Parallel Coding Playbook for Pulumi

Engin Diri — Tue, 02 Jun 2026 00:00:00 +0000

AI coding has two shapes right now. One agent in a loop, sequential work, you babysitting the chat window. Call that 2x. Most teams live here. Five agents in worktrees, parallel work, fresh-context review on every change. Call that 10x. The trick: 2x is mostly prompting, 10x is mostly plumbing.

The parallel coding playbook is a five-pattern setup for running multiple AI coding agents at the same time without them stepping on each other: an issue used as the spec, a plan/build/validate loop, parallel git worktrees, fresh-session review, and a self-healing layer. The whole thing targets application code. The interesting question, and the one I keep ending up at, is what changes when the five agents are touching infrastructure.

2x is prompting, 10x is plumbing

2x is one human, one agent, one repo, one branch. The agent writes, you review, you tell it to try again, it tries again. The bottleneck is your attention. Whatever the agent’s raw throughput, your reading speed sets the ceiling.

10x moves you out of the per-change loop and into the issue loop. You write five issues with sharp acceptance criteria, send each one to its own agent in its own worktree, and let them plan, build, and validate end-to-end. You read five PRs at lunch instead of pair-programming on one all morning.

Concurrent isolation does the work. And isolation is mostly an infrastructure problem.

The five pillars

The five pillars, in one sentence each.

Issue is the spec. The GitHub issue carries the acceptance criteria. The pull request is the artifact that gets validated. Input and output of every implementation are versioned, scoped, and reviewable on their own.
Plan, build, validate. Three stages, three artifacts. A markdown plan you can read in thirty seconds. A build that produces a diff. A validate step that checks the diff against the spec.
Parallel worktrees. Each agent runs in its own git worktree so concurrent changes never trample each other. One repo, five working trees, five branches.
Fresh-session review. A different agent, a different conversation, no shared context, reads the output and judges it. The reviewer never sees the writer’s chat. An agent reviewing its own output in the same context is theater.
Self-healing layer. When the same issue keeps coming back, fix the system that allowed it. Update the rules, the skills, the AGENTS.md. The agent gets better; the bug class disappears.

The application-code version of this playbook leans on ports, node_modules, and databases to get isolation right. The infrastructure version has a different toolbox.

What changes when the agents are touching infrastructure

Walk the pillars again, this time with a Pulumi shop in mind.

Issue is the spec. For application code, the spec describes behavior. For infrastructure, the spec is a Pulumi component contract plus a Pulumi Policies excerpt. “The resulting bucket is private, lives in eu-west-1, has SSE-KMS, and is tagged owner=team-x.” That sentence compiles to a typed component signature and three policy assertions. The agent does not get to interpret “looks right.” The acceptance criteria are deterministic, which is the whole reason this works.

Plan, build, validate. Pulumi already ships the validate step. pulumi preview produces a deterministic, machine-readable diff a second reviewer can judge without the conversation that produced it. The plan is a markdown doc the agent writes before touching code. The build is pulumi up --target against a review stack scoped to the resources the issue covers. The validate step is the preview output plus the Pulumi Policies verdict.

Parallel worktrees. Worktrees alone are not enough. Two worktrees pointing at the same Pulumi stack will fight over state on the first concurrent up. The unit of isolation for infrastructure is the stack, not the worktree. Each worktree gets its own ephemeral review stack and its own ESC environment for credentials. State branches with the work, credentials branch with the work, and the cloud account does not see five agents elbowing each other.

Fresh-session review. The hardest part of the application-code version is keeping the reviewer cold. For infrastructure, the substrate hands you the cold context. The pulumi preview JSON has no memory of the prompt that produced it. A separate agent reading it has the same starting point a human reviewer has: a diff, a stack name, a policy report. Pulumi Neo reasons over the state graph directly, so the reviewer grounds every claim in what the change actually does, not what the writer says it does. Reviewer quality still depends on how well your policies cover the stack, but the cold-context part comes built in.

Self-healing layer. Most Pulumi Policies rule messages today read like assertions. “S3 bucket has no encryption.” A self-healing layer needs them to read like instructions. “S3 bucket has no encryption. Set serverSideEncryptionConfiguration with SSE-KMS to fix.” That single rewrite is the difference between an agent flailing and an agent fixing the violation on the first try. When the same rule keeps tripping, the fix is upstream of the next pull request: in the rules, in the skills, in the policy itself.

The five catches, infra edition

Every parallelism story has a catch list. The application-code version lists port conflicts, node_modules sprawl, database conflicts, token blowouts, and PR pile-up. The infrastructure equivalents map almost one to one.

Port conflicts become stack-name collisions. Two agents naming their stack dev and racing each other into Pulumi Cloud. The fix is the same hash-the-path trick the app-code playbook uses: derive the stack name from pulumi.getProject() plus a hash of the worktree path. Resource names follow the same pattern. Collisions go away.

node_modules sprawl becomes provider plugin sprawl, mostly already solved. Three worktrees each pulling their own copy of pulumi-aws would add up fast, except Pulumi already shares plugins through a single cache at ~/.pulumi/plugins. Identical provider versions are reused across worktrees automatically. Per-worktree language SDKs (node_modules, venv) still need the usual care, but the provider layer is free.

Database conflicts become state conflicts. Two agents racing each other into pulumi up on the same stack is the same hazard as two agents writing to the same migrated database. The app-code playbook reaches for Neon branches or per-worktree SQLite files to isolate state. The infra answer is simpler: each worktree gets its own review stack. State branches with the work, by construction.

Token blowouts become cloud spend per ephemeral stack. The cost vector flips. For app code, the worry is LLM bills. For infrastructure, the worry is what your five agents just spun up in five review stacks. The mitigations are boring and they work. Use TTL stacks to tear review stacks down on a schedule. Avoid retainOnDelete on review-stack resources so the teardown actually frees them. Cap retries per spec. Watch the bill.

PR pile-up is the same problem. Five reviewed diffs are still five things waiting on the merge queue. The infra-flavored mitigations: stack-scoped reviewers (the human who owns the stack approves the change to it), the Pulumi Cloud audit log for grouping by stack and time, and auto-merge for the narrow class of changes where the preview diff is clean and every policy passes. That last one is where most of the throughput hides.

Where to start, this afternoon

Three steps, in order, on a stack with a small blast radius.

Write an AGENTS.md for the repo. Five paragraphs is enough. The component library, the stack naming convention, the policy rules, the review-stack TTL, and the one thing in this repo that bites every newcomer. Neo reads AGENTS.md natively, as do most coding agents. This file is the spec for how the agent should behave even before you write a spec for what it should build.
Cut a 24-hour review-stack TTL. Spin up a review stack on PR open, tear it down on PR close or after 24 hours, whichever comes first. This is the gate that turns “ephemeral” from a slogan into a line item that does not appear on next month’s bill.
Run three issues in parallel. Pick three open issues that touch unrelated resources. Spin up three worktrees, three review stacks, three ESC environments. Let each agent run end-to-end against its own stack. Then have a fourth agent read each preview JSON cold and produce a one-paragraph review. Read three PRs plus the reviewer’s summary at lunch.

That last step is the measurement. The first time you run it, half of the changes will fail validation. The second time, fewer. By the third time you will know whether your spec quality, your policies, and your stack hygiene are good enough to scale this to five, then ten, then to every issue tagged infra:fix.

If three issues finish cleanly, you have the substrate. If they do not, the gap is almost always in the spec or the policy rules, not the agent. Fix the spec, tighten the rule, run it again.

Five stacks before lunch

10x is five concurrent agents, working from five issues, against five stacks, behind five fresh-session reviews. The substrate is already there. Stacks isolate state. ESC isolates credentials. pulumi preview is the deterministic artifact a fresh reviewer can read cold. Pulumi Policies is the self-healing layer when you write the rule messages as instructions.

The remaining work is small and mostly wiring. Write the AGENTS.md. Cut the TTL. Pick three issues that touch unrelated resources. Read three PRs at lunch. Five stacks before the room empties out is a realistic Monday.

See how Pulumi Neo runs your stacks

The Dark Factory Pattern for Infrastructure: Running Pulumi Lights-Out

Engin Diri — Tue, 05 May 2026 00:00:00 +0000

The original dark factory was Fanuc’s robotics plant in Oshino, Japan, where the lights are off because nobody is on the floor. Robots build robots. Parts move through the line for weeks at a time without a person walking past them.

The same pattern is now showing up in software. Three engineers at StrongDM shipped roughly 32,000 lines of production code without writing or reviewing any of it. Stripe’s “Minions” agent system merges over a thousand pull requests every week. In January, Dan Shapiro of Glowforge published a five-level autonomy ladder that landed cleanly enough to become the shorthand most people now use, and BCG put out a piece calling it the dark software factory.

Almost every public writeup so far is about application code. The harder question is what this looks like for infrastructure.

What a dark factory actually is

Shapiro’s ladder is the cleanest framing I’ve seen. He borrows it from the SAE’s self-driving levels, and it fits surprisingly well:

Level	What it is	Driving analogy
0	Spicy autocomplete	Stick shift; you do everything.
1	Coding intern (boilerplate)	Cruise control.
2	Junior developer (interactive pair)	One hand on the wheel.
3	AI writes the majority; you review every PR	Eyes still on the road.
4	Spec-driven; agent runs unattended for hours; you review later	Sleeping at the wheel, you can still wake up.
5	Dark factory; no human review of code before production	No steering wheel at all.

Most teams are at level 2 or 3. A few of the more aggressive ones are at 4. Level 5 is the experiment. Most teams won’t get there safely, and probably shouldn’t try to. The interesting design question is what has to be true for level 5 to be safe at all, and that question gets sharper when the thing being shipped is infrastructure.

A dark factory is not a coding harness. A harness is the framework an agent runs inside; the dark factory is the surrounding system that makes a harness’s output mergeable without a human reading the diff. Copilot and Cursor sit at the other end: interactive, the human stays in the loop on every keystroke. The dark factory takes the human out of the per-change loop entirely and puts them at the top, writing the spec and the acceptance criteria.

The wall between generator and validator

Strip the dark factory down to its layers and there are four of them.

flowchart LR
A[Inputs<br/>Humans] --> B[Code Generation<br/>Autonomous]
B --> C[Validation<br/>Autonomous, isolated]
C -->|pass| D[Merge & Deploy<br/>Autonomous + existing CI/CD]
C -->|fail| B
A -.->|holdout scenarios<br/>generator never sees these| C

The single most important rule is that Code Generation and Validation must be completely isolated. The generator never sees the acceptance scenarios. A separate evaluator does, and it judges the generator’s output against scenarios the generator could not have memorized.

The reason is sycophancy. LLMs are too eager to agree with their own prior turns and too willing to declare victory on something they just produced. Without isolation, the same model that wrote the change is the one telling you it’s fine. The practical concern is direct: a test stored in the same codebase as the implementation will get lazily rewritten to match the code, not the other way around. It isn’t malice; it’s the agent doing exactly what it was asked, badly. The wall is what stops that.

StrongDM’s pattern for this is holdout scenarios: plain-English BDD acceptance tests stored where the generator cannot reach them. Each scenario runs three times against an ephemeral deployment, two of three must pass, and the overall pass rate has to clear 90% before the change moves forward. If the generator fails, it gets a one-line failure message (“SQL Injection Detection failed: endpoint returned 500”), not the scenario text. It cannot game the test.

Without that wall, you don’t have a quality gate. You have theater.

Why infrastructure is the harder version

Application code factories can lean on tests, linters, and type checkers. Infrastructure adds blast radius, drift, secrets, irreversible actions, and multi-region state. A code dark factory shipping a broken UI causes a bad user experience. An infrastructure dark factory shipping a broken IAM policy ends in a postmortem.

A few things make this manageable on Pulumi specifically.

The orchestrator does not need to be invented. The Pulumi Automation API is the engine as an SDK in Python, TypeScript, Go, .NET, Java, or YAML, which is the same surface a dark factory orchestrator runs on. Credentials don’t have to be long-lived: ESC and OIDC issue short-lived ones per run, so the agent never sees a static secret.

Policy doesn’t have to be probabilistic: CrossGuard enforces deterministic rules at preview time. Execution doesn’t have to happen on a laptop: Pulumi Cloud Deployments runs pulumi up inside a governed runner with audit logs and approval rules already wired. And the reasoning layer doesn’t have to start from scratch: Pulumi Neo is grounded in your state graph and ships with three modes (Auto, Balanced, Review) that line up cleanly with Shapiro’s levels 5, 4, and 3.

That doesn’t make Pulumi a dark factory by itself. It means the parts that an application-code factory has to build from scratch are pieces a Pulumi shop already has: a credential broker, a policy engine, a governed runner, a state-aware reasoning layer, an audit trail.

And one more piece nobody talks about: pulumi preview produces a clean, deterministic validation artifact, and CrossGuard evaluates that artifact without ever seeing the conversation that produced the program. That’s the same context-free judgment the holdout pattern depends on, applied at the policy layer instead of the acceptance-test layer. For infrastructure, half the wall is already built.

The interesting work is the part that nobody ships in a box.

The interesting work

What no platform ships for you is the wall: the holdout scenarios for infrastructure, the isolated evaluator that runs them, and the agreement on which stacks are even allowed to run lights-out.

The happy-path orchestrator is small. It pulls a spec, runs preview, hands the preview to an isolated evaluator (with its own credentials and its own access to the cloud, no access to the generator’s prompt or output), and branches on the verdict. Auto mode runs up immediately. Balanced mode submits a deployment that requires approval. Review mode opens a PR for a human. Every branch records a stack version traceable in the audit log. Retries, observability, secret rotation, and the rest of the production-grade plumbing add up to real code, but the shape is small.

The wall is the part that takes a week to get right. You write five plain-English scenarios for one stack (“after pulumi up, the bucket is private, has SSE-KMS, lives in eu-west-1, and is tagged owner=team-x”) and a janky evaluator that runs preview and up against an ephemeral copy, queries the cloud, and asks a separate model whether the resulting state satisfies the scenario. Triple-run, 90% pass gate. Then you watch it for a few weeks before you let anything auto-apply.

A four-phase rollout

This is the same path the application-code factories walked, with the gates tightened.

Phase 1: better context, this afternoon

Write an AGENTS.md for your most active stack repo. Pulumi Neo reads it natively, as do most coding agents. While you’re there, look at your CrossGuard rules and rewrite the error messages as instructions. Not “S3 bucket has no encryption” but “S3 bucket has no encryption. Set serverSideEncryptionConfiguration with SSE-KMS to fix.” That single change is the difference between an agent flailing and an agent fixing the policy violation on the first try. Wire pulumi preview as a build-before-push gate so PRs don’t show up just to fail CI.

Phase 2: spec-driven with holdouts, this week

Pick one stack with a small blast radius. A review-stack lifecycle is ideal. Write five plain-English holdout scenarios for it and the janky evaluator above. Humans still approve every PR. Don’t auto-merge yet. You’re earning the data, not declaring trust.

Phase 3: take the human out of the merge

Only after the three measurable gates hold over twenty PRs (scenario pass rate above 90%, false positive rate below 5%, human override rate below 10%) flip auto-apply on for that one stack. Add a weekly drift sweep that goes through the same scenario gate as everything else.

Phase 4: lights out

Expand the auto-apply flag to every stack with strong scenario numbers. Wire your issue tracker so tickets tagged infra:fix flow through the pipeline. Mock the cloud APIs that are slow or flaky enough to make scenario evaluation expensive. At this point the orchestrator is configuration, not architecture.

What could go wrong

None of these have clean fixes. The mitigations below reduce risk; they don’t eliminate it. Any team running level 5 should expect to eat one or two of these in the first year.

The validator approves a bad change. This is the obvious one. The standard mitigation is layered: triple-run each scenario with a 2-of-3 threshold, a 90% gate over the run set, a human audit of the first fifty auto-applied changes, and your existing policies still run after the validator says yes.

The agent gets a destroy permission it shouldn’t have. There’s a class of operations that should not sit in the autonomous loop yet: dropping a database, deleting a hosted zone, rotating a root key, anything that crosses a regulated data boundary. Scope what each agent identity can do at the credential layer, require human approval for anything destructive, and start every stack at Review mode. Tag changes, security-group adjustments, and instance resizes can run autonomously today. Release-branch cuts and config promotions can probably run by next quarter. The destructive class earns its way in over months.

You need all three of those layers. Approvals without policy means anything a human approves in a hurry ships. Policy without approvals means a sufficiently clever spec eventually finds the gap. Both without a human kill switch means an incident at 3 a.m. has nobody to escalate to.

Costs blow up. Cap retries at three per spec, alert on token spend per run, and remember that StrongDM reported roughly $1,000 per day per engineer-equivalent. That’s still cheaper than a salary, but only if you put the cap in place before you find out.

Where to start

Most of what a dark factory needs already exists in any reasonably mature platform. Whatever you have for state, policy, credentials, audit, and a deployment runner is the substrate. The interesting work is not building the factory. It’s the wall: the holdout scenarios that make the gap between “the model says it’s fine” and “the system is actually fine” mean something.

For most teams, Phase 1 alone is the win. Full Level 5 may stay out of reach indefinitely, and that’s fine. The path itself forces useful work: clearer specs, named bottlenecks, the deterministic gates humans had been running in their heads.

Write an AGENTS.md and five holdout scenarios for one stack this week. That’s enough to get a real signal on whether the pattern fits your team. The rest of the path is the same problem the application-code factories have already worked through, with the gates set tighter.

Treating Prompts Like Code: A Content Engineer's AI Workflow

Cam Soper — Mon, 09 Mar 2026 00:00:00 +0000

Pulumi has a lot of engineers. It has marketers, solution architects, developer advocates. Everyone has something to contribute to docs and blog posts — domain expertise, hard-won lessons, real-world examples. What they don’t all have is familiarity with our Hugo setup, our style guide, our metadata conventions, or where a new document is supposed to live in the navigation tree. I joined Pulumi in July 2025 as a Senior Technical Content Engineer. A few weeks in, my sole teammate departed. The docs practice was now, functionally, me.

The problem was clear enough: how do you take one docs engineer’s accumulated knowledge and make it available to everyone who needs it, without that engineer becoming a bottleneck?

I started packaging it. Here’s what that looked like in practice.

The real problem AI solves

Everyone talks about AI making you faster. That’s not wrong, but it’s not the most interesting part — at least not for me.

The most interesting part is what it does to the starting problem. I have an ADHD brain (not formally diagnosed, but with enough self-recognition to know what’s going on). I know what that means for my relationship with most tasks: I can see the problem, I understand it, I want to fix it, and then the sheer weight of starting crushes me flat.

When I’m stuck on a task, the issue is almost never that I don’t know what to do. It’s that my brain is trying to hold the entire finished product in working memory while simultaneously producing the first step. That’s an enormous cognitive tax, and for an ADHD brain it’s often insurmountable.

Talking through a problem conversationally is a completely different cognitive load. I can tell Claude “here’s the issue, here’s what I’m trying to accomplish, here’s what’s weird about it,” and suddenly I’m not staring at a blank page anymore. I’m in a conversation. The scaffold exists. I can build on it.

That dynamic isn’t new for me. In a previous role writing training modules at Microsoft, I did some of my best work, not because the work was easy, but because I had a collaborator. A friend to think out loud with. Someone to say “okay, so what are we actually trying to say here?” That conversational scaffolding was the difference between spinning and shipping.

In my current role as a team of one, AI turned out to be that collaborator.

This isn’t really a productivity story. It’s closer to a cognitive accommodation story. And I’d bet a lot of people — diagnosed or not — will recognize what I’m describing.

Treating prompts like code

If conversational scaffolding could lower my own activation energy, the next question was obvious: could I build that for anyone who needed it? I knew I wanted to use AI to solve this problem, but I didn’t want to just write a bunch of one-off prompts. That would be a maintenance nightmare, and it wouldn’t scale beyond me. I needed a system. Claude Code calls these reusable prompts skills — other platforms have the same idea under names like plugins or extensions. My first real experiment was /docs-review — a reusable prompt that would run my writing through a consistent set of criteria before I committed it. Nothing fancy. I just wanted a reliable bar that didn’t depend on my mood or how much coffee I’d had.

Then it occurred to me: every PR to our docs repo should get this automatically. So I wired it into our CI/CD pipeline. Meagan, my manager, loved it — and after a few weeks, she noticed that PR quality had improved dramatically. On almost every PR, contributors were now spontaneously pushing an “Addressing feedback” commit right after the automated review posts — catching and fixing issues before I ever saw the PR.

That’s when something clicked: I wasn’t writing prompts anymore. I was writing modules — reusable, composable pieces of my own expertise.

The insight was straightforward, but it changed how I thought about the whole system: if multiple skills need the same context — our style guide, our review criteria, our content standards — that context should live in one place and get consumed by everything that needs it. Just like a shared library. Just like any decent software project.

I created a REVIEW-CRITERIA.md file as the single source of truth for what a “good” docs PR review looks like at Pulumi. Every skill that does any kind of review pulls from it. Change it once, and everything gets smarter at once. Likewise with our style guide, our Hugo conventions, our navigation structure. All of that lives in central reference files that any skill can pull from. If something changes, I change it in one place and all the skills get the update.

This also matters for token efficiency — which sounds like a nerdy footnote but isn’t, especially when automated reviews are running on every PR. Duplicating context across skills bloats token usage fast. Modularizing keeps it lean. Your CI/CD pipeline doesn’t care about elegance, but it definitely cares about cost.

The mental model I kept coming back to: Don’t Repeat Yourself. It’s the same principle that makes good software maintainable. It turns out it makes good AI workflows maintainable too.

The skill catalog

From there, the system grew organically. Whenever I found myself doing something more than once, I asked: “Can I turn this into a skill?” Here’s a sampling of what that produced:

/fix-issue — takes a GitHub issue and recommends a concrete plan of attack, so I go from “here’s a ticket” to “here’s what I’m doing” without the spinning-up tax.

/shipit — runs pre-commit checks, writes a focused commit message, and drafts a PR description.

/pr-review — full doc review on a PR branch: style guide, code examples, screenshots, optional test deployment, then an Approve/Merge/Request Changes dialog with a drafted comment.

/slack-to-issue — converts #docs Slack conversations into properly formed GitHub issues. Slack is where decisions happen; issues are where work gets tracked.

/glow-up — runs an older doc through the modern style guide and flags outdated screenshots, for digging out of accumulated technical debt.

/new-doc and /new-blog-post — guide anyone through adding a new document or blog post with the right location, metadata, and navigation wiring. Engineers, marketers, whoever. The barrier to contributing just dropped significantly.

/docs-tools — helps other repo users discover that any of this exists. Discoverability is a real problem with internal tooling.

Slack’s built-in Claude integration isn’t the same Claude running your Claude Code workflows — they don’t share context or custom instructions. If you want consistent criteria across both surfaces, you need to bring your own backend. That’s exactly what /slack-to-issue handles.

Other people started contributing skills to the repo — not because I asked, but because the pattern was legible enough to extend. Someone built a skill for SEO analysis. Marketing added their own review criteria. Engineers contributed workflows I never would have thought to build.

The thing I’d built as a personal survival tool had become a shared platform. That happened because I treated the prompts like code: modular, reusable, documented, open for contribution.

Honest limitations

It’s not a replacement for human judgment. These are probabilistic tools — they’re right most of the time, not all of the time. /pr-review doesn’t approve PRs autonomously. It highlights things and then asks me, the human, to read them and make the call. The AI does the first pass; I do the last one. That’s not a workaround for a limitation — that’s the design.

The system isn’t finished, either. It’s probably never finished. I’m still tweaking review criteria, still finding edge cases where a skill produces something weird, still adding new tools as new pain points emerge. Treating prompts like code means treating them like software: you ship, you iterate, you maintain. There’s no version 1.0 and done.

And the ADHD angle is real but it’s not magic. There are still days where the paralysis wins. AI lowers the activation energy for starting; it doesn’t eliminate it. I’m still the one who has to show up. I suppose I could automate that too, but then we’d be in a whole different kind of dystopia.

Know your models and their costs. At Pulumi we primarily use Claude, and I work in Claude Code; for most tasks I reach for Sonnet rather than Opus. Opus is excellent, but it’s significantly more expensive, and well-crafted instructions to Sonnet handle the vast majority of my work just as effectively.

Treat it like a coworker. Don’t just issue commands and wait for output. Ask what it thinks. Push back when it’s wrong. Explain your reasoning. The more you engage conversationally, the better the results tend to be. That extends to alignment, too — before diving into a complex task, talk through the approach first. A few minutes of alignment up front beats iterating on a misunderstood spec. I’ve gone as far as adding personal instructions to my config — things like playing along when I’m pretending to be Captain Picard, or using colorful language when the context calls for it. (Yes, those are literal config settings.) That sounds frivolous, but it isn’t: a tool you actually enjoy using is a tool you’ll reach for instead of avoid.

Modularize your workflow. Don’t write one giant monolithic prompt that tries to do everything. Break it into focused skills that do one thing well and share common context through a central reference file. Easier to maintain, easier to debug, cheaper to run.

Version control your prompts. Your skills are code. Treat them like code. Commit them, review them, iterate on them. If a skill starts producing weird output after a tweak, you’ll want to know what changed.

Think about token burn rate. This matters most when running automation in CI/CD. Keep your skills focused — a skill that checks style doesn’t need to load your Hugo navigation conventions. The model only reads what you give it, so give it only what it needs.

Not everything needs to be a prompt. This one is underappreciated: skills can include scripts, and that’s often the right call. When my team moves a doc in the repo, it needs to happen via git mv to preserve history, and we need to add a redirect alias to the front matter to prevent 404s and protect SEO. That’s not something I want an AI to reason through from scratch every time — it’s a solved problem. So it’s a script. The skill just knows the script exists and what it does. Claude orchestrates; the script executes. That’s a cleaner, more reliable division of labor than asking an LLM to reinvent the wheel on every run.

Not everything needs to be generative. Corollary to the last point: if you need deterministic output, don’t use probabilistic tools. We have a skill that generates the meta image for blog posts — procedurally, not generatively. No AI-generated imagery. We have a brand to protect, and “let the AI vibe it out” isn’t a content strategy. The skill follows our visual standards programmatically and produces something consistent every time. Know what you’re automating and why.

What’s next

The next frontier is bringing some of this tooling to the less technical members of the team — marketing, in particular. The skills I’ve built assume a certain comfort level with terminals and repos. That’s fine for engineers. It’s a barrier for everyone else. A friendly interface would lower that bar significantly — that’s the direction I’m currently exploring.

If you’re a technical writer, a developer advocate, or a solo practitioner figuring out how AI fits into your workflow, the approach described here is a solid starting point. The tools matter, but the mental model matters more: treat your prompts like code. Make them reusable. Document them. Share them.

Our docs repo is public, so the skills are there for anyone who wants them. If you’re building something similar, steal freely — or contribute back.

The blank page is still there. It’s just a lot less intimidating when you’ve got a good collaborator and a solid set of tools.

See our docs skills for inspiration

GitOps Best Practices I Wish I Had Known Before

Engin Diri — Thu, 19 Feb 2026 00:00:00 +0000

Getting started with GitOps can feel like trying to herd cats through a YAML factory while the factory is on fire. It’s one of those things that seems like it ought to be simple (just use Git!), but in practice is much more complex — and you may not realize how much more complex until you’re weeks or more into a project. After years of running GitOps workflows in production across dozens of clusters, I’ve collected a list of best practices that I’m hoping can save you from having to make many of the mistakes I’ve made. Think of it as the GitOps cheat sheet I wish I’d had from Day 1.

If you’re not familiar with the formal definition, the OpenGitOps project distills it into four principles:

Declarative desired state
Versioned and immutable storage
Automatic pulling
Continuous reconciliation

But those principles only define the what. This post is also about the how — the practical lessons that can make or break a GitOps implementation.

In this post, I’ll walk you through the GitOps best practices I’ve picked up from production experience, community talks, and more than a few late-night incident calls. Whether you’re just getting started with GitOps or looking to level up, these tips should help you avoid the potholes.

The GitOps mindset in a nutshell.

1. Git is your single source of truth (no, really)

This is the bedrock that the principles of GitOps are built on: every piece of your environment’s desired state lives in a Git repository. No exceptions. No “I’ll just fix it real quick with kubectl edit.” No “let me patch this configmap by hand because the PR process takes too long.”

The moment you make a manual change to your cluster, you’ve created drift between what Git says the world should look like and what it actually looks like. Drift will quietly wreck your GitOps workflows if you let it.

Put everything in Git: Kubernetes manifests, Helm values, Kustomize overlays, policy rules, even your GitOps tool configuration itself.
No manual kubectl edits. If it’s not in Git, it doesn’t exist. Period. Train your team to treat direct cluster changes like touching a hot stove.
You get an audit trail for free. Git gives you a complete history of who changed what, when, and why. That’s your compliance audit trail baked right in.

Pro Tip: Enable branch protection rules on your GitOps repos from Day 1. This prevents anyone from pushing directly to main and bypassing the review process. Future you will thank present you during the next audit.

2. Declarative over imperative, always

If you’re still running sequences of kubectl create, kubectl patch, and kubectl delete commands to manage your cluster, you’re not really doing GitOps yet. Declarative means you define what you want the end state to look like, not the step-by-step instructions to get there.

Think of it like ordering at a restaurant. Imperative: “Go to the kitchen, grab flour, knead dough, preheat the oven to 200 degrees, shape the pizza, add sauce…” Declarative: “One margherita pizza, please.” Let the system figure out how to make it happen.

Your manifests describe the end result. The GitOps operator reconciles reality to match.
It’s idempotent by design. Apply the same manifest ten times, get the same result. No side effects, no surprises.
Rollbacks are easier: just revert a commit. The operator sees the previous desired state and reconciles. A caveat though: this only works for stateless resources; database schema migrations, CRD version changes, persistent volume modifications, and rotated secrets don’t always revert cleanly by rolling back to a previous commit. Be sure to plan rollbacks involving stateful resources carefully.

Pro Tip: If you find yourself writing shell scripts that run a sequence of kubectl commands, stop and ask yourself: “Can I express this as a declarative manifest instead?” Nine times out of ten, the answer is yes.

3. Pull-based deployments are the way

Traditional CI/CD is push-based: your pipeline builds an artifact and then pushes it to the cluster. GitOps flips this. An agent running inside your cluster continuously polls a Git repository and pulls changes when it detects them.

Why does this matter? With push-based deployments, your CI system needs credentials to access your cluster. That’s a wide attack surface. With pull-based, the agent already lives inside the cluster and only needs read access to your Git repo.

Tools like ArgoCD and FluxCD run controllers inside your cluster that watch your Git repos.
Your CI pipeline never needs kubeconfig access. The agent handles deployment.
The agent doesn’t just deploy once; it continuously ensures the cluster matches the desired state in Git.

Pro Tip: You may be tempted to set your reconciliation interval to something aggressive like 1 minute so you always know exactly which version is deployed. That works for a while, but at scale (200+ applications polling every minute) you can blow through GitHub’s API rate limits (5,000 requests/hour for authenticated users) and put real pressure on the Kubernetes API server.

A better approach: set up webhook receivers. Both ArgoCD and FluxCD support incoming webhooks from GitHub, GitLab, and Bitbucket. Your Git provider pings the GitOps controller on every push, so reconciliation kicks off in seconds instead of waiting for the next poll. That alone kills most of the API rate limit pain at scale. You can then relax polling to a 5 or 10 minute fallback for the rare case where a webhook doesn’t fire.

4. Separate app code from deployment config (when it hurts not to)

Let me be upfront: if you’re a small team with one or two services, keeping your Kubernetes manifests next to your application source code in the same repo is fine. A monorepo is simpler to manage and one less thing to automate. Don’t split repos just because a blog post told you to.

That said, ArgoCD’s official best practices call separate repos “highly recommended,” and there are real reasons for that once you grow. Application code and deployment configuration have different lifecycles. You might bump resource limits in a Helm values file without touching a single line of app code. Or you might refactor your entire codebase without changing any deployment parameters. In a shared repo, every config tweak triggers your full CI pipeline, and ArgoCD invalidates the manifest cache for all applications in the repo on every commit, not just the ones that changed. That cache invalidation alone can become a performance problem with dozens of apps in one repo.

When config tweaks start blocking app code from getting through CI, it may be time to split.
When different teams need different access levels (not everyone who pushes app code should modify production deployment config), it’s time to split.
When you’re running microservices with independent release cadences that step on each other, it’s time to split.

If none of those apply yet, don’t split. You’ll know when the pain arrives.

Pro Tip: When you do split, a common pattern is two repos: one for application source code, one for deployment configuration (Helm charts, Kustomize overlays, environment-specific values). Some larger orgs go further with a third repo for environment overrides, following the fleet repo model from the FluxCD maintainers. Either way, pair it with image update automation (Flux Image Automation Controller or ArgoCD Image Updater) so image tag bumps across repos don’t turn into manual toil.

5. Use directories, not branches, for environments

This is the number one anti-pattern I encounter in the wild. Teams create a dev branch, a staging branch, and a prod branch, then cherry-pick or merge between them for promotions. It sounds logical, but it’s a trap. Branches diverge, cherry-picks get missed, configs meant for dev sneak into staging or prod, and before you know it, your environments have drifted, and you can’t tell what’s actually different between them.

A folder structure like environments/dev/, environments/staging/, environments/prod/ makes differences visible in a single diff command.
Moving a change from dev to prod is just copying or updating files across directories, reviewed via a standard pull request.
You’ll never accidentally skip a commit or introduce an environment-specific change where it doesn’t belong. No more cherry-pick roulette.

One thing to note: the commonly used rendered manifests pattern actually does use per-environment branches, but not in a human-managed way. In that pattern, config changes are still committed to main (using named directories as above), and an automated CI process pushes environment YAML to read-only environment branches, which ArgoCD reads from. If you want to try it, ArgoCD’s built-in source hydrator and Kargo Render can automate the rendering for you.

To keep things DRY, you can look to a tool like Kustomize, which lets you use a shared base environment and then apply per-environment patches. Your base directory holds the common manifests, and each environment directory contains only the differences. For example:

gitops-repo/
├── base/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
├── environments/
│ ├── dev/
│ │ ├── kustomization.yaml # replicas: 1, limits: 256Mi
│ │ └── patches/
│ ├── staging/
│ │ ├── kustomization.yaml # replicas: 2, limits: 512Mi
│ │ └── patches/
│ └── prod/
│ ├── kustomization.yaml # replicas: 3, limits: 1Gi
│ └── patches/

This minimizes duplication while keeping environment boundaries clear.

Pro Tip: If you’re working at the IaC layer, Pulumi stack configuration solves a similar problem: each stack (dev, staging, prod) has its own config file with environment-specific values, while the program logic stays shared. It’s the same principle of “one source, per-environment overrides” applied above the Kubernetes manifest level.

6. Validate before you merge

Catching errors after they’ve been applied to your cluster is expensive. Catching them in CI before the merge is cheap. Shift left as hard as you can.

Your GitOps CI pipeline should validate everything it can before the change reaches your cluster: YAML syntax, Kubernetes schema validation, policy compliance, dry-run rendering.

Tools like yamllint and kubeconform catch syntax errors and schema violations before they become runtime failures. (Note: kubeval, which you’ll see in older guides, is no longer maintained — its own README points to kubeconform as the replacement.)
Run helm template or kustomize build in CI to verify that your templates render without errors.
Use OPA/Conftest or Kyverno to enforce organizational policies (no privileged containers, required labels, resource limits set) before merge.

Here’s a minimal GitHub Actions workflow that validates your GitOps manifests on every PR:

name: Validate GitOps Manifests
on:
 pull_request:
 paths: ["environments/**"]

jobs:
 validate:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - name: Validate YAML schemas
 run: kubeconform --strict environments/**/*.yaml
 - name: Build and verify Kustomize output
 run: |
 for env in environments/*/; do
 kustomize build "$env" > /dev/null
 done
 - name: Diff against live cluster
 run: |
 kustomize build environments/staging/ | \
 kubectl diff -f - || true

Pro Tip: Add a kubectl diff step in your CI pipeline that shows exactly what would change in the cluster if the PR were merged. This gives reviewers a concrete view of the impact, not just the YAML diff.

7. Tag with commit SHAs, not “latest”

Using latest as your image tag in a GitOps workflow is playing fast and loose with your deployments. You have no idea what version is actually running, you can’t reliably roll back, and your Git history becomes meaningless because the same manifest could deploy completely different code depending on when it was synced.

Tag your container images with the Git commit SHA that produced them. This creates a direct link between your source code, your build, and your deployment.
Once a tag is pushed, it should never be overwritten. No re-tagging, no “oops let me push a fix to the same tag.”
Need to roll back? Just revert the commit that updated the image tag. The previous SHA still points to the exact same image it always did.

Pro Tip: Set up an admission controller or CI policy that rejects any manifest using latest or any other mutable tag. Make it impossible to deploy without a pinned version.

8. Automate drift detection and reconciliation

Drift is when your cluster’s actual state doesn’t match the desired state in Git. It happens when someone runs a manual kubectl command, when an autoscaler changes a replica count, or when a CRD controller modifies a resource behind your back.

The whole point of GitOps is continuous reconciliation. Your operator should constantly compare the live state against Git and bring things back in line.

Configure your GitOps tool to automatically revert manual changes, but build your exclusion lists (the set of fields and resources you intentionally skip during reconciliation) before you turn on auto-sync, not after. Controllers like Istio, cert-manager, and Crossplane legitimately modify resources, and auto-remediating those changes creates reconciliation loops that can destabilize a cluster. Start with a conservative exclusion list and tighten it over time.
Even with auto-remediation, you want to know when drift happens. Set up alerts. It might indicate a process problem or a team member who needs coaching.

Pro Tip: Be precise about what you reconcile. HPAs, VPAs, cert-manager annotations, and external-dns records all legitimately modify resources. Use ignore rules to exclude fields that are intentionally dynamic. In ArgoCD, that looks like this:

spec:
 ignoreDifferences:
 - group: apps
 kind: Deployment
 jsonPointers:
 - /spec/replicas
 - group: admissionregistration.k8s.io
 kind: MutatingWebhookConfiguration
 jqPathExpressions:
 - .webhooks[]?.clientConfig.caBundle

Also, don’t overlook resource ordering. ArgoCD sync waves and Flux’s dependsOn let you enforce that CRDs are applied before custom resources and namespaces before workloads. Getting ordering wrong is one of the most common causes of failed syncs in production.

9. Practice progressive delivery

Progressive delivery lets you gradually roll out changes, verify they’re working, and automatically roll back if something goes wrong.

GitOps and progressive delivery fit well together. Your Git repo describes the desired rollout strategy, and controllers in the cluster execute it.

Canary deployments: send a small percentage of traffic to the new version. If error rates spike, automatically roll back.
Blue-green deployments: run two identical environments, switch traffic once the new one is verified healthy.
Argo Rollouts extends Kubernetes with canary, blue-green, and analysis-driven rollout strategies that integrate with ArgoCD.
Flagger is the Flux ecosystem equivalent, with automated canary analysis, A/B testing, and blue-green deployments using service mesh or ingress controllers.
Kargo takes it a step further by orchestrating promotions across multiple stages and environments. Instead of manually wiring up pipelines to move changes from dev to staging to prod, Kargo automates the entire promotion workflow on top of ArgoCD. I did a deep dive on GitOps promotion tools and why they belong in your toolkit:

Pro Tip: Combine progressive delivery with automated analysis. Tools like Argo Rollouts and Kargo can query Prometheus metrics during a canary and automatically promote or roll back based on error rates, latency percentiles, or custom metrics.

10. Policy-as-code: your automated guardrails

Reviews are great, but humans make mistakes. Policy-as-code adds an automated layer that enforces organizational rules before a change can land in your cluster. Think of it as guardrails on a mountain road: they don’t slow you down, they just keep you from going over the edge.

OPA/Gatekeeper lets you define policies in Rego that the admission controller enforces at deploy time (e.g., no containers running as root, all deployments must have resource limits).
Kyverno is a Kubernetes-native policy engine that uses familiar YAML to define and enforce policies, including mutation and generation.
Starting with Kubernetes 1.26+, you can write CEL-based ValidatingAdmissionPolicies natively without any external tooling. A lightweight option if you don’t need the full feature set of OPA or Kyverno.
Pulumi CrossGuard lets you write policy packs in TypeScript or Python that validate infrastructure before it’s deployed. If your IaC and GitOps layers are bridged (see the next section), CrossGuard can enforce rules across both.
Run policy checks in your CI pipeline so violations are caught during the PR, not after deployment.

Pro Tip: Start with a small set of high-impact policies (required labels, no privileged containers, resource limits) and expand over time. Trying to enforce 50 policies on Day 1 will create so much friction that teams will revolt.

11. Bridge your IaC and GitOps (don’t choose one)

I hear this all the time: “We use Terraform for infrastructure and ArgoCD for deployments, and they live in completely separate worlds.” Sound familiar? Most teams treat IaC and GitOps as independent workflows, but they’re really two halves of the same story.

IaC handles “Day 0,” creating the cloud resources your cluster depends on: VPCs, the Kubernetes cluster itself, IAM roles, databases, DNS zones. GitOps handles “Day 2,” managing what runs inside the cluster: applications, addons, configurations. The gap between them is where things get messy. Your Helm charts need IAM role ARNs. Your GitOps-deployed addons need to know the cluster’s OIDC provider endpoint. That metadata has to flow from the IaC layer to the GitOps layer somehow.

The gitops-bridge pattern solves this cleanly: IaC provisions cloud resources and writes metadata (role ARNs, account IDs, endpoints) into Kubernetes resources (like ConfigMaps or Secrets) that your GitOps tool can consume directly.

Keep each tool in its lane. Don’t force Terraform’s Helm provider to fight with ArgoCD over resource ownership.
Pulumi lets you define your cloud infrastructure in real programming languages (TypeScript, Python, Go), which makes it natural to compute and pass metadata to your GitOps layer. You can use stack outputs to expose values like role ARNs and cluster endpoints, then feed them into your GitOps manifests. The Pulumi Kubernetes Operator even lets ArgoCD reconcile Pulumi stacks via GitOps.
Statsig went from “1-2 devs clicking around cloud consoles with SEVs left and right” to fully self-service by using Pulumi to generate manifests and ArgoCD to deploy them.
As you grow beyond a handful of clusters, think multi-cluster from the start. Patterns like ArgoCD ApplicationSets with cluster generators and Flux’s Kustomization targeting become important for fleet management. The gitops-bridge pattern works well here because your IaC layer can register new clusters and write their metadata into Kubernetes resources, which ApplicationSets automatically pick up.

Here’s how the gitops-bridge looks in practice. Pulumi provisions cloud resources and writes metadata into a ConfigMap that ArgoCD consumes:

import * as k8s from "@pulumi/kubernetes";

const clusterMetadata = new k8s.core.v1.ConfigMap("cluster-metadata", {
 metadata: {
 name: "cluster-metadata",
 namespace: "argocd",
 labels: {
 "gitops-bridge": "true",
 },
 },
 data: {
 aws_account_id: "123456789012",
 cluster_name: "prod-us-east-1",
 iam_role_arn: "arn:aws:iam::123456789012:role/app-role",
 oidc_provider: "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE",
 },
});

import pulumi_kubernetes as k8s

cluster_metadata = k8s.core.v1.ConfigMap(
 "cluster-metadata",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="cluster-metadata",
 namespace="argocd",
 labels={
 "gitops-bridge": "true",
 },
 ),
 data={
 "aws_account_id": "123456789012",
 "cluster_name": "prod-us-east-1",
 "iam_role_arn": "arn:aws:iam::123456789012:role/app-role",
 "oidc_provider": "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE",
 },
)

Your ArgoCD Applications or Helm values can then reference this ConfigMap, closing the loop between IaC and GitOps without hardcoding cloud-specific values.

Pro Tip: If you’re using Terraform today, the gitops-bridge pattern works there too. But if you’re tired of wrangling HCL and want type-safe infrastructure code with real programming constructs, give Pulumi a look. The bridge from IaC to GitOps becomes much more natural when both sides speak a real programming language.

12. Be pragmatic, not dogmatic

Michael Crenshaw from Intuit gave a talk titled “How GitOps Should I Be?” at a CNCF conference. Intuit runs 45 ArgoCD instances managing 20,000 applications across 200 clusters. The approach they took: “Be strategic, be pragmatic. GitOps most of the time.”

One of the largest GitOps adopters in the world explicitly deviates from pure GitOps when the situation calls for it. Dogmatic adherence to principles that don’t serve you is just another form of technical debt.

Intuit found that GitHub’s SLA of 20 minutes meant they couldn’t rely on GitOps for region evacuation, so they wrote a kubectl cron job that bypasses Git for failover. Pragmatic.
Massive Jenkins ecosystems don’t get rewritten overnight. Intuit’s compromise: the last step in the Jenkins pipeline calls argocd app sync. It’s push-based, yes, but the source of truth stays in Git. The Jenkins pipeline doesn’t own the desired state; it just tells ArgoCD “go reconcile now” instead of waiting for the next poll. It’s a stepping stone while teams migrate off Jenkins, not a permanent architecture.
Continuous reconciliation with self-heal (auto-reverting manual changes) is where most teams deviate. In practice, many teams start with automated drift detection and alerting (so you know when someone runs a manual kubectl command) but leave auto-remediation off until they’ve built confidence in their exclusion lists. That’s a reasonable middle ground.

Pro Tip: Document your intentional deviations. Create an ADR (Architecture Decision Record) for each case where you’ve chosen not to follow pure GitOps, explaining why and what the plan is to close the gap (if ever). This turns exceptions into conscious decisions rather than accidental shortcuts.

Final thoughts

GitOps isn’t a silver bullet, but it’s one of the best approaches we have for managing Kubernetes at scale. The practices in this list aren’t theoretical. They’re what I’ve learned from running GitOps in production, watching talks from teams at Intuit and Statsig, and making every mistake in the book at least once.

If I had to boil it all down to one sentence: treat your GitOps repo like production code, because it is.

Start with the basics (Git as source of truth, declarative config, pull-based delivery) and layer on the advanced practices as your team matures. Don’t try to implement all 12 on Day 1. Pick the three that address your biggest pain points and iterate from there.

And remember, even the largest GitOps adopters in the world aren’t doing it perfectly. They’re being pragmatic. You should be too.

How Pulumi fits into your GitOps workflow

Several of the practices in this post touch on problems Pulumi is built to solve. If you’re bridging IaC and GitOps, managing environment-specific configuration, or enforcing policy across both layers, here’s where to start:

Pulumi IaC: Define cloud infrastructure in TypeScript, Python, or Go and use stack outputs to feed metadata into your GitOps manifests.
Pulumi ESC: Manage secrets and configuration across environments with fine-grained access controls, integrated with Kubernetes via the External Secrets Operator or the Secret Store CSI Driver.
Pulumi Kubernetes Operator: Let ArgoCD reconcile Pulumi stacks via GitOps, closing the loop between your IaC and Day 2 workflows.

The Claude Skills I Actually Use for DevOps

Engin Diri — Mon, 09 Feb 2026 00:00:00 +0000

When Claude Code first released skills, I ignored them. They looked like fancy prompts, another feature to add to the pile of things I would get around to learning eventually. Then I watched a few engineers demonstrate what skills actually do, and something clicked. By default, language models do not write good code. They write plausible code based on what they have read. Plausible code turns into bugs, horrible UX, and infrastructure that breaks at 3am.

Skills fill that gap. They package engineering expertise into something Claude can use. The workflows and judgment matter more than the raw information. Without skills, every conversation starts from zero. You explain the same conventions and correct the same mistakes. Every morning, back to zero.

Think about what separates a junior engineer from a senior one. Both can write code that compiles. Both can deploy infrastructure that runs. The difference is that the senior engineer knows the patterns that prevent problems before they happen. They know when to use component resources instead of plain resources. They know that creating infrastructure inside an apply() callback (Pulumi’s way of transforming outputs that are not known until deployment) breaks preview. They know that hardcoded credentials will eventually end up in a git log somewhere embarrassing.

This knowledge takes years to accumulate through painful experience. Skills let you transfer that knowledge to Claude in minutes. And here is the thing that makes them practical: if you find yourself doing the same type of task with different content each time, that is a skill waiting to be built. You encode the process once, then feed it new inputs forever.

The mechanic, the tools, and the manual

I heard an analogy recently that made skills click for me. Imagine Claude as a mechanic. A capable mechanic who knows engines, can diagnose problems, and fix most cars that come through the shop.

MCP servers are like giving that mechanic a set of tools. Wrenches, diagnostic equipment, lift systems. Without tools, the mechanic cannot do much. With tools, the mechanic can work on whatever comes through the door.

But what happens when someone brings in a Formula 1 race car? Or a 1967 Ford Mustang that has been modified beyond recognition? The mechanic knows engines in general, but these specific vehicles require specific knowledge. The F1 car has procedures that must be followed in exact order. The vintage Mustang has quirks that only someone who has worked on that model would know.

Skills are the user manuals and standard operating procedures for these specific vehicles. They tell the mechanic what needs to happen and when. They encode the expertise of someone who has done this work a thousand times.

Or think about it from a carpenter’s perspective. Skills are the process to make the table: the measurements, the design, the exact steps. MCPs are the tools: the saw, the hammer, the drill. You need both. The process alone is theoretical, and tools without a process just sit in the garage.

For DevOps engineers working with Pulumi, this matters because infrastructure as code has its own quirks and patterns. Generic AI assistance produces code that looks reasonable but breaks conventions the community learned the hard way. Skills teach Claude those conventions.

Why skills instead of MCPs

Before skills clicked for me, I tried solving the expertise problem with MCPs. I kept adding servers until I noticed Claude getting slower and making worse decisions. Turns out the GitHub MCP alone eats 46,000 tokens across 91 tools before you type anything. Cursor eventually capped MCPs at 40 tools because too many options made everything worse.

Slash commands were another option, but you had to remember to invoke them. Anthropic apparently agreed, because in January 2026 they merged slash commands into skills. One unified system instead of two.

Skills avoid this through progressive disclosure. Claude reads just the description at startup, maybe a hundred tokens. The full procedures only load when Claude decides they are relevant. Unlike those massive system prompts that used to eat through your context window, skills stay out of the way until they are needed. For DevOps engineers running long infrastructure sessions with dozens of resources, this matters. You keep your context budget for the actual work instead of burning it on instructions. Skills can also fork context, spinning up isolated subagents that do work without polluting your main conversation. Think of it like handing a colleague a written brief. They go work on it, hand back a summary, and never sit in on your conversation.

I still use MCPs for connecting Claude to external systems. The Pulumi MCP server lets Claude query the registry and validate code. But MCPs give Claude access to things. Skills teach Claude how to think about things. Different jobs. They get more useful when you combine them. One engineer built a financial reporting skill that connects to his Mercury bank account via MCP, pulls every transaction for a given month, classifies the expenses into categories, and generates a styled HTML report with totals and breakdowns. A skill that knows your deployment process connecting to MCPs that talk to your actual infrastructure is the same idea, just pointed at ops instead of accounting.

Skills are portable. They follow an open standard, so a skill you write for Claude Code works in Cursor, GitHub Copilot, or anywhere else that supports agent skills. You can even copy the skill content into ChatGPT as a starting prompt. No vendor lock-in.

Teaching Claude to write Pulumi like an expert

The first time you ask Claude to help with a Pulumi project, the process is painful. You have to explain the patterns you want. You correct mistakes. You explain why creating resources inside apply() breaks things. By the third or fourth project, you start copying your corrections from previous conversations.

I built the dirien/claude-skills pulumi-typescript skill after going through this painful process too many times. It knows the patterns that prevent common mistakes: Pulumi ESC (Environments, Secrets, and Configuration) integration, OIDC (OpenID Connect) instead of hardcoded access keys, ComponentResource abstractions (reusable groups of related resources), and proper output structuring so dependent stacks can consume them cleanly.

Skill	What it teaches Claude
`pulumi-typescript`	Pulumi with TypeScript, ESC secrets management, component patterns, and multi-cloud deployment

npx skills add https://github.com/dirien/claude-skills --skill pulumi-typescript

Skills install as markdown files in your project’s .claude/skills/ directory, so they travel with your repo and are easy to review.

The next time you ask Claude to create infrastructure, it applies these patterns automatically. You do not have to remember to invoke the skill or correct the same mistakes repeatedly.

The official Pulumi skills

Pulumi maintains its own skills repository at pulumi/agent-skills, which they announced recently. The repo includes skills for ComponentResource patterns, Automation API, and migration from Terraform, CDK, CloudFormation, and ARM. The two I use daily are pulumi-esc and pulumi-best-practices.

The pulumi-esc skill teaches Claude how to work with Pulumi ESC (Environments, Secrets, and Configuration). It knows the difference between pulumi env get, pulumi env open, and pulumi env run. It sets up OIDC for dynamic credentials, integrates with external secret stores like AWS Secrets Manager and Vault, and structures layered environment composition so your dev, staging, and production configs inherit from a shared base.

The pulumi-best-practices skill catches the mistakes that burn you in production. It stops Claude from creating resources inside apply() callbacks, enforces proper parent relationships in ComponentResources, encrypts secrets from day one, and makes sure pulumi preview runs before any deployment. These are the patterns that took me years to internalize, and now Claude follows them by default.

Skill	What it teaches Claude
`pulumi-esc`	Environment, secrets, and configuration management with OIDC, dynamic credentials, and secret store integration
`pulumi-best-practices`	Resource dependencies, ComponentResource patterns, secret encryption, and safe refactoring

npx skills add https://github.com/pulumi/agent-skills --skill pulumi-esc
npx skills add https://github.com/pulumi/agent-skills --skill pulumi-best-practices

Making Claude a monitoring expert by default

You deploy something, it works, and six months later something breaks and you realize you never added monitoring. We have all been there. The monitoring skills from the community teach Claude to add observability from the start.

The jeffallan/claude-skills repository contains a monitoring-expert skill that knows Prometheus, Grafana, and DataDog.

Skill	What it teaches Claude
`monitoring-expert`	Structured logging, metrics, distributed tracing, alerting, and performance testing for production systems

npx skills add https://github.com/jeffallan/claude-skills --skill monitoring-expert

In my testing, deploying a static website with these skills installed looks different from vanilla Claude. Instead of just creating the S3 bucket and CloudFront distribution, Claude asked about error rate thresholds before writing any code. It suggested CloudWatch alarms and created an SNS topic for alerts. The results are not always this clean. Sometimes the monitoring suggestions are generic or miss your specific SLO requirements. But the baseline shifted from “no monitoring at all” to “monitoring that needs tuning,” and that is a better starting point.

Kubernetes configuration that actually passes security review

Kubernetes has hundreds of configuration options. Most deployments use a handful of them. The problem is that the important options like security contexts, resource limits, and pod disruption budgets are easy to forget when you are focused on getting something to run.

The jeffallan/claude-skills kubernetes-specialist skill focuses on configurations that production deployments actually need. Without it, ask Claude for a deployment and you get something that runs: the right image, the right ports, maybe a service. With the skill, the same request comes back with runAsNonRoot: true in the security context, resource requests and limits that reflect actual usage patterns, liveness and readiness probes with sensible intervals, and a pod disruption budget. These are the things that make the difference between “it works in staging” and “it survives a node failure in production.” The skill also understands when RollingUpdate makes sense versus Recreate, which is the kind of judgment call that usually requires context a generic model does not have.

The wshobson/agents repository fills in the gaps around Kubernetes with CI/CD, cost management, and deployment workflows:

Skill	What it teaches Claude
`kubernetes-specialist`	Production cluster management, security hardening, and cloud-native architectures
`cost-optimization`	Cloud cost reduction across AWS, Azure, and GCP with right-sizing and reserved instances
`github-actions-templates`	CI/CD workflows, Docker builds, Kubernetes deployments, security scanning, and matrix builds
`gitops-workflow`	ArgoCD and Flux CD for automated Kubernetes deployments

npx skills add https://github.com/jeffallan/claude-skills --skill kubernetes-specialist
npx skills add https://github.com/wshobson/agents --skill gitops-workflow
npx skills add https://github.com/wshobson/agents --skill github-actions-templates
npx skills add https://github.com/wshobson/agents --skill cost-optimization

Debugging like a senior engineer

The obra/superpowers repository contains a skill that changed how I debug with Claude. The systematic-debugging skill implements a four-phase framework: root cause investigation, pattern analysis, hypothesis testing, and implementation.

Without this skill, Claude tends to suggest solutions immediately. Something is broken, here are five things that might fix it. This feels helpful but often wastes time because none of the suggestions address the actual problem.

With the systematic debugging skill, Claude approaches problems differently. It asks clarifying questions. It wants to see logs. It builds a model of what is happening before suggesting changes. When it proposes a fix, it explains why that fix addresses the root cause. Sometimes skills find problems you did not know about. One engineer pointed a skills-equipped Claude at a set of SEO pages and discovered they had been decaying for months with nobody watching. The infrastructure parallel is obvious: configuration drift, unused resources, permissions that expanded over time. A debugging skill that investigates before prescribing will find these things.

Skill	What it teaches Claude
`systematic-debugging`	Root cause investigation, pattern analysis, hypothesis testing, and verified implementation

npx skills add https://github.com/obra/superpowers --skill systematic-debugging

Catching security issues before they ship

Two skills cover different sides of security review. The wshobson/agents k8s-security-policies skill handles Kubernetes-specific hardening: NetworkPolicies, Pod Security Standards, RBAC, OPA Gatekeeper constraints, and service mesh mTLS configuration. The sickn33/antigravity-awesome-skills security-review skill covers application-level concerns like secrets management, SQL injection, XSS prevention, and input validation.

I asked Claude to check a Pulumi program that created an S3 bucket. Without the security skills, Claude confirmed the code was correct and moved on. With the skills loaded, it flagged that the bucket had no server-side encryption configured, the bucket policy allowed s3:* from an overly broad principal, and there was no access logging enabled. On the Kubernetes side, the k8s-security-policies skill catches things like missing default-deny NetworkPolicies and containers running as root. These skills are not a replacement for deterministic tools like tfsec, checkov, or trivy. Those catch known issues every time. Skills are probabilistic and work best as an extra layer during development, not as your only security gate.

Skill	What it teaches Claude
`k8s-security-policies`	Network policies, pod security standards, RBAC, and admission control for defense-in-depth
`security-review`	Secrets management, input validation, SQL injection, XSS/CSRF prevention, and dependency auditing

npx skills add https://github.com/wshobson/agents --skill k8s-security-policies
npx skills add https://github.com/sickn33/antigravity-awesome-skills --skill security-review

Incident response before you need it

At 3am when something breaks, you want runbooks. The incident-runbook-templates skill from wshobson/agents helps Claude create these before you need them. It includes a four-level severity model (SEV1 through SEV4) with response time expectations, escalation decision trees, and communication templates for status updates.

When you ask Claude to document your deployment process, it produces runbooks with diagnostic steps, rollback protocols, and verification checks. It knows kubectl commands for Kubernetes recovery and SQL procedures for PostgreSQL troubleshooting. The output needs editing. Generated runbooks tend to be thorough on the happy path but thin on the failure modes that matter most at 3am. I treat them as a first draft that gets me to 60% in minutes instead of hours, then fill in the gaps from experience.

Skill	What it teaches Claude
`incident-runbook-templates`	Detection, triage, mitigation, resolution, and communication procedures for production incidents

npx skills add https://github.com/wshobson/agents --skill incident-runbook-templates

General-purpose DevOps and SRE skills

The skills above target specific problems. The jeffallan/claude-skills repository also includes two broader skills that cover the day-to-day work that does not fit neatly into one category.

The devops-engineer skill gives Claude a senior DevOps engineer persona covering CI/CD pipelines, container management, deployment strategies like blue-green and canary, and infrastructure as code across AWS, GCP, and Azure. It enforces constraints I care about: no deploying to production without approval, no secrets in code, no unversioned container images.

The sre-engineer skill focuses on reliability: SLO/SLI definitions, error budget calculations, golden signal dashboards, and toil reduction through automation. It produces Prometheus/Grafana configs, remediation runbooks, and reliability assessments. If you run production systems and want Claude to think about error budgets instead of just uptime, this is the skill.

Skill	What it teaches Claude
`devops-engineer`	CI/CD pipelines, container management, deployment strategies, and infrastructure as code across clouds
`sre-engineer`	SLI/SLO management, error budgets, monitoring, automation, and incident response

npx skills add https://github.com/jeffallan/claude-skills --skill devops-engineer
npx skills add https://github.com/jeffallan/claude-skills --skill sre-engineer

Vetting the skills you install

Before you install everything in sight, a warning. Skills run with the same permissions as your AI agent. A malicious skill can exfiltrate credentials, download backdoors, or disable safety mechanisms, and it will look like your agent doing it.

Snyk researchers published ToxicSkills in February 2026 after scanning 3,984 skills from public registries. 13.4% had critical-level vulnerabilities, and they found 76 confirmed malicious payloads. The attack techniques included base64-encoded commands that steal AWS credentials, skills that direct you to download password-protected executables from attacker infrastructure, and jailbreak attempts that try to disable safety mechanisms. 91% of malicious skills combine code-level malware with prompt injection, so they attack on two fronts simultaneously.

Treat skills like you treat any third-party dependency:

Read the source before installing. Skills are markdown and YAML files. If you cannot read the full skill in a few minutes, that is a red flag.
Check the repository. Look at stars, contributors, and commit history. A single-commit repository from an unknown account deserves scrutiny.
Run uvx mcp-scan@latest --skills to scan installed skills for known malicious patterns, prompt injection, and credential exposure.
Be cautious with skills that fetch external content at runtime. The Snyk research found 17.7% of skills on ClawHub pull from third-party URLs, which means the skill’s behavior can change after you install it.
Stick to known repositories. Every skill recommended in this post comes from a repository with visible maintainers and community activity.

Eight malicious skills were still publicly available on ClawHub when Snyk published their findings. The skills ecosystem is young, and the vetting infrastructure is still catching up.

Putting it together

Stacking skills is where this pays off. Install the Pulumi skills and Claude writes better infrastructure code. Add monitoring and security on top and you start catching problems that used to slip through to production.

A note on stacking: I have not hit conflicts running all of these simultaneously, but more skills means more descriptions for Claude to evaluate at startup. If you notice Claude getting slower or making odd choices, pare back to the skills you actually use for that project. Start with the Pulumi and monitoring skills, add others as you need them.

Here is how to set up a new project with all of them:

mkdir -p pulumi-skills-demo && cd pulumi-skills-demo
pulumi new aws-typescript --name skills-demo --yes

npx skills add https://github.com/dirien/claude-skills --skill pulumi-typescript
npx skills add https://github.com/pulumi/agent-skills --skill pulumi-esc
npx skills add https://github.com/pulumi/agent-skills --skill pulumi-best-practices
npx skills add https://github.com/obra/superpowers --skill systematic-debugging
npx skills add https://github.com/jeffallan/claude-skills --skill monitoring-expert
npx skills add https://github.com/jeffallan/claude-skills --skill kubernetes-specialist
npx skills add https://github.com/wshobson/agents --skill gitops-workflow
npx skills add https://github.com/wshobson/agents --skill github-actions-templates
npx skills add https://github.com/wshobson/agents --skill cost-optimization
npx skills add https://github.com/wshobson/agents --skill incident-runbook-templates
npx skills add https://github.com/jeffallan/claude-skills --skill devops-engineer
npx skills add https://github.com/jeffallan/claude-skills --skill sre-engineer
npx skills add https://github.com/wshobson/agents --skill k8s-security-policies
npx skills add https://github.com/sickn33/antigravity-awesome-skills --skill security-review

Then try these two prompts to see how many skills activate at once:

# Static website — triggers Pulumi TypeScript, monitoring, and security skills
Create a Pulumi TypeScript program for a static website on AWS with S3, CloudFront, OIDC credentials via Pulumi ESC, CloudWatch monitoring, and /security-review the infrastructure before deploying

# EKS cluster — stacks Kubernetes, GitOps, incident response, cost, and SRE skills
Create a Pulumi TypeScript program for an EKS cluster with /kubernetes-specialist security hardening, /gitops-workflow for ArgoCD deployment, /incident-runbook-templates for the cluster, /cost-optimization recommendations, and /sre-engineer SLO definitions for the services

The first prompt triggers the Pulumi TypeScript, monitoring, and security review skills in a single conversation. The second stacks Kubernetes, GitOps, incident response, cost, and SRE skills on one cluster build. You get infrastructure code, operational runbooks, and security policies from a single request.

What changes

Fair warning: not every skill works perfectly on the first try. Some need iteration. Some produce output that you have to review and tweak before it matches your standards. Skills do not replace your judgment.

That said, after a few weeks with these skills installed, I stopped correcting the same mistakes. The code Claude writes now looks like code I would write, not code I would have to fix. That is the whole point. Skills just stop you from repeating the same corrections across every conversation.

Get started

Every skill in this post includes its install command. Pick the section that matches your biggest pain point, run the npx skills add command, and try it on your next task. Skills work in Claude Code, Cursor, GitHub Copilot, and anything else that supports the Agent Skills standard.

The Pulumi Agent Skills announcement has more details, and the GitHub repository has the source. If you want something that goes further, with organizational context and deployment governance, look at Pulumi Neo. Neo is grounded in your actual infrastructure, not internet patterns. The 10 things you can do with Neo post shows what that looks like in practice.

Give it one project. That is all it took for me.

Try Pulumi for Free

CDKTF is deprecated: What's next for your team?

Adam Gordon Bell — Thu, 18 Dec 2025 10:00:00 -0800

In July, 2020, CDK for Terraform (CDKTF) was introduced, and last week, on December 10, it was officially deprecated. Support for CDKTF has stopped, the organization and repository have been archived, and HashiCorp/IBM will no longer be updating or maintaining it, leaving a lot of teams out there without a clear path forward.

For most teams, that means it’s time to start looking for a replacement.

It’s an unfortunate situation to suddenly find yourself in as a user of CDKTF, but you do have options, and Pulumi is one of them. In this post, we’ll help you understand what those options are, how Pulumi fits into them, and what it’d look like to migrate your CDKTF projects to Pulumi.

What are the alternatives to CDKTF?

Teams migrating away from CDKTF generally have three options:

Option 1: Fall back to HCL

HashiCorp’s official recommendation is to export your projects to HashiCorp Configuration Language (HCL) and manage them with Terraform. CDKTF even has a command that makes this fairly simple:

cdktf synth --hcl

Of course, if you’re using CDKTF, you probably chose it specifically to avoid HCL. So while possible, this probably isn’t the choice most teams would make unless they had to.

Option 2: Migrate to AWS CDK

If your team is all-in on AWS, another option would be to migrate to AWS CDK. It’s widely used, officially supported, the programming model is similar to CDKTF’s, and both CDK and CDKTF transpile to an intermediate format (CloudFormation YAML and Terraform JSON, respectively) that gets passed on to their underlying tools for deployment.

But while their programming and deployment models are conceptually similar, their resource models and APIs are entirely different. Here’s the code for an S3 bucket written in AWS CDK, for example:

import * as s3 from 'aws-cdk-lib/aws-s3';

const bucket = new s3.Bucket(this, 'my-bucket', {
 bucketName: 'my-example-bucket',
 versioned: true,
 publicReadAccess: false,
});

And here’s the code for a similarly configured bucket in CDKTF:

import { S3Bucket } from '@cdktf/provider-aws/lib/s3-bucket';

const bucket = new S3Bucket(this, 'my-bucket', {
 bucket: 'my-example-bucket',
 versioning: {
 enabled: true,
 },
 acl: 'private',
});

Notice how different these APIs are — and this is just one simple resource with only a few properties; imagine having to rewrite dozens or hundreds of them. Beyond that, there’s also the problem of state: How would you go about translating the contents of a Terraform state file containing hundreds of resources into the equivalent CloudFormation YAML or JSON?

Despite their surface similarities, CDKTF and AWS CDK have little in common. Migration would essentially mean a ground-up rewrite that’d also leave you without the multi-cloud support you already have with CDKTF. For most teams, that makes this option a practical non-starter.

Option 3: Migrate to Pulumi

This is where we should acknowledge our obvious bias — but we genuinely believe that for most users of CDKTF, Pulumi really is the simplest and most broadly compatible alternative.

Like CDKTF, Pulumi lets you build and manage your infrastructure with general-purpose languages like TypeScript, Python, Go, C#, and Java, and it supports organizing your code into higher-level abstractions called components, which you can think of like CDKTF constructs. Both organize cloud resources into stacks (think dev, prod), and both track deployment state similarly, with local, remote, and cloud-hosted options available.

Many of Pulumi’s most popular providers (e.g., the AWS provider) are also built from open-source Terraform schemas, which means their resource models will be nearly identical to what you’re used to with CDKTF. Here’s what an S3 bucket looks like in Pulumi, for example:

import * as aws from '@pulumi/aws';

const bucket = new aws.s3.Bucket('my-bucket', {
 bucket: 'my-example-bucket',
 versioning: {
 enabled: true,
 },
 acl: 'private',
});

You can also use any Terraform provider with Pulumi, and you can even reference Terraform modules directly from within your Pulumi code.

Pulumi is also different from CDKTF in several ways. One is that rather than transpile your source code to a format like JSON as CDKTF does (and then deploying it separately later), Pulumi uses its own declarative deployment engine that resolves the resource graph at runtime and provisions cloud resources directly, which is much faster and more flexible. You can learn more about the deployment model in How Pulumi Works.

Given the API similarities, the support for all Terraform providers and modules, the ability to coexist alongside Terraform-managed projects, and the built-in support for conversion (which we’ll cover next), we think Pulumi is the best alternative for most teams looking to migrate.

What migrating to Pulumi looks like

Migrating a CDKTF project to Pulumi generally happens in three steps:

Conversion, which translates your CDKTF code into a new Pulumi program
Import, which reads the contents of your CDKTF state into a new Pulumi stack
Refactoring, which brings the code in the new program into alignment with the stack’s currently deployed resources

Conversion and import

Migration starts with exporting your CDKTF project to HCL with cdktf synth. From there, Pulumi’s built-in convert and import commands handle creating the new program and importing your state:

# Export your project to HCL.
cdktf synth --hcl

# Convert the HCL into a new Pulumi project.
pulumi convert --from terraform --language typescript

# Create a new Pulumi stack.
pulumi stack init dev

# Import your CDKTF stack's resources into your new Pulumi stack.
pulumi import --from terraform ./terraform.dev.tfstate

The converter automatically translates Terraform input variables, data sources, resources, and outputs into their Pulumi equivalents. You can read more about how this works in Converting Terraform HCL to Pulumi.

Refactoring

Once you’ve imported your state, you’ll often have to make some adjustments to the code to bring it in line with the new Pulumi stack. For instance, pulumi import marks new resources protected by default, to prevent them from being accidentally deleted — but since the code produced by pulumi convert doesn’t include the protect resource option, you’ll need to add it yourself. Fortunately the import step also emits code that you can copy into your program to make this process a little easier.

Refactoring can get a bit more complicated when custom logic and higher-level abstractions are involved, as fidelity to the original CDKTF code is often lost in the translation to HCL. In these situations, having the help of an LLM to recapture that original logic or translate your CDKTF constructs into Pulumi components can be a big time-saver.

An end-to-end example

The best way to get a feel for how this works, though, is to try it yourself.

The pulumi/cdktf-to-pulumi-example repository on GitHub contains a CDKTF project with multiple stacks written in TypeScript, along with a guide that walks you through the process of migrating that project to Pulumi. The guide covers everything we’ve discussed here so far, including:

Converting the CDKTF project into a new Pulumi project
Importing its actively running resources into Pulumi stacks
Modifying the generated code to align with imported state
Performing an initial deployment with Pulumi to complete the migration process

The walkthrough takes only a few minutes to complete, and it’s a great way to stand up an example of your own to get more familiar with Pulumi.

github.com/pulumi/cdktf-to-pulumi-example

What’s next?

If you’re moving on from CDKTF and looking for an alternative, there are a few possible paths forward. For teams that want to keep using real languages and avoid a ground-up rewrite, Pulumi offers the clearest way forward.

To learn more about how Pulumi works, how it differs from CDKTF and from Terraform, how to handle additional conversion scenarios, and more, we recommend:

Diving into the Pulumi docs to get familiar with core concepts and features of the platform
Reading Migrating from Terraform or CDKTF to Pulumi for more detailed, Terraform-specific migration guidance
Joining us in the Pulumi Community Slack to ask questions and learn from others who’ve successfully made the leap from Terraform and CDKTF to Pulumi
Checking out Pulumi for All Your IaC — Including Terraform and HCL to learn more about Pulumi’s native support for Terraform and HCL

And of course, feel free to reach out! We’d love to help in any way we can.

Feature Flagging for Your Infrastructure

Elisabeth Lichtie — Sat, 06 Dec 2025 00:00:00 +0000

One of Pulumi’s foundational benefits is that it allows you to manage your infrastructure as software with rich programming languages, robust testing, and CI/CD patterns that you’d use with your application code. This post will cover applying another classic software development technique to your infrastructure: feature flagging. You can use feature flags to control change rollout, reduce the risk of new releases, and speed up the development of your infrastructure, the same way you do with your applications.

The examples in this post range from simply creating a flag and using it in a Lambda function to fully integrating with LaunchDarkly to build a comprehensive flagging system for your infrastructure.

Creating flags with Pulumi

Check out the code here: 01: Flag Application With LaunchDarkly.

When you define flags in your infrastructure as code, you can configure them alongside all the supporting infrastructure your code needs. You can also encourage flag creation as a best practice in Pulumi components or templates.

With Pulumi, you can use any Terraform provider to manage your feature flags, whether that’s LaunchDarkly, Statsig, Split, Flagsmith, or any other feature flagging tool with Terraform support. This lets you define flags, targeting rules, and the infrastructure they control in a single codebase. The example uses the LaunchDarkly provider to create flags and supporting infrastructure:

const flag = new launchdarkly.FeatureFlag("example-flag", {
 key: "exampleFlag",
 projectKey: "default",
 variationType: "boolean"
}, { provider });

The Lambda function evaluates the flag at runtime using the LaunchDarkly SDK key from environment variables, allowing you to toggle behavior without redeploying code.

Using flags to control infrastructure

Check out the code here: 02: Flaggable Infrastructure.

Before deploying this code, first deploy either 03 (for ESC Only) or 04 (for ESC and LaunchDarkly) so that the environment is created and available.

Once you have flags defined, you can use them to control infrastructure provisioning. Apply the same best practices you would for application feature flags: keep flags focused, remove them when they’re no longer needed, and document their purpose. In this example, a boolean flag controls whether internet-facing resources are created:

const config = new pulumi.Config();
const flags: Record<string, any> | undefined = config.getObject("flags");
const enableInternetAccess = flags ? flags["enableInternetAccess"] : false;

if (enableInternetAccess) {
 // Create Internet Gateway, public subnet, route table, and routes
}

To enable automatic infrastructure updates when flags change (as shown in examples 3 and 4), configure deployment settings in your stack. This requires:

The Pulumi GitHub App installed on your repository
Webhook configuration to trigger deployments when configuration values change

With these in place, changing a flag value automatically triggers a deployment.

Configuring values with ESC

Check out the code here: 03: ESC Auto-updating.

Ingesting config from ESC

Pulumi ESC provides a centralized way to manage configuration and secrets. For teams that don’t need the full feature set of a dedicated flagging service, you can store flag values directly in an ESC environment. Your infrastructure stack imports this environment and reads flag values at deployment time:

const pulumiProv = new pulumiservice.Environment("flaggingEnv", {
 name: "config",
 organization: pulumi.getOrganization(),
 yaml: new pulumi.asset.StringAsset(`values:
 pulumiConfig:
 flags:
 enableInternetAccess: 'true'`),
});

This approach keeps all configuration in one place, eliminates dependencies on external services, and works seamlessly with Pulumi’s existing stack configuration system. You can update flag values through the Pulumi CLI, API, or Cloud console.

Automatically triggering deployments with webhooks

There is some risk involved with automatically triggering updates. You can leave out the webhook and deploy manually if that’s safer for your team.

Set up a Pulumi Cloud webhook to automatically deploy infrastructure when ESC environment values change. The example creates an AWS Lambda function behind API Gateway that receives webhook events and triggers deployments via the Pulumi Deployments API:

const webhook = new pulumiservice.Webhook("escEnvironmentWebhook", {
 organizationName: pulumi.getOrganization(),
 displayName: "ESC Environment Change Webhook",
 payloadUrl: api.apiEndpoint.apply((endpoint) => `${endpoint}/webhook`),
 active: true,
 filters: ["environment_revision_created"],
});

When you update a flag value in ESC, the webhook fires, the Lambda identifies which stacks use that environment, and triggers their redeployment. This creates a continuous delivery pipeline for infrastructure configuration changes.

Ingesting flag values from LaunchDarkly

Check out the code here: 04: LaunchDarkly Auto-updating.

Ingesting flags from LaunchDarkly using ESC Connect

If your team already uses LaunchDarkly for application feature flags, you can integrate those same flags with your infrastructure using ESC Connect. This is also a good option if you need enterprise-grade features like advanced targeting rules or percentage rollouts.

This approach unifies application and infrastructure flag management. The ESC environment uses fn::open::external to query a Lambda function that fetches current flag values from LaunchDarkly:

values:
 pulumiConfig:
 flags: ${customSecrets.response}
 customSecrets:
 fn::open::external:
 url: ${getterUrl}
 request:
 secretName: ANY_SECRET
 secret: true

Your infrastructure code remains unchanged: it still imports the ESC environment and reads flag values the same way as in example 3. The difference is that flag values now come from LaunchDarkly, giving you access to advanced features like targeting rules, percentage rollouts, and flag scheduling.

Automatically triggering deployments with webhooks

There is some risk involved with automatically triggering updates. You can leave out the webhook and deploy manually if that’s safer for your team.

LaunchDarkly’s webhook system can trigger infrastructure updates whenever a flag changes. Here’s how the automation flow works:

graph LR
A[Update Flag in LaunchDarkly] --> B[LaunchDarkly Webhook]
B --> C[API Gateway]
C --> D[Lambda Function]
D --> E[Pulumi Deployments API]
E --> F[Infrastructure Updates]

When the infrastructure deploys, it retrieves the current flag values through ESC:

graph LR
A[Infrastructure Stack] --> B[ESC Environment]
B --> C[fn::open::external]
C --> D[Flag Getter Lambda]
D --> E[LaunchDarkly SDK]
E --> |Flag Value| D
D --> |Flag Value| C
C --> |Flag Value| B
B --> |Flag Value| A

The Lambda function is configured as a webhook endpoint in LaunchDarkly:

const launchDarklyWebhook = new LaunchDarklyWebhook("myWebhook", {
 pulumiProv: provCreation.pulumiEnv,
 flagKey: launchdarklyFlag.key,
 pulumiTokenSecret,
 lambdaRole,
}, { provider });

This creates a full automation loop: change a flag in LaunchDarkly’s UI, and your infrastructure automatically updates to reflect the new value. This pattern works with any feature flagging service that supports webhooks.

ESC Only vs ESC with LaunchDarkly

Both approaches enable automated infrastructure updates based on flag changes, but they suit different use cases:

Aspect	ESC Only (Example 3)	ESC with LaunchDarkly (Example 4)
Complexity	Simpler setup with fewer moving parts	More components but unified with application flags
Dependencies	No external services required	Requires LaunchDarkly subscription
Flag Management	Edit via Pulumi CLI or Cloud console	Edit via LaunchDarkly UI with full feature set
Advanced Features	Basic flag storage, versioning, and approvals	Targeting rules, percentage rollouts, experiments, scheduling
Best For	Teams wanting simple config management	Teams already using LaunchDarkly or needing sophisticated flag controls
Key Benefit	Centralized Pulumi-native configuration	Unified management of application and infrastructure flags

Conclusion

Infrastructure deserves the same development practices as application code. Feature flags let you control rollouts, reduce deployment risk, and coordinate changes across your entire system, not just in your applications, but in the infrastructure that supports them.

Pulumi makes infrastructure feature flagging practical. Whether you choose the simplicity of ESC-only flag management or integrate with an enterprise service like LaunchDarkly, you can build infrastructure that responds dynamically to configuration changes while maintaining the same development practices you use for application code.

Change Management with the Pulumi Kubernetes Operator and Kargo

Elisabeth Lichtie — Tue, 25 Nov 2025 00:00:00 +0000

The Pulumi Kubernetes Operator (PKO) enables you to manage Pulumi stacks as Kubernetes resources, but it doesn’t provide much guidance on change management. Kargo fills this gap by providing controlled, staged promotions with verification steps. Together, they let you keep your infrastructure defined in Pulumi while managing multi-environment rollouts in a systematic way.

This pattern is especially useful for platform teams that need to balance delivery speed with governance as environments and expectations scale.

Going to KubeCon Europe 2026?

Visit Pulumi at Booth 784 for live demos of Pulumi Kubernetes Operator and Kargo workflows for managing infrastructure change at scale.

What you can do with Argo CD, PKO, and Kargo

When you combine these tools, you gain several change management capabilities:

Control release timing: Determine when updates to your Pulumi program are released to different environments, rather than deploying everywhere at once.
Visualize deployments: See which environments are running which versions of your infrastructure code at a glance.
Automatic verification: Automatically verify deployment success before promoting changes to the next environment.
Approval gates: Set up discrete approval requirements before promotion, ensuring human review when needed.
Change tracking: Maintain a clear audit trail of what changed, when, and how it moved through your environments.

About Kargo

Kargo is a continuous promotion platform that manages how changes move through application lifecycle stages. While it’s commonly used for application deployments, it also works for infrastructure code managed by PKO.

The key concepts you’ll work with:

Freight: A packaged set of artifacts that move together through your pipeline. When integrating with PKO, this will typically be the pulumi program that defines your infrastructure and the git commit with the latest version.

Stages: Promotion targets that represent different lifecycle phases (like dev, staging, production). Each stage corresponds to a different PKO Stack resource managing that environment’s infrastructure.

Promotions: The process of moving freight from one stage to the next. Kargo updates the Stack resources to point to the promoted freight.

Warehouses: Monitors that watch your Git repository for infrastructure code changes and package new commits or releases as freight.

Projects: Organizational units that group related stages and warehouses, mapping to Kubernetes namespaces for access control.

When Kargo promotes freight to a stage, it updates the corresponding PKO Stack resource with the new Git reference. PKO then reconciles the stack, and your infrastructure changes are applied.

Example architecture

Workflow

To demonstrate how these tools work together, let’s look at an example setup that manages infrastructure code through a controlled promotion pipeline:

This architecture includes six main components working together:

Security Scanner repository: Contains your Pulumi infrastructure code. Kargo watches this repository for new commits or releases.
Kargo: Controls the release pipeline. When it detects new code in the security scanner repository, it triggers an update to promote that code to the next stage.
Kargo manifests repository: Stores the Kubernetes manifests that define PKO Stack resources. Kargo updates this repository with the new Git references when promoting freight.
Argo CD: Watches the Kargo manifests repository and creates or updates Stack objects in Kubernetes when it detects changes.
Pulumi Kubernetes Operator: Watches Stack objects and deploys them to Pulumi, triggering stack updates or previews.
Pulumi: Performs the actual infrastructure deployments and previews based on PKO’s instructions.

The workflow follows this pattern: Kargo detects new infrastructure code, updates the manifests repository with the new Git reference, Argo CD syncs the Stack objects to Kubernetes, PKO triggers Pulumi to deploy the changes, and Kargo verifies the successful deployment. This creates a closed loop where infrastructure changes are systematically promoted and verified at each stage.

Dashboard and Management

The Kargo dashboard brings operational visibility to this entire workflow. The timeline view at the top shows every freight version moving through your pipeline, making it immediately clear which infrastructure code version is running in each environment and how long it has been there.

The flow diagram above maps out your complete promotion path with color-coded stages. You can see at a glance where changes are flowing smoothly and where they are waiting at approval gates or blocked by failed verifications.

Status indicators on each stage show real-time health information, eliminating the need to manually check multiple systems or run status commands. This single-pane view reduces context switching, speeds up troubleshooting when issues arise, and gives an understanding of exactly what infrastructure is deployed where.

You can see the complete setup for this example, including configuration files and manifests, in the pulumi-operator-with-kargo-change-management repository.

Setting up the environment

The example repository contains three Pulumi projects that deploy in sequence:

cluster: Creates a basic Amazon EKS cluster where everything will run.
cluster-setup: Installs the required dependencies including PKO, cert-manager, Argo CD, Argo Rollouts, and AWS Cognito for authentication.
kargo: Deploys Kargo via Helm and provides examples to set up the project with stages, approval gates, and analysis templates for the promotion pipeline.

Configuring Kargo

Once Kargo is installed, you need to configure several components to set up your promotion pipeline.

Creating a project

Start by creating a Kargo project, which automatically creates a corresponding Kubernetes namespace.

Git credentials

Kargo needs access to your Git repositories to monitor for changes and update manifests. Create a secret with your Git credentials, labeled so Kargo recognizes it:

apiVersion: v1
kind: Secret
metadata:
 name: github-creds
 namespace: kargo-managed-stack
 labels:
 kargo.akuity.io/cred-type: git
stringData:
 repoURL: https://github.com/YOUR_REPO
 username: YOUR_USERNAME
 password: GITHUB_TOKEN

Setting up approver roles

To control who can approve promotions at approval gates, you can create custom roles with the promote permission. This requires three resources: a ServiceAccount with OIDC claim mapping, a Role with promotion permissions, and a RoleBinding to connect them. The easiest way to set up these resources is through the Kargo UI.

The screenshot shows a role configured with the promote verb on specific stages (like approvalgateqa), which allows users bound to this role to approve promotions at that gate. The role also includes permissions to create promotions and view other resources.

Project configuration

Use a ProjectConfig resource to control promotion policies for your project. This example enables automatic promotion for preview and deployment stages:

apiVersion: kargo.akuity.io/v1alpha1
kind: ProjectConfig
metadata:
 name: kargo-managed-stack
 namespace: kargo-managed-stack
spec:
 promotionPolicies:
 - stageSelector:
 name: devpreview
 autoPromotionEnabled: true
 - stageSelector:
 name: dev2preview
 autoPromotionEnabled: true
 - stageSelector:
 name: dev
 autoPromotionEnabled: true
 - stageSelector:
 name: dev2
 autoPromotionEnabled: true
 - stageSelector:
 name: qapreview
 autoPromotionEnabled: true
 - stageSelector:
 name: qa
 autoPromotionEnabled: true

This configuration automatically promotes freight through preview stages and dev environments, while approval gates remain manual.

Pulumi access token

Kargo needs access to the Pulumi API to verify that infrastructure deployments completed successfully. Create a secret in your project through the Kargo UI:

Navigate to your project in the Kargo UI
Navigate to Settings → Credentials
Select Create a new credential with:
- Name: pulumi-api-token
- Type: Opaque
- Key: PULUMI_ACCESS_TOKEN
- Value: Your Pulumi access token

This token allows Kargo’s analysis templates to query the Pulumi API and confirm that stack updates and previews ran successfully before promoting freight to the next stage.

Defining stages

Stages represent different points in your promotion pipeline. This example uses three types of stages to control how infrastructure changes move from development to production.

Preview stages

Preview stages run pulumi preview operations to show what changes would be applied without actually deploying them. These stages help catch issues early by validating that the infrastructure code is valid and showing the expected changes:

devpreview and dev2preview: Run previews for the dev environments
qapreview: Runs a preview for QA after passing approval gates

Preview stages automatically promote to their corresponding update stages once the preview completes successfully and verification passes.

Update stages

Update stages run pulumi up to actually deploy the infrastructure changes:

dev and dev2: Deploy changes to parallel dev environments for testing
qa: Deploy to the QA environment after all approvals

Update stages are configured to automatically promote once the deployment succeeds and verification confirms the infrastructure is in the expected state.

Approval gates

Approval gates are stages that require manual approval before freight can proceed. They act as checkpoints where designated approvers review changes before they move forward:

approvalgatedev: Manual checkpoint before dev deployments
approvalgateqa: Functional review gate before QA deployment
approvalgateqasec: Security review gate before QA deployment

The QA preview stage uses availabilityStrategy: All, meaning freight must pass through both approvalgateqa and approvalgateqasec before proceeding. This ensures both functional and security teams approve changes before they reach QA.

Custom approver roles

Different approval gates may require different approvers. Using the custom roles described earlier, you can grant specific users or groups the promote permission on individual approval gates. For example, security team members might have the promote permission on approvalgateqasec, while functional testers have it on approvalgateqa. This ensures the right people review changes at each checkpoint.

Verification

After each stage completes, Kargo can verify that the operation succeeded before promoting to the next stage. This example uses analysis templates to query the Pulumi API and confirm deployments completed successfully.

Kargo uses Argo Rollouts for verification, so Argo Rollouts must be installed in your cluster.

Analysis templates

Analysis templates define how to verify a stage. They can make HTTP requests, run queries, or execute any checks you need. For Pulumi verification, the templates:

Wait for operations to complete: Include a delay to give Pulumi time to finish the preview or update operation
Query the Pulumi API: Make HTTP requests to the Pulumi API to retrieve the stack’s latest update information
Verify the commit ID: Check that the deployed commit matches the freight being promoted
Confirm success: Ensure the operation completed without errors

The analysis template uses the PULUMI_ACCESS_TOKEN secret created earlier to authenticate with the Pulumi API.

Custom verification standards

Analysis templates are flexible enough to enforce other standards your organization requires as well.

This gives you a way to codify your deployment standards and ensure they’re enforced automatically as infrastructure changes move through environments.

Conclusion

Combining the Pulumi Kubernetes Operator with Kargo gives you systematic control over how infrastructure changes move through your environments. Rather than manually managing deployments or pushing changes directly to production, you get automated promotions with verification steps, approval gates where human review is needed, and a clear audit trail of what changed and when. This approach is particularly valuable for teams managing multiple environments where infrastructure changes need to be tested and approved before reaching production. By defining your promotion pipeline as code with stages, approval gates, and verification templates, you ensure that your infrastructure changes follow the same rigorous process every time.

If you want to put this pattern into practice, you can see it live at KubeCon Europe 2026 (Booth 784) or register for our upcoming Zero to Production in Kubernetes workshop.

Try Pulumi for Free

Day 2 Operations: Drift Detection and Remediation

Mitch Gerdisch — Fri, 24 Oct 2025 00:00:00 +0000

Welcome to the fourth post in our IDP Best Practices series. Today we’re diving into the world of drift detection and remediation, those critical day 2 operations that keep your infrastructure aligned with its intended configuration long after the initial deployment.

You’ve built a beautiful platform with robust guardrails, comprehensive templates, and well-defined golden paths. Your developers are productive, deployments are smooth, and everything seems perfect. Then reality hits. An on-call engineer makes an emergency change through the AWS console during a 3 AM incident. A team member tweaks a security group rule to debug a connection issue and forgets to revert it. Auto-scaling adjusts capacity based on load patterns. Before you know it, your actual infrastructure has quietly diverged from what your code describes.

This post continues our journey through the IDP Best Practices series. We’ve covered strategy and self-service infrastructure, built golden paths with components and templates, and established deployment guardrails through policy as code. Now we’re tackling what happens after deployment when the real world starts making changes to your carefully crafted infrastructure.

Series roadmap:

How to Build an Internal Developer Platform: Strategy, Best Practices, and Self-Service Infrastructure
Build Golden Paths with Infrastructure Components and Templates
Deployment Guardrails with Policy as Code
Day 2 Operations: Drift Detection and Remediation (you are here)
Extend Your IDP for AI Applications: GPUs, Models, and Cost Controls
Next-Gen IDPs: How to Modernize Legacy Infrastructure with Pulumi

The Reality of Infrastructure Drift

Let me paint you a picture that might feel uncomfortably familiar. It’s Saturday afternoon, and you’re enjoying a barbecue with friends when your phone buzzes with that dreaded alert: production is down. You rush to your laptop, but you’re not on the corporate VPN. The AWS console is accessible, but you can’t reach the application itself to properly diagnose the issue.

Faced with a growing incident and impatient stakeholders, you make a quick decision. You modify the security group directly in the console, temporarily allowing access from anywhere (0.0.0.0/0) just so you can get in and fix the problem. Twenty minutes later, the issue is resolved, everyone’s relieved, and you return to your barbecue. That temporary security group change? It stays there, a silent time bomb waiting to be discovered in your next security audit.

This scenario plays out in organizations every day. Your infrastructure has now drifted from its intended state, and it will continue drifting, accumulating more changes, until someone or something notices and fixes it. Each untracked change increases the gap between what you think you have and what actually exists in production.

Understanding the Prevention vs. Detection Paradigm

When building a robust platform, you need to think in layers of defense. Your first instinct might be to prevent all drift from happening in the first place, and that’s absolutely the right starting point. But prevention alone isn’t enough in the messy reality of production operations.

Prevention: The First Line of Defense

Prevention is what we’ve been building throughout this series. You establish code reviews where teammates scrutinize infrastructure changes before they go live. You implement automated testing that validates configurations before deployment. You deploy CrossGuard policies that act as guardrails, blocking misconfigurations before they can cause damage. You create standardized components that encode best practices, making it easier to do the right thing than the wrong thing.

These preventative measures work brilliantly for changes that flow through your normal IaC pipeline. They catch typos, enforce standards, and maintain consistency across your infrastructure. But they have a critical blind spot: they can’t see changes that bypass your pipeline entirely.

Detection and Remediation: The Safety Net

This is where drift detection enters the picture. Think of it as continuous verification, constantly comparing what should exist with what actually exists. While prevention blocks most issues at the gate, detection catches everything that finds another way in.

What kinds of changes slip past prevention? Emergency modifications made during incidents when there’s no time for a proper code review. Console changes made by well-meaning engineers debugging production issues. External factors you can’t control, like AWS changing service limits or deprecating API endpoints. Even accidental modifications by team members who thought they were in the staging environment.

The truth is, no matter how robust your preventative controls, you need detective controls to maintain infrastructure integrity. Prevention and detection work together, creating a comprehensive defense against infrastructure chaos.

What Is Infrastructure Drift?

Infrastructure drift sounds technical, but the concept is surprisingly simple. Drift occurs when your actual infrastructure state doesn’t match your declared state in code. It’s the gap between intention and reality, between what you coded and what’s actually running.

To understand drift properly, you need to grasp three distinct states that Pulumi manages. First, there’s the desired state, which is what your Pulumi program declares in code. This is your intention, your blueprint for how things should be. Then there’s the current state, stored in Pulumi’s state file, which tracks what Pulumi believes exists based on its last interaction with your cloud provider. Finally, there’s the actual state, the ground truth of what really exists in your cloud provider right now.

The journey to drift typically follows a predictable pattern. You start by running pulumi up to deploy your resources, and everything is in perfect harmony. Pulumi creates the resources, records their configuration in its state file, and all three states align perfectly. Time passes, and your resources continue running in the cloud, serving traffic and doing their job.

Then someone makes a manual change. Maybe they modify a resource directly in the console, or an external system alters a configuration. Suddenly, your actual state has diverged from both your current state (what Pulumi thinks exists) and your desired state (what your code says should exist). You now have drift, and it will persist until you detect and address it.

Pulumi’s Drift Detection and Remediation Workflow

Pulumi provides a complete workflow for handling drift that’s both powerful and surprisingly straightforward. Rather than treating drift as an exceptional case, Pulumi makes it a routine part of infrastructure management.

1. Drift Detection with `pulumi refresh`

The foundation of drift detection is the pulumi refresh command. When you run this command, Pulumi springs into action, querying your cloud providers to discover the actual state of your resources. It then compares this reality against what it has recorded in its state file:

# Run drift detection
pulumi refresh --preview-only

# This command:
# 1. Reads your current Pulumi state
# 2. Queries cloud providers for actual resource configurations
# 3. Identifies any differences
# 4. Reports what has drifted

The --preview-only flag is particularly useful here. It shows you exactly what’s drifted without making any changes to your state file. You get a clear report of the differences, allowing you to review and understand the drift before deciding how to handle it.

2. State Reconciliation with `pulumi refresh`

After reviewing the drift, you face a decision. If the changes are legitimate and should be preserved, you can update your state file to match reality:

# Update state file to match cloud reality
pulumi refresh

# This command:
# 1. Detects drift (as above)
# 2. Updates state file to match actual cloud state
# 3. Prepares for remediation via pulumi up

This reconciliation step is crucial. It acknowledges the current reality and updates Pulumi’s understanding of your infrastructure without making any actual changes to your resources. You’re essentially telling Pulumi, “This is what actually exists right now.”

3. Drift Remediation with `pulumi up`

With your state file now reflecting reality, you can use pulumi up to remediate the drift. This command compares the updated state against your code and determines what changes are needed to restore your infrastructure to its intended configuration:

# Remediate drift - restore to desired state
pulumi up

# This command:
# 1. Compares updated state with your Pulumi program
# 2. Identifies required changes
# 3. Applies changes to restore infrastructure

This three-step dance of detect, reconcile, and remediate gives you complete control over how you handle drift. You can review changes, decide whether to keep or revert them, and ensure your infrastructure returns to its intended state.

Automating Drift Detection with Pulumi Deployments

Running pulumi refresh manually works great for small teams and simple infrastructures, but it doesn’t scale. As your platform grows to dozens or hundreds of stacks across multiple environments, manual drift detection becomes impossible. This is where Pulumi Deployments transforms drift detection from a manual chore into an automated safeguard.

Setting Up Automated Drift Detection

The journey to automated drift detection starts with connecting your infrastructure code to Pulumi’s deployment system. Let’s walk through the setup process.

Step 1: Configure Deployment Settings

Your first step is connecting Pulumi to your source control system using Deployment Settings. This integration allows Pulumi to access your infrastructure code and run automated operations:

Pulumi offers native integrations with all major version control systems. If you’re using GitHub, you get full app integration with PR previews, making it easy to see the impact of changes before merging. GitLab users enjoy similar functionality with merge request automation. For other Git providers or self-hosted solutions, you can use raw Git integration with direct repository access using credentials.

Step 2: Create Drift Detection Schedules

With your source control connected, you can now configure automated drift detection to run on whatever schedule makes sense for your organization. Some teams check hourly for production environments, while others might run daily checks for development stacks:

For environments where you’re confident in your automation, you can take it a step further and enable automatic remediation:

With auto-remediation enabled, Pulumi not only detects drift but automatically runs pulumi up to restore your infrastructure to its intended state. This is powerful but should be used carefully, especially in production environments.

Step 3: Configure Webhooks for Notifications

Detecting drift is only valuable if the right people know about it. Pulumi’s webhook system ensures your team stays informed when drift occurs:

You can route notifications wherever your team actually pays attention. Send alerts directly to Slack channels where your ops team congregates. Push notifications to Microsoft Teams if that’s your collaboration platform. Use custom webhooks to integrate with PagerDuty, Datadog, or any other monitoring system. You can even use deployment triggers to automatically run dependent stacks when drift is detected and fixed.

Handling Common Drift Scenarios

Every organization faces similar drift scenarios. Understanding how to handle these common cases will prepare you for real-world operations.

Scenario 1: Emergency Hotfix

The most common drift scenario occurs during incidents. Someone makes an emergency change to fix production, and now you need to decide whether to keep or revert that change:

# 1. Acknowledge the drift
pulumi refresh --stack production

# 2. Review the changes
pulumi preview --stack production

# 3. Either incorporate or revert
# Option A: Accept the change
pulumi refresh --stack production --yes

# Option B: Revert to code
pulumi up --stack production --yes

The key here is having a clear process. First, acknowledge the drift by running refresh to update your state. Then review the changes carefully. Finally, make an explicit decision: either accept the change by updating your code to match, or revert it by running pulumi up to restore the original configuration.

Scenario 2: Legitimate External Changes

Not all drift is bad. Auto-scaling groups adjust capacity based on load. Managed services update configurations automatically. Some changes are legitimate and should be preserved:

// Mark resources that can change externally
const autoScalingGroup = new aws.autoscaling.Group("app-asg", {
 minSize: 2,
 maxSize: 10,
 // ... other config
}, {
 ignoreChanges: ["desiredCapacity"] // Ignore scaling changes
});

The ignoreChanges option tells Pulumi to ignore specific properties when detecting drift. Use it for fields that legitimately change outside your control, like auto-scaling capacity or last-modified timestamps.

Scenario 3: Configuration Drift

Sometimes drift occurs in configuration values rather than resource structure. IP allowlists, feature flags, and environment-specific settings often drift. Handle these with Pulumi’s configuration system:

# Use Pulumi's configuration system
config = pulumi.Config()
allowed_ips = config.get_object("allowed_ips") or ["10.0.0.0/8"]

security_group = aws.ec2.SecurityGroup("app-sg",
 ingress=[{
 "from_port": 443,
 "to_port": 443,
 "protocol": "tcp",
 "cidr_blocks": allowed_ips # Managed via configuration
 }]
)

By externalizing configuration values, you make them easier to update without code changes. This pattern works particularly well for values that change frequently or differ between environments.

The Value of Automated Drift Detection

After implementing drift detection for hundreds of stacks, patterns emerge that tell a compelling story about its value. The transformation begins subtly but grows profound as teams discover what they’ve been missing.

The first change most organizations notice is in their security posture. Where once unauthorized changes might lurk undetected for weeks, only surfacing during quarterly audits or worse, during incident investigations, now they’re caught within minutes. Security teams who previously dreaded audit season find themselves with continuous compliance data at their fingertips. Every change, whether legitimate or not, creates an audit trail that transforms compliance reporting from a scrambling fire drill into a routine export of already-collected data.

As teams settle into this new reality, operational improvements become apparent. Incidents that used to take hours to diagnose now resolve in minutes because engineers can immediately check whether drift contributed to the problem. One team reported that their mean time to recovery dropped by 40% simply because they could eliminate drift as a cause within seconds rather than spending hours manually comparing configurations. Change management, once a bureaucratic nightmare of spreadsheets and approval chains, transforms into an automated system where every modification is tracked, timestamped, and traceable to its source.

Then come the unexpected discoveries. Drift detection becomes a cost optimization tool that nobody anticipated. It uncovers orphaned resources spinning away forgotten in some corner of your cloud account, created during a debugging session six months ago. It catches that instance someone manually upgraded to a larger size during a performance investigation and forgot to downsize. One organization discovered they were spending $15,000 monthly on resources that weren’t even supposed to exist, all caught by their drift detection system in its first week of operation.

Perhaps most importantly, drift detection changes how teams work. Engineers stop wasting time on manual infrastructure audits. Debugging accelerates when you have clear reports showing exactly what changed and when. The constant anxiety about whether production matches your code disappears, replaced by confidence that any discrepancies will be caught and reported immediately. Teams report feeling more empowered to move quickly, knowing they have a safety net that will catch configuration drift before it causes problems.

The cumulative effect is transformative. Organizations running drift detection report not just fewer incidents and lower costs, but a fundamental shift in how they think about infrastructure management. It becomes proactive rather than reactive, confident rather than anxious, automated rather than manual. What starts as a simple scheduled check evolves into a cornerstone of operational excellence.

Getting Started with Drift Detection

Ready to implement drift detection for your IDP? Here’s a practical action plan that you can execute today.

Quick Start Checklist

Begin with the simplest possible drift detection: a manual check on your production stack. Run pulumi refresh --preview-only --stack production to see if you have any existing drift. This takes just 15 minutes and often reveals surprising discrepancies.

Next, invest 30 minutes in setting up automated detection. Configure Pulumi Deployments, create an hourly detection schedule for your most critical stack, and set up Slack notifications so you’ll know immediately when drift occurs.

Spend the next week monitoring and learning. Review the drift patterns that emerge. Identify common causes. Are certain resources drifting repeatedly? Are changes happening at predictable times? Document which changes are legitimate and which represent problems.

After a week of observation, you’re ready to implement remediation. Start with non-production environments where mistakes have lower impact. Enable auto-remediation gradually, monitoring success rates and adjusting your configuration based on what you learn.

Finally, optimize and scale your drift detection. Adjust schedules based on the patterns you’ve observed. Implement custom workflows for complex scenarios. Expand coverage to all your environments and stacks. What starts as a simple check becomes a comprehensive system protecting your entire infrastructure.

Conclusion: Day 2 Operations as a Competitive Advantage

Drift detection and remediation represent more than just good hygiene; they’re competitive advantages in a world where infrastructure complexity keeps growing.

Teams with robust day 2 operations move faster because they have confidence in their infrastructure state. They recover quickly from incidents because they can immediately identify what changed. They maintain security without slowing development by automatically detecting and reverting unauthorized modifications. Most importantly, they scale operations without scaling headcount, automating work that would otherwise require armies of engineers manually checking infrastructure.

The math is compelling. A single undetected security group change could lead to a breach costing millions. One overlooked configuration drift might cause hours of downtime. Yet implementing comprehensive drift detection takes just hours of setup and minutes of ongoing maintenance. The ROI is immediate and substantial.

Our IDP journey continues with “Extend Your IDP for AI Applications: GPUs, Models, and Cost Controls”. As AI workloads become central to modern applications, we’ll explore how to adapt your platform for machine learning workflows. You’ll learn about GPU orchestration, model deployment pipelines, and the unique cost management challenges that AI infrastructure presents.

But don’t wait for the next post to get started with drift detection. Even a simple hourly check can prevent major incidents. Set up basic detection today, and your future self will thank you the next time you’re debugging a production issue at 3 AM. More importantly, your on-call team will appreciate not having to debug issues caused by accumulated drift that could have been prevented.

Ready to implement drift detection? Start your free Pulumi Cloud trial and set up your first drift detection schedule in minutes. No credit card required.

How to Implement Robust Security Guardrails Using Policy as Code

Adam Gordon Bell — Tue, 30 Sep 2025 00:00:00 +0000

Welcome to the third post in our IDP Best Practices series, where we explore how to implement policy as code with Pulumi CrossGuard to create deployment guardrails that make self-service infrastructure both powerful and safe.

Platform engineering presents a fundamental tension: we want to enable developer velocity while maintaining security and compliance. Every platform team faces the same question: how do you give teams the freedom to deploy infrastructure quickly without compromising on safety, security, or organizational standards? The answer isn’t to choose between speed and safety, but rather to embrace automated guardrails powered by policy as code that make both possible simultaneously.

This post is part of our IDP Best Practices series:

How to Build an Internal Developer Platform: Strategy, Best Practices, and Self-Service Infrastructure
Build Golden Paths with Infrastructure Components and Templates
Deployment Guardrails with Policy as Code (you are here)
Day 2 Operations: Drift Detection and Remediation
Extend Your IDP for AI Applications: GPUs, Models, and Cost Controls
Next-Gen IDPs: How to Modernize Legacy Infrastructure with Pulumi

Want hands-on experience? Access the complete demo code and policy examples from this workshop.

The Platform Engineering Challenge: Speed vs. Safety

Let me tell you a story that perfectly captures the platform engineering dilemma. Statsig, a fast-growing feature flag platform processing an incredible 2 trillion events daily, had one infrastructure engineer named Jason who handled all infrastructure requests. Everything worked smoothly until Jason went on parental leave. Suddenly, the entire engineering organization ground to a halt. Infrastructure requests piled up, deployments slowed to a crawl, and the team faced a stark realization: their entire infrastructure capability depended on a single person.

This crisis forced them to confront a fundamental question that every growing engineering team eventually faces: How do you enable self-service infrastructure without sacrificing security, compliance, or operational stability? The answer wasn’t to hire more Jasons or to lock down infrastructure even tighter. Instead, they discovered that deployment guardrails with policy as code could transform their infrastructure conversation from “talk to the infrastructure person” to “ship with confidence, knowing guardrails will catch any issues.”

Think of it like building a system of roads with guardrails. When you create well-designed infrastructure components with proper safety barriers, teams can drive fast and confidently, knowing they’re protected from going off the cliff. The guardrails don’t slow them down; they enable speed by removing fear.

Understanding Platform Engineering Layers

Before we dive into implementing guardrails, we need to understand the architecture of a modern platform and where policies fit within it. Think of your platform as having three distinct layers, each with different security and access requirements.

At the foundation, you have your Layer 1: Foundational Infrastructure, which includes security controls, shared networking, and identity management systems like OIDC and IAM. Platform teams typically manage this layer with strict access controls because it forms the security backbone of your entire infrastructure.

Building on that foundation is Layer 2: Shared Infrastructure, which encompasses VPCs, compute platforms, and load balancers. These are standardized components that teams can use but with some room for customization based on specific needs.

The most dynamic layer is Layer 3: The Workloads Layer, where deployable artifacts, pipelines, and operability tools live. This is where the rubber meets the road, where most self-service activity happens and, consequently, where guardrails become absolutely critical. Developers need maximum freedom at this layer to innovate and ship quickly, but this freedom also introduces the most risk. It’s precisely here that policy-driven guardrails prove their worth, enabling that freedom while automatically preventing dangerous configurations.

What Are Deployment Guardrails?

Deployment guardrails are automated policies that act as your infrastructure’s safety net. Rather than relying on manual reviews or hoping developers remember all the security requirements, guardrails automatically prevent misconfigurations before they reach production. They enforce security standards without human intervention, guide developers toward best practices through immediate feedback, and enable safe self-service by catching issues at the earliest possible moment.

A helpful analogy is to think of guardrails like type checking in programming languages. Just as TypeScript doesn’t restrict your ability to write JavaScript but rather catches type errors before runtime, deployment guardrails don’t limit your infrastructure creativity. They simply ensure you’re following secure patterns and catch potentially dangerous configurations before they cause problems in production.

Introducing Pulumi CrossGuard: Policy as Code

Pulumi CrossGuard is Pulumi’s policy as code framework that brings the same engineering rigor to compliance and security that you apply to your application code. Instead of maintaining policy documents in wikis or relying on manual reviews, you can write policies in familiar programming languages like Python, TypeScript, or Go. These policies then enforce themselves across all your cloud resources and providers, running at different stages of the deployment lifecycle and integrating seamlessly with your CI/CD pipelines for automated enforcement.

Key Policy Types

CrossGuard supports two fundamental types of policies, each serving different validation needs:

Resource Policies: Validate individual resources

def restrict_dangerous_ports(args: ResourceValidationArgs, report_violation: ReportViolation):
 if args.resource_type == "aws:lb/targetGroup:TargetGroup":
 port = args.props.get("port")
 dangerous_ports = ["22", "23", "3389"]
 if str(port) in dangerous_ports:
 report_violation("Dangerous port detected. Avoid using SSH, Telnet, or RDP ports.")

Stack Policies: Validate relationships across resources

def validate_microservice_encryption(args: StackValidationArgs, report_violation: ReportViolation):
 microservice_components = []
 s3_buckets = []

 for resource in args.resources:
 if resource.resource_type == "custom:infrastructure:Microservice":
 microservice_components.append(resource)
 elif resource.resource_type == "aws:s3/bucket:Bucket":
 s3_buckets.append(resource)

 if microservice_components and s3_buckets:
 for bucket in s3_buckets:
 encryption = bucket.props.get("serverSideEncryptionConfiguration")
 if not encryption:
 report_violation("S3 bucket must have encryption enabled when used with microservice components.")

Building Practical Guardrails: Real-World Examples

Let’s walk through building guardrails for a microservice platform, starting with the component from our previous workshop.

Example 1: Port Security Policy

Problem: Developers might accidentally expose services on dangerous ports like SSH (22) or RDP (3389).

Solution: A policy that blocks dangerous port configurations.

import pulumi_policy as policy

def restrict_dangerous_ports_validation(args, report_violation):
 """Prevent services from running on dangerous ports."""
 if args.resource_type == "aws:lb/targetGroup:TargetGroup":
 port = args.props.get("port")
 dangerous_ports = ["22", "23", "3389", "5432", "3306"]

 if str(port) in dangerous_ports:
 report_violation(
 f"Port {port} is not allowed. This port is commonly used for "
 f"administrative services and should not be exposed via load balancer."
 )

restrict_dangerous_ports = policy.ResourceValidationPolicy(
 name="restrict-dangerous-ports",
 description="Prevent services from using dangerous ports",
 validate=restrict_dangerous_ports_validation,
 enforcement_level=policy.EnforcementLevel.MANDATORY
)

Example 2: Resource Limits Policy

Problem: Teams might request oversized resources, leading to cost overruns.

Solution: A policy that enforces reasonable resource limits with advisory warnings.

def limit_memory_validation(args, report_violation):
 """Limit memory allocation for microservices."""
 if args.resource_type == "custom:infrastructure:Microservice":
 # Extract memory from component tags or properties
 memory_str = args.props.get("memory", "512Mi")

 # Parse memory value (assuming format like "2Gi", "1024Mi")
 if memory_str.endswith("Gi"):
 memory_gb = float(memory_str[:-2])
 if memory_gb > 1:
 report_violation(
 f"Memory allocation of {memory_str} exceeds recommended limit of 1Gi "
 f"for microservices. Consider optimizing your application or "
 f"contact the platform team for exceptions."
 )

limit_memory = policy.ResourceValidationPolicy(
 name="limit-microservice-memory",
 description="Enforce reasonable memory limits for microservices",
 validate=limit_memory_validation,
 enforcement_level=policy.EnforcementLevel.ADVISORY
)

Example 3: Cross-Resource Security Policy

Problem: When microservices use S3 buckets, encryption should be mandatory.

Solution: A stack policy that validates encryption across related resources.

def microservice_s3_encryption_validation(args, report_violation):
 """Ensure S3 buckets used with microservices have encryption enabled."""
 microservice_resources = []
 s3_buckets = []

 # Collect relevant resources
 for resource in args.resources:
 if resource.resource_type == "custom:infrastructure:Microservice":
 microservice_resources.append(resource)
 elif resource.resource_type == "aws:s3/bucket:Bucket":
 s3_buckets.append(resource)

 # If we have both microservices and S3 buckets, check encryption
 if microservice_resources and s3_buckets:
 for bucket in s3_buckets:
 sse_config = bucket.props.get("serverSideEncryptionConfiguration")
 if not sse_config:
 report_violation(
 f"S3 bucket '{bucket.name}' must have server-side encryption "
 f"enabled when used with microservice components."
 )

microservice_s3_encryption = policy.StackValidationPolicy(
 name="microservice-s3-encryption",
 description="Ensure S3 buckets used with microservices are encrypted",
 validate=microservice_s3_encryption_validation,
 enforcement_level=policy.EnforcementLevel.MANDATORY
)

Policy Enforcement Models

Pulumi CrossGuard supports multiple enforcement models to fit different workflows, and understanding when to use each model is crucial for effective policy implementation.

The Preventative Model

The most common approach is the preventative model, where policies run before resources are created. This gives you fast feedback and prevents bad deployments from ever reaching the cloud. When you run pulumi preview, policies are evaluated immediately, and if violations are found during pulumi up, the deployment is blocked. This model works particularly well for validating port configurations, resource limits, and other settings that you can check without needing to see the actual deployed resource.

# Policies run automatically during preview and deployment
pulumi preview # Policies evaluated here
pulumi up # Policies block deployment if violations found

The Detective Model

Sometimes you need to validate the actual cloud state, including auto-generated values like ARNs or IP addresses. That’s where the detective model comes in. These policies run after resources are created, making them perfect for compliance scanning and drift detection. While they can’t prevent initial deployment of non-compliant resources, they excel at catching configuration drift and validating properties that only exist after deployment.

CI/CD Integration

The third model integrates policies directly into your deployment pipeline. This ensures consistent enforcement across all teams and creates natural deployment gates. For example, you might configure GitHub Actions to run policy validation on every pull request, blocking merges if violations are found. This approach combines the best of both worlds: early feedback during development and guaranteed enforcement before production.

# GitHub Actions example
- name: Run Policy Validation
 run: |
 pulumi preview --policy-pack ./policies
 if [ $? -ne 0 ]; then
 echo "Policy violations found. Deployment blocked."
 exit 1
 fi

Policy Remediation: Beyond Detection

Modern policy frameworks don’t just detect violations; they can automatically fix them:

def auto_tag_resources(args, report_violation):
 """Automatically add required tags to resources."""
 if args.resource_type == "aws:s3/bucket:Bucket":
 tags = args.props.get("tags", {})

 if "Department" not in tags:
 # Instead of just reporting, fix the issue
 if not hasattr(args, 'tags'):
 args.props["tags"] = {}
 args.props["tags"]["Department"] = "Engineering"
 return args.props # Return modified properties

 return None # No changes needed

auto_tag_policy = policy.ResourceValidationPolicy(
 name="auto-tag-resources",
 description="Automatically add required tags",
 validate=auto_tag_resources,
 enforcement_level=policy.EnforcementLevel.REMEDIATE
)

Server-Side Policy Enforcement

For enterprise deployments, Pulumi provides server-side policy enforcement that ensures policies can’t be bypassed. The process starts by publishing your policies to your Pulumi organization with pulumi policy publish ./my-policies. Once published, you can create policy groups that combine multiple policies with specific enforcement levels, targeting particular stacks or environments while configuring exceptions for special cases. The beauty of this approach is that policies run automatically without requiring CLI flags, providing consistent governance across your entire organization without relying on developers to remember to include policy packs in their commands.

Compliance-Ready Policies

While custom policies address your specific organizational needs, compliance requirements often follow industry standards. Pulumi provides hundreds of pre-built policies for common compliance frameworks:

import { PolicyManager } from "@pulumi/policy";
import { policyManager } from "@pulumi/compliance-ready-policies/policy-manager";

new PolicyPack("aws-compliance-ready-policies-typescript", {
 policies: [
 ...policyManager.selectPolicies({
 vendors: ["aws"],
 services: ["ec2", "s3"],
 severities: ["medium", "high", "critical"],
 topics: ["encryption"],
 frameworks: ["pcidss"],
 }, "mandatory"),
 ],
});

This automatically includes policies for major compliance frameworks like PCI DSS for payment card industry standards, SOC 2 for security and compliance controls, ISO 27001 for information security management, and CIS Benchmarks for security configuration standards.

Best Practices for Policy Implementation

After implementing policies at dozens of organizations, we’ve learned that successful policy adoption follows predictable patterns. The key is to adopt a product mindset: think of your policies as a product serving your development teams, not as bureaucratic rules imposed from above. Start small and iterate based on real feedback rather than trying to implement a comprehensive policy framework from day one.

Start Small and Iterate

Begin with just two or three critical policies that address your most pressing risks. Use advisory enforcement initially, which warns developers about violations but doesn’t block deployments. This gives your team time to understand and adapt to the policies. Only after gathering feedback and refining the policies should you graduate to mandatory enforcement.

Provide Clear, Actionable Error Messages

Nothing frustrates developers more than cryptic policy violations. Your error messages should explain not just what’s wrong, but how to fix it:

def good_error_message(args, report_violation):
 if violation_detected:
 report_violation(
 "S3 bucket encryption is required for compliance. "
 "Add serverSideEncryptionConfiguration to fix this. "
 "See: https://docs.company.com/s3-encryption"
 )

Embrace Progressive Enforcement

Think of enforcement levels as a dial, not a switch. Start with advisory mode to warn about issues, move to mandatory to block deployments, and eventually implement remediation to automatically fix common issues. This progression gives teams time to adapt while gradually raising the security bar.

Test Your Policies Thoroughly

Policies are code, and like any code, they need testing. Write unit tests for your policies to ensure they catch violations correctly and don’t produce false positives:

# Test policies with unit tests
def test_dangerous_port_policy():
 # Mock resource with port 22
 args = create_mock_args("aws:lb/targetGroup:TargetGroup", {"port": 22})
 violations = run_policy(restrict_dangerous_ports, args)
 assert len(violations) == 1
 assert "dangerous port" in violations[0].message.lower()

Document Escape Hatches

No matter how well-designed your policies are, there will always be edge cases. Rather than forcing teams to work around policies in dangerous ways, provide clear, documented escape hatches. Create processes for requesting exceptions, handling emergency exemptions, and appealing for policy changes. As one platform engineer put it, you need “guardrails outside the guardrails” because platform adoption will never be 100%, and that’s okay. This transparency builds trust and ensures policies remain effective enablers rather than bureaucratic obstacles to circumvent.

Real-World Success: Statsig’s Transformation

Remember Statsig’s infrastructure crisis when Jason went on parental leave? Let’s look at how they transformed their platform using the guardrails approach we’ve been discussing. After implementing policy-driven guardrails, their infrastructure story completely changed.

As Tyrone Wong, Infrastructure Engineer at Statsig, explains: “We had a single infra owner. Parental leave forced us to build self-service, and that’s when everything clicked. Developers open a PR, Pulumi previews the change right in the PR, and CI blocks risky changes. That’s how you move fast safely. The magic was turning ’talk to the infra person’ into ‘ship with guardrails.’”

The results speak for themselves. Deployment times dropped from 1.5 weeks to mere minutes. The infrastructure ticket queue that once haunted their Slack channels disappeared entirely. Security actually improved through automated policy enforcement, eliminating the human error factor. Most importantly, developer satisfaction soared as teams gained the autonomy to ship infrastructure changes without friction or fear.

When asked if developers were actually using the self-service platform, Tyrone’s response was telling: “Yeah, people are actually using it because there’s changes they wanted to get done. Previously it became this giant effort with lots of meetings, people being concerned, and Google docs about what the changes would be. Now people can just put together a PR, we have our discussions there, see if anything gets flagged, and move forward.”

Building Your Policy Strategy

Implementing deployment guardrails isn’t a big-bang transformation; it’s a journey that unfolds in phases. Based on patterns we’ve seen across successful implementations, the first week or two should focus on assessment. Start by understanding your current state, auditing existing infrastructure patterns to identify what teams are actually deploying. Look for common misconfigurations that have caused incidents or near-misses, and document your security and compliance requirements as specific, enforceable rules. Most importantly, survey your developers to understand their pain points with the current infrastructure process, as this assessment forms the foundation for policies that solve real problems rather than creating new ones.

In weeks three and four, build your foundation by implementing three to five core policies that address your most critical risks. Set up CI/CD integration so policies run automatically on every pull request, starting with advisory enforcement to gather feedback without blocking deployments. Create clear documentation and runbooks that explain not just what the policies do, but why they exist and how to work with them.

By the second month, you’re ready to expand. Add compliance-specific policies for regulatory requirements and implement server-side enforcement to ensure policies can’t be bypassed. Create formal processes for policy exemptions and exceptions, and begin measuring policy effectiveness through metrics like violation rates and remediation times.

Remember that policy implementation is never “done.” Continuously monitor violation patterns to identify areas where policies might be too strict or too lenient. Refine policies based on developer feedback and incident data, add automated remediation for common violations to reduce manual fixes, and gradually expand coverage to new services and teams using lessons learned from early adopters.

Measuring Policy Success

You can’t improve what you don’t measure, and policy effectiveness is no exception. Successful teams track metrics across three critical dimensions to validate their policy strategy.

From a security perspective, start by monitoring your policy violation rate: the percentage of deployments that trigger policy warnings or blocks. This tells you whether your policies are catching real issues or creating unnecessary friction. Track time to remediation to understand how quickly teams fix violations when they occur. Most importantly, measure the reduction in security incidents caused by misconfigurations. If your policies aren’t reducing incidents, they’re not addressing the right risks.

Security is only half the equation, though. The developer experience dimension is equally important. Monitor deployment velocity to ensure policies aren’t slowing teams down. Regular developer satisfaction surveys reveal whether teams see policies as helpful guardrails or annoying obstacles. Track support ticket volume; successful policy implementation should reduce infrastructure-related requests as teams become more self-sufficient.

Finally, don’t forget to measure the operational health of your policy system itself. Policy coverage shows what percentage of your resources are protected by policies, while the exemption rate reveals whether policies are practical or if teams constantly need exceptions. Track remediation automation to see how many violations are fixed automatically versus requiring manual intervention. Together, these metrics paint a complete picture of your policy program’s health and help you continuously improve your approach.

Common Pitfalls and How to Avoid Them

Even with the best intentions, policy implementations can go wrong. Through years of helping organizations implement policy as code, we’ve identified four critical pitfalls that can derail your efforts.

The first and most common trap is over-engineering policies. It’s tempting to create sophisticated policies that handle every edge case, but overly complex policies are hard to understand and maintain. Teams won’t trust what they don’t understand. Instead, start with simple, clear policies that address specific risks. Use descriptive naming and provide examples of both compliant and non-compliant configurations. Remember, a simple policy that everyone follows is better than a perfect policy that everyone circumvents.

Equally dangerous is falling into “policy theater”: implementing policies because you think you should, not because they address real risks. This creates friction without improving security. Every policy should trace back to an actual incident, security requirement, or compliance need. If you can’t explain why a policy exists with a concrete example, it probably shouldn’t exist.

Poor developer experience is another common failure point. Nothing kills policy adoption faster than cryptic error messages and unclear remediation steps. When a developer hits a policy violation at 5 PM on a Friday, they need clear guidance on how to fix it. Invest heavily in documentation, helpful error messages, and examples. Consider your policies’ error messages as part of your product’s user experience.

Finally, inadequate testing can erode trust quickly. Policies that break legitimate use cases will be worked around or disabled. Before enforcing any policy, test it thoroughly with real scenarios from your production workloads. Run policies in advisory mode first to catch false positives, and have teams attempt real deployments with policies enabled to ensure they don’t block valid configurations.

The Future of Policy as Code

As infrastructure becomes increasingly complex and distributed, policy as code is evolving beyond simple rule enforcement to become an intelligent layer that understands context and intent. Three major trends are shaping this evolution.

First, we’re seeing the emergence of AI-enhanced policies with smart detection systems that use machine learning to identify anomalies that rule-based policies might miss. These systems learn from your infrastructure patterns and can provide contextual recommendations that adapt based on actual usage. Even more exciting is predictive enforcement: imagine policies that can identify risky patterns before they become violations, guiding developers away from problems they haven’t encountered yet.

Second, the multi-cloud reality is driving the need for universal governance. Organizations need policies that work seamlessly across AWS, Azure, and GCP, with federated enforcement that maintains consistency across multiple cloud accounts and regions. Compliance automation is also maturing, with systems that automatically collect evidence for audits and generate compliance reports without manual intervention.

Finally, policy enforcement is moving closer to where developers actually work. IDE integration will soon provide real-time policy feedback as you write infrastructure code, catching issues before you even attempt to deploy. Self-service exemption workflows will let developers request and receive policy exceptions through automated approval processes. Perhaps most intriguingly, we’re seeing the development of learning policies: systems that improve and adapt based on developer feedback and usage patterns, becoming more helpful over time rather than more restrictive.

Conclusion: Enabling Safe Self-Service at Scale

We started this post with a fundamental tension in platform engineering: the need for both speed and safety. Through the lens of Statsig’s transformation and the technical deep-dive into Pulumi CrossGuard, we’ve seen that this isn’t actually a tension that needs resolving. It’s a false dichotomy that policy as code eliminates entirely.

The key insight from successful platform teams like Statsig is that guardrails don’t restrict freedom; they enable it. When developers know that automated policies will catch dangerous configurations, they gain the confidence to move fast and experiment. When platform teams know that policies automatically enforce security and compliance standards, they can focus on building better platforms instead of reviewing every change. This is the magic of policy as code: it transforms infrastructure governance from a bottleneck into an accelerator.

But perhaps the most important lesson is that policy as code isn’t about saying “no” to developers. It’s about intelligent automation that makes the secure path the path of least resistance. It’s about catching mistakes before they become incidents. It’s about encoding your organization’s hard-won knowledge into systems that help every developer benefit from that experience.

As you embark on your own journey to implement deployment guardrails, remember that perfection isn’t the goal; progress is. Start small, iterate based on feedback, and gradually expand your coverage. Your developers will thank you for the clarity and confidence that comes with well-designed guardrails, and your security team will sleep better knowing that policies are enforced automatically and consistently.

The path from manual reviews to automated guardrails is well-traveled and well-documented. Our complete policy examples provide real-world implementations you can adapt to your needs, while the CrossGuard documentation offers deep technical details for advanced use cases. If you’re on AWS, AWSGuard’s pre-built policies offer immediate value, and our compliance-ready policy catalog addresses specific regulatory requirements.

The future of infrastructure management isn’t about choosing between developer autonomy and operational control. It’s about using policy as code to achieve both, creating platforms that are simultaneously powerful and safe, flexible and compliant, fast and secure.

In our next post, we’ll explore Day 2 Platform Operations, diving into how to maintain infrastructure compliance after deployment and automatically remediate configuration drift. Because getting to production is just the beginning; keeping your infrastructure secure and compliant over time is where the real challenge lies.

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

Golden Paths in IDPs: A Complete Guide to Reusable Infrastructure with Pulumi Components and Templates

Engin Diri — Wed, 20 Aug 2025 10:00:00 +0200

Welcome to the second post in our IDP Best Practices series. In this article, we explore how to create golden paths, pre-architected, reusable infrastructure patterns that help standardize and accelerate cloud development.

Modern cloud platforms offer endless options, over 200 AWS services, sprawling Azure catalogs, and countless DevOps tools. The result? Developers face decision fatigue and inconsistent implementations. Golden paths solve this by providing ready-to-use, production-grade infrastructure that encodes your organization’s best practices, security policies, and operational standards.

In this guide, you’ll learn how to build golden paths for your Internal Developer Platform using two core Pulumi constructs: Components, reusable infrastructure building blocks, and Templates, predefined, deployable patterns. You’ll see how to create infrastructure abstractions that are written once, shared across teams, and consumed in any language, turning weeks of setup into minutes of developer-ready infrastructure.

This post is part of our IDP Best Practices series:

How to Build an Internal Developer Platform: Strategy, Best Practices, and Self-Service Infrastructure
Build Golden Paths: Guide to Reusable Infrastructure with Pulumi Components and Templates (you are here)
Policy as Code for Safer IDPs: Enabling Developer Self-Service with Guardrails
Day 2 Operations: Drift Detection and Remediation
Extend Your IDP for AI Applications: GPUs, Models, and Cost Controls
Next-Gen IDPs: How to Modernize Legacy Infrastructure with Pulumi

The complete code examples from this post are available on GitHub.

The Platform Engineering Layer Cake: A Model for IDPs

To understand where golden paths fit into an Internal Developer Platform (IDP), think in layers. This three-tier model structures your platform to deliver increasing levels of abstraction, reuse, and developer value:

Layer 1: Infrastructure Layer

This is the foundation: raw cloud resources include VMs, databases, networks, and storage, which are the fundamental building blocks from AWS, Azure, GCP, and other providers. Pulumi gives you programmatic access to these resources through native providers, but working at this level requires deep infrastructure knowledge.

Layer 2: Platform Layer - Components

This is where the magic happens. Pulumi Components take those raw resources and package them into higher-level abstractions. Instead of manually configuring 20+ AWS resources for a secure web application, you create a component that handles all that complexity and exposes just the configuration that matters:

const app = new SecureWebApplication("my-app", {
 instanceType: "t3.medium",
 minReplicas: 2,
 maxReplicas: 10,
 enableWAF: true,
 environment: "production"
});

Part 1: Building Reusable Infrastructure Components

Reusable infrastructure components act as the foundation of your platform. They encapsulate complexity while remaining composable and scalable. Let’s build a real-world example: a component that deploys containerized microservices to AWS Fargate.

The Power of Multi-Language Components

Here’s what makes Pulumi components revolutionary: write once, consume anywhere. Your platform team can author components in TypeScript, but application teams can consume them in Python, Go, .NET, Java, or even YAML. This separation of concerns is crucial for scaling platform engineering across diverse teams.

Building a Microservice Component Step-by-Step

Let’s create a component that encapsulates everything needed to deploy a production-ready microservice:

// microservice-component.ts
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

export interface MicroserviceArgs {
 appPath: string; // Path to application code
 cpu?: number; // CPU units (256, 512, 1024, etc.)
 memory?: number; // Memory in MB
 port?: number; // Application port
 desiredCount?: number; // Number of tasks
}

export class MicroserviceComponent extends pulumi.ComponentResource {
 public readonly url: pulumi.Output<string>;
 public readonly serviceName: pulumi.Output<string>;
 public readonly clusterName: pulumi.Output<string>;

 constructor(name: string, args: MicroserviceArgs, opts?: pulumi.ComponentResourceOptions) {
 super("custom:infrastructure:Microservice", name, {}, opts);

 // Set defaults
 const cpu = args.cpu || 256;
 const memory = args.memory || 512;
 const port = args.port || 8080;
 const desiredCount = args.desiredCount || 2;

 // Create ECR repository for Docker images
 const repository = new awsx.ecr.Repository(`${name}-repo`, {
 forceDelete: true
 }, { parent: this });

 // Build and push Docker image
 const image = new awsx.ecr.Image(`${name}-image`, {
 repositoryUrl: repository.url,
 context: args.appPath,
 platform: "linux/amd64",
 }, { parent: this });

 // Create Application Load Balancer
 const loadBalancer = new awsx.lb.ApplicationLoadBalancer(`${name}-lb`, {
 defaultTargetGroup: {
 port: port,
 protocol: "HTTP",
 healthCheck: {
 path: "/health",
 interval: 30,
 timeout: 5,
 healthyThreshold: 2,
 unhealthyThreshold: 3,
 }
 }
 }, { parent: this });

 // Create ECS cluster
 const cluster = new aws.ecs.Cluster(`${name}-cluster`, {}, { parent: this });

 // Create Fargate service
 const service = new awsx.ecs.FargateService(`${name}-service`, {
 cluster: cluster.arn,
 taskDefinitionArgs: {
 container: {
 name: name,
 image: image.imageUri,
 cpu: cpu,
 memory: memory,
 essential: true,
 portMappings: [{
 containerPort: port,
 targetGroup: loadBalancer.defaultTargetGroup,
 }],
 },
 },
 desiredCount: desiredCount,
 assignPublicIp: true,
 }, { parent: this });

 // Export outputs
 this.url = pulumi.interpolate`http://${loadBalancer.loadBalancer.dnsName}`;
 this.serviceName = service.service.name;
 this.clusterName = cluster.name;

 // Register outputs
 this.registerOutputs({
 url: this.url,
 serviceName: this.serviceName,
 clusterName: this.clusterName,
 });
 }
}

Documenting for Discoverability and Adoption

To make your component easy to use, add comprehensive documentation and examples to a README.md file:

# MicroserviceComponent

Abstraction for resources needed when using AWS container services.

A component to abstract the details related to:
- Creating a docker image and pushing it to AWS ECR.
- Deploy to ECS Fargate using the docker image.

# Inputs

* appPath: Path to local folder containing the app and Dockerfile.
* port: Port to expose via an ALB.
* cpu (Optional): CPU capacity. Defaults to 256 (i.e. 0.25 vCPU).
* memory (Optional): Memory capacity. Defaults to 512 (i.e. 0.5GB).
* containerName (Optional): Name of the container. Defaults to "my-app".

# Outputs

* publicUrl: The DNS name for the loadbalancer fronting the app.

# Usage
## Specify Package in `Pulumi.yaml`

Add the following to your `Pulumi.yaml` file:
Note: If no version is specified, the latest version will be used.

... omit for brevity ...

This documentation will help developers understand how to use your component effectively, including required inputs, outputs, and example usage.

Publishing Your Component via Private Registry

Once your component is ready, publish it to your Pulumi Private Registry:

# Tag your component version
git tag v1.0.1

# Publish to your organization's registry
pulumi package publish https://registry.pulumi.com/myorg/microservice-component

Now any team in your organization can discover and use your component, regardless of their language preference. All they need is to navigate to the Components section in the Pulumi IDP and search for microservice-component.

Part 2: From Building Blocks to Golden Paths

While components are powerful, they’re just the starting point. Golden path templates layer on opinionated scaffolding, workflows, and compliance best practices that guide developers to production.

Golden Path Maturity: From Zero to Product-Grade Platforms

Teams evolve from ad hoc deployments to mature, productized templates with versioning, metrics, and governance. These maturity levels reflect how deeply your platform enables safe, consistent delivery.

Organizations typically progress through three stages of golden path maturity:

Stage 1: No Golden Paths

Every team reinvents the wheel
Inconsistent practices across projects
High cognitive load on developers
Security and compliance gaps

Stage 2: Dawn of Templates

Basic cookie-cutter templates emerge
Some standardization begins
Manual processes still dominate
Limited support and evolution

Stage 3: Templates as Products

Golden paths have dedicated owners
Regular release cycles and versioning
Migration guides for updates
Metrics track adoption and success
Continuous improvement based on feedback

What Makes a Golden Path Golden?

Drawing from Spotify’s pioneering work, golden paths share these characteristics:

Pre-architected and Supported: The platform team owns and supports the path
Optional but Recommended: Developers can deviate, but staying on the path ensures support
Transparent Abstractions: The implementation is visible, not a black box
Extensible: Teams can add project-specific resources without breaking the pattern
Evolutionary: Templates improve based on feedback and new requirements

Building a Go Microservice Golden Path

Let’s create a complete golden path for Go microservices that includes:

Application scaffolding with best practices
Infrastructure deployment using our component
CI/CD pipeline configuration
Observability setup
Security controls

Step 1: Template Structure

go-microservice-boilerplate/
├── Pulumi.yaml # Infrastructure definition
├── src/ # Application code
│ ├── main.go # Go microservice
│ ├── Dockerfile # Container definition
│ └── go.mod # Dependencies
└── README.md # Documentation

Step 2: Application Scaffolding

The src/main.go file contains a minimal Go microservice with all the best practices and compliance guardrails baked in. This helps developers to quickly get started and extend the application without worrying too much about selecting the right libraries or frameworks.

// source/main.go
package main

import (
 "context"
 "flag"
 "net/http"
 "os"
 "os/signal"
 "time"

 "github.com/labstack/echo/v4"
 "github.com/labstack/echo/v4/middleware"
 "go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho"
 "go.opentelemetry.io/otel"
 "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
 "go.opentelemetry.io/otel/sdk/resource"
 sdktrace "go.opentelemetry.io/otel/sdk/trace"
 semconv "go.opentelemetry.io/otel/semconv/v1.34.0"
)

func initTracer() (*sdktrace.TracerProvider, error) {
 exporter, err := otlptracehttp.New(context.Background())
 if err != nil {
 return nil, err
 }

 resource, err := resource.Merge(
 resource.Default(),
 resource.NewWithAttributes(
 semconv.SchemaURL,
 semconv.ServiceNameKey.String("${PROJECT}"),
 semconv.ServiceVersionKey.String("1.0.0"),
 ),
 )
 if err != nil {
 return nil, err
 }

 tp := sdktrace.NewTracerProvider(
 sdktrace.WithBatcher(exporter),
 sdktrace.WithResource(resource),
 sdktrace.WithSampler(sdktrace.AlwaysSample()),
 )

 otel.SetTracerProvider(tp)
 return tp, nil
}

func echoHandler(c echo.Context) error {
 message := c.QueryParam("message")
 if message == "" {
 message = "Hello from Go microservice!"
 }

 return c.JSON(http.StatusOK, map[string]string{
 "echo": message,
 "service": "${PROJECT}",
 "timestamp": time.Now().UTC().Format(time.RFC3339),
 })
}

func healthHandler(c echo.Context) error {
 return c.JSON(http.StatusOK, map[string]string{
 "status": "healthy",
 "service": "${PROJECT}",
 })
}

func main() {
 healthCheck := flag.Bool("health-check", false, "Run health check and exit")
 flag.Parse()

 if *healthCheck {
 resp, err := http.Get("http://localhost:8080/health")
 if err != nil || resp.StatusCode != http.StatusOK {
 os.Exit(1)
 }
 os.Exit(0)
 }

 tp, err := initTracer()
 if err != nil {
 panic(err)
 }
 defer func() {
 if err := tp.Shutdown(context.Background()); err != nil {
 panic(err)
 }
 }()

 e := echo.New()

 e.Use(middleware.Logger())
 e.Use(middleware.Recover())
 e.Use(middleware.CORS())
 e.Use(otelecho.Middleware("${PROJECT}"))

 e.GET("/echo", echoHandler)
 e.GET("/health", healthHandler)

 go func() {
 if err := e.Start(":8080"); err != nil && err != http.ErrServerClosed {
 e.Logger.Fatal("shutting down the server")
 }
 }()

 quit := make(chan os.Signal, 1)
 signal.Notify(quit, os.Interrupt)
 <-quit
 ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
 defer cancel()
 if err := e.Shutdown(ctx); err != nil {
 e.Logger.Fatal(err)
 }
}

Step 3: Infrastructure Template with YAML

Here’s how you define the infrastructure using Pulumi YAML. You can instantly spot one of the features of Pulumi Components. We consume the micorservice-component in YAML without knowing that it was authored in a different language. This is the power of Pulumi Components: write once, consume anywhere.

And we see another feature of Pulumi Components: As it is a first-class citizen, we can embed the component and add additional resources like auto-scaling policies, alarms, and more directly in the YAML file. This allows us to create a complete golden path for deploying Go microservices on AWS ECS with auto-scaling capabilities.

# Pulumi.yaml
name: ${PROJECT}
description: ${DESCRIPTION}
runtime: yaml
packages:
 component-microservice: https://github.com/smithrobs/component-microservice@v1.0.1
template:
 description: A template for deploying a Go microservice on AWS ECS with auto-scaling capabilities.
resources:
 microserviceComponent:
 type: component-microservice:MicroserviceComponent
 properties:
 appPath: ./src
 port: 8080
 containerName: ${PROJECT}
 ecsTarget:
 type: aws:appautoscaling:Target
 name: ecs_target
 properties:
 maxCapacity: 4
 minCapacity: 1
 resourceId: service/${microserviceComponent.clusterName}/${microserviceComponent.serviceName}
 scalableDimension: ecs:service:DesiredCount
 serviceNamespace: ecs
 ecsPolicyUP:
 type: aws:appautoscaling:Policy
 name: ecs_policy_up
 properties:
 name: ecs_policy_up
 policyType: StepScaling
 resourceId: ${ecsTarget.resourceId}
 scalableDimension: ${ecsTarget.scalableDimension}
 serviceNamespace: ${ecsTarget.serviceNamespace}
 stepScalingPolicyConfiguration:
 adjustmentType: ChangeInCapacity
 cooldown: 60
 metricAggregationType: Maximum
 stepAdjustments:
 - metricIntervalUpperBound: 0
 scalingAdjustment: 1
 ecsPolicyDown:
 type: aws:appautoscaling:Policy
 name: ecs_policy_down
 properties:
 name: ecs_policy_down
 policyType: StepScaling
 resourceId: ${ecsTarget.resourceId}
 scalableDimension: ${ecsTarget.scalableDimension}
 serviceNamespace: ${ecsTarget.serviceNamespace}
 stepScalingPolicyConfiguration:
 adjustmentType: ChangeInCapacity
 cooldown: 60
 metricAggregationType: Maximum
 stepAdjustments:
 - metricIntervalLowerBound: 0
 scalingAdjustment: -1
 serviceCPUHighUtilization:
 type: aws:cloudwatch:MetricAlarm
 name: service_cpu_high_utilization
 properties:
 name: service_cpu_high_utilization
 comparisonOperator: GreaterThanOrEqualToThreshold
 evaluationPeriods: 2
 metricName: CPUUtilization
 namespace: AWS/ECS
 period: 60
 statistic: Average
 threshold: 80
 alarmActions:
 - ${ecsPolicyUP.arn}
 dimensions:
 ClusterName: ${microserviceComponent.clusterName}
 ServiceName: ${microserviceComponent.serviceName}
 serviceCPULowUtilization:
 type: aws:cloudwatch:MetricAlarm
 name: service_cpu_low_utilization
 properties:
 name: service_cpu_low_utilization
 comparisonOperator: LessThanOrEqualToThreshold
 evaluationPeriods: 2
 metricName: CPUUtilization
 namespace: AWS/ECS
 period: 60
 statistic: Average
 threshold: 10
 alarmActions:
 - ${ecsPolicyDown.arn}
 dimensions:
 ClusterName: ${microserviceComponent.clusterName}
 ServiceName: ${microserviceComponent.serviceName}

outputs:
 publicUrl: ${microserviceComponent.publicUrl}

Step 4: Documentation and Examples

Add a README.md file to your template directory to provide clear instructions on how to use the template, including prerequisites, configuration options, and example commands:

# Go Microservice Golden Path

A **golden path** template that gets your Go microservice from code to production on AWS in minutes, not weeks.

## What You Get

This golden path provides everything your development team needs to deploy production-ready Go microservices:

✅ **Production-ready Go microservice** with Echo framework and OpenTelemetry tracing
✅ **AWS infrastructure that scales** - ECS with auto-scaling from 1-4 instances
✅ **Security by default** - Hardened containers, IAM roles, security groups
✅ **Monitoring built-in** - Health checks, load balancer monitoring, CloudWatch alarms
✅ **One-command deployment** - `pulumi up` handles everything
✅ **No AWS expertise required** - Complex ECS setup abstracted away

## For Development Teams: What to Expect

### Your Experience

**Day 1: Getting Started**

- Clone this repo, run `pulumi up`
- Your service is live on AWS in 5-10 minutes
- Public URL provided automatically - no manual setup needed

**Day 2-N: Development Workflow**

- Write your Go code in `microservice/main.go`
- Test locally with `go run main.go`
- Deploy changes with `pulumi up`
- AWS automatically rebuilds and redeploys your container

**Production Operations**

- Service automatically scales with CPU load (80% up, 10% down)
- Health checks ensure unhealthy containers are replaced
- Load balancer distributes traffic across healthy instances
- Distributed tracing helps debug issues across services

### What's Handled For You

You **don't** need to learn or configure:

- ECS clusters, services, and task definitions
- Application Load Balancers and target groups
- Auto-scaling policies and CloudWatch alarms
- Security groups and IAM roles
- Container registries and image building
- Health check configuration

You **do** focus on:

- Writing your Go application logic
- Adding your business endpoints
- Testing your service locally
- Deploying with confidence

Making Templates Available in Pulumi IDP

Publishing your template to Pulumi IDP enables true self-service. Head to Settings → Integrations → Organization Template Sources and add your template repository.

Once published, developers can discover and deploy it directly from the IDP interface.

Now developers can deploy through multiple interfaces:

CLI Deployment:

pulumi new https://github.com/myorg/go-microservice-boilerplate

No-Code Deployment: Navigate to Pulumi IDP → Templates → Deploy with Pulumi → Configure and launch

Pulumi Deployment: Navigate to Pulumi IDP → Templates → Deploy with Pulumi → Configure and launch

Best Practices for Reusable Infrastructure Components and Templates

Well-designed components and templates are the foundation of scalable, self-service infrastructure. These best practices ensure your abstractions are maintainable, discoverable, and production-ready.

1. Design for Day 2 Operations from Day 1

Think beyond deployment. A golden path must also support ongoing operations:

How will teams safely update their infrastructure?
What happens during scaling events?
How do you handle disaster recovery?
What metrics and logs are needed for troubleshooting?

2. Expose Complexity Progressively

Provide sensible defaults that work, while enabling customization for advanced use cases:

interface ComponentArgs {
 // Required - what users must provide
 appName: string;

 // Common customizations with good defaults
 instanceType?: string; // default: "t3.micro"
 replicas?: number; // default: 2

 // Advanced options for power users
 networkConfig?: NetworkConfig;
 securityPolicies?: SecurityPolicy[];
 customMetrics?: MetricDefinition[];
}

4. Use Semantic Versioning Everywhere

Clear versioning for both components and templates indicate stability and reliability:

Major versions (1.0.0 → 2.0.0): Breaking changes
Minor versions (1.0.0 → 1.1.0): New features, backward compatible
Patch versions (1.0.0 → 1.0.1): Bug fixes

This lets teams adopt updates with confidence and control.

5. Test Your Abstractions

Don’t ship black boxes. Create automated tests to validate key functionality and resource creation. Focus on:

Smoke tests that validate resource existence
Output validation to ensure correctness
Integration tests to confirm end-to-end behavior

import { expect } from "chai";
import * as pulumi from "@pulumi/pulumi";

describe("MicroserviceComponent", () => {
 it("should create required resources", async () => {
 const component = new MicroserviceComponent("test", {
 appPath: "./test-app",
 port: 3000
 });

 const resources = await pulumi.runtime.allResources();
 expect(resources).to.include("aws:ecs/cluster:Cluster");
 expect(resources).to.include("aws:ecs/service:Service");
 expect(resources).to.include("aws:lb/loadBalancer:LoadBalancer");
 });
});

How to Measure Golden Path Success

Golden paths aren’t complete until they deliver measurable value. Use these KPIs to assess performance and drive iteration:

Adoption Metrics

Template usage rate: Percentage of new projects using golden paths
Component reuse: Number of stacks consuming shared components
Time to first deployment: Deployment time and frequency from code to production

Quality Metrics

Security compliance rate: Percentage of deployments passing security policies
Stability: Deployment incident frequency
Mean time to recovery (MTTR): Time to recover from production issues

Developer Experience Metrics

Developer satisfaction: Survey teams about their platform experience
Support ticket volume: Ticket volume related to infrastructure
Contribution: Number of PRs or issues submitted to platform templates

Real-World Results: Success Stories

Organizations using golden paths report significant improvements in speed and reliability:

Snowflake reduced deployment time from 1.5 weeks to less than a day
Mercedes-Benz decreased infrastructure provisioning from weeks to minutes
Starburst Data cut deployment time from 2 weeks to 3 hours

These results show that golden paths are not just developer tools, they’re a competitive advantage. To learn more, download the whitepaper: The Golden Path to Cloud Success: Your IDP Roadmap.

Common Pitfalls and How to Avoid Them

Pitfall 1: Over-Abstraction

Problem: Creating components so abstract they’re unusable Solution: Start with concrete use cases, then generalize based on actual patterns

Pitfall 2: Insufficient Escape Hatches

Problem: Golden paths become golden cages Solution: Always provide ways to extend or override default behavior

Pitfall 3: Poor Versioning Strategy

Problem: Breaking changes without migration paths Solution: Maintain backward compatibility and provide clear upgrade guides

Pitfall 4: Lack of Ownership

Problem: Templates become orphaned and outdated Solution: Assign clear ownership and establish maintenance schedules

The Future of Golden Paths: AI and Beyond

The future of golden paths is intelligent, cross-cloud, and fully integrated with modern workflows.

AI-Enhanced Templates

Imagine templates that adapt based on your application’s actual behavior, automatically tuning resources and configurations for optimal performance and cost.

Cross-Cloud Portability

Components that abstract not just resources but entire cloud providers, enabling true multi-cloud golden paths.

GitOps-Native Workflows

Templates that include not just infrastructure but complete GitOps pipelines, from code commit to production deployment.

Your First Steps to Golden Paths

Ready to get started? Here’s your action plan:

Audit Current Patterns: Document the infrastructure patterns your teams use most frequently
Build one reusable component: Pick your most common pattern and build a reusable component
Create Your First Template: Build a complete golden path for one project type
Gather Feedback: Deploy with a pilot team and iterate based on their experience
Scale Gradually: Expand your component library and template catalog based on demand
Measure and Iterate: Track adoption and continuously improve based on metrics

Conclusion: From Fragmentation to Flow

Golden paths aren’t about restricting creativity or enforcing rigid standards. They’re about removing friction from the development process, encoding expertise into reusable patterns, and empowering developers to move at the speed of business.

By building a library of components and templates, you transform your Internal Developer Platform from a collection of tools into a force multiplier for your entire engineering organization. You give developers the gift of not having to solve solved problems, while maintaining the flexibility to innovate where it matters.

Start small. Solve one problem well. Expand from there. With the right foundation, your developer platform will evolve into a system of self-service, speed, and stability.

Ready to Build Your Golden Paths?

Code examples: Explore golden path examples on GitHub
Build with Components: Pulumi Components documentation
Enable Self-Service: Pulumi IDP documentation
Join the Community: Connect with platform engineers in our Pulumi Slack

Next in our series: Policy as Code for Safer IDPs. You will learn how to add automated guardrails that make sure every deployment meets your security and compliance standards without slowing down development.

How to Build an Internal Developer Platform: Strategy, Best Practices, and Self-Service Infrastructure

Mitch Gerdisch — Thu, 14 Aug 2025 05:13:02 +0200

Welcome to the first post in our IDP Best Practices series. In this guide, we’ll walk through the strategic foundations for designing an Internal Developer Platform that empowers developers without sacrificing governance, security, or operational control.

At Pulumi, we’ve worked with hundreds of teams facing the same core challenge: How do you give developers the infrastructure access they need, while maintaining the governance and security your organization requires?

That tension is at the heart of every IDP conversation. Teams want to move faster and innovate, but also need to stay compliant, control costs, and maintain operational stability.

The good news? You can do both, with a clear strategy and the right approach. This series shares proven best practices for designing, building, and scaling IDPs using Pulumi.

These lessons come from real-world implementations across industries and company sizes—and are built to grow with you.

This post is part of our IDP Best Practices series. You can explore the full series below:

How to Build an Internal Developer Platform: Strategy, Best Practices, and Self-Service Infrastructure (you are here)
Build Golden Paths: Guide to Reusable Infrastructure with Pulumi Components and Templates
Policy as Code for Safer IDPs: Enabling Developer Self-Service with Guardrails
Day 2 Operations: Drift Detection and Remediation
Extend Your IDP for AI Applications: GPUs, Models, and Cost Controls
Next-Gen IDPs: How to Modernize Legacy Infrastructure with Pulumi

Understanding the Platform Engineering Layers in Your Internal Developer Platform

When we work with customers on their platform strategy, we often start by referring to the “platform engineering layer cake.” Here is a quick walkthrough of each layer and how Pulumi IDP constructs fit this layer cake approach:

Layer 1: Infrastructure Layer - This is your raw cloud resources: VMs, databases, networks, storage. These are the fundamental building blocks that exist in AWS, Azure, GCP, and other cloud providers.

Layer 2: Platform Layer - This is where Pulumi Components live. Components take those raw infrastructure resources and package them into higher-level abstractions that encapsulate best practices, security policies, and organizational standards. For example, instead of manually configuring 15 different AWS resources to create a secure web application, you create a component that handles all that complexity and exposes just the configuration options that matter to your developers.

Layer 3: Developer Experience Layer - This is where Pulumi Templates and the Private Registry come into play. Templates provide ready-to-deploy patterns that developers can customize, while the private registry makes everything discoverable and manageable at scale.

In our workshop, we focused specifically on those top two layers because that’s where the transformation happens. This is where you turn raw infrastructure into something developers can actually use productively without becoming infrastructure experts themselves.

Here’s how this maps to what you’re actually building:

Infrastructure Layer → Raw Cloud Resources

Your foundation is still the same: AWS EC2 instances, Azure Virtual Networks, GCP Cloud Functions. Pulumi’s providers give you access to these resources using real programming languages instead of YAML or proprietary DSLs.

Platform Layer → Pulumi Components

This is where you create your reusable building blocks. A component might encapsulate:

A complete microservice infrastructure pattern (load balancer, auto-scaling group, database, monitoring)
A secure data pipeline (storage, processing, access controls, encryption)
A standardized Kubernetes application deployment (ingress, service, deployment, secrets management)

Components are written once in any supported language (TypeScript, Python, Go, .NET, Java) but can be consumed by teams using any of these languages. This is crucial. Your platform team might write components in Go, but your application teams can consume them in Python or TypeScript.

Developer Experience Layer → Templates + Private Registry

Templates are complete, deployable projects that use your components. They provide:

Getting started patterns: “Deploy a new microservice”
Reference architectures: “Deploy a three-tier web application”
Environment patterns: “Set up a complete dev/staging/prod pipeline”

The Private Registry is your distribution mechanism. It’s where components and templates become discoverable, versioned, and manageable. Developers don’t need to hunt through Git repositories or Slack channels to find the right building blocks. Everything is catalogued, documented, and accessible through familiar package management workflows.

IDP Example: Building a Web Application Platform Step-by-Step

Let’s make this concrete with an example from our workshops. Say you want to enable teams to deploy secure web applications:

Infrastructure Layer: You need an Application Load Balancer, EC2 Auto Scaling Group, RDS database, VPC with proper subnets, Security Groups, IAM roles, CloudWatch monitoring, and S3 bucket for static assets. That’s about 20+ individual AWS resources that need to be configured correctly and securely.

Platform Layer (Components): You create a WebApplication component that encapsulates all this complexity. Your component exposes simple configuration options like:

const app = new WebApplication("my-app", {
 instanceType: "t3.medium",
 minSize: 2,
 maxSize: 10,
 databaseSize: "small",
 environment: "production"
});

Behind the scenes, your component handles all the security configurations, networking setup, monitoring, and best practices. You publish this component to your private registry.

Developer Experience Layer (Templates): You create templates like:

web-app-starter: A simple web application using your WebApplication component
microservice-template: API service with database and monitoring
full-stack-template: Frontend, backend, and database in a complete environment

Developers can now run pulumi new web-app-starter and get a production-ready, secure web application in minutes instead of weeks. The template uses your component, which handles all the underlying infrastructure complexity.

This layered approach solves the core challenge we see every platform team struggling with: how do you provide developers with the infrastructure they need without creating operational bottlenecks or compromising security standards.

Why Internal Developer Platforms Matter in 2025 and Beyond

While DevOps brought us incredible advances in how we ship and maintain software, we’ve watched many organizations struggle with the unintended consequences: tool sprawl, Day 2 operational pain, and developers who are drowning in infrastructure complexity.

That’s why platform teams have emerged. You exist to solve these challenges by building tools and workflows that enable your internal customers (the developers) to provision infrastructure and deploy software without getting blocked. But here’s the thing I’ve learned from working with hundreds of platform teams: success isn’t just about the tools you choose. It’s about the strategy behind how you implement them.

5 Core Components of a Successful Internal Developer Platform

Through years of working with platform teams and analyzing what separates successful IDPs from those that struggle to gain adoption, we’ve identified five essential components that every platform strategy needs. These aren’t theoretical concepts. They’re battle-tested patterns that work across organizations of all sizes and industries.

1. Abstractions

Let me be clear about something: abstractions aren’t about dumbing down infrastructure for developers. The abstraction is really where you’re hiding the complexity of the underlying infrastructure from your end users, but you’re doing it intentionally to provide appropriate interfaces for different personas in your organization.

We’ve seen too many platform teams get this wrong by either over-abstracting (creating black boxes that developers can’t customize) or under-abstracting (exposing too much complexity). The sweet spot is creating component resources that encapsulate your infrastructure patterns and best practices into reusable building blocks that can be consumed across different programming languages and deployment scenarios, while still providing escape hatches when needed.

2. Blueprints

Blueprints are your templatized, well-architected patterns that developers can use to bootstrap their infrastructure. But here’s what I want you to understand about blueprints: they’re not just starting points that you throw over the wall to developers. They’re carefully designed patterns that embody your organizational best practices and architectural decisions.

When we see successful platform teams, their templates serve as both accelerators and guardrails. They give developers a fast path to production-ready infrastructure while ensuring that everything they deploy follows your organization’s standards. That’s the power of good blueprint design: speed and compliance working together, not against each other.

3. Workflows

Here’s where many platform teams get trapped: they try to build one workflow to rule them all. But in our experience working with organizations across every industry, you need to support multiple consumption patterns because you have multiple types of users. You need:

No-code workflows for users who want point-and-click deployment (think product managers or junior developers who just need to spin up a database)
Low-code workflows using tools like Pulumi YAML for configuration-driven infrastructure (perfect for developers who understand infrastructure but don’t want to write Go or TypeScript)
Full-code workflows for developers who need maximum flexibility and want to leverage the full power of general-purpose programming languages

The key insight? These aren’t three different platforms. They’re three different interfaces to the same underlying components and templates.

4. Security guardrails

This is probably the most important distinction we can share with you: you want to have guardrails, not gates. Too many platform teams create approval processes and manual checkpoints that completely undermine the self-service promise they’re trying to deliver.

Guardrails are different. They start with the best practices and security built into the reusable abstractions. But even the most secure solutions need a safety net. This safety net is policy-as-code that runs automatically, preventing violations before deployment happens. When we work with customers on implementing Pulumi CrossGuard, we’re not creating new friction. We’re embedding security, compliance, and cost controls directly into the deployment process. The developer gets immediate feedback, and you get the assurance that nothing goes to production without meeting your standards.

5. Self-service

Everything we’ve talked about so far builds toward this moment: enabling developers to provision and manage infrastructure independently. But self-service isn’t just about providing a UI. It’s about carefully orchestrating all the other components to create experiences that are both powerful and intuitive.

When we walked through the demo in our workshop, what you saw was the culmination of thoughtful platform design. Developers can discover available services, deploy infrastructure through whatever interface makes sense for them, and manage their resources over time, all without requiring tickets or manual intervention from platform teams. That’s the promise of a well-designed IDP: developer autonomy without operational chaos.

How to Implement Your IDP Strategy

Supporting Different Personas in Your Internal Developer Platform Design

One of the biggest mistakes we see platform teams make: they try to build for their most sophisticated users first. But here’s what I’ve learned from working with hundreds of organizations: you need to think about all your personas from day one.

In most organizations, you’re serving:

Infrastructure engineers who want full control and flexibility (they’ll use the full-code workflows)
Application developers who need infrastructure but prefer higher-level abstractions (perfect for low-code approaches)
Non-technical users who want simple, point-and-click deployment (your no-code users)

The breakthrough insight that separates successful platform teams from struggling ones is this: all these personas can be served by the same underlying components and templates, just consumed through different interfaces. You don’t need to build three different platforms—you need to build one platform with three different consumption models.

Why a Private Registry Is Key to IDP Adoption

Here’s something we always tell platform teams: if you don’t solve the discoverability problem, your beautiful components and templates will sit unused in Git repositories where no one can find them. That’s why the private registry isn’t just a nice-to-have. It’s the foundation that makes everything else work.

When you establish a private registry as your source of truth for components, templates, providers, and policies, you’re solving two critical problems at once. First, discoverability: developers can actually find and explore what’s available. Second, lifecycle management: you can see where each package is being used, track version drift, and understand the impact of changes before you make them.

The workflow is beautifully simple: you publish standardized building blocks with a single pulumi publish command, and developers discover and consume these assets through familiar package management workflows they already know.

Aligning Your IDP to Organizational Context and Services

One pattern we see in every successful IDP implementation is that the platform reflects how the organization actually works. Your developers don’t think in terms of individual stacks or resources. They think in terms of services, applications, and business functionality.

That’s why we built Pulumi Services - to let teams logically group stacks and ESC environments in ways that make sense to your organization. Pulumi Services improve organization by helping your team model infrastructure in a way that’s familiar - simplifying Day 2 operations by providing additional context like links to relevant dashboards, Slack channels and more. When something breaks at 2 AM, your on-call engineer needs to understand dependencies, track usage, and manage infrastructure in the context of the business services that are actually impacted.

How to Measure the Success of Your Internal Developer Platform

Here’s something that might surprise you: the most successful platform teams we work with don’t measure their success by how many cool features they’ve built. They measure success by the business outcomes they’re driving:

Reduced time-to-market through faster infrastructure provisioning (because at the end of the day, your platform exists to help the business move faster)
Improved developer productivity by eliminating infrastructure bottlenecks (happy developers ship more features)
Enhanced security posture through consistent policy enforcement (security that doesn’t slow people down is security that actually gets used)
Lower operational overhead through standardization and automation (because your platform team’s time is better spent on innovation than toil)

The numbers speak for themselves. We’ve worked with customers like Snowflake who reduced deployment times from one and a half weeks to less than a day. Starburst Data cut their infrastructure deployments from two weeks to just three hours. These aren’t vanity metrics. They’re business transformations.

IDP Strategy: 5 Key Steps to Build a Strong Foundation

If you’re just starting your IDP journey, here’s the roadmap we give to every platform team we work with:

Start with user research: Before you write a single line of code, go talk to your developers. Understand your different personas and their actual needs, not what you think they need. Do your users like to work from the command line? Do they prefer interacting with a UI?
Identify common patterns: Look for infrastructure patterns that get repeated across teams. These are your goldmine—the patterns that, once abstracted and templatized, will provide immediate value.
Begin with components: Build reusable infrastructure building blocks before you worry about fancy UIs or workflow orchestration. You need solid foundations before you can build the house.
Implement progressive disclosure: Give people simple interfaces for common use cases, but always provide escape hatches for complex scenarios. The moment your abstraction becomes a prison, developers will route around it.
Measure and iterate: Track adoption and gather feedback religiously. Your platform is only as good as its adoption rate, and adoption only happens when you’re solving real problems for real people.

The Future of IDPs and Platform Engineering

Trends come and go, but platform engineering is here to stay. It’s not just another tool or methodology. This is how modern organizations scale infrastructure to match the speed of innovation. The companies that win are the ones that treat their Internal Developer Platform (IDP) as a strategic foundation, not just a tool.

Your job isn’t just to manage infrastructure and choose the right tools - it’s to enable developers to move faster while keeping governance, security, and costs under control. Get that balance right, and your platform becomes a force multiplier across your entire engineering department.

The fastest way to get there? Focus on these five essential components:

Abstractions
Blueprints
Workflows
Security guardrails
Self-service

Together, they turn your IDP into more than just infrastructure automation, they create a platform that developers trust and your business relies on.

Ready to Build a Successful IDP that Scales?

Pulumi makes it easy to go from static IaC to dynamic, self-service infrastructure with real programming languages and built-in guardrails.

Explore Pulumi IDP or sign up for free to get started today.

Governance as an Enabler: Scaling Safely and Confidently

Adam Gordon Bell — Tue, 17 Jun 2025 00:00:00 +0000

In previous articles in this series, we’ve explored how platform engineering transforms infrastructure chaos into consistent provisioning, empowers engineering teams through self-service infrastructure, optimizes workflows, embeds security directly into your platform, and provides observability as a superpower. Each pillar builds upon the previous ones, creating a cohesive foundation that accelerates innovation and productivity.

But as your platform scales, new challenges inevitably emerge. You’ve empowered engineering teams with self-service infrastructure, streamlined workflows, and embedded security directly into your platform. But as your platform scales, new challenges emerge: How do you ensure consistency, compliance, and cost control without slowing your teams down?

In this article, we’ll explore how Platform Engineering transforms governance from a manual, bureaucratic process into an automated, built-in enabler, helping your organization scale safely and confidently. By embedding governance directly into your platform, you can maintain control, ensure compliance, and manage costs effectively, all while preserving the autonomy and speed your engineering teams have come to expect.

The Problem: Governance as a Manual Bottleneck

Dealing with manual compliance checks and red tape

With increased team autonomy and self-service capabilities, how do you ensure consistency, compliance, and cost control across your entire organization?

Governance often feels like a necessary evil: manual, bureaucratic, and slow. Application teams see it as red tape, while operations teams struggle to maintain control. Manual compliance checks, lengthy audits, and unclear or inconsistent policies create friction and frustration. Teams may bypass governance processes entirely, leading to shadow IT, inconsistent resource configurations, and hidden risks.

The consequences of manual, bureaucratic governance are clear:

Increased compliance risks and audit failures: Without automated enforcement, compliance becomes reactive and error-prone, increasing the likelihood of regulatory violations and audit findings.
Unpredictable cloud costs and budget overruns: Without clear guardrails, self-service infrastructure can lead to resource sprawl, wasted resources, and unexpected cloud bills.
Reduced team autonomy and slower innovation: Manual governance processes reintroduce bottlenecks, slowing down deployments and undermining the agility your platform was designed to achieve.

The Solution: Embedding Governance into Your Platform

Successfully scaling safely with embedded governance

Governance should live inside your platform, not off to the side as a separate process. To make that happen, build these four capabilities into your IDP:

Policy-as-Code for Automated Compliance: Declare rules (like approved regions or required tags) as code. The platform enforces them whenever infrastructure is created or updated, so compliance happens automatically.
Platform-Level RBAC for Permission Boundaries: Decide who can act on projects, stacks, and templates before any cloud credentials run. This early check prevents unauthorized requests from ever reaching cloud provider.
Audit Logs and Drift Detection for Real-Time Visibility: Record every deployment, who ran it, and what changed. Continuously compare live infrastructure to the desired state in code. If someone bypasses approved processes, the platform flags it and alerts the team.
Resource Lifecycle and Deployment Controls: Automatically retire idle environments after a set time (Resource TTLs) so forgotten test clusters don’t rack up bills. Also if needed, gate production changes behind lightweight approval workflows. Routine dev or staging updates roll out instantly, but high impact production changes wait until a reviewer signs off.

Let’s dive into each of these.

A. Policy-as-Code for Compliance and Operational Standards: Automating Trust and Consistency

An engineering team is ready to launch a new service. They’ve tested it and everything looks good, until the deployment fails. Not because of a bug. Because it’s targeting an unapproved cloud region.

Now they’re stuck. A compliance review kicks off. Slack threads fly. A ticket gets filed. What should’ve been a smooth release turns into a delay, all because of a policy someone missed.

Policy-as-code prevents this.

When teams deploy something that breaks the rules (like using an unapproved region), the platform blocks it automatically. The error shows up right away, with a clear message. Nothing gets provisioned, and nobody has to file a ticket.

If you’re already using intent-based components (“I need a Java service with Kafka and PostgreSQL”), most details are handled for you: tags, regions, naming. But people still override things. That’s why policy-as-code matters.

Think of it as a safety net. A menu of components handle the defaults. Policies catch anything that slips through. Together, they keep your platform consistent without slowing anyone down.

B. Role-Based Access Control (RBAC): Balancing Autonomy and Control

Stopped by lengthy approval processes

As your platform grows, managing permissions manually gets messy. If an engineer needs to fix a production issue but doesn’t have access, they file a ticket and wait, sometimes for days. Give developers too many rights, and they might change production by accident. Both options slow teams down and increase risk.

The fix is an RBAC model built into your platform. First, the platform decides who can deploy, who can publish components, and who can manage templates. This check runs before any cloud credentials are used, so invalid requests get blocked early. Second, the cloud IAM layer controls which API calls are allowed, like creating an EC2 instance or updating a database.

This pairs well with a two-level intent-based approach. Teams describe what they need (“I want a Python Lambda with an SQS queue”), and the platform enforces access only for users with the right scopes. Everyone gets just enough access to do their job, no more, no less.

A Platform with RBAC makes permissions clear, reduces mistakes, and keeps everything auditable. Devs move fast, spinning up resources as needed, while strong guardrails stay in place. The result is a scalable, least-privilege model that balances autonomy and control, so your organization can grow safely.

C. Auditability, Traceability, and Drift Detection: Ensuring Visibility and Trust

Anxious about audit failures and compliance risks

An ops engineer spots a production database misbehaving. A quick check shows someone changed its configuration outside the approved workflow. Without an audit trail or drift detection, the team scrambles to figure out who made the change and when. Meanwhile, the incorrect setting stays active, posing a security and compliance risk. No one can fix it without guessing.

A platform with audit logs records every action: who deployed, when, and what changed. Drift detection watches live infrastructure and compares it to the desired state in code. If someone bypasses the workflow (say, editing a database setting in the console), the platform flags it and alerts: “User Alice changed max_connections on prod-db-01 at 3:42 PM, which no longer matches the expected state.” Now the team can pinpoint the change, talk to the right person, and revert or update the code, restoring consistency in minutes, not hours.

Together, audit logs and drift detection give you real-time visibility into every change. You stop playing detective. You see who did what, when, and how it deviated from code, all in one place. That transparency speeds audits, catches unauthorized changes fast, and builds trust across teams. With automatic traces of every change, your platform scales without surprises.

D. Resource Lifecycle and Deployment Controls: Scaling Responsibly and Safely

An engineer spins up a test environment, then walks away. Another pushes a change straight to production without review. The abandoned test cluster runs up cloud costs; the unreviewed prod tweak risks an outage. Without automation, both lead to wasted spend and stressful cleanups.

A modern platform handles this with ephemeral environments where possible and approval gates where it matters.

In dev and staging, engineers can move quickly. They can create test or preview environments, often tied to users or pull requests, that shut down automatically after a set time. TTL rules keep things tidy without manual cleanup.

Production, by contrast, is gated. High-impact changes, like modifying a database schema or adjusting a load balancer, require approval. Before anything is provisioned, sign-off is required. Every approval (or denial) is logged: who, when, and why.

This setup keeps development fast and flexible, while making production changes deliberate and auditable. Your platform stays clean, cost-effective, and safe without getting in the way.

Drowning in resource sprawl and unexpected costs

Real-World Example: Governance Enablement in Action

An engineering team opens a pull request for a new customer-facing service. The platform spins up a preview environment using a template with secure defaults, pre-approved modules, and policy checks. CI runs tests and policy checks in preview (names, regions, encryption, compliance) so the team catches issues early. If a rule fails (say, an unencrypted database), the PR fails before it reaches main. After the PR merges to main, it deploys to production, confident all policy validations have passed.

Idle QA environments shut down after 48 hours, so forgotten clusters don’t rack up bills. Sensitive production changes, like updating a load balancer or altering a critical schema, are carefully reviewed via pull request. Once approved, the platform deploys automatically and logs every action. Drift detection flags console edits, letting the team revert or update code in minutes.

Result: Governance becomes an invisible safety net. Engineers move fast, knowing policy-as-code, RBAC, TTL cleanup, approval gates, and change tracking catch mistakes. Ops stays in control without firefighting or chasing orphan resources. The platform scales safely, balancing freedom with built-in guardrails.

Metrics: Measuring Governance Enablement

To ensure your governance practices truly empower your organization, it’s essential to track clear, actionable metrics. These metrics help you understand the effectiveness of your governance processes, identify areas for improvement, and ensure governance remains frictionless and enabling:

Time Spent on Manual Compliance Checks and Audits: Measure how much time your teams spend manually verifying compliance or performing audits. Effective governance automation should significantly reduce this overhead, freeing teams to focus on higher-value tasks.
Number of Compliance Violations or Audit Findings: Track how frequently compliance violations or audit issues occur. Effective governance should reduce these incidents, demonstrating that automated policies and guardrails are working as intended.
Cloud Cost Predictability and Budget Adherence: Monitor how accurately your cloud spending aligns with forecasts and budgets. Good governance practices, such as automated tagging, resource lifecycle management, and cost controls, should improve predictability and reduce unexpected cost overruns.
Engineering Team Satisfaction with Governance Processes: Regularly survey engineering teams to gauge their satisfaction with governance processes. High satisfaction indicates that governance is enabling rather than hindering their workflows.

Tracking these metrics helps you continuously improve your platform’s governance practices, ensuring they remain effective and frictionless.

Working within clear, automated guardrails

Pulumi and Governance Enablement

Pulumi provides built-in governance features that help you scale safely and confidently, embedding compliance, consistency, and control directly into your platform:

CrossGuard (Policy as Code) Define and enforce compliance and operational policies automatically. CrossGuard checks every resource against your organization’s standards before deployment, preventing non-compliant resources and reducing manual audits.
Role-Based Access Control (RBAC) and Teams Manage permissions with precision. Pulumi’s RBAC ensures teams get exactly the access they need, no more, no less, so developers can move quickly within clear boundaries and ops can reduce risk.
Audit Logs and Drift Detection Capture a full history of every change and compare live infrastructure to the desired state in code. Audit logs simplify compliance reviews, drift detection spots unauthorized edits, and teams can fix issues in minutes.
Time-to-Live (TTL) Stacks / Ephemeral Environments Spin up short-lived environments for testing or previews. You can assign a TTL to any stack so it shuts down automatically after a set period. That keeps forgotten test resources from racking up costs and ensures your platform stays clean.

By leveraging Pulumi’s governance features, including CrossGuard, RBAC, audit logs, and TTL stacks, your platform becomes a powerful enabler. You automate compliance, maintain consistency, and empower engineering teams to innovate quickly and safely.

Conclusion: Governance as a Platform Feature

Having autonomy with built-in controls

Governance doesn’t have to slow you down. By embedding governance directly into your platform, you empower engineering teams to innovate quickly while ensuring compliance, consistency, and control. Instead of manual checks, governance becomes automatic, transparent, and frictionless, enabling your organization to scale safely and confidently.

Your engineering teams gain autonomy and speed, your operations teams gain visibility and control, and your organization gains the confidence to innovate at scale.

You’ve now seen all six pillars of a modern internal developer platform—provisioning, self-service, developer experience, security, observability, and governance. If you’d like to see how Pulumi makes building and running a platform like this simpler, check out Pulumi IDP.

Observability as a Developer Superpower

Adam Gordon Bell — Tue, 10 Jun 2025 00:00:00 +0000

Frustratedly trying to figure out what's actually happening

In previous articles in this series, we’ve shown how platform engineering turns infrastructure chaos into consistency, gives teams self-service tools, smooths developer workflows, and bakes security into the platform. Each pillar builds on the last. Together, they create an internal developer platform that cuts friction and speeds innovation.

Even so, teams still face a big challenge: seeing what’s really happening. Whether things go wrong or run smoothly, engineering teams need clear, actionable insights into their systems. Without observability, you end up guessing, reacting slowly, and hunting through scattered data.

This article shows how observability can be a superpower, giving teams the visibility, insights, and confidence to build better software. Embedding observability into your platform lets teams spot, understand, and fix problems fast, turning reactive firefighting into proactive innovation.

The Problem: Data Overload Without Insights

Teams drown in metrics, logs, and traces but lack useful insights. In practice, this shows up in three common friction points:

Tool sprawl: Teams use separate tools for metrics, logs, and traces. They waste time flipping between dashboards and stitching data together.
Alert fatigue: Teams get hit with noisy, context-free alerts. With no clear priority or context, key alerts get lost, causing missed issues or slow responses.
Reactive debugging: Troubleshooting turns into a late-night fire drill. Teams spend hours digging through logs and metrics after users have already noticed the problem.

When observability is limited to post-mortems and fragmented dashboards, your team wastes time reacting instead of preventing problems, and innovation grinds to a halt.

The Solution: Observability as an Engineering Superpower

Exhausted from hours of reactive debugging when the problem could have been caught earlier

The solution isn’t just about bolting on more monitoring tools. it’s about baking visibility, context, and guidance into your platform. To do this, embrace three key principles:

Centralized Service Dashboards & Service List Surface every running service (or database, function, etc.) in one “Services” portal, complete with health badges (CPU, error rate), on-call owner info, and one-click links to that service’s metrics, logs, and traces. By unifying all telemetry behind a single service card, you eliminate context-switching and help engineers find exactly what they need in seconds, not minutes.
Actionable Alerts and Insights Replace vague, noisy notifications with context-rich, prioritized alerts that include severity, correlated root-cause data, and recommended next steps (“Database latency jumped 200% since last deploy: rollback or scale up replicas”). Group and surface only the most critical issues first to reduce alert fatigue and speed up resolution.
Embedding Observability into Engineering Workflows Ship every new microservice with built-in logging, metrics, and tracing by including those hooks in your platform’s service templates and CI/CD pipelines. When instrumentation is automatic, “oops, I forgot to add a span” moments disappear, and teams gain immediate visibility into performance and errors from day one.

When observability becomes a superpower, engineering teams gain the visibility and insights they need to confidently build, deploy, and operate software. Instead of drowning in data, they proactively identify and resolve issues, optimize performance, and innovate with confidence.

A. Centralized Service Dashboards & Service List

Reading the Service Catalog

Imagine you’re paged at 2 AM because “OrderService” is failing, but you don’t know where to look. Metrics live in Grafana, logs are in Elasticsearch, traces in Jaeger, and you still have to hunt down who’s on call. You spend precious minutes clicking through multiple UIs and Slack channels just to figure out who owns the service and where its telemetry lives.

A centralized service list solves this by surfacing every running microservice (or database, or function) in one place. In your platform’s web portal, you land on a “Services” page that shows OrderService alongside CPU and error‐rate badges, an on-call owner, and links to its real-time dashboard, filtered logs, and trace waterfall. No matter which team spun it up, you know exactly where to click: metrics, logs, traces, deployment history, and contact info all live behind a single service card.

By embedding a service list into your platform, you eliminate context switching and reduce onboarding friction. If a service isn’t listed, it isn’t properly instrumented, so gaps stand out immediately. In practice, this “one‐pane‐of‐glass” approach means engineers spend seconds finding the right dashboard and the right person, instead of minutes piecing together fragments across disconnected tools.

B. Actionable Alerts and Insights: Reducing Noise and Accelerating Response

Digging through alert noise

It’s 4 AM and your phone buzzes with three simultaneous alerts, each with vague descriptions like “High CPU usage detected” or “Error rate increased.” Without clear context or recommended actions, you must manually investigate each alert, digging through logs and metrics to determine severity and root cause. This manual triage process is slow, frustrating, and error-prone, increasing the risk of missing critical issues or delaying resolution.

Engineering teams often face a constant barrage of alerts, many of which lack clear context, severity, or actionable next steps. This flood of noisy, ambiguous notifications creates alert fatigue, causing your team to overlook critical issues or waste valuable time investigating false positives.

A platform approach should focus on actionable metrics. Instead of vague notifications, clearly state the issue (“Database latency increased by 200%”), provide relevant context (“Latency spike correlated with recent deployment”), and suggest immediate actions (“Rollback recent deployment or scale database resources”). Additionally, alerts are automatically grouped and prioritized based on severity and impact, ensuring you focus on the most critical issues first.

By holistically approaching alerts and insights, you significantly reduce alert fatigue and noise, accelerate incident response and resolution, and empower engineering teams with greater confidence and autonomy. Your team spends less time manually triaging alerts and more time proactively resolving issues, improving reliability, productivity, and overall team satisfaction.

C. Embedding Observability into Engineering Workflows: Visibility from Day One

At 2 AM, where are the logs for this service.

You’ve just deployed a brand-new microservice to production, only to discover performance issues or unexpected behavior. Sure, you should remember to add tracing, logging, and metrics by hand, but in practice, things slip through the cracks. It isn’t until real-world traffic hits that you realize you forgot to instrument X or Y, and now you’re scrambling to retroactively add code, redeploy, and wait for data to appear, delaying resolution and frustrating your team.

If your platform’s service templates already include all the necessary logging, metrics gathering, and tracing out of the box, it makes life a lot easier. Embedding observability into those templates and engineering workflows ensures every new microservice ships with built-in instrumentation.

This proactive approach reduces “oops, I forgot” moments, accelerates troubleshooting, and increases team productivity and satisfaction, ultimately improving the reliability and quality of your software.

Real-World Example: Observability Superpower in Action

Applying an Actional Metric

At 3:15 AM, your pagerduty alert goes off: “CheckoutService latency spiked 150%.” You log into your platform’s Services portal and immediately see CheckoutService highlighted with a red latency badge and Todd Rivera listed as the on-call owner. Rather than scouring multiple dashboards, you click its service card to jump straight to the metrics, logs, and trace views.

The alert itself is remarkably precise: “CheckoutService latency rose 150% at 3:10 AM following the v2.3.1 deployment. PaymentGatewayService upstream error rate jumped from 0.2% to 2.3%. Recommendation: rollback v2.3.1 or scale PaymentGateway pods. Contact Todd Rivera.” Instantly, you know where the problem lies, which upstream service is impacted, and what the next step should be. In the trace waterfall, you spot a 200 ms delay on CheckoutService calls to PaymentGateway. The logs, automatically instrumented by your service template, filter to TimeoutException entries all timestamped at 3:10 AM. Opening the “Ask Platform” AI widget, you type, “Why did CheckoutService latency spike at 3:10 AM?” The AI responds: “Likely cause: v2.3.1 added index idx_created_at to PaymentGateway’s transactions table, causing an 80 ms delay per request. Rollback v2.3.1 or patch queries to remove the new index.”

Armed with this precise diagnosis, you open a rollback pull request and, after Todd’s ok, deploy it within minutes.

CheckoutService latency and PaymentGateway errors immediately return to baseline. By moving from alert to resolution entirely within the platform (thanks to built-in instrumentation, actionable alerts, and AI-driven analysis, you’ve squashed a major incident before most users ever noticed.

Metrics: Measuring Observability Enablement

To ensure your observability practices truly empower engineering teams, it’s essential to track clear, actionable metrics. These metrics help you understand the effectiveness of your observability tools and processes, identify areas for improvement, and demonstrate the tangible impact observability has on your organization.

Key metrics to measure observability enablement include:

Mean Time to Detection (MTTD):
How quickly are issues identified after they occur? Effective observability should significantly reduce the time it takes to detect problems, enabling faster responses and minimizing user impact.
Mean Time to Resolution (MTTR):
How quickly are issues resolved once detected? With clear, actionable insights and unified observability, your teams should resolve issues faster, reducing downtime and improving reliability.
Engineering Team Satisfaction with Observability Tools:
Regularly survey your teams to gauge their satisfaction with observability tools and workflows. Higher satisfaction indicates that observability is effectively embedded into engineering workflows, reducing friction and increasing productivity.
Adoption Rate of Observability Tools and Dashboards:
Track how widely observability tools and dashboards are adopted across teams. Increased adoption indicates that your teams find these tools valuable, intuitive, and helpful in their daily work.
Reduction in Alert Noise and False Positives:
Measure the volume and accuracy of alerts over time. Effective observability should reduce noisy, irrelevant alerts, ensuring your teams focus on meaningful, actionable notifications.

Tracking these metrics helps you continuously improve your observability practices, ensuring they remain effective and empowering. By regularly reviewing and acting on these insights, you can proactively enhance team productivity, reliability, and overall satisfaction.

Pulumi and Observability Enablement

Platform insights

Pulumi’s platform features let you explore observability without bolting on extra tools. Key Pulumi features that enable observability include:

Pulumi Insights:
Provides unified visibility and powerful search across all your cloud resources. Your teams can quickly discover, explore, and understand their infrastructure, eliminating manual searches and reducing cognitive load.
Centralized Service List: Pulumi IDP’s Services portal gives you a single place to register each microservice, database, or cloud resource and link to its dashboards, logs, and traces.
Pulumi Co-Pilot:
Delivers AI-powered troubleshooting and insights directly within your workflows. Your teams can ask natural-language questions about their infrastructure (such as “What infrastructure changed yesterday?”) and receive immediate, actionable answers.
Built-In Instrumentation via IDP Components:
When you author components and templates in Pulumi IDP, you can bake in standard logging, metrics, and tracing hooks. Every service spun up from those templates ships with consistent instrumentation on day one.

With Pulumi, observability can become an integrated part of your platform, accelerating innovation, improving reliability, and empowering engineering teams to confidently build, deploy, and operate software.

Conclusion: Observability as a Platform Feature

Happy resolving an incident in minutes.

Observability isn’t just about plugging in more tools. It’s about baking-in consistent instrumentation, measurement, and context so every engineer (platform, DevOps/SRE, or application) knows exactly where to look and how to act.

Service-Templates with Built-In Telemetry By providing service templates that already include logging, metrics gathering, and tracing, you eliminate “Oops, I forgot to instrument X” moments. Every new microservice inherits a standard setup, so you never have to retroactively add code or scramble when traffic first hits production.
Consistent Service Dashboards & Centralized Service List Instead of hunting across eight different dashboards, engineers always start from a single “Service List” page. From there, one click takes them to that service’s metrics overview, log stream, or trace waterfall. This unified entry point reduces cognitive load and cuts straight to “Where’s the problem?”
Measuring Alert Quality and Actionability A truly mature platform doesn’t just send alerts. It tracks whether those alerts are helpful or noise. By measuring “ratio of actionable alerts vs. false positives,” you continuously fine-tune thresholds and eliminate alert fatigue. The result? Engineers trust their notifications and respond faster to real incidents.
AI-Driven Context and Natural-Language Troubleshooting On top of unified telemetry and alert quality metrics, AI can instantly correlate recent deployments, configuration changes, and error spikes. Engineers can ask, “Why did latency jump at 3 AM?” or “What changed in production last night?” in plain English, and the platform provides a clear, context-enriched answer. This additional layer turns reactive firefighting into proactive problem prevention.

When you combine these elements (components and templates, a single service dashboard, alert quality measurement, and AI/natural-language querying), you transform observability into a genuine superpower. Issues are spotted, triaged, and fixed before customers even notice.

Next time, we’ll dive into the final pillar, Platform Governance, showing how to enforce policy, manage costs, and keep your platform secure and compliant as it scales.

From Configuration Chaos to Programming Languages

Daniel Ward — Tue, 03 Jun 2025 00:00:00 +0000

This post is based on our video interview with Daniel Ward, a Software Developer/Consultant at Lean TECHniques and Microsoft MVP. Daniel shares his strategies for moving teams from configuration chaos to programming languages for infrastructure, including the 10% rule for change, viral adoption, and why Pulumi beats YAML for complex systems. Watch the video below or read on. - Adam Gordon Bell

Imagine your infrastructure codebase has evolved from a few files to thousands of lines across dozens of configuration files. Your team spends hours deciphering CloudFormation templates, your AWS CDK deployments take 30+ minutes, and introducing any change becomes a multi-day effort. This is the reality many organizations face when their infrastructure-as-code approach outgrows their tooling.

As a consultant and developer who helps organizations improve their software delivery processes, I’ve seen firsthand how the right technologies—specifically using actual programming languages like C#—can transform infrastructure management instead of forcing teams to use declarative configuration files.

My Background: Coaching Teams Through Technical Transformation

My background is primarily as a .NET developer—I’m now a Microsoft MVP in my first year—though I work with TypeScript and other languages as needed. About six years ago, I moved into consulting, or what we call “coaching” at Lean Techniques.

The idea isn’t just to write code for clients or tell them what to do and leave. Our goal is that once we leave, they don’t need us anymore. We’re essentially trying to put ourselves out of a job.

Organizations bring me in for various reasons—from implementing automated testing after costly failures to migrating from mainframes to the cloud. Sometimes it’s about product delivery challenges, like month-long sprints that never complete on time.

It’s about bringing places up to a modern standard. As we say at Lean Techniques, turning IT from a cost center into a competitive advantage.

If you’re ready to move from sprawling configuration files to code-centric infrastructure, here are eight practical steps to guide you:

1. Follow the 10% Rule for Sustainable Change

One of my guiding principles comes from Gerald Weinberg’s book “The Secrets of Consulting”: the 10% rule. When entering a new team or organization, don’t try to improve more than 10% at a time.

There are a few reasons for this. One is that you can actually end up making the client look bad. If you improve everything at once, their boss might ask why this consultant fixed everything in six months when the team couldn’t fix it in five years.

But there’s a deeper reason related to social dynamics. Any social group, in programming or outside of work—if you come in new and try to change how the group works before familiarizing yourself with it, people will be resistant. They’ll ask “Who do you think you are? You just got here!” And they’re right—you could be wrong about some things. You have to understand how they work to understand what the problems are.

This applies directly to introducing new technologies like infrastructure as code. Rather than overhauling everything at once, I recommend starting small.

If a place is completely new to infrastructure as code, I’ll tell them to start small. None of their engineers know how to do this yet, and I’m not going to do all the work for them and leave, because then none of their engineers will know how to maintain it.

2. Quantify the Cost of Inaction to Overcome Resistance

Even with a measured approach, resistance is inevitable. I’ve experienced teams that are exceptionally open to improvement, often coming from difficult situations and eager for change. I’ve also worked with teams that resist most suggestions, where engineers feel the consultant was forced upon them by management.

It’s always easier if you can do pull-based versus push-based change—where the team has asked for your help instead of the manager telling them “I’m inflicting this help on you.”

For those looking to influence change within their organization, I’ve found that influencing upper-level management comes down to numbers. They deal with too many teams to get into specifics—they want numbers. If you can show how many hours your team spends each week manually pushing code or waiting on another team, and multiply that by engineer salaries, you’re showing how much money is being wasted.

This approach is far more effective than what can come across as “Here’s a new tool I want to play with”—an anti-pattern that reinforces the stereotype of technical people just wanting to use new technologies.

3. Build Consensus with Nemawashi Conversations

I’ve found a Japanese concept called “nemawashi” (literally “turning the roots”) provides another effective approach to organizational change:

Before introducing an idea to a manager or the whole team, individually go to every person on the team first and get their thoughts. If most people respond positively, you can then approach the decision-maker saying, “Everyone on the team wants this—what do you think?”

This isn’t just about collecting checkmarks but genuinely gathering feedback. If you’re getting pushback from every single person, maybe it’s not as great an idea as you thought. Either way, you learn something from it.

4. Drive Viral Adoption Through Early Wins

Rather than forcing change on resistant teams, I prefer what I call “viral adoption”:

We try to work with teams that ask for our help instead of being pushed onto teams. People are way less resistant if you can show them you’re making their job easier. We work with one team who wants our help, make them really effective, and then other teams notice: “How come they’re releasing so many times a week? How do they have so few defects? How is their pipeline so fast?”

This success creates natural curiosity, leading other teams to either request similar help or study and adopt the successful team’s practices. Either way, I’m happy that things are improving.

5. Align Dev and Ops with Infrastructure as Code

When it comes to infrastructure as code specifically, I’ve noticed it’s becoming table stakes for many organizations, though implementation quality varies widely.

It’s hard to have actual continuous delivery without infrastructure as code. Without it, developers can’t define their own infrastructure and have to wait on another team to do it. By continuous, I mean teams should be able to deploy every 30 minutes if they want. If you have to wait one to three business days for the operations team to create your resources, that’s not continuous.

Resistance to infrastructure as code typically comes from two directions:

From developers: “I already have my plate full—now I have to learn cloud and deployment too?”
From operations: “That’s my job—I don’t want to lose it.”

I see this as a win-win opportunity for both sides. For operations teams, the job shifts from mundane tasks like “create a lambda for me” or “replace this code bundle” to more creative platform engineering work. For developers, the benefit is clear: no more waiting on operations to proceed with their work.

6. Invest in Platform Engineering for Reusable Components

Platform engineering has emerged as a solution to these challenges. Instead of every team reinventing solutions for common needs like logging, monitoring, or deployment pipelines, platform teams create standardized, reusable components.

It seems wasteful if every single team has to manage their own side of the Splunk instance or Dynatrace. These are solved problems that every team needs. A platform team can look across the organization, see what can be standardized, and remove those problems from individual teams.

This approach is particularly powerful with tools like Pulumi, which allows creating reusable component resources for common patterns: “It’s very common for teams to create a Lambda-backed API Gateway or a Lambda hooked up to SQS. These are super common things, so here’s a component resource for that so you don’t have to write it yourself.”

The key difference in mindset is that platform teams aren’t enforcing rules—they’re building tools that make developers’ lives easier. Instead of saying “You must do logging this way,” it’s “Here’s an endpoint you can use—you’re good to go.”

7. Avoid YAML Pitfalls: Embrace Programming Languages

One of my pain points is the limitations of configuration languages like YAML for complex infrastructure. I recently gave a presentation about a GitHub Actions workflow that grew to over 1,000 lines, calling shared actions that call other shared actions, making it nearly impossible to understand what’s happening.

I couldn’t use breakpoints or the normal libraries I would use. I’ve seen a 69,000-line YAML file for Prometheus and Kubernetes—that’s an auto-generated example, but even a thousand lines calling several other files is too much.

The fundamental issue is abstraction. Programming languages allow loops, conditionals, and functions to remove repetition and build larger structures. Configuration languages often lack these capabilities, leading to copy-pasting and unmanageable growth.

With declarative approaches, you end up using a lot of magic strings. It’s easy to forget one when you change values. The tooling is also an issue—I’m familiar with IDEs and libraries for programming languages, but the equivalent tools for YAML just aren’t as mature.

8. Choose Pulumi for Flexible, Code-First IaC

When comparing infrastructure-as-code options, I see three major players: AWS CDK, Terraform, and Pulumi. While AWS CDK offers similar capabilities to Pulumi for AWS resources, it relies on CloudFormation under the hood:

CloudFormation is just a black box—it’s proprietary. If something goes wrong, you have to hope someone’s already asked about it on GitHub. When deployments take 15-30 minutes, there’s not much you can do—it’s just how CloudFormation works.

Compared to Terraform, which uses HCL (HashiCorp Configuration Language), Pulumi offers the flexibility of actual programming languages:

There’s nothing wrong with declarative approaches in some cases, but at a certain point—when you have big systems and chunks of resources you want to share between teams—it becomes more cumbersome than helpful. That’s why Terraform is working on their own CDK now.

What sets Pulumi apart is its flexibility: Even if you have platform teams who aren’t developers and want to set standards, they can use YAML with Pulumi and then transform those component resources into other languages for dev teams to use. You get the best of all worlds.

Start Small, Think Big: Small Changes, Big Impact

Whether you’re introducing infrastructure as code or any significant technical change, my advice remains consistent: start small, focus on solving real problems, and let success spread naturally.

By understanding the human elements of change—the 10% rule, nemawashi, and viral adoption—you can transform resistance into enthusiasm. And by choosing tools like Pulumi that bridge the gap between developers and operations, you can create infrastructure that’s both powerful and maintainable.

I’m currently writing a blog series on Pulumi at daninacan.com. I also speak at conferences occasionally, so check out my speaking page if you’re interested.

Ready to escape configuration chaos? Start small, focus on solving real problems, and get started with Pulumi to leverage the full power of programming languages for infrastructure. Join our community Slack to connect with engineers making the same transformation.

Security as an Enabler: Building Trust into Your Platform

Adam Gordon Bell — Fri, 11 Apr 2025 00:00:00 +0000

In previous articles, we looked at how platform engineering fixes infrastructure chaos, enables self-service, and improves developer workflows. These pillars work together to boost both developer productivity and organizational speed.

But there’s still one critical element that can make or break all this progress: security.

Traditional security efforts — even “shift-left” initiatives — often create friction instead of clearing the way for innovation. Embedding security directly into your platform changes that. By weaving in policy-as-code, centralized secrets management, and identity-based authentication, you turn security from a blocker into an enabler. And with the right metrics, you can measure how well your platform balances protection and speed.

When security is baked into your internal developer platform, it stops being a gatekeeper. Developers move faster — and with more confidence.

Done right, your teams gain autonomy without compromising safety. Security people become partners instead of blockers. Your organization builds trust that speeds up innovation rather than slowing it down.

The Problem: Security as a Gatekeeper

Traditional security practices often create friction rather than enable innovation. Developers run into several pain points: last-minute security reviews that delay deployments, unclear policies that feel arbitrary, and security teams that act more like blockers than partners.

This friction has real consequences. When security becomes a bottleneck, developers find ways around it. They take shortcuts. All of this increases risk through inconsistent security practices.

The end result is that innovation slows down. Developer autonomy shrinks. And the agility your platform was supposed to deliver disappears.

Many organizations try to fix this by “shifting security left” - moving security checks earlier in development. But just dumping security responsibilities onto developers isn’t the answer. It overwhelms them and can still create inconsistencies.

The real solution isn’t just about changing when security happens. It’s about changing how it works.

Key Security Functions within a Platform

Effective platform security involves several key functions, regardless of how they are distributed across teams or individuals in your organization:

Defining Security Strategy & Policy:
- Developing and maintaining security policies, standards, and architectural best practices tailored to the organization.
- Providing security guidance, education, and support across engineering teams.
- Performing threat modeling, security architecture reviews, and proactive security assessments.
Monitoring & Responding to Threats:
- Actively monitoring systems to detect security events and potential threats in real-time.
- Managing security tooling (like SIEMs, vulnerability scanners) and orchestrating incident response procedures.
- Investigating incidents, coordinating remediation efforts, and managing security automation.
Integrating Security into Development & Infrastructure:
- Embedding security controls directly into the development lifecycle and infrastructure provisioning processes.
- Implementing secure-by-default configurations, automated security checks (SAST, DAST, SCA), and policy-as-code guardrails.
- Ensuring security considerations are built into CI/CD pipelines, service templates, and infrastructure modules from the start.

Collaboration between these functions is crucial. A successful platform ensures that policy definition, operational monitoring, and practical implementation work together seamlessly to create a secure environment without hindering velocity.

The Solution: Embedding Security into Developer Workflows

The key to solving security friction isn’t just adding more checkpoints earlier. It’s weaving security directly into your platform and your developers’ daily work.

Adding automated scans to your CI/CD pipeline is a good start. Tools like Dependabot, Snyk, and Trivy catch problems early and alert developers quickly. But these “shift-left” checks alone aren’t enough. Developers still need to interpret findings and fix issues, often without adequate support.

Platform engineering takes a different approach. It builds security in from the start, rather than treating it as a final hurdle or dumping complex responsibilities on developers. It uses secure-by-default components and automated guardrails that work continuously.

Let’s see what this looks like in practice.

Policy-as-Code for Security: Automating Trust

If you’ve implemented the intent-based approach we covered earlier, your developers already use secure-by-default modules. They simply request what they need (“I need a Java service with Kafka and PostgreSQL”), and the platform handles security details like IAM roles, encryption, and permissions.

But secure defaults aren’t enough. You still need automated guardrails. Policy-as-Code adds this enforcement layer, keeping security standards consistent—even when someone tries to bypass or change the modules. It’s a backup system: secure modules provide your first defense, and Policy-as-Code catches anything that might slip through.

For Example:

IAM Roles with Least Privilege:
- Secure Default (Module): The platform team’s IAM module automatically generates roles with least-privilege permissions. Developers don’t specify IAM roles directly.
- Policy-as-Code (Additional Guardrail): A policy explicitly checks all IAM roles provisioned by the platform, ensuring no role grants overly broad permissions (e.g., no wildcard permissions). If a module is accidentally modified or misconfigured, the policy catches it immediately.
Mandatory Encryption at Rest:
- Secure Default (Module): The platform team’s database module automatically provisions databases with encryption at rest enabled. Developers don’t specify encryption settings directly.
- Policy-as-Code (Additional Guardrail): A policy explicitly checks all storage resources to ensure encryption at rest is always enabled. If a developer or platform engineer accidentally modifies the module or bypasses it, the policy immediately flags the issue.

When you add Policy-as-Code to your platform, security becomes automatic and clear. Developers get instant feedback they can trust, and security teams maintain control without slowing things down. Your teams can move fast without taking shortcuts that create risk.

Protecting Sensitive Credentials by Default

Policy-as-Code secures your infrastructure, but you still need to protect how applications access sensitive resources like databases and APIs.

Here’s what often happens: A developer copies database passwords into a local .env file. They accidentally commit this file to Git, exposing the credentials to anyone with repository access. Or they inadvertently bake these secrets into a Docker image that others can download. Worse yet, if their laptop gets hacked, attackers can use these credentials to break into production systems.

When these breaches happen, everything stops. Teams scramble to rotate credentials, investigate the damage, and fix systems—creating delays and increasing risk.

Centralized secrets management provides a robust solution to credential security. It stores sensitive information in a secure vault and offers controlled access methods at runtime. Developers work with the secrets management solution instead of handling credentials directly, which significantly reduces exposure risk.

Platform teams build these secure patterns into their service templates. They create clear pathways for accessing credentials that guide developers toward secure practices in both development and production environments.

With this approach, your deployments become both faster and safer. Developers follow simpler, more consistent security procedures, while security teams gain confidence that credentials are properly protected throughout your systems.

Identity and Access Management: Frictionless Security through Identity

Your platform must also solve another security problem: how people and systems safely access cloud resources.

Developers hate juggling multiple credentials—cloud provider keys, passwords, or tokens. These credentials can expire, get lost, or leak, causing headaches and security risks.

Consider a common scenario: A developer trying to fix a broken production system finds their credentials have expired. Now they’re stuck hunting for new credentials instead of fixing the problem.

Identity-based authentication fixes this. Developers just use their regular login. When developers log in normally, they automatically get secure access to cloud resources through OIDC or SAML. No more managing credentials. Access stays secure and current.

This approach simplifies access and removes barriers. Developers get secure access to resources when they need them, speeding up work without weakening security.

Identity-based authentication, combined with policy-as-code guardrails and centralized secrets management, creates a comprehensive security system that protects without impeding work. To see how these practices function together, let’s look at a real example.

Real-World Example: Security Enablement from Build to Incident Response

A developer deploys a new microservice. They select a secure template from the service catalog. Policy-as-code instantly checks that encryption and permissions are set correctly, and credentials are setup in a centralized secrets management environment.

As the service builds, automatic scans check dependencies and container images for vulnerabilities. The system creates a dependency list (SBOM), tracking every component used. Build pipelines verify and sign code, so only trusted code reaches production.

Once deployed, security monitoring begins automatically, showing the service’s security status in real time. Later, someone discovers a vulnerability in a library used by the microservice. Thanks to the dependency list, the security team immediately knows which services are vulnerable. Monitoring spots suspicious activity and sends clear alerts to the team’s dashboards.

Clear incident response plans help teams contain and fix the problem. The team patches the vulnerable component, using the same secure pipelines to quickly roll out the fix.

In this scenario, security helps rather than blocks at every stage. From building to responding to incidents, your platform helps teams build secure software quickly.

But how do you know if your security practices are truly enabling your teams? To answer that, you need clear, actionable metrics.

Metrics: Measuring Security Enablement

To check if your security approach works, track two things:

Security Incident Rate:
Count how many security problems reach production. Fewer incidents means your security is working.
Developer Security Friction Score:
Include security questions in your regular developer surveys. Ask how much security processes slow their work or cause frustration. Improving scores show that security fits naturally into their workflow.

These two measures combined tell you your security works and at what cost, so your team can build good software fast.

Pulumi and Security Enablement

Achieving these goals, a decreasing incident rate coupled with a low developer friction score, requires a platform built with the right foundations. Pulumi helps you construct such a platform by providing built-in capabilities that directly embed security into your infrastructure workflows:

Policy as Code (CrossGuard): Automatically enforce security and compliance standards.
Secure Secrets Management (Pulumi ESC): Centralize and securely inject secrets without manual handling.
Identity-Based Authentication: Simplify secure access to cloud resources using existing identities.

With Pulumi, security becomes an integrated, frictionless part of your platform—accelerating innovation while building trust.

Security as a Platform Feature

Security can help, not hinder. By building security into your platform, you help developers work faster and safer. Security becomes a feature, not an obstacle.

Security completes the platform foundation we’ve built in previous articles on Provisioning, Self-Service, and Developer Experience, creating a platform that speeds up development.

But developers still need to see what’s happening in their systems. Next time, we’ll explore how Observability helps developers spot and fix problems quickly.

Improve Developer Experience: Increase Dev Productivity with Internal Developer Platforms

Adam Gordon Bell — Thu, 13 Mar 2025 00:00:00 +0000

In the last article in this Platform Engineering Pillars series, we explored how self-service infrastructure frees developers from bottlenecks and dependency gates. By providing reusable infrastructure modules and intent-based configurations, platform teams dramatically reduce infrastructure friction. This self-service model powers faster deployments, increased autonomy, and fewer delays.

However, infrastructure provisioning alone isn’t enough to improve developer experience. Even with efficient provisioning, developers can still face inconsistent local setups, sluggish CI/CD pipelines, poor documentation, and fragmented tooling. These obstacles quietly reduce developer productivity, slow developer velocity, and increase operational overhead.

Your platform’s ability to support daily workflows directly impacts core success metrics like time to first commit, time to production, and developer satisfaction.

When improving developer experience as part of a platform engineering initiative, measure:

Time to First Commit – How quickly a developer can become productive in a new environment or project. Pay particular attention to new hires.
Time to Production – How fast code moves from commit to successful production release.
Developer Satisfaction – Regularly survey friction points and prioritize fixing the top issues.

Friction logs are also a powerful way to identify roadblocks, recording every obstacle encountered during common workflows. This reveals hidden friction and provides a roadmap for improvement.

A great developer experience accelerates iteration, improves problem-solving, and drives better products. An internal developer platform (IDP) designed for DevEx is a fundamental competitive advantage.

The Service Catalog: Foundation for Developer Experience

Developer friction often starts with something simple: not knowing what’s already available or who owns it. Without a single, reliable source of truth, developers reinvent the wheel, duplicate work, and waste time chasing information.

A well-structured service catalog in your internal developer platform addresses this by providing a central repository for all services and applications. At its most basic, it’s just a web page with each service’s README, service metadata, and operational information.

But, properly structured, this delivers some large benefits:

Context at a Glance: Every catalog entry follows a standardized template, capturing core information such as purpose, interfaces (REST, gRPC, events, etc.), language/runtime, and environment requirements. New developers can quickly grasp a service’s role and easily jump into productive work.
Discoverability and Reuse: Without a catalog, teams often recreate services they don’t realize already exist. Duplicate services not only waste resources but lead to fragmented knowledge and inconsistencies. With a catalog, developers quickly find and leverage existing solutions, increasing reusability and reducing churn.
Ownership Transparency: Each entry in the catalog explicitly names team ownership, CODEOWNERS files, and pertinent contact details like Slack channels or on-call rotations. Clear ownership streamlines communication, issue resolution, and collaboration, saving precious time and avoiding frustrations.

A service catalog supports autonomy, reduces wasted cycles, and increases developer velocity by making essential information instantly accessible.

But a Service Catalog alone isn’t enough. To fully streamline and enhance your development workflow, you must pair it with consistent local environments and reliable CI/CD processes. Both of which can be serviced by standardized service templates.

Service Templates: Golden Paths for Improving Developer Productivity

Earlier in this series, we introduced service templates as a way to scaffold consistent, ready-to-go services. In an internal developer platform, templates aren’t just for infrastructure, they’re the golden paths that define how developers start and succeed.

If 90% of new services are Go GRPC services, then a carefully crafted, opinionated go GRPC service blueprint will remove a lot of friction. It will give developers everything they need to begin delivering real business value immediately. A few select curated options of common starting points will help guide developers to generate projects complete with standardized structure, built-in quality checks, documentation, and clear guidelines for contribution.

A great service template provides:

Project Bootstrapping

The template quickly scaffolds an entire service—including all necessary files and directories. Everything essential for your organization’s standards should be included, from CI/CD configurations and dependency management to directory structures and environment-specific configuration files.

Built-in Quality Tools

Templates embed guardrails that maintain consistency and high-quality standards across teams:

Linting rules and formatters: Tools like ESLint, flake8, or Checkstyle that ensure code style and consistency across your codebase.
Testing frameworks: Unit tests, integration tests, and contract testing tools set up and pre-configured (JUnit, pytest, Jest, Pact), helping teams maintain high quality right out of the gate.

Documentation and Contribution Guides

Many platform teams overlook how clear, consistent documentation is crucial for initial and ongoing developer productivity. Every new service should arrive out-of-the-box with:

README: Describing at a glance what the project does, why it exists, and how to run it.
Documentation: Service template establishes a place for service-specific documentation that will be surfaced in the Service catalog. Markdown explanations, example input/output, and real-world use-cases.
Contribution guidelines and coding standards: Clearly documenting the expectations, conventions, and standards every developer can follow with confidence.

Embedding this documentation into templates ensures that even brand-new services are discoverable, understandable, and maintainable — right from their first commit.

A great template means the service catalog becomes more than just an organizational listing. These thoughtfully designed templates can directly power it. Every new service entering your catalog aligns neatly with your organization’s best practices in code quality, documentation, and process standards.

When paired with your service catalog, templates ensure every new service aligns with best practices — boosting developer productivity and developer velocity from day one.

Streamlining Local Development to Reduce Friction

One of the fastest ways to improve time to first commit is to standardize local development workflows through your internal developer platform.

Your service templates provide the ideal mechanism for simplifying how quickly developers can get projects running locally. By embedding standardized local-development tooling into these templates, you ensure they’re available consistently across your service catalog. Good templates typically contain:

A preconfigured, containerized environments: Using Dev Containers, Docker Compose, or similar tools lets developers launch an environment that closely resembles production with minimal friction, completely avoiding dependency drift or local environment inconsistencies.
Simple, standardized command runner: Makefiles, just scripts, package.json scripts—whatever works as they’re consistently documented and easy to run commands such as make build, make test, or make lint. Teams might diverge on preferred tooling over time; if so, simply record clearly in the project’s readme (and thus in your service catalog) exactly what’s needed.
Built-in documentation: Each service template includes clear instructions in a README outlining step-by-step processes for setup, running tests, and other everyday developer tasks.

With these pieces fully integrated into your catalog and templates, you significantly reduce friction and speed up development cycles, moving you meaningfully closer to improving your core developer-experience metrics.

The next critical area we need to address is how smoothly and quickly code moves from a local commit to a production release.

CI/CD Integration: Fast, Reliable Pipelines for Experienced Developers

Experienced developers cite unreliable or slow pipelines as major productivity killers. Their frustrations often revolve around flaky builds, unreliable tests, and cumbersome environment setups. Slow or unpredictable pipelines break developer flow, causing frustration, delays, and lower productivity.

A strong internal developer platform integrates fast, stable, and predictable CI/CD processes:

Stable, Fast Feedback:
Every new service comes pre-configured with build acceleration strategies like intelligent caching (Gradle, Bazel, Docker layers) and automatic parallel test execution. Stable and speedy pipelines help teams iterate quickly and confidently.
Ephemeral, Self-service Environments:
Instead of battling shared staging environments – which get blocked by other teams— developers spin up short-lived testing environments directly from pull requests. Need to test your payment-service changes against the latest user-authentication service? Create a dedicated, temporary environment on-the-fly, validate interactions, then shut it down automatically when finished.
Test Reliability:
Strictly enforce policies to quarantine flaky tests, quickly escalate notifications to responsible developers, and provide clear paths for fixing instabilities. Proactive flakiness management ensures credibility and reliability for test pipelines over time.

Reliable CI/CD pipelines protect developer focus, reduce cognitive load, maintain developer velocity, and prevent wasted time, increasing developer productivity.

Developer experience is a vast, nuanced topic, and there’s much more a comprehensive platform can include than we’ve been able to explore fully here. Even just within your Service Catalog and developer workflows, valuable additions might include:

Onboarding Guides: Step-by-step guides rapidly bringing new developers up to speed on teams, tools, and processes.
Frictionless Product Backlog Integration: Quick access from codebases to the product backlog to connect code directly with the business context.
Code Coverage & Quality Dashboards: Easy-to-use visualizations offering fast insights into quality metrics and test coverage.
Deprecation Notices & Change Management: Automated notifications about changes, upcoming maintenance, or decommission plans to keep developers informed and prepared.
Pre-built Common Components: Shared libraries and components solving everyday needs like logging, authentication, validation, and configuration.

And that’s truly just scratching the surface. But remember: Platform engineering is an iterative journey, you don’t need to tackle every single area all at once. If you start with friction logs and surveys to regularly identify and address pain points affecting daily workflows, you’ll already be heading confidently in the right direction.

Conclusion: IDPs as a DevEx Multiplier

A great developer experience is not optional — it’s a force multiplier for developer productivity and developer velocity. An internal developer platform with service catalogs, golden path templates, streamlined local dev, and integrated CI/CD removes friction at every step of the developer journey.

The result? Faster onboarding, quicker delivery, and happier teams.

With Pulumi, platform teams can automate consistent environments, implement golden paths, and enable self-service infrastructure that scales with your organization.

Organization Templates enables standardized service creation and consistent developer experiences
Stack References for managing dependencies between environments and services
Pulumi Deployments to streamline CI/CD workflows and enable self-service infrastructure
Review Stacks for creating ephemeral, on-demand testing environments
Automation API for programmatically managing infrastructure and implementing platform workflows

By prioritizing developer experience in your platform engineering strategy, you empower teams to move faster, build better, and maintain momentum, making your internal developer platform one of the most valuable assets in your organization.

Self-Service Infrastructure: From Tickets to Tools

Adam Gordon Bell — Thu, 06 Mar 2025 00:00:00 +0000

Previous articles in this series explored platform engineering principles and how Infrastructure as Code creates a solid foundation. But there’s still an important challenge to address: the infrastructure provisioning process itself. Without proper modularity and a clear separation between intent and infrastructure details, things get messy—leading to friction, delays, and unnecessary complexity.

The Missing Layer: Abstraction Through Modularity

The root issue here is how teams define and work with infrastructure. Without good abstraction layers, infrastructure code typically becomes unwieldy and complex.

You end up with

Highly detailed specifications that require deep domain knowledge
Environment-specific configurations with subtle variations across resources
Complex interdependencies that are difficult to trace and verify

This approach burdens developers with low-level details they don’t need to worry about, while platform teams become gatekeepers, reviewing intricate configurations. Alternatively, a traditional ticketing model takes over, discarding the benefits of modern practices.

Solution: A Platform Engineering Approach

What’s missing is a proper abstraction layer between raw infrastructure code and developer needs. That layer is the platform. A Platform for self-service infrastructure. Without modular, reusable infrastructure patterns, every new application deployment becomes an exercise in starting from scratch and defining every detail instead of just stating what the application requires and deferring details to the platform.

Common Anti-Patterns Without Proper Abstraction

Without this modular platform approach to cloud infrastructure, organizations typically fall into one of two problematic patterns.

The Copy-Paste Anti-Pattern

Developers copy infrastructure code from previous projects. They find something similar, duplicate large chunks of configuration, make a few changes, and hope everything is right, and submit it for review. This leads to a brittle system with siloed knowledge. Security fixes must be applied manually across every instance – no one has a full picture of the infrastructure – and Day 2 Operations become error prone.

And this isn’t limited to infrastructure: CI scripts get copied and modified, Dockerfiles and Kubernetes manifests get copied and modified.

It feels like control, but it’s not. Developers still rely on platform specialists to validate changes and catch mistakes.

The Ticketing Anti-Pattern

Alternatively, some organizations rely on rigid ticketing systems for infrastructure changes. A platform specialist applies writes and applies all changes, ensuring consistency — but also creating bottlenecks. Developers might wait days or weeks for simple modifications, leaving the process entirely unfinished.

This guarantees expert oversight but strips developers of autonomy, making the platform team a bottleneck for every deployment.

Neither anti-pattern strikes a balance between standardization and autonomy, leading to either sloppy infrastructure (copy-paste) or slow deployments (ticketing).

The Resulting Problems

Day 1 Bottleneck: Slow, Manual Infrastructure = Developer Gridlock

When provisioning becomes a bottleneck, developers with finished code wait days for infrastructure. This disrupts focus and forces frequent context switching. They juggle tasks to stay productive and then have to rebuild focus when the requested infrastructure is ready.

These delays slow the entire delivery pipeline. Teams add buffer time for infrastructure delays, projects slip, and the business sees technology as slow-moving. Meanwhile, the real culprit is slow provisioning that kills momentum.

Day 2 And Beyond Chaos: Sprawling, Unpredictable Infrastructure = Operational Nightmare

Day 1 bottlenecks frustrate developers, but the real long-term cost is patching hell. Copy-paste infrastructure creates hundreds of snowflakes, each needing manual attention for every security fix or version bump. One-off configurations and custom environments pile up, creating technical debt no one fully understands.

Operations teams are overwhelmed by the basic, critical task of simply keeping the lights on across a sprawling, inconsistent landscape.

The Solution: A Platform of Options and Self Service

A Self-service platform is the solution to these problems. The platform is this curated list of modules, providing application developers with a set of pre-built options that let them work at a higher level of abstraction. By choosing from these modules, developers express their intent for infrastructure, leaving the platform to handle the implementation details. Let me explain.

A strong self-service approach divides responsibilities into two clear layers:

Level 1: Platform Team (Module Definition)

Builds and updates reusable infrastructure modules
Bakes in security, compliance, and best practices
Hides complexity so developers don’t need to worry about copy and pasting low-level details
Sets guardrails for scalability, security, and compliance
Updates modules as technologies and needs change

Level 2: Application Developers (Module Consumption)

Browse a catalog of ready-made modules
Compose pre-built modules to set up application infrastructure
Spend time on code, not cloud configurations
Operate within built-in guardrails
Deploy applications faster using standardized components

This setup keeps things fast and flexible. Developers get the autonomy to build without reinventing infrastructure, while platform teams maintain control where it matters.

The Power of Intent-Based Specification

Layer of Abstraction to the Rescue

The two-level approach works by replacing over-specification with intent-based design.

Over-specification happens when developers specify too many infrastructure details that:

Require expertise they might not have
Tightly coupled infrastructure to specific implementations
Make future changes hard or even impossible

Intent-based specification means developers state what their app needs, not how to implement it, meaning:

Platform teams can update underlying infrastructure without breaking applications
New capabilities can be added without disrupting existing applications

Focusing on what developers need, not how it’s built, creates a system that’s easier to maintain and evolve for both platform teams and developers.

Let’s walk through an example.

Practical Example: Java Application with Kafka, PostgreSQL, and Redis

The platform team provides reusable modules:

Java Application Module

This module handles everything needed to run a Java app. It covers the Java runtime, memory settings, JVM options, scaling rules, health checks, monitoring, and Kubernetes namespacing. It also manages containers, security scans, resource limits, and logging integration—so developers don’t have to.

Kafka Integration Module

Similarly, this module simplifies Kafka integration, handling broker connections, authentication, topics, and security. It takes care of SASL/SSL authentication, replication factors, ACLs, and encryption. Developers just specify topics and schemas.

And so on for common infrastructure components used in the org.

Level 2: Developer Implementation

Developers use a simpler interface. To deploy a new Java service with Kafka, PostgreSQL, and Redis, they only need to specify the required input parameters in the platform-defined modules:

# Developer's intent-based configuration
application:
 name: order-processing-service
 module: java-application
 version: 17
 memory: 2GB
 scaling:
 min: 2
 max: 10

integrations:
 - module: kafka
 topics:
 - name: incoming-orders
 partitions: 5
 - name: processed-orders
 partitions: 3
 consumer_group: order-processors

 - module: postgresql
 database: orders_db
 size: medium
 backup: enabled

 - module: redis
 size: small
 ttl: 3600

( Note: YAML not required. This pseudo-code could be UI driven, or it could be written in your programming language of choice. The point is it specifies intent and defers specifics to the platform. )

Solving Day 1 and Day 2 Challenges

The intent-based specification solves immediate and long-term problems:

The Self-Service Transformation in Action:

Developer Experience	Platform Response
Submits simple intent-based configuration	Expands into complete infrastructure
Focuses on application needs, not infrastructure details	Provisions resources with appropriate sizing and security controls
Specifies “what” is needed, not “how” to configure it	Handles networking, monitoring, backups, and credentials
Limited expertise required in Kubernetes, networking, or security	Implements best practices automatically across all deployments
	Maintains and updates the underlying implementation over time

This approach transforms infrastructure from a developer burden into an organizational asset that becomes more valuable over time, rather than decaying into technical debt.

Developer Interaction Models: Choosing Your Platform Interface

Hopefully, now you are bought in on the value of separate layers of abstraction and building a platform to expose those.

The next key decision is how developers will actually use this platform. The ideal interface balances ease of use with the necessary flexibility. Organizations and requirements vary, so a spectrum of interaction models exists, each with its own trade-offs. Tree common approaches are: CLI-Driven Infrastructure as Code, the Self-Service Portal, and a Hybrid model.

Aspect	CLI-Driven	Self-Service Portal	Hybrid Approach
Developer Experience	Code-based, using IaC modules and components directly	Click-based UI with forms and catalogs	Portal for common tasks, code for customization
Code Visibility	Full visibility and control	Limited or no visibility	Visible but with varying levels of access
Governance	Trust-based or PR reviews	Enforced through portal constraints	Guided paths with managed exceptions
Learning Curve	Higher (requires IaC knowledge)	Lower (minimal infrastructure knowledge)	Moderate (basic UI with optional advanced use)
Best For	Experienced teams needing flexibility	Teams prioritizing speed and standardization	Organizations with diverse skill levels

1. CLI-Driven Infrastructure as Code: Empowering Developers with Familiar Tools

In this approach, platform teams build reusable infrastructure modules while developers consume them directly through code. Developers use familiar IaC tools like Pulumi CLI to instantiate these modules, with changes flowing through version control like any other code.

The key strength is transparency and familiarity. Developers maintain visibility and ownership of their intent-based definitions. This code-centric workflow integrates naturally with existing development practices.

However, this flexibility creates governance challenges and requires either strong trust or approval processes that might reintroduce bottlenecks. The approach also assumes developers have some IaC knowledge, making it best suited for organizations with mature DevOps practices and experienced teams.

2. Self-Service Portal: Click-Ops Simplicity for Rapid Provisioning

The self-service portal approach creates a web interface where developers provision infrastructure without writing code. They simply browse a catalog, fill out forms, and click buttons to deploy standardized resources—all pre-configured by the platform team.

This approach keeps infrastructure code completely centralized and hidden from developers. The portal translates simple selections (“Add Database: Medium Size”) into the appropriate underlying configurations, ensuring perfect compliance with organizational standards.

The result is dramatically reduced friction and learning curve, enabling even infrastructure novices to deploy resources in minutes. However, this simplicity comes at the cost of flexibility. Customizations and unique needs typically require platform team involvement, making this model ideal for organizations with predictable workloads or where standardization outweighs customization needs.

3. Hybrid Approach: Code-Backed Self-Service

The hybrid approach generates and modifies infrastructure code via pull requests from UI interactions. When developers configure resources through the portal, the system commits code to repositories automatically, creating a transparent workflow where changes are visible, reviewable, and auditable through standard Git processes.

This model provides both simplicity and flexibility. Routine tasks remain point-and-click while complex scenarios allow direct code editing within the same PR workflow. The approach maintains a complete audit trail through commit history while giving platform teams visibility into all changes.

A significant tradeoff is implementation complexity: building a system that generates commits and PRs based on version control integration requires substantial investment. Organizations must weigh this upfront cost against the long-term benefits of combining UI simplicity with code transparency.

Lifecycle Management

More effective lifecycle management is an advantage of a platform approach. Consider migrating from Kafka to AWS Kinesis. Traditionally, this would involve significant application developer effort, including updating connection details, authentication, and application logic. However, with the two-level abstraction and intent-based approach described earlier, this migration becomes simpler. Platform teams introduce a new Kinesis module matching the existing Kafka module’s intent-based interface. Data is mirrored, and developers can plan a switchover. Migration will not be free, but efforts and coordination will be reduced.

Life-Cycle management is a huge topic in Day 2 operations and outside the scope of what can be covered here, but introducing abstractions and a platform of modules will pay off long term.

Looking Ahead: Operational Excellence and Beyond

The two-level abstraction model does not just simplify provisioning. It also lays a strong foundation for operational excellence. By standardizing infrastructure through reusable modules and intent-based specifications, we create consistency and visibility that significantly simplify critical operational concerns such as:

Drift Detection and Reconciliation: Standardized modules make it easier to detect and correct deviations from desired state.
Continuous Compliance and Security: Centralized module definitions allow security and compliance controls to be consistently enforced and audited.
Cost Management and FinOps: Intent-based specifications enable clearer cost attribution and optimization opportunities.
Observability and Incident Response: Consistent infrastructure patterns simplify monitoring, alerting, and troubleshooting.

Later articles in this series will explore each of these areas in greater depth, demonstrating how a platform approach provides a powerful foundation for addressing them effectively.

Self-Service with Pulumi

The two-level architecture of IDP self-service creates a powerful interface between platform teams and application developers. By embracing intent-based specification, this approach enables developers to clearly express what they need with minimal complexity while giving platform teams the flexibility to implement and evolve the underlying infrastructure.

As you build out your self-service approach, consider how Pulumi might fit in. With Pulumi, you get:

Component resources for creating reusable infrastructure modules that encapsulate best practices.
Organization templates for building a catalog of infrastructure templates.
Packages in your language of choice for a code-first self-service workflow.
A Platform-in-the-box experience for those who want a portal-first experience for self-service infrastructure.
A Backstage plugin for those who prefer to use Backstage for an IDP.

A great self-service workflow and abstraction layer are key to making platform engineering work. Next in this series, we’ll tackle developer experience directly.

Provisioning: From Chaos to Control

Adam Gordon Bell — Thu, 27 Feb 2025 00:00:00 +0000

Provisioning is the first pillar of platform engineering. Without consistent infrastructure provisioning – the automated creation and management of the underlying cloud resources – the rest of the platform suffers. Self-service, governance, and streamlined developer workflows all depend on it. Ultimately, a self-service layer on top of your cloud infrastructure is the goal, enabling developers to quickly and safely provision the resources they need, while adhering to organizational best practices and policies. But before self-service, the foundation of a good IDP is a robust and reliable provisioning system.

By defining cloud resources as code and automating deployments, platform engineering teams ensure every environment – development, staging, and production – stays consistent and maintainable. This cuts down on configuration drift, reduces manual work, and supports auditable, collaborative workflows for every change.

Let’s explore how platform engineering teams can achieve this by version-controlling infrastructure, automating deployments, separating environments properly, and limiting console interventions. By applying these principles, teams can create a platform where developers can move fast without breaking things, and where infrastructure supports innovation rather than slowing it down.

Top 5 IaC anti-patterns

Console-Only Changes: Making “quick fixes” in production without updating IaC leads to drift, confusion, and hidden risks.
Hard-Coded Secrets: Sensitive values in code or config files are a security liability. Use encrypted state or a secrets manager.
Copy-Paste Config: Reusing half-baked snippets across projects creates cruft. Break out shared modules or templates.
No Clear File Structure: Throwing all configs into a single file or folder obscures dependencies. Organize by environment, service, or module.
Skipping Peer Reviews: Infrastructure mistakes can be costly. A simple pull request process can catch errors before deployment.

💡 Tip: Read our list of IaC Best Practices 💡

Building a reliable IaC foundation

It’s not enough to simply define your resources in code. A reliable IaC foundation should ensure that infrastructure changes move smoothly from idea to production, without surprising developers or straining operations. Here’s how to structure these workflows so they’re both transparent and resilient:

Version control as the single source of truth

Whether you’re using GitHub, GitLab, or something else, every infrastructure definition needs to live in version control. This is just the basics – and hopefully, you are already doing this, as you have been for years. Version-controlled changes allow you to track history, compare revisions, and peer review. A pull request might catch a misconfigured subnet or a database that was accidentally left publicly accessible, mistakes that could be catastrophic if they slip into production. This also means anyone can propose improvements or bug fixes, expanding the pool of expertise beyond a single “infrastructure guru.”

Automating changes with CI/CD

Next, consider how to apply infrastructure modifications after they’re merged into main. A CI pipeline can automatically run a “plan” or “preview,” then only proceed with the actual update if no blockers appear. Policy checks can also be added, enforcing rules such as “no public S3 buckets” or “use encrypted volumes only.” By codifying these conditions, you avoid manual verification steps and reduce the chance of risky configurations slipping past manual review.

Trunk-based infrastructure workflow

Protected main Branch: No direct commits
Pull Request Merges: Integrate changes via Pull Requests
Automated PR Checks: Previews and Policy-as-Code run on every PR
Merge: Merge to main triggers automated deployments

Distinguish each environment

Not all environments have the same requirements or budgets, so it’s wise to separate dev, staging, and production configurations. You might share core modules across these environments, but keep each environment’s parameters and secrets in independent config files or stack definitions. This deliberate separation ensures that dev changes are tested in a realistic but contained environment before they can impact production. As a result, teams can experiment more confidently without risking live workloads.

Modularity and reusability

A key principle of effective IaC, and a cornerstone of any good Internal Developer Platform, is reusability. You shouldn’t be copying and pasting large chunks of infrastructure code. Instead, you should build reusable components (or modules, or constructs, depending on the tool’s terminology) that encapsulate best practices and can be easily consumed by developers. This achieves several critical goals:

Consistency: Ensures that similar resources are provisioned in the same way, reducing configuration drift.
Abstraction: Hides complexity from developers. They don’t need to know how a secure, compliant database is created, just that it is.
Maintainability: Changes to the underlying implementation of a component can be rolled out to all consumers of that component.
Speed: Developers can provision infrastructure faster by using pre-built components.
Reduced Errors: Standard components means that there will be less opportunities to make mistakes.

Think of these reusable components as the “building blocks” of your IDP. Your platform team creates and maintains them, and developers use them to assemble their applications.

Regardless of the specific IaC tool you choose, reusable components generally share these characteristics:

Parameterized Inputs: Components accept inputs (parameters, properties, configuration – the term varies) that allow users to customize their behavior. For example, a “database” component might accept a database_size parameter.
Defined Outputs: Components expose outputs that other parts of the infrastructure can consume. For example, a “database” component might output the database’s connection string.
Encapsulation: The internal implementation details of the component are hidden from the consumer. The consumer interacts with a well-defined interface (inputs and outputs).
Versioned: Components should be versioned (e.g., using semantic versioning) to allow for controlled updates and rollbacks.
Discoverable: Components should be easily discoverable by developers, often through a central catalog or registry.

Example: An IAM roles component

IAM roles are one of the most commonly misconfigured AWS resources, making them a perfect candidate for abstraction into a reusable component. Developers frequently make mistakes when setting up roles, leading to overly permissive access, broken permissions, or operational headaches.

Common IAM role mistakes

Overly Broad Permissions – Developers often grant * permissions instead of following the principle of least privilege.
Missing Assume Role Policies – Without a proper trust policy, services may not be able to use the role at all.
Directly Attaching Policies to Users – Instead of assigning roles to groups, users might manually attach policies, leading to hard-to-audit permission structures.
Lack of Naming Conventions & Tagging – Unstructured role names and missing tags make governance difficult.

How a reusable IAM module fixes this

A platform team can provide a self-service IAM role module that enforces best practices while still allowing customization. This module would:

✅ Require a Least Privilege Policy – The module only allows explicitly defined permissions, preventing wildcards.
✅ Auto-Generate Trust Policies – Ensures correct settings, reducing misconfigurations.
✅ Enforce Naming Standards & Tagging – Ensures roles are easily identifiable and trackable.

By using a reusable IAM role module, developers can self-service their permissions while the platform team ensures security, consistency, and maintainability. This removes a major source of security risk and operational overhead.

No console interventions

No matter how carefully IaC is managed, emergencies may occasionally necessitate direct console changes. However, such interventions are strictly forbidden except under an official “break glass” procedure. If a quick fix is applied in the cloud console, it’s critical that the change is immediately documented and returned to code. Regular drift detection can help with this: once a day or once a week, automated scans can compare actual cloud resources to your declared infrastructure. When it detects a mismatch, you decide whether the code or the running resource is the correct version. Either way, human “hotfixes” never linger as hidden surprises for long.

By combining version control, automation pipelines, environment-based configs, and drift checks, you build a solid provisioning foundation. Once you have that in place, the next step is making these patterns easy to use, so teams can start new services without reinventing the wheel.

But first, let’s cover some metrics and roadblocks.

Metrics that matter

The classic DORA metrics focus on software delivery. Adapting and modifying them for infrastructure provisioning can help measure how effectively your platform delivers reliable, self-service cloud resources:

Lead Time for Infrastructure Changes: How fast does a merged PR turn into running infrastructure? Shorter indicates efficient automation and fewer manual steps.
Infrastructure Drift Frequency: Are developers constantly making console tweaks, or does your infrastructure stay in sync with code? A low drift rate means your code and actual resources stay aligned.
Infrastructure Change Failure Rate: What percentage of infrastructure changes result in incidents or rollbacks? A low failure rate indicates a robust, well-tested infrastructure pipeline, reducing wasted effort and potential outages.
Platform Self-Service Rate: What percentage of provisioning changes are implemented by the requesting team (without needing to involve a central operations team)? A high self-service rate shows that your platform empowers developers and reduces bottlenecks.

Common roadblocks

Adopting Infrastructure as Code helps teams automate provisioning, but common pitfalls can slow progress before they see results. Left unaddressed, problems hidden configuration tweaks, scattered code, or reliance on a single “infrastructure hero” make it hard to keep environments consistent and secure.

By spotting these roadblocks early, teams can build a stronger foundation for a repeatable, collaborative IaC workflow.

Knowledge silos: Relying on just one person (“Go ask Sarah”) for all infrastructure tasks creates a risky bottleneck. The broader team can’t replicate or improve the process, slowing new initiatives and leaving you vulnerable if that individual is out of office or leaves the company. Moving to a collaborative, version-controlled IaC model ensures that no single engineer is a gatekeeper for new environments or configuration changes.
Overly complex or unstructured IaC: Bloated or disorganized infrastructure code leads to confusion, duplication, and frequent errors. Engineers might copy-paste configurations from previous projects or cram all resources into a single file. By refactoring IaC into smaller modules, adopting a clear folder structure, and removing dead code, teams gain consistency and accelerate ramp-up times for new services.
Learning curves: Teams new to IaC often encounter a steep learning curve. There is initially a lot to learn, and resistance from people used to quickly working through cloud UI can slow down full adoption. But time invested in learning tools and setting cultural norms pay off in agility and a platform strategy with guidance and golden paths can help reduce the burden.
Cultural inertia: The adoption of a platform engineering approach is a shift in how organizations manage infrastructure, and like all, change is not merely technical but also organizational. Resistance to change and organizational inertia can make it difficult to implement new ways of getting work done. Expecting, understanding, and working through this resistance is key to overcoming it.
Lack of comprehensive approach: IaC tools are great for provisioning, but without a full platform approach, covering monitoring, logging, and the full life cycle of cloud resources, provisioning efforts may fall short. We will see in this series how the pillars of a comprehensive approach strengthen themselves.

These roadblocks highlight why a thoughtful, well-structured approach to provisioning is essential. But that’s not enough because the next step is to make these consistent patterns easily accessible so teams can spin up entire services without starting from scratch. And that’s where “golden path” templates come in.

Platform team provisioning responsibilities

Setting up the initial IaC foundation
Creating and maintaining templates
Automating workflows
Monitoring usage and compliance
Internal Platform Evangelism

Service templates

Even with a solid IaC foundation and reusable modules, spinning up a brand-new service can still require developers to understand how to pull in the underlying infrastructure components and how to use them. That’s where templates come in.

These are pre-vetted, ready-to-use blueprints that capture your organization’s best practices for a complete service or application, including infrastructure, application, and security components. They are the key to enabling true self-service.

Think of service templates as the next step up from the reusable components we discussed earlier. While a component might represent a single “building block” (like a secure S3 bucket or a database), a service template represents an entire structure built from those blocks (like a complete web application with a database, load balancer, and containerized application). They combine those building blocks with sensible default configurations and best practices for things like logging, Kubernetes setup, and security baked in.

Beyond bundling your organization’s best practices into a single repository, these service templates can also include a preconfigured CI/CD pipeline.

This means that when a team spins up a new service from the template, they inherit not just the infrastructure modules and best-practice configurations, but a fully operational CI/CD workflow. No extra setup is required. Developers get immediate feedback on their changes, and the merge to main triggers automatic deployments that keep every environment consistent.

Instead of cobbling together snippets from old repos, teams can scaffold out a service by simply selecting the template that fits their need – whether it’s a containerized web app, a serverless function, or a data-processing pipeline – and providing a few key parameters.

These curated paths cut onboarding time, reduce configuration missteps, and ensure that the IaC modules are used in new services. If developers don’t have to remember which tags to apply or how to route logs, or even where to import infrastructure modules from, they can focus on delivering features rather than wrestling with infrastructure details. And as your platform matures, and new versions of your modules are introduced, you can update these templates, automatically sharing improvements with every new project.

Bringing it all together: The impact of effective provisioning

With reusable modules and service templates in place, and a deployment workflow that takes changes from git commits all the way to your cloud account, your platform engineering team creates a foundation that transforms how developers interact with infrastructure. This transformation yields measurable benefits:

Lower Infrastructure Change Failure Rate: Automated validations and peer reviews catch issues early, reducing the number of failed deployments.
Increased Platform Self-Service Efficiency: Developers can provision resources in minutes instead of days, boosting productivity and reducing bottlenecks.
Reduced Lead Time for Infrastructure Changes: Streamlined automation shortens the time from code commit to deployed infrastructure.
Minimized Infrastructure Drift: Fewer manual console interventions mean your actual infrastructure stays aligned with what’s declared in code.

(In the upcoming article, we’ll dig deeper into how governance guards these golden paths, ensuring that new projects comply with security and corporate policies right from day one.)

Managing infrastructure provisioning with Pulumi

As you build out your platform engineering plan, consider how Pulumi might fit in. With Pulumi, you get:

Infrastructure as Code in the language of your choice.
Component resources for creating reusable infrastructure modules that encapsulate best practices.
Pulumi Deployments for a fully managed infrastructure deployment solution.
And a Platform in the box experience.

Reliable infrastructure provisioning is the cornerstone that supports your entire platform engineering initiative. By implementing the practices outlined in this article – version-controlled IaC, automated CI/CD workflows, environment separation, and golden-path templates – you establish a foundation that enables consistency, speed, and trust.

But provisioning alone isn’t enough. In upcoming articles, we’ll explore how policy-as-code and broader governance tie into this foundation, ensuring that all those newly created services meet security and compliance standards from the moment they’re deployed. But first, we’ll look at expanding the provisioning foundation covered here into a true self-service platform, so teams can quickly spin up services without risking the organizational chaos that often accompanies speed.

Provisioning is the foundation – next in this series we’ll build on it.

Platform Pillars: Build Platforms, Not Infrastructure

Adam Gordon Bell — Wed, 26 Feb 2025 00:00:00 +0000

Software drives innovation. Development teams face pressure to ship features faster. But speed collides with infrastructure complexity. Developers struggle with cloud setups, juggle scattered tools, and wait on operations teams for resources. The result is friction and slower innovation.

This is where Platform Engineering comes in. It helps developers move faster by creating tools that actually work. A good internal platform lets teams self-serve infrastructure, find documentation, follow best practices, and focus on what they do best: writing useful software.

Building a platform isn’t about finding one perfect tool. It’s about assembling the right pieces, or pillars. These pillars define what every successful internal developer platform needs.

This series explores these key pillars of Platform Engineering, offering a practical guide to building platforms that remove barriers to developer speed. Each pillar addresses a specific challenge organizations face when scaling developer productivity. The first challenge is overcoming infrastructure chaos.

Problem: Infrastructure chaos and bottlenecks

Developers face many pain points with manual provisioning, which is why Infrastructure as Code has become so essential:

Inconsistent environments: Dev, staging, and production drift apart. “Works on my machine” becomes everyone’s excuse.
Slow processes: Console clicking and waiting on ops creates bottlenecks. Developers wait days for basic infrastructure.
Configuration drift: Manual configs lead to errors. Quick production fixes become undocumented mysteries that cause problems later.
Lack of repeatability: Spinning up new environments or replicating existing ones is a complex, time-consuming, and often unreliable process.

That’s why provisioning is the first pillar to master if you want to build a successful developer platform.

Problem: Dependency bottlenecks and limited developer autonomy

Even with automated provisioning, another challenge emerges - developers still struggle when tools are provided without considering usability.

Hard to find what you need: Developers waste time hunting for the right infrastructure modules. They end up searching docs, bothering colleagues, or rebuilding something that already exists.
Too much complexity: Most developers aren’t infrastructure experts. Learning all the options and parameters creates hesitation and mistakes.
Lack of real autonomy: Developers have the tools but not the confidence to use them. They still need approvals or fear breaking production.
Inconsistent workflows: Teams create their own ways of using infrastructure, making collaboration harder.

By implementing self-service infrastructure provisioning, you transform your platform from a centrally controlled system to a developer-centric environment. You unlock developer autonomy, accelerate development cycles, and free up operations teams to focus on higher-value activities.

Problem: Inefficient and fragmented development processes

Infrastructure is just one piece of the puzzle. Developers still face daily friction moving code to production.

A typical developer’s day is full of frustrations:

Inconsistent local environments: Variations in local setups, dependency conflicts, and operating system differences lead to debugging headaches and wasted time getting started.
Slow CI/CD pipelines: Long builds, flaky tests, and mysterious pipeline configs block progress. Developers stare at progress bars instead of writing code.
Cumbersome testing and staging setups: Provisioning environments for testing and staging can be a manual process. Environments may not accurately mirror production, leading to surprises and integration issues late in the development cycle.
Lack of discoverability and reusability: Nobody knows what code already exists, so everyone rebuilds it. Duplication becomes the norm.
Developer toil and context switching: Developers spend more time on environment setup and troubleshooting than on writing features.

These inefficiencies add up, creating significant friction in the developer workflow. Developers spend more time wrestling with processes and less time focused on building and innovating. A Platform approach can help solve these challenges.

Problem: Security as a bottleneck and source of friction

Speed and efficiency matter, but security can’t be compromised. Traditionally, security acts as a gate – a necessary hurdle before releasing software. Platform Engineering aims to transform security from a bottleneck into an enabler.

For many developers, security feels like a roadblock, leading to common friction points:

Last-minute security: Checks happen at the end of development, causing costly rework when issues are found.
Restrictive policies: Security rules feel arbitrary and unclear, slowing down deployments.
The “No” department: Security teams are seen as blockers who enforce policies without offering solutions.
Speed vs. security mindset: Everyone believes you must choose between going fast and being secure.
Security is “Someone else’s job”: Developers think security is only the security team’s responsibility.

This “security as a gate” approach not only slows things down but leads to workarounds as developers try to bypass security to meet deadlines.

By integrating security into your platform, you transform security from a gatekeeper into a built-in feature of the development process. Security becomes an enabler of trust and speed, letting developers innovate with confidence.

Problem: Data overload without insights

Developers in modern cloud environments often find themselves in a paradox: they are drowning in data, yet starving for actionable information. Monitoring systems pump out metrics, logs, and traces, but finding what matters feels impossible. An ideal internal developer platform shifts the focus from data collection to actionable insights.

Here’s what developers typically experience:

Tool sprawl: Teams use different tools for metrics, logs, and traces. Developers waste time jumping between dashboards and manually connecting the dots.
Alert noise: Developers get bombarded with alerts lacking context or clear next steps. Important signals get buried in the noise.
Dashboard overload: Existing dashboards overwhelm developers with granular metrics that are hard to connect to application behavior.
Reactive debugging: Troubleshooting becomes a time-consuming fire drill. Developers spend hours sifting through logs to pinpoint issues.
Missing proactive insights: Observability is only used for firefighting. Developers lack insights to optimize performance or prevent issues before they affect users.
Disconnected from production: Many developers feel blind to how their code actually runs in production, limiting their ability to improve it.

This data flood without clear insights leaves developers reactive rather than proactive, hampering their ability to build reliable applications.

By improving observability in your platform, you transform it from an operations function into a developer superpower. Developers gain the insights they need to build better applications with greater confidence, using AI to focus on what truly matters.

Problem: Balancing autonomy with control and cost efficiency

Empowering developers with self-service is essential, but unchecked autonomy leads to chaos. As platform adoption grows, organizations need to maintain control without creating bottlenecks. This is where Governance as Code becomes critical, establishing guardrails while preserving speed.

Self-service creates new challenges:

Configuration drift: In decentralized environments, infrastructure setups diverge from standards, creating security risks.
Cloud cost explosion: Without proper controls, self-service can lead to wasted resources and budget overruns. Teams over-provision or forget to clean up unused environments.
Compliance headaches: Meeting regulatory requirements becomes harder when infrastructure is created in an uncontrolled way.
Audit gaps: Without clear governance, tracking who created what and when becomes nearly impossible.
Manual governance slows everything: If governance relies on approvals and tickets, it defeats the purpose of self-service.

These challenges highlight the need for automated governance that scales with your platform, ensuring control without slowing developers down.

By implementing Governance as Code, you ensure that self-service doesn’t conflict with security, compliance, and cost efficiency. Governance becomes an automated part of your platform, enabling responsible scaling.

Conclusion

These six challenges represent the core obstacles standing between your organization and true developer velocity. When properly addressed, each transforms into a capability that empowers developers and accelerates innovation.

The upcoming articles in this series will explore each area in depth, with practical strategies and real-world examples to help you build an effective internal developer platform.

Ready to implement these principles in your organization? Check out “Platform Engineering: Recommended Practices for Infrastructure as Code”. It provides detailed guidance on building scalable internal developer platforms.

How to structure infrastructure code for scalability
Managing development environments with developer stacks and Git branches
Designing modular and reusable Pulumi projects
Implementing cross-stack dependencies with stack references
Setting up RBAC, security, and governance at scale
Automating infrastructure management with stack orchestration

Get the Free Platform Engineering Ebook →

And don’t miss the next article in this series, where we’ll explore provisioning as the foundation of your developer platform journey.

Kubernetes Best Practices I Wish I Had Known Before

Engin Diri — Mon, 20 Jan 2025 00:00:00 +0000

Kubernetes best practices are the production-tested rules that keep clusters reliable, secure, and affordable: set resource requests and limits on every container, isolate workloads with namespaces and NetworkPolicies, enforce least-privilege RBAC, automate health checks, ship via GitOps, validate every change with policy-as-code, generate an SBOM for every image, and manage Kubernetes itself with infrastructure as code instead of hand-rolled YAML. The 20 practices below cover what production teams actually do in 2026 — not what tutorials suggest.

TL;DR: 20 Kubernetes best practices for 2026

Set resource requests and limits on every container.
Use namespaces plus ResourceQuota and LimitRange for isolation.
Run one container per Pod unless you genuinely need a sidecar.
Manage manifests with Helm, Kustomize, or Pulumi — never raw copies.
Use the Gateway API (ingress-nginx is retired) for north-south traffic.
Configure liveness, readiness, and startup probes for every workload.
Enforce RBAC with the principle of least privilege from day one.
Apply Pod Security Admission at the restricted level by default.
Lock down east-west traffic with NetworkPolicy (default-deny).
Store secrets in an external secret manager (Pulumi ESC, Vault, AWS Secrets Manager) — not as plain Kubernetes Secrets.
Autoscale with HPA, VPA, and Cluster Autoscaler / Karpenter.
Monitor with Prometheus, Grafana, and OpenTelemetry; alert before users do.
Deploy with GitOps (Argo CD or Flux) — Git is the only source of truth.
Gate every change with policy-as-code (Pulumi CrossGuard, OPA/Gatekeeper, Kyverno).
Generate and verify an SBOM for every image; sign with Sigstore/cosign.
Manage cost with FinOps practices: right-size, schedule, and chargeback by namespace.
Keep clusters and add-ons patched on a documented cadence.
Use labels and annotations consistently for ownership, cost, and SLO tracking.
Separate dev, staging, and prod clusters — or virtualize with vCluster.
Treat clusters as cattle, not pets: rebuild from code, never edit live.

Last updated April 30, 2026 — added GitOps, policy-as-code, SBOM/supply-chain, autoscaling, NetworkPolicy, and FinOps sections; added an anti-pattern table; restructured headings around the questions teams actually ask.

Going to KubeCon Europe 2026? Tame Kubernetes complexity with code.

Pulumi demos, platform engineering talks, AI-powered Neo in action, and yes, free plushies. Booth 784.

The ‘Kubernetes is easy’-iceberg meme is a classic example of how Kubernetes can be deceivingly complex

What are Kubernetes best practices?

Kubernetes best practices are the operational patterns that keep workloads available, secure, observable, and cost-efficient on a Kubernetes cluster. They span four areas: workload configuration (resources, probes, images), security (RBAC, NetworkPolicy, Pod Security, secrets, supply chain), delivery (GitOps, policy-as-code, IaC), and platform operations (autoscaling, monitoring, upgrades, FinOps). The rest of this post explains each one with a copy-pasteable rule, a definition of the underlying Kubernetes object, and a common anti-pattern to avoid.

Common Kubernetes anti-patterns vs. the correct approach

Common mistake	What goes wrong	Correct approach
No resource requests or limits	Noisy-neighbor evictions, OOMKills, unschedulable Pods	Set requests for every container; use VPA recommendations to tune
Everything in the `default` namespace	No isolation, no quota, no RBAC boundary	One namespace per team or environment + ResourceQuota + RBAC
`cluster-admin` for service accounts	Full-cluster blast radius on compromise	Per-namespace Role + RoleBinding (least privilege)
Plain Kubernetes `Secret` objects	Base64 ≠ encryption; secrets leak via etcd, kubectl, audit logs	External Secrets Operator + Pulumi ESC, Vault, or AWS Secrets Manager
Open east-west traffic	One compromised Pod can reach every service	Default-deny NetworkPolicy; allow only required flows
`kubectl apply -f` from a laptop	No history, no review, no rollback	GitOps with Argo CD or Flux; Git is the source of truth
Manual `latest` image tags	Non-reproducible deploys, supply-chain risk	Pinned digests (`@sha256:...`) with cosign verification
Ingress-nginx in 2026	Project retired March 2026 — no security fixes	Migrate to the Gateway API (Envoy Gateway, Istio, Kgateway)
Hand-rolled YAML in dozens of repos	Drift, copy-paste bugs, no testing	Pulumi, Helm, or Kustomize with reviews
One shared `prod` cluster for everything	Blast radius spans every team	Separate clusters or vCluster virtual clusters

1. How do you set Kubernetes resource requests and limits?

What are resource requests and limits? A resource request is the minimum CPU and memory the kube-scheduler guarantees a container; a resource limit is the maximum the kubelet allows it to consume before throttling CPU or OOMKilling memory. Pods without requests are scheduled blind, so a noisy neighbor can evict your workload at any time.

The rule: every container in production sets a request for both CPU and memory, and a memory limit. CPU limits are optional and frequently counter-productive — they cause throttling spikes that look like outages.

Start at 100–200m CPU and 128–512Mi memory for a typical web service.
Tune from real data using Vertical Pod Autoscaler with updateMode: Off (recommendation-only) or Prometheus histograms.
Use a per-namespace LimitRange so teams cannot ship a container with no requests at all.

2. How should you structure Kubernetes namespaces?

What is a Kubernetes namespace? A namespace is a logical partition of cluster resources. It scopes object names, RBAC bindings, NetworkPolicies, and quotas — but does not provide hard isolation by itself.

The rule: never deploy production workloads to default. Map namespaces to ownership boundaries — usually one per team, environment, or tenant — and pair every namespace with three things:

A ResourceQuota to cap aggregate CPU, memory, and object counts.
A LimitRange to enforce per-container defaults.
A RoleBinding that grants only the permissions that team needs.

This is the smallest unit that gives you predictable cost, blast-radius containment, and an RBAC perimeter.

3. Should you run multiple containers in one Pod?

What is a Pod? A Pod is the smallest deployable unit in Kubernetes — one or more containers that share a network namespace, IPC, and storage volumes, and that are scheduled and scaled as a single unit.

The rule: one container per Pod by default. Add a second container only when it must share the Pod’s network or volumes — for example, a service-mesh sidecar (Istio/Linkerd), a logging shipper, or an init container that prepares state before the main process starts. Multi-container Pods couple lifecycles and scaling, so reach for them deliberately, not by default.

You might also like:

4. Should you use Helm, Kustomize, or Pulumi for Kubernetes manifests?

The rule: never copy-paste raw YAML across services. Pick one of these:

Helm — Go-templated charts; the de-facto distribution format for off-the-shelf software (Prometheus, cert-manager, Argo CD).
Kustomize — overlay-based, no templating; ships in kubectl. Best for “same app, different environment” diffs.
Pulumi — real programming languages (TypeScript, Python, Go, Java, .NET) for Kubernetes plus the cloud resources around it (EKS, GKE, AKS, DNS, IAM). Type checking, tests, and the Pulumi Kubernetes Operator for GitOps-style reconciliation.

If you are already managing cloud infrastructure with code, extending the same Pulumi program to Kubernetes is the smallest cognitive jump — the cluster, its IAM, its DNS, and its workloads live in one stack and one review.

5. How should you handle ingress and networking on Kubernetes in 2026?

What is the Gateway API? The Gateway API is the successor to the legacy Ingress resource. It splits responsibility between infrastructure (GatewayClass, Gateway) and application (HTTPRoute, GRPCRoute, TLSRoute) teams, and it is the SIG-Network direction for north-south traffic.

The rule: new clusters should standardize on the Gateway API. The widely deployed ingress-nginx controller retired in March 2026, and remaining deployments should migrate to Envoy Gateway, Istio, Linkerd Gateway, or Kgateway.

Terminate TLS at the gateway and automate certificates with cert-manager.
Use path-based and host-based routing instead of one gateway per service.
Protect every gateway with a WAF (managed cloud WAF or Coraza on Envoy).

ingress-nginx is retiring in March 2026.

Here is a guide on How to Move to the Gateway API: post ingress-nginx Retirement.

6. What’s the difference between liveness, readiness, and startup probes?

Probes tell the kubelet whether to restart a container, route traffic to it, or wait for it. Configure them per workload:

Liveness probe — restarts a hung container. Use sparingly; a bad liveness probe is a self-inflicted DoS.
Readiness probe — gates traffic from Services and Gateway routes. Use for every network-facing workload.
Startup probe — gives slow-booting apps (JVM, ML models) time to come up before liveness checks begin. Use whenever cold-start exceeds 10 seconds.

See the upstream guide for HTTP, TCP, and exec probe syntax. Tune failureThreshold, periodSeconds, and timeoutSeconds based on real latency, not defaults.

7. How do you secure a Kubernetes cluster?

Cluster security is layered: identity, workload posture, network, and secrets. Get all four right.

What is RBAC in Kubernetes?

RBAC (Role-Based Access Control) is the Kubernetes authorization model that maps subjects (users, groups, ServiceAccounts) to verbs (get, list, create, delete) on resources, scoped to a namespace (Role) or the whole cluster (ClusterRole).

Grant least privilege — never bind humans or workloads to cluster-admin.
Prefer per-namespace Role + RoleBinding over ClusterRoleBinding.
Audit with kubectl auth can-i --as=... and tools like rbac-lookup.

What is Pod Security Admission?

Pod Security Admission is the built-in admission controller that enforces the three Pod Security Standards (privileged, baseline, restricted) at the namespace level.

Label every namespace pod-security.kubernetes.io/enforce=restricted by default.
Drop NET_RAW and all capabilities; run as non-root with a read-only root filesystem.
Use seccompProfile: RuntimeDefault and a tight securityContext for every container.

What is a NetworkPolicy?

A NetworkPolicy is a Kubernetes object that defines allowed ingress and egress traffic for selected Pods at the IP/port level. Without one, every Pod can reach every other Pod by default.

Apply a default-deny ingress and egress policy in every namespace.
Allow only the flows your service actually needs (DNS, the database, the upstream API).
For richer L7 controls (mTLS, JWT auth), layer a service mesh on top.

How should you manage Kubernetes secrets?

Built-in Secret objects are base64-encoded, not encrypted. Treat them as a transport, not as storage.

Store the source of truth in Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault.
Project secrets into Pods with the External Secrets Operator or the Secret Store CSI Driver.
Enable etcd encryption-at-rest with a KMS provider so leaked etcd snapshots stay opaque.
Rotate automatically; never commit secrets to Git, even encrypted.

8. How do you set up Kubernetes observability?

Observability rests on the three pillars: metrics, logs, and traces. The 2026 default stack is OpenTelemetry-first.

Metrics — Prometheus (or a Prometheus-compatible TSDB like Mimir, Thanos, VictoriaMetrics) plus Grafana dashboards.
Logs — Fluent Bit or the OpenTelemetry Collector shipping structured JSON to Loki, Elastic/OpenSearch, or a managed service. Never depend on kubectl logs for incidents — Pods are ephemeral.
Traces — OpenTelemetry SDKs in your apps emitting to Tempo, Jaeger, or a vendor backend. Correlate trace IDs with logs.
Alerts — page on SLO burn rate, not on every crashed Pod. Burn-rate alerting catches real user impact without firing on every restart.

9. How do you automate Kubernetes deployments with GitOps?

What is GitOps? GitOps is a deployment model where the desired cluster state lives in a Git repository and a controller continuously reconciles the live cluster to match. Changes happen through pull requests, not kubectl apply.

The rule: every cluster has a controller — Argo CD, Flux, or the Pulumi Kubernetes Operator — that owns reconciliation. Humans never kubectl apply to production.

Use app-of-apps (Argo) or Kustomization trees (Flux) to bootstrap whole clusters from a single root.
Keep manifests, Helm values, and Pulumi stack references in Git, with environment promotion via PR.
Combine GitOps with Pulumi Deployments so cloud infrastructure (VPCs, EKS clusters, IAM, DNS) and the workloads on top promote through the same review pipeline. See improving GitOps with the Pulumi Operator for a worked example.
Configure automated rollback on health-check failure, and drift detection alerts when someone edits live state.

10. How often should you upgrade Kubernetes?

Kubernetes ships a minor release roughly every four months and supports each one for about 14 months. Treat upgrades as routine maintenance, not a project.

Stay within two minor versions of upstream — older versions miss CVE patches.
Test upgrades in a non-prod cluster, run a conformance test, then promote.
Back up etcd before control-plane changes; managed services (EKS, GKE, AKS) handle most of this for you.
Upgrade add-ons (CNI, CSI, Gateway controller, cert-manager, metrics-server) on the same cadence — pin versions in code so the upgrade is auditable.
Watch the Kubernetes deprecation guide and run pluto or kubent to find deprecated APIs before they break.

11. How should you label and annotate Kubernetes resources?

Labels and annotations are the metadata layer that everything — Services, NetworkPolicies, monitoring, cost tools, GitOps — keys off of. Consistency matters more than cleverness.

Adopt the recommended labels: app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/version, app.kubernetes.io/component, app.kubernetes.io/part-of, app.kubernetes.io/managed-by.
Add organization-specific labels for owner, cost-center, environment, and SLO tier — your FinOps and on-call processes will need them.
Reserve annotations for non-identifying metadata: build SHA, change-ticket, last-deployed timestamp, ingress controller hints.

12. How should you separate Kubernetes environments?

The rule: production is its own cluster. Dev and staging can share — sometimes. Mixing prod with anything else couples blast radius, upgrade cadence, and quotas.

Separate clusters per environment is the safe default. With managed control planes (EKS, GKE, AKS) the cost is small.
Virtual clusters with vCluster give each team a Kubernetes API of their own without the operational cost of a real cluster.
Namespace-only segregation can work for dev/staging if you enforce strict NetworkPolicy, RBAC, and ResourceQuota boundaries.
Provision all of them from the same Pulumi program with different stacks — dev, staging, prod — so promotion is a config change, not a copy.

13. How do you optimize Kubernetes container images?

Start from a distroless (gcr.io/distroless) or chainguard/static base. Alpine is fine if you accept musl.
Build with multi-stage Dockerfiles so build tools never ship in the final image.
Pin the base image by digest (@sha256:...), not by tag — latest and 1.21 move under you.
Scan every image with Trivy, Grype, or your registry’s built-in scanner. Block builds on Critical/High CVEs.
Smaller images mean faster pulls, faster autoscaling, and a smaller attack surface.

14. What’s a reliable Kubernetes logging strategy?

Centralize to Loki, Elastic/OpenSearch, or a managed service via Fluent Bit or the OpenTelemetry Collector running as a DaemonSet.
Structure logs as JSON with consistent field names (trace_id, span_id, service, level).
Retain by tier — 7 days hot, 30–90 days warm, archive to object storage for compliance.
Redact secrets and PII at the collector, not after the fact.

You might also like:

15. How should you autoscale workloads on Kubernetes?

What is the Horizontal Pod Autoscaler? The HPA scales the replica count of a Deployment or StatefulSet up and down based on CPU, memory, or custom/external metrics.

What is the Vertical Pod Autoscaler? The VPA adjusts a Pod’s CPU and memory requests over time based on observed usage. Run it with updateMode: Off (recommendation-only) in production and apply the recommendations through your IaC tool.

What are Cluster Autoscaler and Karpenter? Cluster Autoscaler adds and removes nodes to match Pod demand. Karpenter is a faster, group-less node provisioner now widely used on EKS and gaining traction on other clouds.

The rule: every production workload has an HPA. Every cluster has a node-level autoscaler. Set sensible min/max replicas, define a PodDisruptionBudget, and gate scale-down behavior so traffic spikes don’t pile up at restart time. Pair with KEDA for event-driven scaling (queues, Kafka lag, scheduled scale).

16. How do you enforce policy-as-code on Kubernetes?

What is policy-as-code? Policy-as-code expresses governance rules — security baselines, tagging, region restrictions, cost guardrails — as code that runs in CI and at admission time, blocking non-compliant changes before they reach the cluster.

The rule: every change goes through at least one policy engine.

Pre-deploy — validate Pulumi or Helm output with Pulumi CrossGuard or Conftest in CI.
Admission-time — install OPA Gatekeeper or Kyverno and require, for example, signed images, a team label, and a non-root securityContext on every Pod.
Continuous — scan running clusters with Pulumi Insights or kubescape for drift from policy.

See the benefits of policy-as-code and enforcing policy-as-code on discovered resources for end-to-end examples.

17. How do you secure the Kubernetes software supply chain?

What is an SBOM? A Software Bill of Materials is a machine-readable inventory of every component in a container image — packages, versions, licenses, hashes — typically in SPDX or CycloneDX format.

The rule: every image you run in production has a verified SBOM and a verified signature.

Generate SBOMs at build time with Syft or docker sbom.
Sign images and SBOMs with Sigstore/cosign keylessly via GitHub OIDC.
Verify signatures at admission with Kyverno or Connaisseur.
Generate provenance attestations (SLSA) so you can prove which build pipeline produced a given image.
Pin base images by digest and rebuild on a cadence so CVE patches actually land.

This is what regulators and customers ask for in 2026 — Executive Order 14028, the EU Cyber Resilience Act, and most enterprise procurement checklists all require it.

18. How do you manage Kubernetes cost (FinOps)?

The rule: Kubernetes cost is a labeling and right-sizing problem before it is a discount problem.

Right-size with VPA recommendations, Goldilocks, or your APM’s recommendation engine. Most workloads request more CPU than they actually use; right-sizing is usually the largest single saving.
Schedule non-prod clusters off overnight and on weekends.
Spot/preemptible nodes for fault-tolerant workloads; use Karpenter’s consolidation to compact bin-packing.
Show back / charge back by namespace and label using OpenCost or Kubecost. Without per-team attribution, no team owns cost.
Provision the underlying cluster with IaC — see Pulumi for Kubernetes — so node types, autoscaling limits, and reserved capacity are reviewable artifacts. Pulumi’s hidden-cost analysis walks through where the money actually goes.

19. Why should you treat Kubernetes clusters as cattle, not pets?

The old “cattle, not pets” adage applies more strictly to Kubernetes than to VMs. Manual edits to a live cluster will be reconciled away, drift between environments, or vanish on the next Pod restart.

Fix problems in code — YAML, Helm chart, or Pulumi program — and let the controller redeploy.
If you can’t redeploy a cluster from Git in under an hour, you have a pet.
Use ephemeral preview environments for PRs; use blue/green or progressive delivery (Argo Rollouts, Flagger) for production cutovers.

20. Why use Pulumi to manage Kubernetes?

Native YAML scales until your team doesn’t. Once you have more than a handful of services, dependencies between cloud resources and Kubernetes objects, or more than one cluster, infrastructure as code in a real programming language wins on every axis.

Real languages — TypeScript, Python, Go, Java, .NET — for type-safe, testable Kubernetes infrastructure. Loops, conditionals, and unit tests instead of templating.
One stack, full topology — manage the cloud (EKS/GKE/AKS, VPC, IAM, DNS) and the workloads in it together. See easily create and manage AWS EKS clusters with Pulumi.
Reusable components — abstract platform patterns into Pulumi packages other teams import instead of copy-pasting.
GitOps reconciliation — the Pulumi Kubernetes Operator reconciles a cluster to a Pulumi stack on every Git push.
Policy-as-code built in — CrossGuard blocks non-compliant changes before pulumi up.
Secrets done right — Pulumi ESC federates secrets and configuration across stacks, environments, and Kubernetes clusters.

For a deeper comparison of hand-written YAML, Terraform, and Pulumi for Kubernetes, see YAML, Terraform, Pulumi: what’s the smart choice for deployment automation with Kubernetes, and the beyond YAML write-up on where Kubernetes automation is heading in 2026.

See how Pulumi helps you manage Kubernetes Services and Deployments with code instead of YAML—bringing structure, reuse, and type safety to your configs.

By adopting Pulumi, you can avoid the complexity of juggling endless YAML files and gain a more streamlined, maintainable workflow for your Kubernetes infrastructure.

Final thoughts

Kubernetes rewards discipline. The 20 practices above — resource hygiene, namespaced isolation, RBAC, NetworkPolicy, Pod Security, external secrets, probes, the Gateway API, observability, GitOps, policy-as-code, signed images and SBOMs, autoscaling, FinOps, multi-cluster strategy, and IaC — are the dial settings that separate clusters that page their owners every week from clusters that don’t.

Pick the three weakest spots in your environment and fix those first. Then promote the fixes through code, not through kubectl.

Want to learn how to put these practices into action? Meet us at KubeCon Europe 2026 (Booth 784) or register for our upcoming Zero to Production in Kubernetes workshop.

Try Pulumi for Free

Secret Rotation with Pulumi ESC

Claire Gaestel — Tue, 10 Dec 2024 15:43:02 -0800

Pulumi ESC now natively supports secrets rotation that makes secrets lifecycle management much easier. Check out the launch blogpost and docs.

Managing secrets in modern cloud applications can be challenging, particularly when it comes to rotation policies. While dynamic secrets (like AWS IAM temporary credentials) handle this automatically, many systems still rely on static secrets that require periodic rotation.

Static secrets, like database passwords or API keys, should be rotated regularly to maintain security, and services depending on these secrets need time to transition to new credentials to avoid downtime. This makes rotating credentials error-prone, and often forgotten.

In this post, we’ll explore an approach for automating static secret rotation using Pulumi ESC combined with Pulumi IaC.

A pattern for extending ESC

We can take advantage of ESC’s integration with Pulumi Deployments to create an ergonomic way of managing rotation schedules ourselves.

We’ll start by defining a custom declarative configuration format for managing a rotation schedule:

# we'll create our own "extended" fn config for ESC
xfn::pulumi-scheduled-update:
 # stack reference to a pulumi program that manages a rotated secret
 stack: "example-rotator-stack-reference/dev"
 # schedule that will drive an automatic scheduled deployment
 scheduleCron: "0 0 * * 0"
 # trigger a manual unscheduled rotation whenever this value changes
 trigger: "break-glass"

Using webhooks, we’ll monitor when the environment is updated, then search for these configuration blocks, and use them to drive scheduled deployments of a stack that rotates credentials. Using ESC’s composition primitives, we can combine this rotation schedule config with a fn::open::pulumi-stacks provider to read the current secret value from the rotator stack to provide to our application.

Here is how the solution fits together: Let’s break down each component and see how they work together to solve our rotation challenges.

The Rotator: Managing Credential Lifecycles

At the heart of our solution is a generic Rotator component that manages credential pairs. It’s designed to handle any type of static secret while ensuring zero-downtime rotations. The way it does this is by maintaining two versions of each secret: “current” and “previous”. Each time the stack is deployed, “current” is replaced with a newly provisioned credential, the old value is demoted to “previous”, and the old previous credential is decommissioned. Crucially, the demoted secret remains valid, which allows consuming services enough time to switch over to the new secret.

const creds = new Rotator("rotating-creds", {
 // rotate credentials on every deployment.
 trigger: Date().toString(),
 // create a set of equivalent credentials that will be rotated between.
 // a particular credential should be replaced whenever `trigger` is updated.
 construct: (name, trigger) => {
 // in this example we're just creating passwords,
 // but these could be mysql users or anything else.
 return new random.RandomPassword(name, {
 length: 10,
 keepers: {trigger}
 })
 },
})

// export when the last rotation happened
export const lastUpdate = creds.lastUpdate.apply(date => date.toISOString());
// export the currently active credential, which will be imported by the downstream ESC environment.
export const current = creds.current.result;

The Scheduler: Orchestrating Rotations

The scheduler component acts as an orchestrator, watching an ESC environment for updates and managing deployment schedules of the rotator stacks. Whenever a change to the environment is saved, a webhook invokes the scheduler, which parses the environment definition to find xfn::pulumi-scheduled-update configuration blocks. Based on these schedule configurations, it creates scheduled deployments for the referenced rotator stacks automatically.

Bringing It Together: Environment Configuration

Once wired together, the magic happens in the ESC environment configuration, where we compose a rotation schedule with dynamic credential retrieval, creating a rotated credential that is automatically kept up to date:

values:
 db-credentials:
 stack: database-credential-rotator/dev
 schedule:
 xfn::pulumi-scheduled-update:
 stack: ${db-credentials.stack}
 scheduleCron: "0 0 * * 0" # weekly rotation
 trigger: "break-glass" # manual trigger
 secrets:
 fn::open::pulumi-stacks:
 stacks:
 credentials:
 stack: ${db-credentials.stack}

 api-keys:
 stack: api-key-rotator/dev
 schedule:
 xfn::pulumi-scheduled-update:
 stack: ${api-keys.stack}
 scheduleCron: "0 0 1 * *" # monthly rotation
 secrets:
 fn::open::pulumi-stacks:
 stacks:
 keys:
 stack: ${api-keys.stack}

 environmentVariables:
 DB_CONNECTION_STRING: ${db-credentials.secrets.credentials.current}
 API_KEY: ${api-keys.secrets.keys.current}

This configuration demonstrates how our custom extension is able to seamlessly integrate with ESC to manage multiple rotating secrets with different schedules declaratively, co-located with dynamic retrieval of the latest credentials from the rotator stack outputs. Applications consuming this environment will automatically receive the latest credentials without any additional configuration.

Trying it out

Open the example environment and take note of the current secrets:

❯ pulumi env open esc-rotation-demo/test db-credentials.secrets
{
"credentials": {
"current": "OBA=wxS:VT",
"lastUpdate": "2024-11-26T17:49:03.000Z",
"previous": "MUgPXkJ+kE"
}
}

Now lets force a rotation by changing the manual trigger:

❯ pulumi env set esc-rotation-demo/test \
db-credentials.schedule.xfn::pulumi-scheduled-update.trigger \
break-glass2

After a few minutes we can see that the secret has rotated successfully! We can also observe that the previous secret remains valid, giving currently deployed consumers time to update to the new credentials.

$ pulumi env open esc-rotation-demo/test db-credentials.secrets
{
"credentials": {
"current": "X2WSmF3+29",
"lastUpdate": "2024-11-26T17:57:05.000Z",
"previous": "OBA=wxS:VT" <---- old "current" has been demoted
}
}

Conclusion

Secret rotation doesn’t have to be a manual, error-prone process. By leveraging Pulumi ESC and this component architecture, we can automate and streamline the rotation of static secrets while maintaining system stability and security. The solution is flexible enough to handle various types of secrets while being simple to configure and maintain. This approach offers several advantages:

Automated: Set-and-forget secret rotation
Zero-downtime: Smooth transitions between credentials
Flexible: Works with any type of static secret
Declarative: Configuration through ESC environments
Auditable: Clear tracking of rotation history
Scalable: Easy to manage across multiple environments

The complete example is available for your perusal here.

This pattern of extending ESC through configuration-driven components is a powerful technique that can be applied to other use cases. We’re excited to see what else you come up with 🙂

Why Every Platform Engineer Should Care About Kubernetes Operators

Engin Diri — Tue, 10 Dec 2024 09:31:38 +0100

In one of my recent talks, I mentioned that the foundation of a successful Kubernetes-powered platform is the use of Kubernetes Operators, as they are a great way to automate operational tasks and the lifecycle of complex applications and services on Kubernetes.

In case you missed the talk, here is the link to the recording:

After the talk, I received feedback from the audience that they would like to have more information about Kubernetes operators and some of the advanced operators I would recommend to use in their Kubernetes-powered platforms. And who am I to reject such a great opportunity to share my knowledge with the community?

As you guess, the base for a Kubernetes Operator is, well, Kubernetes. So, if you are not familiar with Kubernetes, let me give you a brief introduction. Kubernetes, or K8s in short, is by far the most powerful open-source container orchestration platform out there - especially for its capability to maintain a desired state. This means you, as the platform admin, can define how you want your cluster and the workload on top of it to look, and Kubernetes (aka the control plane) executes the necessary steps to ensure the desired state is achieved using a feedback loop.

And how does Kubernetes achieve this complex operational task management? Kubernetes leverages the concept of controllers and operators. Both ensure that the cluster resources are in the desired state. But why is there a distinction between controllers and operators inside the Kubernetes ecosystem? There is also great confusion when folks talk about controllers and operators, as the differences are in the details and operators are a subcategory of controllers.

Kubernetes Controllers vs. Operators

K8s controllers are control loops that watch your resources and reconcile the current state with the desired state in a continuous loop. Think about how a thermostat works in your house. The thermostat watches the current temperature and, if it is below the desired temperature, it turns on the heating. If it is above the desired temperature, it turns off the heating. This is how a controller works in Kubernetes - it watches the resources and conditions defined in the desired state and executes the necessary steps to make sure the current state matches the desired state.

Here are some typical Kubernetes controller key functions as part of the maintain, observe, and enforce loop of the control loop:

Function	Description
Rollouts and Rollbacks	Deployment controller takes care of rolling out new versions of your application and rolling back to the previous version in case of failure.
Cluster maintenance	Node controller watches the nodes in the cluster and makes sure that the nodes are in the desired state.
Event handling	Event controller watches for events and takes action based on the events.
Schedule	Controller to run jobs and cron jobs at a specific time.
Resource enforcement	Resource controller watches the resource usage and enforces resource limits in a namespace.
State enforcement	Different controllers watch the state of the resources and enforce the desired state.

As you can see, controllers automate the routine operational tasks, enforce policies, handle failures and take actions to continuously maintain the desired state of the cluster.

Some of the typical use cases for controllers are:

Limit resource usage - Prevent the overload of the cluster by enforcing resource limits with controllers like ResourceQuota.
Auto-scaling - Automatically scales the number of pods based on the different metrics with HorizontalPodAutoscaler.
Task scheduling - Run jobs and cron jobs at a specific time with Job and CronJob controllers.
Run stateful workloads - Deploy and manage stateful workloads like databases with StatefulSet controller, by ensuring that the pods are created in a specific order and have a stable network identity.
Scaling deployments - Deploy and manage stateless workloads like web applications with the Deployment controller, by ensuring that the desired number of pods is running and handling the rolling updates and rollbacks.

Controllers, as you see, reduce the manual effort for the platform engineering teams and gives the cluster self-managing capabilities.

With all the above points, the benefits of controllers are clear - Let me list some of the main ones:

Automation - Controllers automate the routine operational tasks and enforce policies.
Availability - Controllers ensure that the resources are always available, addressing failures and taking actions to maintain the desired state.
Efficiency - Controllers provide an efficient way to manage the resources and workloads in the cluster with quota management and auto-scaling.
Reliability - Controllers help to make workload management more reliable with features like replica management, pod creation and deletion, and scheduling.
Flexibility - Controllers provide a flexible way to handle a variety of workloads and resources in the cluster, such as DaemonSet, StatefulSet, jobs, cron jobs, deployments, and services.
Observability - Controllers provide a view into the cluster resources and the state of the resources via the Kubernetes API.

With Controllers explained, let’s move on to the next level of automation in Kubernetes and talk about Kubernetes Operators.

Kubernetes Operators

As mentioned before, Kubernetes Operators are a subcategory of controllers that use API extensions (Custom Resource) to complete the automation task. Operators are a set of independent controllers, with each controller responsible for its own task. And while controllers can share similar functions with a controller, it is only focused on one domain and only uses Custom Resources to manage the domain.

Keep in mind that controllers work without the need for custom resources or a link to a specific domain.

Operators are custom-built controllers that focus on the deployment, management, and operation of a specific application or service. Some key functions of operators include:

Function	Description
Security	Operators integrate security best practices and policies, such as encryption, access control, and secret management, to protect the application. You can customize these measures and adjust them to the specific needs of the application.
Workload Lifecycle management	Operators manages the full lifecycle of complex stateful applications through deployment, upgrades, and scaling. It even applies logic to automate operations like backup and restore.
Error Handling	Operators detects and handles errors, such as application crashes, network failures, and resource constraints.
Migration	Operators help to migrate the application to a new version, a new cluster, or a new cloud provider.
Disaster Recovery	Operators help to recover the application from a disaster, such as data loss, network failure, or hardware failure.

Operators are a great way to encode the operational knowledge of the platform engineering team into automated controllers. This way, the platform engineering team can focus on the strategic tasks and let the operators handle the boring routine operational tasks. This reduces the manual effort and increases the reliability of the platform.

Some of the typical use cases for operators are:

Database management - Deploy and manage databases such as MySQL, PostgreSQL, and MongoDB.
Storage management - Deploy and manage storage such as Ceph, GlusterFS, or NFS.
Logging and monitoring - Simplify the deployment and management of logging and monitoring solutions like Prometheus, Grafana, or ELK stack.
CI/CD - Automate the deployment and management of CI/CD pipelines like Jenkins, GitLab, or Tekton. - Backup and restore - Automate the backup and restore of applications and data.
Messaging and event streaming - Simplify the deployment and management of messaging and event streaming solutions like Kafka, RabbitMQ, or NATS.
Machine learning - Automate the deployment and management of machine learning workloads such as TensorFlow, PyTorch, or Kubeflow.

With the applications-centric approach provided by operators, the platform engineering team can now enable fully automated operations while enhancing the reliability, efficiency, and observability of the platform and ensuring adherence to best practices and compliance requirements.

And similar to controllers, operators also have distinct benefits:

Simplification - Operators simplify the deployment and management of complex applications and services by providing a declarative way (via Custom Resources).
Higher Productivity - Operators increase the productivity of the platform engineering team by automating the routine operational tasks and allowing users to focus on delivering business value.
Extensibility - Extending the K8s API with Custom Resource Definitions (CRDs) allows administrators to support new applications and services without changing the core Kubernetes code.
Consistency - Operators ensure that they deploy and run the applications and services consistently across different environments, such as on-premises, cloud, or hybrid.
Modularity - Operators are modular and focused on a specific domain.
Legacy Transformation - Operators help to transform legacy applications into cloud-native applications by automating the deployment and management of complex applications.

Here is a visual representation of a Kubernetes Operator in action, which I found very nice and helpful:

Kubernetes Operators: what are they? Some examples Credit: SparkFabrik

Comparison of Kubernetes Controllers and Operators

Here is a quick summary of the differences between Kubernetes controllers and operators:

Feature	Kubernetes Controllers	Kubernetes Operators
Definition	Core control loops within Kubernetes that reconcile the current state with the desired state of resources.	Custom-built controllers with domain-specific logic, typically extending Kubernetes API with Custom Resource Definitions (CRDs).
Scope	General-purpose automation for cluster-wide resource management and workload operations.	Focused on application or service-specific lifecycle management and operations.
Custom Resources (CRDs)	Not mandatory. Most controllers work with built-in Kubernetes resources like Pods, Deployments, and StatefulSets.	Mandatory. Operators depend on CRDs to define and manage application-specific resources.
Domain-Specific Logic	Limited. Primarily focuses on generic Kubernetes operations like scheduling, scaling, and resource enforcement.	Extensive. Includes application-specific tasks such as backups, upgrades, migrations, and disaster recovery.
Automation	Automates operational tasks like resource scheduling, scaling, and maintaining cluster states.	Automates both operational and domain-specific tasks, often encoding deep operational knowledge of specific applications.
Complexity	Relatively low complexity as they focus on Kubernetes-native workflows.	Higher complexity due to domain-specific implementations and integrations with external systems.
Observability	Provides insight into cluster resource states via Kubernetes API and built-in events.	Enhances observability with application-specific metrics, logs, and health checks tailored to the managed application.
Use Cases	- Pod scaling (e.g., HPA) - Resource enforcement (e.g., ResourceQuota) - Rolling updates and rollbacks - Scheduling jobs (e.g., CronJob).	- Database management (e.g., MySQL Operator) - CI/CD pipelines (e.g., Tekton) - Backup and restore - Monitoring solutions (e.g., Prometheus Operator) - Stateful applications (e.g., Kafka Operator).
Deployment	Part of Kubernetes core components or extensions like kube-controller-manager.	External tools developed by vendors, community, or in-house, deployed using Custom Resources.
Key Benefits	- Cluster-wide automation - Ensures resource availability - Efficient and reliable workload management.	- Simplifies application lifecycle management - Increases productivity with domain-specific automation - Consistent deployments across environments.
Flexibility	Flexible for general Kubernetes operations but limited to built-in resources.	Highly flexible and extensible for custom applications and domain-specific requirements.
Target Audience	Platform administrators and Kubernetes users looking to automate Kubernetes-native workflows.	Developers and platform engineers needing advanced automation and lifecycle management for specific applications or services.
Maintenance	Managed and updated by Kubernetes core or community projects.	Requires domain-specific knowledge and regular updates to accommodate application changes and best practices.

With this detailed explanation of Kubernetes controllers and operators, I would like to share some advanced Kubernetes operators that I recommend for use in your Kubernetes-powered platform.

Advanced Kubernetes Operators

As we have seen, Kubernetes Operators are the go-to way to automate operational tasks and lifecycle management. I should mention that not all operators are created equal. Some operators are more advanced and provide more features than others. That is why, for example, OperatorHub.io categorises operators into five capability levels:

Basic Install - Installs the application on the cluster.
Seamless Upgrades - Manages Application upgrades.
Full Lifecycle Management - Manages the full lifecycle of the application.
Deep Insights - Provides monitoring and metrics.
Auto Pilot - Automatically optimizes and tunes the application.

The more advanced the operator, the more features it provides and the more automation it offers. At its most advanced level, the Auto Pilot, the operator automatically optimizes and tunes the application based on the workload and environment.

Here is a list of advanced Kubernetes operators that I recommend using in your Kubernetes-powered platform:

CloudNativePG Operator

CloudNativePG Operator Credit: cloudnative-pg.io

CloudNativePG Operator is a PostgreSQL operator that covers the full lifecycle of PostgreSQL databases on Kubernetes. It simplifies the deployment, management, and scaling of PostgreSQL databases by providing a declarative way to define and manage PostgreSQL clusters using custom resources.

Some of the key features of CloudNativePG Operator are:

Automated Backups - Schedule and manage backups of PostgreSQL databases.
High Availability - Deploy and manage high-availability PostgreSQL clusters.
Scaling - Scale the number of PostgreSQL instances based on the workload.

Installation

Either use the Helm chart or the Kubernetes manifests to deploy the CloudNativePG Operator:

helm repo add cloudnative-pg https://cloudnative-pg.io/charts/
helm repo update
helm upgrade --install cloudnativepg cloudnative-pg/cloudnative-pg --namespace cnpg-system --create-namespace

Or use Pulumi to deploy the CloudNativePG Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const ns = new k8s.core.v1.Namespace("cnpg-system", {
 metadata: {
 name: "cnpg-system",
 },
});

new k8shelm.Release("cloudnativepg", {
 chart: "oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 namespace: ns.metadata.name,
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

ns = k8s.core.v1.Namespace(
 "cnpg-system",
 metadata=meta.ObjectMetaArgs(
 name="cnpg-system",
 ),
)

release = k8shelm.Release(
 "cloudnativepg",
 chart="oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 namespace=ns.metadata["name"],
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 ns, err := corev1.NewNamespace(ctx, "cnpg-system", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("cnpg-system"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "cloudnativepg", &helmv3.ReleaseArgs{
 Chart: pulumi.String("oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg"),
 Namespace: ns.Metadata.Name().Elem(),
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class MyStack : Stack
{
 public MyStack()
 {
 var ns = new Namespace("cnpg-system", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "cnpg-system"
 }
 });

 var release = new Release("cloudnativepg", new ReleaseArgs
 {
 Chart = "oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 Namespace = ns.Metadata.Apply(m => m.Name),
 CreateNamespace = true
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is a simple example of a PostgreSQL cluster with backup configured on S3 API-compatible storage:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: cluster-example-with-backup
spec:
 instances: 3
 primaryUpdateStrategy: unsupervised

 # Persistent storage configuration
 storage:
 storageClass: standard
 size: 1Gi

 # Backup properties
 # This assumes a local minio setup
 backup:
 barmanObjectStore:
 destinationPath: s3://backups/
 endpointURL: http://minio:9000
 s3Credentials:
 accessKeyId:
 name: minio
 key: ACCESS_KEY_ID
 secretAccessKey:
 name: minio
 key: ACCESS_SECRET_KEY
 wal:
 compression: gzip

Or use Pulumi:

new k8s.apiextensions.CustomResource("cluster-example-with-backup", {
 apiVersion: "postgresql.cnpg.io/v1",
 kind: "Cluster",
 metadata: {
 name: "cluster-example-with-backup",
 },
 spec: {
 instances: 3,
 primaryUpdateStrategy: "unsupervised",

 // Persistent storage configuration
 storage: {
 storageClass: "standard",
 size: "1Gi",
 },

 // Backup properties
 backup: {
 barmanObjectStore: {
 destinationPath: "s3://backups/",
 endpointURL: "http://minio:9000",
 s3Credentials: {
 accessKeyId: {
 name: "minio",
 key: "ACCESS_KEY_ID",
 },
 secretAccessKey: {
 name: "minio",
 key: "ACCESS_SECRET_KEY",
 },
 },
 wal: {
 compression: "gzip",
 },
 },
 },
 },
});

from pulumi_kubernetes import apiextensions

postgresql_cluster = apiextensions.CustomResource(
 "cluster-example-with-backup",
 api_version="postgresql.cnpg.io/v1",
 kind="Cluster",
 metadata=meta.ObjectMetaArgs(
 name="cluster-example-with-backup",
 ),
 spec={
 "instances": 3,
 "primaryUpdateStrategy": "unsupervised",
 # Persistent storage configuration
 "storage": {
 "storageClass": "standard",
 "size": "1Gi",
 },
 # Backup properties
 "backup": {
 "barmanObjectStore": {
 "destinationPath": "s3://backups/",
 "endpointURL": "http://minio:9000",
 "s3Credentials": {
 "accessKeyId": {
 "name": "minio",
 "key": "ACCESS_KEY_ID",
 },
 "secretAccessKey": {
 "name": "minio",
 "key": "ACCESS_SECRET_KEY",
 },
 },
 "wal": {
 "compression": "gzip",
 },
 },
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "cluster-example-with-backup", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("postgresql.cnpg.io/v1"),
 Kind: pulumi.String("Cluster"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("cluster-example-with-backup"),
 },
 OtherFields: k8s.UntypedArgs{
 "instances": pulumi.Int(3),
 "primaryUpdateStrategy": pulumi.String("unsupervised"),
 "storage": pulumi.Map{
 "storageClass": pulumi.String("standard"),
 "size": pulumi.String("1Gi"),
 },
 "backup": pulumi.Map{
 "barmanObjectStore": pulumi.Map{
 "destinationPath": pulumi.String("s3://backups/"),
 "endpointURL": pulumi.String("http://minio:9000"),
 "s3Credentials": pulumi.Map{
 "accessKeyId": pulumi.Map{
 "name": pulumi.String("minio"),
 "key": pulumi.String("ACCESS_KEY_ID"),
 },
 "secretAccessKey": pulumi.Map{
 "name": pulumi.String("minio"),
 "key": pulumi.String("ACCESS_SECRET_KEY"),
 },
 },
 "wal": pulumi.Map{
 "compression": pulumi.String("gzip"),
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class PostGreSQLClusterArgs : CustomResourceArgs
{
 [Input("postgresql.cnpg.io/v1")]
 public Input<string> ApiVersion { get; set; }
 [Input("Cluster")]
 public Input<string> Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Others { get; set; }
 [Input("spec")]
 public InputMap<object> Spec { get; set; }

 public PostGreSQLClusterArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}
class MyStack : Stack
{
 public MyStack()
 {
 var postgresqlCluster = new Pulumi.Kubernetes.ApiExtensions.CustomResource("cluster-example-with-backup", new PostGreSQLClusterArgs("postgresql.cnpg.io/v1", "Cluster")
 {
 ApiVersion = "postgresql.cnpg.io/v1",
 Kind = "Cluster",
 Metadata = new ObjectMetaArgs
 {
 Name = "cluster-example-with-backup",
 },
 Spec = new InputMap<object>
 {
 { "instances", 3 },
 { "primaryUpdateStrategy", "unsupervised" },
 { "storage", new InputMap<object>
 {
 { "storageClass", "standard" },
 { "size", "1Gi" },
 }
 },
 { "backup", new InputMap<object>
 {
 { "barmanObjectStore", new InputMap<object>
 {
 { "destinationPath", "s3://backups/" },
 { "endpointURL", "http://minio:9000" },
 { "s3Credentials", new InputMap<object>
 {
 { "accessKeyId", new InputMap<object>
 {
 { "name", "minio" },
 { "key", "ACCESS_KEY_ID" },
 }
 },
 { "secretAccessKey", new InputMap<object>
 {
 { "name", "minio" },
 { "key", "ACCESS_SECRET_KEY" },
 }
 },
 }
 },
 { "wal", new InputMap<object>
 {
 { "compression", "gzip" },
 }
 },
 }
 },
 }
 },
 }
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

CloudNativePG Operator is a great way to deploy and manage PostgreSQL databases on Kubernetes. It simplifies a wide range of tasks, including backups, high availability, and scaling, making it a must-have for PostgreSQL users.

You might also like:

Flux Operator

Flux Operator Credit: fluxcd.control-plane.io/operator

The Flux Operator is a full autopilot solution that provides an alternative to the traditional Flux Bootstrap procedure, removing the operational burden of managing Flux across multiple clusters by automating the installation, configuration, and upgrade of Flux fully.

Some of the key features of Flux Operator are:

GitOps Deployment - Automate the deployment of applications and services using GitOps.
Lifecycle Management - Manage the full lifecycle of Flux, including upgrades and rollbacks.
Deep Insights - Provide deep insights into the state of Flux and the applications deployed with Flux.

Installation

Use the Helm chart to deploy the Flux Operator:

helm upgrade -i flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator --namespace flux-system --create-namespace

Or use Pulumi to deploy the Flux Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const namespace = new k8s.core.v1.Namespace("flux-system", {
 metadata: {
 name: "flux-system",
 },
});

new k8s.helm.v3.Release("flux-operator", {
 chart: "oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 namespace: namespace.metadata.name,
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

flux_namespace = k8s.core.v1.Namespace(
 "flux-system",
 metadata=meta.ObjectMetaArgs(
 name="flux-system",
 ),
)
flux_operator = k8shelm.Release(
 "flux-operator",
 chart="oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 namespace=flux_namespace.metadata["name"],
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 namespace, err := corev1.NewNamespace(ctx, "flux-system", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("flux-system"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "flux-operator", &helmv3.ReleaseArgs{
 Chart: pulumi.String("oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator"),
 Namespace: namespace.Metadata.Name().Elem(),
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

 var namespaceFluxSystem = new Namespace("flux-system", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "flux-system",
 },
 });

 var fluxOperator = new Release("flux-operator", new ReleaseArgs
 {
 Chart = "oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 Namespace = namespaceFluxSystem.Metadata.Apply(metadata => metadata.Name),
 CreateNamespace = true,
 });
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is an example of a FluxInstance Custom Resource that defines a Flux instance:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
 name: flux
 namespace: flux-system
 annotations:
 fluxcd.controlplane.io/reconcileEvery: "1h"
 fluxcd.controlplane.io/reconcileTimeout: "5m"
spec:
 distribution:
 version: "2.x"
 registry: "ghcr.io/fluxcd"
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"
 components:
 - source-controller
 - kustomize-controller
 - helm-controller
 - notification-controller
 - image-reflector-controller
 - image-automation-controller
 cluster:
 type: kubernetes
 multitenant: false
 networkPolicy: true
 domain: "cluster.local"
 kustomize:
 patches:
 - target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s

Or use Pulumi:

new k8s.apiextensions.CustomResource("flux", {
 apiVersion: "fluxcd.controlplane.io/v1",
 kind: "FluxInstance",
 metadata: {
 name: "flux",
 namespace: "flux-system",
 annotations: {
 "fluxcd.controlplane.io/reconcileEvery": "1h",
 "fluxcd.controlplane.io/reconcileTimeout": "5m",
 },
 },
 spec: {
 distribution: {
 version: "2.x",
 registry: "ghcr.io/fluxcd",
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests",
 },
 components: ["source-controller", "kustomize-controller", "helm-controller", "notification-controller", "image-reflector-controller", "image-automation-controller"],
 cluster: {
 type: "kubernetes",
 multitenant: false,
 networkPolicy: true,
 domain: "cluster.local",
 },
 kustomize: {
 patches: [
 {
 target: {
 kind: "Deployment",
 name: "(kustomize-controller|helm-controller)",
 },
 patch: `
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s
 `,
 },
 ],
 },
 },
});

from pulumi_kubernetes import apiextensions

flux_instance = k8s.apiextensions.CustomResource(
 "flux",
 api_version="fluxcd.controlplane.io/v1",
 kind="FluxInstance",
 metadata={
 "name": "flux",
 "namespace": flux_namespace.metadata["name"],
 "annotations": {
 "fluxcd.controlplane.io/reconcileEvery": "1h",
 "fluxcd.controlplane.io/reconcileTimeout": "5m",
 },
 },
 spec={
 "distribution": {
 "version": "2.x",
 "registry": "ghcr.io/fluxcd",
 "artifact": "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests",
 },
 "components": [
 "source-controller",
 "kustomize-controller",
 "helm-controller",
 "notification-controller",
 "image-reflector-controller",
 "image-automation-controller",
 ],
 "cluster": {
 "type": "kubernetes",
 "multitenant": False,
 "networkPolicy": True,
 "domain": "cluster.local",
 },
 "kustomize": {
 "patches": [
 {
 "target": {
 "kind": "Deployment",
 "name": "(kustomize-controller|helm-controller)",
 },
 "patch": [
 {
 "op": "add",
 "path": "/spec/template/spec/containers/0/args/-",
 "value": "--concurrent=10",
 },
 {
 "op": "add",
 "path": "/spec/template/spec/containers/0/args/-",
 "value": "--requeue-dependency=5s",
 },
 ],
 }
 ]
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "flux", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("fluxcd.controlplane.io/v1"),
 Kind: pulumi.String("FluxInstance"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("flux"),
 Namespace: pulumi.String("flux-system"),
 Annotations: pulumi.StringMap{
 "fluxcd.controlplane.io/reconcileEvery": pulumi.String("1h"),
 "fluxcd.controlplane.io/reconcileTimeout": pulumi.String("5m"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "distribution": pulumi.Map{
 "version": pulumi.String("2.x"),
 "registry": pulumi.String("ghcr.io/fluxcd"),
 "artifact": pulumi.String("oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"),
 },
 "components": pulumi.StringArray{
 pulumi.String("source-controller"),
 pulumi.String("kustomize-controller"),
 pulumi.String("helm-controller"),
 pulumi.String("notification-controller"),
 pulumi.String("image-reflector-controller"),
 pulumi.String("image-automation-controller"),
 },
 "cluster": pulumi.Map{
 "type": pulumi.String("kubernetes"),
 "multitenant": pulumi.Bool(false),
 "networkPolicy": pulumi.Bool(true),
 "domain": pulumi.String("cluster.local"),
 },
 "kustomize": pulumi.Map{
 "patches": pulumi.Array{
 pulumi.Map{
 "target": pulumi.Map{
 "kind": pulumi.String("Deployment"),
 "name": pulumi.String("(kustomize-controller|helm-controller)"),
 },
 "patch": pulumi.Array{
 pulumi.Map{
 "op": pulumi.String("add"),
 "path": pulumi.String("/spec/template/spec/containers/0/args/-"),
 "value": pulumi.String("--concurrent=10"),
 },
 pulumi.Map{
 "op": pulumi.String("add"),
 "path": pulumi.String("/spec/template/spec/containers/0/args/-"),
 "value": pulumi.String("--requeue-dependency=5s"),
 },
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class FluxInstanceArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public FluxInstanceArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}
 var flux = new Pulumi.Kubernetes.ApiExtensions.CustomResource("flux", new FluxInstanceArgs("fluxcd.controlplane.io/v1", "FluxInstance")
 {
 ApiVersion = "fluxcd.controlplane.io/v1",
 Kind = "FluxInstance",
 Metadata = new ObjectMetaArgs
 {
 Name = "flux",
 Annotations =
 {
 { "fluxcd.controlplane.io/reconcileEvery", "1h" },
 { "fluxcd.controlplane.io/reconcileTimeout", "5m" },
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "distribution", new Dictionary<string, object>
 {
 { "version", "2.x" },
 { "registry", "ghcr.io/fluxcd" },
 { "artifact", "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests" }
 }
 },
 { "components", new[]
 {
 "source-controller",
 "kustomize-controller",
 "helm-controller",
 "notification-controller",
 "image-reflector-controller",
 "image-automation-controller"
 }
 },
 { "cluster", new Dictionary<string, object>
 {
 { "type", "kubernetes" },
 { "multitenant", false },
 { "networkPolicy", true },
 { "domain", "cluster.local" }
 }
 },
 { "kustomize", new Dictionary<string, object>
 {
 { "patches", new[]
 {
 new Dictionary<string, object>
 {
 { "target", new Dictionary<string, object>
 {
 { "kind", "Deployment" },
 { "name", "(kustomize-controller|helm-controller)" }
 }
 },
 { "patch", new[]
 {
 new Dictionary<string, object>
 {
 { "op", "add" },
 { "path", "/spec/template/spec/containers/0/args/-" },
 { "value", "--concurrent=10" }
 },
 new Dictionary<string, object>
 {
 { "op", "add" },
 { "path", "/spec/template/spec/containers/0/args/-" },
 { "value", "--requeue-dependency=5s" }
 }
 }
 }
 }
 }
 }
 }
 }
 },
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

The Flux Operator is another excellent example of an advanced Kubernetes operator, providing full lifecycle management with numerous automation features.

Strimzi Operator

Strimzi Operator Credit: strimzi.io

Last but not least, the Strimzi Operator is a full lifecycle management operator for Apache Kafka on Kubernetes. As event-driven architectures are becoming increasingly popular, Apache Kafka has become the de-facto standard for building scalable and reliable event streaming platforms. The Strimzi Operator makes it easier to deploy, manage, and scale Apache Kafka clusters on Kubernetes.

Some of the key features of Strimzi Operator are:

Cluster Management - Deploy and manage Apache Kafka clusters.
Mirror Clusters - Deploy and manage mirror clusters for disaster recovery.
Streaming Capabilities - With Kafka Connect you can stream data between Kafka and other systems.

Installation

Use the Helm chart to deploy the Strimzi Operator:

helm repo add strimzi https://strimzi.io/charts/
helm repo update
helm install strimzi strimzi/strimzi-kafka-operator -n kafka --create-namespace

Or use Pulumi to deploy the Strimzi Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const namespaceKafka = new k8s.core.v1.Namespace("kafka", {
 metadata: {
 name: "kafka",
 },
});

new k8s.helm.v3.Release("strimzi-kafka-operator", {
 chart: "strimzi-kafka-operator",
 namespace: namespaceKafka.metadata.name,
 repositoryOpts: {
 repo: "https://strimzi.io/charts/",
 },
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

kafka_namespace = k8s.core.v1.Namespace(
 "kafka",
 metadata=meta.ObjectMetaArgs(
 name="kafka",
 ),
)

strimzi_kafka_operator = k8shelm.Release(
 "strimzi-kafka-operator",
 chart="strimzi-kafka-operator",
 namespace=kafka_namespace.metadata["name"],
 repository_opts={
 "repo": "https://strimzi.io/charts/",
 },
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 namespaceKafka, err := corev1.NewNamespace(ctx, "kafka", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("kafka"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "strimzi-kafka-operator", &helmv3.ReleaseArgs{
 Chart: pulumi.String("strimzi-kafka-operator"),
 Namespace: namespaceKafka.Metadata.Name(),
 RepositoryOpts: helmv3.RepositoryOptsArgs{
 Repo: pulumi.String("https://strimzi.io/charts/"),
 },
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;
class MyStack : Stack
{
 public MyStack()
 {
 var namespaceKafka = new Namespace("kafka", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "kafka",
 }
 });

 var strimziKafkaOperator = new Release("strimzi-kafka-operator", new ReleaseArgs
 {
 Chart = "strimzi-kafka-operator",
 Namespace = namespaceKafka.Metadata.Apply(ns => ns.Name),
 RepositoryOpts = new RepositoryOptsArgs
 {
 Repo = "https://strimzi.io/charts/",
 },
 CreateNamespace = true,
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is an example of a Kafka Custom Resource that defines a Kafka cluster:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
 name: example-kafka-node-pool
 labels:
 strimzi.io/cluster: example-kafka-cluster
spec:
 replicas: 1
 roles:
 - controller
 - broker
 storage:
 type: ephemeral
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
 name: example-kafka-cluster
 annotations:
 strimzi.io/kraft: enabled
 strimzi.io/node-pools: enabled
spec:
 kafka:
 replicas: 1
 version: 3.8.0
 storage:
 type: ephemeral
 metadataVersion: 3.8-IV0
 listeners:
 - name: plain
 port: 9092
 type: internal
 tls: false
 - name: tls
 port: 9093
 type: internal
 tls: true
 config:
 offsets.topic.replication.factor: 1
 transaction.state.log.replication.factor: 1
 transaction.state.log.min.isr: 1
 default.replication.factor: 1
 min.insync.replicas: 1
 entityOperator:
 topicOperator: {}
 userOperator: {}

This example defines a Kafka cluster with one replica and two listeners, one for plain and one for TLS and uses ephemeral storage.

To create a topic, you can use the following KafkaTopic Custom Resource:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
 name: example-kafka-cluster
 labels:
 strimzi.io/cluster:
spec:
 partitions: 5
 replicas: 1
 config:
 retention.ms: 7200000
 segment.bytes: 1073741824

Or use Pulumi:

new k8s.apiextensions.CustomResource("example-kafka-node-pool", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "KafkaNodePool",
 metadata: {
 name: "example-kafka-node-pool",
 labels: {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 namespace: "kafka",
 },
 spec: {
 replicas: 1,
 roles: ["controller", "broker"],
 storage: {
 type: "ephemeral",
 },
 },
});

new k8s.apiextensions.CustomResource("example-kafka-cluster", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "Kafka",
 metadata: {
 name: "example-kafka-cluster",
 annotations: {
 "strimzi.io/kraft": "enabled",
 "strimzi.io/node-pools": "enabled",
 },
 namespace: "kafka",
 },
 spec: {
 kafka: {
 replicas: 1,
 version: "3.8.0",
 storage: {
 type: "ephemeral",
 },
 metadataVersion: "3.8-IV0",
 listeners: [
 {
 name: "plain",
 port: 9092,
 type: "internal",
 tls: false,
 },
 {
 name: "tls",
 port: 9093,
 type: "internal",
 tls: true,
 },
 ],
 config: {
 "offsets.topic.replication.factor": 1,
 "transaction.state.log.replication.factor": 1,
 "transaction.state.log.min.isr": 1,
 "default.replication.factor": 1,
 "min.insync.replicas": 1,
 },
 },
 entityOperator: {
 topicOperator: {},
 userOperator: {},
 },
 },
});

new k8s.apiextensions.CustomResource("example-kafka-topic", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "KafkaTopic",
 metadata: {
 name: "example-kafka-topic",
 labels: {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 namespace: "kafka",
 },
 spec: {
 partitions: 5,
 replicas: 1,
 config: {
 "retention.ms": 7200000,
 "segment.bytes": 1073741824,
 },
 },
});

from pulumi_kubernetes import apiextensions

kafka_node_pool = k8s.apiextensions.CustomResource(
 "example-kafka-node-pool",
 api_version="kafka.strimzi.io/v1beta2",
 kind="KafkaNodePool",
 metadata={
 "name": "example-kafka-node-pool",
 "labels": {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "replicas": 1,
 "roles": ["controller", "broker"],
 "storage": {
 "type": "ephemeral",
 },
 },
)

kafka_cluster = k8s.apiextensions.CustomResource(
 "example-kafka-cluster",
 api_version="kafka.strimzi.io/v1beta2",
 kind="Kafka",
 metadata={
 "name": "example-kafka-cluster",
 "annotations": {
 "strimzi.io/kraft": "enabled",
 "strimzi.io/node-pools": "enabled",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "kafka": {
 "replicas": 1,
 "version": "3.8.0",
 "storage": {
 "type": "ephemeral",
 },
 "metadataVersion": "3.8-IV0",
 "listeners": [
 {
 "name": "plain",
 "port": 9092,
 "type": "internal",
 "tls": False,
 },
 {
 "name": "tls",
 "port": 9093,
 "type": "internal",
 "tls": True,
 },
 ],
 "config": {
 "offsets.topic.replication.factor": 1,
 "transaction.state.log.replication.factor": 1,
 "transaction.state.log.min.isr": 1,
 "default.replication.factor": 1,
 "min.insync.replicas": 1,
 },
 },
 "entityOperator": {
 "topicOperator": {},
 "userOperator": {},
 },
 },
)

kafka_topic = k8s.apiextensions.CustomResource(
 "example-kafka-topic",
 api_version="kafka.strimzi.io/v1beta2",
 kind="KafkaTopic",
 metadata={
 "name": "example-kafka-topic",
 "labels": {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "partitions": 5,
 "replicas": 1,
 "config": {
 "retention.ms": 7200000,
 "segment.bytes": 1073741824,
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-node-pool", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("KafkaNodePool"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-node-pool"),
 Namespace: pulumi.String("kafka"),
 Labels: pulumi.StringMap{
 "strimzi.io/cluster": pulumi.String("example-kafka-cluster"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "replicas": pulumi.Int(1),
 "roles": pulumi.StringArray{
 pulumi.String("controller"),
 pulumi.String("broker"),
 },
 "storage": pulumi.Map{
 "type": pulumi.String("ephemeral"),
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-cluster", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("Kafka"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-cluster"),
 Namespace: pulumi.String("kafka"),
 Annotations: pulumi.StringMap{
 "strimzi.io/kraft": pulumi.String("enabled"),
 "strimzi.io/node-pools": pulumi.String("enabled"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "kafka": pulumi.Map{
 "replicas": pulumi.Int(1),
 "version": pulumi.String("3.8.0"),
 "storage": pulumi.Map{
 "type": pulumi.String("ephemeral"),
 },
 "metadataVersion": pulumi.String("3.8-IV0"),
 "listeners": pulumi.Array{
 pulumi.Map{
 "name": pulumi.String("plain"),
 "port": pulumi.Int(9092),
 "type": pulumi.String("internal"),
 "tls": pulumi.Bool(false),
 },
 pulumi.Map{
 "name": pulumi.String("tls"),
 "port": pulumi.Int(9093),
 "type": pulumi.String("internal"),
 "tls": pulumi.Bool(true),
 },
 },
 "config": pulumi.Map{
 "offsets.topic.replication.factor": pulumi.Int(1),
 "transaction.state.log.replication.factor": pulumi.Int(1),
 "transaction.state.log.min.isr": pulumi.Int(1),
 "default.replication.factor": pulumi.Int(1),
 "min.insync.replicas": pulumi.Int(1),
 },
 },
 "entityOperator": pulumi.Map{
 "topicOperator": pulumi.Map{},
 "userOperator": pulumi.Map{},
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-topic", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("KafkaTopic"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-topic"),
 Namespace: pulumi.String("kafka"),
 Labels: pulumi.StringMap{
 "strimzi.io/cluster": pulumi.String("example-kafka-cluster"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "partitions": pulumi.Int(5),
 "replicas": pulumi.Int(1),
 "config": pulumi.Map{
 "retention.ms": pulumi.Int(7200000),
 "segment.bytes": pulumi.Int(1073741824),
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class KafkaNodePoolArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaNodePoolArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class KafkaClusterArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaClusterArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class KafkaTopicArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaTopicArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class MyStack : Stack
{
 public MyStack()
 {
 var kafkaNodePool = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaNodePool", new KafkaNodePoolArgs("kafka.strimzi.io/v1beta2", "KafkaNodePool")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "KafkaNodePool",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-node-pool",
 Labels =
 {
 { "strimzi.io/cluster", "example-kafka-cluster" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "replicas", 1 },
 { "roles", new List<string> { "controller", "broker" } },
 { "storage", new Dictionary<string, object> { { "type", "ephemeral" } } }
 }
 });

 var kafkaCluster = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaCluster", new KafkaClusterArgs("kafka.strimzi.io/v1beta2", "Kafka")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "Kafka",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-cluster",
 Annotations =
 {
 { "strimzi.io/kraft", "enabled" },
 { "strimzi.io/node-pools", "enabled" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "kafka", new Dictionary<string, object> {
 { "replicas", 1 },
 { "version", "3.8.0" },
 { "storage", new Dictionary<string, object> { { "type", "ephemeral" } } },
 { "metadataVersion", "3.8-IV0" },
 { "listeners", new List<Dictionary<string, object>> {
 new Dictionary<string, object> {
 { "name", "plain" },
 { "port", 9092 },
 { "type", "internal" },
 { "tls", false }
 },
 new Dictionary<string, object> {
 { "name", "tls" },
 { "port", 9093 },
 { "type", "internal" },
 { "tls", true }
 }
 }},
 { "config", new Dictionary<string, object> {
 { "offsets.topic.replication.factor", 1 },
 { "transaction.state.log.replication.factor", 1 },
 { "transaction.state.log.min.isr", 1 },
 { "default.replication.factor", 1 },
 { "min.insync.replicas", 1 }
 }}
 }},
 { "entityOperator", new Dictionary<string, object> {
 { "topicOperator", new Dictionary<string, object>() },
 { "userOperator", new Dictionary<string, object>() }
 }}
 }
 });

 var kafkaTopic = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaTopic", new KafkaTopicArgs("kafka.strimzi.io/v1beta2", "KafkaTopic")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "KafkaTopic",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-topic",
 Labels =
 {
 { "strimzi.io/cluster", "example-kafka-cluster" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "partitions", 5 },
 { "replicas", 1 },
 { "config", new Dictionary<string, object> {
 { "retention.ms", 7200000 },
 { "segment.bytes", 1073741824 }
 }}
 }
 });
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

The Strimzi Operator demonstrates the power of Kubernetes Operators by providing full lifecycle management for Apache Kafka on Kubernetes. It takes care of the heavy lifting of deploying and managing Kafka clusters, making the Strimzi Operator a valuable tool that should be part of every Kubernetes-powered platform, as it covers a wide range of use cases.

Conclusion

In this blog post, I highlighted the importance of Kubernetes Operators for platform engineers and how they can harness the advanced automation capabilities of operators to simplify deploying, managing, and scaling applications and services on Kubernetes.

I provided some examples of advanced Kubernetes operators that I recommend, but if you want to try using Kubernetes Operators in your Kubernetes-powered platform, here are some great guides to get you started with Pulumi and Kubernetes:

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

Pulumi Patterns and Practices Platform (P3): Some Assembly Required

Troy Howard — Mon, 11 Nov 2024 00:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Setting up an internal developer platform (IDP) can be a daunting task. There are a lot of tools out there that do some of the work for you, but none of them do all of it. Pulumi P3 is no different. Pulumi Patterns & Practices Platform (P3) is a reference architecture that we will be describing, and providing code for, through this series of articles.

We will never try to sell you on the idea that you can simply download a package, click next a few times, and achieve transformative success. That’s because any effective IDP will require some customization and integration to work within your environment.

Tools that purport to have it all figured out have only figured out how to manipulate you into a false narrative they have constructed in a vacuum, where all your organizational needs fit neatly into a few boxes they’ve decided on for you. And also charge you for. In addition to everything else you’re being charged for. Ultimately you’ll still need to build a lot yourself and these products rarely give guidance on how to do that.

When we first started hearing about our customers using Pulumi as an internal developer platform (IDP), we were frankly surprised, as our goals were primarily for Pulumi to be the best developer experience in infrastructure. But it makes sense. All the parts are there, some assembly required. Our goal with Pulumi Patterns and Practices Platform (P3) is to help with that assembly process.

Starting with our previous blog post on the topic, and continuing here, we are examining this use case, and attempting to formalize that into a collection of reusable components and some guidance on how you can skip the marketing pitches and pricing charts, and get straight to the hard work of building your own highly customized internal developer platform with Pulumi at its core.

Pulumi P3: Bill of Materials

Previously we identified the essential qualities of an effective IDP. Those were consistency, reproducibility, visibility, security and compliance, auditability, developer experience. In the last half of the post we discussed which parts of Pulumi could be used to meet those needs. That looks like:

Consistency: component resources, organization templates, drift detection
Reproducibility: stacks, deployments, versioned data
Visibility: Pulumi Insights, Pulumi Copilot
Security and Compliance: RBAC, GitHub Teams, SAML-SSO, Pulumi ESC, Pulumi Crossguard
Auditability: audit logging
Developer Experience: Python/Go/JavaScript/C#, popular IDE support, command-line tools, deeply hackable

That’s all great, and much of that is already built-into Pulumi without the need for you to do anything at all. So, what parts do you actually need to set up and configure? Here’s the bill of materials (BOM) to set up your own instance of Pulumi P3:

Bill of Materials:

Authentication and Identity Management:
- A GitHub organization that matches your Pulumi Cloud organization
- GitHub Teams users and roles that match your organizational structure and security needs
Secrets, Configuration, and Policy:
- Pulumi ESC environments to manage secrets across clouds
- Pulumi Crossguard policy packs that capture your company policies
Developer Experience:
- A set of reusable multi-language components for cross-cutting concerns/common services
- A set of organization templates that match your common use cases

Let’s go through each of those and briefly discuss what it looks like to set that up.

Authentication and identity management

We highly recommend using GitHub for code management. So much so that we have deeply integrated GitHub into Pulumi Cloud across a number of features. While we support alternatives such as GitLab, this will be the easiest and more feature-rich way to configure your platform.

In Pulumi Cloud, you have the ability to create organizations. A Pulumi Cloud organization can help you manage teams, roles, stacks, settings, and provide a dashboard across the entire organization. Pulumi Cloud also allows you to use a variety of identity providers to log in, including GitHub.

For simplicity’s sake, we suggest that you start with your GitHub organization. Create the GitHub organization, set up teams, and add members to those teams, assigning either admin or user roles to each member.

Next, in Pulumi Cloud, create an organization with exactly the same name as your GitHub organization, and choose GitHub as your identity provider. When a Pulumi organization is backed by a GitHub organization, then only members of that GitHub organization may be added to the Pulumi organization. Similarly, as soon as someone loses access to the GitHub organization, they will no longer have access to the Pulumi organization. You will also be able to import your GitHub teams directly into Pulumi Cloud. Then assign your users to the same roles in Pulumi Cloud teams as they have in the associated GitHub teams.

Figure: Mapping GitHub orgs, teams, and roles to Pulumi

Finally, you can map teams to stacks to grant access at specific permission levels. If you’re not familiar with Pulumi Stacks, a stack is a materialized instance of a specific set of cloud resources, as defined in a Pulumi program.

Pulumi ESC: Managing credentials, configuration, and other secrets

In order to deploy a stack you will need secrets such as cloud credentials and other configuration values that are provided to the deployment engine. Pulumi ESC is a secure system for managing secrets. They are organized by environments.

An example set of environments might look something like this:

Example: AWS login/credentials

# aws-creds ESC environment
values:
 creds:
 fn::open::aws-login:
 oidc:
 roleArn: arn:aws:iam::123456789012:role/pulumi-environments-oidc
 sessionName: pulumi-environments-session
 duration: 1h
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.creds.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.creds.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.creds.sessionToken}

Example: Default production environment to use us-east-1 region

# aws-production ESC environment
imports:
 - aws-creds
values:
 aws:
 region: us-east-1
 pulumiConfig:
 aws:region: ${aws.region}

Example: Default staging environment to use us-west-2 region

# aws-staging ESC environment
imports:
 - aws-creds
values:
 aws:
 region: us-west-2
 pulumiConfig:
 aws:region: ${aws.region}

Here we define three environments for AWS:

aws-creds: sets up login via OpenID Connect (OIDC) and provides standard environment variables containing AWS credentials to the Pulumi program
aws-production: imports everything from aws-creds then sets the region to us-east-1
aws-staging: does the same, but sets the region to us-west-2.

Using that in a Pulumi program is as simple as adding the following settings to your stack config:

# Pulumi.staging.yaml
environment:
 - aws-staging

In this manner, you can configure separate environments for staging and production, with a complex set of configuration values and secrets, using different environments for each one. From the developer’s perspective they would only need to change aws-staging to aws-production when they go to deploy their stack.

Another strong benefit of this approach is that all secrets will be encrypted both in-flight and at-rest. Pulumi waits until the last moment to decrypt secrets at runtime. By default, uses automatic, per-stack encryption keys provided by Pulumi Cloud, but you could use a provider of your own choosing instead.

Pulumi Crossguard: Policy-as-Code

Pulumi Crossguard allows you to check and enforce policies on your deployments. Policies are rules, written in code, that run during deployments to check that the resources are conforming to the necessary criteria. You can use off-the-shelf policies like AWSGuard and Pulumi Compliance-Ready Policies or write your own.

Either way you end up with a policy pack that you can apply to your entire Pulumi organization via Pulumi Cloud.

Here’s an example policy that checks for the presence of a tag user:Stack on a S3 bucket:

from pulumi_policy import (
 EnforcementLevel,
 PolicyPack,
 ResourceValidationPolicy,
)

def s3_check_required_tags(args, report_violation):
 if args.resource_type == "aws:s3/bucket:Bucket":
 if ("tags" not in args.props or
 "user:Stack" not in args.props["tags"]):
 report_violation("S3 Bucket is missing required user:Stack tag.")

PolicyPack(
 name="bucket-tags",
 enforcement_level=EnforcementLevel.MANDATORY,
 policies=[
 ResourceValidationPolicy(
 name="s3-tags",
 description="Ensure required tags are present on S3 buckets.",
 validate=s3_check_required_tags,
 ),
 ],
)

If the tag isn’t on the resource, it blocks the deployment with an error message. The error message would look something like this:

Policies:
❌ bucket-tags@v0.0.1
- [mandatory] s3-tags (aws:s3/bucket:Bucket: my-bucket)
Ensure required tags are present on S3 buckets.
S3 Bucket is missing required user:Stack tag.

This allows you to implement company-specific policies that can be as simple or complex as you need them to be.

To apply this across your entire organization, you can publish this policy pack to Pulumi Cloud, with the following commands:

$ pulumi policy publish myorg
$ pulumi policy enable myorg/my-policy-pack latest

Some other great features of Crossguard are the ability to version policies, define multiple policy groups, and create remediation policies that automatically fix policy violations when possible. We will cover these topics in a future post where we go deeper on how to use policies effectively.

Multi-Language Components (MLC)

In Pulumi, a component resource is something that your developers can import in their Pulumi program, instantiate and modify. These are made available via a provider, which is in turn, made available to Pulumi via a provider package. There are many of these already available in the Pulumi Registry. However, in a custom internal developer platform you can define your own components, and bake appropriate settings/configuration directly into the underlying code.

A multi-language component (MLC) is even more useful. You can author your component in your language of choice and then generate a SDK that surfaces that component into all of the languages that Pulumi supports. For example, your platform team might be comfortable writing in Python, but the developers that write your microservices might use Go, and the developers who write the front-end apps might use Node.js. Both teams might need to deploy apps and infrastructure into your Kubernetes cluster. With multi-language components you can write a component in Python that abstracts away all the details of your custom Kubernetes cluster, and make that available to both teams, in both Go, Node.js, and any other language that Pulumi supports.

To build a MLC, you’ll follow these basic steps to create the component, provider, provider package, and generate the multi-language SDK:

Fork one of the component provider boilerplate repos for Python, TypeScript, or Go.
Update the package and code-generator configuration files, which name your component and package, define the inputs and outputs, and declare the dependencies.
Implement the component in your preferred language.
Generate an SDK for the other languages.
Deploy the package.

Here’s a quick example of creating a custom S3 Bucket component in Python, that complies with the tagging policy we built earlier:

from pulumi_aws import s3
import pulumi

class TaggedBucket(pulumi.ComponentResource):

 def __init__(self, name, opts = None):
 super().__init__('mycorp:index:TaggedBucket', name, None, opts)

 # Create a bucket and add a custom tag to it.
 bucket = s3.Bucket(
 f'{name}-bucket',
 tags={
 'user:Stack': pulumi.get_stack()
 },
 opts=ResourceOptions(parent=self))

 self.register_outputs({
 'bucket': bucket,
 'websiteUrl': bucket.website_endpoint,
 'bucketDnsName': bucket.bucketDomainName
 })

This shows the component implementation in isolation from the provider/packaging/SDK boilerplate. In this code sample, we’re creating a component called TaggedBucket that creates a S3 bucket, and adds a tag user:Stack with the current stack name as its value. A developer could now use this in a TypeScript Pulumi program as such, and this resource would automatically have the tags added to it.

import * as mycorp from "mycorp/mycorp-components";

const taggedBucket = new mycorp.TaggedBucket("example");

export const bucket = taggedBucket.bucket;
export const url = taggedBucket.websiteUrl;
export const dnsName = taggedBucket.bucketDnsName;

If you want to see how to create MLCs in more detail, check out this video that walks you through the entire process, and this repo for the code shown in the video. In a follow-up post in this series, we will build some reference MLCs that do things like implement a time-to-live (TTL) for stacks in your staging environment, automate drift detection, and automatically instrument your developer’s deployments with observability tools integrated by default.

Organization templates and the New Project Wizard

The final piece that ties all this together are organization templates. You may have used some of our built-in templates when you learned how to use Pulumi. These are great for basic use cases, but the real magic happens when you bring together your custom components and custom security environments to create personalized templates which represent the internal use cases for your organization.

Pulumi’s New Project Wizard reads these templates and provides an in-browser way to create a new project and deploy it. Running one of these templates will commit and push code to GitHub, and trigger an initial deployment – all in a few clicks and without leaving the browser.

Each template needs the following parts:

A Pulumi.yaml describing the template and its configuration values
A GitHub repo (public or private) containing the code for the templated Pulumi program

Here’s an example of a simple template using the components and environments we described above.

# Pulumi.yaml
name: ${PROJECT}
description: ${DESCRIPTION}
runtime: python
template:
 description: A Python Pulumi program that creates a tagged bucket.

"""__main__.py: A minimal Pulumi program"""

import pulumi
import mycorp

# Create an AWS resource (S3 Bucket)
tagged_bucket = mycorp.TaggedBucket('my-bucket')

# Export the name of the bucket
pulumi.export('bucket_name', tagged_bucket.bucket)

# Pulumi.production.yaml
environment:
 - aws-production

# Pulumi.staging.yaml
environment:
 - aws-staging

The Pulumi.yaml sets up the template and will populate the name and description from the settings provided during the template dialogue. The custom TaggedBucket component will create an S3 bucket, which will be tagged with user:Stack set to the name of the stack. Default stack configurations are provided for the staging and production environments which map to our two ESC environments, aws-production and aws-staging.

How it all works together

With all of that in place, from the developer’s perspective, all they need to do is create a new project from the template, answering three questions: the stack name, the name of the project, and an optional description.

If the developer names the stack staging it will automatically apply the aws-staging ESC environment, which will include the AWS credentials and set the region to us-west-2. However, if the developer names the stack production it will get the aws-production ESC environment setting it to use the us-east-1 region. The name of the stack will be stored in a tag on the resource.

Pulumi Crossguard will apply the bucket-tags policy check to see if the resource has the required user:Stack tag set and will allow the deployment to proceed only if it has that tag. If a developer created a standard S3 Bucket instead of using our internal TaggedBucket component, and failed to add the required tag, they will get an error message from our custom policy when they try to deploy.

Later, we can create additional automation that might do something like delete anything tagged staging after two weeks, or run drift detection on anything tagged production. We will be exploring these concepts in more detail in later posts.

More to Come

While setting up the Pulumi Patterns and Practices Platform (P3) reference architecture is not a simple click-to-deploy, hopefully this high-level tour of the various parts you need to assemble shows that really, it is only a matter of creating a few carefully constructed YAML files and snippets of code, and wiring them together properly. You can start small and build out your platform over time.

The next few posts in this series will go beyond these simple examples, showing much more complicated implementations of all of these pieces, and recommend some best practices for managing your infrastructure with this platform.

And if you are already ready to get your hands on Pulumi after this introduction, feel free to create an account and follow some of our Getting Started guides to see how easy simple use cases are and begin to imagine how that same developer experience will scale up to your entire organization.

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

The Pulumi Cloud is a fully managed service that helps you adopt Pulumi’s open source SDK with ease. It provides built-in state and secrets management, integrates with source control and CI/CD, and offers a web console and API that make it easier to visualize and manage infrastructure. It is free for individual use, with features available for teams.

Create an Account

The Guide to Platform Engineering: 7 Steps to Get It Right

Sara Huddleston — Tue, 22 Oct 2024 07:47:03 +0000

In today’s fast-paced digital landscape, organizations are increasingly adopting platform engineering to optimize their software delivery and operations. Gartner predicts that by 2026, 80% of large software engineering organizations will have platform engineering teams to provide reusable services, components, and tools for application delivery. Additionally, by 2027, 80% of large enterprises will leverage platform engineering to scale DevOps initiatives in hybrid cloud environments effectively.

This shift is driven by the rise of cloud adoption, where many enterprises face the challenge of uncoordinated application teams deploying workloads in different ways across various cloud platforms. This siloed approach often results in a lack of standardization, security risks, and operational inefficiencies.

Platform engineering offers a strategic solution to these issues. This guide provides the essential steps to successfully implement platform engineering, from laying the foundation to scaling internal developer platforms (IDPs) for future growth.

On this article:

Step 1: Securing Executive Buy-in

The first step in your platform engineering journey is securing executive buy-in. This high-level support is essential, as platform engineering teams must create an organization-wide strategy that integrates internal developer portals (IDPs) and self-service capabilities. Leadership needs to understand how platform engineering can address inefficiencies caused by siloed application teams and uncoordinated cloud deployments.

To do this, present a clear roadmap with measurable outcomes, such as improved delivery speed and security. Use metrics like DORA (Deployment Frequency, Lead Time, Mean Time to Recovery) to demonstrate how platform engineering enhances security, standardization, and efficiency. Outline the required resources (headcount, budget, tooling) and align the project with business objectives.

Step 2: Staffing the Platform Engineering Team

Once you have executive backing, the next step is to assemble your platform engineering team. This team will develop and maintain your internal developer platform (IDP), ensuring it offers the organization secure, scalable, and self-service solutions.

Successful platform engineering teams are characterized by a strong customer focus, empathy for the needs of application teams, and the ability to act as the “glue” that binds the organization together. Key roles and responsibilities within the platform team include:

Customer-facing roles: Serve as the primary point of contact for application teams, understanding their needs and advocating for their requirements within the platform.
Infrastructure expertise: Possess deep knowledge of the underlying infrastructure, whether on-premises or in the cloud, to ensure the platform is reliable and scalable.
DevOps and automation skills: Leverage infrastructure as code (IaC) tools and techniques to automate the provisioning and management of the platform.
FinOps and cost optimization expertise: Understand the organization’s financial systems and processes to ensure the platform is cost-effective and aligned with budgetary constraints.
Software development capabilities: Develop the platform’s core components, including self-service interfaces and reusable infrastructure modules, using best practices in software engineering.

By assembling a team with this diverse set of skills and a customer-centric mindset, you’ll be well-positioned to build a platform that truly meets the needs of your application teams.

Step 3: Defining the Platform Mandate

Next, clearly define the platform team’s mandate to set expectations. This document should outline the platform’s purpose, the services it provides, and how it supports internal developer portals (IDPs) and self-service capabilities.

The platform mandate should serve as a central reference point for the platform team and its customers, ensuring everyone is aligned on the team’s mission and its value. Key elements to include in the platform mandate include:

Introduction: A high-level overview of the platform team’s purpose and the organizational outcomes it aims to achieve.
Supported systems: A list of the infrastructure, services, and tools that the platform team is responsible for managing and supporting.
Support model: Details on how application teams can access platform services, including any self-service capabilities and the team’s response times for different types of requests.
Team structure: An overview of the platform team’s members and their areas of expertise, making it easy for application teams to identify the right point of contact.

A well-defined mandate aligns the team and its customers with the platform’s value and purpose.

Step 4: Implementing the Pre-Production Pipeline

With a mandate in place, begin building the core capabilities, starting with a pre-production pipeline. This pipeline standardizes how application code moves from development to deployment, emphasizing security and consistency.

Key components of the pre-production pipeline include:

Version control: Implement a centralized version control system to manage application code and infrastructure as code (IaC) configurations.
Continuous integration (CI): Automate the build and testing of application artifacts, such as container images or deployment packages.
Artifact management: Provide a secure and versioned repository for storing and distributing the application artifacts produced by the CI process.
Static analysis: Integrate tools that can scan application code and dependencies for security vulnerabilities and other issues.
Application delivery: Automate the deployment of application artifacts to pre-production environments, such as staging or testing.

By establishing a standardized pre-production pipeline, you’ll ensure that all application teams follow a consistent, secure, and efficient process for getting their code into a production-ready state.

Step 5: Embracing Infrastructure as Code

Leveraging infrastructure as code (IaC) is crucial for managing the platform’s infrastructure. It enables consistent, scalable, and secure operations by automating the provisioning of resources, monitoring, and enforcing security policies.

Key areas where IaC can be applied within the platform include:

Foundational infrastructure: Provision and manage the underlying cloud resources, such as virtual networks, storage, and compute, that form the platform’s foundation.
Runtime platforms: Deploy and configure the runtime environments where application workloads will be executed, such as Kubernetes clusters or serverless functions.
Observability and monitoring: Set up the logging, metrics, and alerting systems that provide visibility into the platform’s health and performance.
Security and compliance: Implement security controls, such as secrets management and access policies, to ensure the platform meets regulatory and organizational requirements.
Pipelines and automation: Use IaC to define and version-control the platform’s own deployment and management pipelines, ensuring consistency and repeatability.

With infrastructure as code, the platform engineering team can ensure reliable, scalable, and secure infrastructure across the organization.

Step 6: Implementing Policy as Code

Alongside your IaC efforts, it’s important to set up a robust policy-as-code framework within your platform. Policy-as-code allows you to enforce security, compliance, and operational policies programmatically. This provides governance at scale, ensuring all teams follow the same rules.

Policy as code can be applied in two key ways:

Preventative controls: Implement policies that proactively validate and reject non-compliant infrastructure changes before they are provisioned, providing fast feedback to application teams.
Detective controls: Establish policies that continuously monitor the deployed infrastructure, triggering alerts or remediation actions when deviations from the desired state are detected.

By combining IaC and policy as code with self-service provisioning, you maintain security and compliance while giving teams autonomy.

Step 7: Evolving Towards Self-Service

As your platform matures, the ultimate goal is to empower your application teams with self-service capabilities, allowing them to provision the infrastructure and resources they need quickly and autonomously. This self-service model can take various forms, from pre-defined infrastructure modules to fully automated deployment pipelines.

When implementing self-service capabilities, keep the following platform engineering best practices in mind:

Start small and iterate: Don’t try to boil the ocean. Begin with a few well-defined use cases, such as CI/CD pipelines or database provisioning, and gradually expand the self-service offerings as your platform team gains experience and confidence.
Focus on user experience: Ensure that the self-service interfaces, whether web-based or command-line, are intuitive and easy to use. Gather feedback from application teams and continuously refine the user experience.
Maintain stability and reliability: Even with self-service, the platform team is responsible for ensuring the underlying infrastructure and services remain stable and reliable. Implement robust monitoring, alerting, and incident response processes to maintain platform health.
Embrace a product mindset: Treat the platform as a product, with the application teams as your customers. Continuously gather feedback, measure key performance indicators, and iterate on the platform’s capabilities to meet evolving needs.

By gradually transitioning towards a self-service model, you’ll empower your application teams to move faster while maintaining the necessary guardrails and controls to ensure the platform’s long-term success.

Conclusion: Embracing the Platform Engineering Journey

Building a successful platform engineering strategy involves following a clear roadmap. But remember, implementing the platform engineering strategy is a journey, not a destination. By following the steps outlined in this guide, you’ll be well on your way to building a robust, secure, scalable, and customer-centric internal developer platform (IDP) that empowers your organization to deliver software more efficiently and effectively.

With the right approach and mindset, your platform engineering efforts will become a strategic enabler for your organization’s digital transformation.

Build secure, scalable platforms with confidence—register for the Platform Engineering Course

Frequently Asked Questions

What is Platform Engineering?

There are many definitions for platform engineering, one of which is designing, building, and maintaining internal platforms that streamline software delivery by providing reusable tools, services, and workflows for developers. It focuses on scalability, efficiency, and self-service capabilities for internal teams.

For a more in-depth explanation of real-world use cases, see What is platform engineering.

Who are the Platform Customers?

The platform customers are the end users, typically internal teams (e.g., application developers and DevOps engineers), who use the platform’s tools and services to build, deploy, and manage software.

What is an Internal Developer Platform (IDP)?

A centralized platform that offers developers the tools and environments they need to build, test, and deploy applications. It abstracts away the complexity of underlying infrastructure, allowing teams to focus on coding and delivery.

What is a Platform Team?

The platform engineering team is a dedicated group of engineers responsible for managing the internal platform. They ensure the platform meets the needs of the organization, providing reliability, security, and scalability.

What is a Self-Service Infrastructure?

A platform feature that allows development teams to provision, configure, and manage infrastructure resources on demand without relying on other teams, fostering agility.

What is Infrastructure as Code (IaC)?

The process of managing and provisioning infrastructure through code, allowing infrastructure to be versioned, automated, and managed consistently across environments.

For a more in-depth explanation, see the What is Infrastructure as Code? page.

What is Policy as Code (PaC)?

Policy as Code (PaC) defines security, compliance, and operational policies in code to automate their enforcement across infrastructure and application deployments.

What is Developer Experience (DevEx)?

The overall experience developers have with using the platform, tools, and workflows provided by platform engineering. Optimizing DevEx ensures that developers can work more efficiently and effectively.

For a more in-depth explanation, read Beyond Productivity: Developer Experience is Business Critical.

Pushing Pulumi ESC Secrets into External Platforms

Komal Ali — Tue, 17 Sep 2024 00:00:00 -0300

This content now lives in the Pulumi docs. See Sync secrets to external platforms for the maintained version.

Managing secrets across multi-cloud infrastructures has long been a challenge for developers and operations teams. This article explores Pulumi IaC-based strategy to centrally define secrets and configuration in Pulumi ESC and automatically sync these values across the external platforms where they will be utilized, effectively reducing secret sprawl and manual overhead.

The Secret Management Conundrum

Traditional secret management often leads to a phenomenon known as secret sprawl. This occurs when secrets are duplicated across multiple systems, including various repositories, cloud providers, and CI/CD pipelines. The result is a tangled web of inconsistencies that makes tracking and managing secrets a Herculean task.

Updating secrets manually across all these systems is not only tedious but also error-prone. Each time a secret changes, it needs to be updated in every repository, cloud platform, and CI/CD tool where it’s used. This process is time-consuming and frustrating, often leading to mistakes that can compromise security.

Moreover, the lack of a centralized system makes it difficult to track where secrets are stored and who has access to them. This complicates security audits and compliance efforts. When a secret is compromised, the process of revoking and rotating it across multiple systems can be slow and disruptive, potentially leading to application downtime.

ESC Sync with Infrastructure as Code

Using Pulumi building blocks like ESC and IaC, we can create a pattern where secrets and configuration can be centrally defined in ESC, and then automatically synced across the external platforms where they will be utilized.

Consider the following ESC environment, which imports an environment “my-project/my-imported-env” and defines a set of secrets and configuration values to be synced:

imports:
 - my-project/my-imported-env@stable
values:
 sync:
 awsSecretsManager:
 value:
 myConfigKey: ${my-imported-env-foo}
 myNestedKey:
 haha: ${my-imported-env-bar}
 mySecret: ${my-imported-env-password}
 name: name-in-secrets-manager

The sync block defines the secrets and configuration values to be synced. In this example, we are syncing values to AWS Secrets Manager. The value field contains the actual values to be synced, while the name field specifies the name of the secret in AWS Secrets Manager.

To automate the synchronization process, we define a Pulumi program that creates a Pulumi ESC environment, Pulumi IaC stack, and Pulumi deployment settings for the target stack. We then define a set of pre-run commands that extract the values from the environment and set them as configuration values in the target stack. The syncCronSchedule variable specifies how often the synchronization process should run.

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();
const orgName = pulumi.getOrganization();

// projectName is the name of the project where the target stack is located
const projectName = config.require("projectName");

// stackName is the name of the target stack
const stackName = config.get("stackName") || "dev";

// repository is the GitHub repository where the target stack is located (for deployment settings)
const repository = config.require("repository");

// how often you want to sync. Default is hourly
const syncCronSchedule = config.get("syncCronSchedule") || "0 * * * *"

// envPath is the path to the environment file that contains the secrets or configuration to be synced
const envPath = config.get("envPath") || "syncEnv.yaml";

const env = new service.Environment("env", {
 organization: orgName,
 project: projectName,
 name: stackName,
 yaml: new pulumi.asset.FileAsset(envPath)
});

const stack = new service.Stack("esc-sync-aws-secretsmanager", {
 organizationName: orgName,
 projectName,
 stackName,
})

const fullyQualifiedStackName = pulumi.interpolate`${orgName}/${projectName}/${stackName}`;
const fullyQualifiedEnvName = pulumi.interpolate`${orgName}/${projectName}/${env.name}`;

const settings = new service.DeploymentSettings("deployment_settings", {
 organization: orgName,
 project: stack.projectName,
 stack: stack.stackName,
 github: {
 repository,
 },
 sourceContext: {
 git: {
 branch: "main",
 repoDir: "sync/target",
 }
 },
 operationContext: {
 preRunCommands: [
 'pulumi login',
 pulumi.interpolate`pulumi config env add ${projectName}/${env.name} -s ${fullyQualifiedStackName} --yes`,
 pulumi.interpolate`pulumi env open ${fullyQualifiedEnvName} sync.awsSecretsManager.value > sync.json`,
 pulumi.interpolate`pulumi config set -s ${fullyQualifiedStackName} secretName $(pulumi env open ${fullyQualifiedEnvName} sync.awsSecretsManager.name)`,
 ]
 }
});

const schedule = new service.DeploymentSchedule("update_schedule", {
 organization: orgName,
 project: settings.project,
 stack: settings.stack,
 scheduleCron: syncCronSchedule,
 pulumiOperation: "update",
})

Finally, we define a Pulumi program that creates the AWS Secrets Manager secret and secret version based on the values extracted from the environment file. The sync.json file contains the values to be synced, which are read and set as the secret string in the secret version. This Pulumi program is executed on a schedule to ensure that the secrets and configuration values are up-to-date between Pulumi ESC and AWS Secrets Manager.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from "fs";

const config = new pulumi.Config();
const name = config.require("secretName");

// Read a json file from the local filesystem using node.js fs module
const json = fs.readFileSync("sync.json", "utf8");

const secret = new aws.secretsmanager.Secret(name, {
 recoveryWindowInDays: 0,
})

const secretVersion = new aws.secretsmanager.SecretVersion(`${name}-version`, {
 secretId: secret.id,
 secretString: json,
})

// Export the name of the secret
export const secretName = secret.name;

This pattern allows you to define secrets and configuration values in a single location and automatically sync them across external platforms. The same pattern can be extended to create secrets in other platforms like Azure Key Vault, GCP Secrets Manager, GitHub Secrets, HashiCorp Vault and countless others.

Webhooks: An Event-Driven Alternative Approach

In addition to the scheduled synchronization approach, Pulumi ESC also supports webhooks. Webhooks allow you to respond to events in real-time, such as when a secret is updated in ESC, and take immediate action to sync the new value to an external platform.

In the example below, instead of a scheduled deployment, we define a webhook that triggers when the Pulumi ESC environment is updated. This ensures that the secrets and configuration values are always in sync between Pulumi ESC and the external platform.

const webhook = new service.Webhook("webhook", {
 organizationName: orgName,
 projectName: settings.project,
 stackName: settings.stack,
 displayName: "Sync to AWS Secrets Manager",
 environmentName: pulumi.interpolate`${env.project}/${env.name}`,
 filters: [WebhookFilters.ImportedEnvironmentChanged],
 format: WebhookFormat.PulumiDeployments,
 payloadUrl: pulumi.interpolate`${stack.projectName}/${stack.stackName}`,
 active: true,
})

When the webhook is triggered (i.e. the “my-project/my-imported-env” environment is updated), a deployment is automatically triggered, and the secrets and configuration values are synced to AWS Secrets Manager.

Embracing a New Era of Secret Management

Using Pulumi IaC to synchronize secrets and configuration across platform boundaries represents a giant step forward in secret management. By centralizing and automating the handling of sensitive data, it addresses the key challenges of traditional approaches. Secret sprawl becomes a thing of the past, and the risks associated with manual updates are significantly reduced.

Whether you’re working with AWS Secrets Manager, Azure Key Vault, GCP Secrets Manager, GitHub Secrets or other platforms, Pulumi ESC provides a streamlined solution that ensures your secrets are always secure and up-to-date. It offers the convenience and security of a single source of truth for all your secrets, allowing you to focus on building and improving your applications rather than wrestling with secret management.

We encourage you to explore Pulumi’s documentation and dive into the Pulumi ESC Sync Examples repository to see how you can implement this powerful tool in your own projects. Join our vibrant community on Slack to discuss your experiences, ask questions, and share insights. Together, we can make secret management simpler, more secure, and more efficient.

Pulumi Patterns and Practices Platform (P3): A reference architecture for large-scale organizations

Troy Howard — Mon, 05 Aug 2024 00:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Infrastructure management is all fun and games until you find yourself scrolling through 1000+ resources in your AWS console. Worse, when one rogue product team wants to use Azure and your data team wants to be on GCP, you’re ARM wrestling in Azure and watching your economies of scale tip the wrong direction as you’re copy-pasting CloudFormation templates into yet another git repo. This. Needs. To. Be. A. Platform!

And in that moment of overwhelm, you will be sold to, nurture-emailed every week, and told all your problems will be solved by implementing an IDP (internal developer platform, as if you’ve never seen this acronym before). An IDP that costs a lot of money and a lot of time to implement beyond default settings. An IDP that really only solves half of your problems. Your internal team offers to build something that feels more like welding together random pieces of code into an abstract found-art sculpture built from junkyard refuse, already 5 years out of date. How long will this investment be useful before you have to start over?

It’s exhausting. If there was a good solution on the market, you wouldn’t be reading this article. So let’s talk about what you really need, and how Pulumi can help.

An effective internal developer platform

There are quite a few listicles out there professing to authoritatively tell you the 5, or 7, or 11 essential components of an internal developer platform. Personally, I trust our customers to tell us, and here’s what they have said they need:

Consistency: Bring some order to the chaos. As your company and your infrastructure grows, it gets more and more complicated to maintain consistency. You might already have established design patterns that you want to replicate, but don’t have any way to encode those practices in your current tools. There’s a lot of copy/paste of reusable blocks, but no way to apply DRY principles or to modularize/templatize the important parts (hint: all the parts are important!).

Reproducibility: Repeatable behaviors, who dat? If you run your deploy twice do you get the same results each time? What if you replicate your production environment to create a test environment, are they actually identical? How much more work does it take to get them to be? Will you get the same version of the training dataset every time you run your AI workloads? It’s anyone’s guess. A lack of reproducibility slows down development, makes debugging more difficult, and makes that reuse we just talked about harder to achieve.

Visibility: When your node count, and user count starts to go beyond about 50-100 resources (computing or human) you quickly run into a problem of visibility. It can be very difficult to get a handle on what’s happening, how many resources you have, where they are, and how much they cost. Any system that purports to be able to manage 1000 nodes or more must have deeply integrated analytics, dashboards, charts, and be searchable, across all your clouds, all your users, and every kind of resource.

Security and Compliance: Good fences make good neighbors. RBAC, policy-as-code, excellent secrets management, integration with your existing identity providers. These are the things you need to build security and policy guardrails you can rely on. Without them? It’s just a powder keg of liability waiting to catch a spark.

Auditability: What happened and who did it? This is like a high-stakes game of Clue. How quickly can you figure out who ran that bad deployment? Was it Colonel Mustard in the library with the candlestick? Or Blake the new Front-End Developer with overly-broad permissions in AWS? Being able to answer these questions needs to happen quickly. Quickly, like minutes, not hours or days. And it might have happened 6 months ago. Oof.

Developer Experience: In the ideal world, developers drive their own DevOps. The platform team provides self-service tools and streamlined workflows that allow your engineers to provision new resources, so your team doesn’t have to. And you know, if the developers don’t like the user experience, they won’t use it at all, and will invent their own tools. You will have ROGUE SYSTEMS to hunt down and argue against in tedious overly-technical meetings. This is not what you want. We need to keep the developers happy to prevent this.

A holistic view of the Patterns and Practices Platform reference architecture

Pulumi has a broad surface area of products and features that address these needs. Designed with integration in mind from the beginning, our tools orchestrate well, presenting a smooth and streamlined workflow for both operations teams and developer teams.

We have an idea of how you can use all the Pulumi products together to deliver a comprehensive internal platform for security, infrastructure management, and deployments. Call it an internal platform for developer platform engineers (IPfDPE), if you want. We call it the realization of a vision we’ve been working hard to build for many years.

Pulumi Patterns and Practices Platform (P3) is a reference architecture that we will be describing, and providing code for, through this series of articles. We’ll be diving deep into not just what you can do with our tools, but how to do it, and provide code for a reference implementation that you can use to jump start the process.

Here’s a quick overview to give you an idea of how we’ll be addressing those needs in Pulumi Patterns and Practices Platform (P3).

Consistency

Pulumi can help bring consistency to your software catalog by encoding design patterns into reusable component resources and by building custom organization templates that provide a no-code or low-code way to start a new project. Templates help get projects off the ground faster and ensure consistent code structure, policy compliance, and best practices.

Figure: An internal developer portal using custom templates in Pulumi Cloud

Beyond that, because Pulumi is multi-cloud (AWS, Azure, Google Cloud, and more) and multi-language (JavaScript, Python, Go, C#, Java) you can enjoy the same consistency across all your environments and all your developer teams, regardless of the languages they prefer, or cloud tooling they need.

Another core aspect of consistency is drift detection. Pulumi automatically detects and remediates cloud resources that have deviated from the expected state stored in Pulumi Cloud. This tech is better than ibuprofen at getting rid of developer-created headaches.

Reproducibility

Since 2010, scientists have felt that we are in a crisis – a reproducibility crisis – wherein we cannot easily reproduce an experiment in order to verify published results. Similarly, the software industry is entering into a reproducibility crisis of its own, especially around AI training workflows, where it is increasingly difficult to recreate crucial build and prod environments. Pulumi Stacks make it very easy to manage both configuration and state across multiple environments, and make reproducing a deployment within Pulumi a matter of a few basic operations.

You can use Pulumi programs to capture all of the necessary resources for an AI training workload, including things like versioned data using dynamic tables with time-travel functionality in Snowflake. That means you can be sure that not only will your deployment be on the infrastructure you need, it will also have the exact version of data, every time, which is essential to A/B testing and debugging your models.

Visibility

Every resource under management by Pulumi is visible within Pulumi Insights. From this single-pane-of-glass interface, you can search for resources across all cloud environments. Pulumi Copilot provides a state-of-the-art AI chat interface to ask complex questions and get immediate results. Pulumi Insight’s analytics gives you the ability to identify anomalies or trends in resource usage and dig into cost, security, and compliance concerns.

Figure: Search for any resource with Pulumi Insights

Security and Compliance

In the modern parlance, when you say DevOps, you mean DevSecOps. Pulumi is designed to be secure by default. Pulumi Cloud offers full role-based access control (RBAC) functionality including deep integration with GitHub teams and SAML-based SSO, managed secrets and flexibly-defined secure environments with Pulumi ESC, and policy-as-code provided by Pulumi Crossguard. Most importantly all of these features are deeply integrated across the platform, creating an air-tight system with all the guardrails you need for managing security and access.

Auditability

Every action a user takes in Pulumi can be tracked via the audit log which is searchable in two clicks from the Pulumi Cloud homepage dashboard. Audit logs can be filtered by user with one more click. Creating automated backups of your audit logs is a first-class feature. You will never have to worry about responding quickly when someone asks about an event that happened in your system. Also, each deployment and update has logs directly visible from the Pulumi Cloud app, regardless of how it was initiated.

Figure: Viewing the audit log in Pulumi Cloud

Developer Experience

Probably the most compelling aspect of Pulumi is the developer experience. Developers love Pulumi, because they get to use their preferred tools. General purpose programming languages, visual IDEs, command-line tools, and products with an API-driven architecture are what developers want, and it’s what Pulumi delivers in spades.

With Pulumi templates and custom internal component resources in place, developers can drive their own DevOps, provisioning their own infrastructure resources and managing their own deployments directly, reducing bottlenecks in platform teams. Product engineering teams can self-service with a stream-lined workflow that stays compliant with company policy by default. Deep in the code of their favorite programming languages, your developers will never even know they are following the company rules.

Figure: Using C# to write a Pulumi program in VS Code

More to Come

So now that we’ve made a case for how Pulumi can be applied to meet the most pressing needs of a larger organization, hopefully you will realize that the Pulumi Patterns and Practices Platform (P3) reference architecture we are presenting here is more than just infrastructure-as-code. P3 is a Pulumi-powered platform for teams, where your developer portal is not just a catalog of software, but a fully functional control-plane across all your cloud environments.

Stay tuned for the following series of posts where we will use Pulumi to implement the P3 reference architecture for a fully-featured internal developer platform (IDP, or IPfDPE if you prefer). That said, you may already have invested in some popular in cloud-native tools like Backstage or Kubernetes. Pulumi plays well with others, and you will be delighted to see how you can use Pulumi to cover the gaps in the CNCF ecosystem.

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

Pulumi in a Cloud Native World

Engin Diri — Fri, 02 Aug 2024 00:00:00 +0000

In today’s complex digital landscape, organizations are increasingly turning to platform engineering to optimize their software delivery processes and maximize efficiency. The growing complexity of modern applications, coupled with the need for rapid, secure, and scalable deployments, has created a pressing demand for robust Internal Developer Platforms (IDPs).

IDPs are a key component of modern platform engineering strategies. An IDP is a self-service layer that sits on top of an organization’s infrastructure, abstracting away complexity and providing developers with the tools and environments they need to build, test, and deploy applications efficiently.

Key features of an IDP:

Developer Control Plane. Curated experiences that empower developers by meeting them at their level of expertise, whether it’s an abstracted developer portal, custom CLI, or shared IaC templates.
Integration & Delivery. Automations to version control, test, trace, and deploy all infrastructure from resources, configurations, environments, and secrets as well as orchestration automations to manage provisioning workflows.
Monitoring & Logging. Components to log, monitor, and observe all infrastructure for greater operational control as well as optimize against unnecessary costs.
Security & Identity. Security and compliance guardrails that regulate every piece of infrastructure from policies to fine-grained access controls to secrets.
Resources. Providers that support modern cloud architectures such as Kubernetes, containers, serverless, generative AI, machine learning, data lakes, hybrid cloud/on-premises, and more.

By implementing an IDP, organizations can significantly reduce cognitive load on developers, accelerate time-to-market, improve security and compliance, and foster a culture of innovation. IDPs act as a force multiplier, allowing development teams to focus on creating business value while the platform handles the underlying complexity of cloud-native infrastructure.

The Cloud Native Computing Foundation (CNCF) landscape offers a large ecosystem of tools and technologies that can be leveraged to build these platforms from scratch. However, the sheer number of options can be overwhelming, making it difficult for organizations to determine the best combination of tools for their specific needs.

At Pulumi’s Customer Experience Team, we’ve observed that adopting Infrastructure as Code often serves as a catalyst for organizations to reassess their entire software delivery process. This reassessment frequently leads to the implementation of more streamlined, automated, and secure methodologies – the core principles of platform engineering mentioned above.

To provide guidance for our customers, we have created a reference architecture called the KEBAP stack, this stack offers a structured approach to integrate these technologies into an effective platform engineering solution using Pulumi.

The KEBAP Stack Reference Architecture

The KEBAP stack consists of the following (most commonly used) CNCF projects:

Kubernetes (with Kyverno as a bonus)
External Secrets Operator
Backstage
Argo CD
Pulumi

Have a look on the high-level architectural diagram of the KEBAP stack to get a first idea of what we are going to talk about in the following sections:

Breaking Down the KEBAP Stack

Let’s take a closer look at each component of the KEBAP stack and understand what their role. The KEBAP is layered in such a way that each component builds on the capabilities of the previous one. The five layers of the KEBAP stack are designed to address different capabilities of our internal developer platform:

Kubernetes

In modern cloud computing, the hub-and-spoke architecture pattern is widely used to manage multiple Kubernetes clusters. In this pattern, a central control plane (hub) manages a set of clusters (spokes) that run applications and services. The hub provides a unified view of the entire infrastructure, enabling centralized management, monitoring, and security.

The control plane cluster in the KEBAP stack is the hub cluster and contains the control plane logic for the entire IDP.

Pulumi

Pulumi manages the lifecycle of all cloud infrastructure, providing developers with self-service capabilities. It allows you to use familiar programming languages to define your infrastructure as code, offering more flexibility and power than traditional YAML or domain-specific languages.

The Pulumi Kubernetes Operator enables you to manage your infrastructure directly from within Kubernetes, seamlessly integrating with your existing Kubernetes workflows.

Since we use GitOps as our deployment method, we define the infrastructure in a Git repository and then let Argo CD deploy it to the hub cluster. As soon as the infrastructure definition is deployed, the Pulumi Kubernetes Operator will take over and create the defined resources in the selected cloud provider.

Here is an example Pulumi definition of a new Kubernetes cluster using DigitalOcean:

apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: ${{values.name}}
program:
 variables:
 clusterName: "do-cluster"
 clusterRegion: "nyc1"
 nodePoolName: "default"
 nodeCount: 1
 version: 1.30.1-do.0

 resources:
 do_cluster:
 type: digitalocean:KubernetesCluster
 properties:
 name: "${clusterName}"
 region: "${clusterRegion}"
 version: "${version}"
 destroyAllAssociatedResources: true
 nodePool:
 name: "${nodePoolName}"
 size: "s-2vcpu-2gb"
 nodeCount: "${nodeCount}"
 # cut for brevity
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: ${{values.name}}
spec:
 stack: ${{values.stack}}
 envRefs:
 DIGITALOCEAN_TOKEN:
 type: Env
 env:
 name: DIGITALOCEAN_TOKEN
 programRef:
 name: ${{values.name}}
 destroyOnFinalize: false

Argo CD

Argo CD handles the deployment layer, ensuring that the current deployment matches what is declaratively defined in the source. It’s emerging as the de facto standard due to its rich front-end and powerful features. Argo CD implements the GitOps model, continuously monitoring your Git repositories and automatically updating the deployed applications to match the desired state.

In the KEBAP stack, we use Argo CD to deploy the Pulumi infrastructure defined in the Git repository to our hub cluster.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: ${{values.name}}-${{values.stage}}-${{ (values.owner | parseEntityRef).name }}
 namespace: argocd
spec:
 destination:
 namespace: pulumi-operator
 server: https://kubernetes.default.svc
 project: ${{ (values.owner | parseEntityRef).name }}
 source:
 path: "gitops/teams/clusters/${{values.name}}-${{values.stage}}-cluster"
 directory:
 recurse: false
 exclude: '{catalog-info.yaml}'
 targetRevision: main
 repoURL: https://github.com/my-backstage-demo/backstage-infrastructure-provisioning-templates-workshop.git
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - ServerSideApply=true
 - CreateNamespace=true

External Secrets Operator (ESO)

The External Secrets Operator (ESO) addresses the challenge of securely managing secrets in Kubernetes. It integrates with various secret stores (such as Pulumi ESC, AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault). This allows you to configure access to the secret store once for all your required secrets, enhancing security and simplifying management.

We even take it a step further by using ESO to manage the secrets from Pulumi ESC (Environments, Secrets, and Configurations) with the Pulumi Provider for ESO. This way, we use the Pulumi not only for the infrastructure but also for the secrets management reducing the number of tools we need to manage.

To give the Pulumi Kubernetes Operator access to the secrets, we create following ClusterSecretStore object:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
 name: secret-store
spec:
 provider:
 pulumi:
 organization: <org>
 environment: <env>
 accessToken:
 secretRef:
 name: pulumi-access-token
 key: PULUMI_ACCESS_TOKEN
 namespace: argocd

And then we can define the ExternalSecret with all the secrets that we want to manage and pass to our Operator as environment variables extraEnv:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: pulumi-operator-secrets
spec:
 refreshInterval: 1h
 secretStoreRef:
 kind: ClusterSecretStore
 name: secret-store
 data:
 - secretKey: pulumi-access-token
 remoteRef:
 key: "backstage.pulumi-pat"
 - secretKey: do-token
 remoteRef:
 key: "backstage.do"

Backstage

Backstage is an open platform for building developer portals. It provides a centralized place for managing software catalogs, documentation, and tooling. This layer helps in organizing microservices and infrastructure, streamlining the process for developers to create, manage, and explore services.

In our KEBAB stack, we installed the Pulumi plugin for Backstage, which allows us to manage our infrastructure as code directly from the Backstage UI by cookie-cutting the Pulumi project and making the result available in our source control repository. From there, Argo CD takes over and deploys the definition to the hub cluster as described above.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
 name: kubernetes-template
 title: Kubernetes Cluster
 description: |
 A template for creating a new Kubernetes Cluster.
 tags:
 - pulumi
 - kubernetes
spec:
 steps:
 - id: pulumi-new-component
 name: Cookie cut the component Pulumi project
 action: pulumi:new
 input:
 name: "${{ parameters.component_id }}-infrastructure"
 description: ${{ parameters.description | dump }}
 organization: ediri
 stack: ${{ parameters.stack }}
 template: "https://github.com/my-silly-organisation/microservice-civo/tree/main/infrastructure-${{ parameters.cloud }}-${{ parameters.language }}"
 config:
 "node:node_count": "${{ parameters.nodeCount }}"
 folder: .

Bonus: Kyverno

Kyverno can be used to enforce policies and automate security and operational best practices in Kubernetes clusters. Kyverno is a policy engine designed for Kubernetes that allows you to define policies as code and enforce them at runtime. This ensures that your Kubernetes clusters are secure and compliant with your organization’s policies.

In the KEBEP stack, we use Kyverno to create a ClusterPolicy object to automatically add vCluster clusters to the Argo CD as spoke clusters. This will generate a Secret object in the argocd namespace for each vCluster cluster the dev teams create.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: sync-secret-vcluster
 annotations:
 argocd.argoproj.io/sync-wave: "5"
spec:
 generateExistingOnPolicyUpdate: true
 rules:
 - name: sync-secret-vcluster-nonprod
 match:
 any:
 - resources:
 names:
 - "*-dev-*"
 - "*-staging-*"
 - "*-prod-*"
 kinds:
 - Secret
 exclude:
 any:
 - resources:
 namespaces:
 - kube-system
 - default
 - kube-public
 - kyverno
 context:
 - name: namespace
 variable:
 value: "{{ request.object.metadata.namespace }}"
 - name: name
 variable:
 value: "{{ request.object.metadata.name }}"
 - name: ca
 variable:
 value: "{{ request.object.data.\"certificate-authority\" }}"
 - name: cert
 variable:
 value: "{{ request.object.data.\"client-certificate\" }}"
 - name: key
 variable:
 value: "{{ request.object.data.\"client-key\" }}"
 - name: vclusterName
 variable:
 value: "{{ replace_all(name, 'vc-', '') }}"
 jmesPath: 'to_string(@)'
 - name: stage
 variable:
 value: "{{ split(vclusterName, '-')[2] }}"
 jmesPath: 'to_string(@)'
 - name: team
 variable:
 value: "{{ split(vclusterName, '-')[3] }}"
 jmesPath: 'to_string(@)'
 generate:
 kind: Secret
 apiVersion: v1
 name: "{{ vclusterName }}"
 namespace: argocd
 synchronize: true
 data:
 kind: Secret
 metadata:
 labels:
 argocd.argoproj.io/secret-type: cluster
 clusterType: "vcluster"
 env: "{{ stage }}"
 team: "{{ team }}"
 stringData:
 name: "{{ vclusterName }}"
 server: "https://{{ vclusterName }}.{{ namespace }}:443"
 config: |
 {
 "tlsClientConfig": {
 "insecure": false,
 "caData": "{{ ca }}",
 "certData": "{{ cert }}",
 "keyData": "{{ key }}"
 }
 }

Putting all the Pieces Together

As we have seen, each component of the KEBAP stack plays a crucial role in the overall architecture. Let’s how the different workflows look like when we put all the pieces together. We go through the following scenarios:

Ignite the Control Plane Cluster With Pulumi
Ordering a New Development Kubernetes Cluster
Ordering a New Production Kubernetes Cluster
Ordering Other Infrastructure

Ignite the Control Plane Cluster With Pulumi

We use Pulumi code to create the control plane cluster in our cloud provider of choice. After the creation of the control plane, we install the Argo CD in a very bare-bones way:

const argocd = new ArgoCD("argocd", {
 initialObjects: initialObjects
}, {
 providers: {
 kubernetes: k8sProvider,
 },
});

The idea is to let then Argo CD fetch the rest of the configuration from the Git repository and apply it to the cluster. And here comes the power of GitOps: It will also manage the Argo CD itself. This way, we separated the workload running on the cluster from the creation of the cluster itself.

Ordering a New Development Kubernetes Cluster

Starting point is Backstage, where we can order a new development Kubernetes cluster. As we want to keep the costs low for development cluster, we opted here for the vCluster project. The vCluster project allows us to create a Kubernetes cluster on demand, which runs as containers on the control plane cluster. The moment the cluster is created, we add it to the Argo CD as spoke cluster. This way, we can roll out applications to the development cluster in the same way as we would do for production clusters.

See the definition of the ClusterPolicy object in the Kyverno section.

Ordering a New Production Kubernetes Cluster

The Backstage catalog is again the starting point to order a new production Kubernetes cluster. This time, we use the Pulumi code to create the cluster in the cloud provider of choice. After the creation of the cluster, we add it to the Argo CD as a production cluster. This way, we can roll out applications to the production cluster in the same way as we would do for development clusters.

Ordering Other Infrastructure

Any other infrastructure components such as databases, message queues, and storage will be also ordered via Backstage. And similar to the ordering of a new production Kubernetes cluster, we use Pulumi to create the cloud infrastructure.

Conclusion

A lot of organizations are building their internal developer platforms using CNCF projects to fulfill the needs of their internal customers. The KEBAP stack provides a structured approach to integrate these technologies into an effective platform engineering solution. We also saw that Pulumi plays a crucial role in this stack by taking care of the management of the cloud infrastructure.

This KEBAP stack offers several benefits, including:

A secure, automated, and auditable environment
Reproducible and programmable infrastructure-as-code
Unified management of cloud resources, Kubernetes, and application deployments
Faster onboarding for new team members
Easier compliance with regulatory requirements due to increased auditability

But there is also a downside: The KEBAP stack requires a significant investment in time and resources to set up and maintain as you are in full charge of the integration of the components as well as operating them. All the projects under the umbrella of CNCF are not automatically implying that they are compatible with each other out of the box. Any platform engineering team will need to invest continuously time and resources to keep the stack up-to-date and running. This is a not negligible investment to consider for any organization.

For this reason, we have another reference architecture that uses components from all of the Pulumi Platform for an end to end fully-managed solution. This solution takes care of the integration of the different components of an internal developer portal and provides a carefree experience of managing your infrastructure at scale.

Continue to upskill and learn about platform engineering:

Read Pulumi Patterns and Practices (P3): A Pulumi-based reference architecture for large-scale organizations
Learn how to build Developer Portals with Pulumi.
Register for one of our Platform Engineering or DevOps workshops.

Next-level IaC: Powering Up Pulumi with Package Ecosystems

Troy Howard — Mon, 13 May 2024 00:00:00 +0000

Every experienced tech professional I know has a programming language they love. But is it the syntax and symbols that make it so loveable? Not really. It’s the community and package ecosystem surrounding the language that makes a real impact on your heart… and on your productivity!

If we look at some of the biggest success stories in tech — Python, Node.js, Ruby, Perl, and Go — the common thread between all of them is an extensive ecosystem of packages, libraries, modules (or whatever you decide to call them… ahem, Gems?!). A great language will allow you to build anything you can imagine, but a great ecosystem will have already written it for you, and made it available in a convenient to install-and-use package.

One of the amazing things about Pulumi is that it is built around general-purpose programming languages, and that means your Pulumi programs have access to the entire ecosystem of packages that come with each language. This is a stark difference between Pulumi and other infrastructure automation tools that use proprietary domain-specific languages with not much in the way of community around them. Some tools might allow you to write custom code, but they certainly don’t make it convenient, and still… you have to write it yourself, which just adds so much overhead to a project where your core concern isn’t writing that custom function, but rather shipping your own product on the infrastructure you are trying to automate.

The flexibility that Pulumi gives you, and the access to these powerful libraries of code, means your productivity can skyrocket when you get to write in your favorite language, using your preferred tools and the package ecosystem you already know, love, and rely on every day.

Let’s have a look at a concrete example where this package ecosystem can get you out of a tough spot.

Use Case: Determining S3 MIME types on-the-fly

A popular example of cloud infrastructure automation is creating an S3 bucket and pushing some files to it. This is a common task that many of us have had to do, and most infrastructure automation tools have a way to accomplish this.

Here’s one way to do that in Pulumi:

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';

// Create an S3 bucket where we will upload the file.
const bucket = new aws.s3.Bucket("my-bucket", {
 // You can apply additional bucket settings here.
});

const filePath = "path/to/your/local/file.txt"; // Replace with the path to the file you want to upload

// Upload the file to the S3 bucket.
const bucketObject = new aws.s3.BucketObject("my-bucket-object", {
 bucket: bucket, // Reference to the bucket created above.
 key: "file.txt", // This will be the file's name in the bucket.
 source: pulumi.FileAsset(filePath), // Contents of the file.
 contentType: "text/plain", // MIME-type of the file. Adjust if you upload a different type of file.
});

// Export the name of the bucket
export const bucketName = bucket.id;

// Export the URL of the file in the S3 bucket
export const objectUrl = pulumi.interpolate`s3://${bucket.id}/${bucketObject.key}`;

import pulumi
import pulumi_aws as aws

# Create an S3 bucket
bucket = aws.s3.Bucket('my-bucket')

# The path to the file to upload to S3
file_path = "path/to/your/local/file.txt" # Replace with the path to the file you want to upload

# Upload the file to the S3 bucket
bucket_object = aws.s3.BucketObject('my-bucket-object',
 bucket=bucket.id, # Reference to the bucket created above.
 key='file.txt', # This will be the file's name in the bucket.
 source=pulumi.FileAsset(file_path),
 content_type='text/plain') # MIME-type of the file. Adjust if you upload a different type of file.

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

# Export the URL of the file in the S3 bucket
pulumi.export('object_url', pulumi.Output.concat('s3://', bucket.id, '/', bucket_object.key))

However, something that stands out as a weak link here is the line:

contentType: "text/plain", // MIME-type of the file. Adjust if you upload a different type of file.

content_type='text/plain') # MIME-type of the file. Adjust if you upload a different type of file.

Yeah, hard-coding the MIME type for every file I need to upload is a not going to scale. In fact, I’m probably going to get a random directory full of files I need to iterate over, each with different file types, and it will change every time it runs, and we won’t know what type of file it is ahead of time.

Packages to the rescue!! Turns out that determining the MIME type of a file only involves importing a community-created module like mime-types in TypeScript, or using the mimetypes module from Python’s standard library, and a single function call to guess the MIME type:

import * as mime from 'mime-types'

// Detect the MIME type of the file
const mimeType = mime.lookup('path/to/your/local/file.txt');

import mimetypes

# Detect the MIME type of the file
mime_type, _ = mimetypes.guess_type('path/to/your/local/file.txt')

So, to work that into our Pulumi program from above, we only need to make a few small edits:

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';
import * as mime from 'mime-types';

// Create an S3 bucket where we will upload the file.
const bucket = new aws.s3.Bucket("my-bucket", {
 // You can apply additional bucket settings here.
});

const filePath = "path/to/your/local/file.txt"; // Replace with the path to the file you want to upload

// Upload the file to the S3 bucket.
const bucketObject = new aws.s3.BucketObject("my-bucket-object", {
 bucket: bucket, // Reference to the bucket created above.
 key: "file.txt", // This will be the file's name in the bucket.
 source: pulumi.FileAsset(filePath), // Contents of the file.
 contentType: mime.lookup(filePath) || "application/octet-stream", // MIME-type of the file
});

// Export the name of the bucket
export const bucketName = bucket.id;

// Export the URL of the file in the S3 bucket
export const objectUrl = pulumi.interpolate`s3://${bucket.id}/${bucketObject.key}`;

import pulumi
import pulumi_aws as aws
import mimetypes

# Create an S3 bucket
bucket = aws.s3.Bucket('my-bucket')

# The path to the file to upload to S3
file_path = "path/to/your/local/file.txt" # Replace with the path to the file you want to upload

# Detect the MIME type of the file
mime_type, _ = mimetypes.guess_type(file_path)

# Upload the file to the S3 bucket
bucket_object = aws.s3.BucketObject('my-bucket-object',
 bucket=bucket.id, # Reference to the bucket created above.
 key='file.txt', # This will be the file's name in the bucket.
 source=pulumi.FileAsset(file_path),
 content_type=mime_type or 'application/octet-stream') # MIME-type of the file

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

# Export the URL of the file in the S3 bucket
pulumi.export('object_url', pulumi.Output.concat('s3://', bucket.id, '/', bucket_object.key))

Well that was pretty straightforward, wasn’t it? Trying to do this in another tool might take all day to figure out the right way to implement it, but in Pulumi it worked exactly like you would expect it to, in no time at all.

It’s a real game-changer when you can just quickly grab something complicated like MIME typing from a package. But when you can’t, you’re going to be stuck figuring out how to write that functionality yourself, from scratch, using things like Bash and primitive data types, and then find yourself copy-pasting that stuff all over the place. All the while fully aware of how much time you’re wasting by having to do so. That’s a real pain! It’s the kind of pain most people managing infrastructure have been through before.

The possibilities are endless

What other packages can you imagine using in your infrastructure solutions? Maybe you could use the text-to-image library to generate an image badge for your website? Maybe use the crypto module to encrypt a file before uploading it to S3? Or possibly using octokit to check on the state of something in git before taking the next step in your automation?

Whatever you can do with code in your favorite language, you can do in Pulumi as part of your infrastructure automation. I’m not saying you should use cowsay to generate custom MOTD banners for your ssh logins, I’m just saying you could.

Next Steps

If you haven’t already, install Pulumi today, and follow our self-directed Getting Started guides to learn more about making the most of Pulumi’s next-level infrastructure management features at your organization.

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

Next-level IaC: Drop those wrapper scripts and let your language do that for you

Christian Nunciato — Tue, 30 Apr 2024 00:00:00 +0000

Our users are always telling us (particularly the ones who come to Pulumi from other IaC tools) that being able to use general-purpose languages to manage their infrastructure was a game changer for them.

I know it was for me. As a JavaScript developer, when I discovered Pulumi and saw that I could do pretty much everything I was doing with Terraform but with TypeScript, I was immediately hooked; that’s all it took. Just being able to write my resource declarations in a language I knew well (and that my IDE understood) was huge.

But it’s easy to forget, even when you’ve been using Pulumi for a while, that you aren’t just getting the language alone — you’re getting the full capability of the language’s runtime environment also. Not just JavaScript, but all of Node.js. Not just C#, but everything the .NET framework, runtime, and ecosystem have to offer.

This is quite a big deal, actually. Whereas with other tools, you’re often bound by the limits of the tool’s DSL, with Pulumi, you have no such constraints; you can do just about anything your chosen language can do. That means much more than just declaring long lists of cloud resources — it means unlocking all kinds of capabilities that can help bring your infrastructure-management practice to a whole new level.

In this post, the first of a new series we’re calling Next-level IaC, we’ll show you a few small but useful ways you can take advantage of this reality — first, by spotting some opportunities to simplify your deployments.

Run pre-deployment tasks with Pulumi

Fundamentally, a Pulumi program is just a regular program. That means you can take actions programmatically at runtime — before the Pulumi engine gets going.

Suppose you were tasked with writing the code to deploy and manage a static website. Conceptually, you might break this process up into a few separate steps:

Build the website
Deploy the supporting infrastructure (e.g, the cloud storage, maybe a content-delivery network)
Copy the content of the website into the infrastructure

This you might lead you to reach for Bash to write up a shell script like the one below, which uses Hugo (a popular static-site generator) to build the website, Pulumi to provision the infrastructure, and the AWS CLI to push the built website up into the AWS cloud:

#!/bin/bash

# Build the website.
hugo --destination ./public

# Provision the infrastructure.
pulumi up --yes

# Copy the website content into the infrastructure.
aws s3 sync ./public s3://$(pulumi stack output bucketName)

And this would definitely work. But it’s also unnecessary. You can avoid both the Bash wrapper and the dependency on the AWS CLI by using your chosen language’s built-in support for spawning shell processes:

import * as aws from "@pulumi/aws";
import * as childProcess from "child_process";

// Build the website.
const result = childProcess.execSync("hugo --destination ./public", { stdio: "pipe", cwd: "./www" });

// Provision a storage bucket for the website.
const bucket = new aws.s3.Bucket("bucket", {
 website: {
 indexDocument: "index.html",
 },
});

// Copy the website home page into the bucket.
const homepage = new aws.s3.BucketObject(
 "index.html",
 {
 bucket: bucket.id,
 content: fs.readFileSync("./www/public/index.html", "utf-8"),
 contentType: "text/html",
 acl: "public-read",
 },
);

import pulumi_aws as aws
import subprocess

# Build the website.
result = subprocess.run(
 ["hugo", "--destination", "./public"],
 stdout=subprocess.PIPE,
 cwd="./www",
 check=True,
 shell=True,
)

# Provision a storage bucket for the website.
bucket = aws.s3.Bucket(
 "bucket", website=aws.s3.BucketWebsiteArgs(index_document="index.html")
)

# Copy the website home page into the bucket
homepage = aws.s3.BucketObject(
 "index.html",
 bucket=bucket.id,
 content=open("./www/public/index.html", "r").read(),
 content_type="text/html",
 acl="public-read",
)

Running processes like these synchronously, as above, means they’ll complete before Pulumi starts registering resources — important here because the BucketObject resource needs that public folder to exist for the deployment to succeed. Fortunately it does, so all is well:

$ pulumi up
Updating (dev)
Type Name Plan Info
+ pulumi:pulumi:Stack example-dev create 15 messages
+ ├─ aws:s3:Bucket bucket create
+ └─ aws:s3:BucketObject index.html create
Diagnostics:
pulumi:pulumi:Stack (example-dev):
Start building sites ...
| EN
-------------------+-----
Pages | 4
Paginator pages | 0
Non-page files | 0
Static files | 0
Processed images | 0
Aliases | 0
Sitemaps | 1
Cleaned | 0
Total in 11 ms
Resources:
+ 3 created

Also notice how the example captures the output of the process in result and writes it to the terminal. Standard logging like this tends not to be possible with static languages (at least not without some hackery) because there’s no real way to do it programmatically. Similarly, those that do support spawning processes tend to require them to return strictly formatted JSON strings — which, as the example shows, not all processes do. (Forcing you to have to wrap them with yet more shell scripts.)

Simple stuff, but flexible and powerful. With one line, you’ve kept your deployment process simple, dropping two tools you don’t need thanks to the built-in capabilities of Pulumi and your language of choice.

Fetch data to generate resources dynamically

To build on the static-website scenario, suppose you wanted to retrieve some data from an external source and use it to provision some cloud resources dynamically. For example, you might want to let your marketing team create short URLs on their own for social media posts. For this, you might once again turn to Bash and the AWS CLI to pull those URLs from your corporate CMS (e.g., with curl), either before or after running pulumi, and create the corresponding redirects imperatively using a follow-up shell command of some sort.

Again, a totally reasonable approach. But again, why bother, when you have the full capability of Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET at fingertips? Just add a few lines to your program to fetch the data and be on your way:

// ...

// Fetch some redirects from a hypothetical CMS.
const redirects = fetch(`${process.env.CMS_ENDPOINT}/redirects.json`)
 .then(response => response.json())
 .then(items =>
 items.forEach((redirect: any, i: number) => {
 // Create an S3 website redirect for each one.
 new aws.s3.BucketObject(
 `redirect-${i}`,
 {
 bucket: bucket.id,
 key: redirect.from,
 websiteRedirect: redirect.to,
 acl: "public-read",
 },
 { dependsOn: [ownershipControls, publicAccess] },
 );
 }),
 );

#...
import json
import os
import requests

# ...

# Fetch some redirects from a hypothetical CMS.
response = requests.get(f"{os.environ['CMS_ENDPOINT']}/redirects.json")
redirects = json.loads(response.text)

# Create an S3 website redirect for each one.
for i, redirect in enumerate(redirects):
 aws.s3.BucketObject(
 f"redirect-{i}",
 bucket=bucket.id,
 key=redirect["from"],
 website_redirect=redirect["to"],
 acl="public-read",
 )

Notice the example parses the response using the language’s built-in JSON support. If your CMS happened to return data in some other format, or required some additional credentials for authorization, no problem — your language and its package ecosystem are going to be there to help you.

Also notice the example reads the source URL directly from the environment. This is typically more cumbersome (if even possible) with other tools, requiring you to name your environment variables in tool-specific ways and then tack on additional blocks of configuration before you’re able to use them. As you can see, Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET makes this much more convenient.

Finally, notice the redirects are translated into cloud resources using a familiar JavaScript TypeScript Python Go C# Java Pulumi YAML F# VB looping construct. We’ll cover this topic in much more depth in an upcoming post. (Stay tuned!)

Hook the event loop to run tasks after deployment

Finally, you can do nifty things like run tasks programmatically after a Pulumi deployment completes — once again, without ever having to leave the Pulumi program.

If your hypothetical website were sitting behind a CloudFront CDN, for example, you might want some way to clear the CDN’s cache after a deployment. This generally requires some imperative action, like using the AWS CLI to run aws cloudfront create-invalidation sometime later. How might you do something like this with Pulumi, given Pulumi programs exit automatically when their operations complete?

One way — again, since a Pulumi program is just a regular Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET program — would be to write a JavaScript TypeScript Python Go C# Java Pulumi YAML F# VB function to create an invalidation request, and then ask the Node.js Node.js Python Go .NET Java Pulumi YAML .NET .NET runtime to run that function before exiting.

The mechanics of this vary slightly by language, but it works, and it’s a very handy way to keep all of your deployment logic in one place — again avoiding the need to relegate anything to a Bash script:

// ...
import * as cloudfront from "@aws-sdk/client-cloudfront";

const config = new pulumi.Config("aws");
const region = config.require("region");

// Create a CloudFront distribution for the website.
const cdn = new aws.cloudfront.Distribution("cdn", {
 enabled: true,
 defaultRootObject: "index.html",
 origins: [{
 originId: bucket.arn,
 domainName: bucket.websiteEndpoint,
 // ...
 }],
 // ...
});

function createInvalidation(id: string) {
 // Only invalidate after a deployment.
 if (pulumi.runtime.isDryRun()) {
 console.log("This is a Pulumi preview, so skipping cache invalidation.");
 return;
 }

 process.on("beforeExit", () => {
 const client = new cloudfront.CloudFrontClient({ region });
 const command = new cloudfront.CreateInvalidationCommand({
 DistributionId: id,
 InvalidationBatch: {
 CallerReference: `invalidation-${Date.now()}`,
 Paths: {
 Quantity: 1,
 Items: ["/*"],
 },
 },
 });

 client
 .send(command)
 .then(result => {
 console.log(`Invalidation status for ${id}: ${result.Invalidation?.Status}.`);
 process.exit(0);
 })
 .catch(error => {
 console.error(error);
 process.exit(1);
 });
 });
}

// Register a function to be invoked before the program exits.
cdn.id.apply(id => createInvalidation(id));

The relevant code is the call to process.on("beforeExit"), which registers a function to be invoked just before the program exits using the resolved ID of the distribution. (For more on how the apply() method works, see Inputs and Outputs.) The function returns early for Pulumi previews — no sense clearing the cache if the site hasn’t changed — and uses the AWS SDK for JavaScript to submit the invalidation request to CloudFront, naming it uniquely with a timestamp and logging the result to the console:

# ...
import boto3
import asyncio

def create_invalidation(id):
 # Don't bother invalidating unless it's an actual deployment.
 if pulumi.runtime.is_dry_run():
 print("This is a Pulumi preview, so skipping cache invalidation.")
 return

 client = boto3.client("cloudfront")
 result = client.create_invalidation(
 DistributionId=id,
 InvalidationBatch={
 "CallerReference": f"invalidation-{time.time()}",
 "Paths": {
 "Quantity": 1,
 "Items": ["/*"],
 },
 },
 )

 print(
 f"Cache invalidation for distribution {id}: {result['Invalidation']['Status']}."
 )


# Register a function to be invoked before the program exits.
cdn.id.apply(lambda id: atexit.register(lambda: create_invalidation(id)))

The relevant code is the call to atexit.register(), which registers a function to be invoked just before the program exits using the resolved ID of the distribution. (For more on how the apply() method works, see Inputs and Outputs.) The function returns early for Pulumi previews — no sense clearing the cache if the site hasn’t changed — and uses Boto3 to submit the invalidation request to CloudFront, naming it uniquely with a timestamp and logging the result to the console:

$ pulumi preview
Previewing update (dev)
Type Name Info
pulumi:pulumi:Stack example-dev 16 messages
Diagnostics:
pulumi:pulumi:Stack (example-dev):
...
This is a Pulumi preview, so skipping cache invalidation.

$ pulumi up
Updating (dev)
Type Name Status Info
pulumi:pulumi:Stack example-dev 16 messages
Diagnostics:
pulumi:pulumi:Stack (example-dev):
...
Cache invalidation of distribution EAJOVC1QNGT13: InProgress.

Techniques like these are a great way to eliminate unnecessary scripts, glue code, and other dependencies while keeping your deployment logic clear, well contained, and amenable to change.

Wrapping up

Hopefully this first post has given you a few new ways to think about how to do more with Pulumi and your language of choice. Over the next few weeks, as this series unfolds, we’ll share a lot more, so keep an eye on this space!

You’ll find the full source for the examples above on GitHub:

Happy coding!

Next Steps

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

Create an Account

IaC Best Practices: Summarizing Key Learnings

Scott Lowe — Mon, 08 Apr 2024 00:00:00 +0000

Welcome to the final post in our series of articles focused on Infrastructure as Code best practices. In this post, we’ll wrap up our recommendations for IaC with Pulumi, summarizing the best practices from previous posts as well as highlighting some areas of potential future growth for the team at Zephyr Archaeotech Emporium—the fictional company at the center of the story throughout this series.

For ease of navigation, here are links to all the blog posts in the series:

IaC Best Practices: Understanding Code Organization and Stacks
IaC Best Practices: Enabling Developer Stacks and Git Branches
IaC Best Practices: Structuring Pulumi Projects
IaC Best Practices: Applying Stack References
IaC Best Practices: Implementing RBAC and Security
IaC Best Practices: Using Automation API
IaC Best Practices: Summarizing Key Learnings (this post)

Recapping IaC Best Practices

As this series has progressed, we’ve been showing you how Zephyr’s use of Pulumi has changed in response to how the organization, the team, and the application has changed. In case you haven’t been following along—or in case you need a refresher—here’s a brief summary of what’s happened in the series:

Zephyr started with a single project in a single repository with two stacks, one for production and one for testing. (Details are available in the first blog post in the series.)
Zephyr quickly added per-developer stacks as a way to enhance developer productivity. (You can read about per-developer stacks in the second post in the series.)
As Zephyr continued to grow, their single Pulumi project grew into three different Pulumi projects: one for base infrastructure, one for their Kubernetes platform, and one for the online store application. (The third post in the series describes the reasoning for adopting multiple projects.)
Accompanying the switch to multiple projects, Zephyr implemented stack references (to pass information between stacks in different projects) and applied role-based access control (RBAC) to the stacks in Pulumi Cloud. (Refer back to the fourth post and the fifth post in the series for more details on each of these areas.)
Zephyr realized a need to split out their data layer, going from three projects to four projects, and along the way saw an opportunity to add a higher level of orchestration using the Pulumi Automation API. (Refer back to our sixth post for more details on Zephyr’s initial use of Automation API.)

Since the last blog post, the Zephyr team has continued to explore Automation API, and has added an Automation API program that allows their developers to deploy the Zephyr online store directly from the associated GitHub repositories—developers don’t even need to clone down the repositories first.

To see the latest Automation API programs for Zephyr, refer to the zephyr-automation GitHub repository.

Areas of future growth for Zephyr

We’ve stated before that best practices are always point-in-time recommendations based on the requirements of the organization/team/individual using Pulumi. What this means, in practice, is that there is almost always room for growth and evolution. Organizations, teams, and individuals rarely remain static, and therefore their use of Pulumi is also unlikely to remain entirely static.

So, what are some potential areas of future growth for Zephyr?

Splitting the deployment of the microservices: We alluded to this in the previous post. Right now, all the microservices that comprise the online store are deployed together. What if one particular team needs to deploy only their service? The current approach can certainly accommodate that—Pulumi is declarative and will only change what needs to be changed—but what’s better for the Zephyr team? Does Zephyr need to optimize for allowing service teams to move independently? Is it about reducing blast radius, so that undesired changes aren’t introduced to another service? These are the sorts of considerations that drive this decision (and the third post in our series discusses structuring Pulumi projects in more detail).
Using component resources: Currently, the Zephyr team is using the standard API objects supplied by the Pulumi provider. For example, the various microservices for the Zephyr online store are defined using standard Kubernetes objects, like this:
```
const assetsNs = new k8s.core.v1.Namespace("assets-ns", {
 metadata: ...})

const assetsService = new k8s.core.v1.Service("assets-service", {
 metadata: ...})

const assetsDeployment = new k8s.apps.v1.Deployment("assets-deployment", {
 metadata: ...})
```
Component resources would allow the Zephyr team to define a single logical resource that contains “child” resources. This would allow the Zephyr team to create a Microservice resource that incorporates the standard Kubernetes Namespace, Deployment, and Service objects. Deploying an instance of a Microservice resource deploys all the child objects defined in the component resource. This can simplify the Pulumi code through further use of DRY (Don’t Repeat Yourself).
Making use of Pulumi Deployments: Zephyr’s CI/CD pipelines are not a topic we’ve discussed in the series. Given that Zephyr uses GitHub, they could use GitHub Actions (GHA); Pulumi has an action for GHA workflows. Zephyr could also use Pulumi Deployments, which offers an extensive set of features—OIDC support for dynamic credentials, Review Stacks (per-PR stacks), support for GitHub Enterprise, and GitOps-style workflows, to name a few. Support for Pulumi Deployments is also baked into Automation API, meaning that Zephyr could extend their existing Automation API code to incorporate Pulumi Deployments.
Addressing secrets and configuration management: We’ve also not discussed how Zephyr could manage their secrets and configuration data. Pulumi ESC is a good option here, offering seamless integration with Pulumi for infrastructure as code while also supporting a variety of other DevOps tools and use cases.

Even though it’s working well, clearly there’s a lot of room for the Zephyr team to continue to enhance and improve their infrastructure as code implementation!

Summarizing Best Practices

Throughout the series, we’ve provided guidelines on how to best utilize Pulumi. While we recommend reading the entire series, in this section will “summarize our summaries” and provide some broad, overarching practices.

With regard to Pulumi projects, keep it as simple as possible, but no simpler. While tools like Automation API can help with orchestrating multiple projects, it remains true that in most cases a single project is simpler. Use multiple projects when necessary, but only when necessary.
Use stacks freely. Using multiple projects has a cost in complexity. Using multiple stacks, on the other hand, does not—so use them freely to model different environments, to provide per-developer test environments, to test changes before rolling them into production, or for any other use case where you need a separate, independent instance of the infrastructure defined in your Pulumi program.
Focus on the requirements. It can sometimes be too easy to focus on the technology instead of the requirements. There are a lot of very cool things you can do with Pulumi, but try not to get distracted from the reason you’re using infrastructure as code: repeatable, reliable, consistent deployments of your cloud infrastructure.
Don’t forget about security along the way. Make sure you are properly securing your infrastructure. Pulumi CrossGuard can help here by providing “policy as code” guardrails for cloud infrastructure. Pulumi Cloud’s RBAC functionality helps with ensuring the right users have the right permissions to the right sets of cloud infrastructure resources.

Reviewing the Code

You can view the code we’ve presented in this series in the following GitHub repositories:

The “zephyr-infra” repository contains the base infrastructure definitions.
The “zephyr-k8s” repository contains the Pulumi code to stand up the Kubernetes cluster onto which Zephyr deploys their online store application.
The “zephyr-data” repository manages Zephyr’s “data layer” for persistent data used by the online store application.
The “zephyr-app” repository is Zephyr’s online store. This repository has both the application code as well as the Pulumi code for deploying the application (see the “infra” directory).
In the “zephyr-automation” repository you’ll find Automation API programs used by Zephr to orchestrate deployments of the different projects involved in supporting the online store.

In each of these repositories, there are different branches that reflect the state of the code as of a particular blog post. The “main” branch reflects the final state of the code, as described in this wrap-up post.

Wrapping up

This series has used the story of a fictional company, Zephyr Archaeotech Emporium, to discuss some recommendations around how to best utilize Pulumi for solving their infrastructure as code needs. While this company is fictional, the benefits of Pulumi and infrastructure as code are real. We invite you to try for yourself—sign up for a free Pulumi Cloud account today!

You might also like:

Your Perfect Infrastructure May Not Be So Perfect

Replicating Data to Support Multi-Region Applications

Adora Nwodo — Wed, 27 Dec 2023 11:03:10 +0000

In the previous article, we covered multi-region scaling, its importance, and how you can use Pulumi stacks to represent multiple regions and environments. The big takeaway from that article is that scaling your application across multiple regions is an important architectural decision to enable scalability and availability, but it comes with its own set of considerations. One of these considerations is around the data that your application needs or uses. Data replication is typically necessary in multi-region architectures because if you have multiple application instances worldwide making calls to just one database instance, the latency of your application will be high. After all, it will take a long time for that database instance to perform data operations and send back a result to the user through the application. Some requests might have repeated timeouts. In this article, we will cover data replication in multi-region applications and how it plays a pivotal role in distributed systems.

Why is data replication important?

In almost every instance of using multi-region scaling with an application, data replication becomes imperative for a few different reasons:

Latency reduction: Replicating data closer to end users minimizes data access latency, providing a seamless and responsive user experience.
Availability: When data is distributed across different locations, applications may be able to remain available even when there are regional outages or disruptions. There are some architectural/design decisions around this that we’ll discuss in the next section, but generally speaking a globally distributed approach creates redundancy. This means that if one region experiences an outage, application users can still access data from alternative locations.
Load balancing: Data replication enables effective load balancing. To minimize overload on a single server or region, traffic can be distributed among numerous replicas, optimizing resource utilization and boosting overall system performance.

Data Replication and the CAP Theorem

Any discussion of data replication among multiple regions invariably needs to address the CAP theorem. In a nutshell, the CAP theorem (also known as Brewer’s theorem) states that a distributed data store can only provide two of three guarantees: consistency, availability, and partition tolerance. Generally, these three guarantees are defined in this way:

Consistency – all clients always have the same view of the data: Choosing strong consistency assures that all clients, regardless of their geographical location in the distributed system, see the same data. However, the pursuit of uniformity frequently results in increased latency, particularly when dealing with low bandwidth and/or high latency networks.
Availability – all clients can always read and write the data: Prioritizing high availability means that the system responds to client read and write requests even in the event of network partitions. Accepting eventual consistency and accepting some transient divergence among replicas may be required to achieve availability.
Partition Tolerance – the system will continue to work despite partitions: Partition tolerance is non-negotiable in cloud computing, when systems span numerous regions and components are distributed. It ensures that the system continues to function even in the face of physical partitions or network outages, but doing so will require strategic compromises in terms of consistency or availability.

To reiterate, application and infrastructure architects have no choice but to tolerate network partitioning, as networks outside their control or influence—either the cloud provider’s network or the internet or both—typically fall in the replication path. Therefore, application and infrastructure architects have to choose the behavior of the data store when (not if) network partitioning occurs:

Does the data store cancel the operation, preserving consistency but affecting availability?
Does the data store proceed with the operation, prioritizing availability at the cost of consistency?

As mentioned earlier, though, more than just network partitioning comes into play here. Network bandwidth affects the balance between preserving consistency or optimizing for availability in the context of multi-region applications. Network configurations also come into play; what impact will a larger MTU (Maximum Transmission Unit) have on data replication? Decisions here are full of trade-offs. For example, prioritizing replication speed to ensure data consistency across regions may lead to increased latency, affecting the system’s ability to meet stringent availability requirements. An extension to the CAP theorem, known as PACELC, captures the trade-off between latency and consistency even when network partitions aren’t present.

Therefore, the challenge lies in finding the right balance. Striking a compromise between data consistency and system performance (in the form of availability) is a delicate task requiring a careful analysis of the principles of the CAP theorem, guiding system architects in making informed choices that align with the specific needs and priorities of their distributed applications.

Using Pulumi to configure data replication

Now that you’ve seen the impact of CAP theorem-related decisions on distributed data stores and the applications that use them, let’s take a look at a specific example. Let’s say you are building a distributed system and you want to have a central database with multiple replicas across the different availability zones (or regions) where your application will be deployed.

Cloud providers such as Azure, AWS, and Google Cloud all have databases that offer multi-region capabilities (AWS has DynamoDB, Azure has Cosmos DB, and Google Cloud has Cloud Spanner). Thanks to Pulumi’s integration with a variety of cloud providers, including (but not limited to) Azure, AWS, and Google Cloud, you can use Pulumi to deploy instances of these multi-region databases in your preferred programming language. This allows you to take advantage of these offerings and still reap the benefits of using infrastructure as code with Pulumi.

Let’s look at an example of using Pulumi to define an Azure Cosmos DB account with multi-region replication:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure-native";

const resourceGroup = new azure.resources.ResourceGroup("myResourceGroup");

const cosmosDbAccount = new azure.documentdb.DatabaseAccount("myCosmosDbAccount", {
 resourceGroupName: resourceGroup.name,
 location: resourceGroup.location,
 locations: [
 { locationName: "East US" },
 { locationName: "West US" },
 // Add additional Azure regions for replication
 ],
 databaseAccountOfferType: "Standard",
 consistencyPolicy: {
 defaultConsistencyLevel: "Session",
 },
});

In this example, the locations parameter is used to specify the Azure regions where Cosmos DB replicas will be deployed, enabling global data distribution. Aligning these locations with the locations where your application code is deployed allows you to minimize latency due to reading from the database.

Additionally, you can also enable multi-region writes. This is typically useful for write-active scenarios, ensuring that write operations can be directed to any region with a writable replica, and can reduce latency associated with writes to the database. This reduction in write latency often translates directly into improved performance for a multi-region application.

const cosmosDbAccount = new azure.documentdb.DatabaseAccount("myCosmosDbAccount", {
 resourceGroupName: resourceGroup.name,
 … // more code
 enableMultipleWriteLocations: true,
});

Consistency

You will need to choose a consistency level that aligns with your application requirements, such as “Session,” “Bounded staleness,” “Eventual,” “Strong,” or “Consistent prefix.” This page has more details on the different consistency levels, but let’s look at a couple specific configurations.

Recall that strong consistency ensures that all nodes or replicas in the system view the same data simultaneously, regardless of which node they are accessing. In other words, after performing a write operation, any subsequent read action from any node must return the most recent write value. This consistency mode guarantees the linear ordering of processes, and the system behaves like a single, coherent entity. Strong consistency prioritizes consistency over availability in the event of network partition. It also hinders performance as the write must be acknowledged by multiple nodes.

You can prioritize strong consistency by configuring the Azure Cosmos DB account to use the “Strong” consistency level. This ensures that all read and write operations observe the most recent version of the data across all replicas.

const cosmosDbAccount = new azure.documentdb.DatabaseAccount("myCosmosDbAccount", {
 resourceGroupName: resourceGroup.name,
 consistencyPolicy: {
 defaultConsistencyLevel: "Strong",
 },
});

Cosmos DB also supports tunable consistency. Tunable consistency means that the consistency can be changed for each read and write request. Tunable consistency levels can be used in instances when strict strong consistency is not required. When configured this way, the application now assumes the responsibility for determining the consistency.

This next code snippet shows how to configure the Cosmos DB account with a “Bounded Staleness” consistency level, which allows you to balance consistency and latency based on application requirements.

const cosmosDBAccount = new azure.documentdb.DatabaseAccount("myCosmosDB", {
 resourceGroupName: resourceGroup.name,
 consistencyPolicy: {
 // Leverage tunable consistency levels for specific scenarios
 defaultConsistencyLevel: "BoundedStaleness",
 maxStalenessPrefix: 100,
 },
});

Now that the responsibility for assigning consistency is with the client, the code snippet below shows how to make API calls with tunable consistency for both reads and writes using Azure Cosmos DB SDK:

import { CosmosClient } from "@azure/cosmos";

const endpointUri = "Your_Cosmos_DB_Endpoint_URI";
const primaryKey = "Your_Cosmos_DB_Primary_Key";
const databaseId = "Your_Database_Id";
const containerId = "Your_Container_Id";

const cosmosClient = new CosmosClient({ endpoint: endpointUri, key: primaryKey });
const database = cosmosClient.database(databaseId);
const container = database.container(containerId);

async function writeDocumentWithTunableConsistency() {
 const document = { id: "1", PartitionKey: "MyPartitionKey", Message: "Hello, Cosmos DB!" };

 const requestOptions = {
 consistencyLevel: "BoundedStaleness",
 maxStalenessPrefix: 100,
 };

 await container.items.create(document, requestOptions);

 console.log("Document written with tunable consistency.");
}

async function readDocumentWithTunableConsistency() {
 const documentId = "Your_Document_Id";

 const requestOptions = {
 consistencyLevel: "Eventual",
 };

 const { resource } = await container.item(documentId, "Your_Partition_Key").read(requestOptions);

 console.log(`Document read with tunable consistency. Message: ${resource.Message}`);
}

(async () => {
 await writeDocumentWithTunableConsistency();
 await readDocumentWithTunableConsistency();
})();

Availability

In the context of the CAP and PACELC theorems, availability refers to the guarantee that every request will receive a non-error response—but without the guarantee that it contains the most recent write. As we’ve discussed throughout this article, when dealing with distributed data stores this means navigating the trade-offs between consistency and availability. If you opt for solid consistency then you ensure uniform data views but you reduce availability during partitions. If you opt to prioritize availability, then you have no choice but to deal with consistency issues.

This tradeoff becomes especially relevant when uninterrupted system availability is necessary, such as high-traffic applications or systems with stringent service level agreements (SLAs).

It’s also important to understand that availability is not the same as fault tolerance, especially in a CAP theorem context. Going back to our example of using multiple Cosmos DB replicas to support a multi-region application, Cosmos DB (as shown earlier) absolutely supports placing replicas in different availability zones or different regions. This enables cross-region replication, ensuring that the data exists in multiple locations. In the event one of these locations fails or becomes unreachable or inoperable, the application can tolerate this fault (meaning the application isn’t necessarily completely down and there is little or no data loss) but availability may still be impacted. For example, choosing to prioritize consistency over availability means that when this fault occurs the database may block, restrict, or delay operations until the fault is resolved.

So how does one tune availability with a distributed data store? Consistency and availability are two sides of the same coin; adjusting consistency affects availability. With Cosmos DB, you tune availability by adjusting the consistency settings on the database instance. If you opt for strict consistency, then availability is impacted; if you opt for a less strict consistency level, then availability is not impacted as much (or not at all).

Here’s how to configure CosmosDB to prioritize availability:

const cosmosDbAccount = new azure.documentdb.DatabaseAccount("myCosmosDbAccount", {
 resourceGroupName: resourceGroup.name,
 location: resourceGroup.location,
 locations: [
 { locationName: "East US" },
 { locationName: "West US" },
 // Add additional Azure regions for replication
 ],
 consistencyPolicy: {
 defaultConsistencyLevel: "Eventual",
 },
 enableAutomaticFailover: true,
});

When prioritizing availability, you might choose to use a default consistency level of “Eventual,” as shown in the example above. Eventual consistency prioritizes high availability by enabling updates to propagate gradually across replicas. While it has decreased read latency, it accepts temporary inconsistencies that resolve over time. As you might expect, your application will need to be prepared to deal with these temporary inconsistencies.

A more balanced option is called “BoundedStaleness.” This choice introduces staleness in read operations, allowing for better availability during network partitions. “BoundedStaleness” provides a controlled staleness window (maxStalenessPrefix), enabling developers to fine-tune the balance between consistency and availability. Bounded Staleness offers a sort of middle ground, allowing users to define a controlled maximum lag for read operations and therefore ensuring a specified level of consistency while maintaining availability during network partitions. This model suits scenarios where a compromise between consistency and availability is necessary, and the application can tolerate a defined level of data staleness.

Your application’s requirements influence the decision between these options.

By also enabling enableAutomaticFailover, the code ensures that Cosmos DB automatically performs failover in the event of a regional or node failure. Automatic failover improves availability by redirecting traffic to healthy instances and minimizing downtime during unexpected disruptions.

Partition tolerance

Partition tolerance isn’t a tunable/configurable parameter. Application and infrastructure architects need to accept that network partitions will occur, and instead need to configure the data store to behave in an expected fashion (either prioritizing consistency at the expense of availability, or prioritizing availability at the cost of consistency).

The CAP theorem posits that, during network partitions, a system must make a trade-off between consistency and availability. In Cosmos DB, tweaking the consistency level influences data presentation to clients. Opting for solid consistency ensures uniform data views but may reduce availability during partitions. Conversely, prioritizing availability over consistency allows for continuous operation, even with potential data discrepancies. High consistency levels may result in increased latency and reduced availability during partitions, while prioritizing availability ensures responsiveness, even in partitioned scenarios.

Understanding these trade-offs is crucial for designing resilient systems. It allows for informed decisions about the system’s behavior during network partitions, ultimately contributing to effectively managing partition tolerance in distributed systems and databases.

Conclusion

Data replication is important for ensuring the resilience and performance of stateful distributed services. This article not only emphasized the significance of data replication in distributed systems, but it also demonstrated how Pulumi, in collaboration with Azure, enables developers to architect and manage resilient, performant, and globally distributed infrastructures for their stateful applications. As cloud-native architectures expand, the importance of infrastructure as code (IaC) in ensuring optimal data replication outcomes becomes increasingly important, and Pulumi can be used to provision the infrastructure required.

Did you learn something new? We’d like to know. If you’d like to try this out for yourself, you can get started with Pulumi Cloud here. You can also join our community to be a part of the continuous conversation, and we can’t wait to see what you build!

Scaling Applications Across Multiple Regions

Adora Nwodo — Wed, 06 Dec 2023 20:24:40 +0000

As a team building a distributed cloud service that will be used by different people around the world, you will need strategies to cater to users globally, ensuring uninterrupted service even in the face of disruptions. Have you ever wondered how businesses operate and deliver high-level performance across different regions? Well, that is the result of scaling applications across multiple regions — in a bid to ensure constant access and flexibility, organizations distribute their applications and databases across various geographic regions so that if one part faces issues, the service remains available elsewhere.

In this article, we will explore the significance of multi-region scaling and show using Pulumi for multi-region deployment.

Understanding multi-region scaling

When distributed systems adopt (or implement) multi-region scaling, they position themselves for high availability because the redundancy minimizes downtime and ensures that the service stays up and running. If one region experiences issues, such as server downtime or network problems, the application remains accessible to users in other regions. This also leads to better performance in their systems because a fast and responsive experience for users across the globe is achieved.

It is easier to reach a global audience when you have scaled geographically. This means that if you have customers in Europe and the US and you’re a multi-regional cloud service, by distributing your traffic across different instances, you can improve your customers’ experience by serving users worldwide and also catering to diverse regions and time zones. Specific data protection and privacy regulations are also met, as multi-region scaling allows systems to adhere to these regulations by keeping data within defined boundaries.

Although multi-region deployments are needed for scaling, availability, and performance, there could be downsides associated with them if the design process for being multi-region isn’t well thought out. Some of the challenges that could be encountered are:

Complexity: Each region may have unique requirements, and ensuring consistency in configuration can be challenging. Troubleshooting across regions can add complexity to the infrastructure setup.
Cost: Expanding to multiple regions often increases infrastructure costs. Businesses need to balance the cost of maintaining multiple resources with the benefits of high availability and performance.
Deployment consistency: Inconsistencies in resource configuration can lead to performance issues and errors, impacting the user experience.
Security compliance: Meeting security and compliance standards in different regions may require additional effort. Each jurisdiction may have unique data protection and privacy regulations, and this can be a challenge.

The Pulumi solution

A stack is a distinct deployment target within a Pulumi project. Stacks enable you to organize, version, and manage your infrastructure code effectively. They are flexible enough to do more than merely plan deployments across different stages of development. You may better control infrastructure orchestration with them as an adaptable tool for managing deployments in a variety of circumstances. Each stack represents a separate instance of your infrastructure, allowing you to deploy and manage your resources in different environments or configurations. To learn more about stacks, you can read this.

Consider how Pulumi Stacks can be used effectively in the following ways:

Deploying to different environments: You can use stacks to deploy into different environments, like development, staging, and production. Stacks allow separate configurations to be maintained throughout these contexts.
Deploying to different regions: You can use stacks to effortlessly manage deployments in multiple regions (e.g. westus2, westeurope, northeurope etc.). This enables you to scale your infrastructure across geographical locations while maintaining a centralized and consistent approach to resource allocation. These deployments could be in the same environment, but different regions (e.g. production westeurope, and production westus2), and you can use stacks to handle the different permutations.
Deploying to different cloud provider accounts: Stacks can also be used to automate deployments across several cloud provider accounts. Even when working with different cloud platforms, your deployment process can be unified and consistent.

Each of these stacks have their own configuration, but there are also some configuration values that are shared across stacks. To obey the DRY (Don’t Repeat Yourself) programming rule, you can pair stacks with Pulumi ESC (Environments, Secrets, and Configuration). This pairing results in a flexible deployment strategy. The combination empowers users to segregate configurations for each environment, manage secrets securely, and maintain a centralized store for configuration data. This ensures a clean separation of concerns, contributing to improved security and maintainability in multi-environment deployments. Pulumi ESC is a pivotal aspect of managing stacks, especially as the number of stacks grows. The idea is to store configuration settings separately from your code, ensuring that each stack has configuration values tailored to its specific needs.

In the next section, we’ll show how to deploy applications across different regions using Pulumi stacks and Pulumi ESC.

Define regions and stacks

Let’s look at an example of deploying to three distinct regions — production westus2, production westeurope, and staging southeastasia — using stacks. In this scenario, we will use three stacks: prodwu2, prodwe, and stagingsea. Each of these stacks will have a stack configuration file associated with it, named Pulumi.<stack>.yaml. In this example, we are using stacks both to deploy into different environments (production and staging), but also to deploy into different regions (West US 2, West Europe, and Southeast Asia).

You can create your stacks by running:

pulumi stack init prodwu2
pulumi stack init prodwe
pulumi stack init stagingsea

After creating your stacks, you can update your Pulumi configuration YAML files with their corresponding values (typically done with the pulumi config set command).

Pulumi.prodwu2.yaml:

config:
 multi-region:appServiceTier: PremiumV2
 multi-region:appServiceSize: P1V2
 multi-region:environment: production
 multi-region:location: westus2

Pulumi.prodwe.yaml:

config:
 multi-region:appServiceTier: PremiumV3
 multi-region:appServiceSize: P1V3
 multi-region:environment: production
 multi-region:location: westeurope

Pulumi.stagingsea.yaml:

config:
 multi-region:appServiceTier: Standard
 multi-region:appServiceSize: S1
 multi-region:environment: staging
 multi-region:location: southeastasia

Leveraging Pulumi ESC

For configuration values that are shared across stacks, Pulumi ESC allows you to create them once and reference them in multiple stacks. In the referenced codebase, there are other values e.g. resourceNamePrefix, storageKind, storageSkuName, and tenantId. Since these values are the same across all our stacks, we can leverage ESC so that we can declare them once and reuse them everywhere.

Pulumi ESC is a feature of Pulumi Cloud and in order to use ESC, you should be using Pulumi Cloud as your backend.

To leverage Pulumi ESC, you first need to initialize a new environment like this:

pulumi env init shared-multi-region

You can use the pulumi env command, as shown.

After which, you can set all the value for the configurations you need. Remember to add it as part of the pulumiConfig object, so that your Pulumi stack understands and can pick it up.

pulumi env set shared-multi-region pulumiConfig.resourceNamePrefix mrapp
pulumi env set shared-multi-region pulumiConfig.storageKind "Standard_LRS"
pulumi env set shared-multi-region pulumiConfig.storageSkuName "Standard_LRS"
pulumi env set shared-multi-region pulumiConfig.tenantId "00000000-0000-0000-0000-000000000000" # Your tenantId

After successfully adding your config values to your new environment, you can reference it across all your stacks like this:

config:
 multi-region:appServiceTier: Standard
 multi-region:appServiceSize: S1
 multi-region:environment: staging
 multi-region:location: southeastasia

environment:
 imports:
 - shared-multi-region

Here, shared-multi-region has more configurations and you can add references to them in your stack without copying and pasting multiple lines.

Update Pulumi code

Once you have set up the Pulumi ESC environment and linked it to your Pulumi stack configuration, you also need to update your Pulumi code to use the configuration values instead of any hard-coded values. This is shown in the code snippet below using the config.require method:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

// Define the configuration
const stackName: string = pulumi.getStack();
const config = new pulumi.Config();

const appServiceTier = config.require("appServiceTier");
const appServiceSize = config.require("appServiceSize");
const resourceNamePrefix = config.require("resourceNamePrefix"); // gotten from ESC
const storageKind = config.require("storageKind"); // gotten from ESC

// ... more code

// Create an Azure Resource Group
const resourceGroup = new azure.resources.ResourceGroup(resourceGroupName, {
 resourceGroupName: resourceGroupName,
 location: location,
 tags
});

// Create an Azure Storage Account
const storageAccount = new azure.storage.StorageAccount(storageAccountName, {
 accountName: storageAccountName,
 resourceGroupName: resourceGroup.name,
 kind: storageKind,
 location: location,
 sku: {
 name: storageSkuName
 },
 tags
});

// ... more code

Deploy stacks

Your Pulumi stacks should be ready to deploy now—you’ve updated the code to use configuration values that either come from the stack or from Pulumi ESC, and you’ve done both the per-stack configurations and the Pulumi ESC configuration values. Now it’s just a matter of running pulumi up:

pulumi up --stack prodwu2
pulumi up --stack prodwe
pulumi up --stack stagingsea

Now, you have seen how to use stacks to represent regions and environments. With this approach, the dynamism happens through your configuration and your code doesn’t have to change per region.

To see a full application with more detail, click here.

Considerations for high availability

Once you’ve configured your multi-region deployments, you should consider strategies for high availability to ensure that applications remain operational and accessible even in the face of failures.

Placing workloads across multiple regions

Effectively distributing workloads across various geographic regions is a key strategy for maintaining high availability. You can employ one of the following methods to distribute traffic across multiple regions:

Active/Passive with hot standby: In this approach, one region actively handles traffic while the other remains on hot standby, meaning the services in the secondary region are always running.
Active/Passive with cold standby: Similar to hot standby, in this case, the services in the secondary region aren’t allocated until needed for failover. This minimizes costs but might increase recovery time.
Active/Active: Both regions are active, and requests are load-balanced between them. If one region becomes unavailable, it is taken out of rotation.

In the context of infrastructure as code using Pulumi, these strategies directly translate to the management of multiple stacks. In active/hot standby and active/active configurations, there will typically be two Pulumi stacks, each with its allocated set of resources. This allows for the independent management and deployment of resources in each region.

On the other hand, in an active/cold standby scenario, while there might be a Pulumi stack defined for the cold standby site, it won’t have any resources initially allocated. Part of the failover process would involve running a pulumi up specifically for the cold standby region, triggering the allocation of resources as needed. This dynamic resource allocation during failover helps optimize costs, as resources are provisioned only when necessary, contributing to a more efficient use of cloud resources.

Replicating data

Implementing appropriate data replication solutions is an important factor in enhancing system resilience. This involves creating duplicates of our databases and storing them in several locations to ensure that our data remains consistent and accessible. Such replication is especially important for sustaining smooth operations if one region has disruptions or malfunctions.

Exploring the details of data replication exposes the most difficult aspects of being multi-region with high availability. Finding the correct balance between Recovery Point Objective (RPO) and Recovery Time Objective (RTO) is an important factor. RPO refers to how frequently we store our data, but RTO refers to how soon we can get things back on track if there is a problem. Finding the correct balance entails considering how frequently we update our data and how long it takes to recover from unanticipated errors.

Services such as AWS Aurora Global Databases or Azure Cosmos DB can be used to simplify this complex operation. These services are intended to enable multi-region database replication as simple as possible, while also maintaining high availability across distributed systems.

Configuring load balancing, failover, and disaster recovery

Configuring load balancing, failover, and disaster recovery is needed to achieve high availability in a system. These strategies help distribute traffic efficiently, ensure continuous operation, and mitigate the impact of potential failures or disasters.

Load balancing: Choose a way to balance the work of your servers—it could be by using a cloud service like Azure Traffic Managers. Then, decide how you want to divide the job among your servers, like taking turns or picking the one with the least number of tasks to process. Make sure you set up health checks to keep an eye on how well your servers are doing. If you’re working in different parts of the world, use global balancing to send people to the closest or healthiest place for your data.
Failover: Think about things that might go wrong, like servers crashing or network problems. If something breaks, set up a plan to automatically switch to a healthy backup. Monitor your servers to catch anomalies quickly. Make sure you have failover infrastructure ready to take over if there is ever an outage. Also, make sure the tool that shares the work (the load balancer) doesn’t break itself. If something goes wrong, set up a way for your system to keep working, even if it’s not at full power. This is known as graceful degradation.
Graceful degradation: Graceful degradation is a concept that prepares systems for potential failures. When faced with unanticipated challenges such as server crashes or network outages, a system designed with graceful degradation adjusts its performance to maintain key functionality. Consider a web application that is significantly reliant on external APIs for real-time data. If one of these APIs experiences a temporary glitch, a system that embraces graceful degradation may seamlessly switch to presenting cached data or simplifying functionality, ensuring users maintain access to core functionalities even when service outages occur. This strategy is also applicable to cloud-based services, where a system intelligently scales down non-essential features under high loads in order to prioritize key activities, proving the robustness of systems designed using gentle degradation principles.
Disaster recovery: Be ready for the worst by copying your important data to different places. Have clear steps on how to bring back your data and programs if everything goes wrong. Spread your tools and services in different areas to reduce problems if one place has a big issue. Make sure your system can easily switch to another area if a big problem happens. Plan for disasters by making step-by-step guides on what to do. Test your disaster plans often to be sure they work well and to find any areas that need fixing.

Security considerations

It is one thing to multi-scale, it’s another responsibility to secure the infrastructure. Securing a multi-regional infrastructure introduces challenges. In multi-region setups, data privacy can be tricky due to different regulations, like GDPR in Europe, or The California Consumer Privacy Act (CCPA). Using differential privacy and data residency controls are important when handling data according to each region’s rules. Pulumi policy as code also enables the proactive design and enforcement of compliance-ready policies. Ensuring compliance in industries like healthcare and finance adds complexity, requiring a tailored approach based on specific regulations.

Expanding infrastructure in multi-regional setups makes them more vulnerable to cyber threats. To tackle this, it’s important to use good security practices like micro-segmentation, intrusion detection systems, and regular penetration testing. Being proactive in incident response planning, controlling access effectively, and employing strong encryption methods are also important for a secure cloud infrastructure.

Conclusion

As we navigate the complexities of multi-region deployments, Pulumi can help in simplifying infrastructure management. It is important to note that embracing the challenges of multi-region scaling is a strategic move for technology projects navigating the global landscape. With Pulumi stacks and ESC configurations, you can seamlessly manage deployments and configurations, ensuring adaptability and security.

Did you learn something new? We’d like to know. If you’d like to explore further or discuss your specific use case, you can schedule a consultation with a solutions engineer. You can also join our community to be a part of the continuous conversation, and we can’t wait to see what you build!

Going Beyond With Advanced Infrastructure as Code Use Cases

Joe Duffy — Tue, 21 Nov 2023 03:53:33 +0000

This is the third of a three-part series originally published on The New Stack. Read Part 1 and Part 2.

Engineers who modernize their Infrastructure as Code with Pulumi get two classes of benefits:

Infrastructure as Code to develop cloud infrastructure with code.
Pulumi Cloud, which tames cloud infrastructure management at scale.

We’ve covered a fair bit of the first above, but have yet to scratch the surface for the second.

Many Infrastructure as Code solutions require that you explicitly manage something typically referred to as “state.” This state is an artifact produced that keeps track of what you think your infrastructure looks like, as defined by your Infrastructure as Code, so that it can be easily compared to what your actual infrastructure looks like. This is how diffs and updates can be done. Every Infrastructure as Code tool stores this infrastructure state, which is really just metadata about all the cloud resources, properties and dependencies. However, how much the tool exposes you to it varies.

Managed services like AWS CloudFormation and Azure Resource Manager (ARM) abstract this state away behind a service. Other tools like Terraform require you to explicitly manage that state by hand, often storing it in an S3 bucket or equivalent on your chosen cloud. The managed service approach tends to be easier to use and also avoids common pitfalls including security ones (like accidentally committing secrets to your Terraform state file in plain text) as well as reliability ones (like a code merge gone wrong that takes out your infrastructure). Pulumi Cloud offers both options, but the default is to use it like a managed service.

Pulumi Cloud is available for unlimited free usage, has a free tier for teams and offers advanced editions for larger teams and enterprises. Pulumi’s SDK is fully open source and can be used without the service, but the cloud service is the easiest, most secure and most reliable way to use Pulumi Cloud.

Pulumi Cloud’s architecture allows your cloud credentials to remain on your client. This when combined with features like identity integration and secrets management, addresses many pressing security concerns, and can mean that using the service is even more secure than managing Infrastructure as Code state on your own.

The service enables integration with your chosen identity provider — such as GitHub, GitLab, Atlassian, Okta, Azure ActiveDirectory or any SAML/SSO provider — and provides a complete visual history of who changed what infrastructure and when, very similar to what GitHub offers for code commits. It also makes it easier to see and control what’s going on in your team with organizations, role-based access controls and policy enforcement. It enables rich workflows via webhooks and gives users the ability to deploy either with the click of a button or straight out of a git repo.

You can get a better sense of what functionality is available in the Pulumi Cloud by reading this overview page. Although the Pulumi Cloud is trusted by many Global 2000 companies, is regularly audited by outside firms for well-architected and security best practices, and is SOC 2 Type II compliant, there is also a self-hosted edition available that can be run anywhere.

Testing Your Infrastructure as Code

A benefit of using general-purpose languages is that we get to test our code. Most of us wouldn’t think of writing application code without testing it, and Infrastructure as Code is no different. It’s not just that we can test it, but we can use standard tools and practices.

Testing can come in many different forms, including unit tests, property tests and integration tests. The following table summarizes the differences between the three approaches:

	Unit Tests	Property Tests	Integration Tests
Provision real infrastructure	No	Yes	Yes
Require the Pulumi CLI	No	Yes	Yes
Time to execute	Milliseconds	Seconds	Minutes
Language	Same as Pulumi program	Node.js or Python	Any language
Validation target	Resource inputs	Resource inputs and outputs	External endpoints

First is basic unit testing which, thanks to using regular programming languages, can be done in the usual way. For instance, the built-in Python and Go test frameworks “just work,” and as do popular third-party frameworks like Mocha (for JavaScript), NUnit (for .NET), JUnit (for Java), and so on. This can be good at ensuring a function that computes CIDR blocks, for instance, is correct. Testing infrastructure itself is a bit more involved, simply because many properties for resources aren’t known a priori until a deployment is done. But when combined with Pulumi Cloud’s infrastructure mocking framework, it’s possible to test infrastructure topologies and properties too. Read more about testing.

Next are assertions. The idea behind this is to embed assertions within your actual program, so that unexpected situations are caught immediately. This can be associated with any program state, including the resources themselves. Read more about assertions.

Next up is integration testing. For Infrastructure as Code, the typical meaning of an integration test is to spin up a dedicated stack solely for purposes of testing, often referred to as an “ephemeral stack.” Because Pulumi Cloud makes it easy to create independent, isolated copies of your infrastructure, thanks to the projects and stack model, these workflows are surprisingly easy to achieve. This can then be integrated into your typical testing processes, whether those are run by hand or automated as part of your CI/CD process. It’s common to spin up an ephemeral stack whose lifetime is tied to a single pull request, for instance, to validate that the changes being proposed are correct: It is created (and tested) when the pull request is opened, updated (and tested) anytime a new push to it is made and destroyed when it’s closed. Read more about integration testing and ephemeral stack techniques.

Finally, lots of more exotic tests are possible. For instance, you can fuzz-test your Infrastructure as Code by adding randomization to ensure the infrastructure behaves correctly in the face of strange inputs. You can chaos test your infrastructure by disabling instances or scaling key services below their typical thresholds to understand how your application behaves. You can stress test your infrastructure by placing unreasonable demands and seeing what breaks. The possibilities here are endless, but by using Infrastructure as Code in a general-purpose language, you have a lot more tools than is typically available to you.

To learn more about testing Infrastructure as Code more generally, with specific code examples, please see the Automated Testing of Pulumi Programs user guide.

Enforcing Security, Compliance, Cost and More with Policy as Code

An adjacent approach to both Infrastructure as Code and testing is something called policy as code. Just like Infrastructure as Code, the idea is to encode definitions in a real code artifact. The difference is that those are policies that analyze and flag problems with your infrastructure. The benefits of expressing these in code, however, are very similar to Infrastructure as Code: You can version policies as your requirements change over time, it’s reviewable by the team, and it’s just more expressive.

These policies can cover literally anything that can be analyzed and checked. This commonly includes security (such as ensuring databases aren’t open to the internet), compliance (that only FIPS endpoints are used for government-critical infrastructure), cost (that a deployment about to be done doesn’t exceed a team’s budget), and best practices (that all resources are tagged according to the team’s conventions). The idea is similar to linters and static analysis, both of which just work with Infrastructure as Code written in general-purpose languages, just purpose-built for infrastructure analysis.

Just as Pulumi Cloud lets you use general-purpose languages for your Infrastructure as Code, it lets you use general-purpose languages for policy as code too. This benefits from the richness of those languages and enables advanced use cases like integrating with cost management or ticketing systems like ServiceNow to enforce IT policies. Pulumi Cloud also supports the Open Policy Agent (OPA) language Rego to make it easier to leverage existing policies.

If we violate a policy, we get an error (for mandatory checks) or warning (for advisory ones) before the mistake even makes its way into production:

This might be “boring” from an engineer’s point of view but tends to be very important for an organization, especially for folks who are building out a platform team.

A benefit of using the Pulumi Cloud is that you can manage and enforce policies across the entire organization without individual engineers needing to even know how to run them. To learn more about policy as code, refer to the user guide for CrossGuard, Pulumi’s policy as code.

Deploying Your Infrastructure as Code Continuously

Most engineering teams ship their applications continuously thanks to CI/CD systems like GitHub Actions, Atlassian BitBucket, GitLab Pipelines, CircleCI, Travis, TeamCity, Spinnaker, or one of the built-in cloud CI/CD services like AWS CodePipeline, Azure DevOps or Google CloudBuild. This unlocks powerful git-driven workflows to push code out into production without needing to deploy things manually. But many teams don’t yet do that for infrastructure.

Adopting Infrastructure as Code helps you to do two things. First, you can verify and ship infrastructure changes the same way you ship application changes, by pushing and reviewing code, a practice we’ve already reviewed as being a core strength of Infrastructure as Code. Second, you can harmonize your application and infrastructure delivery pipelines, so they aren’t as disparate and disconnected. This helps to ship faster while scaling up to many more complex environments as your software sees more success.

The shift to containers, serverless and managed services has blurred the line between “application code” and “infrastructure code.” Is a serverless function on the infrastructure side of the line or application? What about a container registry? A pub/sub-topic? By using entirely different tools and delivery pipelines for applications versus infrastructure, which has historically been true of most infrastructure tools, we introduce a false dichotomy that slows down the team.

Pulumi Cloud offers two solutions for automating infrastructure delivery: First, any of the above CI/CD systems can be used to not just deliver application code, but also your Infrastructure as Code too. Second, Pulumi Cloud offers a built-in system called “Pulumi Deployments” to get up and running and trigger deployments based on git workflows. Read more in the Pulumi Deployments and/or Continuous Delivery user guides.

Advanced IaC Automation with Automation API

What if Infrastructure as Code were a normal library, in your language of choice, rather than a CLI? That was the question we asked before creating the automation API. Why is that an interesting question? It’s interesting because it elevates Infrastructure as Code to a core capability of any larger piece of software, which can be automated, rather than just a tool that needs to be run by hand by humans. We saw above that a small step toward this is to put Pulumi Cloud into your CI/CD, but the automation API takes it to a whole new level.

The scenarios this unlocks are mentioned earlier — custom tools and libraries, internal infrastructure provisioning portals, and even entire SaaS products — all of which need Infrastructure as Code as a central capability. Indeed, some of your favorite SaaS products, and even those that can be managed with Pulumi, use Pulumi on the backend to provision and manage infrastructure across clouds and across clusters.

Interestingly, we have found that, on average, those who leverage the automation API versus traditional Infrastructure as Code use cases are able to manage more than 10 times the infrastructure per engineer. That’s a big deal and the goal of automation: Doing more with less, experiencing less manual toil and just shipping faster with confidence.

Pulumi’s approach of using general-purpose languages made automation API a natural and unique extension of Infrastructure as Code into new realms. To learn more about Automation API, check out this page, which describes some common scenarios with complete examples.

Detecting IaC Drift and Doing Something about It

A problematic scenario with any Infrastructure as Code tool is something called “drift”: this happens when someone manually changes your infrastructure outside of the Infrastructure as Code tool. In that case, the actual infrastructure has “drifted” from what it was supposed to be, given your Infrastructure as Code definitions. Detecting drift is one thing, remediating it is another.

As a concrete example, imagine an engineer on the team is debugging an issue in production. To do so, they crack open port 22 so they can SSH into a Bastion host. (There are probably better approaches but as we know, when there’s a live incident, speed matters and sometimes shortcuts are taken.) Unfortunately, after resolving the incident, the engineer has forgotten to remove port 22 access. This is a potential security risk that we want to catch and resolve.

The two potential remediations to drift are typically:

Reapply the Infrastructure as Code, thereby eliminating the drift, and in this case, closing port 22 access
Incorporate the infrastructure changes into our program, so that the drifted state becomes our last-known-good on a go-forward basis. It’s also sometimes interesting to compare different environments to see if the infrastructure within them has drifted, although thanks to the Pulumi model of projects and stacks, this is uncommon in practice. It’s more of a concern with per-stack configuration, for which the Pulumi Cloud offers features to detect and resolve.

The Pulumi Cloud refresh command can be used to detect drift and then you can decide how to remediate it. All of the building blocks are there to either reapply changes, ingest them back into the program or some combination thereof. The combination of refresh, plus automation API, as well as the Pulumi Deployments capability, enables you to detect and remediate drift on demand or on a scheduled basis. Read more about drift detection and remediation or watch the video.

Learn More about Infrastructure as Code

The Pulumi Getting Started guide is the easiest way to get up and running with Infrastructure as Code. In addition to that, plus exploring the Pulumi Templates and Examples, we recommend some additional resources:

The tutorial series, Infrastructure as Code Recommended Practices, is a great overview of many concerns you will encounter along your Infrastructure as Code journey. This ranges from how to organize your project code and stacks, how to do local development and testing of your code, how to evolve these structures as your needs change, automation scenarios and more.

The Using Pulumi page is a collection of tutorials and getting started guides on specific infrastructure as code topics, such as deeper dives into fundamental concepts, how to simplify your infrastructure projects, and more.

We hope this article has given you a good grasp of the core Infrastructure as Code concepts you’re likely to encounter, no matter the tool you choose, as well as demonstrated some of the unique benefits Pulumi Cloud offers by using general-purpose and popular programming languages.

IaC Best Practices: Using Automation API

Scott Lowe — Wed, 26 Jul 2023 00:00:00 +0000

Welcome to the sixth post in our series of blog posts focused on Infrastructure as Code (IaC) best practices. So far in this series, you’ve seen how Zephyr Archaeotech Emporium—the fictional company at the center of this series—uses Pulumi to manage their online retail store. You read how Zephyr’s initial use of Pulumi changed to use short-lived per-developer stacks. Later, as Zephyr continued to grow, you saw how Zephyr restructured their Pulumi projects and stacks, incorporated Stack References, and used Pulumi Cloud’s role-based access control (RBAC) functionality to control access to their stacks. This post focuses on how Zephyr takes advantage of the Pulumi Automation API to bring an even greater level of orchestration to the stacks that represent their online store.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. These practices have unfolded organically, as a direct response to Zephyr’s evolving needs. The aim is to demonstrate that best practices are not set in stone, but are rather “point in time” recommendations that adapt as your company grows.

For ease of navigation, here are links to all the posts in the series:

IaC Best Practices: Understanding Code Organization and Stacks
IaC Best Practices: Enabling Developer Stacks and Git Branches
IaC Best Practices: Structuring Pulumi Projects
IaC Best Practices: Applying Stack References
IaC Best Practices: Implementing RBAC and Security
IaC Best Practices: Using Automation API (you are here)
IaC Best Practices: Summarizing Key Learnings

Managing Complexity in a Multi-Project Pulumi Setup

Over the course of this series, you’ve seen how Zephyr’s use of Pulumi has grown and changed. For the sake of completeness, here’s a quick recap:

Zephyr started with a single project in a single repository with two stacks, one for production and one for testing. (Details are available in understanding code organization and stacks post.)
To unlock developer velocity, Zephyr quickly added per-developer stacks. (You can read about per-developer stacks post in the series.)
As Zephyr grew and expanded, their single Pulumi project grew into three different Pulumi projects: one for base infrastructure, one for their Kubernetes platform, and one for the online store application. (The structuring Pulumi projects post describes the reasoning for adopting multiple projects.)
Accompanying the switch to multiple projects, Zephyr implemented stack references (to pass information between stacks in different projects) and applied role-based access control (RBAC) to the stacks in Pulumi Cloud. (Refer back to the stack references and RBAC & security posts in the series for more details on each of these areas.)

To see the state of Zephyr’s Pulumi implementation before the changes described in this article, please look at the blog/multi-project branch in each of the GitHub repositories (zephyr-infra, zephyr-k8s, and zephyr-app).

Since we last checked in on Zephyr, one significant change has emerged. As traffic to the Zephyr online store continued to increase, Zephyr’s operations team was finding it difficult to scale the relational databases running on Kubernetes (specifically, the orders and catalog databases). After evaluating their options, the team decided to migrate to Amazon Aurora with MySQL compatibility. Using the considerations outlined for structuring Pulumi projects, they decided to create a fourth Pulumi project, named “zephyr-data”, to manage the data infrastructure for the online store. Initially, this data infrastructure project manages only the Aurora clusters and database instances for the orders and catalog databases.

Because Zephyr was already using a modular approach to their overall application stack, inserting a new project for managing the online store’s data layer involved only minimal changes:

A new Pulumi program had to be written. This new program used stack references to leverage the existing infrastructure created by the base infrastructure stack and allowed traffic to the databases from the security group created by the Kubernetes platform stack. You’ll find this new program in the zephyr-data repository on GitHub (look in the blog/automation-api branch).
The Pulumi program for the Kubernetes platform stack only needed to export the security group created for the cluster nodes.
The Pulumi program for deploying the online store microservices referenced the database endpoints created by the new data infrastructure stack, and no longer spun up containerized instances of MariaDB for the orders and catalog databases.

The resulting four project architecture looks like this:

To see the state of the code after the changes described above—the addition of the zephyr-data project and the changes to use Amazon Aurora—please review the blog/automation-api branch in each of the Zephyr repositories.

Using a modular approach with multiple projects has its benefits. However, one of the trade-offs of splitting your IaC code into separate Pulumi projects is some additional relative complexity. With additional complexity comes additional room for error. To stand up a production-like environment, developers now had to stand up four different stacks (base infrastructure, Kubernetes platform, data infrastructure, and online store) in a specific order. If a user gets the order wrong, then the online store won’t come up properly.

With only four projects, this is not an insurmountable challenge by any stretch. However, what if it was 10 projects? Or 15 projects? Those numbers are not out of reach! Right now Zephyr has a single project to deploy all the various services involved for the online store. What if the application team decides to split up that project so that individual services can be deployed independently of each other? (After all, being able to deploy microservices independently of one another is one of the benefits of such an architecture.)

The Zephyr team started asking some questions: How can we make this process simpler? How can we remove room for error? How can we prepare our teams for greater scale?

Simplifying Advanced Deployments with Automation API

The challenges facing the Zephyr teams are not unique to Zephyr. Many IaC vendors recommend using a modular approach. Unfortunately, most IaC products rely on less-than-ideal approaches—like Bash scripts or Makefiles—to “stitch” together the different modules.

In some cases, organizations can use existing build automation tools and CI/CD pipelines to assist. What’s really needed, though, is a way to programmatically control the IaC tooling, and that’s what Pulumi’s Automation API is. The Pulumi Automation API allows Zephyr’s teams to use a general purpose programming language to automate Pulumi operations like performing an update, refreshing the stack state, or destroying a stack. Like Pulumi itself, the Pulumi Automation API supports multiple languages and supports multiple state backends, including Pulumi Cloud. Further, there is no requirement that Pulumi Automation API programs are written in the same language as the Pulumi programs they automate. In the case of Zephyr, which thus far has chosen to use TypeScript for their Pulumi programs, this means they are free to use any supported language for their Pulumi Automation API programs.

When the Zephyr team started evaluating options for simplifying the deployment of their online store and associated infrastructure, they decided to move forward with the Pulumi Automation API for these reasons:

Tight integration with Pulumi and Pulumi Cloud
Support for Pulumi Deployments (something we’ll examine in the next blog post)
Support for a broad range of programming languages, just like Pulumi itself
No requirement to use the same language as the Pulumi programs being automated (in fact, a Pulumi Automation API program could automate multiple programs written in different supported languages!)

Let’s take a look at Zephyr’s initial foray into using Pulumi Automation API.

Performing Stack Operations Efficiently

You can view the Pulumi Automation API program that Zephyr wrote in the local-source directory of the pulumi/zephyr-automation repository on GitHub, in the blog/automation-api branch.

Zephyr’s team decided to write their Pulumi Automation API program in Golang. One of the first things you’ll note about the Pulumi Automation API program is that it looks more like a “regular” Go program than a typical Pulumi Go program.

For example, a typical Pulumi Automation API program written in Go starts out like any other Go program:

func main() {
 // All your Go code is here
}

In a typical Pulumi Go program, on the other hand, the main() function starts off with a pulumi.Run call, like this:

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // All the Pulumi Go code is here
 })
}

The easiest way to understand this difference is to consider the way each program is invoked. A Pulumi Automation API program written in Go is executed by running go run main.go, like running any other Go program. In contrast, a Pulumi Go program is invoked by the pulumi CLI.

Zephyr’s Automation API program starts out with checking for a command-line argument to indicate the desired mode of the operation—should it be creating/updating resources, or destroying resources? In a standard Pulumi Go program, this would be determined by the invocation of the pulumi CLI (for example, whether you ran pulumi up or pulumi destroy). Here, though, that falls to the user to check:

// Determine mode of operation; default is to refresh/update
destroy := false
argsWithoutProg := os.Args[1:]
if len(argsWithoutProg) > 0 {
 if argsWithoutProg[0] == "destroy" {
 destroy = true
 }
}

Next, the program reads some configuration data from a YAML configuration file; this configuration data includes things like the relative file system location of the Pulumi programs that this Automation API program is going to execute:

// Get configuration data
data, err := os.ReadFile("config.yaml")
if err != nil {
 fmt.Printf("Error reading configuration file: %v\n", err)
 os.Exit(1)
}
var env Environment
err = yaml.Unmarshal(data, &env)
if err != nil {
 fmt.Printf("Error parsing configuration information: %v\n", err)
 os.Exit(1)
}

Separating configuration from code in this manner allows you to use the language’s existing ability to create binary executables that can be easily distributed to your teams. For example, Zephyr has the option of running go build to create a binary executable for distribution to internal teams.

From there, the Automation API program gets into setting up the stacks it will use:

// Set up base stack
baseStackName := auto.FullyQualifiedStackName(env.Organization, env.BaseProject.Name, env.StackName)
baseStack, err := auto.UpsertStackLocalSource(ctx, baseStackName, env.BaseProject.Location)
if err != nil {
 fmt.Printf("Failed to create or select %s stack: %v\n", env.BaseProject.Nickname, err)
 os.Exit(1)
}
fmt.Printf("Successfully created/selected %s stack\n", env.BaseProject.Nickname)

In this particular program, Zephyr decided to start with a local workspace. (You can read more about workspaces in the Pulumi Automation API docs.) This is reflected in the use of the UpsertStackLocalSource method shown above, which will create or select a stack for a Pulumi program found in a local filesystem location.

Users can also use UpsertStackInlineSource, which embeds the Pulumi program into the Automation API program as a function. There are trade-offs to each approach:

Using local source (i.e., the Pulumi program is stored outside of the Automation API program) has the benefit of allowing users to independently revise the Pulumi program or the Automation API program; this may be the ideal approach where organizations have separate teams that will create, manage, or maintain the Automation API programs. In Zephyr’s case, where there is a platform team, this made a lot of sense for them. This does also mean, as you might guess, that the Pulumi programs must be reachable by the Automation API program (this typically means users need to clone down the appropriate repositories to their system first).
Using inline source (where the Pulumi program is embedded in the Automation API program) may be a better fit when there is a single team managing or maintaining both the Pulumi program(s) as well as the Automation API program(s), or when—for whatever reason—these programs should not be revised independently of one another. This has the benefit of removing the need for users to clone or copy separate Pulumi programs onto their system before being able to use the Automation API program.

After all four stacks—base, platform, data, and app—have been set up for use within the Pulumi Automation API program, then comes the “real work”: performing operations on those stacks. To reduce duplicate code, the Zephyr program makes use of three functions:

The refreshStack function (starting on line 199 of main.go) performs the equivalent of a pulumi refresh against a stack.
The updateStack function (starting on line 218 of main.go) performs an update against the stack. This is like running pulumi up against a stack.
Finally, the deleteStack function (starting on line 236 of main.go) destroys the resources in a stack. This equates to running pulumi destroy against a stack.

Using these functions, lines 94 through 197 perform the key task of taking action to stand up or tear down the Zephyr online store and all associated infrastructure. If the user passes the parameter “destroy” on the command line (meaning the Pulumi Automation API program is invoked using go run main.go destroy), then it will tear down the stacks. Otherwise, the Pulumi Automation API program will perform a refresh and then an update—much the same way that the pulumi CLI itself behaves.

Along the way, you can see the Pulumi Automation API program passing configuration values to the stacks being orchestrated (this is lines 177 through 182, for example):

// Set config values for app stack
appStack.SetConfig(ctx, "platformOrgName", auto.ConfigValue{Value: env.Organization, Secret: false})
appStack.SetConfig(ctx, "dataOrgName", auto.ConfigValue{Value: env.Organization, Secret: false})
appStack.SetConfig(ctx, "platformProjName", auto.ConfigValue{Value: env.PlatformProject.Name, Secret: false})
appStack.SetConfig(ctx, "dataProjName", auto.ConfigValue{Value: env.DataProject.Name, Secret: false})
appStack.SetConfig(ctx, "platformStackName", auto.ConfigValue{Value: env.StackName, Secret: false})
appStack.SetConfig(ctx, "dataStackName", auto.ConfigValue{Value: env.StackName, Secret: false})

Among other things, this means you can take your existing Pulumi programs, and—without making any changes—orchestrate them using an Automation API program! Further, in reviewing the code shown above, you can see that the Pulumi Automation API ties into Pulumi’s secrets functionality.

In the end, using Pulumi’s Automation API enabled Zephyr’s platform team to simplify the workflow for standing up the online store application and all of its associated infrastructure into a simple and straightforward process:

Developers ensure they have cloned the appropriate source repositories onto their system.
Developers run go run main.go (or go run main.go destroy) to stand up or tear down the online store and all associated infrastructure.

Although Zephyr uses multiple GitHub repositories, the Pulumi Automation API has no requirements for multiple repositories. You could use a monorepo, if that better suits your particular requirements.

Summarizing Best Practices: Automation API

This post discussed the following best practices when using Pulumi Automation API:

Use the Automation API to orchestrate operations on multiple stacks. In cases where you’ve split Pulumi projects (per the recommendations in the post on structuring Pulumi projects), you’ve introduced a potential opportunity for using Automation API to orchestrate operations. This is a great counter-balance to offset the relative increase in complexity that comes from breaking large Pulumi projects into smaller, more focused Pulumi projects.
Separate configuration from code. This is already a generally-accepted recommendation in most programming circles, and it applies to Automation API programs as well. Doing so enables you, if you wish, to distribute binary artifacts to teams instead of requiring them to execute the source code of your Automation API programs.
Use local source to decouple Automation API programs from Pulumi programs when separate teams are involved. When separate teams are creating, managing, or maintaining the Automation API programs and the Pulumi programs, using local source instead of inline source enables the teams to operate somewhat independently.
When there is a single team managing both Automation API programs and Pulumi programs, use inline source. This is also true when, perhaps due to compliance or regulatory concerns, the Automation API program should be more tightly coupled to the Pulumi programs it is automating.

The Zephyr series wraps up in the next post, where we summarize all our recommendations so far. We’ll also take a look at areas of potential future growth for the team at Zephyr. Don’t miss it!

IaC Best Practices: Implementing RBAC and Security

Scott Lowe — Tue, 23 May 2023 00:00:00 +0000

This post continues our series of blog posts focused on IaC best practices. In earlier posts, we introduced Zephyr Archaeotech Emporium, the fictional company that sits at the center of this series, and discussed Zephyr’s primary use case for Pulumi: managing their online retail store. You read how Zephyr’s initial use of Pulumi changed to incorporate the use of short-lived per-developer stacks. Later, as Zephyr continued to grow, you saw how Zephyr restructured their Pulumi projects and stacks, and incorporated the use of Stack References. This post is a complement to the post on structuring Pulumi projects, concentrating on the use of role-based access control (RBAC) and security in Zephyr’s multi-project configuration.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. You’ve seen these practices emerge over the course of the series—not all immediately, and not all right away. Structuring the blog series in this way is a deliberate decision; many best practices are “point in time” recommendations: they are context-dependent and the recommendations for any given company, like Zephyr, may change as the company and its needs change.

For ease of navigation, here are links to all the blog posts in the series:

IaC Best Practices: Understanding Code Organization and Stacks
IaC Best Practices: Enabling Developer Stacks and Git Branches
IaC Best Practices: Structuring Pulumi Projects
IaC Best Practices: Applying Stack References
IaC Best Practices: Implementing RBAC and Security (the post you’re reading)
IaC Best Practices: Using Automation API
IaC Best Practices: Summarizing Key Learnings

Evolving Project Structure to Enhance Security and Scalability

So far, you’ve read about how Zephyr started out with a single project and a single Git repository. Adding short-lived per-developer stacks, as described in the post on best practices for developer stacks and Git branches, didn’t really impact this structure. However, as the company grew, Zephyr’s project and Git repository structure evolved into its current form:

The “zephyr-infra” project handles base infrastructure (VPC and related constructs).
The “zephyr-k8s” project automates the Kubernetes platform layer, building atop the base infrastructure components supplied by the “zephyr-infra” project.
Finally, the “zephyr-app” project handles the deployment of the online retail store application onto Kubernetes, layering on top of the platform provided by the “zephyr-k8s” project.

You can review the reasons for this structure in the post on best practices for structuring Pulumi projects. One of the reasons listed in the third Zephyr blog post was security:

…security is a factor that affects how projects and stacks should be structured. In Zephyr’s case, the ability for one team to affect another team’s resources was present when everything was a single project. With multiple projects, Zephyr can use stack permissions and role-based access control (RBAC) in Pulumi Cloud to appropriately control who can affect what.

The last part of that quote—the part about being able to “appropriately control who can affect what”—is another way of describing what’s known as the principle of least privilege. If you’re unfamiliar with this term, we recommend following the link and getting more information, because the principle of least privilege stands at the core of all the recommended practices you’ll read in this post.

Pulumi supports a number of self-managed backends, including Amazon S3 (and S3-compatible servers like Minio or Ceph), Azure Blob Storage, and Google Cloud Storage. The RBAC functionality described in this post applies only to the managed Pulumi Cloud backend.

Understanding the RBAC Building Blocks in Pulumi Cloud

To understand how to secure your Pulumi projects and stacks in Pulumi Cloud, it is important to first explain the basic building blocks you’ll use:

Organization: An organization represents a customer/company within Pulumi Cloud. Generally speaking, a single company like Zephyr is represented by a single organization.
Member: Each member represents a user account that is part of an organization. In Zephyr’s case, each Zephyr employee that needs to interact with Pulumi is a member of Zephyr’s organization in Pulumi Cloud.
Team: A team is a grouping of members, and is the primary component involved in assigning per-stack permissions in the Pulumi Cloud RBAC model. How you structure teams within your organization will depend on a great many different factors we’ll explore in this post.
Stack: A stack is a separate instance of the resources created by a Pulumi program within a project. Each stack has its own independent state and its own configuration values. With respect to RBAC, each stack is a separate entity to which one or more teams can be granted permission. In Zephyr’s case, there is a long-lived production stack for each of the separate projects, and a persistent (or long-lived) test stack for each of the separate projects. There are also the per-developer stacks, which are typically short-lived.

This diagram, also used in the post on best practices for stack references, shows the stacks and projects for Zephyr’s online store:

Now, how are these different components assembled together? At its most basic level, you would use teams to group members together, and then assign stack permissions to the team. This diagram helps illustrate the idea:

A couple of important points emerge from the above diagram:

The relationship between stacks and teams is many-to-many. That is, any given stack can have many teams assigned permissions, and any given team may be assigned permissions to many stacks.
Similarly, the relationship between teams and members is also many-to-many (any given team can—and almost always will—have multiple members, and any given member can be part of multiple teams at the same time).

Assembling Security for Stacks

The first, and perhaps the most important, best practice with regards to securing your stacks is to actually use Pulumi Cloud’s RBAC functionality. While this may seem obvious, there are practical implications to this statement that are useful to explore.

Pulumi Cloud has the ability to set a default stack permission for all members of an organization. In the screenshot below, you can see that the default permission has been set to Write:

In this configuration, you are effectively disabling Pulumi Cloud’s RBAC functionality. Permissions in Pulumi Cloud are additive; that is, permissions only state what you can do, not what you can’t do. Therefore, if the default stack permission is set to Write, then assigning a team read-only permissions to a stack has no effect—because members of that team can read from the stack (by virtue of their inclusion in a team) and can write to the stack (by virtue of the default stack permission). The team members’ read-only permission is added to the default stack permission for their effective permission.

For this reason, Pulumi recommends setting the default stack permission to Read or None. Setting it to Read means that all members of the organization can see and read from stacks that aren’t their own, but can’t change or modify the others stacks. Setting it to None means that members have permissions to their own stacks by default—gaining any form of access to stacks other than their own requires being added to a team that has permissions to those stacks.

When it comes to assigning a team access to a stack, there are three options:

Stack reader (read-only access; also called “Stack reader access” in some areas of the Pulumi Cloud UI)
Stack editor (read/write access, but can’t delete the stack; also called “Stack write access” in the Pulumi Cloud UI)
Stack admin (full access; you may also see it called “Stack admin access”)

When the “Allow organization members to create stacks and transfer stacks to this organization” is checked, like in the screenshot above, then the user (organization member) that creates a stack automatically becomes the stack admin for that stack. The user’s ability to delete the stack, however, is affected by the “Allow stack admins to delete stacks” setting.

Full details on what each of these permissions grants is detailed in the projects and stacks documentation (see the “Stack Permissions” section). In accordance with the principle of least privilege, Pulumi recommends you grant the minimum level of access necessary in order for a user (organization member) to meet their job responsibilities. You accomplish this by adding the member to a team, and then granting the team the minimum level of necessary access. You can’t assign stack permissions directly to users.

Because team membership is central to the use of stack permissions, Pulumi has several ways to make it easier to manage the relationship between teams and stacks:

You can use the Pulumi Cloud user interface to assign a team access to a stack.
You can use Pulumi Cloud’s REST API to control team access to stacks.
You can use the Pulumi Cloud provider for Pulumi to programmatically manage team access to a stack using Pulumi itself.
You can use the --teams flag to pulumi stack init to assign stack editor access to one or more teams when you create the stack.

In general, Pulumi recommends declaratively managing your Pulumi Cloud RBAC configuration with the Pulumi Cloud provider (formerly known as the Pulumi Service provider), unless you have specific requirements that would prevent it. However, be aware that the Pulumi Cloud provider doesn’t allow you to change a member’s role in an organization (every member is either a “member” or an “admin” within an organization); that would have to be done via the Pulumi Cloud user interface or REST API.

Pulumi also recommends using the --teams flag with pulumi stack init to ensure that new stacks are properly assigned team permissions upon creation. You can then go back and fine-tune those team permissions—which are set to “Stack editor” or “Stack write access” when you use the --teams flag—using a Pulumi program with the Pulumi Cloud provider. Alternately, teams can use a team access token when creating a new stack, and that will also grant the team permission to the newly-created stack.

Finally, Pulumi recommends allowing users to create their own stacks. Otherwise, only an organization admin would be allowed to create new stacks, and this is likely to cause a bottleneck in provisioning cloud infrastructure—one of the very issues that infrastructure as code tools like Pulumi are attempting to solve! (It may be necessary in some cases to restrict this functionality, such as in certain highly regulated industries or verticals.) Preventing users from creating their own stacks would also prevent the use of per-developer stacks like Zephyr uses, which as you’ve seen can be enormously helpful in accelerating development and deployment velocity.

Inspecting RBAC Implementation

Zephyr’s RBAC implementation in Pulumi Cloud is represented in part by a diagram that you also saw earlier in this post:

Zephyr set the organization default stack permission to Read. This enabled teams to read all stacks, and enabled the continued use of stack references without any additional configuration. Teams were created to grant “Stack editor” (also called “Stack write access”) to stacks based on the overall organizational structure of the company. Thus, the Platform team gained access to the non-production zephyr-infra and zephyr-k8s stacks, while the Application team was given access to the non-production zephyr-app stacks. However, only members of the Production team were given access to the production stacks in all three projects, ensuring that access to update those stacks—and thus the actual resources running Zephyr’s online store—remained relatively controlled.

Summarizing Best Practices: RBAC & Security

The post covered the following guidelines for securing your stacks in Pulumi Cloud:

Use Pulumi Cloud’s RBAC functionality by setting your organization’s default stack permissions to Read or None. This is generally the safest way to implement the principle of least privilege, unless you have a good use case for allowing all organization members full write access to all stacks.
When assigning team permissions, grant the minimum level of access necessary. If a user isn’t responsible for making updates to a stack, then give that user stack reader permission through a team membership (remember that you can’t assign stack permissions directly to organization members, you have to use a team). Keep in mind, though: if the organization level stack permission has been set to Read, then members already have the equivalent of stack reader permissions on all stacks, so creating a team to grant stack reader permission is unnecessary and superfluous.
Declaratively manage your Pulumi Cloud RBAC configuration where possible using the Pulumi Cloud provider from within a Pulumi program.
Use the --teams flag with pulumi stack init to ensure that teams have stack permissions upon stack creation. You can also use a team access token to accomplish the same result.
Allow users to create their own stacks. Generally, turning this functionality off (found in the Settings > Access Management screen of Pulumi Cloud) will end up creating a bottleneck, as only organization admins would be allowed to create new stacks.

In the next IaC Best Practices post, the focus is on Automation API. Stay tuned to see how Zephyr uses Automation API to further streamline and automate the deployment of their cloud resources!

IaC Best Practices: Applying Stack References

Scott Lowe — Fri, 31 Mar 2023 00:00:00 +0000

This is the fourth post in a series of blog posts focused on Zephyr Archaeotech Emporium—our fictional company—and their use of Pulumi to manage their online retail store. In the first three posts, you saw how Zephyr’s initial use of Pulumi changed as the company grew, and how the use of short-lived per-developer stacks helped Zephyr’s application development team meet the demands of a fast-growing company. This post is a complement to the earlier post on structuring Pulumi projects, discussing how Zephyr uses Stack References to link their projects together and sharing some IaC best practices around the use of Stack References.

As you may have read in earlier Zephyr posts, the ultimate goal of the Zephyr series is to share best practices on the use of Pulumi to manage your infrastructure and application resources (using a fictional company and a somewhat complex containerized application as the use case). The series exposes those practices over time—not all right away, and not without also discussing the context for the recommendations. This is deliberate, demonstrating how “point-in-time” recommendations change based on the needs of the company and its requirements.

Here are links to all the blog posts in the series:

IaC Best Practices: Understanding Code Organization and Stacks
IaC Best Practices: Enabling Developer Stacks and Git Branches
IaC Best Practices: Structuring Pulumi Projects
IaC Best Practices: Applying Stack References (you are here)
IaC Best Practices: Implementing RBAC and Security
IaC Best Practices: Using Automation API
IaC Best Practices: Summarizing Key Learnings

Reviewing Zephyr’s Current Status

This post continues the previous post in the series. Zephyr’s infrastructure footprint and Pulumi codebase is the same; that is, Zephyr now has three Pulumi projects (zephyr-infra, zephyr-k8s, and zephyr-app). Each of these projects handles a different aspect of Zephyr’s online store:

The “zephyr-infra” project handles base infrastructure (VPC and related constructs).
The “zephyr-k8s” project handles the Kubernetes platform layer, building atop the infrastructure components provided by the “zephyr-infra” project.
The “zephyr-app” project handles the deployment of the online store services onto Kubernetes, layering on top of the platform provided by the “zephyr-k8s” project.

The reasons for this structure are explained in structuring Pulumi projects post. For Zephyr, it came down to supporting multiple teams and allowing the application to evolve independently of the infrastructure that supports it.

You can continue to use the blog/multi-project branch in the GitHub repositories to see the state of Zephyr’s code and projects for this blog post.

It’s clear that there are dependencies across projects: The zephyr-k8s project needs to know the VPC ID and the subnet IDs from the zephyr-infra project. Similarly, the zephyr-app project needs to have access to the Kubernetes cluster details from the zephyr-k8s project. As the Zephyr team went about building the code that’s now in use in their multi-project architecture, they needed to decide how best to handle these cross-project dependencies. The Zephyr team knew that hard-coding output values from one project as configuration values in another stack wasn’t the ideal way (and in fact Pulumi strongly recommends against hard-coding output values), but what was the best solution to use? The answer to that question is stack references.

Linking Stacks with Stack References

Stack references allow you to access output values from one stack in another stack. In other words, stack references offer a way to programmatically “link” stacks in different projects for the purpose of sharing information. You can find more details in the documentation on stack references.

Using stack references, the Zephyr team can make the necessary connections they need:

The code handling the Kubernetes cluster can reference the VPC ID and subnet IDs from the base infrastructure layer.
Similarly, the code that is responsible for deploying the online store services can reference the necessary Kubernetes connection details (the Kubeconfig) for the Kubernetes cluster.

You can see how the Zephyr team did this for their projects, including code samples, toward the end of this post (in the section titled “Examining Zephyr’s use of stack references”). This graphical representation also illustrates Zephyr’s current repository and project structure as well as the stack references:

It’s worthwhile to note that any information that needs to be accessible from another stack via a stack reference must be exported as a stack output in the source stack. If you don’t mark it as a stack output, then it can’t be used in a stack reference. Adding stack outputs after the fact requires little effort and has no impact on existing infrastructure.

Also, any value retrieved via a stack reference is treated as a Pulumi Output, and therefore may require some extra work to transform values (such as the use of Output.apply). The recent addition of OutputDetails support in Pulumi—you can read more about OutputDetails in this blog post announcing the functionality—helps considerably in this situation. Some SDKs also have language-specific mechanisms that can help; for example, using Go’s .AsStringArrayOutput() method on a StackReference.GetOutput statement makes referencing subnet IDs from other project much easier.

While stack references are conceptually straightforward and not difficult to implement, there are some best practices to be mindful of regarding the use of stack outputs and stack references:

When it comes to stack outputs, export what is needed. If it needs to be accessed from outside the stack, export it; otherwise, don’t. Why? Stack outputs form the application programming interface (API) by which other Pulumi programs interact with this Pulumi program. Like in any API implementation, expose what the user cares about or is likely to need, and leave the rest as an implementation detail. You can add more stack outputs after the fact with only minimal code changes, and as has been mentioned already this is done with a quick pulumi up that has no affect on existing infrastructure (you’re modifying the stack object itself).
Be judicious in your use of stack references. This is sort of the “flip side” of the previous bullet. If you need to reference an output from another stack, then do so; otherwise, don’t. If you find yourself needing lots of stack references, then the way in which you’ve structured your projects may need to be re-examined so as to minimize the coupling between projects. Additionally, there is a potential “speed of light” concern here; if where you are executing your Pulumi code is across a high-latency link from where you store your Pulumi state, stack references will be affected by that latency.
If you need to expose lots of outputs on a stack, consider exporting a structured data object. It’s possible to construct a JSON object (or dict or struct, depending on your language) to hold all the stack outputs, and then export that object. Then a stack reference can read that object, resulting in a single call to the backend. Be aware, though, that you’ll need to write the necessary code to understand/import/unmarshall that JSON object in the referring stack, so there is a small amount of extra work required in this situation.
In line with parameterizing as much of your code as possible, also be sure to parameterize your stack references. A stack reference is built using an organization name, a project name, and a stack name. Don’t hardcode these values; instead, use configuration values to allow the users to specify from which source stack(s) the values will be referenced. You’ll see an example of that in the next section.
Plan for handling cascading stack updates. If a stack has outputs being consumed via stack references, what happens when those stack outputs change? The Pulumi Service supports webhooks that can help, and Pulumi is currently exploring options for having Pulumi Deployments help with this (see Pulumi Cloud Requests GitHub issue). In practice this should be uncommon, and frequent occurrences may be a good indicator that your resource partitioning (how resources are split among multiple projects) needs to be adjusted.

One thing not listed above is security—and that’s because a later post in the series focuses on this topic. You’ll want to keep the security of your stacks in mind when planning for the use of stack references. The teams managing the stacks where the stack references exist will need at least read-only access to the stacks where the outputs exist. (For more on this front, refer to the IaC Best Practices post on RBAC and security.)

Examining Use of Stack References

With these best practices in mind, you can examine the Zephyr team’s implementation to see how they put these recommendations into action.

In the code for the base infrastructure stack, you can see that Zephyr exported the essential values needed by the Kubernetes platform stack:

// Export some values for use elsewhere
export const vpcId = eksVpc.vpcId;
export const privSubnetIds = eksVpc.privateSubnetIds;
export const pubSubnetIds = eksVpc.publicSubnetIds;

In the Kubernetes platform stack, the Zephyr team parameterized the values needed for the stack reference. This particular approach, by the way, is key to preserving the per-developer stacks to which the Zephyr team has grown accustomed (each developer needs to specify the correct organization, project, and stack name):

// Grab some configuration values
// Some code omitted here
const infraOrgName = config.require("infraOrgName");
const infraProjName = config.require("infraProjName");
const infraStackName = config.require("infraStackName");

// Create a StackReference to get information from base stack
const infraSr = new pulumi.StackReference(`${infraOrgName}/${infraProjName}/${infraStackName}`);

As with the base infrastructure stack, only the key value needed by other stacks is exported (in this case, the Kubeconfig required to access this cluster):
```
// Export some values for use elsewhere
export const kubeconfig = eksCluster.kubeconfig;
```

Finally, in the application stack, the stack reference values are again parameterized, and the Kubeconfig—necessary for the application stack to deploy onto the provisioned Kubernetes cluster—is referenced via a stack reference:

// Grab some configuration values
const config = new pulumi.Config();
const k8sOrgName = config.require("k8sOrgName");
const k8sProjName = config.require("k8sProjName");
const k8sStackName = config.require("k8sStackName");

// Create a StackReference to get Kubeconfig from base stack
const kubeSr = new pulumi.StackReference(`${k8sOrgName}/${k8sProjName}/${k8sStackName}`);

All of the GitHub links in the paragraphs above reference the blog/multi-project branch in each repository, which has the code as of this blog post (and the earlier blog posts).

Summarizing Best Practices: Stack References

This post covered the following guidelines for working with Pulumi:

Export the minimum of what’s needed from your stacks. It’s not difficult to add stack outputs later, and you can do it with no impact to existing infrastructure.
Be judicious in the use of stack references. Reducing stack references where possible helps keep projects from becoming overly coupled together, much in the same way as minimizing stack outputs.
Use a structured data object for large numbers of values that need to be referenced. This enables Pulumi to retrieve all the data with a single call, but be aware you’ll need to write extra code to understand the data structure in the referring stack.
Parameterize your stack references. Don’t hardcode organization, project, or stack name values. Instead, pass these in as configuration values (with default values applied, where applicable).
Plan to build mechanisms to handle cascading stack updates. Using the Pulumi Service’s webhook functionality can be useful here. Frequent instances of cascading updates might be an indicator that your project structure needs to be revisited for optimal resource partitioning.

The next post in the IaC best practices series focuses on security and role-based access control (RBAC)—a topic you won’t want to miss! Before then, the next visit to the “Zephyr universe” will be to take a look at an oft-overlooked use case for Pulumi: local testing. Come see how to use Pulumi to automate local testing for development!

IaC Best Practices: Structuring Pulumi Projects

Christian Nunciato — Fri, 17 Mar 2023 00:00:00 +0000

This is the third post in a series of blog posts focused on Zephyr Archaeotech Emporium—our fictional company—and their use of Pulumi to manage their online retail store. In the first post, you saw how Zephyr initially decided to go with a single Pulumi project for managing deployments of their online retail store application. In this post, you’ll see how Zephyr’s use of Pulumi changes as their company grows and evolves.

The first post about organizing code and stacks specifically mentions that the series will show “how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.” While the ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application, it’s important to note that many best practices are “point in time” recommendations: best practices are context-dependent and the recommended practices for a given organization will change as the company and its infrastructure needs also change. This blog post is an example of that evolution in action: the practices that worked well for Zephyr while they were smaller and had smaller infrastructure and less teams working on that infrastructure are no longer ideal as the company has grown.

Here are links to all the blog posts in the series:

IaC Best Practices: Code Organization and Stacks
IaC Best Practices: Enabling Developer Stacks and Git Branches
IaC Best Practices: Structuring Pulumi Projects (this post)
IaC Best Practices: Applying Stack References
IaC Best Practices: Implementing RBAC and Security
IaC Best Practices: Using Automation API
IaC Best Practices: Summarizing Key Learnings

Growing Development Teams

As a result of launching its online presence, Zephyr saw amazing success. The market for arcane artifacts and novel curiosities is booming, and Zephyr is becoming a leader in this market. Funded by this success, Zephyr has grown rapidly. Along with this growth, Zephyr’s software engineering and IT teams also grew and expanded. Fortunately, Zephyr’s use of per-developer stacks with short-lived ephemeral Git branches positioned the development teams well for this growth.

For a look at the state of Zephyr’s Pulumi code as it stood as of the end of the second post in the series, see the blog/dev-stacks branch of the pulumi/zephyr-app repository on GitHub.

Naturally this growth has resulted in some separation of duties. The IT team now has responsibility for the infrastructure that supports the online retail store. Within that team, some team members are focused on “core” infrastructure, while other team members are focused only on Kubernetes (a natural decision given Zephyr’s use of Kubernetes in their architecture). The application team continues to remain responsible for the online store application itself.

This left Zephyr again asking some questions about the best path forward:

Should they use a single project, or multiple projects? Zephyr had used a single Pulumi project up until now, but growth and organizational change meant they wanted to make sure they were using Pulumi in a way that best reflected their needs. What are the advantages and disadvantages of each approach?
If multiple projects, how many projects? What’s the best way to determine how many projects are needed? What factors play a role in this decision?

Structuring Pulumi Projects

One project, or multiple projects? And if multiple projects, how many projects? These are the core questions the Zephyr needed to resolve. In the end, there are a number of key factors that determine the answers to these questions:

Use case: Zephyr has only a single application (so far!), but in situations where there are multiple applications then keeping those applications (and sometimes their associated infrastructure, depending on other factors) in separate Pulumi projects is generally recommended. In some ways, “use case” is an aggregation of other factors—a new application with a distinct set of resources that do not share a lifecycle with existing resources and are potentially managed by a different team.
Company/department/team structure: How a company/department/team is organized should certainly be taken into account when structuring your Pulumi projects. In Zephyr’s early days with Pulumi, a single team managed everything so a single project was fine. Now with multiple teams, Zephyr will likely find using multiple projects to be more effective.
Security: Closely related to the previous bullet, security is a factor that affects how projects and stacks should be structured. In Zephyr’s case, the ability for one team to affect another team’s resources was present when everything was a single project. With multiple projects, Zephyr can use stack permissions and role-based access control (RBAC) in Pulumi Cloud to appropriately control who can affect what. (The IaC Best Practices blog post on RBAC and security discusses this topic in more detail.)
The relationship between resources: The relationships between resources—such as which resources are dependent upon other resources—is another factor that feeds into your Pulumi project structure. A “shared infrastructure layer” is often best handled in a separate project of its own, but may pull in resources that might otherwise have been kept with an application in its own project. (In some respects, this is a “counter-balance” to the first bullet point.) On the other hand, resources that belong exclusively to a specific application or function might be best bundled in the same Pulumi project.
Resource lifecycle: Do resources have a shared lifecycle, meaning they are always created, updated, or deleted together? If so, then those resources should be placed in the same project. If the resources do have a shared lifecycle, then placing them in the same project simplifies lifecycle management. A great example of resources with a shared lifecycle would be a Kubernetes Deployment and the associated Kubernetes Service.
Change rate of resources: Some resources may change very infrequently. Consider the example of an AWS VPC and associated objects. Once it is created, it may not change for extended periods of time, if ever. On the other hand, some resources may change frequently (in other words, these resources have a high change rate). A fast-moving software development team may revise the Kubernetes Deployment regularly to pull in a new container image with updated code. In a single project, combining resources that have a high change rate with resources that have a low change rate may be suboptimal.
Blast radius: Finally, there is the consideration of what happens in the event of human error, and limiting the “blast radius.” With a single project, all the resources in that project are within the blast radius—all of them could potentially be affected in the event of human error. Using separate Pulumi projects can limit the blast radius. It’s not something that should drive the decision all by itself, but it is a factor to be considered.

Now that you have an idea of some of the factors that affect how users structure their Pulumi projects, what are the advantages and disadvantages of each approach?

For single projects, the key advantage is simplicity. There’s no denying that a single project is simpler and less complex than working with multiple projects. This advantage, though, is also the key disadvantage. A single project does not afford the flexibility to let different teams manage different aspects of the infrastructure, for example, nor does it offer the ability to split resources that don’t have a shared lifecycle. And, as mentioned earlier, blast radius is a concern.

For multiple projects, the key advantage is flexibility. It’s true that the added flexibility comes at the cost of some added complexity (a future post will explore one aspect of that added complexity in the form of StackReferences). However, you gain the ability to separate infrastructure resources according to team ownership/responsibility, relationship to other resources (i.e., grouping resources in a “shared infrastructure layer”), or any of the other dimensions mentioned above.

In some ways, the decision points for single vs. multiple projects are analogous to the decision points for monolith vs. microservice: independent scaling and flexibility versus simplicity.

In general, most customers end up using multiple Pulumi projects per use case. As you’ve seen, though, there are a number of factors that influence that decision and will affect how many projects end up in use.

Moving Forward with Projects

Zephyr had a number of factors in play, not the least of which was the organizational change resulting from their growth and success. In the end, Zephyr decided to restructure their use of Pulumi for the online retail store into three separate projects:

An infrastructure project: This project handled the AWS VPC and related infrastructure resources.
A platform project: This project managed the Kubernetes cluster(s).
An application project: This project handled the actual online retail store itself.

You can use a combination of code refactoring and pulumi import to assist in the process of restructuring Pulumi projects, but it some cases it may be easier to create new infrastructure and migrate over to it.

In the code organization and stacks post, you read about some of the options for where to store Pulumi code relative to the applications it supports. At that time, Zephyr had decided to go with what’s called a monorepo approach, meaning that the Pulumi code was stored in the same repository as the application code. In conjunction with the project reorganization, Zephyr also decided to change how their code was organized:

Code for the infrastructure project moved to a separate “zephyr-infra” repository
A new “zephyr-k8s” repository housed the code for the platform project
The code for the online retail store application itself remained in the same “zephyr-app” repository

Note that relocating code (to a different GitHub repository or different filesystem location within a repository) generally has no impact on Pulumi. As long as the project name and stack name do not change, then changing the filesystem location won’t affect anything—keeping in mind, of course, that filesystem paths and such may need to be adjusted in code to account for any such changes.

After restructuring their projects, Zephyr’s arrangement of Pulumi projects and stacks, along with their associated GitHub repositories, looks something like this:

These repositories are available for you to review on GitHub (zephyr-infra, zephyr-k8s, and zephyr-app); use the branch/tag selector to find the blog/multi-project branch in each repository to see the state of the code as of this blog post. Be aware that the process for standing up the Zephyr online store is now a bit more complex; check the instructions in each repository for more details if you’d like to give these projects a spin.

Summarizing Best Practices: Pulumi Project Structures

This post covered the following recommended practices for working with Pulumi:

Use multiple Pulumi projects when you need the added flexibility that multiple Pulumi projects offer.
Align your Pulumi projects according to external factors like company/department/team structure, resource dependencies, lifecycle, and use case/application.
Change the filesystem location for Pulumi code as needed without impacting the resources managed by that code, as long as the Pulumi project name and stack name don’t change.

The next post will focus on some specific technical details related to Zephyr’s project reorganization; specifically, how they used StackReferences to share information between their Pulumi projects. Stay tuned!

IaC Best Practices: Enabling Developer Stacks & Git Branches

Christian Nunciato — Fri, 10 Mar 2023 00:00:00 +0000

In the first post about code organization and stacks, we introduced Zephyr, a fictional company that uses Pulumi to manage its online retail store. Following on from that post, which discusses code organization and stacks, this post explores two more questions users frequently ask when working with Pulumi in teams — namely, How can I best enable multiple developers to collaborate on a Pulumi project? And how can I use Git and Git branching to support this kind of collaboration? In this post, we’ll provide some guidance and best practices around these topics, using Zephyr and its online store as the use case.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series — not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and its retail application changes to accommodate its growth.

Here are links to all of the posts in the series:

IaC Best Practices: Understanding Code Organization and Stacks
IaC Best Practices: Enabling Developer Stacks and Git Branches (this post)
IaC Best Practices: Structuring Pulumi Projects
IaC Best Practices: Applying Stack References
IaC Best Practices: Implementing RBAC and Security
IaC Best Practices: Using Automation API
IaC Best Practices: Summarizing Key Learnings

Existing IaC Workflow

When we last met up with the Zephyr team, they were off and running, managing their newly refactored online store, Zephyr Archaeotech Emporium, with a single Pulumi project and two Pulumi stacks — one for development (dev) and another for production (prod). The team had chosen to use one Git repository (a monorepo) to manage the code for the online store and its infrastructure after refactoring the store into a set of containerized microservices deployed with Kubernetes on Amazon EKS.

For a snapshot of the code as it was at the end of the first post in the series, see the blog/original branch of the pulumi/zepyhr-app repository on GitHub.

Up to now, the two-stack approach has worked well for the team. It’s allowed them to focus on building new features, integrating those features into the shared dev environment by developing and testing locally (which we’ll cover in a future post) and merging their feature branches regularly into the base branch, main, using standard GitHub pull-request workflows. Around once a day, as the team’s changes are merged into main, the team lead (and original author of the Pulumi program) deploys them into the dev environment with Pulumi with the following commands:

git pull origin main
cd infra
pulumi stack select zephyr/dev
pulumi up

This builds new container images for any services whose code has changed since the last stack update, pushes those images to Amazon ECR, and updates the EKS cluster to deploy the services automatically.

Then, around once a week, as new features are ready for customers, the team lead deploys the main branch of the repository into production similarly:

git pull origin main
cd infra
pulumi stack select zephyr/prod
pulumi up

The deployment process is still a bit more manual than the team would like — but since updates typically take only a few minutes, the team lead is generally satisfied with the status quo.

A few more details worth noting out about Team Zephyr’s current setup:

Both of the project’s stack-configuration files, Pulumi.dev.yaml and Pulumi.prod.yaml, are checked into version control. This is safe and considered good practice (even with secrets) because it allows the team to review or change the configuration of a given stack easily by viewing or editing the code in each file.
The main branch is the only long-lived branch in the repository; all others (e.g., developers’ feature branches) are deleted once they’re merged into main.
Thanks to this workflow, the application deployed into the dev environment tracks closely with the latest code in the main branch. As such, the dev stack serves as a kind of pre-prod or integration environment that not only allows the team to dogfood new features as they’re added, but also to catch bugs or other regressions before they escape into production. When things do go wrong, the team can review the list of recent stack updates to identify the source of the defect and get it fixed.
Zephyr is using the Pulumi Service backend with a single organization (that everyone on the team belongs to) and separate AWS accounts for the dev and prod environments. Everyone on the team has read/write access to the AWS account used for development, but only the team lead has access to the AWS account used for production.

Growing Pains

Life for Team Zephyr was good — for a while.

Then one day during a particularly active sprint, a developer who’d started working on an improvement to the checkout process realized it might benefit from an updated version of Redis, which appeared to be presently pinned at 6.0 in the Pulumi program:

// ...
const checkoutRedisDeployment = new k8s.apps.v1.Deployment("checkout-redis-deployment", {
 spec: {
 template: {
 spec: {
 containers: [{
 image: "redis:6.0-alpine",
 // ...

A quick glance at the dev and prod stacks in the Pulumi Service confirmed this was the case.

After a quick spike to validate the enhancement locally, the developer felt confident that a Redis upgrade was the way to go — but since the deployed version of Redis would ultimately dictate the code that’d have to be written to finish the feature for real, they couldn’t really be sure without testing the theory with a full deployment. What they really wanted was to be able to deploy a Redis upgrade, along with the associated code changes, into an AWS environment as much like production as possible — the dev environment, say — without having to go through the process of making a pull request, getting it merged, and waiting for the team lead to deploy the dev stack only to realize later that the change didn’t actually work as expected, and be forced to revert.

In situations like these, it can be tempting to suggest that the developer simply pick up and use the dev stack — especially on small, fast-moving teams like Zephyr’s where everyone already has access to the Pulumi stack and the shared cloud environment. Indeed, Pulumi makes this sort of thing easy: to deploy their local changes into the dev environment, our developer need only check out their feature branch, select the dev stack, run pulumi up, and then wait a few moments for the changes to be deployed:

cd infra
pulumi stack select zephyr/dev
pulumi up

This approach definitely works — but it presents at least a few problems:

The team would lose all of the practical benefits of having the dev stack track closely with the latest content in main.
The team lead’s deployment of main into dev later that day would overwrite the developer’s own feature-branch deployment. There’s a good chance the lead would notice the diff of the deployment preview and deduce that someone else on the team had possibly deployed into dev out of band:
```
 kubernetes:apps/v1:Deployment (create):
 spec: {
 template: {
 spec: {
 containers: [
 [0]: {
- image: "redis:6.0-alpine"
+ image: "redis:7.0-alpine"
 ...
 }
 ]
 }
 }
 }
 ...
```
But if they didn’t, the developer’s ad-hoc deployment would be lost. At best, it’d force both team members to drop what they were doing and get in touch to figure out what to do next. At worst, it could lead to confusion and the potential for shipping untested code.
In addition to raising the risk of developers overwriting each other’s work, active collaboration the same stack increases the likelihood of deployment conflicts. On the one hand, this is a good thing in that Pulumi prevents same-stack updates from running concurrently:
```
$ pulumi up
error: [409] Conflict: Another update is currently in progress.
```
But on the other, it’s a not-so-good thing in that it’s a reduction in development velocity: The update that can’t run because another update is in progress is an update that will have to wait until sometime later. The more often this happens, the longer it ultimately takes for the team to get the validation it needs to keep moving forward.

So facing these issues, what should the Zephyr team do? The dev stack had originally been created for rapid development — and that worked well, when there was only one developer working on infrastructure. With several, however, things clearly get complicated. Should the team be using a dev stack at all? If so, how? And if not, what should they be doing instead?

Stepping Back to Reassess

With what we’ve learned so far, we can return to the questions we raised at the beginning of this post and begin to sketch out a few recommendations for Team Zephyr. Those questions, again, were:

How can I best enable multiple developers to collaborate on a Pulumi project? For Zephyr, an ideal answer to this question would give every member of the team the ability to iterate on both application and infrastructure code concurrently, deploying as often as needed without having to worry about disrupting the work or the velocity of anyone else on the team.
How can I use Git and Git branching to support this collaboration? So far, the feature-branch/pull-request workflow has been serving the Zephyr team well. Should they continue on the same path, merging frequently to main and deploying to dev and prod from there? Or is there some alternative workflow that might serve the team better, either now or at some time in the future?

Let’s answer these questions now by taking a look at how the Zephyr team handled them.

Using Developer Stacks

We’ve seen a few ways that having multiple developers share a single stack for ad-hoc deployment can lead to various problems. But we’ve also seen at least one situation in which being able to run ad-hoc deployments — without the risk of colliding with existing environments — can be quite valuable. In fact there are many reasons you might want to do this — and fortunately Pulumi makes it quite easy to do so.

The best practice is to allow developers to create and manage their own Pulumi stacks. In this workflow, developers working on feature branches are able to iterate freely on both application and infrastructure, deploying as often as necessary into a shared cloud environment (typically a sandbox environment, isolated from production) and destroying cloud infrastructure when it’s no longer needed.

For the developer, the typical workflow looks something like this:

Check out a new feature branch
Make changes to application and/or infrastructure code
Select your personal developer stack with pulumi stack select
Deploy your stack into your team’s shared cloud environment with pulumi up
Commit your code and submit a pull request, optionally including a link either to the Pulumi update (e.g., when using the Pulumi Service) or to certain key resources belonging to the stack — for example, an application’s primary HTTP endpoint

The benefits of a workflow like this one are many:

No more resource-naming collisions. Because Pulumi names cloud resources uniquely by default, no two stacks will ever produce cloud resources with the same names.
No more deployment conflicts. When every developer has a stack of their own, they can deploy to the sandboxed environment as often as necessary, without having to wait for another deployment to finish first.
Everyone on the team has their own production-like environment to use for end-to-end testing.

For Zephyr, the quickest path to adopting this workflow was to use the dev stack as a foundation for new developer stacks. To do that, team members simply ran the following Pulumi commands:

pulumi stack init zephyr/developer-name --copy-config-from zephyr/dev
pulumi up

Developer stacks are considered ephemeral, so they’re torn down with pulumi destroy when not in use in order to cut down on cost. Stack configuration files (e.g., Pulumi.developer-name.yaml) are checked into version control and stored alongside other stacks like Pulumi.dev.yaml and Pulumi.prod.yaml.

Keeping Feature-Branch Workflows

We’ve also seen that the Zephyr team’s existing Git workflow (short-lived feature branches merged frequently into a long-lived base branch) has worked well for them — and it turns out this workflow also works well when combined with the use of developer stacks.

For example, after creating and deploying a new stack to validate and test the proposed Redis upgrade, our Zephyr developer simply included a link to the application (deployed with their dev stack) in the associated GitHub pull request. Reviewers were then able to follow the link to the online store to see the effects of the shopping-cart change firsthand, which meant they were able to approve and get the change merged quickly and confidently knowing it’d work when deployed into production.

Naming Stacks Clearly

Lastly, having adopted the use of developer stacks, the Zephyr team realized the name “dev” probably didn’t work quite as well as it once had for the dev stack. Since the stack and its long-running deployment had essentially been serving (and continued to serve) as an environment for testing changes before releasing them into production, the team decided to rename the dev stack test instead. To do that, they simply ran:

pulumi stack select zephyr/dev
pulumi stack rename zephyr/test

Viewing the Next Iteration of Code

You can view the second iteration of Zephyr’s Pulumi and application code — the iteration that corresponds to the decisions described in this blog post — by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to the blog/dev-stacks branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Developer Stacks and Git Branches

In this post, we covered several new guidelines for working collaboratively with Pulumi, all aimed at optimizing for high team and developer velocity:

Use short-lived feature branches and merge them frequently into a single base branch.
Deploy the base branch regularly into a long-running pre-prod environment to continuously test the integration of the components of the system.
Enable developers to create and use dev stacks of their own as sandboxes for development, testing, and collaboration.
Name your stacks clearly and in alignment with the environments they deploy into — test, staging, production, etc.

The next Zephyr blog post will dive into how the Zephyr team structures their Pulumi projects. Stay tuned!

Fast Docker Image Builds with Pulumi

Monica Rodriguez — Wed, 08 Mar 2023 14:00:00 -0800

How do I speed up Docker image builds with Pulumi? Use BuildKit (the default since Docker 23), enable a registry or layer cache so repeated builds reuse work, write a multi-stage Dockerfile so production images skip build-time dependencies, and reach for the dedicated Docker Build provider when you need buildx features like multi-platform images, build secrets, or Docker Build Cloud. With these techniques together, repeat builds in a Pulumi program commonly drop from minutes to seconds.

TL;DR

BuildKit is the default in current Docker releases and in the Pulumi Docker providers — you generally do not need to opt in.
Cache your layers somewhere persistent. A registry cache (cacheFrom / cacheTo) is what makes CI builds fast across runners and contributors.
Multi-stage Dockerfiles are the single biggest win for image size and rebuild speed when you have a compile or bundle step.
Use pulumi/docker-build for new projects. It exposes the full buildx surface (multi-platform, secrets, multiple cache backends, Docker Build Cloud).
Use pulumi/docker v4+ if you already depend on it — its docker.Image resource also rebuilds only on context change and supports BuildKit and registry caching.

What actually moves the needle

The numbers below are representative of what teams report from a typical Node.js or Python web app (single Dockerfile, ~300 MB final image). Your mileage varies with project size, network, and runner specs — treat this as a directional comparison, not a benchmark suite.

Configuration	Cold build	Repeat build (no source change)	Repeat build (small source change)
Legacy builder, no caching	minutes	minutes	minutes
BuildKit, local layer cache only	minutes	seconds (cache hit)	tens of seconds
BuildKit + multi-stage Dockerfile	minutes	seconds (cache hit)	tens of seconds
BuildKit + multi-stage + registry cache (`cacheFrom`/`cacheTo`)	minutes (first run)	seconds (cache hit)	seconds–tens of seconds, even on a fresh CI runner
`docker-build` provider with buildx + Docker Build Cloud	under 2 min	seconds (cache hit)	seconds

The headline pattern: each row past the first is dominated by what the build can skip, not by raw CPU. Pulumi’s job is to wire those caches up declaratively so they keep working across every pulumi up.

How does Pulumi build Docker images?

Pulumi has two providers for image builds, and which one you should use depends on how new your project is.

The Docker provider (pulumi/docker) is the original. Its docker.Image resource can build a local context and push to any registry as part of a Pulumi deployment, alongside resources for containers, networks, and volumes. Since v4 it rebuilds only when the build context changes and uses BuildKit by default.
The Docker Build provider (pulumi/docker-build) is a newer, dedicated provider focused exclusively on image builds. It exposes Docker’s buildx interface, so you get multi-platform builds, multiple cache backends, build secrets, and Docker Build Cloud as first-class features.

For new projects, prefer docker-build. For existing programs that already use docker.Image, you do not have to migrate — docker.Image continues to receive the speed and caching improvements covered below.

Why are my Docker builds slow?

Three things make Docker builds slow in practice, and Pulumi addresses each one declaratively.

No cache, or the wrong cache. A fresh CI runner has no local Docker layers. Without a remote cache, every build reinstalls every dependency.
Cache busting on noise. If your Dockerfile copies package.json and the rest of your source in the same step, every source change invalidates the dependency-install layer. Order layers from least-changing to most-changing.
Building dev tooling into the production image. Compilers, linters, and test frameworks have no business in a runtime image. Multi-stage builds let you discard them.

How do I enable BuildKit and registry caching in Pulumi?

BuildKit has been the default builder since Docker 23 and is the default in both Pulumi providers. The piece you usually have to configure is where the cache lives — local disk on a developer laptop is fine, but for CI you almost always want a registry cache so layers persist across runners.

With the Docker Build provider (recommended)

import * as docker_build from "@pulumi/docker-build";

const image = new docker_build.Image("my-image", {
 tags: ["docker.io/pulumibot/demo-image:latest"],
 context: { location: "./app" },
 dockerfile: { location: "./app/Dockerfile" },
 platforms: [
 "linux/amd64",
 "linux/arm64",
 ],
 cacheFrom: [{ registry: { ref: "docker.io/pulumibot/demo-image:cache" } }],
 cacheTo: [{ registry: { ref: "docker.io/pulumibot/demo-image:cache", mode: "max" } }],
 push: true,
});

cacheFrom and cacheTo together are the important pair: the first run populates the cache, every later run on any machine pulls it. mode: "max" exports cache for every layer, not just the final ones.

With the Docker provider (`docker.Image`)

import * as docker from "@pulumi/docker";

const image = new docker.Image("my-image", {
 imageName: "docker.io/pulumibot/demo-image:latest",
 build: {
 context: "app",
 dockerfile: "Dockerfile",
 platform: "linux/amd64",
 cacheFrom: {
 images: ["docker.io/pulumibot/demo-image:cache-base"],
 },
 },
});

The docker.Image resource accepts a list of cache images via cacheFrom.images. Tag a previous build with a stable cache tag (for example, :cache-base) and reference it here so subsequent builds reuse its layers.

How do I write a Dockerfile that builds fast?

The Dockerfile matters at least as much as the Pulumi code wrapping it. Two patterns do most of the work.

Order layers least-volatile first. Copy lockfiles and run dependency installs before copying source code. That way changing a .ts file does not bust the npm ci layer.

# Dependencies — only invalidates when lockfiles change
COPY package.json package-lock.json ./
RUN npm ci

# Source — changes constantly, but the dependency layer above survives
COPY . .
RUN npm run build

Use multi-stage builds to drop tooling. Compile in a fat image, copy the artifact into a slim one. The runtime image stays small and the build stage gets cached independently.

FROM node:20 AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-slim AS runtime
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
CMD ["node", "dist/server.js"]

When you combine multi-stage with a registry cache (cacheTo mode max), CI runs on fresh machines reuse both stages — the row in the table above where small source changes finish in 10–30 seconds.

How do I build multi-platform images with Pulumi?

Use the docker-build provider and pass a list to platforms. BuildKit will build all variants in parallel and push a manifest list, so a single tag serves whatever architecture the puller needs.

const image = new docker_build.Image("api", {
 tags: ["ghcr.io/example/api:latest"],
 context: { location: "./api" },
 platforms: [
 "linux/amd64",
 "linux/arm64",
 ],
 push: true,
});

If you are still on docker.Image, the build.platform field accepts a single target (for example, "linux/amd64"); for true multi-arch manifests, prefer docker-build.

How do I see Docker build output in Pulumi?

Filtered Docker build and push logs render directly in pulumi up output for both providers, so you can debug a failing layer without leaving your Pulumi run. With the Docker Build provider, structured progress shows per-stage timing — useful when you are trying to figure out which layer is the slow one.

How do I migrate an older `docker.Image` program?

If you are upgrading from a pre-v4 program, pulumi-docker v4 reshaped the docker.Image resource and renamed several supporting types, but does not manage backend state, so updating the package version and adjusting types is generally enough. The full migration guide lives in the v4.0.0 release notes.

If you are starting fresh or want the full BuildKit feature set, install pulumi-docker-build instead and follow the Docker Build provider getting started guide.

Getting started

Docker Build provider — recommended for new projects.
Docker provider — the original, including docker.Image, containers, and networks.
Introducing the Docker Build provider — deeper dive on buildx, secrets, and Docker Build Cloud.

The cheapest performance win in most Pulumi programs is wiring up a registry cache and a multi-stage Dockerfile. Start there, then reach for buildx features as the project grows.

IaC Best Practices: Understanding Code Organization & Stacks

Aaron Kao — Mon, 20 Feb 2023 00:00:00 +0000

This is the first in a series of blog posts that explores how a fictional company—Zephyr Archaeotech Emporium—uses Pulumi to manage their online retail store. This post explores a couple of common questions that users ask when working with Pulumi; specifically, where should I store my Pulumi code? And how do I support multiple environments with Pulumi? This post will provide some guidance and Infrastructure as Code best practices around these topics, using Zephyr and their online store as the use case.

The ultimate goal of this series is to discuss recommended practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series—not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.

Here are links to all the blog posts in the series:

IaC Best Practices: Understanding Code Organization and Stacks (this post)
IaC Best Practices: Enabling Developer Stacks and Git Branches
IaC Best Practices: Structuring Pulumi Projects
IaC Best Practices: Applying Stack References
IaC Best Practices: Implementing RBAC and Security
IaC Best Practices: Using Automation API
IaC Best Practices: Summarizing Key Learnings

Setting Up the Scenario

Zephyr is short for Zephyr Archaeotech Emporium, the fictional company in our scenario. Zephyr is an online retailer that specializes in the sale of “rare arcane artifacts and replicas.” Over the past few years, the company has experienced a significant increase in its online presence, making it easier for customers to purchase unique and mysterious items. Zephyr’s collection includes a variety of rare and unusual objects that are difficult to find elsewhere, making it a popular destination for collectors, enthusiasts, and adventurers.

Moving to a Containerized Architecture

Zephyr’s application is the second generation of their online store. Their first generation was a monolith that they manually deployed. It was sufficient in Zephyr’s early days, but as word about Zephyr spread and the volume of traffic to their online store increased, Zephyr found it increasingly difficult to deploy small, incremental changes to the store’s code. As a result, the velocity of their development efforts slowed. In preparation for the next phase of their growth—driven by robust sales of arcane artifacts—Zephyr’s team evaluated a number of different architectures that would enable them to increase development velocity and give them greater flexibility in scaling different aspects of their online store. In the end, they settled on a containerized architecture deployed to Kubernetes because some of the existing team was already familiar with these technologies.

Here’s a diagram of their second-generation application architecture:

As you can see in the diagram, the application has been broken down into microservices. The UI service communicates with other backend services via HTTP APIs; the various backend services communicate with various databases like MySQL and Redis for data persistence and/or caching. The backend services are responsible for implementing the functionality of a typical e-commerce site like orders, checking out, and managing the inventory catalog. While the choice of Kubernetes does afford Zephyr some much-needed functionality—like the ability to easily scale different services within the application—it also brings with it an added level of complexity that did not exist with their previous monolithic architecture. The Kubernetes manifest to deploy their application tops out at just over 1,000 lines of YAML!

Astute readers may note that Zephyr’s application looks very similar to this application; in truth, this series uses a fork of the AWS Containers retail store example as the application in this fictional scenario. (Many thanks to the AWS Containers DA team for their hard work here!) This is an admittedly over-engineered application, but it does afford the opportunity to address a range of customer scenarios in this blog series.

Managing Infrastructure and Application Deployments

As part of the switch to their new architecture, Zephyr decided they wanted to use Pulumi to manage both the infrastructure and the application deployment. Why Pulumi? Zephyr’s team recognized that moving to microservices on Kubernetes was going to make it more difficult to create reproducible, consistent implementations of their application. Zephyr’s team also knew that adopting infrastructure as code would help them with fast and repeatable deployments with little additional complexity. Another bonus was being able to use programming languages they already knew.

As they prepared to embark on using Pulumi, a couple of questions came up for the Zephyr team:

Where should they store their Pulumi code? The ability for Zephyr’s developers to use a full programming language to manage the infrastructure and deployment of their online store is a huge plus, but should this code go in the same repository as their application code? Or a different repository?
How do they address the need for multiple environments? It’s clear to the Zephyr development team that they’ll need multiple instances of the online store (for things like testing, QA, or development). What’s the best way to handle this when using Pulumi?

Let’s examine each of these issues in a bit more detail, and see how Zephyr chose to proceed.

Storing Pulumi Code

When it comes to answering the question of where to store Pulumi code relative to the application(s) it supports, there are two basic options:

In the same repository as your application code: This approach has the advantage of being simpler, but may not offer the necessary flexibility when dealing with multiple teams or varying levels of access control.
In a separate repository from your application code: Using a separate repository adds some complexity, but does provide greater flexibility to more easily address a variety of organizational requirements and varying levels of access control.

So which approach is best? That will depend on a number of different factors, many of which are outlined in the organizing projects & stacks documentation. Some other factors that users need to take into consideration include:

Who is responsible for maintaining the code? If the application code and the Pulumi code are managed by different teams, then using separate repositories may be the best approach.
What sort of access controls are needed? If the organization needs different access controls for the application code and the Pulumi code, then separate repositories are generally needed.
What is the relationship between the infrastructure resources managed by Pulumi and the applications? If the infrastructure is “shared,” meaning it is used by multiple applications, then keeping the Pulumi code in a separate repository may be preferred. If the infrastructure is dedicated to a specific application, then keeping the application code and the Pulumi code together may be preferred.
What about CI/CD? If an organization will use a CI/CD pipeline, that may affect how you organize your code. (This is something we’ll delve into more deeply later in this series.)

It’s worth stressing that the answers to these questions may change over time as organizations grow and their applications evolve. Addressing that change and its effects on an organization’s Pulumi projects and stacks is something we’ll be discussing later in this series.

The discussion above is working from the assumption that Pulumi users will store their Pulumi code in a version control system, like Git. Strictly speaking, using a version control system isn’t required for Pulumi to function (there is one exception we’ll touch on later in this series when we discuss Pulumi Deployments), but we very strongly recommend that all users use Pulumi in conjunction with version control.

In the case of Zephyr, their team felt like a monorepo approach (storing Pulumi code and application code together) was most appropriate for their specific requirements. Zephyr is a relatively small organization with a single team of developers who are responsible for managing both the cloud infrastructure as well as the development and deployment of the online store application. Zephyr’s team also decided to use a single Pulumi project—for now. (Be sure to follow this blog series to see how that evolves, and why!)

Addressing the Need for Multiple Environments

The second question Zephyr encountered is how to handle the need for multiple instances of their application. In addition to a production instance—which is the instance behind Zephyr’s public-facing online store—Zephyr also felt they needed an environment for the developers to use in testing changes to the online store.

This use case—needing to have multiple, separate instances of the infrastructure and applications created by a single Pulumi program—is exactly what Pulumi stacks were designed to address. Each stack is a separate instance of the resources created by a Pulumi program within a project. Further, each stack has its own independent state, and each stack has its own configuration values. Stacks can be short-lived (meaning the associated resources are also short-lived), or stacks can be long-lived. Aside from the cloud resources created by the Pulumi program, stacks are lightweight and simple to create or delete.

Zephyr decided to initially start with two stacks: a production stack (named “prod”) and a development stack (named “dev”). As you’ll observe throughout this series, this is a decision that is easily adjusted over time as your organization’s requirements change.

Viewing the First Iteration of Code

You can view the first iteration of Zephyr’s Pulumi and application code—the iteration that corresponds to the decisions described in this blog post—by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to viewing the blog/original branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Organizing Code and Stacks

This article introduced you to Zephyr, the fictional company for this blog series, and reviewed the application architecture for Zephyr’s online store (a containerized application running on Kubernetes). Throughout the blog series, this application serves as the use case around which various recommended practices are discussed.

In this post the following guidelines were covered:

Use version control for your Pulumi code. Although it isn’t strictly required in many cases, a number of the benefits of adopting infrastructure as code are lost without the use of version control.
Place your Pulumi code into version control repositories in a way that aligns with organizational requirements. There is no “right” or “wrong” approach; choose the approach that enables you to satisfy your organization’s requirements around access control, code/repo ownership, and other organizational requirements.
Leverage Pulumi stacks to create multiple instances of cloud resources with a single Pulumi program. Pulumi stacks were created to address the need for organizations to create multiple instances of cloud resources, each with its own state and its own set of configuration data, from a single Pulumi program.

For now, Zephyr has decided to go with a monorepo approach—a single repository that contains both their application code and the Pulumi code to manage the infrastructure resources. All of the resources are defined in a single Pulumi project, with multiple stacks that correspond to development and production environments. Over the course of this series, we’ll see how Zephyr’s use of Pulumi changes as Zephyr grows and their application evolves.

The next Zephyr blog post will examine how the Zephyr team adjusts their use of stacks and Git branches to support developer productivity. Stay tuned!

FinOps With Pulumi

Matt Small — Tue, 14 Feb 2023 00:00:00 +0000

What is FinOps?

The FinOps Foundation eloquently defines FinOps as “an evolving cloud financial management discipline and cultural practice that enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions.” Simply put, FinOps is the continuous effort to control cloud spend.

Just as organizations have adopted operations-focused best practices into software development cycles and have considered how to best insert security best practices along the way, financial best practices may also be codified by developers writing cloud programs.

Adopting a FinOps practice brings real world OpEx(Operating Expenses) considerations to the decisions that cloud engineers regularly make. When you add FinOps to your Cloud Center of Excellence, technical teams bring financial context into service level and recovery objectives, scaling and placement decisions, and business continuity planning. Developers are typically more conscious of the resources they are using and consider the impact of their application design decisions at both development, test and production scale. Therefore organizations usually realize substantial OpEx savings while better forecasting their growth.

Who is Responsible for Cloud Financial Decisions?

In most smaller organizations, developers themselves are often responsible for the decisions that impact cost. In larger organizations, this responsibility typically falls to a central IT or Cloud Platform team, often seen as the cost center for infrastructure, and the best team to optimize that spend continuously. Some organizations are evolving toward FinOps as a practice, with dedicated teams responsible for enabling the organization to make smarter financial decisions and negotiating with cloud vendors for discounts.

Ultimately, the cloud account owner is responsible for the cloud bill, and failure to pay that bill will result in the suspension of cloud services. Given that infrastructure is often code, developers or IT teams are often made responsible for the task of optimization. Optimization recommendations may range from simply deleting orphaned cloud resources to re-architecting the entire application to use a modern serverless framework that would consume less resources overall.

Establishing FinOps Practices

The first step of establishing many FinOps practices is determining the allocation of costs. It is easy to allocate costs to individual users launching cloud services, but it becomes compoundingly hard once pipeline automation, team structure, and business unit reporting come into play. As an organization scales, it grows increasingly difficult to allocate costs across different business units and teams, across environments such as development versus production, or across application component services such as the data layer. Further, how does an organization account for supporting systems and services running alongside applications in the cloud, including third-party commercial software?

When starting your FinOps Journey is essential to understand cost allocation, typically done with consistent and enforced tagging of cloud resources. Creating a Tag Policy is covered below.

FinOps: Reactive vs. Proactive Optimization

Most FinOps processes today are reactive; once you have received your bill, you will observe (and pay) the costs and resources accounted for, review your architecture and make necessary adjustments such as reserving, reallocating or removing resources. Next month, you will do it again, also factoring in your company’s cloud growth, application design changes, team shifts and other dynamics that occurred in the meantime. Many platforms, tools, and job descriptions are oriented toward this reactive optimization process.

Proactive optimization and enforcement means providing real time feedback on how cloud engineering architectural changes may impact cost and creating preventative policies which do not allow for budget-busting cloud services to be provisioned. This is an evolving practice in many modern cloud-centric organizations. As more cloud operations are defined as code, this preventative, proactive optimization model is becoming more popular.

Financial optimization can also come in the form of strategic sourcing; most cloud providers offer multiple discount mechanisms, including large enterprise discount plans for committed consumption that may extend to third party SaaS software purchased through the cloud marketplaces, as Pulumi is. This level of optimization will provide dramatic savings on your top-line spend.

FinOps with Pulumi

Using cloud engineering practices with Pulumi, you have the opportunity to implement both proactive and reactive models to control your cloud spend.

As an infrastructure as code platform, Pulumi is in the unique position to be proactive about managing your costs.

You can design guardrails on your provisioning to limit expensive resource types or quantities,
Direct resources to reserved allocations,
Ensure resources are tagged and traceable to the correct cost center,
Connect resources to your observability and dedicated FinOps platforms,
Provide preventative visibility and control.

Consider uses for CrossGuard Policy as Code, multi-language components for defining resource abstractions, and Automation API to package up orchestration into web service endpoints, as described in Pulumi Features below.

If you are proactively managing your infrastructure, then reactively, the largest contributor to financial overrun is drift; the misalignment between the desired state and the actual state. Pulumi is used to define your desired state; however, many things may cause the actual state to begin to deviate from the desired state in your code, such as manual actions from the cloud console or ad-hoc deployment pipelines.

Drift detection, which is the name given to continuously scanning your environment to compare the actual state with the desired state and alerting and/or automatically remediating the difference in state, is a primary method of managing these creeping cloud costs. Leverage an automation system, such as a CI tool, to implement continuous scanning against your Pulumi code as described below.

Remediation of any infrastructure mismanagement requires infrastructure reconfiguration action to be taken; often resources need to be replaced with better suited, more cost effective resources, and resources that were provisioned outside of acceptable policies may have to be destroyed. However, any remediation should carefully consider the underlying reasoning for the misalignment in the first place. That should include both the reason for the initial deployment specifications as well as any decisions for why that’s no longer acceptable. Be sure to consider all architectural decisions, such as disaster recovery and business continuity planning, regional latency, and any other impact to service level objectives (SLOs). Understanding the context will help you code preventative measures into your Pulumi programs going forward and the self-documenting nature of infrastructure-as-code will ensure your decisions are memorialized for future developers and operators, for example, in a Stack README as described.

Automation of remediation is a goal for many cloud engineering organizations; if the perfect state is defined in code, then any drift may indicate the need to reconcile back to the desired perfect state. Given that some cloud resources may necessarily drift due to limitations in how they are managed, a combination of alerting and remediation of known patterns is implemented. For example, you might choose to detect and deprovision any newly instantiated resource not provisioned by Pulumi, but allow manual resizing of resources if needed. Just be sure to capture any infrastructure and improvements you want to make default and/or repeatable as infrastructure as code.

Best First Action: Implement a Tag Policy

Getting started with a FinOps usually falls to the resource owners or Cloud Engineering teams themselves. The more resources you have, the more complex and noisy things are going to get and the harder it will be to both allocate costs and resources where they are needed, as well as identify and eliminate the unneeded ones.

Implementing Tag Policy is the best first action to take. Tags are the set of required cloud metadata tags for any resources provisioned by your organization. While not every resource in every cloud may be tagged, most are. Getting into the practice early will help you maintain accountability for what you are running, demystify your cloud bills as they grow larger, and help update the application architecture or make cost-effective product decisions long-term.

A Basic Tag Policy for FinOps will include:

Owner - the person responsible for this resource that you would contact if you had to (e.g., johndoe@pulumi.com)
Origin - the pipeline, process, and/or system through which this came to be (e.g., ArgoCD, Jenkins, Pulumi)
Business Unit - the team responsible typically maps to the cost center associated (e.g., Sales, Engineering)
Environment - the teams/customers this deployment services (e.g., Dev, Staging, Production)
Application Component - the part of the application that this resource serves (e.g., data pipeline, authentication, front end)*

You will now be able to report on and manage resources in those tag groups. It’s advisable that you do not create too many required tags, or it will quickly get difficult to manage and it increases the likelihood of multiple tags applying, which reduces the insightfulness of the data.

Read more about Tag Policy implementation with Pulumi below, and for extended reading, refer to this post, Automatically Enforcing AWS Resource Tagging Policies from CEO, Joe Duffy.

Pulumi Features in Context

Multi-Language Components

Multi-language components allow you to create repeatable infrastructure patterns, modular abstractions of resources and/or groups of resources that meet your desired configuration standards. Packaging up resources within and across clouds, implementing standard deployment best practices, and making those components available to developers reduces the cognitive load that they have when building new applications from those pre-assembled, ready-to-run components.

Use these guardrails to enforce Tag Policies, for example that every tag for “Production” is actually “Production” and not “Prod” or “production” (yes, case sensitivity matters). Or you may ensure teams only deploy resources such as VMs to regions where you have allocated Reserved Instances or other financial benefits in your application architecture. With MLCs end users have access to your component and not to the underlying infrastructure resources, guaranteeing repeatabality, however, in order to realize the benefits of MLCs, your developers must be empowered to leverage them, but limited from using other methods.

CrossGuard

CrossGuard Policy as Code allows an Organization to define global rules for how resources may be provisioned and prevent resources from being provisioned by Pulumi in any other configuration. Pulumi Stacks that do not meet the Organization’s financial, security, data protection, or other pre-defined standards are prevented from provisioning any resources, and the developer is instructed to correct their desired configuration or program in accordance with the Organization-wide policies.

Policy as code ensures that any Pulumi code written and run meets an organization’s stated policies. It is the ultimate preventative weapon to prevent financial, security, or other placement and provisioning decisions that may lead to exposure or overrun.

Go here for examples of FinOps policies to get you started!

Automation API

Of the many things Automation API allows you to do, one is you can create an HTTP endpoint for a Pulumi program. Organizations have used this to control easy on/off switches for applications such as provisioning/deprovisioning single tenant SaaS architectures, short lived environments, and complex testing and blue/green deployments. Because of the encapsulation of a Pulumi program and the simplification this provides, this endpoint may be directly exposed to those responsible for making the financial decisions in the moment, such as a Sales Engineer who must turn on a new POC environment for testing who then must be sure to shut that off when completed. Oftentimes, these decisions are automated in processes and pipelines.

Pulumi FinOps in Practice: Sweeping up Dev and QA

A common automation in FinOps is to destroy things that don’t meet your Tag Policy. These “Sweeper Bots” will continuously query AWS for resources with specific tags and delete them. Read more about how Pulumi has implemented this using Pulumi to set tags and Lambda to sweep through and remove resources that meet certain specifications in this post, Controlling AWS Costs with Lambda and Pulumi.

Conclusion

In the manner of “Crawl, Walk, Run”, it’s best to start by implementing policies as code in a manner that will cause the most benefit while causing the least amount of change and/or downtime. Make sure you have the appropriate billing tags applied to all supported resources, ensuring you are able to allocate costs to the appropriate business units in your company. Then add in additional policies to make sure you are using instances covered by any reserved instance or similar savings plans. Continue evolving your infrastructure and policy as code strategy to incorporate both preventative and reactive checks and balances.

Have questions about using Pulumi when implementing FinOps in your IaC pipelines? Contact us and let us know your questions or request a demo.

Using Pulumi Securely

Tushar Shah — Thu, 13 Oct 2022 00:00:00 +0000

Cloud computing’s greatest strength and weakness is the proliferation of a massive number of services globally. To adequately assess and mitigate the inherent risks for your company, customers, and employees, cloud architects are typically responsible for a vast surface area of potential endpoints and vectors of attack.

Cloud Security Posture

AWS likes to talk about a Shared Responsibility Model, their friendly way of saying, “we’ve got our stuff covered, and if you’re compromised, it’s almost certainly your fault.” They have built up practices and recommendations, like the AWS Well-Architected Framework. Their pillars of architecture philosophy give you your best chances to securely use their resources (following these recommendations is easy with Crosswalk for AWS from Pulumi). Increasing high-profile attacks against private and open-source software supply chains have developers increasingly on high alert and conscious of incorporating more security into their DevOps.

Risks associated with maintaining a good cloud security posture extend to your infrastructure as code with Pulumi and it’s important to leverage the features and capabilities of the Pulumi platform along with good industry practices to secure your code, configuration and secrets.

Pulumi’s Hierarchy

Pulumi has three levels to consider access for: users, teams and organizations.

Organizations are a space for you to collaborate on shared projects and stacks. You may have multiple cloud accounts from a single cloud provider or multiple cloud provider accounts tied to an organization in any combination. It’s recommended that you segment organizations and cloud accounts to limit access and the “blast radius” of security or financial events that may occur within them.

Teams are groups of users within an organization with specific permissions to stacks. Consider using these to break up access to environments such as dev, staging and production. You should look to mirror your code repository or IAM teams/groups as much as possible.

Users are tied to personal identities, often with an identity provider and should be used wherever a human is using Pulumi. These identities should not be used for any systems, tools or pipeline-based provisioning.

Security Best Practices with Pulumi

The following recommendations should help you get started and prepare you to scale your company, application and cloud engineering team long term.

Manage Federated Identity

Connect your user and Pulumi organization with an Identity Provider such as Github, Gitlab or BitBucket, your Single Sign On (SSO) system and/or another System for Cross-Domain Identity Management (SCIM). Your code repositories hold your Infrastructure as Code, and most organizations are motivated to keep access tightly controlled. SSO improves this process across tools and puts more control into your organization’s hands. SCIM enables you to manage your users and groups centrally in your Identity Provider (IdP) and then synchronize those users and groups to the Pulumi Service. Leveraging your existing onboarding and offboarding process will help limit the risk of bad actors in Pulumi itself.

Build Pulumi Teams

Teams improve on basic role-based access control (RBAC) for administrators and users by allowing you to group users and their access to specific stacks in your organization. For example, a team called productionreadonly could be created and scoped so that all members have only limited access to read the production stacks, while another team, productionadmins, would retain full privileges to those stacks.

Import Manually Created Resources

Manually created cloud resources are error-prone and difficult to audit, iterate and improve on. If you’ve created manual resources, consider importing them to Pulumi programs.

Tag your Stacks

We have previously covered Tag Policies as a best practice for the cloud resources themselves. Still, you can also tag your Pulumi stacks to help organize and visualize your application resources in logical groupings that will help you assess, respond and automate incident remediation. Here are two examples of using tagging your stacks automatically: in Python and TypeScript.

Deploy from Pipelines

Automate every step of your deployment process for anything beyond local development by creating CI/CD pipelines. Manual approval steps in your pipeline should be avoided. Consider creating automated checks to ensure your infrastructure is created only with approved Pulumi Packages and automatically ensure your infrastructure is in compliance by using CrossGuard.

Leveraging Automation API as the programmatic interface for running Pulumi programs without the Pulumi CLI is a strongly typed and safe way to use Pulumi in embedded contexts such as web servers without having to shell out to a CLI. For example, Elkjop Nordic uses Automation API to provide a self-service portal that serves as a secure IT vending machine full of their application and infrastructure building blocks.

Log Everything

Track key system events such as user and pipeline activity, attempted and restricted activity, and identity and access control changes. Use Pulumi audit logs to simplify this process out of the box. If applicable, we would suggest automatically exporting Pulumi Audit Logs to your systems.

Use Tokens

Organization Access Tokens, Team Access Tokens and Personal Access Tokens securely connect your automation pipelines and development environments with Pulumi without the risks of association user/password combinations. Machines talk to Pulumi with tokens of various types, and it is always advisable to use Tokens over Users where possible. More on the types of tokens is below.

Organizational Access Tokens

These are scoped to the entire Pulumi organization. Use these for tooling with broad access across stacks and resources deployed in that organization. For example, if your production environment runs in an isolated cloud account and organization, then an Organization Access Token is likely appropriate for your CI/CD pipeline that deploys production.

Team Access Tokens

Scoped to the stack access of a Pulumi team. Use these when resources associated with different environments or services are commingled within a single cloud account or organization. In general, it is often recommended to create a Team Access Token for each CI/CD pipeline, for example, dev vs production.

Personal Access Tokens

Scoped to individual users. Used whenever a developer deploys from their local machine. Be sure your users are part of a Pulumi team to simplify their role-based access within the platform.

Rotate the Tokens

Add Pulumi to your list of tokens to rotate on a regular basis.

Assessing and Implementing

Even if you’ve been using Pulumi for a while, we recommend you regularly take these three steps to continuously audit and improve your cloud security posture as it relates to your infrastructure as code:

Confirm your users’, teams’, and organization access are in line with either or both your code access or cloud access policies; make any necessary updates to access, permissions or scope of privilege.
Audit any pipeline that uses Pulumi and ensure that they use fresh, rotated tokens.
Review your audit logs regularly, noting any abnormal or unexpected activity. It is advisable to export these audit logs and process them with a security event management system.

Some of the features mentioned, such as teams, SSO/SCIM and audit logs, are only available to Enterprise and/or Business Critical Edition users and those on our 14-day trial. If you would like to implement and/or test any of these features please start a trial or contact us for access.

Patterns for Drift Detection with Pulumi

David Flanagan — Thu, 16 Jun 2022 09:26:46 -0500

Portions of this blog post are out of date. See the Pulumi Deployments drift detection blog post for an updated overview and examples.

Let’s face it, at some point someone is going to modify your carefully-crafted and automated infrastructure without updating your Pulumi program. These changes cause the desired state of our Pulumi program’s to be inconsistent with the state of the world. These inconsistencies are often referred to as “drift”. In this article, I want to cover a couple of patterns for detecting and reconciling this drift with your Pulumi programs.

Why is Drift Detection Important?

Reconciling drift as quickly as possible is very important in the infrastructure as code (IaC) space because the longer drift is left unattended, the harder it becomes for automation to reconcile in a manner that can be considered safe for your organization, SLAs, and customers happiness. Imagine this as if it were your car. If a single tyre needs replaced, you might be OK for a while and not notice any problems driving. If you leave that unattended and another tyre runs flat, then you’ve got a slightly more serious problem. As these smaller problems accumulate you’re applying a multiplier of risk to your overall infrastructure, especially during reconciliation. The more resources Pulumi needs to handle during this step, the more likely for some collateral damage.

It’s important that you build a system that can tolerate a nominal amount of drift but actively works to repair that as quickly as possible.

Let’s see how.

Refreshing the World

When you run a pulumi up, Pulumi will create a directed-acyclic graph (DAG) from your Pulumi program. This graph is then compared to your state file, and any changes that need to happen to have the DAG match the state file are applied. Pulumi does not reach out to “the world” to detect if what is in your state file is actually still up to date, at-least not without adding the --refresh option.

That’s the secret sauce to ensuring that we can begin to detect and reconcile drift. Of course, this comes with a couple of caveats that you need to be aware of before adopting --refresh as your default behavior.

Firstly, it’s slower. The --refresh option will take longer because the Pulumi program needs to reach out to all of the providers and fetch information on the resources in your state file.

Secondly, the refresh will sync potentially “untracked” fields within your resources and could trigger more updates to your resources than you’ve had previously. This isn’t a bad thing and Pulumi’s defaulting will hopefully ensure that resources aren’t needlessly updated, but it’s definitely worth keeping in mind.

With that being said, let’s take a look at two common strategies for updating Pulumi programs.

Workflow Approach

The most common approach is to use your CI/CD / build system to run Pulumi whenever your source code changes. This works quite well, but it means that any drift isn’t detected and reconciled until someone pushes some changes. If nobody commits to the repository for 2 days, a week, or even longer - then a few problems can arise. The longer the gap, the more chance of drift causing more problems during the next run. Drift can cause resources to be recreated, which may be unexpected, and if left long enough, the “drift” might become expected behavior within your system and the next reconcile becomes a “break”.

When using the workflow approach for your Pulumi program’s, I highly encourage you to schedule builds as frequently as you can. Whether that’s hourly or daily, the more you run your Pulumi and handle drift, the easier and more comfortable you’ll be with the automation.

If you want to do this with GitHub Actions, you can use the cron settings on your workflow. This is an example of my own production infrastructure:

name: infrastructure

on:
 workflow_dispatch: {}
 push:
 branches:
 - main
 schedule:
 - cron: "6 6 * * *"

CircleCI also makes this rather easy too:

 daily-run-workflow:
 triggers:
 - schedule:
 cron: "6 6 * * *"
 filters:
 branches:
 only:
 - main

Operator Approach

Now, scheduled executions are great, but if you want to take that to the next level - using the Pulumi Kubernetes Operator is the next step. The operator can be configured to run every few minutes, ensuring that your world is always consistent with your Pulumi program. Here’s an example YAML representation of a Pulumi stack:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: some-repo-production
spec:
 stack: production
 projectRepo: https://github.com/some/repo
 branch: "refs/heads/main"
 destroyOnFinalize: true
 continueResyncOnCommitMatch: true
 refresh: true
 resyncFrequencySeconds: 60

The last three settings are the most important part of this configuration. Unfortunately, these are not the defaults - but there’s an issue tracking this change.

continueResyncOnCommitMatch is a setting that tells Pulumi to run even though the code itself hasn’t changed, which when used with refresh: true means we’ll always update the state of the world and reconcile any drift. resyncFreqeuncySeconds is the number of seconds between each reconcile.

Making Refresh Your Default

Before you go updating all your workflow files, or adopting the operator, for all the benefits of continuous reconciliation and drift detection, let’s cover a few safety tips.

Always run a pulumi up --refresh locally / manually first and inspect the changes. Pulumi may wish to replace some resources and if you don’t want that to happen - you need to prepare those resources for your new continuous reconciliation.

Protect

If you have a resource that should NEVER be deleted, make sure you protect it. You can do this by adding a ResourceOption to the resource in your Pulumi program.

{
 protect: true
}

opts=ResourceOptions(protect=True)

pulumi.Protect(true)

new CustomResourceOptions { Protect = true }

CustomResourceOptions.builder()
 .protect(true)

options:
 protect: true

Ignore Changes

If Pulumi wants to replace a resource that you don’t believe needs to be updated, you can ask Pulumi to ignoreChanges on certain properties. Of course, it might be best to have your code match those properties, but in the event that you can’t - you can use the code below to ignore:

{
 ignoreChanges: ["prop"]
}

opts=ResourceOptions(ignore_changes=["prop"])

pulumi.IgnoreChanges([]string{"prop"})

new CustomResourceOptions { IgnoreChanges = { "prop" } }

CustomResourceOptions.builder()
 .ignoreChanges("prop")

options:
 ignoreChanges:
 - prop

Delete Before Replace

If you have any resources that don’t use auto-naming, you’ll also need to ensure they have the deleteBeforeReplace option enabled.

{
 deleteBeforeReplace: true
}

opts=ResourceOptions(delete_before_replace=["prop"])

pulumi.DeleteBeforeReplace(true)

new CustomResourceOptions { DeleteBeforeReplace = true }

CustomResourceOptions.builder()
 .deleteBeforeReplace(true)

options:
 deleteBeforeReplace: true

Once you’ve run a couple of pulumi up --refresh and you’re comfortable with the proposed executions, you can adopt the automation and sleep easier, knowing that you’re handling drift and keeping your infrastructure up to date.

Want to See More?

Watch me demo some of these features live, on YouTube, on my PulumiTV series: Modern Infrastructure.

Hierarchical Config: The Interim Solution

David Flanagan — Tue, 15 Mar 2022 10:00:00 +0000

This blog post discusses hierarchical config before Pulumi ESC was released. ESC makes hierarchical config easy.

A really common question that we receive on the Pulumi team is, “How can we set config at a project level, that can be used across all stacks?”.

When I say “really common” … I mean really, really common.

This issue was first open in 2018 and has received 52 votes from the community. Not only that, we’ve had plenty of similar issues created over the years too.

This is clearly a feature that our community has asked for! We’re happy to say that we delivered the first part of our plans to support hierarchical config in early November 2022. While we believe this new functionality satisfies most customer requests, below are some other approaches you can also use.

Project Level Config

Typically, when you need to access the stack configuration in a Pulumi program, you use the config package.

import { Config } from "@pulumi/pulumi";

const config = new Config();
const region = config.require("region");

import (
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

conf := config.New(ctx, "")
region := conf.Require("region")

import pulumi

config = pulumi.Config();
region = config.require('region');

using Pulumi;

var config = new Config();
var region = config.Require("region");

This code will reach from the Pulumi.<stack-name>.yaml file and give you access to the configuration values for the prefix you want to load.

So if you want project level config that works across all stacks, you can “hard code” the configuration. It sounds too simple, right?

export const projectLevelConfig = {
 region: "us-west-2",
 encryptionKmsKey: "arn:aws:kms:us-west-2:...",
 issueEmail: "bugs@ellingsonmineral.com"
};

type ProjectLevelConfig struct {
 Region string
 EncryptionKmsKey string
 IssueEmail string
}

const projectLevelConfig = ProjectLevelConfig{
 Region: "us-west-2",
 EncryptionKmsKey: "arn:aws:kms:us-west-2:...",
 IssueEmail: "bugs@ellingsonmineral.com"
}

projectLevelConfig = {
 "region": "us-west-2",
 "encryptionKmsKey": "arn:aws:kms:us-west-2:...",
 "issueEmail": "bugs@ellingsonmineral.com"
}

var projectLevelConfig = new Dictionary<string, string>
{
 { "region", "us-west-2" },
 { "encryptionKmsKey", "arn:aws:kms:us-west-2:..." },
 { "issueEmail", "bugs@ellingsonmineral.com"
}

These objects/structs/dictionaries can all be consumed and imported across the various components and files within your Pulumi program, but they cannot be used at an organizational level across multiple Pulumi programs … unless you’ve got a mono-repo. Do you? 😅

Organization Level Config

So if you want to provide organizational level config, you need to fall deeper into the programming language ecosystem that you’re using to build your infrastructure. Every programming language has a way to parse and consume JSON and YAML files. Better yet, every programming language has a way to fetch a remote JSON/YAML file and give you a object/struct/dictionary.

So really, there’s no reason you can’t use a remote JSON file to provide a common configuration object across all your Pulumi programs. That being said, there are definitely reasons you shouldn’t use a remote JSON file. Maintaining a remote JSON file is much more of a burden than it will initially seem. As you add more values to your JSON file and more teams/projects begin to rely on it, you’ll start to feel the pain of schema management. How do I know that the JSON I’m pulling down has the values I need and that fields haven’t been changed or replaced? 😬

So while I 💯 feel like you shouldn’t do this, you can if you really need to. Just make sure you understand the tradeoffs and enforce a schema.

A common way to manage schema is to use a JSON Schema or CUE to define the structure of your JSON file.

By using one of these methods, you can publish a schema that is available within your organization and people can have confidence that the value they pull remotely can be deserialized to a strict type. Using CI/CD you can also ensure the value itself conforms to the schema before updating the public document.

With JSON Schema, you’d define something like:

{
 "type": "object",
 "properties": {
 "region": {
 "type": "string",
 "default": "us-west-2"
 },
 "encryptionKmsKey": {
 "type": "string",
 "default": "arn:aws:kms:us-west-2:..."
 },
 "issueEmail": {
 "type": "string",
 "default": "bugs@ellingsonmineral.com"
 }
 }
}

With CUE, this is much more concise. CUE is really awesome and I encourage you to check it out.

region: string | *"eu-west-2"
encryptionKmsKey: string | *"arn:aws:kms:eu-west-2:..."
issueEmail: string | *"bugs@ellingsonmineral.com"

Pulumi Native Hierarchical Configuration

So we’ve worked out that project level configuration is a data structure within our Pulumi programs and it works pretty darn well.

However, the organization level config is prone and rife with pain, confusion, and delusion. It requires non-trivial tooling and process to ensure that a global document is valid and consistent with downstream consumers. What I’ve shown you is a “if you must” approach that I would use myself. We’ve also not even handled the “merge” of organization level with project level overrides, and then stack level overrides. Of course, with code it’s all possible, but it’s more and more code that you need to write that we don’t want you to need to write, right? Right!

Pulumi wants to make this better for you, which is why we added project-level config support and why we are actively exploring additional enhancements. If you want to follow along with our progress on improving the configuration experience, I suggest you watch/subscribe to this issue.

If you want to share your opinions on how YOU would like this feature to be implemented, jump into the comments. We’d love to hear from you!

Speak soon.

Upgrade Strategies: An Introduction for IaC

Laura Santamaria — Mon, 07 Mar 2022 09:59:56 -0600

When you’re working with infrastructure, you’re inevitably going to need to upgrade or update that infrastructure. Whether it’s an operating system update or a desire to get CPU or memory upgrades, you will need the ability to pick resources and change them as necessary. In the past, this kind of upgrade would be done on the basis of individual resources, with each one being updated and checked either by hand or programmatically before moving onto the next resource. If you’ve ever done a database migration or if you ever did the recommended way of upgrading your computer’s operating system including all of the backup steps, you’re familiar with this process. Stand up the new resource. Check everything works. Move over the data. Check again. Tear down the old infrastructure. In a cloud computing environment, though, you’re often dealing with hundreds or thousands of resources, and doing one-by-one replacement is a nightmare that takes ages. However, there are other options, many borrowed from the application deployment world, that we have available to us because we write infrastructure as code.

Generally, there’s a few strategies for replacing something in a cloud computing environment. You’ll often hear about them as deployment strategies. These strategies generally differ based on the order of operations: Do you create a new resource before you delete the old one? Do you replace some and pause to gather data? Do you YOLO and just toss everything out and build new? All of these strategies are worth considering depending on the needs of your situation. A production system likely needs to maintain uptime, or the amount of time a system is available to the end user, to meet service-level agreements (SLAs), which might include an availability promise such as a promise of 5 nines (99.999%) of uptime. In that case, a YOLO strategy won’t be very acceptable, will it?

As many of these deployment strategies initially came from the application world, we can’t just use the same deployment strategies exactly as they appear for an application because there’s a bit more going on under the hood. We need to consider what we’re building and where we are on our stack. If we have multiple instances of our application running on identical containers, for example, we can treat the containers a bit differently than, say, the single load balancer in front of all of the containers or the node underneath a single-node Kubernetes cluster.

To illustrate all of the following deployment scenarios, let’s imagine we have a system where an application is deployed on some infrastructure. What the application is doesn’t really matter; we’re going to focus on the infrastructure here. We’ll pick a simple situation: There’s a security update for one of your pieces of infrastructure.

Big Bang

Let’s say the infrastructure you’re working with is a large number of virtual machines (VMs) that are running an outdated, insecure version of an operating system. If you weren’t concerned about uptime or the stored data on those systems, you could certainly just tear them down and stand up new ones—this situation could reflect that perhaps you’re working with a development environment that is ephemeral and isn’t in use right now, or you don’t have any load just yet. Some folks refer to this kind of upgrade strategy as a “big bang” strategy, though folks who run data centers would likely say that the big bang strategy still at least required testing and careful consideration of data transitions and you would rarely completely wipe the original system before replacing it with another. If you’re wondering where this upgrade strategy came from, it was originally used in non-cloud-native systems where you likely didn’t have the hardware available to run other deployment strategies. In these cases, you would have maintenance windows and change processes, and the whole process was mapped out well ahead of time. This kind of fast switchover, especially with little to no testing, really is not a good idea for a cloud-native or cloud-based production environment, and it’s frowned upon if you’re working with a cloud-based environment that others are also using. There’s so many better options than scheduling maintenance windows and ripping apart systems when you have the capability to stand up parallel virtual hardware.

Blue-Green Deployments

Now, let’s say that instead of tearing down and then standing up new VMs, you instead were to create an almost identical environment with the new operating system, test it under load, and then transition your traffic over and wait for success before tearing down the old version. This is known as a blue-green deployment, and it’s a fairly popular strategy. A blue-green deployment is, in short, a create-check-delete process. You may have heard this strategy called “red/black deployment” or “a/b deployment.” The rationale behind using “blue” and “green” as names originally comes from needing easy names that didn’t have any kind of connotation of one group of systems being “better” than the other (e.g., “red” is the same color as alert lights, so don’t you want to keep the “black” deployment?). So you certainly can name it whatever you’d like in your company, but the concept is the same. The “blue” environment is currently running. To upgrade, you stand up an almost identical environment with the upgrade you want to perform. That environment is your “green” environment. Then, you run any test that you can against that green environment to ensure it’s ready. Finally, you switch your traffic over to the green system and monitor for any issues. Once you’re sure the environment is stable and can handle the load of your normal traffic (typically measured in hours to days depending on when you made the switch and what your traffic patterns demonstrate), you tear down the blue environment.

Note that, while a blue-green deployment traditionally is standing up and tearing down entire systems, you don’t have to do an entire system if you can do a subsystem instead so long as that subsystem is self-contained. The idea behind the rollout strategy is the same: You use a load balancer to transition traffic from one complete system to another. Since we’re talking infrastructure, the more traditional version needs to be modified by thinking of subsystems. For example, if you’re replacing the load balancer as well, you would stand up the green load balancer, point it at the rest of the blue deployment, make the switch of the traffic to the green load balancer, and then complete the move from the rest of the blue deployment to the green deployment. Then, finally, the blue load balancer and the rest of the blue deployment can be decommissioned.

Canary Deployments

Now, a blue-green deployment isn’t the only upgrade strategy, and it’s not always the best one for a cloud-native system. There’s also a strategy called a canary deployment. In a canary deployment, you stand up new infrastructure and move traffic over in small increments, such as 5% of overall traffic. That small slice of traffic is considered a “canary,” an indicator of failure or success of such a move. While it’s often used in application deployment as a way to gather user feedback on an application change, this strategy is also useful for infrastructure as it highlights issues that only occur under true, random load before they become a problem for a full deployment. This strategy is fairly popular in microservices architectures and with platforms like Kubernetes as it’s much easier to implement than with more traditional, VM-based systems.

Rolling Deployments

The final upgrade strategy we’ll consider is called a rolling deployment. In a rolling deployment, each element of a system is replaced one at a time, with each new instance being checked before decommissioning and removing the old instance. That check is called a health check on some platforms, and the basic idea is that the deployment tool sends a request to the new element and waits for a response that indicates the system is functional and responsive as expected. Once the health check on the new element clears, the old one is removed, and the deployment tool “rolls” to the next element in the system being updated. We find this one in many Kubernetes-based applications, and it can work well for cloud-native infrastructure, as well.

In the next article in this series, we’ll explore how these three strategies are similar and different for infrastructure as code, and why you would use one over the other. Stay tuned!

In the next parts of the series, we’ll try these deployment strategies with Pulumi, exploring how to use code to define each kind with a test system. Watch this space!

Meanwhile, while you’re waiting, we did a few videos on this topic over at PulumiTV, like this one on blue-green deployments with Pulumi and Python on GCP.

Organizational Patterns: A Developer Portal

Matty Stratton — Thu, 30 Dec 2021 11:00:39 -0600

Using Pulumi is more than just writing code and components. In addition to common software development practices, there are also a number of success patterns related to how your company or team builds and deploys Pulumi programs to successfully build, deploy, and manage your infrastructure and applications. In this continuation of a series, I will explore one of these patterns - using the Pulumi Automation API to create a developer portal.

It is becoming more and more common to consider building out centralized web-based “portals” for developers (and other team members) to provision the infrastructure that their services and applications require. In many ways, the pattern of a developer portal takes portions of both the Single Infra Repo and Automation Team patterns covered in previous posts. The difference is that the developer portal is a single, centralized web application that can be used by multiple teams and organizations to provision infrastructure and applications, rather than writing the Pulumi programs themselves.

Depending upon the needs of the various teams, the developer portal could include options to deploy virtual machines into the cloud of their choice (or the organization’s standard cloud; perhaps the choice isn’t…a choice!), to provision cloud resources such as virtual networks, security groups, database instances, or even deployments to a kubernetes cluster.

Generally speaking, the developer portal is created and maintained by a central platform team. This is the team who knows the success patterns/standards that are employed by the organization, and can implement these patterns in the developer portal. The platform team is also responsible for maintaining the infrastructure that the developer portal requires.

One of the great advantages of providing a single portal for provisioning these resources is that all the “other stuff” that is often required (besides the resources from the hyperscaler) can be handled by the developer portal. For example, if the portal is used to provision a virtual machine, the code behind it can register the virtual machine with the company’s infrastructure management system.

A great example of what is possible in a portal can be seen with Komal Ali’s example, the Self Service Infrastructure Platyform. This is example code only, but it shows some of the capabilities of a developer portal.

In the example, the web portal is created using a combination of Python, Flask and Jinja templates (along with the Pulumi Automation API) to provide push-button access to deploy cloud resources according to the standards and patterns of the organization. The code snippets below illustrate some of the concepts (this code is not meant to be run, but rather to illustrate the concepts - the full example code can be found at github.com/komalali/self-service-platyform).

from flask import (Blueprint, current_app, request, flash,
 redirect, url_for, render_template)

import pulumi
import pulumi_aws as aws
import pulumi.automation as auto
import os
from pathlib import Path

bp = Blueprint("virtual_machines", __name__, url_prefix="/vms")
instance_types = ['c5.xlarge', 'p2.xlarge', 'p3.2xlarge']

In the initial snippet above, we are setting up what we need for the Flask application, but the key items to make note of are including the pulumi_aws and pulumi.automation packages, which will be used for the cloud specific resources and automation.


def create_pulumi_program(keydata: str, instance_type=str):
 ami = aws.ec2.get_ami(most_recent=True,
 owners=["amazon"],
 filters=[aws.GetAmiFilterArgs(name="name", values=["*amzn2-ami-minimal-hvm*"])])

 group = aws.ec2.SecurityGroup('web-secgrp',
 description='Enable SSH access',
 ingress=[aws.ec2.SecurityGroupIngressArgs(
 protocol='tcp',
 from_port=22,
 to_port=22,
 cidr_blocks=['0.0.0.0/0'],
 )])

Continuing on, we define the Pulumi program, and set up some of the standard settings that we consider our defaults and that are not selectable by the user.


 server = aws.ec2.Instance('dlami-server',
 instance_type=instance_type,
 vpc_security_group_ids=[group.id],
 key_name=keypair.id,
 ami=ami.id)

 pulumi.export('public_dns', server.public_dns)

This final snippet is what creates the actual instance, and it exports the public DNS of the instance so that it can be accessed by the user and presented in the portal.

Conclusion

This pattern has been implemented and deployed at many larger enterprises, where the platform team has the resources to dedicate to managing the portal as an actual product. This is a key part to remember - the developer portal is a product whose consumers are the developers and other teams in the organization, and it needs to be managed as such!

Don’t forget to check out the previous two posts in this series, Organizational Patterns - A Single Infra Repo and Organizational Patterns - An Automation Team".

Improving the GitOps Pipeline with the Pulumi Operator

David Flanagan — Fri, 24 Dec 2021 17:08:06 +0000

Check out version 2.0 of the Pulumi Kubernetes Operator.

This time last year, I presented Applying the Law of Demeter to GitOps at GitOps Days 2020. The Law of Demeter is a design principle, proposed in 1988, which encourages loose coupling between systems. During this session, I wanted the audience to understand and be able to identify when their applications and continuous delivery pipelines have too much knowledge of the platform in which they’re going to run. As an industry, we’re seeing a great deal of momentum towards Platform Engineering and with this comes a Broca divide, a strict division of responsibilities: to build a platform and to consume a platform.

Instead, the platform will provide everything the application needs to function. We’ve seen a lot of this before, notably via the 12-factor manifesto.

Over the course of the last year I’ve collected my thinking more and more around this pattern, and I condensed it down into a single tweet:

#GitOps isn’t a decision of push vs pull for continuous delivery. It’s more deciding whether your application tells the environment how to deploy itself, or whether the environment tells your application how to run.

You want the latter. You want GitOps.

So what does this all mean in a more practical sense?

Ingress Example

Time and time again I see the same violation of the Law of Demeter with Kubernetes and GitOps: Ingress.

Application delivery teams require Ingress resources within Kubernetes to expose their applications to their customers. Often, Helm or Kustomize is used to provide bases that can be extended for each environment. We can see an example of the boundary violation in the Kustomize examples. I’m not calling out Kustomize here; Pulumi also has examples of this pattern.

Unfortunately, this pattern has become quite common. Leaking environment information down the stack is what we’re trying to avoid, but doing so can be rather tricky.

To understand this challenge in concrete terms, let’s look at an example of a simple Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: minimal-ingress
 annotations:
 nginx.ingress.kubernetes.io/rewrite-target: /
spec:
 rules:
 - host: staging.example.org
 http:
 paths:
 - path: /testpath
 pathType: Prefix
 backend:
 service:
 name: test
 port:
 number: 80

The violation here is host: staging.example.org inside of the application team resources. We now have a constraint binding the platform and the application together. If we continue down this path, the platform team can’t launch new environments without first reaching out to the application teams to ensure they have a base that can provide the correct domain name for these resources.

What we really want is for the platform team to be able to launch platforms whenever they want, increasing their ability to iterate and experiment with the platform. At the same time, by removing the bases and variance from the application teams, we ensure that they have a single blueprint for how to deploy their application. Win/Win.

Sadly, this is a difficult problem to solve; though not impossible.

Pulumi Operator

The Pulumi Operator hit major milestone in October this year, releasing version 1.0.

The Pulumi Operator allows us to leverage the power of Pulumi at the application team level for GitOps. This bond between Pulumi for building platforms and Pulumi for consuming platforms means we can have a strictly typed exchange of “knowns” between our platform and our application that happens at execution time.

Let’s imagine, in TypeScript, that we have the following versioned interface:

export interface Environment {
 name: string;
 baseDomain: string;
 serviceNamespace: string;
 ingressEnabled: boolean;
 networkPoliciesEnforced: boolean;
}

With this contract, the Platform Engineering team can ensure that every cluster, or namespace, has a ConfigMap called environment can be consumed by the Pulumi Operator. In this, the Pulumi program, the GitOps delivery pipeline from the application team, can begin to grok the environment to which it is being deployed. With this knowledge, it can begin to augment the manifests before they’re applied to the cluster.

Let’s go back to our Ingress example and see how this could be done.

First, let’s assume our Platform team have deployed a ConfigMap like such:

apiVersion: v1
kind: ConfigMap
metadata:
 name: environment
data:
 v1: |
 {
 "name": "production",
 "baseDomain": "pulumi.com",
 "serviceNamespace": "utils",
 "ingressEnabled": true,
 "networkPoliciesEnforced": true
 }

The specification for what an environment looks like will differ from organization to organization, but this hopefully provides enough context for some of the things you may wish to encapsulate this way.

Next, our application team defines their delivery via Pulumi, too. The first thing they do as part of the delivery is load the environment config. This data can then be used to augment any resources that need to be environment aware.

import * as k8s from "@pulumi/kubernetes";

const cm = k8s.core.v1.ConfigMap.get("environment", "environment", {});

export interface Environment {
 name: string;
 baseDomain: string;
 serviceNamespace: string;
 ingressEnabled: boolean;
 networkPoliciesEnforced: boolean;
}

const env = cm.data.apply((data): Environment => JSON.parse(data["v1"]));

const ingress = new k8s.networking.v1.Ingress(name,
 {
 metadata: {
 labels: labels,
 },
 spec: {
 rules: [
 {
 host: pulumi.interpolate`mysvc.${env.baseDomain}`,
 http: {
 paths: [
 {
 path: "/",
 backend: {
 serviceName: serviceName,
 servicePort: "http",
 }
 },
 ],
 },
 }
 ]
 }
 }
);

That’s it! We’ve got isolated delivery pipelines for both the platform team and the application team, leveraging GitOps and continuous reconciliation, without leaking any information across the stacks.

Pulumi is already seeing considerable adoption and success with platform teams, and I’m excited to see further adoption of the Pulumi Operator and application teams harnessing the power of GitOps with Pulumi.

Organizational Patterns: An Automation Team

Matty Stratton — Wed, 22 Dec 2021 11:49:08 -0600

I want to approach this pattern with a bit of caution - there are patterns and anti-patterns related to having a single central team focused on automation.

The anti-pattern is to have an “automation team” within your organization who is tasked with creating all infrastructure as code for the entire organization. This is a common pattern, but it is generally rife with issues.

Understanding how to build and deploy a service or application requires a certain amount of domain knowledge of that service. If you have a single set of people across your entire company who need to write Pulumi programs for each and every application or service, you are now expecting that small group to not only be experts in Pulumi, but also every bit of infrastructure and software applications (both third party and created internally) across your organization.

A more successful pattern is to have the automation team act as enablers and providers for the other groups within the company. This is a team of subject matter experts in using Pulumi, and not only do they have the knowledge to build and deploy infrastructure, but they are familiar with various patterns and good practices for how to build and deploy infrastructure.

There are different mechanisms for how this automation team can enable others. One way is to embed them temporarily within a product team to work alongside them to build their automation. Then, when the product team is done, the automation team member can go and help another squad.

Depending upon the size and skills of the company and this group, they may also be responsible for creating training materials for the other groups. These are usually based upon existing Pulumi training materials, but customized for the specific needs and culture of the company.

Reusable components

This central automation team also often provides reusable components that might be shared across multiple development teams. This helps the individual teams to not have to “reinvent the wheel” and to bring in these components for their own particular needs.

As a simple example, a component could be created (and published as a library to be consumed by product teams) that defines a Kubernetes deployment.

import * as deploy from "@mycorp/deploy";

const kuard = new deploy.Deployment("kuard", {
 image: "gcr.io/kuard-project/kuard:latest",
 replicas: 1,
});

What’s happened here in our (incomplete!) example is that the published package only exposes the parts that the team needs to change; the rest is provided by the automation team.

The application team don’t need to duplicate values (such as ports) because the abstractions cater for them. The application team also didn’t need to apply any metadata, health checks, and environment variables because it can be inferred through convention, increasing velocity across the organization. When values are provided by the application team, they override convention and they can still iterate and experiment, providing feedback to the automation team.

Final tips

One of the more important recommendations for this pattern is to avoid trying to create a “mandate” for all teams for adoption. The automation team provides capabilities, such as templated CI/CD pipelines, reusable components, and training materials, but it is still up to the individual teams to decide how to use them.

Conclusion

This is a pattern that works well at a larger organization, where it would be beyond the scope of a single group to be able to wrap their understanding around hundreds of services. However, it is a pattern that can be applied to smaller organizations as well!

Watch for the next post in this series, where we will dig into some other patterns! Or revisit the first post, Organizational Patterns - A Single Infra Repo.

Pulumi Blog: Best Practices

How to Test Infrastructure as Code

Why should you test infrastructure as code?

What types of IaC tests are there?

How do you write unit tests for Pulumi?

Setting up mocks

The program under test

How do you run integration tests against real cloud resources?

Using the Automation API

Using the Go integration framework

How does policy as code complement IaC testing?

How is testing IaC in Pulumi different from Terraform?

How to set up IaC testing with Pulumi

Frequently asked questions

What is infrastructure as code testing?

Can you unit test IaC without deploying to the cloud?

Do you need real cloud credentials to test Pulumi?

What’s the difference between unit and integration tests for IaC?

How do you mock cloud resources in Pulumi?

What test frameworks work with Pulumi?

How do you test Pulumi in CI/CD?

Is policy as code the same as testing?

How is Pulumi testing different from Terratest?

How do you tear down resources after integration tests?

Where to go next

Five Stacks Before Lunch: The Parallel Coding Playbook for Pulumi

2x is prompting, 10x is plumbing

The five pillars

What changes when the agents are touching infrastructure

The five catches, infra edition

Where to start, this afternoon

Five stacks before lunch

The Dark Factory Pattern for Infrastructure: Running Pulumi Lights-Out

What a dark factory actually is

The wall between generator and validator

Why infrastructure is the harder version

The interesting work

A four-phase rollout

Phase 1: better context, this afternoon

Phase 2: spec-driven with holdouts, this week

Phase 3: take the human out of the merge

Phase 4: lights out

What could go wrong

Where to start

Treating Prompts Like Code: A Content Engineer's AI Workflow

The real problem AI solves

Treating prompts like code

The skill catalog

Honest limitations

Lessons to share

What’s next

GitOps Best Practices I Wish I Had Known Before

1. Git is your single source of truth (no, really)

2. Declarative over imperative, always

3. Pull-based deployments are the way

4. Separate app code from deployment config (when it hurts not to)

5. Use directories, not branches, for environments

6. Validate before you merge

7. Tag with commit SHAs, not “latest”

8. Automate drift detection and reconciliation

9. Practice progressive delivery

10. Policy-as-code: your automated guardrails

11. Bridge your IaC and GitOps (don’t choose one)

12. Be pragmatic, not dogmatic

Final thoughts

How Pulumi fits into your GitOps workflow

The Claude Skills I Actually Use for DevOps

The mechanic, the tools, and the manual

Why skills instead of MCPs

Teaching Claude to write Pulumi like an expert

The official Pulumi skills

Making Claude a monitoring expert by default

Kubernetes configuration that actually passes security review

Debugging like a senior engineer

Catching security issues before they ship

Incident response before you need it

General-purpose DevOps and SRE skills

Vetting the skills you install

Putting it together

What changes

1. Drift Detection with `pulumi refresh`

2. State Reconciliation with `pulumi refresh`

3. Drift Remediation with `pulumi up`