Pulumi Blog: Testing

How to Test Infrastructure as Code

Alex Leventer — Tue, 30 Jun 2026 14:00:00 -0700

IaC testing means validating your infrastructure code the same way you test application software—unit tests with mocked cloud providers that run in milliseconds, integration tests that deploy and inspect real resources, and policy checks that enforce compliance rules on every preview and deploy. Together, these layers catch misconfigurations before they reach production.

Untested infrastructure code is a liability. A missing tag in a security group rule, an S3 bucket with public read access, a misconfigured IAM policy—any of these can slip through code review and land in production. Infrastructure bugs are often harder to debug than application bugs, because the feedback loop is slow (deploy, observe, destroy) and the blast radius is large (an outage, a security incident, a surprise AWS bill).

The good news: Pulumi is built on general-purpose programming languages, which means you can test infrastructure code with the exact same tools and frameworks you already use to test application code. No new language to learn. No separate toolchain. Just pytest, Mocha, or go test—pointing at your infrastructure program.

This guide walks through all three testing layers with complete, runnable examples.

Why should you test infrastructure as code?

The case for testing application code is well-established. The case for testing IaC is just as strong—arguably stronger, because the consequences of an undetected bug are more severe.

Catch misconfigurations before they deploy. A unit test that checks whether a security group allows SSH from 0.0.0.0/0 will catch that misconfiguration in milliseconds, before any cloud API call is made. The same test in CI catches it before the code is merged.

Prevent drift and outages. Integration tests that deploy to a staging environment and validate runtime behavior—does the HTTP endpoint return 200? does the database accept connections?—give you confidence that an infrastructure change doesn’t silently break a dependent service.

Refactor safely. Well-tested infrastructure code can be restructured, modularized, and upgraded without fear. When all your properties, tags, and security rules are asserted in tests, you know immediately if a refactor breaks a constraint.

Enforce security and compliance automatically. Policy as code lets you encode your organization’s security rules as executable tests. Mandatory policies block non-compliant deployments entirely; advisory policies surface warnings for human review. These rules run on every pulumi preview and pulumi up—not just at audit time.

Ship infrastructure at software speed. Teams that test IaC ship changes faster, with less fear. The CI pipeline becomes a confidence engine rather than a deployment gate.

The annual DORA State of DevOps research is consistently associated with a pattern among elite-performing engineering teams—those that deploy frequently and recover quickly from failures: comprehensive automated testing across the stack. Infrastructure is no longer exempt from that standard.

What types of IaC tests are there?

IaC testing mirrors the classic software test pyramid: a wide base of fast, cheap unit tests, a smaller layer of integration tests against real infrastructure, and policy checks that enforce rules throughout.

Layer	What it tests	Speed	Cost	Cloud credentials?
Unit tests	Resource properties, tags, security rules, naming conventions	Milliseconds	Free	No
Integration tests	Full stack lifecycle, runtime behavior, cross-resource dependencies	Minutes	Real cloud spend	Yes
Policy as code	Compliance rules, security baselines, organizational standards	Milliseconds	Free	No (runs at preview time)

graph TD
A["🔒 Policy as Code<br/>Compliance &amp; guardrails<br/>Runs at preview time<br/>Milliseconds · Free"] --> B
B["🧪 Integration Tests<br/>Real cloud deployments<br/>Full lifecycle validation<br/>Minutes · Cloud spend"] --> C
C["⚡ Unit Tests<br/>Mocked providers<br/>Property assertions<br/>Milliseconds · Free"]
style A fill:#8A3FFC,color:#fff,stroke:#6B2FD4
style B fill:#0EA5E9,color:#fff,stroke:#0284C7
style C fill:#10B981,color:#fff,stroke:#059669

Start with unit tests—they’re free, fast, and catch the most common mistakes. Add integration tests for critical paths (the deployment pipeline, data infrastructure, security-sensitive resources). Layer in policy as code for compliance requirements that must never regress.

How do you write unit tests for Pulumi?

Pulumi unit tests replace the communication channel between the Pulumi program and the cloud provider with mocks. When the program registers a resource (e.g., aws.ec2.SecurityGroup(...)), the mock intercepts the call, returns dummy state, and never makes a real cloud API call. Tests run in the same process, in the same language, using the same test runner you already use.

Setting up mocks

The import-order rule: You must set up mocks before importing your Pulumi program. If you import the program first, the Pulumi runtime initializes without mocks and will attempt real cloud calls.

# test_infra.py
import unittest
import pulumi

class MyMocks(pulumi.runtime.Mocks):
 def new_resource(self, args: pulumi.runtime.MockResourceArgs):
 # Return [id, state]. State keys must be camelCase.
 return [args.name + "_id", args.inputs]

 def call(self, args: pulumi.runtime.MockCallArgs):
 return {}

# Set mocks BEFORE importing the program
pulumi.runtime.set_mocks(MyMocks())
import infra # import program AFTER setting mocks

class TestInfra(unittest.TestCase):

 @pulumi.runtime.test
 def test_no_public_ssh(self):
 """Security group must not expose SSH to the public internet."""
 def check_security_group(args):
 ingress_rules = args[0]
 for rule in (ingress_rules or []):
 for cidr in (rule.get("cidrBlocks") or []):
 self.assertNotEqual(
 cidr, "0.0.0.0/0",
 "Security group must not expose port 22 to the internet."
 )
 return pulumi.Output.all(infra.group.ingress).apply(check_security_group)

 @pulumi.runtime.test
 def test_server_has_tags(self):
 """EC2 instance must have required tags."""
 def check_tags(args):
 tags = args[0]
 self.assertIn("Environment", tags, "Missing 'Environment' tag")
 self.assertIn("Name", tags, "Missing 'Name' tag")
 return pulumi.Output.all(infra.server.tags).apply(check_tags)

if __name__ == "__main__":
 unittest.main()

Run with: python -m unittest discover -v

// infra.test.ts
import * as pulumi from "@pulumi/pulumi";
import { expect } from "chai";

// Set mocks BEFORE dynamic import of the program
pulumi.runtime.setMocks(
 {
 newResource: (args: pulumi.runtime.MockResourceArgs): { id: string; state: any } => {
 // State keys must be camelCase
 return { id: `${args.name}-id`, state: args.inputs };
 },
 call: (args: pulumi.runtime.MockCallArgs) => ({ ...args.inputs }),
 },
 "my-project",
 "test",
 false // preview = false
);

describe("Infrastructure", function () {
 let infra: typeof import("./index");

 before(async function () {
 // Dynamic import AFTER setting mocks
 infra = await import("./index");
 });

 it("must not expose SSH to the internet", function (done) {
 pulumi.all([infra.group.ingress]).apply(([ingress]) => {
 for (const rule of ingress ?? []) {
 for (const cidr of rule.cidrBlocks ?? []) {
 expect(cidr).to.not.equal(
 "0.0.0.0/0",
 "Security group must not expose port 22 to the internet"
 );
 }
 }
 done();
 });
 });

 it("must have required tags", function (done) {
 pulumi.all([infra.server.tags]).apply(([tags]) => {
 expect(tags).to.include.keys("Environment", "Name");
 done();
 });
 });
});

Run with: mocha -r ts-node/register infra.test.ts

// infra_test.go
package infra_test

import (
 "testing"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/stretchr/testify/assert"
)

// MockResourceMonitor implements pulumi.ResourceMonitor for unit tests.
func TestNoPublicSSH(t *testing.T) {
 err := pulumi.RunErr(func(ctx *pulumi.Context) error {
 infra, err := NewInfra(ctx)
 assert.NoError(t, err)

 infra.Group.Ingress.ApplyT(func(rules []ec2.SecurityGroupIngress) error {
 for _, rule := range rules {
 for _, cidr := range rule.CidrBlocks {
 assert.NotEqual(t, "0.0.0.0/0", cidr,
 "Security group must not expose port 22 to the internet")
 }
 }
 return nil
 })
 return nil
 }, pulumi.WithMocks("my-project", "test", mocks{}))
 assert.NoError(t, err)
}

type mocks struct{}

func (mocks) NewResource(args pulumi.MockResourceArgs) (string, map[string]interface{}, error) {
 return args.Name + "_id", args.Inputs, nil
}
func (mocks) Call(args pulumi.MockCallArgs) (map[string]interface{}, error) {
 return args.Args, nil
}

Run with: go test ./... -v

// InfraTests.cs
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Testing;
using Xunit;

public class InfraTests
{
 [Fact]
 public async Task NoPublicSSH()
 {
 var resources = await Testing.RunAsync<MyStack>();

 var sg = resources.OfType<Pulumi.Aws.Ec2.SecurityGroup>().First();
 var ingress = await sg.Ingress.GetValueAsync();

 foreach (var rule in ingress)
 {
 foreach (var cidr in rule.CidrBlocks)
 {
 Assert.NotEqual("0.0.0.0/0", cidr);
 }
 }
 }

 [Fact]
 public async Task ServerHasRequiredTags()
 {
 var resources = await Testing.RunAsync<MyStack>();

 var server = resources.OfType<Pulumi.Aws.Ec2.Instance>().First();
 var tags = await server.Tags.GetValueAsync();

 Assert.True(tags.ContainsKey("Environment"), "Missing 'Environment' tag");
 Assert.True(tags.ContainsKey("Name"), "Missing 'Name' tag");
 }
}

Run with: dotnet test

// InfraTest.java
package myproject;

import com.pulumi.aws.ec2.Instance;
import com.pulumi.aws.ec2.SecurityGroup;
import com.pulumi.test.Mocks;
import com.pulumi.test.Mocks.CallArgs;
import com.pulumi.test.Mocks.ResourceArgs;
import com.pulumi.test.Mocks.ResourceResult;
import com.pulumi.test.PulumiTest;
import com.pulumi.test.TestOptions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import static org.junit.jupiter.api.Assertions.*;

class InfraTest {

 // Implement Mocks to intercept resource creation. State keys must be camelCase.
 static class MyMocks implements Mocks {
 @Override
 public CompletableFuture<ResourceResult> newResourceAsync(ResourceArgs args) {
 var state = new HashMap<>(args.inputs);
 return CompletableFuture.completedFuture(
 ResourceResult.of(Optional.of(args.name + "_id"), state));
 }

 @Override
 public CompletableFuture<Map<String, Object>> callAsync(CallArgs args) {
 return CompletableFuture.completedFuture(Map.of());
 }
 }

 // PulumiTest.cleanup() must run after every test to reset the runtime state.
 @AfterEach
 void cleanup() {
 PulumiTest.cleanup();
 }

 @Test
 void serverHasRequiredTags() {
 var result = PulumiTest
 .withMocks(new MyMocks())
 .withOptions(TestOptions.builder()
 .projectName("project").stackName("stack").preview(false)
 .build())
 .runTest(ctx -> new Infra());

 var instance = result.resources().stream()
 .filter(r -> r instanceof Instance)
 .map(r -> (Instance) r)
 .findFirst()
 .orElseThrow(() -> new AssertionError("EC2 instance not found"));

 var tags = PulumiTest.extractValue(instance.tags())
 .orElseThrow(() -> new AssertionError("Tags must not be null"));
 assertTrue(tags.containsKey("Environment"), "Missing 'Environment' tag");
 assertTrue(tags.containsKey("Name"), "Missing 'Name' tag");
 }

 @Test
 void noPublicSshExposed() {
 var result = PulumiTest
 .withMocks(new MyMocks())
 .withOptions(TestOptions.builder()
 .projectName("project").stackName("stack").preview(false)
 .build())
 .runTest(ctx -> new Infra());

 var group = result.resources().stream()
 .filter(r -> r instanceof SecurityGroup)
 .map(r -> (SecurityGroup) r)
 .findFirst()
 .orElseThrow(() -> new AssertionError("Security group not found"));

 var rules = PulumiTest.extractValue(group.ingress());
 for (var rule : rules) {
 for (var cidr : rule.cidrBlocks()) {
 assertNotEquals("0.0.0.0/0", cidr,
 "Security group must not expose SSH to the internet");
 }
 }
 }
}

Run with: mvn test (JUnit 5)

The program under test

Both examples above test this program:

# infra.py
import pulumi
from pulumi_aws import ec2

group = ec2.SecurityGroup("web-secgrp",
 ingress=[{
 "protocol": "tcp",
 "from_port": 80,
 "to_port": 80,
 "cidrBlocks": ["0.0.0.0/0"],
 }],
)

server = ec2.Instance("web-server",
 instance_type="t2.micro",
 ami="ami-0b0ea68c435eb488d",
 vpc_security_group_ids=[group.id],
 tags={"Name": "web-server", "Environment": "dev"},
)

// index.ts
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

export const group = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [{
 protocol: "tcp",
 fromPort: 80,
 toPort: 80,
 cidrBlocks: ["0.0.0.0/0"],
 }],
});

export const server = new aws.ec2.Instance("web-server", {
 instanceType: "t2.micro",
 ami: "ami-0b0ea68c435eb488d",
 vpcSecurityGroupIds: [group.id],
 tags: { Name: "web-server", Environment: "dev" },
});

// main.go
package infra_test

import (
 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

type Infra struct {
 Group *ec2.SecurityGroup
 Server *ec2.Instance
}

func NewInfra(ctx *pulumi.Context) (*Infra, error) {
 group, err := ec2.NewSecurityGroup(ctx, "web-secgrp", &ec2.SecurityGroupArgs{
 Ingress: ec2.SecurityGroupIngressArray{
 ec2.SecurityGroupIngressArgs{
 Protocol: pulumi.String("tcp"),
 FromPort: pulumi.Int(80),
 ToPort: pulumi.Int(80),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 },
 })
 if err != nil {
 return nil, err
 }

 server, err := ec2.NewInstance(ctx, "web-server", &ec2.InstanceArgs{
 InstanceType: pulumi.String("t2.micro"),
 Ami: pulumi.String("ami-0b0ea68c435eb488d"),
 VpcSecurityGroupIds: pulumi.StringArray{group.ID()},
 Tags: pulumi.StringMap{
 "Name": pulumi.String("web-server"),
 "Environment": pulumi.String("dev"),
 },
 })
 if err != nil {
 return nil, err
 }

 return &Infra{Group: group, Server: server}, nil
}

// MyStack.cs
using Pulumi;
using Pulumi.Aws.Ec2;
using Pulumi.Aws.Ec2.Inputs;

public class MyStack : Stack
{
 public MyStack()
 {
 var group = new SecurityGroup("web-secgrp", new SecurityGroupArgs
 {
 Ingress = new[]
 {
 new SecurityGroupIngressArgs
 {
 Protocol = "tcp", FromPort = 80, ToPort = 80,
 CidrBlocks = new[] { "0.0.0.0/0" },
 },
 },
 });

 var server = new Instance("web-server", new InstanceArgs
 {
 InstanceType = "t2.micro",
 Ami = "ami-0b0ea68c435eb488d",
 VpcSecurityGroupIds = new[] { group.Id },
 Tags = new InputMap<string>
 {
 { "Name", "web-server" },
 { "Environment", "dev" },
 },
 });
 }
}

// Infra.java
package myproject;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.aws.ec2.SecurityGroup;
import com.pulumi.aws.ec2.SecurityGroupArgs;
import com.pulumi.aws.ec2.Instance;
import com.pulumi.aws.ec2.InstanceArgs;
import com.pulumi.aws.ec2.inputs.SecurityGroupIngressArgs;
import java.util.List;
import java.util.Map;

public class Infra {
 public final SecurityGroup group;
 public final Instance server;

 public Infra() {
 this.group = new SecurityGroup("web-secgrp",
 SecurityGroupArgs.builder()
 .ingress(SecurityGroupIngressArgs.builder()
 .protocol("tcp")
 .fromPort(80)
 .toPort(80)
 .cidrBlocks("0.0.0.0/0")
 .build())
 .build());

 this.server = new Instance("web-server",
 InstanceArgs.builder()
 .instanceType("t2.micro")
 .ami("ami-0b0ea68c435eb488d")
 .vpcSecurityGroupIds(group.id().applyValue(List::of))
 .tags(Map.of("Name", "web-server", "Environment", "dev"))
 .build());
 }
}

Key detail: State property keys in the mock’s new_resource/newResource return value must be camelCase (cidrBlocks, not cidr_blocks), regardless of the language you’re writing in. This is how Pulumi serializes properties internally.

How do you run integration tests against real cloud resources?

Unit tests tell you whether your resource definitions are correct. Integration tests tell you whether your infrastructure actually works. Integration tests deploy real cloud resources, run assertions against them, and then destroy everything—giving you end-to-end confidence at the cost of actual cloud spend and time.

Using the Automation API

The Pulumi Automation API lets you drive pulumi up, pulumi destroy, and every other Pulumi CLI operation programmatically, from within a test. You define the stack inline (using a function that runs your Pulumi program) or point it at an existing project directory.

# test_integration.py
import pytest
import pulumi
import pulumi_aws as aws
from pulumi import automation as auto

def create_bucket_program():
 """The Pulumi program to deploy during the test."""
 bucket = aws.s3.Bucket("test-bucket",
 tags={"Environment": "test", "ManagedBy": "pulumi"},
 )
 pulumi.export("bucket_name", bucket.id)
 pulumi.export("bucket_arn", bucket.arn)

@pytest.fixture(scope="module")
def deployed_stack():
 """Deploy the stack for the test module and destroy it when done."""
 stack = auto.create_or_select_stack(
 stack_name="integration-test",
 project_name="bucket-test",
 program=create_bucket_program,
 )
 stack.set_config("aws:region", auto.ConfigValue(value="us-west-2"))

 # Install provider plugins
 stack.workspace.install_plugin("aws", "v7.34.0")

 # Deploy
 up_result = stack.up(on_output=print)

 yield up_result.outputs

 # Teardown: destroy everything after the test module completes
 stack.destroy(on_output=print)
 stack.workspace.remove_stack("integration-test")

def test_bucket_was_created(deployed_stack):
 """The S3 bucket should have been created with the expected name prefix."""
 bucket_name = deployed_stack["bucket_name"].value
 assert bucket_name.startswith("test-bucket"), f"Unexpected bucket name: {bucket_name}"

def test_bucket_has_tags(deployed_stack):
 """Verify the bucket was created with required tags (via AWS SDK)."""
 import boto3
 s3 = boto3.client("s3", region_name="us-west-2")
 bucket_name = deployed_stack["bucket_name"].value
 tagging = s3.get_bucket_tagging(Bucket=bucket_name)
 tag_map = {t["Key"]: t["Value"] for t in tagging["TagSet"]}
 assert tag_map.get("Environment") == "test"
 assert tag_map.get("ManagedBy") == "pulumi"

Run with: pytest test_integration.py -v

// integration.test.ts
import { LocalWorkspace, ConfigValue, UpResult } from "@pulumi/pulumi/automation";
import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";
import { expect } from "chai";

describe("S3 Bucket Integration", function () {
 this.timeout(300_000); // 5 min — real deploys take time

 let outputs: UpResult["outputs"];
 let stackName = "integration-test-ts";

 before(async function () {
 const stack = await LocalWorkspace.createOrSelectStack({
 stackName,
 projectName: "bucket-test",
 program: async () => {
 const bucket = new aws.s3.Bucket("test-bucket", {
 tags: { Environment: "test", ManagedBy: "pulumi" },
 });
 return {
 bucketName: bucket.id,
 bucketArn: bucket.arn,
 };
 },
 });

 await stack.setConfig("aws:region", { value: "us-west-2" });
 await stack.workspace.installPlugin("aws", "v7.34.0");

 const upResult = await stack.up({ onOutput: console.log });
 outputs = upResult.outputs;
 });

 after(async function () {
 const stack = await LocalWorkspace.selectStack({ stackName, projectName: "bucket-test", program: async () => {} });
 await stack.destroy({ onOutput: console.log });
 await stack.workspace.removeStack(stackName);
 });

 it("creates a bucket with the right name prefix", function () {
 const name = outputs["bucketName"].value as string;
 expect(name).to.match(/^test-bucket/);
 });
});

Run with: mocha -r ts-node/register integration.test.ts

// integration_test.go
package integration_test

import (
 "context"
 "fmt"
 "os"
 "testing"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/auto"
 "github.com/pulumi/pulumi/sdk/v3/go/auto/optup"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/stretchr/testify/assert"
)

func TestBucketIntegration(t *testing.T) {
 ctx := context.Background()

 stack, err := auto.NewStackInlineSource(ctx, "integration-test", "bucket-test",
 func(ctx *pulumi.Context) error {
 bucket, err := s3.NewBucket(ctx, "test-bucket", &s3.BucketArgs{
 Tags: pulumi.StringMap{
 "Environment": pulumi.String("test"),
 "ManagedBy": pulumi.String("pulumi"),
 },
 })
 if err != nil {
 return err
 }
 ctx.Export("bucketName", bucket.ID())
 return nil
 },
 )
 assert.NoError(t, err)

 _ = stack.SetConfig(ctx, "aws:region", auto.ConfigValue{Value: "us-west-2"})
 upResult, err := stack.Up(ctx, optup.ProgressStreams(os.Stdout))
 assert.NoError(t, err)

 t.Cleanup(func() {
 _, _ = stack.Destroy(ctx)
 _ = stack.Workspace().RemoveStack(ctx, "integration-test")
 })

 name := fmt.Sprintf("%v", upResult.Outputs["bucketName"].Value)
 assert.Contains(t, name, "test-bucket")
}

Run with: go test ./... -v -timeout 10m

// IntegrationTests.cs
using System;
using System.Threading.Tasks;
using Pulumi.Automation;
using Xunit;

public class IntegrationTests : IAsyncLifetime
{
 private WorkspaceStack _stack = null!;
 private UpResult _upResult = null!;

 public async Task InitializeAsync()
 {
 var program = PulumiFn.Create(() =>
 {
 var bucket = new Pulumi.Aws.S3.Bucket("test-bucket", new()
 {
 Tags = new() { ["Environment"] = "test", ["ManagedBy"] = "pulumi" },
 });
 return new System.Collections.Generic.Dictionary<string, object?>
 {
 ["bucketName"] = bucket.Id,
 };
 });

 _stack = await LocalWorkspace.CreateOrSelectStackAsync(new InlineProgramArgs(
 projectName: "bucket-test",
 stackName: "integration-test",
 program: program
 ));

 await _stack.SetConfigAsync("aws:region", new ConfigValue("us-west-2"));
 _upResult = await _stack.UpAsync(new UpOptions { OnStandardOutput = Console.WriteLine });
 }

 public async Task DisposeAsync()
 {
 await _stack.DestroyAsync(new DestroyOptions { OnStandardOutput = Console.WriteLine });
 await _stack.Workspace.RemoveStackAsync("integration-test");
 }

 [Fact]
 public void BucketNameHasExpectedPrefix()
 {
 var name = _upResult.Outputs["bucketName"].Value.ToString();
 Assert.StartsWith("test-bucket", name);
 }
}

Run with: dotnet test

// IntegrationTest.java
package myproject;

import com.pulumi.Context;
import com.pulumi.automation.LocalWorkspace;
import com.pulumi.automation.UpOptions;
import com.pulumi.automation.DestroyOptions;
import com.pulumi.automation.ConfigValue;
import com.pulumi.automation.WorkspaceStack;
import org.junit.jupiter.api.*;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.*;

@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class IntegrationTest {

 private static final String PROJECT_NAME = "bucket_test_project";
 private static final String STACK_NAME = "integration-test";

 private WorkspaceStack stack;
 private Map<String, com.pulumi.automation.OutputValue> outputs;

 @BeforeAll
 void setUp() throws Exception {
 stack = LocalWorkspace.createOrSelectStack(PROJECT_NAME, STACK_NAME, (Context ctx) -> {
 var bucket = new com.pulumi.aws.s3.Bucket("test-bucket",
 com.pulumi.aws.s3.BucketArgs.builder()
 .tags(Map.of("Environment", "test", "ManagedBy", "pulumi"))
 .build());
 ctx.export("bucketName", bucket.id());
 });

 stack.workspace().installPlugin("aws", "v7.34.0");
 stack.setConfig("aws:region", new ConfigValue("us-west-2"));

 var upResult = stack.up(UpOptions.builder().onStandardOutput(System.out::println).build());
 outputs = upResult.outputs();
 }

 @AfterAll
 void tearDown() throws Exception {
 if (stack != null) {
 stack.destroy(DestroyOptions.builder().onStandardOutput(System.out::println).build());
 stack.workspace().removeStack(STACK_NAME);
 }
 }

 @Test
 void bucketNameHasExpectedPrefix() {
 var name = (String) outputs.get("bucketName").value();
 assertTrue(name.startsWith("test-bucket"), "Unexpected bucket name: " + name);
 }
}

Run with: mvn test

Using the Go integration framework

If you prefer Go-based integration tests, Pulumi ships a purpose-built Go test framework at github.com/pulumi/pulumi/pkg/v3/testing/integration. It handles the full create → validate → destroy lifecycle for any Pulumi program (written in any language), with an ExtraRuntimeValidation hook for assertions against deployed state.

func TestS3Website(t *testing.T) {
 integration.ProgramTest(t, &integration.ProgramTestOptions{
 Dir: path.Join("..", "my-s3-website"),
 Config: map[string]string{"aws:region": "us-west-2"},
 ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
 url := stack.Outputs["websiteUrl"].(string)
 resp, _ := http.Get("http://" + url)
 assert.Equal(t, 200, resp.StatusCode)
 },
 })
}

Best practice: Always destroy test stacks in a finally block or test teardown fixture. Leaked integration test stacks are a common cause of unexpected cloud spend.

How does policy as code complement IaC testing?

Unit and integration tests validate what your infrastructure does. Policy as code enforces what your infrastructure must conform to—security standards, tagging conventions, compliance requirements—and it runs automatically on every pulumi preview and pulumi up.

Pulumi Policies are currently supported in Python and TypeScript. You write policies in the same language as the rest of your stack, without learning a DSL. A mandatory policy blocks a deployment entirely when violated; an advisory policy surfaces a warning but allows the deploy to proceed.

# __main__.py in your policy pack directory
from pulumi_policy import (
 EnforcementLevel,
 PolicyPack,
 ReportViolation,
 ResourceValidationArgs,
 ResourceValidationPolicy,
)

REQUIRED_TAGS = {"Name", "Environment", "Owner"}

def require_tags(args: ResourceValidationArgs, report_violation: ReportViolation):
 if args.resource_type.startswith("aws:"):
 tags = args.props.get("tags") or {}
 missing = REQUIRED_TAGS - set(tags.keys())
 if missing:
 report_violation(
 f"Resource is missing required tags: {', '.join(sorted(missing))}"
 )

def no_public_s3(args: ResourceValidationArgs, report_violation: ReportViolation):
 if args.resource_type == "aws:s3/bucket:Bucket":
 if args.props.get("acl") in ("public-read", "public-read-write"):
 report_violation("S3 buckets must not have a public ACL.")

PolicyPack(
 name="aws-security-baseline",
 policies=[
 ResourceValidationPolicy(
 name="require-tags",
 description="All AWS resources must have Name, Environment, and Owner tags.",
 enforcement_level=EnforcementLevel.MANDATORY,
 validate=require_tags,
 ),
 ResourceValidationPolicy(
 name="no-public-s3",
 description="S3 buckets must not be publicly accessible via ACL.",
 enforcement_level=EnforcementLevel.MANDATORY,
 validate=no_public_s3,
 ),
 ],
)

// index.ts in your policy pack directory
import * as aws from "@pulumi/aws";
import { PolicyPack, validateResourceOfType } from "@pulumi/policy";

new PolicyPack("aws-security-baseline", {
 policies: [
 {
 name: "require-tags",
 description: "All EC2 instances must have Name, Environment, and Owner tags.",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.ec2.Instance, (instance, args, reportViolation) => {
 const required = ["Name", "Environment", "Owner"];
 const tags = instance.tags || {};
 for (const tag of required) {
 if (!tags[tag]) {
 reportViolation(`Missing required tag: ${tag}`);
 }
 }
 }),
 },
 {
 name: "no-public-s3",
 description: "S3 buckets must not have a public ACL.",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => {
 if (bucket.acl === "public-read" || bucket.acl === "public-read-write") {
 reportViolation("S3 bucket must not have a public ACL.");
 }
 }),
 },
 ],
});

Running a policy pack

# Scaffold a new policy pack from a template
mkdir my-policies && cd my-policies
pulumi policy new aws-typescript

# Run policy checks against a stack
pulumi preview --policy-pack ./my-policies
pulumi up --policy-pack ./my-policies

# Or publish to Pulumi Cloud and enforce organization-wide
pulumi policy publish

As Joe Duffy, Pulumi CEO, puts it: “The smartest agent in the world still needs guardrails, audit trails, and policy enforcement to be trusted with production systems at scale.”

Policy checks complement unit tests: unit tests verify your intent (“I wrote tags into this resource”), while policies enforce your standards (“all resources must have these tags, regardless of what any one developer intended”). For complex compliance environments, publish policies to Pulumi Cloud and assign them via policy groups to enforce them across the stacks or accounts you choose—including organization-wide.

How is testing IaC in Pulumi different from Terraform?

The fundamental difference is language cohesion. With Pulumi, you write infrastructure, tests, and policies all in the same general-purpose language. With Terraform, your infrastructure is HCL but your tests must be something else—either Go (Terratest) or a constrained HCL DSL (terraform test).

	Pulumi	Terratest	`terraform test` (≥ 1.6)
Language for tests	Same language as infra (Python, TypeScript, Go, C#, Java); YAML programs tested via Automation API from any of those languages	Always Go (regardless of infra language)	HCL DSL (`.tftest.hcl`)
Unit tests with mocks	Yes — `pulumi.runtime.set_mocks()`, no cloud credentials needed	No — no mock layer for cloud providers	Limited — `mock_provider` (v1.7+) is HCL-declarative, no programmatic logic
Integration testing	Automation API (any language) or Go framework	Full-featured but Go-only	Yes, via `command = apply` run blocks
Policy / guardrails	Pulumi Policies: Python or TypeScript, runs at preview time	External tools (checkov, tfsec, OPA/Rego — separate toolchain)	Sentinel (enterprise) or external
Execution speed	Unit: milliseconds; integration: depends on cloud	Minutes (real deployments by default)	Plan-only: fast; apply: minutes
Learning curve	Use existing language skills and test frameworks	Requires Go knowledge and understanding of `go test`	Low if you know HCL; limited expressiveness

Terratest is battle-tested and widely adopted, particularly for Terraform module testing. It offers deep integrations with AWS, GCP, Azure, and Kubernetes. Its limitation is that it has no mock layer for cloud providers, so most test suites deploy real infrastructure, which makes them slower and costlier to run on every commit.

terraform test has improved significantly in Terraform 1.6 and 1.7. The command = plan mode avoids real deployments, and mock_provider enables basic unit-like testing. But expressiveness is bounded by HCL’s declarative model—no general-purpose programming, no third-party assertion libraries, and a fixed set of run/assert test constructs.

With Pulumi, there’s no context switch. The same developer who writes the infrastructure writes the tests in the same language, using the same IDE, the same debugger, and the same CI pipeline. That cohesion is a productivity multiplier.

Test your infrastructure with Pulumi

Write unit tests, integration tests, and policy checks in the language you already use, then run them on every pull request.

Get started

How to set up IaC testing with Pulumi

Follow these steps to add a complete test suite to a Pulumi project. The examples use Python, but the pattern is identical in TypeScript, Go, C#, or Java.

Step 1: Install Pulumi and choose a language SDK

# Install Pulumi CLI
curl -fsSL https://get.pulumi.com | sh

# Create a new project (or use an existing one)
pulumi new aws-python --name my-project --stack dev

Step 2: Add a test framework

# Python
pip install pytest pytest-asyncio

# TypeScript
npm install --save-dev mocha ts-node @types/mocha chai @types/chai

# Go uses the built-in testing package — no additional dependencies

Step 3: Implement the Mocks class

Create mocks.py (or equivalent) with your Mocks subclass. Keep it simple for unit tests: return [args.name + "_id", args.inputs] from new_resource.

Step 4: Write unit tests

Create test_infra.py in your project root. Import mocks first, then your program. Write test functions that use @pulumi.runtime.test and pulumi.Output.all(...).apply(...) to assert resource properties.

Step 5: Add integration tests

Create test_integration.py with a pytest fixture that creates a stack, calls stack.up(), yields outputs for assertions, and calls stack.destroy() in teardown. Point this at a dedicated test cloud account.

Step 6: Create a policy pack

mkdir policies && cd policies
pulumi policy new aws-python
# Edit __main__.py to add your rules

Step 7: Wire it all into CI

# .github/workflows/test.yml (GitHub Actions excerpt)
jobs:
 unit-tests:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: pulumi/actions@v6
 - run: pip install -r requirements.txt
 - run: python -m pytest tests/unit/ -v

 integration-tests:
 runs-on: ubuntu-latest
 needs: unit-tests
 environment: staging
 steps:
 - uses: actions/checkout@v4
 - uses: pulumi/actions@v6
 - run: pip install -r requirements.txt
 - env:
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 run: python -m pytest tests/integration/ -v --timeout=300

Frequently asked questions

What is infrastructure as code testing?

Infrastructure as code testing is the practice of writing automated checks that validate your IaC programs before they deploy to production. Just like application testing, it includes unit tests (fast, no cloud credentials, mock the provider), integration tests (real cloud deployments in ephemeral environments), and policy checks (rules that enforce compliance standards on every preview and deploy).

Can you unit test IaC without deploying to the cloud?

Yes. Pulumi’s mocking framework intercepts all cloud API calls and replaces them with in-process stubs. Unit tests run entirely in memory—no cloud credentials, no network calls, no cost. A suite of 50 unit tests runs in under a second. This is the primary reason to write unit tests: fast, free, deterministic feedback on every commit.

Do you need real cloud credentials to test Pulumi?

Only for integration tests. Unit tests use Pulumi’s mock runtime and require no cloud credentials at all. Policy checks also run without credentials at pulumi preview time. Integration tests that use the Automation API do need credentials—they deploy real resources—so they’re typically run in a CI stage with access to a dedicated test account, not on every PR.

What’s the difference between unit and integration tests for IaC?

Unit tests validate resource properties, configurations, and relationships using mocked cloud providers—they run in milliseconds with no cloud credentials. Integration tests deploy actual resources to a real cloud environment, validate runtime behavior (does the endpoint respond? does the database accept connections?), and then destroy everything. Unit tests are cheap and frequent; integration tests are slower, cost real cloud spend, and run less often.

How do you mock cloud resources in Pulumi?

Implement pulumi.runtime.Mocks (Python) or call pulumi.runtime.setMocks() (TypeScript/JavaScript) before importing your Pulumi program. The new_resource method receives every resource registration and returns a fake [id, state] pair. The call method handles data-source lookups. Set mocks with pulumi.runtime.set_mocks(MyMocks()) and then import your program—the runtime will route all resource registrations through your mock instead of making real API calls.

What test frameworks work with Pulumi?

Any language-native test framework works. For Python: pytest (recommended) or unittest. For TypeScript/JavaScript: Mocha, Jest, or Vitest. For Go: the standard testing package. For C#: NUnit or xUnit. For Java: JUnit. The @pulumi/pulumi/automation SDK and pulumi.automation module integrate naturally with all of them. YAML is Pulumi’s declarative configuration language; because it has no control flow or executable statements, you cannot write test logic in YAML itself. YAML programs are tested by pointing the Automation API from Python, TypeScript, Go, C#, or Java at the project directory and asserting on the stack outputs.

How do you test Pulumi in CI/CD?

Split your test suite into two CI stages. Stage 1 (unit tests): runs on every PR commit, no cloud credentials required, fast. Stage 2 (integration tests): runs on merge to main or on a schedule, requires cloud credentials for a dedicated test account, uses the Automation API to deploy ephemeral stacks. The Pulumi GitHub Actions integration simplifies credential injection and stack management in both stages.

Is policy as code the same as testing?

Policy as code is complementary to—but distinct from—testing. Tests validate what your program produces (resource A has property B); policies enforce what your resources must conform to (all resources must have tag X). Policies run automatically at preview time and block non-compliant deployments at deploy time. Together, they create defense in depth: tests verify intent, policies enforce standards regardless of intent.

How is Pulumi testing different from Terratest?

Terratest typically deploys real infrastructure—it has no mock layer for cloud providers—so most test runs provision actual cloud resources, incurring real cost and taking 5–30 minutes. Tests must be written in Go regardless of what language your infrastructure uses. Pulumi’s unit testing framework runs in milliseconds with no cloud credentials, uses your existing language and test runner, and provides true mock isolation. For integration testing, Pulumi’s Automation API offers a similar model to Terratest but in any language.

How do you tear down resources after integration tests?

In Python pytest, use a module-scoped fixture that yields after stack.up() and calls stack.destroy() in the teardown. In TypeScript Mocha, call stack.destroy() in an after() hook. In Go, use defer stack.Destroy(ctx, ...). Always use a try/finally pattern (or equivalent in your language) so teardown runs even when a test assertion fails. Pulumi Cloud also has built-in stack TTL and auto-destroy features for extra safety.

Where to go next

Testing IaC is not an all-or-nothing investment. Start by adding a handful of unit tests to your most critical stack—the one that manages your production VPC, your IAM roles, or your data infrastructure. Pick three to five properties that, if wrong, would cause an outage or a security incident. Write tests for those. Run them in CI. Expand from there.

When you’re ready to go deeper:

Testing overview — Pulumi’s full testing documentation
Unit testing guide — detailed mock API reference with Go and C# examples
Automation API integration testing — advanced patterns (multi-stack tests, config injection, parallel test execution)
Policy as code authoring — writing and publishing Pulumi Policies
Pulumi vs. Terraform — a full comparison of the two platforms

If you’re migrating from Terraform, the Pulumi conversion tool translates your existing HCL to Python, TypeScript, Go, C#, and more, including your test infrastructure. Your Terratest or terraform test suites can be ported to Pulumi’s native test runner—using your existing language—as part of the migration.

Get started with Pulumi for free →

Self-Verifying AI Agents: Vercel's Agent-Browser in the Ralph Wiggum Loop

Engin Diri — Tue, 20 Jan 2026 00:00:00 +0000

In my previous post about the Ralph Wiggum technique, Claude Code built a complete serverless URL shortener on AWS. The setup included Playwright MCP for end-to-end testing. It worked. But I kept wondering if there was something better for AI-driven browser automation. Then Vercel released agent-browser, and I had to try it.

Why browser automation matters for AI coding

When an AI coding agent builds a frontend, someone has to verify it works. Without browser automation, that someone is you. The AI finishes and says “done,” but you can’t trust that claim until you open a browser and click around yourself.

Browser automation changes this. The AI verifies its own work. It builds a component, launches a browser, tests the interaction, confirms behavior matches expectations. If something breaks, it fixes the code and tests again. The validation loop runs without you.

This matters more as AI agents take on larger tasks. A quick component fix might not need automated testing. But when an agent builds an entire dashboard from scratch, you need proof that the deployed application actually works.

The context problem with browser automation

Playwright MCP gives you powerful browser control, but it comes with overhead. Every screenshot, every DOM snapshot, every accessibility tree adds tokens to your context window. GitHub issue #889 documents a 6x token increase between versions 0.0.30 and 0.0.32. Users report single screenshots consuming over 15,000 tokens. Some exhausted their entire five-hour token allocation in just a few automation steps.

The problem is verbose output. Full accessibility trees contain every element on the page with all their properties. Console message logging adds more. Each piece of information might be useful for debugging, but together they overwhelm the context window.

This creates a frustrating tradeoff. You want rich browser interaction for thorough testing, but that richness costs tokens. Fewer tokens means fewer iterations before hitting limits.

Vercel’s “less is more” philosophy

Vercel ran into something similar with their D0 text-to-SQL agent. Their research documents what happened when they reduced tool complexity.

The original architecture used 17 specialized tools. Each tool handled one specific operation: create tables, insert rows, run queries, validate schemas. The approach seemed logical. Specialized tools should produce better results than general ones.

The numbers told a different story. With 17 tools, they got 80% success (4 out of 5 queries), averaging 274.8 seconds and ~102,000 tokens. The worst case took 724 seconds, burned 145,463 tokens, and still failed.

Then they rebuilt with just two tools: ExecuteCommand and ExecuteSQL. Success jumped to 100%. Average execution dropped to 77.4 seconds (3.5x faster). Token usage fell to ~61,000 (37% reduction). The worst case took 141 seconds, used 67,483 tokens, and succeeded.

Vercel’s takeaway: “We were constraining reasoning because we didn’t trust the model to reason.” Fewer tools meant the model could think more freely about how to accomplish tasks. The simplicity reduced confusion and context waste.

Applying “less is more” to browser automation

agent-browser takes the same approach. Instead of separate tools for clicking, typing, scrolling, and navigating, it has a unified CLI with one clever idea: the snapshot + refs system.

When you request a page snapshot, agent-browser returns something like this:

- button "Sign In" [ref=e1]
- textbox "Email" [ref=e2]
- textbox "Password" [ref=e3]
- link "Documentation" [ref=e4]

Those @e1, @e2 references are stable element identifiers. To click the sign-in button, you run click @e1. No CSS selectors. No XPath expressions. No waiting for the DOM to stabilize. The reference points to that exact element.

This cuts out the verbosity of full accessibility trees. You get just enough information to understand the page structure and interact with elements. The AI can reason about what it sees without drowning in metadata.

Installing agent-browser

Setup is simple:

npm install -g agent-browser && agent-browser install

This installs the CLI and downloads a bundled browser. You don’t need to install any additional browsers.

For Claude Code integration, install the official skill from the agent-browser package. You can either copy it from node_modules:

cp -r node_modules/agent-browser/skills/agent-browser .claude/skills/

Or download it directly:

mkdir -p .claude/skills/agent-browser
curl -o .claude/skills/agent-browser/SKILL.md \
 https://raw.githubusercontent.com/vercel-labs/agent-browser/main/skills/agent-browser/SKILL.md

The skill gives Claude Code better context than just running agent-browser --help.

Unlike Playwright MCP, there’s no server configuration. Agent-browser runs as a standalone CLI that Claude Code invokes directly through bash.

The experiment: adding analytics

I wanted a realistic test scenario, so I added an analytics dashboard to the url-shortener-saas project from my previous post. The feature tracks click events with metadata (browser, device, country, referrer) and displays them through interactive charts.

This follows the same “Ralph Wiggum” workflow from my previous post: write a feature-prompt.md that describes what you want, then let Claude Code implement, deploy with Pulumi, and verify with browser automation. Here’s a snippet from the feature prompt:

## Backend Requirements

### New DynamoDB Table: Analytics Events

Create a new DynamoDB table `url-shortener-analytics-{stack}` with:
- Partition Key: shortCode (String)
- Sort Key: timestamp (String, ISO 8601 format)
- Attributes: browser, deviceType, country, referrer, etc.

### New Lambda: analytics.ts

GET /api/analytics/{shortCode}
- Returns timeline, browsers, devices, countries, referrers
- Query params: from, to, granularity

## Success Criteria

1. `pulumi up` deploys successfully without errors
2. All existing E2E tests still pass
3. New analytics E2E tests pass
4. Charts render with real data after generating test clicks

Claude Code reads this, writes the Lambda handlers, updates the Pulumi infrastructure code, and deploys. The infrastructure changes are straightforward:

// Analytics events table
const analyticsTable = new aws.dynamodb.Table("analytics-table", {
 name: `${projectName}-analytics-${stack}`,
 billingMode: "PAY_PER_REQUEST",
 hashKey: "shortCode",
 rangeKey: "timestamp",
 attributes: [
 { name: "shortCode", type: "S" },
 { name: "timestamp", type: "S" },
 ],
 globalSecondaryIndexes: [{
 name: "by-date",
 hashKey: "shortCode",
 rangeKey: "timestamp",
 projectionType: "ALL",
 }],
});

After pulumi up succeeds, the agent needs to verify the deployed feature works. That’s where browser automation comes in. The feature prompt specifies verification requirements:

## End-to-End Verification

After deployment, use the `/agent-browser` skill to verify:
1. Analytics link appears in navigation
2. Analytics page loads without errors
3. Charts render (timeline, browser, device, geographic)
4. URL-specific analytics work from dashboard
5. Click data appears after generating test clicks

The test suite covered six scenarios: homepage load, URL shortening, dashboard view, analytics navigation, analytics overview, and date filter functionality. Nothing fancy, just the basics you’d want to verify after deploying.

I ran the exact same tests with both Playwright MCP and agent-browser to see how much context each consumed.

Agent-browser workflow in practice

Here’s what the CLI interaction looked like during my test run.

Navigate to the deployed application:

$ agent-browser open https://d1232drths1aav.cloudfront.net
✓ Snip.ly - Modern URL Shortener
 https://d1232drths1aav.cloudfront.net/

Snapshot to see available elements (the -i flag filters to interactive elements only):

$ agent-browser snapshot -i
- link "S Snip.ly" [ref=e1]
- link "Home" [ref=e2]
- link "Dashboard" [ref=e3]
- link "Analytics" [ref=e4]
- link "Docs" [ref=e5]
- button "Switch to dark mode" [ref=e6]
- link "Get Started" [ref=e7]
- textbox "Paste your long URL here..." [ref=e8]
- button "Shorten URL" [ref=e9]

That entire homepage snapshot is 280 characters. Playwright MCP returned 8,247 characters for the same page.

Fill the URL input and click the shorten button:

$ agent-browser fill @e8 "https://example.com/agent-browser-e2e-test"
✓ Done

$ agent-browser click @e9
✓ Done

Each action confirmation is 6 characters. Playwright MCP returns the full page state after every click - 12,891 characters when I clicked the shorten button.

Navigate to analytics:

$ agent-browser click @e4
✓ Done

$ agent-browser wait --load networkidle && agent-browser snapshot -i
✓ Done
- link "S Snip.ly" [ref=e1]
- link "Home" [ref=e2]
- link "Dashboard" [ref=e3]
- link "Analytics" [ref=e4]
- button "Last 7 days" [ref=e8]
- button "Last 30 days" [ref=e9]
- button "Last 90 days" [ref=e10]
- link "1 analytics-e2e-test https://example.com/analytics-test-verification 5" [ref=e11]
- link "2 test-url https://www.example.com/this-is-a-very-long-url... 4" [ref=e12]

The analytics snapshot shows date filter buttons and top URLs with click counts. 385 characters versus Playwright MCP’s 4,127.

Test the date filter:

$ agent-browser click @e9
✓ Done

$ agent-browser screenshot analytics-30-day-filter.png
✓ Screenshot saved to analytics-30-day-filter.png

The ref-based workflow is deterministic. Each command operates on a specific element from the snapshot. No guessing about selectors.

The numbers: 82.5% context reduction

Same six tests, both tools:

Metric	Playwright MCP	Agent-Browser	Reduction
Total response characters	31,117	5,455	82.5%
Largest single response	12,891	2,847	77.9%
Average response size	3,112	328	89.5%
Homepage snapshot	8,247	280	96.6%
Dashboard snapshot	12,891	2,847	77.9%

The difference is what each tool returns after every action.

Playwright MCP returns the full accessibility tree after every click:

### Page state
- Page URL: https://d1232drths1aav.cloudfront.net/dashboard
- Page Title: Snip.ly - Modern URL Shortener
- Page Snapshot:
- generic [ref=e2]:
 - banner [ref=e3]:
 - generic [ref=e4]:
 - link "S Snip.ly" [ref=e5] [cursor=pointer]:
 - /url: /
 - generic [ref=e6]: S
 - generic [ref=e7]: Snip.ly
 - navigation [ref=e8]:
 - link "Home" [ref=e9] [cursor=pointer]:
 - /url: /
 - text: Home
 ...
[12,891 characters continues...]

Agent-browser returns:

✓ Done

6 characters versus 12,891 for the same button click.

Six tests consumed ~31K characters with Playwright MCP versus ~5.5K with agent-browser. At roughly 4 characters per token, that’s ~7,800 tokens versus ~1,400. An AI agent could run 5.7x more tests in the same context budget.

What I noticed

Past the numbers, working with agent-browser felt different in ways that are hard to quantify.

The compact snapshots changed the rhythm of iteration. A typical page fit in a few hundred tokens instead of thousands, so Claude Code could run more test cycles before hitting limits. I spent less time worrying about context and more time actually testing.

Element refs removed a frustration I’d had with Playwright. CSS selectors break when someone changes a class name or restructures the DOM. With agent-browser, a button is just @e9. It doesn’t care about styling. That predictability was a relief.

What worked well:

Snapshots stay small enough that you don’t think about them
Refs mean no selector debugging
The CLI just works with Claude Code’s bash tool
No MCP server to configure

Agent-browser is early. Documentation is thin, and I read the source more than once to figure out edge cases.

Where I hit friction:

Modals that appear after API calls needed manual wait logic
Playwright’s waiting mechanisms are more mature
Hidden or dynamically loaded elements sometimes didn’t show up without explicit waits

For my URL shortener tests, agent-browser used fewer tokens per cycle, and the ref-based approach was more predictable than selector-driven automation.

Playwright MCP still wins on depth. Network interception, multi-tab handling, PDF generation, better waiting logic—if you need those, you need Playwright. For complex browser automation, it’s the more capable tool.

Deploy what your agent builds

Pulumi provisions the infrastructure your coding agent builds and tests, so a single pulumi up takes each verified change to the cloud.

Get started

When to use each

Agent-browser fits long autonomous sessions where context budget matters, basic navigation and verification tasks, and setups where you want to skip MCP configuration. CLI-based skills also avoid the schema overhead—tool definitions for multi-tool MCPs eat tokens even when idle.

Playwright MCP fits when you need the advanced stuff: network interception, PDF generation, multi-tab workflows, or sophisticated synchronization. It’s also the right choice if you have existing Playwright test suites. If the full MCP schema feels heavy, Playwright skills offer a lighter wrapper.

Start with agent-browser for AI validation loops. Move to Playwright when you outgrow it.

Test-Driven Infrastructure Development with Pulumi and Jest

Christian Nunciato — Mon, 13 Jun 2022 00:00:00 +0000

When I was a kid growing up in Southern California, there was a phone number you could call to find out what time it was. It was a local number, 853-1212 (easy to remember as the arrangement of the numbers on the keypad made a capital T), and I used it all the time, to set my watch, adjust the alarm clock, fix the display on the VCR. I don’t recall the last time I used it, probably sometime in the mid ’90s, but I do remember clearly the sound of the voice at the other end of the line.

Services like these were called speaking clocks (although I never called them that; to me, it was just “calling Time”), and they date back to the early 1930s or so. In the U.S., most were powered by a machine called an Audichron that used mechanical drums to play back the time of day, often with a little advertisement or the current temperature to go along with it.

By the mid-2000s, though, most of these locally run services had been shut down — by then, we had clocks baked into our phones — and today, only a handful of Audichrons survive. Thanks to the National Institute of Standards and Technology, however, you can still call a phone number to get the time, and while the voice might not be the same, and long distance rates will apply, it’s definitely there, and you can use it. So on the off chance you happen to find yourself with no idea what time it is and only an analog phone line in reach, fear not — old-school telephone tech has your back. For now. Assuming you remember the number.

Recalling all this stuff did make me wonder, though, what a more modern version of a speaking clock might look like. So in this post, we’re going to build one ourselves. We won’t use an actual phone number, but we will use Pulumi and AWS — and because we want to do it right, we’ll take a test-driven approach to developing the infrastructure with Jest, the JavaScript testing framework. We’ll use TypeScript and Node.js for everything, focus on unit tests, and when we’re done, we’ll have a single, serverless, browser-friendly HTTPS endpoint that returns an MP3 audio stream that speaks the current time.

Let’s get started.

Sketching it out

The first thing we’ll need is a runtime environment — someplace to run some server-side JavaScript that can render and deliver an audio file. Until recently, the easiest way to get an HTTP endpoint up and running on AWS has generally been with AWS Lambda and API Gateway, using Lambda to run the requisite code and API Gateway to expose the Lambda to the internet. Pulumi Crosswalk actually makes this really easy, too — but with the release of AWS Lambda Function URLs this April, we now have another option, one that doesn’t need API Gateway at all.

A Lambda Function URL is just what it sounds like: a URL that exposes a Lambda function. Specifically, it’s an AWS cloud resource that consists of a few properties that tell AWS whether to allow anonymous access to the function or to protect it with AWS IAM, and optionally, you can also provide a few cross-origin resource-sharing (CORS) rules to provide (or restrict) access by websites running on other domains.

For an app like this one, a Function URL is a good fit, as it’s simple, quick to deploy (and update), and it gives us all the control we need to get the job done. Architecturally, then, our little app will look something like this:

That about covers the infrastructure — but what about the sound?

As it happens, Google Translate offers a publicly accessible text-to-speech API that can, as long as you don’t overuse it, render a string to MP3 audio. We can use that API to generate the time-of-day message, tack the sound of a beep onto the end of it (for maximum nostalgic effect), and voilà: instant, infinitely scalable, cloud-native speaking clock.

Plan in place, we can kick things off by creating a new Pulumi project.

Create a new TypeScript project

Start by creating a new AWS TypeScript project in the usual way:

$ mkdir audichron-2022 && cd audichron-2022
$ pulumi new aws-typescript

Step through the prompts to create a new stack (you’ll only need one stack for this project), and when the new-project wizard completes, clear out the contents of index.ts entirely, as we’ll be building this program entirely from the ground up.

Install and configure Jest

Since we’re working with TypeScript, we can use ts-jest, which conveniently brings Jest along for the ride:

$ npm install ts-jest @types/jest --save-dev

Jest requires a bit of configuration, so add a new file at the root of the project, called jest.config.ts, with the following code:

import type { Config } from "@jest/types";

const config: Config.InitialOptions = {
 preset: "ts-jest",
};

export default config;

With Jest now installed and configured, and its helpers (like describe and expect) declared globally, you can create a new spec file to hold your tests. Add another file at the root of the project called index.spec.ts, and give it the following temporary content:

import "jest";

describe("My speaking clock", () => {
 it ("works", () => {
 expect(true).toBeTruthy;
 });
});

Lastly, in package.json, add a scripts block with a helper to make running the tests a little easier:

{
 "name": "audichron-2022",
 "devDependencies": {
 "@types/node": "^14",
 "ts-jest": "^28.0.3"
 },
 ...
 "scripts": {
 "test": "jest"
 }
}

Now run the tests, just to make sure you’ve got everything wired up correctly. If all goes well, you should see that the temporary spec we just added happily passes:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 ✓ works

With that, it’s time to get to start writing some real tests.

Start with some failing tests

As devoted practitioners of test-driven development, we’re going to start by writing some unit tests — specifically, some failing unit tests that we can fix by writing the Pulumi code to make them pass.

Recall that our design requires just two cloud resources: a Lambda function and a Lambda function URL. For the function itself, we’ll use the high-level aws.lambda.CallbackFunction resource, one of my favorites for managing for Lambdas because it requires only one property: an inline JavaScript function to handle the event that triggers the Lambda.

For the URL resource — the eventual triggerer of that event — you’ll use an aws.lambda.FunctionURL configured to make the Lambda publicly accessible (i.e., available to anyone on the internet) and embeddable on any domain, making it easy, for example, to embed the URL as the src attribute of an HTML5 audio element.

So to get things going, we’ll need to:

Configure Pulumi to mock AWS resources. We’re writing unit tests, after all, and we want them to be fast, so we’ll need to prevent those tests from provisioning any real cloud infrastructure.
Import your Pulumi resource declarations — the function and the function URL — into index.spec.ts so you can reference them in tests.

In that light, replace the contents of index.spec.ts with the following code, which, after some setup to switch Pulumi into unit-testing mode, contains just one test: a check to make sure that the function URL is configured with the right authorizationType. I’ve included some comments to help explain what each line is doing:

import * as pulumi from "@pulumi/pulumi";
import "jest";

describe("My speaking clock", () => {

 // Define the infra variable as a type whose shape matches that of the
 // to-be-defined resources module.
 // https://www.typescriptlang.org/docs/handbook/2/typeof-types.html
 let infra: typeof import("./resources");

 beforeAll(() => {

 // Put Pulumi in unit-test mode, mocking all calls to cloud-provider APIs.
 pulumi.runtime.setMocks({

 // Mock requests to provision cloud resources and return a canned response.
 newResource: (args: pulumi.runtime.MockResourceArgs): {id: string, state: any} => {

 // Here, we're returning a same-shaped object for all resource types.
 // We could, however, use the arguments passed into this function to
 // customize the mocked-out properties of a particular resource based
 // on its type. See the unit-testing docs for details:
 // https://www.pulumi.com/docs/iac/concepts/testing/unit
 return {
 id: `${args.name}-id`,
 state: args.inputs,
 };
 },

 // Mock function calls and return whatever input properties were provided.
 call: (args: pulumi.runtime.MockCallArgs) => {
 return args.inputs;
 },
 });
 });

 beforeEach(async () => {

 // Dynamically import the resources module.
 // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#dynamic_imports
 infra = await import("./resources");
 });

 describe("function URL", () => {

 it("is publicly accessible", (done) => {
 infra.timeURL.authorizationType.apply(authType => {
 try {
 expect(authType).toBe("NONE");
 done();
 } catch (err) {
 done(err);
 }
 });
 });
 });
});

We can’t quite run these tests just yet, though, as they rely on a module called resources that we haven’t yet defined. Let’s do that now by creating a new file, resources.ts, containing some stubs for the cloud resources we’ve elected to use:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

export const timeFunction = new aws.lambda.CallbackFunction("time-function", {
 callback: () => {
 // Leave the callback implementation empty for now.
 },
});

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "", // Leave this empty, too. (We want our first test to fail.)
});

Now run the tests, and you should see that the public-accessibility test fails — which shouldn’t be surprising, as we haven’t yet supplied an acceptable value:

$ npm test

FAIL ./index.spec.ts
 My speaking clock
 function URL
 ✕ is publicly accessible

 ● My speaking clock › function URL › is publicly accessible

 expect(received).toBe(expected) // Object.is equality

 Expected: "NONE"
 Received: ""

Now let’s turn that sad-looking ❌ into a ✅.

Make the tests pass

Update the FunctionUrl resource to change its authorizationType input to NONE (meaning for anonymous access, no authorization required):

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
- authorizationType: "",
+ authorizationType: "NONE",
});

Once you do that, you should be able to run the test suite again, and this time, see it pass:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible

Excellent: you’re now well on your way to writing more tests with Jest — and we’ll do that in a moment. But first, let’s digress momentarily to explore a few other ways of writing Jest specs with Pulumi.

Tidy up the specs

If you’ve written JavaScript tests before, particularly with tools like Jest, Jasmine, and Mocha, there’s a good chance you’ve gotten used to writing those tests in a particular way. And if you’re like me, you might’ve raised a bit of an eyebrow when you saw how that first test was written. Here it is again, this time with comments that capture the questions I might’ve had myself if I were seeing this code for the first time:

it("is publicly accessible", (done) => {
 infra.timeURL.authorizationType.apply(auth => { // Why do I have to use .apply() here?
 try { // What happens if I don't try/catch?
 expect(auth).toBe("NONE"); // Is it possible to use expect() by itself?
 done(); // Do I have to call done()? What if I don't?
 } catch (err) { // How can I make these tests more readable?
 done(err); // Is all of this code necessary?
 }
 });
});

This is a perfectly valid test, of course; we know because we just ran it. Still, you might wonder, especially since we’re only checking one property, whether this test might be written a bit more concisely — something more like this, maybe:

it("is publicly accessible", () => {
 expect(infra.timeURL.authorizationType).toBe("NONE"); // Alas, nope.
});

The main reason is the asynchronous nature of the objects we’re testing. An aws.lambda.FunctionUrl, after all, isn’t just a plain ol’ JavaScript object — it’s a Pulumi CustomResource, a representation of an eventual resource running in the cloud, with properties that may not be known until sometime after the resource has been deployed. While it’s true we provided authorizationType as a regular JavaScript string, by the time we attempt to read it in the test, it’s been transformed into a pulumi.Output<string> — an asynchronous value, like a promise, that has to be unwrapped with a call to apply() before it can be read.

However, just adding apply and calling expect from within its callback doesn’t quite work, either:

it("is publicly accessible", () => {
 infra.timeURL.authorizationType.apply(authType => {
 expect(authType).toBe("NONE"); // Also nope.
 });
});

Although that might not be obvious at first, since on the surface, the test appears to pass:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible

It isn’t until you force the test to fail that it becomes clearer something’s gone sideways. Try changing the expected authorizationType to something other than NONE and you’ll see what I mean:

$ npm test

PASS ./index.spec.ts
 My time app
 function URL
 ✓ is publicly accessible

Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 total

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function
without a catch block, or by rejecting a promise which was not handled with .catch(). The promise
rejected with the reason "Error: expect(received).toBe(expected) // Object.is equality

Expected: "AWS_IAM"
Received: "NONE"".] {
 code: 'ERR_UNHANDLED_REJECTION'
}

All is not lost, though. We do have other options, and that UnhandledPromiseRejection offers a bit of a clue.

Turn outputs into promises

Jest actually has built-in support for writing asynchronous tests with promises, and there are a couple of ways you can write specs that use them:

Have the function passed to it (or its alias, test) return a promise containing your expectation
Pass an async function to it and await the promise in the usual way

There’s only one problem: pulumi.Outputs may indeed be promise-like, but they aren’t actual promises.

They can, however, be wrapped in promises pretty easily. I like the way Pulumi community member @aviflax handled this with a little utility function that wraps the Output and preserves the type of its underlying value. Here’s that function, followed by a couple of examples that use it:

// Convert a pulumi.Output to a promise of the same type.
function promiseOf<T>(output: pulumi.Output<T>): Promise<T> {
 return new Promise(resolve => output.apply(resolve));
}

// Our test, rewritten to have it() return a Promise<string>.
it("is publicly accessible", () => {
 return promiseOf(infra.timeURL.authorizationType).then(authType => {
 expect(authType).toBe("NONE");
 });
});

// The same test, rewritten to use async/await.
it("is publicly accessible", async () => { // <-- Don't miss that async!
 const authType = await promiseOf(infra.timeURL.authorizationType);
 expect(authType).toBe("NONE");
});

Either approach works equally well, but I’m slightly more partial to the latter for its conciseness. So for now, update index.spec.ts to add the promiseOf helper and adjust the test accordingly, then run the suite one more time to make sure all’s well:

// ...

// Convert a pulumi.Output to a promise of the same type.
function promiseOf<T>(output: pulumi.Output<T>): Promise<T> {
 return new Promise(resolve => output.apply(resolve));
}

describe("My speaking clock", () => {
 // ...

 describe("function URL", () => {

 it("is publicly accessible", async () => {
 const authType = await promiseOf(infra.timeURL.authorizationType);
 expect(authType).toBe("NONE");
 });
 });
});

Assuming you’re back in ✅ territory, let’s move on so we can finish things off.

Finish the app

We’re in good shape: we’ve got scaffolding for our Jest-powered unit tests and we’re generally happy with how they look, so it’s time to round out the rest of the app and its tests so we can deploy. We’ll go through this last part briskly, as we’ve covered most of the important stuff at this point, and pause only briefly to point out what’s important as we go.

Here’s the plan:

We’ll add one more test to make sure the function URL is CORS-friendly
We’ll add another test to make sure the URL resource is bound to the right Lambda function
We’ll add the Lambda code to generate and return the audio file
We’ll deploy, and bask in the warm glow of nostalgia

Let’s get to it.

Add the remaining tests

To index.spec.ts, add two more tests, one to ensure the function is accessible to browsers running on any domain, the other to verify the function URL resource is exposing the right function (if we didn’t, it’d be easy to change that reference by mistake and accidentally expose the wrong function):

describe("My speaking clock", () => {
 // ...

 describe("function URL", () => {
 // ...

 it("is CORS-friendly", async () => {
 const authType = await promiseOf(infra.timeURL.cors);
 expect(authType).toEqual({
 allowOrigins: ["*"],
 allowMethods: ["GET"],
 });
 });

 it("is bound to the right Lambda function", async () => {
 const timeFuncName = await promiseOf(infra.timeFunction.name);
 const timeURLFunc = await promiseOf(infra.timeURL.functionName);
 expect(timeFuncName).toEqual(timeURLFunc);
 });
 });
});

Run the tests, and you should see one failure:

$ npm test

FAIL ./index.spec.ts (14.256 s)
 My speaking clock
 function URL
 ✓ is publicly accessible (3022 ms)
 ✕ is CORS-friendly (3 ms)
 ✓ is bound to the right Lambda function (1 ms)

 Expected: {"allowMethods": ["GET"], "allowOrigins": ["*"]}
 Received: undefined

Tests: 1 failed, 2 passed, 3 total

A quick addition to the FunctionUrl in resources.ts should get you to green:

// ...
export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "NONE",
+ cors: {
+ allowOrigins: ["*"],
+ allowMethods: ["GET"],
+ },
});

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible
 ✓ is CORS-friendly
 ✓ is bound to the right Lambda function

Tests: 3 passed, 3 total

That’s it for the tests! Now for the final additions.

Produce the audio message

The last thing we’ll need to do is render some audio. For this, we’ll need to communicate with Google Translate, our text-to-speech provider of choice, and that means we’ll need an HTTP client. I generally like node-fetch, so let’s go with that:

npm install --save node-fetch@2

Now, finish the program by importing node-fetch into resources.ts, adding a function to produce a speech-friendly string indicating the current time and updating the CallbackFunction’s callback property to send the string to Google Translate to produce the desired result. Here’s the complete file, with comments, for reference:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const fetch = require("node-fetch");

// Convert the current time into a speech-friendly string.
export function getSpeechText() {
 const now = new Date();
 now.setSeconds(now.getSeconds() + 6);

 const local = new Date(now.toLocaleString("en-US", { timeZone: "America/Los_Angeles" }));
 const localHour = local.getHours();
 const localMinute = local.getMinutes();
 const h = localHour > 12 ? localHour - 12 : localHour;
 const m = localMinute < 10 ? `oh ${localMinute}` : localMinute; // So 2:03 -> "two-oh-three"
 const s = local.getSeconds();

 return `At the tone, the time will be ${h} ${m}. And ${s} seconds.`;
}

export const timeFunction = new aws.lambda.CallbackFunction("time-function", {

 // Update the Lambda callback body to convert the current time into an MP3 file.
 callback: async () => {
 const text = getSpeechText();
 const speechURL = `https://translate.google.com/translate_tts?ie=UTF-8&q=${encodeURIComponent(text)}&textLen=${text.length}&tl=en&client=tw-ob`;
 const beepURL = "https://www.pulumi.com/uploads/beep.mp3";
 const ttsResponse = await fetch(speechURL);
 const beepResponse = await fetch(beepURL);
 const speech = await ttsResponse.arrayBuffer();
 const beep = await beepResponse.arrayBuffer();

 // Tack a beep onto the end of the audio returned from Google Translate, then
 // render the whole thing as a base-64 encoded string.
 const body = Buffer.concat([Buffer.from(speech), Buffer.from(beep)]).toString("base64");

 // Return an appropriately shaped HTTP response.
 return {
 body,
 headers: { "Content-Type": "audio/mpeg" },
 isBase64Encoded: true,
 statusCode: 200,
 };
 },
});

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "NONE",
 cors: {
 allowOrigins: ["*"],
 allowMethods: ["GET"],
 },
});

// Export the public URL of our shiny new service.
export const audioURL = timeURL.functionUrl;

Finally, open index.ts (which should still be empty) and add a couple of lines to import the resources module and export the function URL as a Pulumi stack output:

import "./resources";

export { audioURL } from "./resources";

And that’s it. We’re ready to deploy.

Deploy the service

Doublecheck your AWS credentials are configured, then deploy the new service with a single pulumi up:

$ pulumi up

...
Updating (dev)

View Live: https://app.pulumi.com/cnunciato/audichron-2022/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack audichron-2022-dev created
 + ├─ aws:iam:Role time-function created
 + ├─ aws:lambda:Function time-function created
 + ├─ aws:iam:RolePolicyAttachment time-function-7cd09230 created
 + ├─ aws:iam:RolePolicyAttachment time-function-a1de8170 created
 + ├─ aws:iam:RolePolicyAttachment time-function-1b4caae3 created
 + ├─ aws:iam:RolePolicyAttachment time-function-019020e7 created
 + ├─ aws:iam:RolePolicyAttachment time-function-4aaabb8e created
 + ├─ aws:iam:RolePolicyAttachment time-function-74d12784 created
 + ├─ aws:iam:RolePolicyAttachment time-function-e1a3786d created
 + ├─ aws:iam:RolePolicyAttachment time-function-6c156834 created
 + ├─ aws:iam:RolePolicyAttachment time-function-b5aeb6b6 created
 + └─ aws:lambda:FunctionUrl time-url created

Outputs:
 audioURL: "https://52rjmmybwfwcud3nxxhosznpum0cycbw.lambda-url.us-west-2.on.aws/"

Resources:
 + 13 created

Duration: 20s

When the update completes, open the URL in your browser of choice, and you should hear the message you’ve been waiting for — one that might even make Pat Fleet proud:

$ open $(pulumi stack output audioURL)

When you’re ready, you can tear everything down with a pulumi destroy.

Wrapping up

Beyond just having a nifty (and admittedly rather silly) new way to tell time, you should have a much better sense at this point of how you can use Pulumi with Jest to write better, safer infrastructure code.

From here, there’s a bunch more you might think about next: writing more tests to cover the code we just added, exploring some additional flavors of testing in the docs, or having a look at a few examples. You’ll find the full source for this walkthrough up on GitHub as well.

Happy testing!

How to Test Infrastructure with Pulumi

Sophia Parafina — Fri, 23 Jul 2021 00:00:00 +0000

In the Testing Practices for Cloud Engineering article we discussed how to apply software testing methodologies to cloud engineering. We also examined testing regimes starting from the testing pyramid to the trophy and honeycomb models of testing better suited to distributed and cloud architectures. These testing regimes include three types of tests suited for cloud architectures:

unit tests for testing methods and functions within a service
property tests for validating specified service outputs
integration tests to ensure that resources interact as specified

In this article, we’ll do a deep dive into each of these testing methods.

Cloud engineering testing differs from application testing because multiple dependencies between components can have high latency. Let’s examine these dependencies using an example application composed of an application layer, a database layer, and a network layer. We can visualize these dependencies with Pulumi’s resource graph.The graph is constructed from the inputs needed to create a resource.

The resource graph for the application layer shows many dependencies. The application deploys containers in AWS Fargate, which are instantiated with an application service that requires a service definition with many inputs.

Unit tests are ideal for validating inputs and ensuring that cloud resources are created. Let’s look at a simple example with just two resources, an EC2 instance running a web server and a SecurityGroup policy.

We can use mocks to test if the webserver has a name tag. First, we’ll create the mocks and import the infrastructure that uses the mocks.

import unittest
import pulumi

class MyMocks(pulumi.runtime.Mocks):
 def new_resource(self, args: pulumi.runtime.MockResourceArgs):
 outputs = args.inputs
 if args.type == "aws:ec2/instance:Instance":
 outputs = {
 **args.inputs,
 "publicIp": "203.0.113.12",
 "publicDns": "ec2-203-0-113-12.compute-1.amazonaws.com",
 }
 return [args.name + '_id', outputs]
 def call(self, args: pulumi.runtime.MockCallArgs):
 if args.token == "aws:ec2/getAmi:getAmi":
 return {
 "architecture": "x86_64",
 "id": "ami-0eb1f3cdeeb8eed2a",
 }
 return {}

pulumi.runtime.set_mocks(MyMocks())

# Now actually import the code that creates resources, and then test it.
import infra

We can write a test that checks if the web server has tags and specifically a name tag. First, here’s the code that creates the web server instance:

server = ec2.Instance('web-server-www',
 instance_type="t2.micro",
 vpc_security_group_ids=[ group.id ], # reference the group object above
 # Comment out to fail a test:
 tags={'Name': 'webserver'}, # name tag
 # Uncomment to fail a test:
 #user_data=user_data) # start a simple web server
 ami=ami_id)

Using Python’s unittest framework, the test looks like this:

@pulumi.runtime.test
def test_server_tags():
 def check_tags(args):
 urn, tags = args
 assert tags, f'server {urn} must have tags'
 assert 'Name' in tags, 'server {urn} must have a name tag'

 return pulumi.Output.all(infra.server.urn, infra.server.tags).apply(check_tags)

The test has two assertions: one for tags on the server urn and another if the instance has a Name tag. When we call the test, we pass the server urn and tags. Note that we did not instantiate an EC2 instance and used a mock. This is why it can run unit tests frequently with quick results.

Validating cloud resource properties in tests

In addition to unit tests, we want to test the properties of a cloud resource. For example, we want to restrict access to a resource by creating a policy that limits ingress. For this use case, we need a property test to ensure the exact values required by a policy. The resource graph from our simple example shows that a SecurityGroupRule sets ingress to the web server. We can test if port 22 is open, which would enable logging into the server via ssh.

group = ec2.SecurityGroup('web-secgrp', ingress=[
 { "protocol": "tcp", "from_port": 22, "to_port": 22, "cidr_blocks": ["0.0.0.0/0"] },
 { "protocol": "tcp", "from_port": 80, "to_port": 80, "cidr_blocks": ["0.0.0.0/0"] },
])

The code that tests for an open port 22 for any rule that includes from_port and cidr_blocks in the resource definition.

# Test if port 22 for ssh is exposed.
@pulumi.runtime.test
def test_security_group_rules():
 def check_security_group_rules(args):
 urn, ingress = args
 ssh_open = any([rule['from_port'] == 22 and any([block == "0.0.0.0/0" for block in rule['cidr_blocks']]) for rule in ingress])
 assert ssh_open == False, f'security group {urn} exposes port 22 to the Internet (CIDR 0.0.0.0/0)'

 # Return the results of the unit tests.
 return pulumi.Output.all(infra.group.urn, infra.group.ingress).apply(check_security_group_rules)

Because web-secgrp has port 22 open to all IP addresses, we can expect it to fail this test. You can find the complete example for the unit and property tests on Pulumi’s example Github repository.

Integration testing

Integration testing requires deploying resources in either a test environment or an ephemeral environment. In this integration testing example, we’ll deploy a single resource using Pulumi’s Automation API. Spinning up ephemeral environments for integration testing is an excellent use for the Automation API, which provides a programmatic interface for building infrastructure without a CLI or web-based console.

In this simple example, we’ll use Automation API to create an S3 bucket and perform a set of tests that include:

verifying the bucket name and AWS region
upload a test file
delete the file
delete the stack with the bucket

The following code sets up the Pulumi stack and S3 bucket with Automation API. It’s interesting to note that the tests don’t import the Pulumi AWS SDK and uses a combination of Python’s built-in unittest framework and Amazon’s boto3 Python library. This illustrates the extensibility and flexibility of infrastructure as code by letting developers choose the tools and SDKs.

import os
import unittest

import boto3
from pulumi.x import automation as auto

from resource_s3 import BUCKET_NAME, OUTPUT_KEY_BUCKET_NAME, OUTPUT_KEY_REGION


class TestS3(unittest.TestCase):

 @classmethod
 def setUpClass(cls) -> None:
 cls.STACK_NAME = 'staging'
 cls.REGION_NAME = 'eu-north-1'
 cls.WORK_DIR = os.path.join(os.path.dirname(__file__))
 cls.FILE_NAME = 'bucket.txt'

 cls.stack = auto.create_or_select_stack(stack_name=cls.STACK_NAME, work_dir=cls.WORK_DIR)
 cls.stack.set_config("aws:region", auto.ConfigValue(value=cls.REGION_NAME))
 cls.stack.up(on_output=print)
 cls.outputs = cls.stack.outputs()
 cls.s3 = boto3.resource('s3')

The TestS3 class has a second @classmethod decorator for the tests; for example, the following code uploads a file and checks to see if the file is stored in the bucket.

 def test_s3_create_permission(self):
 output_bucket_name = self.outputs.get(OUTPUT_KEY_BUCKET_NAME).value
 created_response = self.s3.Bucket(output_bucket_name).put_object(Key=self.FILE_NAME, Body='Hi')
 self.assertEqual(created_response.key, self.FILE_NAME)

When we run the test, it creates the stack and resource, runs the tests, and tears down the infrastructure. This process can be part of a CI/CD pipeline to run an automated integration test as part of the build process. The complete example for integration testing is available on Github.

Infrastructure testing strategy: final thoughts

Although distributed infrastructure presents some unique challenges because of its asynchronous nature, we can perform efficient and automated testing. Whether it is unit and property testing that uses mocks and stubs or integration testing that requires deployed resources, infrastructure as code lets you use familiar testing frameworks and SDKs. Moreover, Pulumi’s Automation API makes it simple to create stacks and spin up ephemeral infrastructure for integration testing. Learn more about infrastructure testing with Pulumi’s Testing User Guide and get hands-on with our testing examples.

Unit Testing Assets

Lee Zen — Thu, 11 Jun 2020 00:00:00 +0000

When deploying infrastructure, we want to ensure that what we’re deploying matches our expectations. One way to do so is via unit testing. We’ve talked about this concept in previous posts, such as in this overview and this post on deployments with .NET.

Often, when we’re creating cloud resources, we want to ensure that a resource’s underlying assets match certain properties. For example, the entrypoint or handler for a cloud function should be an executable function. Similarly, objects we’ll serve as static assets on a website should not exceed a certain size. We can use Pulumi’s unit testing framework along with language-specific constructs such as introspection or filesystem calls to ensure this type of correctness. We’ll walk through some examples to show just how easy it is to do so.

The examples below are in Python, but you can easily achieve the same types of tests using introspection/reflection in Node, .NET, and Go.

Check that Functions are Executable

When creating a Lambda function, we’re expected to pass in a function handler that gets called upon invocation. Typically, the value is something like mod.handler, where mod is the module name and handler is a function. A basic check would be to make sure that the module exists and that the function is callable.

We can easily construct this unit test like so:

test_infra.py:

import unittest
import pulumi

class MyMocks(pulumi.runtime.Mocks):
 def new_resource(self, type_, name, inputs, provider, id_):
 return [name + '_id', inputs]
 def call(self, token, args, provider):
 return {}

pulumi.runtime.set_mocks(MyMocks())

import infra

class TestInfrastructure(unittest.TestCase):

 # Test that the function entrypoint is actually an executable function.
 @pulumi.runtime.test
 def test_function_handler_callable(self):
 def check_entrypoint(handler):
 import importlib
 from types import ModuleType
 from inspect import signature, Parameter

 # The handler should be of the form <module_name>.<function>
 module_name, function_handler = handler.split('.', 1)

 # Check that we can load the module.
 module = importlib.import_module(module_name)
 self.assertIsInstance(module, ModuleType)

 # Check that the function handler is actually callable.
 fn = getattr(module, function_handler)
 self.assertTrue(callable(fn))

 # Check that it has two parameters: event and context
 sig = signature(fn)
 self.assertListEqual(['event', 'context'], list(sig.parameters.keys()))

 return infra.lambda_function.handler.apply(check_entrypoint)

Here, we use Python’s introspection capabilities to import a module based on a string that we obtain from the Lambda function definition. Similarly, after we import the module, we can verify that the function is callable and has the appropriate arguments for a Lambda function. At this point, we’ve confirmed that we’ve correctly “wired in” a function handler. Separately, we would expect another set of unit tests that verifies the correctness of the handler itself.

Here’s the corresponding infrastructure code to make the test pass, assuming there’s also a file called lambda_function.py, which contains a function handler(event, context).

infra.py:

import pulumi
import pulumi_aws as aws

lambda_role = aws.iam.Role('example-role', assume_role_policy=f"""{{
 "Version":"2012-10-17",
 "Statement":[{{
 "Effect": "Allow",
 "Principal":{{
 "Service": "lambda.amazonaws.com"
 }},
 "Action": "sts:AssumeRole"
 }}]
}}
""")

attach_lambda_role_execution = aws.iam.RolePolicyAttachment('example-attach-execute',
 policy_arn='arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole',
 role=lambda_role.name)

module_name = 'lambda_function'
module_file = f'{module_name}.py'
lambda_function = aws.lambda_.Function('example-function',
 handler=f'{module_name}.handler',
 role=lambda_role.arn,
 runtime='python3.7',
 code=pulumi.AssetArchive({
 module_file: pulumi.FileAsset(module_file)
 })
)

If lambda_function.py doesn’t exist or there’s no function handler within that module, the test will fail.

File Assets

Another example of checking for underlying asset correctness is when providing files to a resource. For example, we might want to check that the content type we provide to an object matches that of the resource:

test_infra.py:

# Test that the bucket object matches the correct mimetype
@pulumi.runtime.test
def test_bucket_object_content_type(self):
 def check_filetype(args):
 source, mimetype = args
 import filetype
 kind = filetype.guess(source.path)
 self.assertEqual(kind.mime, mimetype)

 return pulumi.Output.all(infra.bucket_obj.source, infra.bucket_obj.content_type).apply(check_filetype)

Here, we use the filetype module to guess at the type based on reading the file itself. We then check that the mime type specified on the bucket object matches.

For example, if the file represented by image.png isn’t a PNG, this test would fail:

infra.py:

bucket = aws.s3.Bucket('my-bucket')

bucket_obj = aws.s3.BucketObject('my-bucket-obj',
 bucket=bucket.id,
 content_type='image/png',
 source=pulumi.FileAsset('image.png'))

Secrets

When working with secrets, we want to ensure that provided values are secret and that corresponding outputs are secret. This helps ensure that our state never contains secrets as plaintext values. We can write a simple test to check this:

test_infra.py:

# Test that the secret input is marked as secret
@pulumi.runtime.test
def test_secret_input_and_output(self):
 self.assertTrue(infra.secret_password_v1.secret_string.is_secret().result())

If we try to write something like the following in our code, it’ll fail this test:

infra.py:

secret_password = aws.secretsmanager.Secret('example-secret')
secret_password_v1 = aws.secretsmanager.SecretVersion('example-secret-version',
 secret_id=secret_password,
 secret_string='example-string')

If we change secret_string='example-string' to secret_string=pulumi.Output.secret('example-string') our test will pass.

Conclusion

There are many other examples of resources where it’s useful to check the underlying properties of the input assets. For example, checking the validity of certificates, keys, files, and container images are all assets that benefit from making sure we’re wiring in resources correctly to our cloud infrastructure. We hope you take this opportunity to use Pulumi’s unit testing framework to check your infrastructure’s underlying assets for correctness. Please visit the unit testing guide to learn more.

At Scale Infrastructure Testing With Pulumi

Dustin Farris — Thu, 28 May 2020 00:00:00 +0000

Guest Article: Dustin Farris is an experienced cloud engineering consultant. He’s currently building a new data lake for a large public university using Pulumi. The project handles sensitive student and research data and as a result, his team must meet stringent QA and security requirements. Dustin shows how resource mocking in Pulumi makes testing and verification faster than ever before.

The university IT team has ambitious goals for a new data lake that we’re building. We need to follow AWS Well-Architected standards, adopt cutting-edge tools, and demonstrate provable quality and security from development to production.

Advancements in available cloud services give us the opportunity to deliver better performance, scalability, reliability, and security. We are redesigning our infrastructure to include:

AWS Database Migration Service (DMS) for connectivity to on-prem, legacy databases;
AWS Glue for extracting, transforming and loading (ETL) new data;
AWS Lambda Step Functions for workflows; and
AWS Lake Formation for creating a secure data catalog available for analytics tools.

With this shift, it makes sense to throw out old configurations and pick the best tools for the job rather than be encumbered by legacy DSLs. We use Pulumi as our Infrastructure as Code tool of choice because it has all of the capabilities we need:

Pulumi supports TypeScript - the modern programming language that the team prefers for its static typing capabilities.
It works ‘out of the box’ with modern Integrated Development Environments (IDEs) like Visual Studio Code.
Because Pulumi uses familiar programming languages, we can unit test our code
By virtue of working with TypeScript, it natively supports Mocha - our testing framework of choice.

Watch Dustin Farris and Lee Zen demonstrate unit testing on this episode of Modern Infrastructure Wednesday.

Accelerating Verification

There are many benefits to using Pulumi to develop and test infrastructure. First, the team has the immediate benefit of modern language and IDE support: things like missing arguments are now visible in-line as we define resources. In addition, state tracking and version control capabilities, for all cloud provider resources, make for a terrific experience in terms of productivity.

Second, creating standard infrastructure stacks for the team helps everyone move faster. We create environments for each developer in separate AWS accounts so they can develop features without impacting other developers and we don’t have to wait for central IT to provision resources. Pulumi’s support for ephemeral environments means that developers can spin-up and spin-down stacks quickly, in order to verify that changes to the desired state are actually reflected in the provisioned resources. This makes integration testing far easier, and ephemeral environments keep cloud costs down because they are easily destroyed when they are no longer needed.

Ideally, a unit test should demonstrate that a new component or resource works in isolation from other dependencies. Resource mocking mimics complex infrastructure in lieu of spinning up the dedicated infrastructure for a test (which would be slow and potentially expensive.) This step is critical for preventing configuration mistakes from reaching production.

The unit test suite for our data lake consists of over 700 tests and previously took almost 20 minutes to execute. With improvements to StackReference mocking and the new ‘monitor mock’ feature, these same tests now run in about 20 seconds. That’s 60x faster over our previous performance!

Getting Started with Resource Mocking

Continuously testing resource definitions is a best practice and it’s incredibly easy to get started. In this example, we mock a resource name to attach to an IAM Policy Role. The test checks that we attach the correct policy to the expected role.

test-setup.ts

/**
 * Create the mock resource
 */
pulumi.runtime.setMocks({
 newResource: function (
 resourceType: string,
 name: string,
 inputs: any,
 provider?: string,
 id?: string,
 ): { id: string; state: Record<string, any> } {
 const defaultState = {
 arn: `${name}-arn`,
 name: name,
 ...inputs,
 };
 // Use id unless it's blank or missing.
 const resourceId = id?.trim() ? id : `${name}-id`;
 return { id: resourceId, state: defaultState };
 },
 call: function (): Record<string, any> {
 return {};
 },
});

infra.ts

export const attachment = new aws.iam.RolePolicyAttachment(
 "role-policy-attachment",
 {
 policyArn: policy.arn,
 role: role.name,
 },
);

infra.spec.ts

/**
 * Returns a resource output's value, even if it's undefined.
 */
const getOutput = <T>(output: pulumi.Output<T>): Promise<T | undefined> =>
 (output as any).promise() as Promise<T>;

describe("attachment", () => {
 it("attaches the policy", async () => {
 assert.equal(
 await getOutput(attachment.policyArn),
 await getOutput(policy.arn),
 );
 });
 it("attaches to the role", async () => {
 assert.equal(
 await getOutput(attachment.role),
 await getOutput(role.name),
 );
 });
});

The key part is that we never actually talk to the cloud provider. Instead, we are able to mock the output values that we’d typically expect to have from using the provider such as the ARN. In the above example, you can see we simply mock out the ARN by returning “${name}-arn” instead of whatever the cloud provider would actually assign. There are many powerful ways to use this mocking to ensure that certain output properties are set or to mock the return of certain lookup calls to the cloud provider.

Testing with Policies

In addition to resource testing with mocks, we can test the entire stack with policies. This example ensures that the EKS cluster uses version 1.13 deployed on a custom VPC.

infra.ts

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a custom VPC.
const vpc = new awsx.ec2.Vpc("my-vpc");

// Create a basic EKS cluster.
const cluster = new eks.Cluster("my-cluster", {
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 storageClasses: "gp2",
 deployDashboard: false,
 version: "1.13",
 vpcId: vpc.id,
 subnetIds: vpc.publicSubnetIds,
});

policy.test

const stackPolicy: policy.StackValidationPolicy = {
 name: "eks-test",
 description: "EKS integration tests.",
 enforcementLevel: "mandatory",
 validateStack: async (args, reportViolation) => {
 const clusterResources = args.resources.filter(r => r.isType(aws.eks.Cluster));
 if (clusterResources.length !== 1) {
 reportViolation(`Expected one EKS Cluster but found ${clusterResources.length}`);
 return;
 }

 const cluster = clusterResources[0].asType(aws.eks.Cluster)!;
 if (cluster.version !== "1.13") {
 reportViolation(`Expected EKS Cluster '${cluster.name}' version to be '1.13' but found '${cluster.version}'`);
 }

 const vpcId = cluster.vpcConfig.vpcId;
 if (!vpcId) {
 // 'isDryRun==true' means the test are running in preview.
 // If so, the VPC might not exist yet even though it's defined in the program.
 // We shouldn't fail the test then to avoid false negatives.
 if (!pulumi.runtime.isDryRun()) {
 reportViolation(`EKS Cluster '${cluster.name}' has unknown VPC`);
 }
 return;
 }

 const ec2 = new aws.sdk.EC2({region: aws.config.region});
 const response = await ec2.describeVpcs().promise();
 const defaultVpc = response.Vpcs?.find(vpc => vpc.IsDefault);
 if (!defaultVpc) {
 reportViolation("Default VPC not found");
 return;
 }

 if (defaultVpc.VpcId === vpcId) {
 reportViolation(`EKS Cluster '${cluster.name}' should not use the default VPC`);
 }
 },
}

const tests = new policy.PolicyPack("tests-pack", {
 policies: [stackPolicy],
});

Policy tests run on the entire stack are integration tests. We have two policies. The first policy is to check the EKS version which is done without deploying because the version is an input value. Next, we check to see if the tests are running in preview. The test returns a violation if the VPC has not been deployed because it does not have an output for the policy to validate .

Next Steps

After working with Pulumi to build-out our data lake and implementing robust, end-to-end testing, the team is now moving towards Continuous Integration and Continuous Delivery (CI/CD) using AWS CodePipeline. We are also looking forward to Pulumi’s new policy-as-code framework, CrossGuard, to set guardrails on infrastructure and enforce compliance - further advancing the team’s QA and security goals.

Conclusion

Testing is just one of the superpowers that Pulumi 2.0 delivers. For more information, see our Testing User Guide and worked examples on GitHub in the language of your choice.

Unit Testing Cloud Deployments with .NET

Mikhail Shilkov — Thu, 21 May 2020 00:00:00 +0000

Because Pulumi uses general-purpose programming languages to provision cloud resources, you can take advantage of native tools and perform automated tests of your infrastructure. The full power of each language is available, including access to libraries and frameworks for testing.

This blog post takes a deeper dive into mock-based unit testing of Pulumi programs written in C# and F#.

How Unit Testing Works in Pulumi

Let’s start with a picture showing how Pulumi components interact during a typical deployment process.

Whenever a new resource is instantiated in code, the program makes a remote call to the engine. The engine receives the resource’s input values, validates them, and translates the request to an invocation of a cloud API. The API returns some data, which are translated to resource outputs and sent back to the program.

The remote calls may be slow, unreliable, and non-deterministic, which makes testing of these interactions hard.

Our testing guide outlines several testing methods, but today I want to focus on unit testing.

The Pulumi SDK provides a hook to replace all the remote calls with mocks. Mocks run in the same process and can respond immediately with hard-coded or calculated on-the-fly data.

We can write fully deterministic and blazingly fast automated tests as all remote calls and uncertainty are eliminated. There is no cloud to respond to resource creation, so it’s a developer’s responsibility to mimic the cloud behavior with mocks adequately.

This blog post walks you through an example of unit testing and mocking with the .NET SDK.

Define the Base Structure

In this article, I build a program that deploys a static website to Azure. I use TDD to add new tests and deployment components bit-by-bit.

Let’s start with an empty project—go ahead and create a .NET Core Console Application with the .NET CLI or your favorite IDE.

Install NuGet packages

You are free to choose your unit testing frameworks, mocking and assertions libraries. I’m using NUnit with FluentAssertions, and my program tests Azure resources, so this is my project file:

<Project Sdk="Microsoft.NET.Sdk">

 <PropertyGroup>
 <OutputType>Exe</OutputType>
 <TargetFramework>netcoreapp3.1</TargetFramework>
 <Nullable>enable</Nullable>
 </PropertyGroup>

 <ItemGroup>
 <PackageReference Include="FluentAssertions" Version="5.10.2" />
 <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
 <PackageReference Include="NUnit" Version="3.12.0" />
 <PackageReference Include="NUnit3TestAdapter" Version="3.16.1" />
 <PackageReference Include="Pulumi.Azure" Version="3.*" />
 </ItemGroup>

</Project>

<Project Sdk="Microsoft.NET.Sdk">

 <PropertyGroup>
 <OutputType>Exe</OutputType>
 <TargetFramework>netcoreapp3.1</TargetFramework>
 <Nullable>enable</Nullable>
 </PropertyGroup>

 <ItemGroup>
 <PackageReference Include="FluentAssertions" Version="5.10.2" />
 <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
 <PackageReference Include="NUnit" Version="3.12.0" />
 <PackageReference Include="NUnit3TestAdapter" Version="3.16.1" />
 <PackageReference Include="Pulumi.Azure" Version="3.*" />
 <PackageReference Include="Pulumi.FSharp" Version="2.*" />
 </ItemGroup>

 <ItemGroup>
 <Compile Include="Testing.fs" />
 <Compile Include="WebsiteStack.fs" />
 <Compile Include="WebsiteStackTests.fs" />
 </ItemGroup>

</Project>

Stack

A Pulumi stack is the “unit” of our testing. Every test instantiates a stack, retrieves the resources that the stack defines, and makes assertions about them.

Here is my starting point in WebsiteStack file:

using System.IO;
using Pulumi;
using Pulumi.Azure.Core;
using Storage = Pulumi.Azure.Storage;

public class WebsiteStack : Stack
{
 public WebsiteStack()
 {
 // <-- Cloud resources go here
 }
}

namespace UnitTesting

open Pulumi
open Pulumi.FSharp
open Pulumi.Azure.Core

type WebsiteStack() =
 inherit Stack()

 // <-- Cloud resources go here

This class is precisely what Pulumi expects to start deploying your resources to the cloud. Once the test suite is ready and green, you can deploy the stack with pulumi up.

Mocks

As I explained above, unit tests replace the real Pulumi engine with mocks. Mocks get all calls and respond with predefined values.

A mock class has to implement two methods of Pulumi.Testing.IMocks interface. NewResourceAsync is called whenever a new resource is defined, while CallAsync is invoked when our program retrieves information about existing cloud resources. We’ll focus on the former in this post, but we still need to define both.

While you are free to use your favorite mocking library, I’ll keep it simple and define the mocks as a plain class.

using System.Collections.Immutable;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Testing;

namespace UnitTesting
{
 class Mocks : IMocks
 {
 public Task<(string id, object state)> NewResourceAsync(
 string type, string name, ImmutableDictionary<string, object> inputs, string? provider, string? id)
 {
 var outputs = ImmutableDictionary.CreateBuilder<string, object>();

 // Forward all input parameters as resource outputs, so that we could test them.
 outputs.AddRange(inputs);

 // <-- We'll customize the mocks here

 // Default the resource ID to `{name}_id`.
 id ??= $"{name}_id";
 return Task.FromResult((id, (object)outputs));
 }

 public Task<object> CallAsync(string token, ImmutableDictionary<string, object> inputs, string? provider)
 {
 // We don't use this method in this particular test suite.
 // Default to returning whatever we got as input.
 return Task.FromResult((object)inputs);
 }
 }
}

namespace UnitTesting

open System
open System.Collections.Immutable
open System.Threading.Tasks
open Pulumi.Testing

type Mocks() =
 interface IMocks with
 member this.NewResourceAsync(typeName: string, name: string, inputs: ImmutableDictionary<string, obj>, provider: string, id: string): Task<ValueTuple<string,obj>> =
 // Forward all input parameters as resource outputs, so that we could test them.
 let dict = inputs.ToImmutableDictionary() :> obj

 // <-- We'll customize the mocks here

 // Default the resource ID to `{name}_id`.
 let id = if id = null then sprintf "%s_id" name else id
 Task.FromResult(struct (id, outputs :> obj))

 member this.CallAsync(token: string, inputs: ImmutableDictionary<string, obj>, provider: string): Task<obj> =
 // We don't use this method in this particular test suite.
 // Default to returning whatever we got as input.
 Task.FromResult(null :> obj)

Both methods receive several parameters and need to return a result. Notably, they get the list of inputs—the arguments that our program defined for the given resource. The goal of the mocks is to return the list of outputs, similarly to what a cloud provider would do.

My default implementation returns the outputs that include all the inputs and nothing else. That’s a sensible default: usually, real outputs are a superset of inputs. We’ll extend this behavior as we need later on.

Test Fixture

The next class to add is the container for my future unit tests. I named the file WebsiteStackTests.cs and it defines a test container, called a “test fixture” in NUnit:

using System.Linq;
using System.Threading.Tasks;
using FluentAssertions;
using NUnit.Framework;
using Pulumi;
using Pulumi.Azure.Core;
using Storage = Pulumi.Azure.Storage;

namespace UnitTesting
{
 [TestFixture]
 public class WebserverStackTests
 {
 // <-- Tests go here

 private static Task<ImmutableArray<Resource>> TestAsync()
 {
 return Deployment.TestAsync<WebsiteStack>(new Mocks(), new TestOptions {IsPreview = false});
 }
 }
}

namespace UnitTesting

open System
open System.Collections.Immutable
open System.Threading.Tasks
open NUnit.Framework
open FluentAssertions
open Pulumi
open Pulumi.Azure.Core
open Pulumi.Azure.Storage
open Pulumi.Testing

[<TestFixture>]
type WebserverStackTests() =
 let runTest(): ImmutableArray<Resource> =
 let options = new TestOptions(IsPreview = Nullable<bool> false)
 Deployment.TestAsync<WebsiteStack>(new Mocks(), options)
 |> Async.AwaitTask
 |> Async.RunSynchronously

 // <-- Tests go here

I defined a helper method TestAsync that points Pulumi’s Deployment.TestAsync to our stack and mock classes.

As we progress through the article, I will add tests to this class one-by-one.

Retrieve output values

Finally, I need a small extension method to extract values from outputs. Every Pulumi resource returns values wrapped inside an Output<T> container. While the real engine runs, those values may be known or unknown, but my mock-based tests always return known values. It’s safe to get those values and convert them to a Task<T>:

public static class TestingExtensions
{
 public static Task<T> GetValueAsync<T>(this Output<T> output)
 {
 var tcs = new TaskCompletionSource<T>();
 output.Apply(v => { tcs.SetResult(v); return v; });
 return tcs.Task;
 }
}

let getValue(output: Output<'a>): 'a =
 let tcs = new TaskCompletionSource<'a>()
 output.Apply(fun v -> tcs.SetResult(v); v) |> ignore
 tcs.Task.Result

To learn more about outputs, read this article.

First Test

The setup is done, time to write my first unit test! True to the TDD spirit, I write the tests before I define any resources.

Every Azure resource has to live inside a resource group, so my first test checks that a new resource group is defined.

[Test]
public async Task SingleResourceGroupExists()
{
 var resources = await TestAsync();

 var resourceGroups = resources.OfType<ResourceGroup>().ToList();
 resourceGroups.Count.Should().Be(1, "a single resource group is expected");
}

[<Test>]
member this.SingleResourceGroupExists() =
 let resources = runTest()

 let resourceGroupCount = resources.OfType<ResourceGroup>() |> Seq.length
 resourceGroupCount.Should().Be(1, "a single resource group is expected") |> ignore

This code is great to illustrate the overall structure of each test:

Call the TestAsync method to evaluate the stack.
Find the target resource in the collection of created resources.
Assert a property of the target resource.

In this case, the test validates that there is one and only one resource group defined.

I can run dotnet test and see the expected test failure:

$ dotnet test
...
X SingleResourceGroupExists [238ms]
Error Message:
Expected resourceGroups.Count to be 1 because a single resource group is expected, but found 0.
...
Total tests: 1
Failed: 1
Total time: 0.9270 Seconds

I go ahead and add the following definition to the WebsiteStack constructor.

var resourceGroup = new ResourceGroup("www-prod-rg");

let resourceGroup = new ResourceGroup("www-prod-rg")

This change is enough to make the tests green!

$ dotnet test
...
Test Run Successful.
Total tests: 1
Passed: 1
Total time: 0.8462 Seconds

Storage Account

It’s time to add a test for a second resource in the stack: a Storage Account. To make things more interesting, I want to check that the storage account belongs to our resource group.

There’s no direct link between a storage account object and a resource group object. Instead, we need to check the ResourceGroupName property of the account. The test below expects the account to belong to a resource group called www-prod-rg.

[Test]
public async Task StorageAccountBelongsToResourceGroup()
{
 var resources = await TestAsync();
 var storageAccount = resources.OfType<Storage.Account>().SingleOrDefault();
 storageAccount.Should().NotBeNull("Storage account not found");

 var resourceGroupName = await storageAccount.ResourceGroupName.GetValueAsync();
 resourceGroupName.Should().Be("www-prod-rg");
}

[<Test>]
member this.StorageAccountBelongsToResourceGroup() =
 let resources = runTest()
 let storageAccount = resources.OfType<Account>() |> Seq.tryHead |> Option.toObj
 storageAccount.Should().NotBeNull("Storage account not found") |> ignore

 let resourceGroupName = getValue storageAccount.ResourceGroupName
 resourceGroupName.Should().Be("www-prod-rg", null) |> ignore

Of course, the test fails. I try to fix it with what I think is the minimal change to make the test pass by adding a new resource to the stack and pointing it to the resource group.

var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
{
 ResourceGroupName = resourceGroup.Name,
});

let storageAccount =
 new Account(
 "wwwprodsa",
 new AccountArgs(ResourceGroupName = io resourceGroup.Name))

However, when I rerun the test suite, all three tests fail. They complain about the missing required properties AccountReplicationType and AccountTier on the storage account, so I have to add those.

var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
{
 ResourceGroupName = resourceGroup.Name,
 AccountTier = "Standard",
 AccountReplicationType = "LRS",
 StaticWebsite = new Storage.Inputs.AccountStaticWebsiteArgs
 {
 IndexDocument = "index.html"
 }
});

let storageAccount =
 new Account(
 "wwwprodsa",
 new AccountArgs(
 ResourceGroupName = io resourceGroup.Name,
 AccountTier = input "Standard",
 AccountReplicationType = input "LRS",
 StaticWebsite = input (new AccountStaticWebsiteArgs(IndexDocument = input "index.html"))))

I defined the StaticWebsite property as well: I’ll leave testing these values as an exercise for the reader.

Two resource group tests are back to green again, but the storage account test fails with a new error message.

X StorageAccountBelongsToResourceGroup [84ms]
Error Message:
Expected resourceGroupName to be "www-prod-rg", but found <null>.

What’s going on, and why is it null?

I defined the account name like this: ResourceGroupName = resourceGroup.Name. However, if we look closely, the resourceGroup resource doesn’t have an input property Name defined. www-prod-rg is a logical name for Pulumi deployment, not the physical name of the resource.

Under normal circumstances, the Pulumi engine would use the logical name to produce the physical name of the resource group automatically (see resource names for details). However, my mocks don’t do that.

That’s a good reason to change the Mocks implementation. I add the following lines to the NewResourceAsync method.

// Set the name to resource name if it's not set explicitly in inputs.
if (!inputs.ContainsKey("name"))
 outputs.Add("name", name);

let inputKVs = inputs |> Seq.map(fun kv -> kv.Key, kv.Value) |> Seq.toList
let nameKVs = if inputs.ContainsKey("name") then [] else [("name", name :> obj)]

let outputs = [inputKVs; nameKVs] |> Seq.concat |> Seq.map KeyValuePair
let dict = outputs.ToImmutableDictionary() :> obj

Note that mocks operate on weakly-typed dictionaries, so I need to get the property name right. Pulumi SDKs are open source, so I looked here to double-check the exact value.

After this change in Mocks, my tests go green again.

Test Run Successful.
Total tests: 3
Passed: 3

Website Files

The next step is to upload some files to the static website. Well, instead, to write an automated test that validates the upload with mocks.

I create a wwwroot folder with two HTML files in it and copy the folder to the build’s output. Here is my change in the project file.

<ItemGroup>
 <None Update="wwwroot\**\*.*">
 <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
 </None>
</ItemGroup>

Now, the test is straightforward: it expects two Blob resources in the stack.

[Test]
public async Task UploadsTwoFiles()
{
 var resources = await TestAsync();
 var files = resources.OfType<Storage.Blob>().ToList();
 files.Count.Should().Be(2, "Should have uploaded files from `wwwroot`");
}

[<Test>]
member this.UploadsTwoFiles() =
 let resources = runTest()
 let filesCount = resources.OfType<Blob>() |> Seq.length
 filesCount.Should().Be(2, "Should have uploaded files from `wwwroot`") |> ignore

As intended, the test fails immediately.

X UploadsTwoFiles [95ms]
Error Message:
Expected files.Count to be 2 because Should have uploaded files from `wwwroot`, but found 0.

I extend my stack with the following loop that navigates through the files and creates a storage blob for each of them.

var files = Directory.GetFiles("wwwroot");
foreach (var file in files)
{
 var blob = new Storage.Blob(file, new Storage.BlobArgs
 {
 ContentType = "application/html",
 Source = new FileAsset(file),
 StorageAccountName = storageAccount.Name,
 StorageContainerName = "$web",
 Type = "Block"
 });
}

let files =
 let fileNames = Directory.GetFiles("wwwroot")
 fileNames
 |> Seq.map(fun file ->
 new Blob(
 file,
 new BlobArgs(
 ContentType = input "application/html",
 Source = input (new FileAsset(file) :> AssetOrArchive),
 StorageAccountName = io storageAccount.Name,
 StorageContainerName = input "$web",
 Type = input "Block")))
 |> List.ofSeq

However, this change breaks the stack and all tests in the suite:

Error Message:
Pulumi.RunException : Running program failed with an unhandled exception:
System.InvalidOperationException: Unsupported value when converting to protobuf: Pulumi.FileAsset
at Pulumi.Serialization.Serializer.CreateValue(Object value)
...

Once again, our mocks fail to represent the behavior of the engine accurately. The Pulumi engine knows about the FileAsset class pointing to a file on the disk and how to convert it to an uploaded blob. But, the engine doesn’t copy this property to outputs. I need to adjust the mocks again.

I’m not particularly interested in testing the binary contents of the files now, so I’ll change the Mocks class to ignore the source property and not to include it into the output dictionary.

if (type == "azure:storage/blob:Blob")
{
 // Assets can't directly go through the engine.
 // We don't need them in the test, so blank out the property for now.
 outputs.Remove("source");
}

let outputs =
 [inputKVs; nameKVs]
 |> Seq.concat
 |> Seq.filter (fun (k, _) -> typeName <> "azure:storage/blob:Blob" || k <> "source")
 |> Seq.map KeyValuePair

This change makes my test suite happy again.

Test Run Successful.
Total tests: 4
Passed: 4

Validate Website URL

So far, I’ve been validating the input parameters of resources. What if I want to test something that is usually done by the cloud provider?

For instance, when Azure creates a static website, it automatically assigns a public endpoint. The endpoint is then available in the PrimaryWebEndpoint property after the Pulumi program ran and resources are created. I may want to export this value from stack outputs and validate it in a unit test.

[Test]
public async Task StackExportsWebsiteUrl()
{
 var resources = await TestAsync();
 var stack = resources.OfType<WebsiteStack>().First();

 var endpoint = await stack.Endpoint.GetValueAsync();
 endpoint.Should().Be("https://wwwprodsa.web.core.windows.net");
}

[<Test>]
member this.StackExportsWebsiteUrl() =
 let resources = runTest()
 let stack = resources.OfType<WebsiteStack>() |> Seq.head

 let endpoint = getValue stack.Endpoint
 endpoint.Should().Be("https://wwwprodsa.web.core.windows.net", null) |> ignore

This test doesn’t even compile yet, so I have to define the Endpoint property and assign a value to it. Here is the complete implementation of the stack with the output property defined at the end.

public class WebsiteStack : Stack
{
 public WebsiteStack()
 {
 var resourceGroup = new ResourceGroup("www-prod-rg", new ResourceGroupArgs
 {
 Tags = { { "Environment", "production" } }
 });

 var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
 {
 ResourceGroupName = resourceGroup.Name,
 AccountTier = "Standard",
 AccountReplicationType = "LRS",
 StaticWebsite = new Storage.Inputs.AccountStaticWebsiteArgs
 {
 IndexDocument = "index.html"
 }
 });

 var files = Directory.GetFiles("wwwroot");
 foreach (var file in files)
 {
 var blob = new Storage.Blob(file, new Storage.BlobArgs
 {
 ContentType = "application/html",
 Source = new FileAsset(file),
 StorageAccountName = storageAccount.Name,
 StorageContainerName = "$web",
 Type = "Block"
 });
 }

 this.Endpoint = storageAccount.PrimaryWebEndpoint;
 }

 [Output] public Output<string> Endpoint { get; set; }
}

type WebsiteStack() =
 inherit Stack()

 let resourceGroup =
 new ResourceGroup(
 "www-prod-rg",
 new ResourceGroupArgs(Tags = inputMap(["Environment", input "production"])))

 let storageAccount =
 new Account(
 "wwwprodsa",
 new AccountArgs(
 ResourceGroupName = io resourceGroup.Name,
 AccountTier = input "Standard",
 AccountReplicationType = input "LRS",
 StaticWebsite = input (new AccountStaticWebsiteArgs(IndexDocument = input "index.html"))))

 let files =
 let fileNames = Directory.GetFiles("wwwroot")
 fileNames
 |> Seq.map(fun file ->
 new Blob(
 file,
 new BlobArgs(
 ContentType = input "application/html",
 Source = input (new FileAsset(file) :> AssetOrArchive),
 StorageAccountName = io storageAccount.Name,
 StorageContainerName = input "$web",
 Type = input "Block")))
 |> List.ofSeq

 [<Output>]
 member val Endpoint = storageAccount.PrimaryWebEndpoint with get, set

Now, the test compiles but fails when executed:

X StackExportsWebsiteUrl [85ms]
Error Message:
Expected endpoint to be "https://wwwprodsa.web.core.windows.net", but found <null>.

That’s because there is no call to Azure to populate the endpoint. It’s time to make the last change to my Mocks class to emulate that assignment.

// For a Storage Account...
if (type == "azure:storage/account:Account")
{
 // ... set its web endpoint property.
 // Normally this would be calculated by Azure, so we have to mock it.
 outputs.Add("primaryWebEndpoint", $"https://{name}.web.core.windows.net");
}

let endpointKVs =
 if typeName = "azure:storage/account:Account"
 then ["primaryWebEndpoint", sprintf "https://%s.web.core.windows.net" name :> obj]
 else []

let outputs =
 [inputKVs; nameKVs; endpointKVs]
// ...

And that’s it! My static website is ready and tested!

Test Run Successful.
Total tests: 5
Passed: 5
Total time: 0.9338 Seconds

Note that it still takes less than a second to run my test suite so that I can iterate very quickly.

Get Started

The tests above cover the basics of unit testing with Pulumi .NET SDK. You can take it from here and apply the techniques and practices that you use while testing the application code. You may also try more advanced practices like property-based testing or behavior-driven development—we believe that the mocks enable many testing styles.

Here are several useful pointers to get started with testing in Pulumi:

Testing Guide
Full code for this blog post: C#, F#.
Another example of unit testing in C#

Unit Testing Infrastructure

Sophia Parafina — Tue, 24 Mar 2020 00:00:00 +0000

We’re pleased to announce that unit testing with Node.js, Python, .NET, and Go is supported in recent releases. You can test resources before deploying your infrastructure using familiar tools and test frameworks. Check your resource configuration and responses without the wait of deploying them and speed up infrastructure development and production deployments.

Pulumi brings testing to Infrastructure as Code. Testing your infrastructure code will provide early bug notification, cleaner and more extensible code, and simplify refactoring. We implemented testing with these goals in mind.

Run code without the engine
Write tests using existing test tools and frameworks
Mock any dependencies

Here are examples in Python and Go to get you started.

Python

First we’ll set up an AWS EC2 instance as a web server to test.

import pulumi
from pulumi_aws import ec2

group = ec2.SecurityGroup('web-secgrp', ingress=[
 { "protocol": "tcp", "from_port": 22, "to_port": 22, "cidr_blocks": ["0.0.0.0/0"] },
 { "protocol": "tcp", "from_port": 80, "to_port": 80, "cidr_blocks": ["0.0.0.0/0"] },
])

user_data = '#!/bin/bash echo "Hello, World!" > index.html nohup python -m SimpleHTTPServer 80 &'

server = ec2.Instance('web-server-www;',
 instance_type="t2.micro",
 security_groups=[ group.name ], # reference the group object above
 user_data=user_data, # start a simple web server
 ami="ami-c55673a0") # AMI for us-east-2 (Ohio)

pulumi.export('group', group)
pulumi.export('server', server)
pulumi.export('publicIp', server.public_ip)
pulumi.export('publicHostName', server.public_dns)

Using Python’s unittest framework we can test if the test service conforms to a configuration that restricts an exposed port 22 and use of user_data and enforces the use of tags and membership in a security group.

import unittest
import pulumi

# Create the mock.
class MyMocks(pulumi.runtime.Mocks):
 def call(self, token, args, provider):
 return {}

 def new_resource(self, type_, name, inputs, provider, id_):
 if type_ == 'aws:ec2/securityGroup:SecurityGroup':
 state = {
 'arn': 'arn:aws:ec2:us-west-2:123456789012:security-group/sg-12345678',
 'name': inputs['name'] if 'name' in inputs else name + '-sg',
 }
 return ['sg-12345678', dict(inputs, **state)]
 elif type_ == 'aws:ec2/instance:Instance':
 state = {
 'arn': 'arn:aws:ec2:us-west-2:123456789012:instance/i-1234567890abcdef0',
 'instanceState': 'running',
 'primaryNetworkInterfaceId': 'eni-12345678',
 'privateDns': 'ip-10-0-1-17.ec2.internal',
 'publicDns': 'ec2-203-0-113-12.compute-1.amazonaws.com',
 'publicIp': '203.0.113.12',
 }
 return ['i-1234567890abcdef0', dict(inputs, **state)]
 else:
 return ['', {}]

pulumi.runtime.set_mocks(MyMocks())

# Import the EC2 web server.
import infra

class InfraTests(unittest.TestCase):

 # Test if the service has tags and a name tag.
 @pulumi.runtime.test
 def test_server_tags(self):
 def check_tags(args):
 urn, tags = args
 self.assertIsNotNone(tags, f'server {urn} must have tags')
 self.assertIn('Name', tags, 'server {urn} must have a name tag')

 return pulumi.Output.all(infra.server.urn, infra.server.tags).apply(check_tags)

 # Test if the instance is configured with user_data.
 @pulumi.runtime.test
 def test_server_userdata(self):
 def check_user_data(args):
 urn, user_data = args
 self.assertFalse(user_data, f'illegal use of user_data on server {urn}')

 return pulumi.Output.all(infra.server.urn, infra.server.user_data).apply(check_user_data)

 # Test if service is a member of a security group.
 @pulumi.runtime.test
 def test_server_security_groups(self):
 def check_security_groups(args):
 urn, security_groups = args
 self.assertIsNotNone(security_groups, f'server {urn} does not specify security_groups')
 self.assertGreater(len(security_groups), 0, f'server {urn} does not specify security_groups')

 return pulumi.Output.all(infra.server.urn, infra.server.security_groups).apply(check_security_groups)

 # Test if port 22 for ssh is exposed.
 @pulumi.runtime.test
 def test_security_group_rules(self):
 def check_security_group_rules(args):
 urn, ingress = args
 ssh_open = any([rule['from_port'] == 22 and any([block == "0.0.0.0/0" for block in rule['cidr_blocks']]) for rule in ingress])
 self.assertFalse(ssh_open, f'security group {urn} exposes port 22 to the Internet (CIDR 0.0.0.0/0)')

 # Return the results of the unit tests.
 return pulumi.Output.all(infra.group.urn, infra.group.ingress).apply(check_security_group_rules)

Go

This is the same example, but written in Go using the stretchr framework for unit tests.

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v2/go/aws/ec2"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

type infrastructure struct {
 group *ec2.SecurityGroup
 server *ec2.Instance
}

func createInfrastructure(ctx *pulumi.Context) (*infrastructure, error) {
 group, err := ec2.NewSecurityGroup(ctx, "web-secgrp", &ec2.SecurityGroupArgs{
 Ingress: ec2.SecurityGroupIngressArray{
 ec2.SecurityGroupIngressArgs{
 Protocol: pulumi.String("tcp"),
 FromPort: pulumi.Int(22),
 ToPort: pulumi.Int(22),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 ec2.SecurityGroupIngressArgs{
 Protocol: pulumi.String("tcp"),
 FromPort: pulumi.Int(80),
 ToPort: pulumi.Int(80),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 },
 })
 if err != nil {
 return nil, err
 }

 const userData = `#!/bin/bash echo "Hello, World!" > index.html nohup python -m SimpleHTTPServer 80 &`

 server, err := ec2.NewInstance(ctx, "web-server-www", &ec2.InstanceArgs{
 InstanceType: pulumi.String("t2-micro"),
 SecurityGroups: pulumi.StringArray{group.ID()},
 Ami: pulumi.String("ami-c55673a0"),
 UserData: pulumi.String(userData),
 })
 if err != nil {
 return nil, err
 }

 return &infrastructure{
 group: group,
 server: server,
 }, nil
}

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 infra, err := createInfrastructure(ctx)
 if err != nil {
 return err
 }

 ctx.Export("group", infra.group.ID())
 ctx.Export("server", infra.server.ID())
 ctx.Export("publicIp", infra.server.PublicIp)
 ctx.Export("publicHostName", infra.server.PublicDns)
 return nil
 })
}

We implement the same tests used in the Python example.

package main

import (
 "fmt"
 "sync"
 "testing"

 "github.com/pulumi/pulumi-aws/sdk/v2/go/aws/ec2"
 "github.com/pulumi/pulumi/sdk/v2/go/common/resource"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/stretchr/testify/assert"
)

type mocks int

// Create the mock.
func (mocks) NewResource(typeToken, name string, inputs resource.PropertyMap, provider, id string) (string, resource.PropertyMap, error) {
 switch typeToken {
 case "aws:ec2/securityGroup:SecurityGroup":
 if _, ok := inputs["name"]; !ok {
 inputs["name"] = resource.NewStringProperty(name + "-sg")
 }
 inputs["arn"] = resource.NewStringProperty("arn:aws:ec2:us-west-2:123456789012:security-group/sg-12345678")
 return "sg-12345678", inputs, nil
 case "aws:ec2/instance:Instance":
 inputs["arn"] = resource.NewStringProperty("arn:aws:ec2:us-west-2:123456789012:instance/i-1234567890abcdef0")
 inputs["instanceState"] = resource.NewStringProperty("running")
 inputs["primaryNetworkInterfaceId"] = resource.NewStringProperty("eni-12345678")
 inputs["privateDns"] = resource.NewStringProperty("ip-10-0-1-17.ec2.internal")
 inputs["publicDns"] = resource.NewStringProperty("ec2-203-0-113-12.compute-1.amazonaws.com")
 inputs["publicIp"] = resource.NewStringProperty("203.0.113.12")
 return "i-1234567890abcdef0", inputs, nil
 default:
 return "", inputs, fmt.Errorf("unexpected resource type %v", typeToken)
 }
}

func (mocks) Call(token string, args resource.PropertyMap, provider string) (resource.PropertyMap, error) {
 return args, fmt.Errorf("unexpected function call %v", token)
}

// Applying unit tests.
func TestInfrastructure(t *testing.T) {
 err := pulumi.RunErr(func(ctx *pulumi.Context) error {
 infra, err := createInfrastructure(ctx)
 assert.NoError(t, err)

 var wg sync.WaitGroup
 wg.Add(4)

 // Test if the service has tags and a name tag.
 pulumi.All(infra.server.URN(), infra.server.Tags).ApplyT(func(all []interface{}) error {
 urn := all[0].(pulumi.URN)
 tags := all[1].(map[string]interface{})

 assert.Containsf(t, tags, "name", "missing a name tag on server %v", urn)
 wg.Done()
 return nil
 })

 // Test if the instance is configured with user_data.
 pulumi.All(infra.server.URN(), infra.server.UserData).ApplyT(func(all []interface{}) error {
 urn := all[0].(pulumi.URN)
 userData := all[1].(*string)

 assert.Nilf(t, userData, "illegal use of userData on server %v", urn)
 wg.Done()
 return nil
 })

 // Test if service is a member of a security group.
 pulumi.All(infra.server.URN(), infra.server.SecurityGroups).ApplyT(func(all []interface{}) error {
 urn := all[0].(pulumi.URN)
 securityGroups := all[1].([]string)

 assert.Greaterf(t, len(securityGroups), 0, "illegal security group spec on server %v", urn)
 wg.Done()
 return nil
 })

 // Test if port 22 for ssh is exposed.
 pulumi.All(infra.group.URN(), infra.group.Ingress).ApplyT(func(all []interface{}) error {
 urn := all[0].(pulumi.URN)
 ingress := all[1].([]ec2.SecurityGroupIngress)

 for _, i := range ingress {
 openToInternet := false
 for _, b := range i.CidrBlocks {
 if b == "0.0.0.0/0" {
 openToInternet = true
 break
 }
 }

 assert.Falsef(t, i.FromPort == 22 && openToInternet, "illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0) on group %v", urn)
 }

 wg.Done()
 return nil
 })

 wg.Wait()
 return nil
 }, pulumi.WithMocks("project", "stack", mocks(0)))
 assert.NoError(t, err)
}

Try it out today

We’ve released these features early so that you can try them out before the 2.0 release. To get you started, we’ve added full working examples on Github. With the 2.0 release we’ll have more documentation, examples, and blog posts. In the meantime, we would appreciate feedback on our Community Slack channel.

Test and Optimize Elasticsearch with Pulumi

Itamar Syn-Hershko — Thu, 05 Mar 2020 00:00:00 +0000

Guest Article: Itamar Syn-Hershko, Founder and CTO of BigData Boutique shows how they use Pulumi to benchmark Elasticsearch configurations across cloud providers. Pulumi enables BigData Boutique to test deployments in parallel and gather metrics to produce performant and cost-effective solutions for its customers.

BigData Boutique can rapidly iterate through infrastructure configurations based on client requirements. Familiar programming languages significantly shortens the inner dev loop because developers and operators can use modern programming tools such as IDEs, test frameworks, and CI/CD. By increasing the velocity of infrastructure deployment across many clouds with repeatable and versioned code, Pulumi enables companies to build efficient revenue centers for their business.

How Pulumi Drives Our Elasticsearch Capacity Planning and Cost Optimization Service

Capacity Planning and Cost Optimization of Elasticsearch clusters requires a special level of expertise and automation. Here is how we use Pulumi to launch long-running benchmarks to correctly identify the right configuration for our customers’ Big Data clusters.

At BigData Boutique, we are continually challenged by our customers—whether it’s complex Big Data challenges we are asked to solve, reducing costs for sophisticated systems holding mountains of data, improving performance for high-demand systems, or challenges faced in building a data-driven system.

One of the main challenges our customers are facing with Big Data systems is deploying the right hardware at the right scale. They need to know how to correctly estimate the number of servers required to run their compute workloads and databases, so they don’t pay more than they should, while still keeping those services running without interruption and within the defined KPIs.

To support our customers and help them succeed in their Big Data projects, we provide an array of advanced services, and our unique cost-optimization service addresses this challenge directly. This article outlines how we use long-running benchmarks to correctly identify the right configuration for our customers’ BigData clusters by using the infrastructure as code tool Pulumi for automating the creation of complex infrastructures with conditionals and highly tunable configurations to orchestrate those benchmarks at scale.

Elasticsearch Cost Optimization

Let’s take Elasticsearch as an example. Elasticsearch is a real-time search and analytics engine. It is built to ingest huge amounts of data and to allow one to search through and visualize them in real-time. But to be able to sustain the required amount of data on ingestion and the desired number of queries per second, the right hardware has to be used.

To do this, we have to make some decisions: which disks to use (SSD disks? local, or network storage?), the number of servers (scaling up or out?), CPU and RAM ratios, index mappings, various cluster configurations, and more. These are not easy questions to answer, and that is where our Elasticsearch expertise comes into play. But answering those questions correctly with a high level of certainty also requires rigorous testing for validating assumptions.

Whether our customer is running on-prem or on a cloud, our Elasticsearch Capacity Planning Service exists for one purpose: to find the hardware solution with the optimum balance between cost and performance, and do it scientifically, so the answers are as accurate and as precise as possible.

Elasticsearch Sizing Process

Our process begins with a sample data request. We ask the customer for their data (or a large sample of it), existing index mappings, expected queries, and finally known KPIs and SLAs. Our goal in the initial discovery phase is to reduce the number of free variables as much as possible because the more such variables, the more permutations we will have to test. Ideally, we want very few dimensions of freedom — usually in terms of hardware and configurations—so the number of permutations to test is on the order of dozens and not thousands.

Once the initial framework has been decided, we run fully automated benchmarks to test some of the permutations our experience tells us are the most promising to deliver good results. The benchmarking process is smart enough to omit long tails and outliers, so the results are not going to lie as to the true nature of the configuration that was selected and tested.

Between benchmark iterations, our team uses its years of experience to create more test configurations that optimize various configurations we decide are important based on the results of the previous benchmarks. After several iterations of benchmarking on various configurations, we send the preliminary findings to confirm our results and proposed direction to the customer. The purpose is to communicate the results and validate the trade-offs we decide on during the sizing procedure. Finally, we provide a summary session with our team of Elasticsearch experts and customer technical and business contacts to present the results and recommendations, and to explain the reasoning behind the analysis.

Finding the best Elasticsearch configuration requires careful benchmarking and tech expertise. Yet trade-offs will come into play. For instance, it might seem that the only way to gain another 100ms of performance improvement for a certain specific query would be to spend another $500 per month on hardware. But there may be other options. Our Sizing Service takes the guesswork out of the process. We automate the entire benchmarking procedure in order to make it an exact science.

To fully automate the benchmarking process, we use Pulumi.

Using Pulumi for State-of-the-Art Infrastructure Automation

Pulumi is an infrastructure-as-code tool that allows us to define and run infrastructures using code, as opposed to doing this manually or using other tools that use configuration-based syntax. This lets us create complex infrastructures with conditionals and highly tunable configurations. The Pulumi program we created can be run numerous times with different parameters. Each result is a cluster that is built and configured differently, sometimes even on completely entirely different clouds.

At BigData Boutique, we use that concept to launch multiple infrastructures in parallel, benchmark them, and show a report that highlights which config is better along with the various trade-offs for each. After the benchmark process, the cluster is destroyed, and the “provisioner” instance reports all logs, metrics, and the benchmark report, which we then analyze and visualize.

Generating, visualizing, and analyzing dozens of reports in parallel gives us, the Elasticsearch Engineers, the ability to understand which knobs to turn and what configurations to change to make our customers succeed in their project without spending more than they are willing to.

Conclusion

In sum, leveraging Pulumi helps us to create highly configurable infrastructures to orchestrate benchmarks at scale. As one of many advanced services that we offer, the Elasticsearch Capacity Planning and Cost Optimization Service continues to meet our customers’ most important planning and optimizations needs in an as scientifically rigorous manner as possible. You can read more about it here.

The original article was published on their company blog.

Unit Testing Your Infrastructure with Node.js and Mocha

Joe Duffy — Sat, 13 Jul 2019 00:00:00 +0000

Some parts of this blog post are out-of-date. Please refer to our Testing Guide for the updated overview and tutorials.

Testing your infrastructure using familiar tools like Node.js’s Mocha framework allows you to ensure configuration is correct before provisioning, and that the resulting infrastructure has certain desirable properties afterwards. This can enforce team standards, ensure security guidelines are being followed, and so much more. Because Pulumi uses general purpose languages, you can just embed tests alongside your infrastructure-as-code definitions themselves, using a familiar authoring style and reporting experience. In this post, we’ll explore the ins and outs of unit testing your infrastructure.

Test-Driven Infrastructure

We previously explored many reasons and solutions for testing your infrastructure. In this post, we’ll see a very simple approach that leverages existing test tools and frameworks. For this blog post, we’ll be using Node.js with TypeScript, the Mocha test framework, and the Chai assertion library.

Our example will be testing two things about an Amazon EKS cluster; that it is

Running Kubernetes version 1.13.
Provisioned inside a private VPC, rather than the default one.

In true test-driven development (TDD) form, let’s start with the tests themselves. If you’re familiar with Mocha and Chai, this should look like ordinary testing to you. If you’re not familiar with them, don’t worry – they are quite easy to read and learn: Mocha and Chai.

Here is the contents of our tests/eks.spec.ts test file:

import * as aws from "@pulumi/aws";
import { expect } from "chai";
import * as pulumi from "@pulumi/pulumi";
import { promise } from "./";
import { cluster } from "../";

describe("#Amazon EKS Cluster", () => {
 it("Should use an approved Kubernetes version", async () => {
 const version = await promise(cluster.eksCluster.version);
 expect(version).to.equal("1.13");
 });
 it("Should be provisioned inside a custom, non-default VPC", async () => {
 const vpcId = await promise(cluster.core.vpcId);
 if (vpcId) {
 const defaultVpc = await aws.ec2.getVpc({ default: true });
 expect(vpcId).not.equals(defaultVpc.id);
 } else if (pulumi.runtime.isDryRun()) {
 pulumi.log.warn("Skipped VPC ID check: not known during preview -- will test after apply");
 } else {
 throw new Error("VPC unknown during deployment");
 }
 });
});

This code does a few things worth describing. First, it imports all the packages that we’re going to use. Notably, this includes the normal Chai package for assertions, the AWS and Pulumi SDK packages, the EKS cluster provisioned by this project itself, and a utility function promise that we’ll dive into momentarily. Then it simply uses the Mocha functions describe and it to register test cases that will be run.

The version test fetches a property from our EKS cluster, cluster.eksCluster.version, and then uses the Chai assertion expect(version).to.equal("1.13"). This will fail the test if the version is anything but 1.13 (including if it is unknown).

The VPC test is slightly more complex. For the most part, it is simply checking the cluster.core.vpcId property and ensuring it doesn’t equal the default VPC ID, which we fetch with aws.ec2.getVpc. However, because we might be creating the custom VPC in the program itself, it’s possible we won’t know the ID during previews. (A preview is when Pulumi shows you a dry-run of your deployment without actually doing it yet.) That’s why we check pulumi.runtime.isDryRun and let things slide, with a little warning message. We could always be conservative and fail the test, but in this case, we’ll actually permit the deployment to proceed – but ensure we check it afterwards. This means we might fail the test after the bad cluster has been deployed, but that’s better than not checking at all.

Our Base (Failing) Program

Here is the program we’ll be testing. Keeping with our TDD theme, let’s start with the tests failing to begin with (we are using the default VPC and not specifying a version):

import * as eks from "@pulumi/eks";

// Create a basic EKS cluster.
export const cluster = new eks.Cluster("my-cluster", {
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 storageClasses: "gp2",
 deployDashboard: false,
});

If you want to create a fresh Pulumi project and follow along, simply install the CLI, ensure it is configured for your AWS account, and run pulumi new aws-typescript to create an empty project. Swap out the contents of index.ts with the above.

Before we add our tests, we can run a pulumi preview and all will work just fine:

$ pulumi preview
Previewing update (tdd-infra):

 Type Name Plan
 + pulumi:pulumi:Stack infra-eks-tdd-infra create
 + └─ eks:index:Cluster my-cluster create
 + ├─ eks:index:ServiceRole my-cluster-eksRole create
 + │ ├─ aws:iam:Role my-cluster-eksRole-role create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-eksRole-90eb1c99 create
 + │ └─ aws:iam:RolePolicyAttachment my-cluster-eksRole-4b490823 create
 + ├─ eks:index:ServiceRole my-cluster-instanceRole create
 + │ ├─ aws:iam:Role my-cluster-instanceRole-role create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-03516f97 create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-e1b295bd create
 + │ └─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-3eb088f2 create
 + ├─ pulumi-nodejs:dynamic:Resource my-cluster-cfnStackName create
 + ├─ aws:iam:InstanceProfile my-cluster-instanceProfile create
 + ├─ aws:ec2:SecurityGroup my-cluster-eksClusterSecurityGroup create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksClusterInternetEgressRule create
 + ├─ aws:eks:Cluster my-cluster-eksCluster create
 + ├─ pulumi:providers:kubernetes my-cluster-eks-k8s create
 + ├─ pulumi-nodejs:dynamic:Resource my-cluster-vpc-cni create
 + ├─ kubernetes:core:ConfigMap my-cluster-nodeAccess create
 + ├─ aws:ec2:SecurityGroup my-cluster-nodeSecurityGroup create
 + ├─ kubernetes:storage.k8s.io:StorageClass my-cluster-gp2 create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksExtApiServerClusterIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksClusterIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeInternetEgressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeClusterIngressRule create
 + ├─ aws:ec2:LaunchConfiguration my-cluster-nodeLaunchConfiguration create
 + ├─ aws:cloudformation:Stack my-cluster-nodes create
 + └─ pulumi:providers:kubernetes my-cluster-provider create

Resources:
 + 29 to create

As usual, this shows us all of the resources that Pulumi would provision were we to proceed with a deployment. No tests yet, however.

Adding the Test Harness Boilerplate

Now let’s hook up the tests so they run anytime we do a pulumi preview or pulumi up.

Note that we won’t be running these tests using the mocha CLI. Instead, we will run Mocha programmatically. This lets us run the tests inside of our Pulumi program where all of the necessary functionality is available. It also automatically coordinates test execution with previews and updates.

First, create a test file, tests/index.ts, that runs Mocha. This is basic boilerplate that registers all of the test files in the tests/ directory and will work for any project:

import * as fs from "fs";
import * as Mocha from "mocha";
import * as path from "path";
import * as pulumi from "@pulumi/pulumi";

// runTests executes all test files (*.ts) in the current directory.
export function runTests() {
 // Create a new Mocha test runner (with a long timeout).
 const mocha = new Mocha({ timeout: 1000*60*30 });

 // Only keep the .ts files, and skip this file (index.ts).
 const testDir = __dirname;
 fs.readdirSync(testDir).
 filter(file => file.endsWith(".ts") && file !== "index.ts").
 forEach(file => { mocha.addFile(path.join(testDir, file)); });

 // Now actually run the tests with the desired reporter.
 console.log(`Running Mocha Tests: ${mocha.files}`);
 mocha.reporter("spec").run(failures => {
 process.exitCode = failures ? 1 : 0;
 });
}

// promise returns a resource output's value, even if it's undefined.
export function promise<T>(output: pulumi.Output<T>): Promise<T | undefined> {
 return (output as any).promise() as Promise<T>;
}

The runTests function creates an instance of the Mocha runner, registers all the test files, configures the desired reporter, and runs the tests. We adjust the timeout to be 30 minutes, since provisioning things can take some time and we don’t want Mocha to timeout. For complete information on the options available, see the Mocha documentation; effectively, if you can do it with the mocha CLI, there is a way to do it using the API.

This file also defines the promise function used to access resource properties in our tests. We’ll see why this is needed shortly.

Running the Tests

Now that we have runTests, let’s go back and add a call to it in our primary index.ts file. First we need to import our custom function at the top of the file:

import { runTests } from "./tests";

And now we’ll add a call to runTests at the very bottom of the file:

// Run deployment validation tests.
runTests();

That’s it. Now when we run pulumi up, we will see that our tests are running!

$ pulumi up
Previewing update (tdd-infra):

 Type Name Plan Info
 + pulumi:pulumi:Stack infra-eks-tdd-infra create 1 error; 19 messages
 + └─ eks:index:Cluster my-cluster create
 + ├─ eks:index:ServiceRole my-cluster-eksRole create
 + │ ├─ aws:iam:Role my-cluster-eksRole-role create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-eksRole-4b490823 create
 + │ └─ aws:iam:RolePolicyAttachment my-cluster-eksRole-90eb1c99 create
 + ├─ eks:index:ServiceRole my-cluster-instanceRole create
 + │ ├─ aws:iam:Role my-cluster-instanceRole-role create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-03516f97 create
 + │ ├─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-e1b295bd create
 + │ └─ aws:iam:RolePolicyAttachment my-cluster-instanceRole-3eb088f2 create
 + ├─ pulumi-nodejs:dynamic:Resource my-cluster-cfnStackName create
 + ├─ aws:iam:InstanceProfile my-cluster-instanceProfile create
 + ├─ aws:ec2:SecurityGroup my-cluster-eksClusterSecurityGroup create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksClusterInternetEgressRule create
 + ├─ aws:eks:Cluster my-cluster-eksCluster create
 + ├─ pulumi:providers:kubernetes my-cluster-eks-k8s create
 + ├─ aws:ec2:SecurityGroup my-cluster-nodeSecurityGroup create
 + ├─ kubernetes:core:ConfigMap my-cluster-nodeAccess create
 + ├─ kubernetes:storage.k8s.io:StorageClass my-cluster-gp2 create
 + ├─ pulumi-nodejs:dynamic:Resource my-cluster-vpc-cni create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksClusterIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeClusterIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeInternetEgressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksNodeIngressRule create
 + ├─ aws:ec2:SecurityGroupRule my-cluster-eksExtApiServerClusterIngressRule create
 + ├─ aws:ec2:LaunchConfiguration my-cluster-nodeLaunchConfiguration create
 + ├─ aws:cloudformation:Stack my-cluster-nodes create
 + └─ pulumi:providers:kubernetes my-cluster-provider create

Diagnostics:
 pulumi:pulumi:Stack (infra-eks-tdd-infra):
 Running Mocha Tests: infra-eks/tests/eks.spec.ts
 #Amazon EKS Cluster
 1) Should use an approved Kubernetes version
 2) Should be provisioned inside a custom, non-default VPC
 0 passing (9s)
 2 failing
 1) #Amazon EKS Cluster
 Should use an approved Kubernetes version:
 AssertionError: expected undefined to equal '1.13'
 at Object.<anonymous> (tests/eks.spec.ts:19:35)
 at Generator.next (<anonymous>)
 at fulfilled (tests/eks.spec.ts:4:58)
 2) #Amazon EKS Cluster
 Should be provisioned inside a custom, non-default VPC:
 AssertionError: expected 'vpc-c93b06ae' to not equal 'vpc-c93b06ae'
 + expected - actual
 at Object.<anonymous> (tests/eks.spec.ts:25:38)
 at Generator.next (<anonymous>)
 at fulfilled (tests/eks.spec.ts:4:58)

 error: an unhandled error occurred: Program exited with non-zero exit code: 1

As expected, our tests fail for two reasons:

The Kubernetes version is undefined and not 1.13.
The VPC ID is equal to the default VPC, since the EKS cluster uses that by default.

Fixing Our Program

Now let’s refactor our infrastructure so the tests start passing.

To fix the first problem, we simply need to pass the Kubernetes version explicitly when creating our cluster. That’s as simple as passing a new argument:

export const cluster = new eks.Cluster("my-cluster", {
 ...
 version: "1.13",
});

Note that we could have done what we did with the VPC ID, and change the test to verify that the version is correct after provisioning. It turns out that at the time of this writing, the default version is in fact 1.13, so our test would then pass. But we’d be depending on a brittle default that could change at any moment – so it’s probably better for us to be explicit about this!

If we rerun pulumi up, we’ll see that this test now passes – huzzah:

Diagnostics:
 pulumi:pulumi:Stack (infra-eks-tdd-infra):
 Running Mocha Tests: infra-eks/tests/eks.spec.ts
 #Amazon EKS Cluster
 ✓ Should use an approved Kubernetes version (3704ms)
 1) Should be provisioned inside a custom, non-default VPC
 1 passing (4s)
 1 failing
 1) #Amazon EKS Cluster
 Should be provisioned inside a custom, non-default VPC:
 AssertionError: expected 'vpc-c93b06ae' to not equal 'vpc-c93b06ae'
 + expected - actual
 at Object.<anonymous> (tests/eks.spec.ts:25:38)
 at Generator.next (<anonymous>)
 at fulfilled (tests/eks.spec.ts:4:58)

 error: an unhandled error occurred: Program exited with non-zero exit code: 1

But the VPC test still fails. Let’s fix that one by creating a custom VPC – the awsx.ec2.Vpc component makes this easy – and then passing its resulting ID and subnet IDs as arguments.

const vpc = new awsx.ec2.Vpc("my-vpc");
export const cluster = new eks.Cluster("my-cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.publicSubnetIds,
 ...
});

For reference, here is the complete corrected index.ts file we have now:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import { runTests } from './tests';

// Create a basic EKS cluster.
const vpc = new awsx.ec2.Vpc("my-vpc");
export const cluster = new eks.Cluster("my-cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.publicSubnetIds,
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 storageClasses: "gp2",
 deployDashboard: false,
 version: "1.13",
});

// Run deployment validation.
runTests();

Now if we rerun pulumi up, we will see that the tests pass during the preview, landing us at the prompt asking us whether to deploy our infrastructure:

$ pulumi up
Previewing update (tdd-infra):

 Type Name Plan
 + pulumi:pulumi:Stack infra-eks-tdd-infra create
 + ├─ awsx:x:ec2:Vpc my-vpc create
...
 + └─ eks:index:Cluster my-cluster create
...

Diagnostics:
 pulumi:pulumi:Stack (infra-eks-tdd-infra):
 Running Mocha Tests: infra-eks/tests/eks.spec.ts
 #Amazon EKS Cluster
 ✓ Should use an approved Kubernetes version (839ms)
 ✓ Should be provisioned inside a custom, non-default VPC
 2 passing (928ms)

 warning: Skipped VPC ID check: not known during preview -- will test after apply

Resources:
 + 59 to create

Do you want to perform this update?
 yes
> no
 details

They passed! Notice, however, that the VPC test was skipped. That’s because we can’t know whether the resulting ID is going to end up resolving to the default VPC or not. There are myriad ways that the program could have supplied the ID, including by querying and looking up the default VPC ID dynamically and passing it as an argument.

If we choose yes and deploy the changes, we will a more confidently passing test:

Updating (tdd-infra):

 Type Name Status
 + pulumi:pulumi:Stack infra-eks-tdd-infra created
 + ├─ awsx:x:ec2:Vpc my-vpc created
...
 + └─ eks:index:Cluster my-cluster created
...

Diagnostics:
 pulumi:pulumi:Stack (infra-eks-tdd-infra):
 Running Mocha Tests: infra-eks/tests/eks.spec.ts
 #Amazon EKS Cluster
 ✓ Should use an approved Kubernetes version (666477ms)
 ✓ Should be provisioned inside a custom, non-default VPC (917ms)
 2 passing (11m)

Resources:
 + 59 created

Duration: 12m40s

Voila! A successfully stood up EKS cluster, with built-in TDD security safeguards.

Keeping Track of Failures

Anytime we try to deploy a program that fails, Pulumi is keeping a record of it.

For instance, let’s say somehow the VPC wasn’t known at preview time, and ended up resolving to the default VPC. Even though the deployment occurred, we’d want to know that we now have an issue that we need to go track down.

We can very clearly see such failures in the Pulumi Service:

And if we select it, we’ll see the complete Mocha test output:

We could even use webhooks to fire off a Slack alarm so that nobody misses the issue. This is often very helpful in unattended scenarios, like continuous deployment.

Better to catch these things late than never!

About Resource Properties and Outputs

A brief aside on resource properties is worthwhile, because they are “special” in some important ways when writing test cases.

All properties on resources are of type Output<T>. This behaves a lot like a promise, in that the value isn’t readily available to your program. Instead, you use the apply function to get at it, and Pulumi calls this when your resource’s properties are known.

Sometimes, however, properties cannot be known. A good example is the VPC ID we’re using above. Amazon will automatically assign a VPC ID when the resource is provisioned. That means that during previews, we won’t actually have a known value – it doesn’t even exist yet! We somehow need a way to deal with these situations.

The Output<T>.apply function tries to shield you from these circumstances. It would be maddening to need to check inside of these functions whether a value is known or not, and then you’d have all sorts of conditional code that differs between previews and updates – generally an anti-pattern, because the entire point of a preview is to show you exactly what will be done during an update. Any differences run the risk of showing different results.

Tests are special, however, in that you will need to think about what tests make sense to run before an update, versus those that should run after an update.

That’s why we need the promise<T> function defined earlier. It uses an internal promise() property to get at the raw underlying promise for any resource property, and returns it so you can either await it or call then in the usual promise ways. This does mean we aren’t hidden from the details of whether a preview is running or not.

This function, or functionality like it, will be made available as an official SDK feature as soon as pulumi/pulumi#2287 has been resolved.

By using this function in our tests, we have the following cases to consider:

The value is known, which could be due to the following cases
- It was explicitly supplied in our program
- It was a default populated by the underlying resource provider during a preview
- We are doing an update and the full outputs are available
The value is undefined, which could be due to the following cases
- We are in a preview and the value hasn’t been computed yet
- We have done an update and the property was not made available as an output value

In all cases, the pulumi.runtime.isDryRun function will return true during a preview.

In general, it’s easiest to write tests that just run at deployment time, rather than during a preview, when all of these different states are possible. That has the downside to not catching problems before they get deployed, however. As with many things, this is a tradeoff.

For a more complete overview of Output<T>, please see the Pulumi Programming Model documentation.

Next Steps

In this post, we’ve seen how to test your infrastructure using familiar test driven development (TDD) techniques and frameworks like Node.js’s Mocha and Chai libraries. This is a powerful capability that builds atop Pulumi’s use of general purpose languages, letting you apply existing software development best practices to your infrastructure.

Although we’ve just scratched the surface with our EKS cluster example, this capability can be used to enforce a wide array of team standards, best practices, and security guidelines. If you can express it in your favorite test framework, you can check it!

Everything we’ve seen in this article is open source and free to use – give Pulumi a try today.