The Cloud Engineering Summit replay is available! See all your favorite talks, on demand. Watch Now

Property Testing

Policy as Code (also known as “CrossGuard”) is Pulumi’s offering to set guardrails and enforce compliance for cloud resources. Typically, policy packs would run across multiple projects and stacks to apply organization-wide rules.

Property Testing repurposes the power of policy definitions for developers to define invariants, or properties, that must hold for a specific stack they are working on. While Policy as Code and Property Testing both use the same technology, the goals and workflows are different.

This guide walks you through an example of a property test written in TypeScript. A sample Pulumi program provisions an Amazon EKS cluster. The test ensures two properties of the EKS cluster:

  • Running Kubernetes version 1.13.
  • Provisioned inside a private VPC, rather than the default one.

Blank Project

Our setup consists of two directories. The parent folder contains a typical Pulumi program written in TypeScript and bootstrapped with pulumi new aws-typescript command.

The sub-directory tests contains a policy pack with our future property tests and is created with pulumi policy new aws-typescript (notice the policy argument).

You can see the full layout in the examples repository.

Now it’s time to write the code!

Test-Driven Infrastructure

In true spirit of test-driven development (TDD), let’s start with the tests themselves.

Here is the content of our tests/index.ts test file:

import * as aws from "@pulumi/aws";
import * as policy from "@pulumi/policy";
import * as pulumi from "@pulumi/pulumi";

const stackPolicy: policy.StackValidationPolicy = {
    name: "eks-test",
    description: "EKS integration tests.",
    enforcementLevel: "mandatory",
    validateStack: async (args, reportViolation) => {
        const clusterResources = args.resources.filter(r => r.isType(aws.eks.Cluster));
        if (clusterResources.length !== 1) {
            reportViolation(`Expected one EKS Cluster but found ${clusterResources.length}`);
            return;
        }

        const cluster = clusterResources[0].asType(aws.eks.Cluster)!;

        // TODO 1: validate the cluster version

        // TODO 2: validate the cluster VPC
    },
}

const tests = new policy.PolicyPack("tests-pack", {
    policies: [stackPolicy],
});

This code does a few things worth describing. First, it imports all the packages that we’re going to use. Notably, this includes the Policy SDK package for testing, and the AWS and Pulumi SDK packages. Note that it does not import the Pulumi program with the EKS cluster definition. The tests are going to run against any program that satisfies its invariants.

Then, the code creates a single stack policy to describe the properties of the EKS cluster. The first implicit property is the fact that there is an EKS cluster in the stack at all. If the cluster is not found, or several clusters are found, the test reports a violation (failure).

Now, we can add the tests for our two properties. Add the following code in place of the first TODO item:

if (cluster.version !== "1.13") {
    reportViolation(
        `Expected EKS Cluster '${cluster.name}' version to be '1.13' but found '${cluster.version}'`);
}

The version test checks a property from our EKS cluster, cluster.version, and fails if the version is anything but 1.13 (including if it is unknown).

The VPC test is slightly more involved:

const vpcId = cluster.vpcConfig.vpcId;
if (!vpcId) {
    // 'isDryRun==true' means the test are running in preview.
    // If so, the VPC might not exist yet even though it's defined in the program.
    // We shouldn't fail the test then to avoid false negatives.
    if (!pulumi.runtime.isDryRun()) {
        reportViolation(`EKS Cluster '${cluster.name}' has unknown VPC`);
    }
    return;
}

const ec2 = new aws.sdk.EC2({region: aws.config.region});
const response = await ec2.describeVpcs().promise();
const defaultVpc = response.Vpcs?.find(vpc => vpc.IsDefault);
if (!defaultVpc) {
    reportViolation("Default VPC not found");
    return;
}

if (defaultVpc.VpcId === vpcId) {
    reportViolation(`EKS Cluster '${cluster.name}' should not use the default VPC`);
}

The first part asserts that there is a non-empty VPC identifier assigned to the cluster. Then, the test uses the AWS SDK to retrieve the default VPC. Finally, it compares the two IDs to see if they are equal and report a violation if so.

Our Base (Failing) Program

Here is the program we’ll be testing. Keeping with our TDD theme, let’s start with the tests failing to begin with (we are using the default VPC and not specifying a version):

import * as eks from "@pulumi/eks";

// Create a basic EKS cluster.
const cluster = new eks.Cluster("my-cluster", {
    desiredCapacity: 2,
    minSize: 1,
    maxSize: 2,
    storageClasses: "gp2",
    deployDashboard: false,
});

To run our tests, we need to run the deployment with pulumi up but also pass an extra parameter to point to the tests policies. As expected, the result shows a test failure at the bottom:

$ pulumi up --policy-pack tests

Previewing update (dev):
...

Policy Violations:
    [mandatory]  tests-pack v1  eks-test (pac-ts-eks-dev: pulumi:pulumi:Stack)
    EKS integration tests.
    Expected EKS Cluster 'my-cluster-eksCluster-3187fd6' version to be '1.13' but found 'undefined'

Note that only one test failed: the VPC test requires the actual deployment to run to retrieve a VPC ID because ID is unknown during the preview (the VPC does not exist yet).

Fixing Our Program

Now let’s refactor our infrastructure so that the failing test starts passing. In the TDD spirit, we only fix the version test now, because it was the one with a violation.

To fix the first problem, we need to pass the Kubernetes version explicitly when creating our cluster. That’s as simple as passing a new argument:

const cluster = new eks.Cluster("my-cluster", {
    ...
    version: "1.13",
});

If we rerun pulumi up --policy-pack tests, we’ll see that the preview now passes:

$ pulumi up --policy-pack tests

Previewing update (dev):
...

Policy Packs run:
    Name                Version
    tests-pack (tests)  (local)

Do you want to perform this update?
  yes
> no
  details

Choose ‘yes’ to deploy the infrastructure and rerun the tests. After the update complete, the tests run again to inspect the actual resources. Predictably, the VPC test fails now:

...
Policy Violations:
    [mandatory]  tests-pack v1  eks-test (pac-ts-eks-dev: pulumi:pulumi:Stack)
    EKS integration tests.
    EKS Cluster 'my-cluster-eksCluster-55b01ab' should not use the default VPC

Let’s fix the second test by creating a custom VPC—the awsx.ec2.Vpc component makes this easy—and then passing its resulting ID and subnet IDs as arguments.

const vpc = new awsx.ec2.Vpc("my-vpc");
const cluster = new eks.Cluster("my-cluster", {
    vpcId: vpc.id,
    subnetIds: vpc.publicSubnetIds,
    ...
});

For reference, here is the complete, corrected index.ts file we have now:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a custom VPC.
const vpc = new awsx.ec2.Vpc("my-vpc");

// Create a basic EKS cluster.
const cluster = new eks.Cluster("my-cluster", {
    desiredCapacity: 2,
    minSize: 1,
    maxSize: 2,
    storageClasses: "gp2",
    deployDashboard: false,
    version: "1.13",
    vpcId: vpc.id,
    subnetIds: vpc.publicSubnetIds,
});

If we run pulumi up --policy-pack tests -y, we’ll see that all the tests now pass after the VPC is created and the cluster is replaced:

$ pulumi up --policy-pack tests -y
...
Resources:
    + 30 created
    ~ 2 updated
    +-14 replaced
    46 changes. 13 unchanged

Policy Packs run:
    Name                Version
    tests-pack (tests)  (local)

Voila! A successfully stood up EKS cluster, with built-in TDD security safeguards.

Full Example

A complete runnable example of property tests is available in the examples repository: Property Testing with TypeScript.