Compatibility Testing Pulumi HCL

Pulumi HCL has at its core a simple promise:

A program that works for tofu apply will also work for pulumi up.

This must be true to allow Terraform modules to be shared between tofu config and Pulumi programs. This property makes testing Pulumi HCL simple. Let me explain.

At the end of the day, Pulumi is a system to translate actual state & desired state into a series of imperative actions, so actual state can be reconciled to desired state. Terraform is a system to translate actual state & desired state into a series of imperative actions, so actual state can be reconciled to desired state. How desired state is expressed can be radically different, and the underlying reconciliation engine can be radically different, but at the end of the day, both tools do the same thing:

Executing a Terraform program looks like this:

  flowchart LR
    tf["*.tf files"]
    current["current state"]
    subgraph engine["reconciliation engine"]
        direction LR
        desired["desired state"] --> internal["provider steps"]
    end
    output["provider steps"]

    tf --> desired
    current --> engine
    engine --> output

Executing a Pulumi program is more dynamic, because the reconciliation engine is in more active dialog with the user’s program. That said, the diagram is the same shape. To match semantics, Pulumi HCL dynamically bridges any Terraform provider in the registry. This means that, for the subset of Pulumi programs that are valid OpenTofu programs, both programs take the same input (*.tf files) and produce the same step output (Terraform provider steps). Providers are the part of our model that generates user-observable behavior, which means if we match what providers see, we match what users see. This gives us a really nice definition of correctness for Pulumi HCL¹:

Pulumi HCL correctly interprets an HCL program when it generates the same set of provider steps as tofu does.

How we compatibility test Pulumi HCL

We have created a framework to assert on the property above for Pulumi HCL: tfcompat. Each tfcompat test has 2 components:

• The files of the HCL program
• The providers the program uses

I’ll walk you through an example test case, then explain how the framework works.

An example tfcompat test

This Go test is the full code of TestL2SimpleResource:

// tests/tfcompat/l2_simple_resource_test.go

func TestL2SimpleResource(t *testing.T) {
	t.Parallel()
	tfcompat.RunCase(t, "l2_simple_resource", tfcompat.Case{
		Providers: []tfcompat.Provider{
			{Name: "simple", Factory: providers.SimpleProvider},
		},
	})
}

Factory is a function that produces a new in-memory Terraform provider called "simple". The "l2_simple_resource" in the test is the folder that contains the actual HCL program under test:

# tests/tfcompat/testdata/cases/l2_simple_resource/main.tf

resource "simple_resource" "a_resource" {
  input_one = "hello"
  input_two = true
}

output "some_output" {
  value = simple_resource.a_resource.result
}

This test asserts that Pulumi HCL & OpenTofu both:

• ConfigureProvider the simple provider the same way.
• Call the same plan RPC during pulumi preview & tofu plan.
• Call the same ApplyResourceChange to create the resource.
• Pulumi HCL or OpenTofu didn’t call any other provider RPCs.

One really important takeaway is that nowhere in this test case do we write down what Pulumi HCL should do. tfcompat.RunCase takes a scenario, but it doesn’t take accepted behavior. This will be important later. Before we get there, let me explain how tfcompat.RunCase works.

The anatomy of tfcompat.RunCase

Every tfcompat.RunCase runs 2 parallel processes, then compares the results:

• The Terraform Side: tfcompat.RunCase runs each Terraform provider in-memory, then copies the files in its test directory to a temp dir and runs tofu plan, then tofu apply against the temp dir. We use TF_REATTACH_PROVIDERS to have tofu attach to our in-memory Terraform providers.

• The Pulumi Side: tfcompat.RunCase runs each Terraform provider in-memory & copies the test files in its test directory to a separate test dir, and runs pulumi preview, then pulumi up against the temp dir. We use PULUMI_BRIDGE_REATTACH_PROVIDERS to instruct our dynamic bridge to attach to our in-memory provider.

For both Pulumi & Terraform, the test harness records each provider’s gRPC calls for all providers and it records the stack outputs for both invocations.

After both runs have completed, the test asserts that the outputs of the Pulumi program & the Terraform program match, and that the providers saw the same operations. A test case passes if and only if the providers for OpenTofu & Pulumi saw the same operations, and stack outputs were equal.

Writing tests with LLMs

Because tfcompat tests assert that Pulumi HCL matches OpenTofu, and not the test author’s idea of correctness, we can use LLMs to effectively hunt for bugs. Without additional constraints, telling Claude or Codex to find a bug will produce mostly false positives. Because our tests need only a scenario to test, it is very hard² for LLMs to produce false positives. This allows useful bug finding runs with as simple a prompt as:

I’d like you to do a pass trying to find bugs. You will prove each bug with a genuine failing tfcompat test. Start with 10 sub-agents. Bugs should not be duplicates and bugs should not reflect existing issues. Each failure should be stood up as a draft PR with just the failing test added. These PRs will fail CI. That is intentional. Don’t try to fix the bugs you solved. Keep the sub-agents running until you have found 10 failures. You are responsible for validating that the bugs are real and ensuring that the sub-agents do not create duplicate bugs, so you should create the PRs directly.

This is supported by a couple of skills, but this strategy does genuinely find high-quality bugs.

Because of how easy it is to send LLMs to hunt bugs, I think of this almost as a property-based test with LLMs as both the case generator and the reducer.

Conclusion

Having a strong and testable definition for Pulumi HCL makes it easy & fast to write integration tests, ensuring that our implementation is correct. LLMs are excellent at finding bugs when given the ability to write tests that fail if and only if they show a real divergence between our HCL implementation & OpenTofu, letting us hunt for bugs at LLM scale. All this testing has made us pretty confident that what we’ve shipped is pretty close to full parity with OpenTofu, and we’d love it if you gave it a try.