Pulumi Blog: Ian Wahbe

Compatibility Testing Pulumi HCL

Ian Wahbe — Fri, 14 Aug 2026 00:00:00 +0000

Pulumi HCL has at its core a simple promise:

A program that works for tofu apply will also work for pulumi up.

This must be true to allow Terraform modules to be shared between tofu config and Pulumi programs. This property makes testing Pulumi HCL simple. Let me explain.

At the end of the day, Pulumi is a system to translate actual state & desired state into a series of imperative actions, so actual state can be reconciled to desired state. Terraform is a system to translate actual state & desired state into a series of imperative actions, so actual state can be reconciled to desired state. How desired state is expressed can be radically different, and the underlying reconciliation engine can be radically different, but at the end of the day, both tools do the same thing:

Executing a Terraform program looks like this:

flowchart LR
tf["*.tf files"]
current["current state"]
subgraph engine["reconciliation engine"]
direction LR
desired["desired state"] --> internal["provider steps"]
end
output["provider steps"]
tf --> desired
current --> engine
engine --> output

Executing a Pulumi program is more dynamic, because the reconciliation engine is in more active dialog with the user’s program. That said, the diagram is the same shape. To match semantics, Pulumi HCL dynamically bridges any Terraform provider in the registry. This means that, for the subset of Pulumi programs that are valid OpenTofu programs, both programs take the same input (*.tf files) and produce the same step output (Terraform provider steps). Providers are the part of our model that generates user-observable behavior, which means if we match what providers see, we match what users see. This gives us a really nice definition of correctness for Pulumi HCL¹:

Pulumi HCL correctly interprets an HCL program when it generates the same set of provider steps as tofu does.

If you are familiar with property-based testing, you might be thinking this looks like a testable property. You’re right.

How we compatibility test Pulumi HCL

We have created a framework to assert on the property above for Pulumi HCL: tfcompat. Each tfcompat test has 2 components:

The files of the HCL program
The providers the program uses

I’ll walk you through an example test case, then explain how the framework works.

An example `tfcompat` test

This Go test is the full code of TestL2SimpleResource:

// tests/tfcompat/l2_simple_resource_test.go

func TestL2SimpleResource(t *testing.T) {
 t.Parallel()
 tfcompat.RunCase(t, "l2_simple_resource", tfcompat.Case{
 Providers: []tfcompat.Provider{
 {Name: "simple", Factory: providers.SimpleProvider},
 },
 })
}

Factory is a function that produces a new in-memory Terraform provider called "simple". The "l2_simple_resource" in the test is the folder that contains the actual HCL program under test:

# tests/tfcompat/testdata/cases/l2_simple_resource/main.tf

resource "simple_resource" "a_resource" {
 input_one = "hello"
 input_two = true
}

output "some_output" {
 value = simple_resource.a_resource.result
}

This test asserts that Pulumi HCL & OpenTofu both:

ConfigureProvider the simple provider the same way.
Call the same plan RPC during pulumi preview & tofu plan.
Call the same ApplyResourceChange to create the resource.
Pulumi HCL or OpenTofu didn’t call any other provider RPCs.

One really important takeaway is that nowhere in this test case do we write down what Pulumi HCL should do. tfcompat.RunCase takes a scenario, but it doesn’t take accepted behavior. This will be important later. Before we get there, let me explain how tfcompat.RunCase works.

The anatomy of `tfcompat.RunCase`

Every tfcompat.RunCase runs 2 parallel processes, then compares the results:

The Terraform Side: tfcompat.RunCase runs each Terraform provider in-memory, then copies the files in its test directory to a temp dir and runs tofu plan, then tofu apply against the temp dir. We use TF_REATTACH_PROVIDERS to have tofu attach to our in-memory Terraform providers.
The Pulumi Side: tfcompat.RunCase runs each Terraform provider in-memory & copies the test files in its test directory to a separate test dir, and runs pulumi preview, then pulumi up against the temp dir. We use PULUMI_BRIDGE_REATTACH_PROVIDERS to instruct our dynamic bridge to attach to our in-memory provider.

For both Pulumi & Terraform, the test harness records each provider’s gRPC calls for all providers and it records the stack outputs for both invocations.

After both runs have completed, the test asserts that the outputs of the Pulumi program & the Terraform program match, and that the providers saw the same operations. A test case passes if and only if the providers for OpenTofu & Pulumi saw the same operations, and stack outputs were equal.

Writing tests with LLMs

Because tfcompat tests assert that Pulumi HCL matches OpenTofu, and not the test author’s idea of correctness, we can use LLMs to effectively hunt for bugs. Without additional constraints, telling Claude or Codex to find a bug will produce mostly false positives. Because our tests need only a scenario to test, it is very hard² for LLMs to produce false positives. This allows useful bug finding runs with as simple a prompt as:

I’d like you to do a pass trying to find bugs. You will prove each bug with a genuine failing tfcompat test. Start with 10 sub-agents. Bugs should not be duplicates and bugs should not reflect existing issues. Each failure should be stood up as a draft PR with just the failing test added. These PRs will fail CI. That is intentional. Don’t try to fix the bugs you solved. Keep the sub-agents running until you have found 10 failures. You are responsible for validating that the bugs are real and ensuring that the sub-agents do not create duplicate bugs, so you should create the PRs directly.

This is supported by a couple of skills, but this strategy does genuinely find high-quality bugs.

Because of how easy it is to send LLMs to hunt bugs, I think of this almost as a property-based test with LLMs as both the case generator and the reducer.

Conclusion

Having a strong and testable definition for Pulumi HCL makes it easy & fast to write integration tests, ensuring that our implementation is correct. LLMs are excellent at finding bugs when given the ability to write tests that fail if and only if they show a real divergence between our HCL implementation & OpenTofu, letting us hunt for bugs at LLM scale. All this testing has made us pretty confident that what we’ve shipped is pretty close to full parity with OpenTofu, and we’d love it if you gave it a try.

Pulumi HCL accepts a superset of what OpenTofu accepts. This method only applies to the subset of programs that OpenTofu accepts. ↩︎
Typically, false positives come from an LLM finding a scenario where tofu apply errors, but pulumi up behaves correctly. ↩︎

Emulating Terraform on Pulumi's Engine

Ian Wahbe — Tue, 04 Aug 2026 00:00:00 +0000

The core promise of Pulumi’s HCL support is that you can bring your existing Terraform configuration and modules, and pulumi will run them. If it works in OpenTofu and doesn’t work in Pulumi, we would like to fix that. Given that goal, our HCL interpreter needs to take HCL as input and emit instructions to the Pulumi engine that semantically match how tofu would interpret the same input. This is made harder by the fact that Pulumi and OpenTofu have fundamentally different engine semantics and provider ecosystems. This blog post will explore how we have implemented that mapping well enough to get 96%¹ of our top Terraform modules working on Pulumi. We’ll briefly walk through how Pulumi’s HCL interpreter handles Terraform’s resource semantics, providers, and modules. It will also call out where Pulumi’s HCL support lets you do things that Terraform and OpenTofu will not allow.

Providers

Both Pulumi and Terraform have providers, but they don’t have the same providers. While there are providers that Terraform does not have, Pulumi can always resolve a Terraform provider using Pulumi’s confusingly named terraform-provider provider.² This is the same provider that lets you consume Any Terraform Provider in another Pulumi program with pulumi package add terraform-provider .... The terraform-provider provider acts as a relay: it speaks Pulumi’s protocol to the Pulumi engine, and speaks Terraform’s provider protocol to the Terraform provider it stands up. Because Pulumi HCL needs to work with all Pulumi providers and because terraform-provider lets Pulumi HCL speak to Terraform providers via the Pulumi protocol, Pulumi HCL actually only speaks Pulumi protocols directly:

flowchart LR
subgraph n2Entry[" "]
n2["terraform-provider"]
end
subgraph providerBox["Pulumi Provider"]
direction TD
n2Entry
n3["Terraform Provider"]
end
n0["Pulumi HCL"] <--> n1["Pulumi Engine"]
n1 <--> n2Entry
n2 <--> n3
n2@{ shape: rect}
n3@{ shape: rect}
n0@{ shape: rect}
n1@{ shape: rect}
style n2Entry fill:transparent,stroke:transparent

All internals of github.com/pulumi/pulumi-hcl are implemented in Pulumi’s language protocol.

Because the terraform-provider natively understands Terraform version ranges and defaults to the OpenTofu registry, we can directly translate normal provider requests to the terraform-provider. It takes its arguments as an untyped list of strings, since the interface was originally intended for the command line. Let’s walk through some simple examples:

resource "aws_s3_bucket" "example" {
 bucket = "my-bucket-123"
}

Pulumi’s HCL interpreter sees that there is no terraform.required_providers block, so it cuts the resource token at the first _ and uses the default registry and namespace. This is what the request that goes to the Pulumi engine looks like:

pulumirpc.PackageSpec{
 Source: "terraform-provider",
 Parameters: []string{"registry.opentofu.org/hashicorp/aws"},
}

Because no version was specified, we leave it to terraform-provider to determine the version. It will use the latest version.

Just like Terraform, you can override this with a required_providers block:

terraform {
 required_providers {
 example = {
 source = "my.custom.registry/me/example"
 version = "~> 5.0"
 }
 }
}

resource "example_resource" "another_example" {
}

We perform the same mechanical translation. The source is fully specified, and there is a version, so we pass it along to the Pulumi engine, which passes it along to terraform-provider:

pulumirpc.PackageSpec{
 Source: "terraform-provider",
 Parameters: []string{"my.custom.registry/me/example", "~> 5.0"},
}

Pulumi HCL will automatically translate almost all provider calls through the terraform-provider translator. Providers with the source pulumi/* will instead be routed directly. This is how you can use a native Pulumi provider in Pulumi HCL:

terraform {
 required_providers {
 kubernetes = {
 source = "pulumi/kubernetes"
 version = "4.33.0"
 }
 }
}

resource "kubernetes_yaml_config_file" "app" {
 file = "app.yaml"
}

Our HCL interpreter routes this directly to the Pulumi Kubernetes package:

pulumirpc.PackageSpec{
 Source: "kubernetes",
 Version: "4.33.0",
}

Observe that the version moved from Parameters to the Version field. That’s because providers written with source = "pulumi/*" are talking about the plugin directly.

Resources

Both Pulumi programs and Terraform config exist to express a resource graph. It is their primary purpose. I don’t have the time or the pixels to explain everything, so I’ll restrict myself to three sub-topics here:

Property name translation
Provisioners and conditions
Resource options

Property name translation

If you’ve ever looked at the schema behind any of our providers, you will observe that names are camelCase.

 "aws:s3/bucket:Bucket": {
 "properties": {
 "accelerationStatus": { "type": "string", "description": "...", "deprecationMessage": "..." },
 "bucketDomainName": { "type": "string", "description": "..." },
 "bucketNamespace": { "type": "string", "description": "..." },
 ...

Our codegen actually relies on this fact, which means that even the schemas produced by terraform-provider use camelCase property names. The clever among you will see the problem: Terraform uses snake_case by convention. When Pulumi HCL sends a resource request with property values, it needs to translate from the snake_case text to camelCase for the engine, and then back again on values returned by the engine.

sequenceDiagram
participant Program as HCL program<br/>(snake_case)
participant Interp as HCL interpreter
participant Engine as Pulumi engine<br/>(camelCase)
Interp->>Engine: GetSchema("example_resource")
Engine-->>Interp: schema (property names + types)
Program->>Interp: some_value = "true"
Interp->>Engine: RegisterResource { someValue: true }
Engine-->>Interp: outputs { someValue: true, computedValue: "..." }
Interp-->>Program: example_resource.example.computed_value

Doing this correctly actually requires you to be aware of the resource’s schema; the raw value is not enough. Let’s work through an example to understand why:

resource "example_resource" "example" {
 some_value = "true"
 block {
 inner_value = false
 }
 another_attribute = {
 "trap_one" = 1
 "trap_two" = 2
 }
}

Here are two different value blobs (represented as JSON) that correspond to the inputs for "example" above. Both are valid, depending on the type of "example_resource":

{
 "someValue": "true",
 "block": {
 "innerValue": false
 },
 "anotherAttribute": {
 "trap_one": 1,
 "trap_two": 2
 }
}

{
 "someValue": true,
 "block": [{
 "innerValue": "false"
 }],
 "anotherAttribute": {
 "trapOne": "1",
 "trapTwo": 2
 }
}

Let’s break down the difference between the two potential value blobs:

HCL converts scalar values to the type the provider expects, so if someValue is typed as a string, we keep the "true". If it’s typed as a boolean, we convert and pass the provider the converted value: true.
If block is typed as an object, the Pulumi provider expects to see an object back on the wire. If block is typed as a list of objects, then the Pulumi provider needs to see a list of objects back, even if it’s a list of one.
innerValue can have the same type conversion as someValue. Type conversion doesn’t stop at the top level.
If anotherAttribute is a map type, then its keys are user-provided values and need to be kept as they are. If it’s an object type, then its keys need to be shifted back to camelCase.

Pulumi’s HCL interpreter is thus type aware. It queries the engine for the schema of each provider it translates for and does the correct conversion, tracking the target type as it walks the value for translation. This is not limited to resources; it does the same translation for data sources, modules, and provider blocks.

For Pulumi providers that are bridged from Terraform providers, we apply an additional step. Bridged providers expose an explicit naming table used by pulumi convert --from terraform. Pulumi HCL queries the engine for this mapping also, and uses that to make sure that resource and property names line up exactly with the underlying Terraform provider, even if the property was renamed or hidden in Pulumi.

Provisioners and conditions

Terraform provisioners allow the Terraform config to specify commands to run at various points in the resource lifecycle. Conditions allow the same, but for boolean expressions to be evaluated instead. For example:

resource "example_resource" "example" {
 input = var.input

 lifecycle {
 precondition {
 condition = var.input > 3
 error_message = "var.input must exceed 3"
 }
 }

 provisioner "local-exec" {
 command = "echo An example output is ${self.output} >> file.txt"
 }
}

Pulumi only has one mechanism to hook into the resource lifecycle: hooks. Pulumi’s resource hooks are powerful enough to express both provisioners and conditions. The easiest way I can express this is by comparison to another language. I’ll use TypeScript. The example configuration block above is equivalent to this TypeScript program:

const config = new pulumi.Config();

const precondition = new pulumi.ResourceHook("example_resource:precondition:example", async args => {
 const inputs = args.newInputs as example.ResourceArgs;
 if (inputs.input <= 3) {
 throw new Error("var.input must exceed 3");
 }
});

const provisioner = new pulumi.ResourceHook("example_resource:provisioner:example", async args => {
 const self = args.newOutputs as example.ResourceState;
 const { exec } = require('child_process');
 exec(`echo An example output is ${self.output} >> file.txt`);
});

new example.Resource("example", {
 input: config.requireNumber("input"),
}, {
 hooks: {
 beforeCreate: [precondition],
 beforeUpdate: [precondition],
 afterCreate: [provisioner],
 },
});

The full mapping between Terraform provisioners and conditions and Pulumi hooks can be found below:

Terraform	Pulumi
`lifecycle.precondition`	`beforeCreate`, `beforeUpdate`
`lifecycle.postcondition`	`afterCreate`, `afterUpdate`
`provisioner` (creation-time)	`afterCreate`
`provisioner` with `when = destroy`	`beforeDelete`
`lifecycle.prevent_destroy`	`beforeDelete` (always errors)

Creation-time provisioners bind only to afterCreate: Terraform does not re-run provisioners on update, and neither do we.

Resource options

Our final challenge for resources is Terraform’s various resource options. Resource options in both Terraform and Pulumi exist to give special instructions to the engine concerning a specific resource. We can classify Terraform’s resource options into two kinds:

Those that can be handled at the language level.
Those that need engine support.

Let’s start by going through those that can be handled at the language level without engine support in Pulumi:

count/for_each: The HCL interpreter unrolls count and for_each and sends a request for each underlying resource. These are equivalent to using a for-loop in any of Pulumi’s programming languages.
lifecycle.prevent_destroy: Pulumi doesn’t have an equivalent in-language hook (protect is stored in state). We emulate prevent_destroy with a beforeDelete hook, exactly like our destroy-time provisioners.

For most of the resource options that require real engine support, Pulumi has equivalent options:

lifecycle.replace_triggered_by: Terraform allows specifying either resources or values here. We evaluate every element and feed it to replacementTrigger. Elements that reference a resource also contribute that resource to replaceWith, which covers the case where the referenced resource is replaced without any of its attribute values changing.
lifecycle.ignore_changes: We map this directly to Pulumi’s ignoreChanges. We translate the paths from Terraform’s snake_case to Pulumi’s camelCase for you.
lifecycle.create_before_destroy: To replicate Terraform’s default behavior, we register all resources with deleteBeforeReplace set to true. When lifecycle.create_before_destroy is set, we go back to Pulumi’s default behavior here.
provider: Pulumi has exactly this concept. We pass it through to the engine as is.
depends_on: We pass this directly to the Pulumi engine.

That’s how we map Terraform’s resource options. Of course, Pulumi has its own set of resource options, and we expose those on resources with a pulumi block. You can see the remaining resource options in the Pulumi resource options docs page.

Modules

Terraform has modules, and Pulumi has components. Naturally, we represent Terraform modules as components. Under the hood, Pulumi has two different kinds of components:

In-language components
Multi-language components (MLCs)

In-language components are components that are consumed directly within the language. They don’t need the Pulumi engine’s intervention to serve them. In Pulumi HCL, this is what you get when you write a module block. The same language host running the rest of your program loads that HCL and interprets it.

Pulumi HCL also supports MLCs as both a consumer and a provider. That means it can work with the engine to let you consume MLCs written in other Pulumi languages in HCL, and that you can consume HCL modules in other Pulumi languages.

From HCL’s perspective, an MLC is just like any other resource, so consuming one looks like any other resource construction. Here is what it looks like to consume our AWSx VPC resource (written in TypeScript) in HCL:

terraform {
 required_providers {
 awsx = {
 source = "pulumi/awsx"
 }
 }
}

resource "awsx_ec2_vpc" "example" {
 availability_zone_names = ["us-west-2a", "us-west-2b"]
}

We can consume HCL modules in other Pulumi languages as well. Here is what it looks like to consume the unrelated terraform-aws-modules/vpc/aws module in a Pulumi language:

import * as pulumi from "@pulumi/pulumi";
import * as hcl from "@pulumi/hcl";

const vpc = new hcl.Module("vpc", {
 source: "terraform-aws-modules/vpc/aws",
 version: "5.0.0",
 inputs: {
 name: "example-vpc",
 cidr: "10.0.0.0/16",
 },
});

export const vpcId = vpc.outputs.apply(o => o["vpc_id"]);

import pulumi
import pulumi_hcl as hcl

vpc = hcl.Module("vpc",
 source="terraform-aws-modules/vpc/aws",
 version="5.0.0",
 inputs={
 "name": "example-vpc",
 "cidr": "10.0.0.0/16",
 })

pulumi.export("vpc_id", vpc.outputs["vpc_id"])

package main

import (
 "github.com/pulumi/pulumi-hcl/sdk/go/hcl"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 vpc, err := hcl.NewModule(ctx, "vpc", &hcl.ModuleArgs{
 Source: "terraform-aws-modules/vpc/aws",
 Version: pulumi.StringRef("5.0.0"),
 Inputs: pulumi.Map{
 "name": pulumi.String("example-vpc"),
 "cidr": pulumi.String("10.0.0.0/16"),
 },
 })
 if err != nil {
 return err
 }
 ctx.Export("vpcId", vpc.Outputs.MapIndex(pulumi.String("vpc_id")))
 return nil
 })
}

using System.Collections.Generic;
using Pulumi;
using Pulumi.Hcl;

return await Deployment.RunAsync(() =>
{
 var vpc = new Module("vpc", new ModuleArgs
 {
 Source = "terraform-aws-modules/vpc/aws",
 Version = "5.0.0",
 Inputs =
 {
 { "name", "example-vpc" },
 { "cidr", "10.0.0.0/16" },
 },
 });

 return new Dictionary<string, object?>
 {
 ["vpcId"] = vpc.Outputs.Apply(o => o["vpc_id"]),
 };
});

package myapp;

import com.pulumi.Pulumi;
import com.pulumi.hcl.Module;
import com.pulumi.hcl.ModuleArgs;
import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 var vpc = new Module("vpc", ModuleArgs.builder()
 .source("terraform-aws-modules/vpc/aws")
 .version("5.0.0")
 .inputs(Map.of(
 "name", "example-vpc",
 "cidr", "10.0.0.0/16"))
 .build());

 ctx.export("vpcId", vpc.outputs().applyValue(o -> o.get("vpc_id")));
 });
 }
}

name: example
runtime: yaml
resources:
 vpc:
 type: hcl:index:Module
 properties:
 source: terraform-aws-modules/vpc/aws
 version: "5.0.0"
 inputs:
 name: example-vpc
 cidr: 10.0.0.0/16
outputs:
 vpcId: ${vpc.outputs["vpc_id"]}

If you want strongly typed SDKs for your Terraform modules, you can generate them with pulumi package add.

Conclusion

This has been a brief survey of how we have mapped Terraform’s semantics onto Pulumi’s engine. Providers are bridged into Pulumi, resource options are translated or handled directly in the Pulumi HCL interpreter, and modules are components… in any language.

If you want to try it yourself, start with the get-started guide. And if you find a program that works in OpenTofu but not in Pulumi, that’s a bug: file an issue and we would love to fix it.

This is 56/58 of the top Terraform AWS Module by usage. Failures are due to ephemeral resources. ↩︎
I named it, naming is hard. I wanted you to be able to type pulumi package add terraform-provider <your-provider>. ↩︎

Introducing: Support For Using Any Terraform Provider with Pulumi

Luke Hoban — Thu, 29 Aug 2024 10:00:00 -0600

For the most up-to-date information about using any Terraform provider with Pulumi, see the Any Terraform Provider documentation.

One of our core goals at Pulumi is to provide access to manage any cloud infrastructure with a single unified programming model. Whether it’s multi-cloud (AWS+Azure+Kubernetes), hybrid cloud (GCP+VMWare+Cisco), or managed services (Databricks+GitHub+Cloudflare), Pulumi makes it easy to deploy and manage infrastructure across all of your cloud environments using any of the 150+ cloud providers in the Pulumi Registry.

We’re excited to take this even further by introducing support for using any Terraform or OpenTofu provider from within your Pulumi programs. If there is a long tail Cloud or SaaS platform that has a provider for those ecosystems, it now works with Pulumi as well. And if your organization has built your own custom Terraform or OpenTofu provider to support an internal cloud platform, you can use it from Pulumi as well, without having to publish it to any registry.

Terraform providers are the closest thing the industry has today to a “defacto standard” for describing the resource model of a cloud platform. The vast majority of these providers are implemented by cloud providers, partners, or cloud ecosystem contributors - as a way to describe what resources are available in their platform that can be managed via various kinds of infrastructure tooling. These investments become even more impactful when the work done to define the resource model can be widely leveraged by other tools for a variety of purposes - from IaC to Policy to Asset Inventory and more. There is nothing innately Terraform-specific about these providers - they are simply a standard resource model and the implementation of CRUD (Create-Read-Update-Delete) operations against that model.

At Pulumi, we’ve supported bridging Terraform providers for use in Pulumi from the beginning. Pulumi’s provider model is even more expressive than Terraform’s and we have a defined way to map Terraform providers (as well as other provider ecosystems) into the Pulumi provider model to be used by Pulumi’s CLI and deployment orchestration engine. Many of the most used providers in the Pulumi ecosystem are bridged Terraform providers. However, until now, bridging a new provider required a decent amount of up front and ongoing work, to maintain a separate repository, build and push separate releases, and publish resulting SDKs to every language’s package manager. This meant that while the largest and most used Terraform providers were available in Pulumi, the long tail of providers were not as easily accessible. With this launch, we are now opening up easy access to all of these providers.

Using any Terraform or OpenTofu provider takes just a few seconds using a single command:

$ pulumi package add terraform-provider <your-terraform-provider>

This is enabled by our new Local Packages feature, which lets you generate Pulumi Packages locally into your Pulumi project, instead of relying only on packages published to the Pulumi Registry. Support for generating local packages is available for all Pulumi languages.

Local packages are available as a convenient alternative to the existing support for authoring Native or Bridged Pulumi Packages. You can read more about all of the options available to authoring and using Pulumi Packages in the Pulumi Packages documentation, or about how to generate a local package for a Terraform or OpenTofu provider in the Any Terraform Provider docs.

Walkthrough

Imagine you are working on a new cloud infrastructure project, and come across a technology you want to adopt, but don’t yet see a Pulumi provider in the Pulumi Registry. What do you do?

Let’s walk through a concrete example. We’ll work with a Python project:

$ pulumi new python

Planetscale is an exciting new fully-managed MySQL database platform. Although there isn’t yet an official Pulumi provider for Planetscale in the Pulumi Registry, we can easily use Planetscale from Pulumi if we want to. First, we generate a local package using the pulumi package add command.

$ pulumi package add terraform-provider planetscale/planetscale
Successfully generated a Python SDK for the planetscale package at /Users/lukehoban/github/lukehoban/anytftest/sdks/planetscale

To use this SDK in your Python project, run the following command:

 echo sdks/planetscale >> requirements.txt

 pulumi install

You can then import the SDK in your Python code with:

 import pulumi_planetscale as planetscale

We ask to use the terraform-provider package, which implements support for any Terraform or OpenTofu provider, and then ask specifically for support for the planetscale/planetscale provider. By default, this is pulled from the OpenTofu registry, but you can also specify a local path to the provider, or a fully-qualified reference to any Terraform Registry API-compatible server.

As we see from the output, this generates a local ./sdks/planetscale folder with a Python SDK for using the Planetscale provider.

After running the two commands provided in the output, we now have the local SDK incorporated into our project:

$ echo sdks/planetscale >> requirements.txt
$ pulumi install

We can add some code that uses this new SDK:

import pulumi
import pulumi_planetscale as planetscale

db = planetscale.Database(
 resource_name="mydb",
 organization="lukehoban",
 cluster_size="PS-10"
)

pulumi.export("db_url", db.url)

We get strong typing, completion lists, and error reporting over this SDK from within our IDE.

Finally, we can deploy our code, deploying a new Planetscale database directly from Pulumi:

$ pulumi up
Updating (luke-pulumi-corp/dev2)

View in Browser (Ctrl+O): https://app.pulumi.com/luke-pulumi-corp/anytftest/dev2/updates/2

 Type Name Status
 pulumi:pulumi:Stack anytftest-dev2
 + └─ planetscale:index:Database mydb created (1s)

Outputs:
 + db_url: "https://api.planetscale.com/v1/organizations/lukehoban/databases/mydb-77af236"

Resources:
 + 1 created
 1 unchanged

Duration: 3s

Clicking on the link, we see our new managed database ready for connections! Within just a few seconds, we’ve incorporated support for this new service into our Pulumi infrastructure!

Planetscale is an exciting example, but it is just one of thousands of 3rd party (published in a public registry) and 2nd party (published internally within an organization) providers that are now available for use within Pulumi.

Why Support All Terraform Providers?

Pulumi has a rich and expressive Native Provider model for building providers directly against the full capabilities of the Pulumi Infrastructure as Code resource model. Many of the most important Pulumi providers are built directly on this foundation, including our Azure Native and Kubernetes providers. So why are we excited to support using Terraform and OpenTofu providers from Pulumi?

In short, we believe in giving Pulumi users access to every ecosystem of great providers and components available across the industry and in meeting partners where they are. If a partner or user has built something exciting for managing infrastructure in the cloud, we want Pulumi users to benefit from it. Similarly, we want the partner or user to easily extend the value of their provider to also be able to reach the Pulumi user base, and all of the benefits that Pulumi can offer for Infrastructure as Code management, with no extra effort on their part.

We’ve embraced this approach in many other areas as well:

Pulumi enables deploying and managing the lifecycle of any Kubernetes YAML file or Helm Chart. There’s a great ecosystem and library of Helm charts, and we make those all available to Pulumi users without any wrappers or indirections.
We introduced the Pulumi AWS CDK Adapter to enable Pulumi users to incorporate AWS CDK constructs directly into their Pulumi programs without rewriting in Pulumi first.
And, of course, Pulumi offers features for deploying and managing Azure Resource Manager templates and CloudFormation templates, enabling easy deployment of existing cloud provider IaC artifacts.

With the new support for Any Terraform Provider, we’re bringing this same experience to the Terraform and OpenTofu provider ecosystems.

Use any Terraform provider

Add any Terraform or OpenTofu provider to your Pulumi program with a single command, and manage it alongside the rest of your cloud infrastructure in one workflow.

Learn more

How Does This Work?

Two new technologies in the Pulumi open source project underpin support for using any Terraform or OpenTofu provider.

Parameterized Providers offer support for building a single provider that can implement many different “flavors.” For example, the terraform-provider provider can be parameterized to provide support for any Terraform or OpenTofu provider.

Local Packages offer the ability to generate a Pulumi SDK for the language of your Pulumi project into a local folder, which can be directly incorporated into your project via your existing package manager along with pulumi install.

Local packages are versioned when you include a version in the pulumi package add command, and parameterized providers support many different formats for their parameters. For example, the most general form of the command to use an existing Terraform provider is:

$ pulumi package add terraform-provider [<registry>/]<author>/<name> [version]

Or:

$ pulumi package add terraform-provider /path/to/my/terraform-provider-binary

What’s Next?

Today’s release is a big step in opening up access to even more providers for Pulumi users. But there is still a lot more we have planned to build on top of these new foundations.

First, with locally generated SDKs, the documentation is available within the language SDK you generate (and so lights up in your IDE or development environment!). Still, there is no static docs page available in the Pulumi Registry to find and link to on the web. In the future, we plan to offer the ability to publish documentation for locally bridged providers in the Registry. (tracked in registry#5302) In that case, the registry entry will be only the docs - the SDKs and provider binary will not need to be published/hosted in the Registry itself.

Second, the underlying Local Packages and Parameterized Providers technologies for building different flavors of a single provider are applicable to many other use cases. In addition to having a single provider that can be “parameterized” to act as any Terraform provider, we have plans to apply this technology to additional use cases, such as generating strongly typed providers over any Kubernetes CRD, any Helm Chart, any other Pulumi program, and many more ecosystems of exciting infrastructure components which can be deployed via Pulumi.

Take the new support for using any Terraform or OpenTofu provider from Pulumi for a spin today:

Repairing State With Pulumi Refresh

Ian Wahbe — Thu, 01 Sep 2022 10:19:47 +0200

Under the hood, Pulumi is a desired state engine. This means that you tell Pulumi what you want, Pulumi knows what already exists, and it makes targeted changes to match the state of the world with your desired state. This works great as long as Pulumi understands the state of the world, which it nearly always does. We will discuss how pulumi refresh can be used to bring Pulumi’s state back inline with external state.

The Problem

There are scenarios where Pulumi’s understanding of the world differs from what actually exists. For Pulumi to work correctly, it needs to reconcile its known state with the actual state. Before we explore how Pulumi solves this problem, we should explore how the Pulumi state gets out of sync with the actual state of the world. There are two main ways for Pulumi to get out of sync: out of band changes and terminating Pulumi mid-operation. I will explain each in turn.

Out of Band Changes

Pulumi expects to be the sole controller for the resources it provisions. An out of band change happens when this principle is violated. This is best illustrated with an example. Imagine creating an S3 bucket with Pulumi:

const bucket = new aws_native.s3.Bucket("my-bucket", {
 tags: [{
 key: "pulumi-bucket",
 value: "not important",
 }],
});

Then someone goes into the AWS console and deletes the bucket. What happens if you then go change a tag on the bucket:

const bucket = new aws_native.s3.Bucket("my-bucket", {
 tags: [{
 key: "pulumi-bucket",
 value: "don’t delete me",
 }],
});

Pulumi has recorded creating a bucket, but it does not know that the bucket was deleted. It will try to update the bucket and fail, since the bucket does not exist anymore.

Terminating Pulumi Mid-Operation

Pulumi also gets confused if the CLI or Pulumi Provider process is terminated while an update is taking place. The Pulumi CLI checks in its state before and after each operation it performs. Creating the bucket from above goes something like:

Started pulumi up.
About to create aws_native:s3:Bucket::my-bucket.
Created aws_native:s3:Bucket::my-bucket with ID my-bucket-6e3d099.
Finished pulumi up.

If the Pulumi CLI process is terminated between steps 2 and 3, then Pulumi does not know if “my-bucket” exists or not. Unlike the first example, Pulumi is aware that it does not know if the resource was created. This will be displayed as a pending CREATE operation. Depending on what type of operation was interrupted, it is possible to have pending CREATE, UPDATE, DELETE, READ and IMPORT operations.

The Solution

As of #10394, the solution to both problems is pulumi refresh. The pulumi refresh command modifies Pulumi’s state so it matches the state of the underlying providers. Most discrepancies can be resolved automatically by checking Pulumi’s state against the underlying resource provider. For our AWS bucket example, Pulumi would ask the aws-native provider if bucket my-bucket-6e3d099 exists and what tags it has.

This works for all resources that Pulumi knows about. This set includes Pulumi managed resources that were changed manually, or for pending UPDATE, DELETE, READ and IMPORT operations. Pulumi does not know the resource ID for pending CREATE operations (it might not exist), so it can not resolve everything for you. It needs a little bit of help. For each pending CREATE, Pulumi will ask you what to do:

If the resource was created successfully, you can select import, give Pulumi the ID of the created resource, and Pulumi will bring that resource under management.

If no resource was created, you can select clear to tell Pulumi to clear the pending CREATE. It will recreate the resource for you the next time you run pulumi up.

You can learn more by going to the pulumi refresh documentation or by running pulumi refresh --help.

Command Details

There are some flags you can pass to modify the new behavior:

--import-pending-creates tells Pulumi to import resources that were created during pending operations. For each resource, pass its full URN as one flag invocation and its cloud provider physical ID as the next. This is equivalent to selecting the import option when running in interactive mode, but only for the specified resources.
--clear-pending-creates will remove any pending CREATEs from the state.
--skip-pending-creates will skip all pending CREATEs. This is the default behavior when running the CLI outside of interactive mode.

These flags can be combined:

pulumi refresh \
 --import-pending-creates "urn:pulumi:dev::my-project::aws-native:s3:Bucket::my-bucket" \
 --import-pending-creates "my-bucket-6e3d099" \
 --clear-pending-creates

The above command will run the pending CREATE resolution for our example S3 bucket, and will clear all other pending CREATE operations.

Unlock Programmatic Control by Disabling Default Providers

Ian Wahbe — Fri, 01 Apr 2022 00:00:00 +0000

As of 3.23.0, users can disable the default provider with Pulumi. So what does this mean for you? If you’ve been using Pulumi for a bit, you’ll have encountered provider resources, which are how we abstract the global state of a cloud provider. All resources have an associated provider. If no provider is supplied in the user’s code, a default provider is created to serve the resource. Explicit providers, which are defined by the user in code, allow programmatic and dynamic control of how a resource deploys into a cloud. A Pulumi resource can be instructed to use an explicit provider by setting the provider resource option or by inheriting the provider from the resource’s parent resource.

As an example, consider this basic S3 bucket and object for AWS, with an explicit provider that defines an AWS region:

const west = new aws.Provider("provider", {
 region: aws.Region.USWest2,
});

// Create `bucket` with explicitly set provider `west`
const bucket = new aws.s3.Bucket("my-bucket", {
 website: {
 indexDocument: "index.html",
 },
}, { provider: west });

// Inherit provider `west` from parent `bucket`
const bucketObject = new aws.s3.BucketObject("index.html", {
 acl: "public-read",
 contentType: "text/html",
 bucket: bucket,
 source: new pulumi.asset.FileAsset("index.html"),
}, { parent: bucket });

west = aws.Provider("provider", region="us-east-1")

bucket = aws.s3.Bucket(
 "my-bucket",
 website=aws.s3.BucketWebsiteArgs(index_document="index.html"),
 opts=ResourceOptions(provider=west),
)

bucketObject = aws.s3.BucketObject(
 "index.html",
 acl="public-read",
 content_type="text/html",
 bucket=bucket.id,
 source=asset.FileAsset("index.html"),
 opts=ResourceOptions(parent=bucket),
)

west, err := aws.NewProvider(ctx, "provider", &aws.ProviderArgs{
 Region: aws.RegionUSEast1,
})
if err != nil {
 return err
}

bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
 Website: s3.BucketWebsiteArgs{
 IndexDocument: pulumi.StringPtr("index.html"),
 },
}, pulumi.Provider(west))
if err != nil {
 return err
}

_, err = s3.NewBucketObject(ctx, "index.html", &s3.BucketObjectArgs{
 Acl: s3.CannedAclPublicRead,
 ContentType: pulumi.StringPtr("text/html"),
 Bucket: bucket,
 Source: pulumi.NewFileAsset("index.html"),
}, pulumi.Parent(bucket))
if err != nil {
 return err
}

var west = new Aws.Provider("provider", new Aws.ProviderArgs {
 Region = "us-east-1",
});

var bucket = new S3.Bucket("my-bucket", new S3.BucketArgs {
 Website = new Aws.S3.Inputs.BucketWebsiteArgs {
 IndexDocument = "index.html"
 },
 },
 new CustomResourceOptions { Provider = west },
);

new S3.BucketObject("index.html", new S3.BucketObjectArgs {
 Acl = "public-read",
 ContentType = "text/html",
 Bucket = bucket,
 Source = FileAsset("index.html"),
 },
 new CustomResourceOptions { Parent = bucket },
);

Default providers help keep simple things simple but make other actions impossible. As demonstrated in the above example, AWS providers, whether they are explicit or default, control the deployment region. That means multi-region deployments necessitate explicit providers. Likewise, deploying to a newly created Kubernetes cluster requires explicit providers since Kubernetes providers describe their cluster. There are many scenarios when explicit providers are mandatory for a correct deployment. When it is important that only explicitly configured providers are used, default providers can lead to unpredictable deployments.

Imagine you are trying to create a bucket in each AWS region you have access to. I might write the following code:

const filter = {
 allRegions: false,
 filters: [{
 name: "opt-in-status",
 values: ["opted-in", "opt-in-not-required"],
 }],
};

async function regionalBuckets(
 config: pulumi.Config,
): Promise<Array<aws.s3.Bucket>> {
 const bucketList = new Array<aws.s3.Bucket>();
 for (
 const region in (await aws.getRegions(filter)).names
 ) {
 const provider = new aws.Provider(`${region}-provider`, {
 region: region as aws.Region,
 accessKey: config.requireSecret("key"),
 secretKey: config.requireSecret("secret"),
 });
 const bucket = new aws.s3.Bucket(`${region}-bucket`, {}, { provider });
 bucketList.push(bucket);
 }
 return bucketList;
}

filters = [
 aws.GetRegionsFilterArgs(
 name="opt-in-status", values=["opted-in", "opt-in-not-required"]
 )
]


async def regional_buckets(config: pulumi.Config) -> Output[List[s3.Bucket]]:
 bucket_list = []
 for region in (await aws.get_regions(filters=filters)).names:
 provider = aws.Provider(
 f"{region}-provider",
 region=region,
 access_key=config.require_secret("key"),
 secret_key=config.require_secret("secret"),
 )
 bucket = s3.Bucket(f"{region}-bucket", opts=ResourceOptions(provider=provider))
 bucket_list.append(bucket)
 return bucket_list

var filters = []aws.GetRegionsFilter{{
 Name: "opt-in-status",
 Values: []string{"opted-in", "opt-in-not-required"},
}}

func regionalBuckets(ctx *pulumi.Context, config config.Config) (s3.BucketArray, error) {
 var bucketList s3.BucketArray
 regions, err := aws.GetRegions(ctx, &aws.GetRegionsArgs{Filters: filters})
 if err != nil {
 return nil, err
 }
 for _, region := range regions.Names {
 provider, err := aws.NewProvider(ctx, region+"-provider", &aws.ProviderArgs{
 Region: pulumi.String(region),
 AccessKey: config.RequireSecret("key"),
 SecretKey: config.RequireSecret("secret"),
 })
 if err != nil {
 return nil, err
 }
 bucket, err := s3.NewBucket(ctx, region+"-bucket", nil, pulumi.Provider(provider))
 if err != nil {
 return nil, err
 }
 bucketList = append(bucketList, bucket)
 }
 return bucketList, nil
}

async Task<S3.Bucket[]> regionalBuckets(Pulumi.Config config) {
 var filter = new Aws.Inputs.GetRegionsFilterArgs{
 Name = "opt-in-status",
 Values = {"opted-in", "opt-in-not-required"},
 };
 var regions = await Aws.GetRegions.InvokeAsync(new Aws.GetRegionsArgs{Filters = {filter}});
 var bucketList = new List<S3.Bucket>();
 foreach (var region in regions.Names) {
 var provider = new Aws.Provider(region+"-provider", new Aws.ProviderArgs {
 Region = region,
 AccessKey = config.GetSecret("key")!,
 SecretKey = config.GetSecret("secret")!,
 });
 var bucket = new S3.Bucket(region+"-bucket", null, new CustomResourceOptions {
 Provider = provider
 });
 bucketList.Add(bucket);
 }
 return bucketList.ToArray();
}

This code will create a bucket in each region that the currently logged-in AWS account has access to. If someone else ran the same code on a computer logged in to another account, Pulumi could create different buckets, even though all buckets were provisioned by an explicit provider. This behavior is because we forgot to specify the provider for the call to aws.getRegions. You might not notice that something unexpected happened because there would be no error message, just inconsistent deployments. To prevent this category of problem, Pulumi now offers the ability to disable default providers on a per-stack basis.

By disabling the default aws provider, we would get the following error:

Error: Invoke: Default provider for 'aws' disabled. 'aws:index/getRegions:getRegions' must use an explicit provider.

Disabling default providers has been a popular community request. To meet that request, we’ve added the ability to disable default providers in the 3.23.0 release. The Pulumi config variable pulumi:disable-default-providers represents the list of packages whose default providers are disabled. Attempting to invoke a disabled default provider will fail, raising an error as in the previous example.

How to disable default providers

Disabling a default provider involves adding the relevant package name to the config list pulumi:disable-default-providers. For example, disabling the default provider for Kubernetes and AWS would look like this snippet in the configuration file:

pulumi:disable-default-providers:
- aws
- kubernetes

If you want to enforce that no resource should use a default provider, you would add the following snippet to the configuration file:

pulumi:disable-default-providers: ["*"]

To add the disable-default-providers configuration setting to your stack configuration file using the Pulumi CLI or the Automation API SDK, please see the Resource providers page.

Now that you can disable the default provider, you don’t have to worry about all of the possible unexpected consequences accidentally relying on your system configuration. We can’t wait to find out what you’ll build next! If you want to have an impact on our roadmap, you can go to our public roadmap and vote on issues with the rest of the community. Let us know what you think!

Pulumi Blog: Ian Wahbe

Compatibility Testing Pulumi HCL

How we compatibility test Pulumi HCL

An example tfcompat test

The anatomy of tfcompat.RunCase

Writing tests with LLMs

Conclusion

Emulating Terraform on Pulumi's Engine

Providers

Resources

Property name translation

Provisioners and conditions

Resource options

Modules

Conclusion

Introducing: Support For Using Any Terraform Provider with Pulumi

Walkthrough

Why Support All Terraform Providers?

How Does This Work?

What’s Next?

Repairing State With Pulumi Refresh

The Problem

Out of Band Changes

Terminating Pulumi Mid-Operation

The Solution

Command Details

Unlock Programmatic Control by Disabling Default Providers

How to disable default providers

An example `tfcompat` test

The anatomy of `tfcompat.RunCase`