Pulumi Blog: Zaid Ajaj

Pulumi Google Cloud Provider Version 9.0.0

Guinevere Saenger — Wed, 24 Sep 2025 13:33:50 -0700

We’re excited to announce the v9 release of the Pulumi Google Cloud Provider! This major release contains important updates to Google Cloud resources and functions, and keeps you up to date with what’s new from Pulumi.

The Pulumi Google Cloud provider can be used to provision any of the Google Cloud resources available in the upstream provider. The provider is open source and available on GitHub so you can be part of the community - issues and pull requests are always welcome!

Here are a few links to help you get started if you are new to Pulumi:

Getting Started - A guided walkthrough for creating your first project
Setup & Install - Instructions on installing the Google Cloud provider
How-to guides - Learn how to use the Google Cloud provider to provision specific resources
Templates - Use a quickstart template to create a new project
Pulumi Neo - Ask Pulumi Neo to help you with your project

In case you missed it - new modules

Throughout the past year, we have worked to keep this provider in sync with the upstream provider. Notable new modules are those supporting AI and Google Gemini. New modules added since the last major release include:

What’s New in 9.0

Upstream Improvements

This release includes the latest improvements and changes from the recent major version in the upstream provider. Some themes are:

Improved Import Validation

Multiple resources throughout the provider have improved validation and error messages to ensure the supplied GCP resource ID matches import formats specified in the documentation.

Increased Field Validation

Several resource fields have been updated to reflect requirements of Google Cloud and will fail validation if not configured correctly, to avoid failures later on. Refer to the migration guide for details.

Documentation Improvements

We are always working on improving and updating our documentation. Among other improvements, this release adds 100 new top-level overview descriptions for resources and functions.

Upgrading

You can find the migration guide on the Pulumi registry.

NEW! Use Pulumi Neo to assist with your upgrade

You can use our new automation agent, Neo, to help you with your provider upgrade. In the Pulumi service, ask Neo to help you with your upgrade by providing the migration guide:

Neo will look at the provided migration guide and your stack, and determine a plan of action. Here, Neo has discovered there were some breaking changes, needing updates to your program.

And finally, Neo can open a pull request with the proposed changes to your code.

Converting Bicep code to Pulumi

Zaid Ajaj — Fri, 29 Sep 2023 00:00:00 +0000

Bicep is a DSL developed by Microsoft to simplify the authoring of ARM templates and deploy resources to Azure. Today I will be sharing with you a new Pulumi converter plugin that I have been working on that converts Bicep code to any of the supported Pulumi languages.

The Bicep language is a domain-specific language for Infrastructure-As-Code built by Microsoft that uses declarative syntax to deploy Azure resources. It was made to address the shortcomings of authoring ARM templates that are written in JSON. Compared to ARM templates, Bicep is much more concise and readable. However, here at Pulumi we believe that using general-purpose programming languages that are familiar to developers is the best way to author Infrastructure-As-Code. Besides being able to use a general-purpose language, you and your team can benefit from using Pulumi Cloud as you scale to larger projects and teams with features such as Pulumi Deployments, Organization Access Tokens, Review Stacks and more.

That is why I have been working on a new Pulumi converter plugin that converts Bicep code to any of the supported Pulumi languages using the Pulumi CLI via pulumi convert.

Installing and using the plugin

First, install the converter plugin using the Pulumi CLI:

pulumi plugin install converter bicep

This command will install the plugin from the latest GitHub release of the repository.

Once installed, you can use it to convert Bicep code to Pulumi. For example, let’s say we have the following Bicep code in a file called example.bicep which deploys a storage account:

resource storage 'Microsoft.Storage/storageAccounts@2022-09-01' = {
 name: 'storageaccount'
 location: resourceGroup().location
 kind: 'StorageV2'
 sku: {
 name: 'Standard_LRS'
 }
}

Navigate to the directory where the file is located and run the following command. For example to generate TypeScript code from the Bicep code above, run:

pulumi convert --from bicep --language typescript --out output -- --entry example.bicep

This command will use the Bicep converter plugin we just installed to convert the code in example.bicep to TypeScript and place the generated code in the output directory.

You can change typescript to any of the supported languages such as csharp, python, go, java or yaml and you are good to go.

If you don’t specify the --entry flag, the plugin will look for the first file with the .bicep extension in the current directory.

Let us look at the generated code from the example above:

import * as pulumi from "@pulumi/pulumi";
import * as azure_native from "@pulumi/azure-native";

const config = new pulumi.Config();
// The name of the resource group to operate on
const resourceGroupName = config.require("resourceGroupName");
const currentResourceGroup = azure_native.resources.getResourceGroupOutput({
 resourceGroupName: resourceGroupName,
});
const storage = new azure_native.storage.StorageAccount("storageaccount", {
 accountName: "storageaccount",
 kind: "StorageV2",
 location: currentResourceGroup.apply(currentResourceGroup => currentResourceGroup.location),
 resourceGroupName: currentResourceGroup.apply(currentResourceGroup => currentResourceGroup.name),
 sku: {
 name: "Standard_LRS",
 },
});

import pulumi
import pulumi_azure_native as azure_native

config = pulumi.Config()
# The name of the resource group to operate on
resource_group_name = config.require("resourceGroupName")
current_resource_group = azure_native.resources.get_resource_group_output(resource_group_name=resource_group_name)
storage = azure_native.storage.StorageAccount("storageaccount",
 account_name="storageaccount",
 kind="StorageV2",
 location=current_resource_group.location,
 resource_group_name=current_resource_group.name,
 sku=azure_native.storage.SkuArgs(
 name="Standard_LRS",
 ))

package main

import (
 "github.com/pulumi/pulumi-azure-native-sdk/v2/resources"
 "github.com/pulumi/pulumi-azure-native-sdk/v2/storage"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 cfg := config.New(ctx, "")
 // The name of the resource group to operate on
 resourceGroupName := cfg.Require("resourceGroupName")
 currentResourceGroup, err := resources.LookupResourceGroup(ctx, &resources.LookupResourceGroupArgs{
 ResourceGroupName: resourceGroupName,
 }, nil)
 if err != nil {
 return err
 }
 _, err = storage.NewStorageAccount(ctx, "storageaccount", &storage.StorageAccountArgs{
 AccountName: pulumi.String("storageaccount"),
 Kind: pulumi.String("StorageV2"),
 Location: *pulumi.String(currentResourceGroup.Location),
 ResourceGroupName: *pulumi.String(currentResourceGroup.Name),
 Sku: &storage.SkuArgs{
 Name: pulumi.String("Standard_LRS"),
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AzureNative = Pulumi.AzureNative;

return await Deployment.RunAsync(() =>
{
 var config = new Config();
 // The name of the resource group to operate on
 var resourceGroupName = config.Require("resourceGroupName");
 var currentResourceGroup = AzureNative.Resources.GetResourceGroup.Invoke(new()
 {
 ResourceGroupName = resourceGroupName,
 });

 var storage = new AzureNative.Storage.StorageAccount("storageaccount", new()
 {
 AccountName = "storageaccount",
 Kind = "StorageV2",
 Location = currentResourceGroup.Apply(getResourceGroupResult => getResourceGroupResult.Location),
 ResourceGroupName = currentResourceGroup.Apply(getResourceGroupResult => getResourceGroupResult.Name),
 Sku = new AzureNative.Storage.Inputs.SkuArgs
 {
 Name = "Standard_LRS",
 },
 });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.azurenative.resources.ResourcesFunctions;
import com.pulumi.azurenative.resources.inputs.GetResourceGroupArgs;
import com.pulumi.azurenative.storage.StorageAccount;
import com.pulumi.azurenative.storage.StorageAccountArgs;
import com.pulumi.azurenative.storage.inputs.SkuArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 final var config = ctx.config();
 final var resourceGroupName = config.get("resourceGroupName");
 final var currentResourceGroup = ResourcesFunctions.getResourceGroup(GetResourceGroupArgs.builder()
 .resourceGroupName(resourceGroupName)
 .build());

 var storage = new StorageAccount("storage", StorageAccountArgs.builder()
 .accountName("storageaccount")
 .kind("StorageV2")
 .location(currentResourceGroup.applyValue(getResourceGroupResult -> getResourceGroupResult.location()))
 .resourceGroupName(currentResourceGroup.applyValue(getResourceGroupResult -> getResourceGroupResult.name()))
 .sku(SkuArgs.builder()
 .name("Standard_LRS")
 .build())
 .build());

 }
}

configuration:
 resourceGroupName:
 type: string
resources:
 storageaccount:
 type: azure-native:storage:StorageAccount
 properties:
 accountName: storageaccount
 kind: StorageV2
 location: ${currentResourceGroup.location}
 resourceGroupName: ${currentResourceGroup.name}
 sku:
 name: Standard_LRS
variables:
 currentResourceGroup:
 fn::invoke:
 Function: azure-native:resources:getResourceGroup
 Arguments:
 resourceGroupName: ${resourceGroupName}

Notice how the in the original Bicep code, we reference the location of the implicit resource group via resourceGroup().location. When the converter generates code for the target language, it replaces this implicit resource group with an explicit one by parameterizing the program by a resource group name and using that resource group when deploying the resources.

Early days for the converter

Currently the converter supports most of the features of Bicep such as parameters, variables, modules, resources, outputs and a number of built-in functions. That said, the converter plugin is still in its early days and it’s not yet tested against many real-world Bicep programs. I am actively working on improving it and making it support most of the Bicep language features. The source code is available on this GitHub repository. If you find any issues or have any feedback, please open an issue on the repository.

How does it work?

The converter plugin is a native binary built using .NET in F# and new Pulumi Converter SDK. Using the core Pulumi nuget package, we shipped an experimental API that allows you to easily build converter plugins that can extend the functionality of pulumi convert.

Building a language converter plugin involves taking the source language, in this case Bicep, and converting it to our internal Pulumi Configuration Language, or PCL for short. The converter implements a function of the following shape:

let convertProgram (request: ConvertProgramRequest): ConvertProgramResponse =
 let sourceDirectory = request.SourceDirectory
 let targetDirectory = request.TargetDirectory
 let args = request.Args
 // conversion logic here

In our case, we read the Bicep code from the source directory, convert it to PCL and write out the PCL code in the target directory. Then, Pulumi uses the built-in program generation facilities to take care of the rest of the work of generating the target language code from the PCL code so that we don’t have to worry about language-specific details. The converter only needs to know how to generate PCL.

To build the actual transformation from Bicep to PCL, I made use the of the Azure.Bicep.Core package available for .NET which allowed me to parse the Bicep code and generate a typed Abstract Syntax Tree (AST) from it. Building an AST from the source language allows us to easily traverse the tree, analyze it and symbolically rewrite pieces of it. Working at the AST level also makes it easy to test source code transformation using structure rather than text. Once we have obtained the Bicep AST, we transform it into a Pulumi AST that represents a PCL program. Finally, we print out the Pulumi AST to a string and write it to the target directory.

If you are curious about the specification of the AST that represents Bicep programs, head over to this file from the source code. You will find types such as BicepProgram and BicepSyntax that model almost every aspect of Bicep code. As for the AST of PCL programs, you can find the type definitions here.

The F# language is great for almost everything, but is especially amazing for writing language converters because of its powerful pattern-matching capabilities and the ability to write code in a functional style. Many of us working at Pulumi are big fans of F#!

Here is a snippet of the transformation pipeline written in F# that takes the input Bicep program and converts it to the final PCL text

bicepProgram
|> BicepProgram.dropResourceUnknowns
|> BicepProgram.reduceScopeParameter
|> BicepProgram.parameterizeByTenantAndSubscriptionId
|> BicepProgram.addResourceGroupNameParameterToModules
|> BicepProgram.parameterizeByResourceGroup
|> Transform.bicepProgramToPulumi
|> Transform.modifyComponentPaths rewriteComponentPath
|> Printer.printProgram

You can learn more on the implementation of these functions from the source code on GitHub. I’ve written it in a way that is easy to follow, understand and contribute to. Most of these functions are fully unit-tested and I am working on adding more tests to cover more of the Bicep language features.

Bonus converter: ARM to Pulumi

Those who are familiar with the Bicep CLI, know that it has a built-in decompiler that converts ARM templates to Bicep. I thought it would be fun to build another converter that takes ARM templates to Pulumi but without writing any conversion logic for it. Instead, I would decompile the ARM template to Bicep, then use the logic from the Bicep converter to generate the final PCL code. Fortunately, the decompiler code is embedded in the nuget package Azure.Bicep.Core so we don’t have to rely on the existence of the Bicep CLI in order to make this converter work.

From all of this, I created another converter pulumi-converter-arm that works exactly like the Bicep converter but takes ARM templates as input. Head over to the repository to learn more about how to install it and how to use it.

Depending on how well this ARM converter does its job, we might consider deprecating arm2pulumi in favor of it. However, we first need to test it against more templates and iron out the rough edges. It would be the ideal situation because there are virtually zero maintenance costs, any improvements to the Bicep converter would automatically get picked up here. Also, improvements to the decompiler from the Azure.Bicep.Core nuget package would benefit the converter as well.

Building your own Pulumi language converter

Converter plugins are a great way to extend the Pulumi CLI with pulumi convert to support new languages. When we shipped support for Converting full Terraform Programs To Pulumi, we extended the pulumi convert command to allow installing and using converter plugins that are shipped independently from the Pulumi CLI.

In essence, those plugins are executable binaries that can be written in any language, which serves up a gRPC server that implements the Converter contract:

// Converter is a service for converting between other ecosystems and Pulumi.
// This is currently unstable and experimental.
service Converter {
 // ConvertState converts state from the target ecosystem into a form that can be imported into Pulumi.
 rpc ConvertState(ConvertStateRequest) returns (ConvertStateResponse) {}

 // ConvertProgram converts a program from the target ecosystem into a form that can be used with Pulumi.
 rpc ConvertProgram(ConvertProgramRequest) returns (ConvertProgramResponse) {}
}

This Converter interface is considered experimental and is subject to change in the future.

Now, learning how to setup gRPC servers and implement the Converter contract is not an easy task unless you are familiar with it. For .NET, we shipped the experimental Pulumi converter SDK to the main Pulumi nuget package that makes it extremely easy to build converter plugins.

A bare-bones converter plugin that does nothing looks like this in F#:

open Pulumi.Experimental.Converter
open Pulumi.Codegen

let convertProgram (request: ConvertProgramRequest): ConvertProgramResponse =
 // TODO: Implement the conversion logic here
 ConvertProgramResponse.Empty

convertProgram
|> Converter.CreateSimple
|> Converter.Serve
|> Async.AwaitTask
|> Async.RunSynchronously

This will spin up a gRPC server that implements the Converter contract and will serve it on a random port assigned during startup. The Pulumi converter SDK automatically implements gRPC reflection which allows tools such as Postman to discover the available gRPC services and methods and send example requests to them. See Postman with gRPC.

Publishing converter plugins

The easiest way to make your converter plugin available to others is to publish it on GitHub releases. The Pulumi CLI expects a naming convention pulumi-converter-<name> and will automatically download the plugin from the latest GitHub release of the repository when given the --server flag during pulumi plugin install converter <name> --server <url>.

Converter plugins written in .NET are simple console applications that are compiled to native binaries for each target platform. To see how this is done, refer to this createAndPublishArtifacts function written for the Bicep converter that does the heavy lifting of building the native binaries for each platform, putting them in an archive then publishing them to GitHub releases.

Note: when publishing native binaries for macOS, the Github Action executing the publishing process must be using a macOS runner so that the generated binary from dotnet publish gets signed with a developer certificate. This is required for the binary to be executable on macOS.

Final thoughts

I hope you enjoyed this article and learned something new about the Pulumi converter plugins and Pulumi internals. I am very excited about the Bicep converter plugin and I am looking forward to hearing your feedback on it from the community. It was a lot of fun writing this in F# and made writing complex pieces of code very trivial.

If you have any questions or feedback, don’t hesitate to reach out!

Using Go Generics with Pulumi

Zaid Ajaj — Wed, 20 Sep 2023 00:00:00 +0000

Pulumi loves Go, it’s what powers Pulumi. We’ve kept a close eye on the design and development of support for generics in the Go programming language over the years, a feature that allows developers to write type-safe, concise, and reusable code. We’ve been exploring what it’d look like to improve Pulumi’s Go SDKs with generics and recently published a public RFC detailing our plans. We’ve been making progress on the implementation and are excited to announce preview support for Go generics in our core and AWS Go SDKs. If you’re using Go with Pulumi, we’d love for you to give it a try and share your feedback!

// Given
var a pulumi.IntOutput
var b pulumi.StringOutput

// Before (could panic at runtime if you got something wrong)
o := pulumi.All(a, b).ApplyT(func(vs []interface{}) string { // could panic
 a := vs[0].(int) // could panic
 b := vs[1].(string) // could panic
 return strconv.Itoa(a) + b
}).(pulumi.StringOutput) // could panic

// After (compile-time type-safety)
o := pulumix.Apply2(a, b, func(a int, b string) string {
 return strconv.Itoa(a) + b
})

Why Generics?

Before looking at what’s available in the preview, let’s first review the motivations for adding support for generics, from the RFC.

Unsafe Inputs, Outputs, and Apply

A common complaint about the Pulumi Go SDK is the lack of type safety. The Input and Output types, and the apply operation rely heavily on interface{}. This makes them effectively untyped at compile-time, instead relying on type matching at runtime.

For instance, the apply operation is implemented in Go as the following method:

func (*OutputState) ApplyT(applier interface{}) Output

The contract for ApplyT states that applier must be a function with one of the following signatures:

func(U) T
func(U) (T, error)

Where U must be assignable from the type contained in the underlying Output. If it isn’t, the operation will panic.

The caller will usually want to cast the result into a specific output type. If they choose the wrong output type, this operation will panic.

So given a typical usage like the following, there are two spots that can panic: the ApplyT invocation and the IntOutput cast.

intOutput := stringOutput.ApplyT(func(v string) int {
 return len(v)
}).(pulumi.IntOutput)

Excessive code generation

The Go SDK relies heavily on code generation in an attempt to alleviate the pain of untyped operations. It generates type-specific input and output wrapper types for nearly every type it encounters.

In the core Pulumi SDK, this includes all primitive types, their pointers, containers, and containers of containers. For instance, the core SDK has 22 String* types besides type String itself, covering nearly all variations of the following pattern:

String(Ptr|(Array|Map)?(Array|Map))?(Input|Output)?

This pattern of code generation carries over to code generated for provider SDKs, though not to this degree of nesting. Every type gets an input and output type, a pointer variant if the type is referenced as a pointer, and container variants if the type is referenced from a container.

In general, this generates extremely large packages that are ungainly to navigate—both in source form and in their API reference.

Introducing Go Generics in the Pulumi Go SDK

We can address the above by leveraging Go generics to incorporate type-safety into the core concepts (inputs, outputs, and apply) and reduce the amount of generated code.

Starting with v3.80 of the Pulumi Go SDK, we have added a new pulumix subpackage (containing “extensions”), which includes new generic Input[T] and Output[T] types as well as type-safe generics-based APIs to work with them such as Apply.

With the new support for generics, the previous untyped example can now be written in a type-safe manner as follows:

intOutput := pulumix.Apply(stringOutput, func(v string) int {
 return len(v)
})

This is checked at compile-time: if you mistakenly specified an incorrect type, it will be caught at compile-time rather than panicking at runtime.

We recommend using Go 1.21 or later, which has improved type inference for generics, allowing you to write the call to Apply above without having to explicitly specify generic type parameters.

Adopting Generics in Your Pulumi Programs

Here’s a larger example that uses the non-generic pulumi.All function and ApplyT method to produce a JSON string representing a policy for a BucketPolicy:

package main

import (
 "encoding/json"

 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/cloudfront"
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 ErrorDocument: pulumi.String("404.html"),
 },
 })
 if err != nil {
 return err
 }

 originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
 Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
 })
 if err != nil {
 return err
 }

 _, err = s3.NewBucketPolicy(ctx, "cloudfront-bucket-policy", &s3.BucketPolicyArgs{
 Bucket: bucket.ID(),
 Policy: pulumi.All(bucket.Arn, originAccessIdentity.IamArn).ApplyT(
 func(args []any) (string, error) {
 bucketArn := args[0].(string)
 iamArn := args[1].(string)

 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }).(pulumi.StringOutput),
 }, pulumi.Parent(bucket))
 if err != nil {
 return err
 }

 return nil
 })
}

pulumi.All is used to combine bucket.Arn and originAccessIdentity.IamArn into a single Output and then uses ApplyT construct a policy in JSON using the values.

If we wanted to take advantage of the type-safety provided by the new pulumix subpackage, we could rewrite the above example to use pulumix.Apply. First, import "github.com/pulumi/pulumi/sdk/v3/go/pulumix". Then, replace the use of pulumi.All and ApplyT with the generic pulumix.Apply:

 Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }),

Here, we’re specifically using pulumix.Apply2Err, a variant of pulumix.Apply that accepts two generic arguments and a callback function that supports returning an error. There are other variants of pulumix.Apply that accept up to 8 generic arguments, and variants that return an error and accept a context.Context.

The result of our call to pulumix.Apply2Err in this example is pulumix.Output[string]. We can pass this directly to the Policy argument because it happens to be weakly typed as pulumi.Input, which pulumix.Output[T] implements.

If the Policy argument had been typed as pulumi.StringInput, for example, we would have had to cast the pulumix.Output[string] to pulumi.StringOutput using the pulumix.Cast function:

policy := pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 // ...
 })

// ...

Policy: pulumix.Cast[pulumi.StringOutput](policy),

Here’s the updated example in full:

package main

import (
 "encoding/json"

 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/cloudfront"
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 ErrorDocument: pulumi.String("404.html"),
 },
 })
 if err != nil {
 return err
 }

 originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
 Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
 })
 if err != nil {
 return err
 }

 _, err = s3.NewBucketPolicy(ctx, "bucket-policy", &s3.BucketPolicyArgs{
 Bucket: bucket.ID(),
 Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }),
 }, pulumi.Parent(bucket))
 if err != nil {
 return err
 }

 return nil
 })
}

Notice how we are importing pulumix alongside pulumi. Being a subpackage means you can start using the generics-based types and functions alongside the current APIs as necessary without having to switch your code fully from using the old types to the new generic ones. Instead, you can migrate your code gradually to these new APIs where it makes sense. When needed, there are APIs to convert between generics-based APIs and the old ones using functions such as pulumix.Cast and pulumix.ConvertTyped.

Generics-Based Cloud Provider SDKs

We’ve seen how the pulumix subpackage on its own can be used with existing Pulumi provider SDKs. We can go further and adopt generics in the provider SDKs themselves. To make the transition non-breaking and gradual for your Pulumi programs, we intend to publish generic-based variants of the provider Go SDKs under a subpackage called x (for “extensions”). This allows you to migrate your code resource by resource.

Migrating an AWS Program to leverage Go generics

The following example is an AWS Pulumi program which we’ll migrate into the generics-based API. Here is what pulumi new aws-go gives you today:

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an AWS resource (S3 Bucket)
 bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 },
 })
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("bucketName", bucket.ID())
 return nil
 })
}

To start using the generics-based version of the S3 module from AWS, you need to change the import from "https://github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3" to "https://github.com/pulumi/pulumi-aws/sdk/v7/go/aws/x/s3" (notice the x after aws/), then change input values to use pulumix.Ptr instead of pulumi.String.

Rather than passing values as pulumi.String("foo") and pulumi.Int(42), with generics, you can simply use pulumix.Val("foo") and pulumix.Val(42). Similarly, you can use the pulumix.Ptr function for pointer values.

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/x/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an AWS resource (S3 Bucket)
 bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumix.Ptr("index.html"),
 },
 })
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("bucketName", bucket.ID())
 return nil
 })
}

You’ll also need to update your go.mod to use the preview branch of the Pulumi AWS Go SDK that includes the x subpackage with generic APIs. Run the following:

go get github.com/pulumi/pulumi-aws/sdk/v7@generics
go mod tidy

The changes look minimal, except the fact that the inputs of s3.BucketArgs and the outputs of Bucket are now fully generics-based. Here is the Bucket type you get from NewBucket:

type Bucket struct {
 pulumi.CustomResourceState
 AccelerationStatus pulumix.Output[string] `pulumi:"accelerationStatus"`
 Acl pulumix.Output[*string] `pulumi:"acl"`
 Arn pulumix.Output[string] `pulumi:"arn"`
 Bucket pulumix.Output[string] `pulumi:"bucket"`
 // ....
}

And here are its input arguments:

type BucketArgs struct {
 AccelerationStatus pulumix.Input[*string]
 Acl pulumix.Input[*string]

 Arn pulumix.Input[*string]
 Bucket pulumix.Input[*string]
 // ...
}

Authoring Generics-based Go SDKs

For Pulumi package authors, it is possible to start generating Go SDKs for your providers that include the generics-based variant under the x subpackage using v3.84.0 of Pulumi. By default, this is not enabled so you will have to explicitly opt-in to it using a specific option in the go language section of your Pulumi schema. The new option is called "generics" and it has three possible values:

"none" is the default value when the option is not set. This doesn’t generate the generics-based variant
"side-by-side" is the option we are using for our select providers which generates the variant under the x subdirectory.
"generics-only" if you want your provider to only emit generics-based APIs. Since this is a preview feature, things might still be rough around the edges, so use this option with caution as fixes in the SDK-generation code might result in breaking changes in your APIs.

Here is how this looks in JSON:

{
 "language": {
 "go": {
 "generics": "side-by-side"
 }
 }
}

We eventually intend to make "side-by-side" be the default, such that all providers get a generics-based variant under the x subpackage, so that all users can use the full generics API.

Try it out!

We are excited for you to try out these new APIs and would love to hear your feedback! You can learn more details on the design and add comments to the RFC, or chat with us in the #golang channel on the Pulumi Community Slack.

If you’d like to learn more about building cloud applications with Go and Pulumi, we have a workshop coming up in October. Sign up here.

We’ll also be at GopherCon, if you’d like to meet with us.

Simplify Configuration Management with Project-level Config

Fraser Waters — Wed, 02 Nov 2022 06:00:00 -0700

This blog post discusses project-level config before Pulumi ESC was released. ESC makes this and hierarchical config easy.

One of our most up-voted feature requests (with 78 thumbs ups) is to support hierarchical config. We’re happy to announce that we’ve now released the first part of plans to support this feature.

Pulumi will now allow you to set configuration values in your Pulumi.yaml file, using the given value as a default for all stacks in the project. While we expect even this first level of support will be incredibly useful to many people we also want to assure you that we have many more plans in place to make this feature even better.

How to Use Project-level Configuration

Pulumi has always had the concept of configuration per stack, and the changes for project level configuration haven’t fundamentally changed that. You can still set config for stacks via the pulumi config CLI command, or by editing the Pulumi.<stack>.yaml file. Project level config just gives you the ability to set default values for all stacks in your Pulumi.yaml.

A common request we get for this is making sure that all stacks use the same AWS region, or don’t have default providers enabled. You can now do this by setting these in your Pulumi.yaml.

name: project
runtime: go
config:
 pulumi:disable-default-providers: [“*”]
 aws:region: us-east-1

As you can see this allows setting default configuration values per-stack for configuring the engine or providers, but you can also set default values for project configuration. For example maybe you have a default count of the number of virtual machines to create:

name: project
runtime: go
config:
 vm-count: 2

The only caveat to mention about this system is that if you have object values they must be nested under a key value. So if you want to set the key obj to the object value { foo: "bar" } you would need to write it as such:

name: project
runtime: go
config:
 obj:
 value:
 foo: bar

This allows the engine to distinguish between values and schemas, which we will show below.

Defining Configuration Schemas in Pulumi

A related feature to defining values for config at the project level is being able to define a schema for the config. This allows you to define what configuration settings you expect in your Pulumi.yaml so that the engine can use that information to validate configuration for every stack.

This is done using a very similar declaration to JSON schema, or our Pulumi JSON schema (the schemas from resource providers that drive our code generators). Below is a small example of some of the things you can declare:

config:
 vm-count:
 type: integer
 description: the number of virtual machines to create
 default: 2
 subnets:
 type: array
 description: an array of subnets to create
 items:
 type: string
 vaultKey:
 type: string
 description: which vault key to use for this stack
 secret: true

These are declared in the same config block as the other project level config shown through this post. You can combine the two by declaring your project settings and defining defaults for providers together:

config:
 aws:region: us-east-2
 createVpc:
 type: boolean
 default: false

Pulumi YAML Configuration: What’s Changing?

When we released our support for simple Pulumi programs in YAML we added some support for configuration with it. This was done via the configuration key in the Pulumi.yaml file. We’ll be deprecating support for that key now, as YAML will instead use the new standard project configuration.

What’s Next for Pulumi Configuration?

This is just the start of our support for better config. We posted a long comment to GitHub earlier this year to explain some of our future plans, and we do intend to continue improving this part of the Pulumi system as described in that comment.

FAQ: Project-Level Configuration in Pulumi

What is project-level config in Pulumi?

Project-level config allows you to set default configuration values for all stacks in a Pulumi.yaml file.

Can I override project-level config for specific stacks?

Yes! You can override default values in Pulumi.<stack>.yaml or via the CLI.

How does Pulumi handle object values in config?

Objects must be nested under a value key to distinguish them from schemas.

Is the configuration key still supported in YAML?

No, it is now deprecated in favor of project-level configuration.

Where can I give feedback or ask questions?

Join the Pulumi Community Slack or join the GitHub Discussions.

Improved Pulumi Experience with .NET 6

Zaid Ajaj — Fri, 22 Jul 2022 00:00:00 +0000

In this blog post, we will talk about how Pulumi is now using .NET 6, the latest Long-Term Support version of .NET, as our default across the ecosystem. We will discuss the changes applied to templates, program structure and code generation. We also explain how Pulumi C# projects can benefit from the latest features in .NET 6 and how it simplifies your programs overall. Let’s dive in, shall we?

Templates

We updated all of our .NET templates to use net6.0 as the target framework. This means that projects created with pulumi new require at least the .NET 6 SDK to build and run the projects. The templates also simplified the program structure quite a bit and not longer use MyStack classes to define resources in case of C# and VB.NET templates. F# templates didn’t change that much because these already didn’t use the class approach to define infrastructure.

Simplified program structure

Let us examine a new C# template. Create a new project using pulumi new aws-csharp in a new directory and notice that there is no longer a MyStack.cs file, only Program.cs. Open the latter in an editor and you should see the following:

using Pulumi;
using Pulumi.Aws.S3;
using System.Collections.Generic;

return await Deployment.RunAsync(() =>
{
 // Create an AWS resource (S3 Bucket)
 var bucket = new Bucket("my-bucket");

 // Export the name of the bucket
 return new Dictionary<string, object?>
 {
 ["bucketName"] = bucket.Id
 };
});

This is how a stack with resource definitions and outputs now looks. The Get Started guide has been updated to follow this pattern as well.

Targeting .NET 6 means that we can use features from C# 10 and here we see them in action: using top-level statements that greatly simplify the entry point of the project and reduce it down to a single function call to Deployment.RunAsync. Inside this function is where we define our resources and return outputs for the stack. Top-level statements also allow us to await asynchronous functions without having an explicit Main(string[] args) function or a Program class.

Notice how you no longer have to define the types of your stack outputs upfront. Instead, you simply return a dictionary of key-valued outputs. In fact, the function Deployment.RunAsync is very flexible. When you don’t want to return outputs, don’t!

return await Deployment.RunAsync(() =>
{
 var bucket = new Bucket("my-bucket");
});

Moreover, when you want to await asynchronous functions alongside your resource definitions, just mark the lambda as async. Here is an example where we await function invokes and work with their results.

using Pulumi;
using Pulumi.Aws;
using Pulumi.Aws.Ec2;
using System.Linq;

return await Deployment.RunAsync(async() =>
{
 var vpc = await GetVpc.InvokeAsync(new()
 {
 Default = true
 });

 var zones = await GetAvailabilityZones.InvokeAsync();

 foreach (var range in zones.Names.Select((zone, index) => new { Value = zone, Key = index }))
 {
 var subent = new Subnet($"vpcSubnet-{range.Key}", new()
 {
 AssignIpv6AddressOnCreation = false,
 VpcId = vpc.Id,
 MapPublicIpOnLaunch = true,
 CidrBlock = $"10.100.{range.Key}.0/24",
 AvailabilityZone = range.Value,
 Tags =
 {
 ["Name"] = $"pulumi-sn-{range.Value}",
 },
 });
 }
});

The cherry on top is that you can use target-typed new expressions when instantiating resource or function invoke argument types. Basically instead of writing new Bucket("my-bucket", new BucketArgs { ... }), you can write new Bucket("my-bucket", new () { ... }).

Backward-compatibility

It is worth mentioning that although the new templates and the accompanying getting-started guides are updating to .NET 6, the main Pulumi SDK NuGet package and cloud provider packages are still targeting netcoreapp3.1. This mean that if users are still using .NET SDK v3.1 or v5.0 to build and run your projects, everything will continue to work as always.

Code generation

Pulumi provides tools to translate from several IaC languages. These tools will generate Pulumi programs in any of our supported languages. This includes:

Translating Pulumi YAML with the Pulumi CLI using pulumi convert
Translating Azure Resource Manager (ARM) templates using arm2pulumi
Translating Terraform code with the Pulumi CLI using pulumi convert --from terraform
Translating Kubernetes YAML using kube2pulumi

These tools take their source language as input, convert it into the Pulumi intermediate language, then translate it into one of our supported languages. In the case of C#, the code generation has been updated to produce code that uses the latest features from .NET 6. We’ve also fixed minor issues that were present in the translation logic.

The code generation update has landed in the Pulumi CLI and pulumi convert, the other tools however will pick up these updates soon.

Let us take an example from Pulumi YAML and convert it to C#.

Generate a new Pulumi YAML project using pulumi new azure-yaml, it should give you this YAML program:

name: azure-yaml
runtime: yaml
description: A minimal Azure Native Pulumi YAML program

resources:
 # Create an Azure Resource Group
 resourceGroup:
 type: azure-native:resources:ResourceGroup
 # Create an Azure resource (Storage Account)
 storageAccount:
 type: azure-native:storage:StorageAccount
 properties:
 resourceGroupName: ${resourceGroup.name}
 sku:
 name: Standard_LRS
 kind: StorageV2

variables:
 storageAccountKeys:
 Fn::Invoke:
 Function: azure-native:storage:listStorageAccountKeys
 Arguments:
 resourceGroupName: ${resourceGroup.name}
 accountName: ${storageAccount.name}

outputs:
 # Export the primary key of the Storage Account
 primaryStorageKey: ${storageAccountKeys.keys[0].value}

Now you can take this program and convert it to C# using

pulumi convert --language csharp --out csharp-from-yaml

This command generates a full C# project from the YAML program into the directory ./csharp-from-yaml, then restores the dependencies and builds the project. Now if you examine Program.cs, it contains the converted program and follows the succinct C# 10 syntax that targets .NET 6:

using System.Collections.Generic;
using Pulumi;
using AzureNative = Pulumi.AzureNative;

return await Deployment.RunAsync(() =>
{
 var resourceGroup = new AzureNative.Resources.ResourceGroup("resourceGroup");

 var storageAccount = new AzureNative.Storage.StorageAccount("storageAccount", new()
 {
 ResourceGroupName = resourceGroup.Name,
 Sku = new AzureNative.Storage.Inputs.SkuArgs
 {
 Name = "Standard_LRS",
 },
 Kind = "StorageV2",
 });

 var storageAccountKeys = AzureNative.Storage.ListStorageAccountKeys.Invoke(new()
 {
 ResourceGroupName = resourceGroup.Name,
 AccountName = storageAccount.Name,
 });

 return new Dictionary<string, object?>
 {
 ["primaryStorageKey"] = storageAccountKeys.Apply(listStorageAccountKeysResult => listStorageAccountKeysResult.Keys[0]?.Value),
 };
});

Summary

.NET 6 is the latest Long-Term Support version in the Microsoft ecosystem. Our new .NET 6 support makes for simpler and easier to write Pulumi programs. You can give it a try by installing .NET 6 and using one of the templates with pulumi new, then follow our updated Get Started guide to get up and running quickly. Our code generation tools have been upgraded to emit code that targets the latest features of C# and fixes minor correctness issues in the translation logic.

Enhanced static-code analysis for C# projects

Zaid Ajaj — Fri, 15 Jul 2022 00:00:00 +0000

When I started using Pulumi for the first time, I used C# as my language of choice for defining infrastructure. I start by creating resources and providing their parameters through argument objects. The IDE helps me out with auto-completions and type errors as I go but the compiler didn’t always detect some of the errors I eventually came across.

Consider the following piece of code:

var resourceGroup = new ResourceGroup("resourceGroup");
var storageAccount = new StorageAccount("storageAccount", new StorageAccountArgs
{
 ResourceGroupName = resourceGroup.Name,
 Sku = new SkuArgs { Name = SkuName.Standard_LRS }
});

In this snippet, I create a resource group in Azure, then create a storage account that belongs to that resource group. Pretty simple stuff! The IDE tells me that the code is good and doesn’t report any errors but just to be sure everything is as it should be, I build the project and the code compiles successfully. Now I am confident and ready to deploy the infrastructure.

I run pulumi preview and to my surprise, it fails. The C# program throws an exception at runtime:

Diagnostics:
pulumi:pulumi:Stack (AzureExample-dev):
error: Running program '/Users/zaid/projects/AzureExample/bin/Debug/net6.0/AzureExample.dll' failed with an unhandled exception:
System.ArgumentNullException: [Input] Pulumi.AzureNative.Storage.StorageAccountArgs.Kind is required but was not given a value (Parameter 'Kind')

What happened? It turns out I didn’t provide a required property Kind when instantiating the type StorageAccountArgs for the StorageAccount.

This is an unfortunate result because I had expected that when my program compiled, that it was valid and ready to deploy.

In this case, the fix is simple:

var storageAccount = new StorageAccount("storageAccount", new StorageAccountArgs
{
 ResourceGroupName = resourceGroup.Name,
 Sku = new SkuArgs { Name = SkuName.Standard_LRS },
 // adding this property
 Kind = Kind.StorageV2
});

There is a bigger problem in the API of the resource creation. The compiler should tell you at compile-time that you missed a required property. The underlying cause is because we are using property initializers for StorageAccountArgs with a parameterless constructor. This is the case when creating any resource across the cloud providers available in the Pulumi ecosystem.

See Pulumi #3808 for more details and discussion on this issue.

What can we do to fix the problem?

First solution that comes to mind is to fix the generated .NET API for each cloud provider available with Pulumi: adding the required parameters as part of the constructor of arguments type and removing the parameterless constructor.

This however means introducing a breaking change for each and every existing C# program running with Pulumi using the empty constructor. Here at Pulumi, we want to ensure that changes to our APIs are backward-compatible, so this is a no-go.

The problem above looks like the compiler could have detected if it somehow knew which properties were required or and which were optional. In the case of the generated APIs, this requiredness information is only encoded on properties with a special attribute [Input("propertyName", required: true)] which unfortunately is Pulumi-specific and the compiler has no way of detecting it and reporting an error or a warning to the developer at compile-time.

Solution

What do we do? Static code analysis to the rescue! If the compiler can’t do it, we can build an extension and have detect these types of errors for the developers.

Enter PulumiCSharpAnalyzer: a Roslyn-based code analyzer that adds compile-time checks to your Pulumi code and reports it as part of the build.

Available today as a NuGet package, you can install the analyzer into your project as follows:

dotnet add package PulumiCSharpAnalyzer

Now your IDE, whether that is Rider, Visual Studio or VS Code with OmniSharp should report the error inside the IDE like any compile error or warning. Here is for example how it looks like inside Rider:

Not just your IDE, it integrates into your build pipeline too. When you dotnet build your project, the detected errors will show up there:

> dotnet build
Microsoft (R) Build Engine version 17.2.0+41abc5629 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

 Determining projects to restore...
 All projects are up-to-date for restore.
/Users/zaid/projects/AzureExample/Program.cs(11,63): warning MissingRequiredProperty: Missing required property Kind when initializing properties of type StorageAccountArgs [/Users/zaid/projects/AzureExample/AzureExample.csproj]
 AzureExample -> /Users/zaid/projects/AzureExample/bin/Debug/net6.0/AzureExample.dll

Build succeeded.

Besides detecting missing properties when creating resources, the analyzer also detects missing properties for function invokes which follow a similar pattern:

Finally, an anti-pattern that the analyzer can detect is creating resources within an .Apply(...) call:

We always recommend developers to avoid creating resources inside .Apply(...) calls because the execution plan of Pulumi will be different during the pulumi preview and pulumi up phase.

The reason is that when we have an instance of Output<T> from which we call Apply, the value of the instance is most likely unknown during pulumi preview since it is only computed when Pulumi actually performs the deployment plan against APIs of cloud providers.

Limitations

With the current features of the analyzer, it is already usable across all cloud provider SDKs. That said, there are limitations to the way the analyzer works. At the moment, when determining the missing properties, the logic only detects initialized properties when creating the object and diffs those against the required properties from the schema. This means that if you happened to use property setters instead of property initializers, they are considered “not provided” and will report a warning. For example, the following piece of code will produce a false-positive warning…

var storageAccountArgs = new StorageAccountArgs();
storageAccountArgs.ResourceGroupName = resourceGroup.Name;
storageAccountArgs.Kind = Kind.StorageV2;
storageAccountArgs.Sku = new SkuArgs { Name = SkuName.Standard_LRS };

…even though it is logically equivalent to

new StorageAccountArgs
{
 ResourceGroupName = resourceGroup.Name,
 Kind = Kind.StorageV2,
 Sku = new SkuArgs
 {
 Name = SkuName.Standard_LRS
 }
};

This is something I will be working on in the near future and the fix is relatively doable, but it is just a heads-up that the analyzer is not perfect at the moment.

Conclusion

Install the analyzer today in your C# Pulumi projects and get better static-code analysis in your Pulumi projects. Keep in the mind the limitations outlined above and let us know if you encounter issues that we didn’t take into account before.