Using Go Generics with Pulumi

Pulumi loves Go, it’s what powers Pulumi. We’ve kept a close eye on the design and development of support for generics in the Go programming language over the years, a feature that allows developers to write type-safe, concise, and reusable code. We’ve been exploring what it’d look like to improve Pulumi’s Go SDKs with generics and recently published a public RFC detailing our plans. We’ve been making progress on the implementation and are excited to announce preview support for Go generics in our core and AWS Go SDKs. If you’re using Go with Pulumi, we’d love for you to give it a try and share your feedback!

// Given
var a pulumi.IntOutput
var b pulumi.StringOutput

// Before (could panic at runtime if you got something wrong)
o := pulumi.All(a, b).ApplyT(func(vs []interface{}) string { // could panic
    a := vs[0].(int) // could panic
    b := vs[1].(string) // could panic
    return strconv.Itoa(a) + b
}).(pulumi.StringOutput) // could panic

// After (compile-time type-safety)
o := pulumix.Apply2(a, b, func(a int, b string) string {
    return strconv.Itoa(a) + b
})

Why Generics?

Before looking at what’s available in the preview, let’s first review the motivations for adding support for generics, from the RFC.

Unsafe Inputs, Outputs, and Apply

A common complaint about the Pulumi Go SDK is the lack of type safety. The Input and Output types, and the apply operation rely heavily on interface{}. This makes them effectively untyped at compile-time, instead relying on type matching at runtime.

For instance, the apply operation is implemented in Go as the following method:

func (*OutputState) ApplyT(applier interface{}) Output

The contract for ApplyT states that applier must be a function with one of the following signatures:

func(U) T
func(U) (T, error)

Where U must be assignable from the type contained in the underlying Output. If it isn’t, the operation will panic.

The caller will usually want to cast the result into a specific output type. If they choose the wrong output type, this operation will panic.

So given a typical usage like the following, there are two spots that can panic: the ApplyT invocation and the IntOutput cast.

intOutput := stringOutput.ApplyT(func(v string) int {
    return len(v)
}).(pulumi.IntOutput)

Excessive code generation

The Go SDK relies heavily on code generation in an attempt to alleviate the pain of untyped operations. It generates type-specific input and output wrapper types for nearly every type it encounters.

In the core Pulumi SDK, this includes all primitive types, their pointers, containers, and containers of containers. For instance, the core SDK has 22 String* types besides type String itself, covering nearly all variations of the following pattern:

String(Ptr|(Array|Map)?(Array|Map))?(Input|Output)?

This pattern of code generation carries over to code generated for provider SDKs, though not to this degree of nesting. Every type gets an input and output type, a pointer variant if the type is referenced as a pointer, and container variants if the type is referenced from a container.

In general, this generates extremely large packages that are ungainly to navigate—both in source form and in their API reference.

Introducing Go Generics in the Pulumi Go SDK

We can address the above by leveraging Go generics to incorporate type-safety into the core concepts (inputs, outputs, and apply) and reduce the amount of generated code.

Starting with v3.80 of the Pulumi Go SDK, we have added a new pulumix subpackage (containing “extensions”), which includes new generic Input[T] and Output[T] types as well as type-safe generics-based APIs to work with them such as Apply.

With the new support for generics, the previous untyped example can now be written in a type-safe manner as follows:

intOutput := pulumix.Apply(stringOutput, func(v string) int {
    return len(v)
})

This is checked at compile-time: if you mistakenly specified an incorrect type, it will be caught at compile-time rather than panicking at runtime.

Adopting Generics in Your Pulumi Programs

Here’s a larger example that uses the non-generic pulumi.All function and ApplyT method to produce a JSON string representing a policy for a BucketPolicy:

package main

import (
	"encoding/json"

	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/cloudfront"
	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
			Acl: pulumi.String("private"),
			Website: &s3.BucketWebsiteArgs{
				IndexDocument: pulumi.String("index.html"),
				ErrorDocument: pulumi.String("404.html"),
			},
		})
		if err != nil {
			return err
		}

		originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
			Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
		})
		if err != nil {
			return err
		}

		_, err = s3.NewBucketPolicy(ctx, "cloudfront-bucket-policy", &s3.BucketPolicyArgs{
			Bucket: bucket.ID(),
			Policy: pulumi.All(bucket.Arn, originAccessIdentity.IamArn).ApplyT(
				func(args []any) (string, error) {
					bucketArn := args[0].(string)
					iamArn := args[1].(string)

					policy, err := json.Marshal(map[string]any{
						"Version": "2012-10-17",
						"Statement": []map[string]any{
							{
								"Sid":    "CloudfrontAllow",
								"Effect": "Allow",
								"Principal": map[string]any{
									"AWS": iamArn,
								},
								"Action":   "s3:GetObject",
								"Resource": bucketArn + "/*",
							},
						},
					})
					if err != nil {
						return "", err
					}
					return string(policy), nil
				}).(pulumi.StringOutput),
		}, pulumi.Parent(bucket))
		if err != nil {
			return err
		}

		return nil
	})
}

pulumi.All is used to combine bucket.Arn and originAccessIdentity.IamArn into a single Output and then uses ApplyT construct a policy in JSON using the values.

If we wanted to take advantage of the type-safety provided by the new pulumix subpackage, we could rewrite the above example to use pulumix.Apply. First, import "github.com/pulumi/pulumi/sdk/v3/go/pulumix". Then, replace the use of pulumi.All and ApplyT with the generic pulumix.Apply:

			Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
				func(bucketArn, iamArn string) (string, error) {
					policy, err := json.Marshal(map[string]any{
						"Version": "2012-10-17",
						"Statement": []map[string]any{
							{
								"Sid":    "CloudfrontAllow",
								"Effect": "Allow",
								"Principal": map[string]any{
									"AWS": iamArn,
								},
								"Action":   "s3:GetObject",
								"Resource": bucketArn + "/*",
							},
						},
					})
					if err != nil {
						return "", err
					}
					return string(policy), nil
				}),

Here, we’re specifically using pulumix.Apply2Err, a variant of pulumix.Apply that accepts two generic arguments and a callback function that supports returning an error. There are other variants of pulumix.Apply that accept up to 8 generic arguments, and variants that return an error and accept a context.Context.

The result of our call to pulumix.Apply2Err in this example is pulumix.Output[string]. We can pass this directly to the Policy argument because it happens to be weakly typed as pulumi.Input, which pulumix.Output[T] implements.

If the Policy argument had been typed as pulumi.StringInput, for example, we would have had to cast the pulumix.Output[string] to pulumi.StringOutput using the pulumix.Cast function:

policy := pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
    func(bucketArn, iamArn string) (string, error) {
        // ...
    })

// ...

Policy: pulumix.Cast[pulumi.StringOutput](policy),

Here’s the updated example in full:

package main

import (
	"encoding/json"

	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/cloudfront"
	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
			Acl: pulumi.String("private"),
			Website: &s3.BucketWebsiteArgs{
				IndexDocument: pulumi.String("index.html"),
				ErrorDocument: pulumi.String("404.html"),
			},
		})
		if err != nil {
			return err
		}

		originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
			Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
		})
		if err != nil {
			return err
		}

		_, err = s3.NewBucketPolicy(ctx, "bucket-policy", &s3.BucketPolicyArgs{
			Bucket: bucket.ID(),
			Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
				func(bucketArn, iamArn string) (string, error) {
					policy, err := json.Marshal(map[string]any{
						"Version": "2012-10-17",
						"Statement": []map[string]any{
							{
								"Sid":    "CloudfrontAllow",
								"Effect": "Allow",
								"Principal": map[string]any{
									"AWS": iamArn,
								},
								"Action":   "s3:GetObject",
								"Resource": bucketArn + "/*",
							},
						},
					})
					if err != nil {
						return "", err
					}
					return string(policy), nil
				}),
		}, pulumi.Parent(bucket))
		if err != nil {
			return err
		}

		return nil
	})
}

Notice how we are importing pulumix alongside pulumi. Being a subpackage means you can start using the generics-based types and functions alongside the current APIs as necessary without having to switch your code fully from using the old types to the new generic ones. Instead, you can migrate your code gradually to these new APIs where it makes sense. When needed, there are APIs to covert between generics-based APIs and the old ones using functions such as pulumix.Cast and pulumix.ConvertTyped.

Generics-Based Cloud Provider SDKs

We’ve seen how the pulumix subpackage on its own can be used with existing Pulumi provider SDKs. We can go further and adopt generics in the provider SDKs themselves. To make the transition non-breaking and gradual for your Pulumi programs, we intend to publish generic-based variants of the provider Go SDKs under a subpackage called x (for “extensions”). This allows you to migrate your code resource by resource.

Migrating an AWS Program to leverage Go generics

The following example is an AWS Pulumi program which we’ll migrate into the generics-based API. Here is what pulumi new aws-go gives you today:

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// Create an AWS resource (S3 Bucket)
		bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
			Website: &s3.BucketWebsiteArgs{
				IndexDocument: pulumi.String("index.html"),
			},
    	})
		if err != nil {
			return err
		}

		// Export the name of the bucket
		ctx.Export("bucketName", bucket.ID())
		return nil
	})
}

To start using the generics-based version of the S3 module from AWS, you need to change the import from "https://github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3" to "https://github.com/pulumi/pulumi-aws/sdk/v6/go/aws/x/s3" (notice the x after aws/), then change input values to use pulumix.Ptr instead of pulumi.String.

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/x/s3"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// Create an AWS resource (S3 Bucket)
		bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
			Website: &s3.BucketWebsiteArgs{
				IndexDocument: pulumix.Ptr("index.html"),
			},
		})
		if err != nil {
			return err
		}

		// Export the name of the bucket
		ctx.Export("bucketName", bucket.ID())
		return nil
	})
}

You’ll also need to update your go.mod to use the preview branch of the Pulumi AWS Go SDK that includes the x subpackage with generic APIs. Run the following:

go get github.com/pulumi/pulumi-aws/sdk/v6@generics
go mod tidy

The changes look minimal, except the fact that the inputs of s3.BucketArgs and the outputs of Bucket are now fully generics-based. Here is the Bucket type you get from NewBucket:

type Bucket struct {
	pulumi.CustomResourceState
	AccelerationStatus pulumix.Output[string] `pulumi:"accelerationStatus"`
	Acl pulumix.Output[*string] `pulumi:"acl"`
	Arn pulumix.Output[string] `pulumi:"arn"`
	Bucket pulumix.Output[string] `pulumi:"bucket"`
	// ....
}

And here are its input arguments:

type BucketArgs struct {
	AccelerationStatus pulumix.Input[*string]
	Acl pulumix.Input[*string]

	Arn pulumix.Input[*string]
	Bucket pulumix.Input[*string]
	// ...
}

Authoring Generics-based Go SDKs

For Pulumi package authors, it is possible to start generating Go SDKs for your providers that include the generics-based variant under the x subpackage using v3.84.0 of Pulumi. By default, this is not enabled so you will have to explicitly opt-in to it using a specific option in the go language section of your Pulumi schema. The new option is called "generics" and it has three possible values:

• "none" is the default value when the option is not set. This doesn’t generate the generics-based variant
• "side-by-side" is the option we are using for our select providers which generates the variant under the x subdirectory.
• "generics-only" if you want your provider to only emit generics-based APIs. Since this is a preview feature, things might still be rough around the edges, so use this option with caution as fixes in the SDK-generation code might result in breaking changes in your APIs.

Here is how this looks in JSON:

{
   "language": {
       "go": {
           "generics": "side-by-side"
        }
   }
}

We eventually intend to make "side-by-side" be the default, such that all providers get a generics-based variant under the x subpackage, so that all users can use the full generics API.

Try it out!

We are excited for you to try out these new APIs and would love to hear your feedback! You can learn more details on the design and add comments to the RFC, or chat with us in the #golang channel on the Pulumi Community Slack.

If you’d like to learn more about building cloud applications with Go and Pulumi, we have a workshop coming up in October. Sign up here.

We’ll also be at GopherCon, if you’d like to meet with us.