Using Go Generics with Pulumi
Posted on
Pulumi loves Go, it’s what powers Pulumi. We’ve kept a close eye on the design and development of support for generics in the Go programming language over the years, a feature that allows developers to write type-safe, concise, and reusable code. We’ve been exploring what it’d look like to improve Pulumi’s Go SDKs with generics and recently published a public RFC detailing our plans. We’ve been making progress on the implementation and are excited to announce preview support for Go generics in our core and AWS Go SDKs. If you’re using Go with Pulumi, we’d love for you to give it a try and share your feedback!
// Given
var a pulumi.IntOutput
var b pulumi.StringOutput
// Before (could panic at runtime if you got something wrong)
o := pulumi.All(a, b).ApplyT(func(vs []interface{}) string { // could panic
a := vs[0].(int) // could panic
b := vs[1].(string) // could panic
return strconv.Itoa(a) + b
}).(pulumi.StringOutput) // could panic
// After (compile-time type-safety)
o := pulumix.Apply2(a, b, func(a int, b string) string {
return strconv.Itoa(a) + b
})
Why Generics?
Before looking at what’s available in the preview, let’s first review the motivations for adding support for generics, from the RFC.
Unsafe Inputs, Outputs, and Apply
A common complaint about the Pulumi Go SDK is the lack of type safety. The Input
and Output
types, and the apply
operation rely heavily on interface{}
. This makes them effectively untyped at compile-time, instead relying on type matching at runtime.
For instance, the apply
operation is implemented in Go as the following method:
func (*OutputState) ApplyT(applier interface{}) Output
The contract for ApplyT
states that applier
must be a function with one of the following signatures:
func(U) T
func(U) (T, error)
Where U
must be assignable from the type contained in the underlying Output
. If it isn’t, the operation will panic.
The caller will usually want to cast the result into a specific output type. If they choose the wrong output type, this operation will panic.
So given a typical usage like the following, there are two spots that can panic: the ApplyT
invocation and the IntOutput
cast.
intOutput := stringOutput.ApplyT(func(v string) int {
return len(v)
}).(pulumi.IntOutput)
Excessive code generation
The Go SDK relies heavily on code generation in an attempt to alleviate the pain of untyped operations. It generates type-specific input and output wrapper types for nearly every type it encounters.
In the core Pulumi SDK, this includes all primitive types, their pointers, containers, and containers of containers. For instance, the core SDK has 22 String*
types besides type String
itself, covering nearly all variations of the following pattern:
String(Ptr|(Array|Map)?(Array|Map))?(Input|Output)?
This pattern of code generation carries over to code generated for provider SDKs, though not to this degree of nesting. Every type gets an input and output type, a pointer variant if the type is referenced as a pointer, and container variants if the type is referenced from a container.
In general, this generates extremely large packages that are ungainly to navigate—both in source form and in their API reference.
Introducing Go Generics in the Pulumi Go SDK
We can address the above by leveraging Go generics to incorporate type-safety into the core concepts (inputs, outputs, and apply) and reduce the amount of generated code.
Starting with v3.80 of the Pulumi Go SDK, we have added a new pulumix
subpackage (containing “extensions”), which includes new generic Input[T]
and Output[T]
types as well as type-safe generics-based APIs to work with them such as Apply
.
With the new support for generics, the previous untyped example can now be written in a type-safe manner as follows:
intOutput := pulumix.Apply(stringOutput, func(v string) int {
return len(v)
})
This is checked at compile-time: if you mistakenly specified an incorrect type, it will be caught at compile-time rather than panicking at runtime.
Apply
above without having to explicitly specify generic type parameters.Adopting Generics in Your Pulumi Programs
Here’s a larger example that uses the non-generic pulumi.All
function and ApplyT
method to produce a JSON string representing a policy for a BucketPolicy
:
package main
import (
"encoding/json"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/cloudfront"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
Acl: pulumi.String("private"),
Website: &s3.BucketWebsiteArgs{
IndexDocument: pulumi.String("index.html"),
ErrorDocument: pulumi.String("404.html"),
},
})
if err != nil {
return err
}
originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
})
if err != nil {
return err
}
_, err = s3.NewBucketPolicy(ctx, "cloudfront-bucket-policy", &s3.BucketPolicyArgs{
Bucket: bucket.ID(),
Policy: pulumi.All(bucket.Arn, originAccessIdentity.IamArn).ApplyT(
func(args []any) (string, error) {
bucketArn := args[0].(string)
iamArn := args[1].(string)
policy, err := json.Marshal(map[string]any{
"Version": "2012-10-17",
"Statement": []map[string]any{
{
"Sid": "CloudfrontAllow",
"Effect": "Allow",
"Principal": map[string]any{
"AWS": iamArn,
},
"Action": "s3:GetObject",
"Resource": bucketArn + "/*",
},
},
})
if err != nil {
return "", err
}
return string(policy), nil
}).(pulumi.StringOutput),
}, pulumi.Parent(bucket))
if err != nil {
return err
}
return nil
})
}
pulumi.All
is used to combine bucket.Arn
and originAccessIdentity.IamArn
into a single Output
and then uses ApplyT
construct a policy in JSON using the values.
If we wanted to take advantage of the type-safety provided by the new pulumix
subpackage, we could rewrite the above example to use pulumix.Apply
. First, import "github.com/pulumi/pulumi/sdk/v3/go/pulumix"
. Then, replace the use of pulumi.All
and ApplyT
with the generic pulumix.Apply
:
Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
func(bucketArn, iamArn string) (string, error) {
policy, err := json.Marshal(map[string]any{
"Version": "2012-10-17",
"Statement": []map[string]any{
{
"Sid": "CloudfrontAllow",
"Effect": "Allow",
"Principal": map[string]any{
"AWS": iamArn,
},
"Action": "s3:GetObject",
"Resource": bucketArn + "/*",
},
},
})
if err != nil {
return "", err
}
return string(policy), nil
}),
Here, we’re specifically using pulumix.Apply2Err
, a variant of pulumix.Apply
that accepts two generic arguments and a callback function that supports returning an error. There are other variants of pulumix.Apply
that accept up to 8 generic arguments, and variants that return an error and accept a context.Context
.
The result of our call to pulumix.Apply2Err
in this example is pulumix.Output[string]
. We can pass this directly to the Policy
argument because it happens to be weakly typed as pulumi.Input
, which pulumix.Output[T]
implements.
If the Policy
argument had been typed as pulumi.StringInput
, for example, we would have had to cast the pulumix.Output[string]
to pulumi.StringOutput
using the pulumix.Cast
function:
policy := pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
func(bucketArn, iamArn string) (string, error) {
// ...
})
// ...
Policy: pulumix.Cast[pulumi.StringOutput](policy),
Here’s the updated example in full:
package main
import (
"encoding/json"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/cloudfront"
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
"github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
Acl: pulumi.String("private"),
Website: &s3.BucketWebsiteArgs{
IndexDocument: pulumi.String("index.html"),
ErrorDocument: pulumi.String("404.html"),
},
})
if err != nil {
return err
}
originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
})
if err != nil {
return err
}
_, err = s3.NewBucketPolicy(ctx, "bucket-policy", &s3.BucketPolicyArgs{
Bucket: bucket.ID(),
Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
func(bucketArn, iamArn string) (string, error) {
policy, err := json.Marshal(map[string]any{
"Version": "2012-10-17",
"Statement": []map[string]any{
{
"Sid": "CloudfrontAllow",
"Effect": "Allow",
"Principal": map[string]any{
"AWS": iamArn,
},
"Action": "s3:GetObject",
"Resource": bucketArn + "/*",
},
},
})
if err != nil {
return "", err
}
return string(policy), nil
}),
}, pulumi.Parent(bucket))
if err != nil {
return err
}
return nil
})
}
Notice how we are importing pulumix
alongside pulumi
. Being a subpackage means you can start using the generics-based types and functions alongside the current APIs as necessary without having to switch your code fully from using the old types to the new generic ones. Instead, you can migrate your code gradually to these new APIs where it makes sense. When needed, there are APIs to covert between generics-based APIs and the old ones using functions such as pulumix.Cast
and pulumix.ConvertTyped
.
Generics-Based Cloud Provider SDKs
We’ve seen how the pulumix
subpackage on its own can be used with existing Pulumi provider SDKs. We can go further and adopt generics in the provider SDKs themselves. To make the transition non-breaking and gradual for your Pulumi programs, we intend to publish generic-based variants of the provider Go SDKs under a subpackage called x
(for “extensions”). This allows you to migrate your code resource by resource.
Migrating an AWS Program to leverage Go generics
The following example is an AWS Pulumi program which we’ll migrate into the generics-based API. Here is what pulumi new aws-go
gives you today:
package main
import (
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Create an AWS resource (S3 Bucket)
bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
Website: &s3.BucketWebsiteArgs{
IndexDocument: pulumi.String("index.html"),
},
})
if err != nil {
return err
}
// Export the name of the bucket
ctx.Export("bucketName", bucket.ID())
return nil
})
}
To start using the generics-based version of the S3 module from AWS, you need to change the import from "https://github.com/pulumi/pulumi-aws/sdk/v6/go/aws/s3"
to "https://github.com/pulumi/pulumi-aws/sdk/v6/go/aws/x/s3"
(notice the x
after aws/
), then change input values to use pulumix.Ptr
instead of pulumi.String
.
pulumi.String("foo")
and pulumi.Int(42)
, with generics, you can simply use pulumix.Val("foo")
and pulumix.Val(42)
. Similarly, you can use the pulumix.Ptr
function for pointer values.package main
import (
"github.com/pulumi/pulumi-aws/sdk/v6/go/aws/x/s3"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
"github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Create an AWS resource (S3 Bucket)
bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
Website: &s3.BucketWebsiteArgs{
IndexDocument: pulumix.Ptr("index.html"),
},
})
if err != nil {
return err
}
// Export the name of the bucket
ctx.Export("bucketName", bucket.ID())
return nil
})
}
You’ll also need to update your go.mod
to use the preview branch of the Pulumi AWS Go SDK that includes the x
subpackage with generic APIs. Run the following:
go get github.com/pulumi/pulumi-aws/sdk/v6@generics
go mod tidy
The changes look minimal, except the fact that the inputs of s3.BucketArgs
and the outputs of Bucket
are now fully generics-based. Here is the Bucket
type you get from NewBucket
:
type Bucket struct {
pulumi.CustomResourceState
AccelerationStatus pulumix.Output[string] `pulumi:"accelerationStatus"`
Acl pulumix.Output[*string] `pulumi:"acl"`
Arn pulumix.Output[string] `pulumi:"arn"`
Bucket pulumix.Output[string] `pulumi:"bucket"`
// ....
}
And here are its input arguments:
type BucketArgs struct {
AccelerationStatus pulumix.Input[*string]
Acl pulumix.Input[*string]
Arn pulumix.Input[*string]
Bucket pulumix.Input[*string]
// ...
}
Authoring Generics-based Go SDKs
For Pulumi package authors, it is possible to start generating Go SDKs for your providers that include the generics-based variant under the x
subpackage using v3.84.0 of Pulumi. By default, this is not enabled so you will have to explicitly opt-in to it using a specific option in the go language section of your Pulumi schema. The new option is called "generics"
and it has three possible values:
"none"
is the default value when the option is not set. This doesn’t generate the generics-based variant"side-by-side"
is the option we are using for our select providers which generates the variant under the x subdirectory."generics-only"
if you want your provider to only emit generics-based APIs. Since this is a preview feature, things might still be rough around the edges, so use this option with caution as fixes in the SDK-generation code might result in breaking changes in your APIs.
Here is how this looks in JSON:
{
"language": {
"go": {
"generics": "side-by-side"
}
}
}
We eventually intend to make "side-by-side"
be the default, such that all providers get a generics-based variant under the x
subpackage, so that all users can use the full generics API.
Try it out!
We are excited for you to try out these new APIs and would love to hear your feedback! You can learn more details on the design and add comments to the RFC, or chat with us in the #golang
channel on the Pulumi Community Slack.
If you’d like to learn more about building cloud applications with Go and Pulumi, we have a workshop coming up in October. Sign up here.
We’ll also be at GopherCon, if you’d like to meet with us.