Pulumi Blog: Go

Using Go Generics with Pulumi

Zaid Ajaj — Wed, 20 Sep 2023 00:00:00 +0000

Pulumi loves Go, it’s what powers Pulumi. We’ve kept a close eye on the design and development of support for generics in the Go programming language over the years, a feature that allows developers to write type-safe, concise, and reusable code. We’ve been exploring what it’d look like to improve Pulumi’s Go SDKs with generics and recently published a public RFC detailing our plans. We’ve been making progress on the implementation and are excited to announce preview support for Go generics in our core and AWS Go SDKs. If you’re using Go with Pulumi, we’d love for you to give it a try and share your feedback!

// Given
var a pulumi.IntOutput
var b pulumi.StringOutput

// Before (could panic at runtime if you got something wrong)
o := pulumi.All(a, b).ApplyT(func(vs []interface{}) string { // could panic
 a := vs[0].(int) // could panic
 b := vs[1].(string) // could panic
 return strconv.Itoa(a) + b
}).(pulumi.StringOutput) // could panic

// After (compile-time type-safety)
o := pulumix.Apply2(a, b, func(a int, b string) string {
 return strconv.Itoa(a) + b
})

Why Generics?

Before looking at what’s available in the preview, let’s first review the motivations for adding support for generics, from the RFC.

Unsafe Inputs, Outputs, and Apply

A common complaint about the Pulumi Go SDK is the lack of type safety. The Input and Output types, and the apply operation rely heavily on interface{}. This makes them effectively untyped at compile-time, instead relying on type matching at runtime.

For instance, the apply operation is implemented in Go as the following method:

func (*OutputState) ApplyT(applier interface{}) Output

The contract for ApplyT states that applier must be a function with one of the following signatures:

func(U) T
func(U) (T, error)

Where U must be assignable from the type contained in the underlying Output. If it isn’t, the operation will panic.

The caller will usually want to cast the result into a specific output type. If they choose the wrong output type, this operation will panic.

So given a typical usage like the following, there are two spots that can panic: the ApplyT invocation and the IntOutput cast.

intOutput := stringOutput.ApplyT(func(v string) int {
 return len(v)
}).(pulumi.IntOutput)

Excessive code generation

The Go SDK relies heavily on code generation in an attempt to alleviate the pain of untyped operations. It generates type-specific input and output wrapper types for nearly every type it encounters.

In the core Pulumi SDK, this includes all primitive types, their pointers, containers, and containers of containers. For instance, the core SDK has 22 String* types besides type String itself, covering nearly all variations of the following pattern:

String(Ptr|(Array|Map)?(Array|Map))?(Input|Output)?

This pattern of code generation carries over to code generated for provider SDKs, though not to this degree of nesting. Every type gets an input and output type, a pointer variant if the type is referenced as a pointer, and container variants if the type is referenced from a container.

In general, this generates extremely large packages that are ungainly to navigate—both in source form and in their API reference.

Introducing Go Generics in the Pulumi Go SDK

We can address the above by leveraging Go generics to incorporate type-safety into the core concepts (inputs, outputs, and apply) and reduce the amount of generated code.

Starting with v3.80 of the Pulumi Go SDK, we have added a new pulumix subpackage (containing “extensions”), which includes new generic Input[T] and Output[T] types as well as type-safe generics-based APIs to work with them such as Apply.

With the new support for generics, the previous untyped example can now be written in a type-safe manner as follows:

intOutput := pulumix.Apply(stringOutput, func(v string) int {
 return len(v)
})

This is checked at compile-time: if you mistakenly specified an incorrect type, it will be caught at compile-time rather than panicking at runtime.

We recommend using Go 1.21 or later, which has improved type inference for generics, allowing you to write the call to Apply above without having to explicitly specify generic type parameters.

Adopting Generics in Your Pulumi Programs

Here’s a larger example that uses the non-generic pulumi.All function and ApplyT method to produce a JSON string representing a policy for a BucketPolicy:

package main

import (
 "encoding/json"

 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/cloudfront"
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 ErrorDocument: pulumi.String("404.html"),
 },
 })
 if err != nil {
 return err
 }

 originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
 Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
 })
 if err != nil {
 return err
 }

 _, err = s3.NewBucketPolicy(ctx, "cloudfront-bucket-policy", &s3.BucketPolicyArgs{
 Bucket: bucket.ID(),
 Policy: pulumi.All(bucket.Arn, originAccessIdentity.IamArn).ApplyT(
 func(args []any) (string, error) {
 bucketArn := args[0].(string)
 iamArn := args[1].(string)

 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }).(pulumi.StringOutput),
 }, pulumi.Parent(bucket))
 if err != nil {
 return err
 }

 return nil
 })
}

pulumi.All is used to combine bucket.Arn and originAccessIdentity.IamArn into a single Output and then uses ApplyT construct a policy in JSON using the values.

If we wanted to take advantage of the type-safety provided by the new pulumix subpackage, we could rewrite the above example to use pulumix.Apply. First, import "github.com/pulumi/pulumi/sdk/v3/go/pulumix". Then, replace the use of pulumi.All and ApplyT with the generic pulumix.Apply:

 Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }),

Here, we’re specifically using pulumix.Apply2Err, a variant of pulumix.Apply that accepts two generic arguments and a callback function that supports returning an error. There are other variants of pulumix.Apply that accept up to 8 generic arguments, and variants that return an error and accept a context.Context.

The result of our call to pulumix.Apply2Err in this example is pulumix.Output[string]. We can pass this directly to the Policy argument because it happens to be weakly typed as pulumi.Input, which pulumix.Output[T] implements.

If the Policy argument had been typed as pulumi.StringInput, for example, we would have had to cast the pulumix.Output[string] to pulumi.StringOutput using the pulumix.Cast function:

policy := pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 // ...
 })

// ...

Policy: pulumix.Cast[pulumi.StringOutput](policy),

Here’s the updated example in full:

package main

import (
 "encoding/json"

 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/cloudfront"
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 bucket, err := s3.NewBucket(ctx, "content-bucket", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 ErrorDocument: pulumi.String("404.html"),
 },
 })
 if err != nil {
 return err
 }

 originAccessIdentity, err := cloudfront.NewOriginAccessIdentity(ctx, "cloudfront", &cloudfront.OriginAccessIdentityArgs{
 Comment: pulumi.Sprintf("OAI-%s", bucket.ID()),
 })
 if err != nil {
 return err
 }

 _, err = s3.NewBucketPolicy(ctx, "bucket-policy", &s3.BucketPolicyArgs{
 Bucket: bucket.ID(),
 Policy: pulumix.Apply2Err(bucket.Arn, originAccessIdentity.IamArn,
 func(bucketArn, iamArn string) (string, error) {
 policy, err := json.Marshal(map[string]any{
 "Version": "2012-10-17",
 "Statement": []map[string]any{
 {
 "Sid": "CloudfrontAllow",
 "Effect": "Allow",
 "Principal": map[string]any{
 "AWS": iamArn,
 },
 "Action": "s3:GetObject",
 "Resource": bucketArn + "/*",
 },
 },
 })
 if err != nil {
 return "", err
 }
 return string(policy), nil
 }),
 }, pulumi.Parent(bucket))
 if err != nil {
 return err
 }

 return nil
 })
}

Notice how we are importing pulumix alongside pulumi. Being a subpackage means you can start using the generics-based types and functions alongside the current APIs as necessary without having to switch your code fully from using the old types to the new generic ones. Instead, you can migrate your code gradually to these new APIs where it makes sense. When needed, there are APIs to convert between generics-based APIs and the old ones using functions such as pulumix.Cast and pulumix.ConvertTyped.

Generics-Based Cloud Provider SDKs

We’ve seen how the pulumix subpackage on its own can be used with existing Pulumi provider SDKs. We can go further and adopt generics in the provider SDKs themselves. To make the transition non-breaking and gradual for your Pulumi programs, we intend to publish generic-based variants of the provider Go SDKs under a subpackage called x (for “extensions”). This allows you to migrate your code resource by resource.

Migrating an AWS Program to leverage Go generics

The following example is an AWS Pulumi program which we’ll migrate into the generics-based API. Here is what pulumi new aws-go gives you today:

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an AWS resource (S3 Bucket)
 bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumi.String("index.html"),
 },
 })
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("bucketName", bucket.ID())
 return nil
 })
}

To start using the generics-based version of the S3 module from AWS, you need to change the import from "https://github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3" to "https://github.com/pulumi/pulumi-aws/sdk/v7/go/aws/x/s3" (notice the x after aws/), then change input values to use pulumix.Ptr instead of pulumi.String.

Rather than passing values as pulumi.String("foo") and pulumi.Int(42), with generics, you can simply use pulumix.Val("foo") and pulumix.Val(42). Similarly, you can use the pulumix.Ptr function for pointer values.

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/x/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumix"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an AWS resource (S3 Bucket)
 bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
 Website: &s3.BucketWebsiteArgs{
 IndexDocument: pulumix.Ptr("index.html"),
 },
 })
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("bucketName", bucket.ID())
 return nil
 })
}

You’ll also need to update your go.mod to use the preview branch of the Pulumi AWS Go SDK that includes the x subpackage with generic APIs. Run the following:

go get github.com/pulumi/pulumi-aws/sdk/v7@generics
go mod tidy

The changes look minimal, except the fact that the inputs of s3.BucketArgs and the outputs of Bucket are now fully generics-based. Here is the Bucket type you get from NewBucket:

type Bucket struct {
 pulumi.CustomResourceState
 AccelerationStatus pulumix.Output[string] `pulumi:"accelerationStatus"`
 Acl pulumix.Output[*string] `pulumi:"acl"`
 Arn pulumix.Output[string] `pulumi:"arn"`
 Bucket pulumix.Output[string] `pulumi:"bucket"`
 // ....
}

And here are its input arguments:

type BucketArgs struct {
 AccelerationStatus pulumix.Input[*string]
 Acl pulumix.Input[*string]

 Arn pulumix.Input[*string]
 Bucket pulumix.Input[*string]
 // ...
}

Authoring Generics-based Go SDKs

For Pulumi package authors, it is possible to start generating Go SDKs for your providers that include the generics-based variant under the x subpackage using v3.84.0 of Pulumi. By default, this is not enabled so you will have to explicitly opt-in to it using a specific option in the go language section of your Pulumi schema. The new option is called "generics" and it has three possible values:

"none" is the default value when the option is not set. This doesn’t generate the generics-based variant
"side-by-side" is the option we are using for our select providers which generates the variant under the x subdirectory.
"generics-only" if you want your provider to only emit generics-based APIs. Since this is a preview feature, things might still be rough around the edges, so use this option with caution as fixes in the SDK-generation code might result in breaking changes in your APIs.

Here is how this looks in JSON:

{
 "language": {
 "go": {
 "generics": "side-by-side"
 }
 }
}

We eventually intend to make "side-by-side" be the default, such that all providers get a generics-based variant under the x subpackage, so that all users can use the full generics API.

Try it out!

We are excited for you to try out these new APIs and would love to hear your feedback! You can learn more details on the design and add comments to the RFC, or chat with us in the #golang channel on the Pulumi Community Slack.

If you’d like to learn more about building cloud applications with Go and Pulumi, we have a workshop coming up in October. Sign up here.

We’ll also be at GopherCon, if you’d like to meet with us.

IaC Best Practices: Using Automation API

Scott Lowe — Wed, 26 Jul 2023 00:00:00 +0000

Welcome to the sixth post in our series of blog posts focused on Infrastructure as Code (IaC) best practices. So far in this series, you’ve seen how Zephyr Archaeotech Emporium—the fictional company at the center of this series—uses Pulumi to manage their online retail store. You read how Zephyr’s initial use of Pulumi changed to use short-lived per-developer stacks. Later, as Zephyr continued to grow, you saw how Zephyr restructured their Pulumi projects and stacks, incorporated Stack References, and used Pulumi Cloud’s role-based access control (RBAC) functionality to control access to their stacks. This post focuses on how Zephyr takes advantage of the Pulumi Automation API to bring an even greater level of orchestration to the stacks that represent their online store.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. These practices have unfolded organically, as a direct response to Zephyr’s evolving needs. The aim is to demonstrate that best practices are not set in stone, but are rather “point in time” recommendations that adapt as your company grows.

Managing Complexity in a Multi-Project Pulumi Setup

Over the course of this series, you’ve seen how Zephyr’s use of Pulumi has grown and changed. For the sake of completeness, here’s a quick recap:

Zephyr started with a single project in a single repository with two stacks, one for production and one for testing. (Details are available in understanding code organization and stacks post.)
To unlock developer velocity, Zephyr quickly added per-developer stacks. (You can read about per-developer stacks post in the series.)
As Zephyr grew and expanded, their single Pulumi project grew into three different Pulumi projects: one for base infrastructure, one for their Kubernetes platform, and one for the online store application. (The structuring Pulumi projects post describes the reasoning for adopting multiple projects.)
Accompanying the switch to multiple projects, Zephyr implemented stack references (to pass information between stacks in different projects) and applied role-based access control (RBAC) to the stacks in Pulumi Cloud. (Refer back to the stack references and RBAC & security posts in the series for more details on each of these areas.)

To see the state of Zephyr’s Pulumi implementation before the changes described in this article, please look at the blog/multi-project branch in each of the GitHub repositories (zephyr-infra, zephyr-k8s, and zephyr-app).

Since we last checked in on Zephyr, one significant change has emerged. As traffic to the Zephyr online store continued to increase, Zephyr’s operations team was finding it difficult to scale the relational databases running on Kubernetes (specifically, the orders and catalog databases). After evaluating their options, the team decided to migrate to Amazon Aurora with MySQL compatibility. Using the considerations outlined for structuring Pulumi projects, they decided to create a fourth Pulumi project, named “zephyr-data”, to manage the data infrastructure for the online store. Initially, this data infrastructure project manages only the Aurora clusters and database instances for the orders and catalog databases.

Because Zephyr was already using a modular approach to their overall application stack, inserting a new project for managing the online store’s data layer involved only minimal changes:

A new Pulumi program had to be written. This new program used stack references to leverage the existing infrastructure created by the base infrastructure stack and allowed traffic to the databases from the security group created by the Kubernetes platform stack. You’ll find this new program in the zephyr-data repository on GitHub (look in the blog/automation-api branch).
The Pulumi program for the Kubernetes platform stack only needed to export the security group created for the cluster nodes.
The Pulumi program for deploying the online store microservices referenced the database endpoints created by the new data infrastructure stack, and no longer spun up containerized instances of MariaDB for the orders and catalog databases.

The resulting four project architecture looks like this:

To see the state of the code after the changes described above—the addition of the zephyr-data project and the changes to use Amazon Aurora—please review the blog/automation-api branch in each of the Zephyr repositories.

Using a modular approach with multiple projects has its benefits. However, one of the trade-offs of splitting your IaC code into separate Pulumi projects is some additional relative complexity. With additional complexity comes additional room for error. To stand up a production-like environment, developers now had to stand up four different stacks (base infrastructure, Kubernetes platform, data infrastructure, and online store) in a specific order. If a user gets the order wrong, then the online store won’t come up properly.

With only four projects, this is not an insurmountable challenge by any stretch. However, what if it was 10 projects? Or 15 projects? Those numbers are not out of reach! Right now Zephyr has a single project to deploy all the various services involved for the online store. What if the application team decides to split up that project so that individual services can be deployed independently of each other? (After all, being able to deploy microservices independently of one another is one of the benefits of such an architecture.)

The Zephyr team started asking some questions: How can we make this process simpler? How can we remove room for error? How can we prepare our teams for greater scale?

Simplifying Advanced Deployments with Automation API

The challenges facing the Zephyr teams are not unique to Zephyr. Many IaC vendors recommend using a modular approach. Unfortunately, most IaC products rely on less-than-ideal approaches—like Bash scripts or Makefiles—to “stitch” together the different modules.

In some cases, organizations can use existing build automation tools and CI/CD pipelines to assist. What’s really needed, though, is a way to programmatically control the IaC tooling, and that’s what Pulumi’s Automation API is. The Pulumi Automation API allows Zephyr’s teams to use a general purpose programming language to automate Pulumi operations like performing an update, refreshing the stack state, or destroying a stack. Like Pulumi itself, the Pulumi Automation API supports multiple languages and supports multiple state backends, including Pulumi Cloud. Further, there is no requirement that Pulumi Automation API programs are written in the same language as the Pulumi programs they automate. In the case of Zephyr, which thus far has chosen to use TypeScript for their Pulumi programs, this means they are free to use any supported language for their Pulumi Automation API programs.

When the Zephyr team started evaluating options for simplifying the deployment of their online store and associated infrastructure, they decided to move forward with the Pulumi Automation API for these reasons:

Tight integration with Pulumi and Pulumi Cloud
Support for Pulumi Deployments (something we’ll examine in the next blog post)
Support for a broad range of programming languages, just like Pulumi itself
No requirement to use the same language as the Pulumi programs being automated (in fact, a Pulumi Automation API program could automate multiple programs written in different supported languages!)

Let’s take a look at Zephyr’s initial foray into using Pulumi Automation API.

Performing Stack Operations Efficiently

You can view the Pulumi Automation API program that Zephyr wrote in the local-source directory of the pulumi/zephyr-automation repository on GitHub, in the blog/automation-api branch.

Zephyr’s team decided to write their Pulumi Automation API program in Golang. One of the first things you’ll note about the Pulumi Automation API program is that it looks more like a “regular” Go program than a typical Pulumi Go program.

For example, a typical Pulumi Automation API program written in Go starts out like any other Go program:

func main() {
 // All your Go code is here
}

In a typical Pulumi Go program, on the other hand, the main() function starts off with a pulumi.Run call, like this:

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // All the Pulumi Go code is here
 })
}

The easiest way to understand this difference is to consider the way each program is invoked. A Pulumi Automation API program written in Go is executed by running go run main.go, like running any other Go program. In contrast, a Pulumi Go program is invoked by the pulumi CLI.

Zephyr’s Automation API program starts out with checking for a command-line argument to indicate the desired mode of the operation—should it be creating/updating resources, or destroying resources? In a standard Pulumi Go program, this would be determined by the invocation of the pulumi CLI (for example, whether you ran pulumi up or pulumi destroy). Here, though, that falls to the user to check:

// Determine mode of operation; default is to refresh/update
destroy := false
argsWithoutProg := os.Args[1:]
if len(argsWithoutProg) > 0 {
 if argsWithoutProg[0] == "destroy" {
 destroy = true
 }
}

Next, the program reads some configuration data from a YAML configuration file; this configuration data includes things like the relative file system location of the Pulumi programs that this Automation API program is going to execute:

// Get configuration data
data, err := os.ReadFile("config.yaml")
if err != nil {
 fmt.Printf("Error reading configuration file: %v\n", err)
 os.Exit(1)
}
var env Environment
err = yaml.Unmarshal(data, &env)
if err != nil {
 fmt.Printf("Error parsing configuration information: %v\n", err)
 os.Exit(1)
}

Separating configuration from code in this manner allows you to use the language’s existing ability to create binary executables that can be easily distributed to your teams. For example, Zephyr has the option of running go build to create a binary executable for distribution to internal teams.

From there, the Automation API program gets into setting up the stacks it will use:

// Set up base stack
baseStackName := auto.FullyQualifiedStackName(env.Organization, env.BaseProject.Name, env.StackName)
baseStack, err := auto.UpsertStackLocalSource(ctx, baseStackName, env.BaseProject.Location)
if err != nil {
 fmt.Printf("Failed to create or select %s stack: %v\n", env.BaseProject.Nickname, err)
 os.Exit(1)
}
fmt.Printf("Successfully created/selected %s stack\n", env.BaseProject.Nickname)

In this particular program, Zephyr decided to start with a local workspace. (You can read more about workspaces in the Pulumi Automation API docs.) This is reflected in the use of the UpsertStackLocalSource method shown above, which will create or select a stack for a Pulumi program found in a local filesystem location.

Users can also use UpsertStackInlineSource, which embeds the Pulumi program into the Automation API program as a function. There are trade-offs to each approach:

Using local source (i.e., the Pulumi program is stored outside of the Automation API program) has the benefit of allowing users to independently revise the Pulumi program or the Automation API program; this may be the ideal approach where organizations have separate teams that will create, manage, or maintain the Automation API programs. In Zephyr’s case, where there is a platform team, this made a lot of sense for them. This does also mean, as you might guess, that the Pulumi programs must be reachable by the Automation API program (this typically means users need to clone down the appropriate repositories to their system first).
Using inline source (where the Pulumi program is embedded in the Automation API program) may be a better fit when there is a single team managing or maintaining both the Pulumi program(s) as well as the Automation API program(s), or when—for whatever reason—these programs should not be revised independently of one another. This has the benefit of removing the need for users to clone or copy separate Pulumi programs onto their system before being able to use the Automation API program.

After all four stacks—base, platform, data, and app—have been set up for use within the Pulumi Automation API program, then comes the “real work”: performing operations on those stacks. To reduce duplicate code, the Zephyr program makes use of three functions:

The refreshStack function (starting on line 199 of main.go) performs the equivalent of a pulumi refresh against a stack.
The updateStack function (starting on line 218 of main.go) performs an update against the stack. This is like running pulumi up against a stack.
Finally, the deleteStack function (starting on line 236 of main.go) destroys the resources in a stack. This equates to running pulumi destroy against a stack.

Using these functions, lines 94 through 197 perform the key task of taking action to stand up or tear down the Zephyr online store and all associated infrastructure. If the user passes the parameter “destroy” on the command line (meaning the Pulumi Automation API program is invoked using go run main.go destroy), then it will tear down the stacks. Otherwise, the Pulumi Automation API program will perform a refresh and then an update—much the same way that the pulumi CLI itself behaves.

Along the way, you can see the Pulumi Automation API program passing configuration values to the stacks being orchestrated (this is lines 177 through 182, for example):

// Set config values for app stack
appStack.SetConfig(ctx, "platformOrgName", auto.ConfigValue{Value: env.Organization, Secret: false})
appStack.SetConfig(ctx, "dataOrgName", auto.ConfigValue{Value: env.Organization, Secret: false})
appStack.SetConfig(ctx, "platformProjName", auto.ConfigValue{Value: env.PlatformProject.Name, Secret: false})
appStack.SetConfig(ctx, "dataProjName", auto.ConfigValue{Value: env.DataProject.Name, Secret: false})
appStack.SetConfig(ctx, "platformStackName", auto.ConfigValue{Value: env.StackName, Secret: false})
appStack.SetConfig(ctx, "dataStackName", auto.ConfigValue{Value: env.StackName, Secret: false})

Among other things, this means you can take your existing Pulumi programs, and—without making any changes—orchestrate them using an Automation API program! Further, in reviewing the code shown above, you can see that the Pulumi Automation API ties into Pulumi’s secrets functionality.

In the end, using Pulumi’s Automation API enabled Zephyr’s platform team to simplify the workflow for standing up the online store application and all of its associated infrastructure into a simple and straightforward process:

Developers ensure they have cloned the appropriate source repositories onto their system.
Developers run go run main.go (or go run main.go destroy) to stand up or tear down the online store and all associated infrastructure.

Although Zephyr uses multiple GitHub repositories, the Pulumi Automation API has no requirements for multiple repositories. You could use a monorepo, if that better suits your particular requirements.

Summarizing Best Practices: Automation API

This post discussed the following best practices when using Pulumi Automation API:

Use the Automation API to orchestrate operations on multiple stacks. In cases where you’ve split Pulumi projects (per the recommendations in the post on structuring Pulumi projects), you’ve introduced a potential opportunity for using Automation API to orchestrate operations. This is a great counter-balance to offset the relative increase in complexity that comes from breaking large Pulumi projects into smaller, more focused Pulumi projects.
Separate configuration from code. This is already a generally-accepted recommendation in most programming circles, and it applies to Automation API programs as well. Doing so enables you, if you wish, to distribute binary artifacts to teams instead of requiring them to execute the source code of your Automation API programs.
Use local source to decouple Automation API programs from Pulumi programs when separate teams are involved. When separate teams are creating, managing, or maintaining the Automation API programs and the Pulumi programs, using local source instead of inline source enables the teams to operate somewhat independently.
When there is a single team managing both Automation API programs and Pulumi programs, use inline source. This is also true when, perhaps due to compliance or regulatory concerns, the Automation API program should be more tightly coupled to the Pulumi programs it is automating.

The Zephyr series wraps up in the next post, where we summarize all our recommendations so far. We’ll also take a look at areas of potential future growth for the team at Zephyr. Don’t miss it!

The Easier Way to Create Pulumi Providers in Go

Kyle Dixler — Thu, 19 Jan 2023 00:00:00 +0000

We are excited to announce that we’ve updated the Pulumi Provider Boilerplate to make custom provider implementation easier!

This major update brings a wealth of usability improvements to the Pulumi Provider Boilerplate by incorporating our brand new Pulumi Go Provider SDK.

Note: This update will not have any impact on existing providers that are utilizing an older version of the repository.

Background

The Pulumi Provider Boilerplate (view on GitHub) is a template repository that contains a minimal example of a working provider and has served as a starting point for writing your own provider.

For those who may not be familiar with provider authoring, it involves three distinct parts:

Package schema definition
Provider implementation
SDK generation

Providers are implemented as gRPC servers that handle requests to perform CRUD (create, read, update, delete) operations on resources on behalf of the Pulumi engine. SDK generation, on the other hand, involves generating source code libraries for a Pulumi program to utilize the provided resources.

In the past, implementing a provider using the old provider boilerplate could be difficult because you had to implement:

the provider’s schema
the CRUD operations on a provider’s resources
the gRPC server
SDK code generation to consume these resources in a Pulumi program

Now, using our code-first approach, you can implement your provider in Go by defining the provider’s metadata, resources, and functions, and then compiling the provider binary. The library takes care of the provider schema and gRPC server for you and the Pulumi CLI takes care of the SDK code generation with the following command:

pulumi package gen-sdk <path/to/provider-binary>

You can leave the heavy lifting to us and focus on the implementation details that matter to your organization.

Boilerplate code

Note: The completed boilerplate is attached at the bottom of this document and in the pulumi-provider-boilerplate repository (view on GitHub).

Provider Entrypoint

The code below uses the new pulumi-go-provider v1 APIs. Make sure you are using the latest version of github.com/pulumi/pulumi-go-provider.

The pulumi provider boilerplate program is currently quite short and the main entrypoint is as follows.

package main

import (
 "context"
 "log"

 "github.com/pulumi/pulumi-go-provider/infer"
)

func main() {
 prov, err := infer.NewProviderBuilder().
 WithNamespace("my-team-name").
 WithResources(
 // We tell the provider what resources it needs to support.
 // In this case, a single custom resource.
 infer.Resource(Random{}),
 ).
 Build()
 if err != nil {
 log.Fatal(err.Error())
 }

 _ = prov.Run(context.Background(), "xyz", "v0.0.1")
}

Resource Implementation

Resources consist of arguments, state, and CRUD operations to control them.

// Each resource has in input struct, defining what arguments it accepts.
type RandomArgs struct {
 // Fields projected into Pulumi must be public and hava a `pulumi:"..."` tag.
 // The pulumi tag doesn't need to match the field name, but its generally a
 // good idea.
 Length int `pulumi:"length"`
}

// Each resource has a state, describing the fields that exist on the created resource.
type RandomState struct {
 // It is generally a good idea to embed args in outputs, but it isn't strictly necessary.
 RandomArgs
 // Here we define a required output called result.
 Result string `pulumi:"result"`
}

// Each resource has a controlling struct.
type Random struct{}

Resource behavior is determined by implementing methods on the controlling struct.

// All resources must implement Create at a minumum.
func (Random) Create(ctx p.Context, name string, input RandomArgs, preview bool) (string, RandomState, error) {
 state := RandomState{RandomArgs: input}
 if preview {
 return name, state, nil
 }
 state.Result = makeRandom(input.Length)
 return name, state, nil
}

The Create method is mandatory, but other methods are optional.

Check: Remap inputs before they are typed.
Diff: Change how instances of a resource are compared.
Update: Mutate a resource in place.
Read: Get the state of a resource from the backing provider.
Delete: Custom logic when the resource is deleted.
Annotate: Describe fields and set defaults for a resource.
WireDependencies: Control how outputs and secrets flows through values.

Wrapping up

We encourage everyone to try the new authoring experience, including devs who may have previously found it challenging in comparison to the ease of program authorship.

Additional Example: Command provider

If you want a more involved example, the command provider (view on GitHub) has been rewritten to use the Pulumi Go Provider library entirely and is heavily commented. You can actually compare the project before and after using it to see how it has streamlined provider implementation.

Completed example

The complete boilerplate provider looks as follows:

package main

import (
 "context"
 "log"

 "github.com/pulumi/pulumi-go-provider/infer"
)

// Version is initialized by the Go linker to contain the semver of this build.
var Version string

func main() {
 prov, err := infer.NewProviderBuilder().
 WithNamespace("my-team-name").
 WithResources(
 // We tell the provider what resources it needs to support.
 // In this case, a single custom resource.
 infer.Resource(Random{}),
 ).
 Build()
 if err != nil {
 log.Fatal(err.Error())
 }

 _ = prov.Run(context.Background(), "xyz", Version)
}

// Each resource has a controlling struct.
// Resource behavior is determined by implementing methods on the controlling struct.
// The `Create` method is mandatory, but other methods are optional.
// - Check: Remap inputs before they are typed.
// - Diff: Change how instances of a resource are compared.
// - Update: Mutate a resource in place.
// - Read: Get the state of a resource from the backing provider.
// - Delete: Custom logic when the resource is deleted.
// - Annotate: Describe fields and set defaults for a resource.
// - WireDependencies: Control how outputs and secrets flows through values.
type Random struct{}

// Each resource has in input struct, defining what arguments it accepts.
type RandomArgs struct {
 // Fields projected into Pulumi must be public and hava a `pulumi:"..."` tag.
 // The pulumi tag doesn't need to match the field name, but its generally a
 // good idea.
 Length int `pulumi:"length"`
}

// Each resource has a state, describing the fields that exist on the created resource.
type RandomState struct {
 // It is generally a good idea to embed args in outputs, but it isn't strictly necessary.
 RandomArgs
 // Here we define a required output called result.
 Result string `pulumi:"result"`
}

// All resources must implement Create at a minumum.
func (Random) Create(ctx p.Context, name string, input RandomArgs, preview bool) (string, RandomState, error) {
 state := RandomState{RandomArgs: input}
 if preview {
 return name, state, nil
 }
 state.Result = makeRandom(input.Length)
 return name, state, nil
}

func makeRandom(length int) string {
 seededRand := rand.New(rand.NewSource(time.Now().UnixNano()))
 charset := []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789")

 result := make([]rune, length)
 for i := range result {
 result[i] = charset[seededRand.Intn(len(charset))]
 }
 return string(result)
}

How we manage GitHub at Pulumi with Pulumi

Guinevere Saenger — Tue, 22 Mar 2022 11:29:33 -0700

We recently updated this article to reflect that parentTeamId of the Team resource accepts a string instead of an int since v5.9.1 of the Pulumi Github package.

Ah, GitHub. The home of all developers. The place where we share code. The world’s most awkward social media site. The secret LinkedIn for techies. The tool we use for company org structure, work planning, code ownership, and permissions…

Wait.

That’s quite a lot.

GitHub is good at many things, but a full-on organization management tool it is not.

Have you ever needed your manager to manually enable admin permission on a repo for you? Or have you needed to page the CEO to add you to a team, because your manager was out that day? Have you ever wondered who is on what team? Or which team owns a repo? What if you change teams, or a team changes names? A reorg happens, and the “platform-integrations” team is no more, but we still need to call it that on GitHub because it is the team with all the repository accesses?

When I joined Pulumi in 2021, all of the above happened to me within my first few weeks.

We at Pulumi wanted to reduce this kind of management friction, and we decided to solve it the Pulumi way: with declarative infrastructure using the Pulumi GitHub provider.

The Goal

First, we set a few ground rules. We decided that administrative privileges over Pulumi’s GitHub repositories should flow as follows:

An employee is part of one or more GitHub teams.
Teams have appropriately scoped permissions over each repository they own.
Any Pulumi employee should be able to request code repository access in an official way.

Additionally, any current team membership and repository access should not be disrupted during development of this tool.

Getting Started

Create a new Pulumi program:

$ pulumi new go
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (team-management)
project description: (A minimal Go Pulumi program)
Created project 'team-management'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev) pulumi/prod
Created stack 'prod'

Installing dependencies...

go: downloading github.com/pulumi/pulumi/sdk/v3 v3.26.1
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run 'pulumi up'

$ ls
Pulumi.yaml go.mod go.sum main.go

We now have a pulumi project YAML configuration file and the beginnings of a small Go program all set up.

According to the provider configuration documentation, we need to add a properly scoped token, as well as set “pulumi” as our GitHub organization owner.

$ export GITHUB_TOKEN=YYYYYYYYYYYYYY
$ export GITHUB_OWNER=pulumi

Capturing Current State of Resources

Now, Pulumi is great at creating new infrastructure from scratch via code. But this wasn’t a from-scratch situation. We had to migrate existing resources—GitHub teams—to Pulumi, without disrupting anyone’s access.

Enter Pulumi Import.

What Pulumi Import does, in a nutshell, is find existing infrastructure by unique ID (in the GitHub provider’s case, the team ID), and add them to a Pulumi Stack. You can find the specific import instructions on the registry documentation for each resource. In this case we want the GitHub import instructions.

$ pulumi import github:index/team:Team animals 1234567
Previewing import (pulumi/prod)

 Type Name Plan
 pulumi:pulumi:Stack team-mgmt-prod
 = └─ github:index:Team animals import

Resources:
 = 1 to import
 12 unchanged

Do you want to perform this import? yes
Importing (pulumi/prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 = └─ github:index:Team animals imported

Resources:
 = 1 imported
 1 unchanged

Now that we have imported the Team resource, it is part of our Pulumi Stack. But it is not part of our main.go program yet. Helpfully, Pulumi Import replies with sample code on how to add the imported resource to your Pulumi program, in the language you selected for your project:

Please copy the following code into your Pulumi application. Not doing so will cause Pulumi to report that an update will happen on the next update command.

Please note that the imported resources are marked as protected. To destroy them you will need to remove the `protect` option and run `pulumi update` *before* the destroy will take effect.


package main

import (
 "github.com/pulumi/pulumi-github/sdk/v4/go/github"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := github.NewTeam(ctx, "animals", &github.TeamArgs{
 CreateDefaultMaintainer: pulumi.Bool(false),
 Description: pulumi.String("should get imported"),
 Name: pulumi.String("animals"),
 Privacy: pulumi.String("closed"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

After adding the team to main.go as suggested, this is the output of pulumi preview:

Previewing update (pulumi/prod)

 Type Name Plan Info
 pulumi:pulumi:Stack team-mgmt-prod

Resources:
 2 unchanged

Since pulumi preview shows no changes, we now know that our code reflects the existing infrastructure. It’s a bit funny to think about your program working well when it does nothing, but this was a huge first step in preserving existing infrastructure and ensuring all of our coworkers could continue their daily work uninterrupted!

To finish up, we unprotect the resource:

$ pulumi state unprotect 'urn:pulumi:prod::team-mgmt::github:index/team:Team::animals'

Creating Configuration

Next, we import the remaining teams and refactor the code to handle multiple teams at once. This is also the time to write our organization structure into a YAML configuration file.

---
org: "pulumi-demo--org"
teams:
 - name: "animals"
 slug: "animals"
 description: "A team in charge of animals"
 - name: "newteam"
 slug: "newteam"
 description: "I am a new team"

We then read this information into a Go struct with the necessary properties to pass into the Pulumi provider.

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 Slug string `yaml:"slug"`
}
type Organization struct {
 Org string `yaml:"org"`
 Teams []Team `yaml:"teams"`
}

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

//import accurate team info from yaml
orgFilePath, err := filepath.Abs("./org.yaml")
if err != nil {
 return err
}
yamlFile, err := ioutil.ReadFile(orgFilePath)
if err != nil {
 return err
}
var org Organization
err = yaml.Unmarshal(yamlFile, &org)
if err != nil {
 return err
}

for _, team := range org.Teams {
 ghTeam, err := github.NewTeam(ctx, team.Slug, &github.TeamArgs{
 Description: pulumi.String(team.Description),
 Name: pulumi.String(team.Name),
 Privacy: pulumi.String("closed"),
}, pulumi.Protect(false))
if err != nil {
fmt.Println("encountered error creating new Pulumi GitHub team: ", team.Name)
 return err
}
 return nil
 })
}

Notice how the GitHub provider allows us to use team names to create teams. In contrast, team IDs are only needed for importing to the Stack, not for maintaining state.

Relationships Are Hard

The next step involved a lot of thinking about team memberships and team nesting. GitHub allows teams to be nested. This groups teams by area of responsibility but also allows for certain properties, such as permissions, to be inherited by subteams.

Parent Team Parent
 |
 -------------------------
 | |
Child Teams Child1 Child2
(inherit all permissions)

To establish this relationship, we set a ParentTeamId on child teams. First off, we add a new field to our Teams struct:

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 Slug string `yaml:"slug"`
 ParentTeamId int
 Teams []Team `yaml:"teams"`
}

Here is where things get a little tricky. Any GitHub Team can have subteams, but not every Team has a parent Team. We also did not want to have to hardcode GitHub Team IDs into our configuration files. We needed to meet the following requirements:

Create a parent team, using a team name.
Obtain its team ID.
Create any child teams, also using a team name.
Write the parent team’s ID into the ParentTeamId field of each child team.
Do all of the above in a single pulumi up.

This is where maintaining infrastructure with Pulumi truly shines. In our code, we can use Pulumi Apply to hold on to the promise of a parent team ID, and pass this promise into the appropriate field:

func setupTeams(ctx *pulumi.Context, parentTeam *Team) error {

// set up parent team, i.e. engineering
 ghParentTeam, err := github.NewTeam(ctx, parentTeam.Slug, &github.TeamArgs{
 Description: pulumi.String(parentTeam.Description),
 Name: pulumi.String(parentTeam.Name),
 Privacy: pulumi.String("closed"),
 }, pulumi.Protect(false))
 if err != nil {
 fmt.Println("encountered error creating new Pulumi github team: ", parentTeam.Name)
 return err
 }

//set up nested teams
 for _, childTeam := range parentTeam.Teams {
 // set each child team's parent team ID to the current team ID
 ghChildTeam, err := github.NewTeam(ctx, childTeam.Slug, &github.TeamArgs{
 Description: pulumi.String(childTeam.Description),
 Name: pulumi.String(childTeam.Name),
 Privacy: pulumi.String("closed"),
 ParentTeamId: ghParentTeam.ID(),
 }, pulumi.Protect(false))
 if err != nil {
 fmt.Println("encountered error creating new Pulumi github team: ", childTeam.Name)
 return err
 }
 }
 return nil
}

Running this as part of main.go will result in beautifully nested teams on the GitHub UI. But with Pulumi, we can do even better. We can set pulumi.Parent() on the child teams:

for _, childTeam := range parentTeam.Teams {
 ghChildTeam, err := github.NewTeam(ctx, childTeam.Slug, &github.TeamArgs{#...},
 pulumi.Protect(false),
 pulumi.Parent(ghParentTeam))
 #...
}

With this, we can see the relationship between the resources on pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + └─ github:index:Team newteam created
 + └─ github:index:Team newchildteam created
Resources:
 + 2 created
2 unchanged

Which will be reflected on the Pulumi Stack’s Graph View in the Pulumi Service:

Adding this relationship to Pulumi is mostly a visual nicety in our case; however, certain Pulumi properties get inherited in a helpful way. Note that on GitHub, it is possible to create multiple layers of teams, whereas the example code in this blog post only shows a single layer of subteam structure. We are small enough at Pulumi to not need more, but you may want to expand on the solution if you do.

Adding People

Next, teams should have members! Let’s add them to the YAML config:

org: "pulumi-demo-org"
teams:
 - name: "animals"
 slug: "animals"
 description: "A team in charge of animals"
 members:
 - username: "platypus"
 - username: "owlcat"

A TeamMembership in GitHub is a cross reference between a Team and a User. Fortunately these do not need to be explicitly imported, as they are merely establishing relationships between GitHub Users and Teams. We can add TeamMemberships to the Stack with a Members struct, a new Members[] field on the Team struct, and an extra function. Again, Pulumi lets us use the promised output of the team ID to set the TeamId field in the TeamMembership:

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 ParentTeamId int
 Teams []Team `yaml:"teams"`
 Slug string `yaml:"slug"`
 Members []Member `yaml:"members"`
}

type Member struct {
 UserName string `yaml:"username"`
 Role string `yaml:"role"`
}

#...

func addMembers(ctx *pulumi.Context, members []Member, team *github.Team, teamName string) error {
 for _, member := range members {
 // unique name for TeamMembership
 utmName := teamName + "-" + member.UserName
 _, err := github.NewTeamMembership(ctx, utmName, &github.TeamMembershipArgs{
 TeamId: team.ID(),
 Username: pulumi.String(member.UserName),
 })
 if err != nil {
 return err
 }
 }
 return nil
}

Running pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + ├─ github:index:TeamMembership animals-owlcat created
 + └─ github:index:TeamMembership animals-platypus created

Resources:
 + 2 created
 4 unchanged

Alright! So now we have

Imported existing teams to a Pulumi Stack to track and organize all GitHub teams from a central repository, and
Added team members to teams.

That’s pretty great so far! While there are many org chart tools, what makes this one useful to us is that we can declare the desired org state in a config file, and then let Pulumi figure out what changes should be applied. Let’s automate that!

Add CI

In order for this tool to be used by everyone, we keep code and configuration in a GitHub repository. We can use Pulumi’s GitHub Action to run a pulumi preview on a pull request, and a pulumi up on merge to main.

Here’s what that looks like on the pull request:

name: Pull Request preview
on:
 pull_request:
 workflow_dispatch:

env:
 AWS_REGION: us-west-2
 GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_OWNER: pulumi

jobs:
 pull-request:
 runs-on: ubuntu-latest
 name: Preview
 steps:
 - uses: actions/checkout@v2
 - name: Install Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.17
 - name: Build Binary
 run: |
 go mod tidy
 go build main.go
 - name: Pulumi Preview
 uses: pulumi/actions@v3
 with:
 command: preview
 refresh: true
 stack-name: pulumi/prod
 comment-on-pr: true

And the pulumi up on merge to main:

name: Update
on:
 push:
 branches:
 - "main"
 workflow_dispatch:

env:
 AWS_REGION: us-west-2
 GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_OWNER: pulumi

jobs:
 main:
 runs-on: ubuntu-latest
 name: Update
 steps:
 - uses: actions/checkout@v2
 - name: Install Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.17
 - name: Build Binary
 run: |
 go mod tidy
 go build main.go
 - name: Pulumi update
 uses: pulumi/actions@v3
 with:
 command: up
 refresh: true
 stack-name: pulumi/prod

Note that we are calling refresh: true in both Workflows, which uses Pulumi Refresh to make sure that the existing GitHub resources are aligned with the resource state in our Stack.

Now, anyone with access to the GitHub management repo can:

Create, re-parent, delete, or rename teams and re-create any memberships via pull request to the config file.
Add and remove team members via pull request.
Audit and explicitly maintain org structure via git history and review processes.

But of course… there’s more!

Managing Permissions

The next step involved a lot of thinking about repository permissions and permission access.

Shortly after I joined Pulumi, my team was combined with another team. They brought along all of their repositories—none of which I had access to. Moreover, all of our team names were outdated. I could rename the teams using pulumi-github… but I could not transfer all of the repository access we all needed to our new team. Yet.

On GitHub, parent teams pass permissions down to child teams. But what if we wanted permissions to be more granular?

Here’s where TeamRepositories come in. A TeamRepository has three fields: a team ID, a repository name, and a permission. Just like TeamMemberships, this resource is a link between a Team and a Repository and does not need to be imported.

We gave each team their own config file with permission levels listing repository names:

team: "animals"
permissions:
 - role: "admin"
 repos:
 - "capybaras"
 - "owls"
 - role: "maintain"
 repos:
 - role: "push"
 repos:
 - role: "read"
 repos:

This can be read into our Pulumi program as follows:

// TeamPermissions describes a github team and the levels of permissions available (i.e. admin, maintain)
type TeamPermissions struct {
 TeamName string `yaml:"team-name"`
 Permissions []Permission `yaml:"permissions"`
}

// Permission is a single permission and a list of repos with that permission.
type Permission struct {
 Role string `yaml:"role"`
 Repos []string `yaml:"repos"`
}

#...

func setupTeamRepos(ctx *pulumi.Context, team *github.Team, teamName string) error {

 repoFilePath, err := filepath.Abs("./team-repos/" + teamName + ".yaml")
 if err != nil {
 fmt.Println(err.Error())
 return err
 }
 yamlFile, err := ioutil.ReadFile(repoFilePath)
 if err != nil {
 return err
 }

 var teamPerms TeamPermissions
 err = yaml.Unmarshal(yamlFile, &teamPerms)
 if err != nil {
 return err
 }
 for _, permission := range teamPerms.Permissions {
 for _, repo := range permission.Repos {
 _, err := github.NewTeamRepository(ctx, fmt.Sprintf("%s-%s-%s", repo, teamName, permission.Role), &github.TeamRepositoryArgs{
 Permission: pulumi.String(permission.Role),
 Repository: pulumi.String(repo),
 TeamId: team.ID(),
 })

 if err != nil {
 return err
 }
 }
 }
 return nil
}

And the output of running pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + ├─ github:index:TeamRepository owls-animals-admin created
 + ├─ github:index:TeamRepository capybaras-animals-admin created


Resources:
 + 2 created
 2 changes. 4 unchanged

With this addition, we can now combine teams with zero access disruptions as follows:

Create the new team in the org config.
Create a TeamRepository config for the new team with desired permission levels.
Open a pull request and let CI and Pulumi do the rest.

Now we can develop standards around repository permissions based on teams’ roles and areas of responsibility, and we have tooling in place that can maintain these standards for everyone in the org to see. In fact, we can standardize the meaning of “code ownership” via access levels in this way—one centralized management location rather than asking your grandboss to dig through the UI for you.

Future challenges

We want to automate some of these steps: adding new teammates to the org should also automatically add them to this repository so they can self-serve their onboarding and see Pulumi work in action!

To prevent state drift, we will run a regular reconciliation job in CI.

Of course, we will add other Pulumi providers on top of this org structure so we can sync our other tooling.

Create Amazon EKS clusters in your favorite language

Levi Blackstone — Thu, 04 Feb 2021 00:00:00 +0000

Check out version 3.0 of the Pulumi EKS Provider.

Pulumi’s infrastructure as code tooling combines the programming languages and tools you already know with the full power of cloud infrastructure. But until now, some Pulumi components for cloud infrastructure, like our popular EKS package for Amazon’s Elastic Kubernetes Service, were only available in a subset of the languages supported by Pulumi.

Now, you can use the EKS package–previously only available for TypeScript–in all four Pulumi languages: TypeScript, Python, .NET, and Go. Regardless of the language you choose, you can manage EKS clusters with Pulumi, starting with the v0.22.0 release. Check out our Modern Infrastructure Wednesday video to see it in action:

Making components available in every language

Pulumi’s EKS package is made up of Pulumi components. A component is a collection of Pulumi resources that provides an easy way to package best practices for using cloud resources, even if those best pratices span individual resources or even entire cloud providers. Until now though, components could only be used in Pulumi programs written in the same language as the component.

To change that, we’re creating “multi-language components”. With multi-language components, anyone will be able to author a component in one language and automatically compile it to all other languages supported by Pulumi. This means anyone in the worldwide Pulumi community will be able to take advantage of any component, even if it’s written in a different language. Right now, unless a component author is willing to write code in all four Pulumi languages, their component is only able to reach Pulumi users using the same language SDK.

In addition to expanding the library of publicly available components, this work will also enable organizations to write Pulumi components in the right programming language, or languages, for each team and each product. Infrastructure or platform teams will be able to write components in Go and share them with a webapp team using TypeScript, write components in .NET and share them with a machine learning team using Python, and any other combination that suits their business.

The next step in the story is a seamless authoring experience that enables component creators to write, build, and publish multi-language components. That will be in public preview soon, and we’re excited for you to try it out. If you’d like, subscribe to the multi-language components issue for updates.

Next steps

If you haven’t yet, try out the new pulumi-eks package with this walkthrough.

Announcing Cross-Language Enum Support

Komal Ali — Tue, 15 Dec 2020 00:00:00 +0000

Here at Pulumi, we believe in leveraging the best features of programming languages to create a delightful development experience for our users. Today, we continue our contributions in this area by announcing cross-language support for enum types in our provider SDKs, available in all Pulumi languages - Python, TypeScript, .NET and Go.

Enum types explained

An enum (short for enumerated) type is a data type consisting of a set of named values. In the Pulumi resource model, a property is an enum type when its value can only be set to a finite number of predetermined values.

For instance, consider a simple S3 bucket.

import * as pulumi from "@pulumi/pulumi";
import { s3 } from "@pulumi/aws";

const myBucket = new s3.Bucket("myBucket", { acl: "private" });

import pulumi
from pulumi_aws import s3

my_bucket = s3.Bucket("myBucket", acl="private")

using Pulumi;
using S3 = Pulumi.Aws.S3;

class MyStack : Stack
{
 public MyStack()
 {
 var myBucket = new S3.Bucket("myBucket", new S3.BucketArgs
 {
 Acl = "private",
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v3/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 myBucket, err := s3.NewBucket(ctx, "myBucket", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

In the above code, the S3 Bucket resource has a property called acl, where we pass in the string private to indicate that this is a private bucket.

If we look at the resource docs, we can see that the acl property can only be set to one of a few different values: private, public-read, public-read-write, aws-exec-read, authenticated-read, and log-delivery-write. The acl property is the perfect candidate for an enum type and is emitted as one, so you can use the following code instead.

import * as pulumi from "@pulumi/pulumi";
import { s3 } from "@pulumi/aws";

const myBucket = new s3.Bucket("myBucket", { acl: s3.CannedAcl.Private });

import pulumi
from pulumi_aws import s3

my_bucket = s3.Bucket("myBucket", acl=s3.CannedAcl.PRIVATE)

using Pulumi;
using S3 = Pulumi.Aws.S3;

class MyStack : Stack
{
 public MyStack()
 {
 var myBucket = new S3.Bucket("myBucket", new S3.BucketArgs
 {
 Acl = S3.CannedAcl.Private,
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v3/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 myBucket, err := s3.NewBucket(ctx, "myBucket", &s3.BucketArgs{
 Acl: s3.CannedAclPrivate,
 })
 if err != nil {
 return err
 }
 return nil
 })
}

IDE Superpowers

Enum properties provide discoverable and normalized constants that can be used in place of raw strings, allowing you to move faster with the added boosts of IDE tooling like type hints and autocomplete.

Using the provided constants, you can avoid referring back to the documentation to remember the valid values and save precious moments in the development cycle that would be lost to debugging errors caused by typos.

Optimized for flexibility

While some properties make sense as “strict” enums (i.e., the input value must be one of the enumerated values), there are times when it would be beneficial to have constants for commonly used values without restricting the input to only those values. We call this second category “relaxed” enums, and we model them a little differently in our SDKs.

“Strict” enums

A property is a “strict” enum when the input value must be one of the enumerated values. In this case, the property type is specified as the enum type.

We will use “strict” enums when we are sure that the enum will include all legal values, such as when a provider is auto-generated from a cloud provider specification (like our Azure-Native or Kubernetes providers).

export class StorageAccount extends pulumi.CustomResource {
 constructor(name: string, args: StorageAccountArgs, opts?: pulumi.CustomResourceOptions) {...}
}

export interface StorageAccountArgs {
 readonly accessTier?: pulumi.Input<enums.storage.latest.AccessTier>;
 ...
}

class StorageAccount(pulumi.CustomResource):
 def __init__(__self__,
 resource_name: str,
 access_tier: Optional[pulumi.Input['AccessTier']] = None,
 ...):
 ...

namespace Pulumi.AzureNative.Storage
{
 public partial class StorageAccount : Pulumi.CustomResource
 {
 public StorageAccount(string name, StorageAccountArgs args, ...)
 {
 }
 }

 public sealed class StorageAccountArgs : Pulumi.ResourceArgs
 {
 [Input("accessTier")]
 public Input<Pulumi.AzureNative.Storage.AccessTier>? AccessTier { get; set; }

 ...
 }
}

func NewStorageAccount(ctx *pulumi.Context,
name string, args *StorageAccountArgs, ...) (*StorageAccount, error) {
 ...
}

type StorageAccountArgs struct {
 AccessTier AccessTier
 ...
}

When using TypeScript, “strict” enums will accept both the constant (AccessTier.Cool) or the literal string ("Cool"). If there is an accidental spelling error in the literal, you are immediately alerted to the issue rather than having to wait until runtime.

“Relaxed” enums

When a property is a “relaxed” enum, the property type is specified as the Union of the enum type and the underlying primitive type. This means that you have the convenience of using the enum constants (i.e., s3.CannedAcl.Private), but you may also pass in the raw string (i.e., "private").

In the AWS provider (and other Terraform-based providers), we have opted for only using “relaxed” enums. The reasoning is twofold. Allowing the primitive type maintains backward compatibility and also allows users to use values that may not yet be represented in the Pulumi schema (e.g., a new Managed Policy ARN, EC2 Instance type, etc.).

export class Bucket extends pulumi.CustomResource {
 constructor(name: string, args?: BucketArgs, opts?: pulumi.CustomResourceOptions) {...}
}

export interface BucketArgs {
 readonly acl?: pulumi.Input<string | enums.s3.CannedAcl>;
 ...
}

class Bucket(pulumi.CustomResource):
 def __init__(__self__,
 resource_name: str,
 acl: Optional[pulumi.Input[Union[str, 'CannedAcl']]] = None,
 ...):
 ...

namespace Pulumi.Aws.S3
{
 public partial class Bucket : Pulumi.CustomResource
 {
 public Bucket(string name, BucketArgs? args = null, ...)
 {
 }
 }

 public sealed class BucketArgs : Pulumi.ResourceArgs
 {
 [Input("acl")]
 public InputUnion<string, Pulumi.Aws.S3.CannedAcl>? Acl { get; set; }

 ...
 }
}

func NewBucket(ctx *pulumi.Context,
name string, args *BucketArgs, ...) (*Bucket, error) {
 ...
}

// In Go, relaxed enums retain the type of the underlying primitive.
// However, the CannedAcl constants may still be passed in, as they are
// of the same underlying type.
type BucketArgs struct {
 Acl pulumi.StringPtrInput
 ...
}

Try them out!

You can find enum types integrated into v3.19.0 of the AWS provider and v0.7.1 of the Azure-Native provider, and we will be adding enums to other providers in the coming weeks and months.

Azure-Native: In the native Azure provider, all properties labeled as enums in the OpenAPI spec are represented as such. In all, there are over 1300 enums provided in the SDKs. The native Azure provider uses both “strict” and “relaxed” enums since the OpenAPI specification explicitly defines its properties as such.

AWS: The AWS provider enums are manually identified and maintained as part of the provider schema. We’ve already added many that you might find useful, such as Lambda Runtimes, EC2 Instance Types, and IAM Managed Policies, and will continue to add more in the coming months.

Take the new providers for a spin and let us know what you think in the Community Slack! If there are properties that you would like to see represented as enums, let us know. Or even better, submit a PR like this one to add them to the schema!

Welcoming Go to the Pulumi Family

Evan Boyle — Tue, 19 May 2020 00:00:00 +0000

Over the last 10 years, Go has quickly become the “language of the cloud” for building application servers and services that run in and on today’s cloud platforms. With Pulumi 2.0, Go can also be used to manage and provision modern infrastructure as well. Across any cloud (AWS, Azure, GCP, Kubernetes and more than 50 others!) and across a variety of workloads (containers, serverless, kubernetes, core infrastructure and more), you can now use the rich software engineering strengths of the Go language and ecosystem to manage your cloud infrastructure. The Pulumi open source project itself has been built on Go from day 1, and so we’re really excited to bring full Go support for cloud infrastructure as code to the same language ecosystem that Pulumi itself has been part of.

What is Pulumi?

Pulumi is a modern infrastructure as code tool for the cloud. It is built around four key ideas.

Programming Languages. Pulumi offers best-in-class productivity and software engineering capabilities by allowing you to use existing rich programming languages like JavaScript, Python, .NET and of course Go. Pulumi combines this with still getting the reliability of a desired-state infrastructure as code. The best of both worlds!
Multi-Cloud. Give operators the tools to build consistent workflows across any cloud(s). Whether public, private, or hybrid — you can use the full breadth and depth of any cloud services.
Modern Application Architectures. Pulumi fully supports legacy workloads, but it is designed for the world of modern containers, serverless, and cloud-native architectures. Build the architecture that works for you by mixing and matching infrastructure components.
Cloud Engineering Teams. Pulumi is for the entire team; everyone ranging from developers, infrastructure and operations engineers, and security engineers can use Pulumi. Together, they can tackle cloud solutions together without technology or team silos.

Giving you superpowers

Pulumi gives you the software engineering superpowers of the Go programming language for working with cloud infrastructure. But what can you do with them? In this post, we’ll walk through a few vignettes of how Pulumi can be used to work with serverless, containers and Kubernetes workloads across GCP, Azure, AWS and self-managed clouds.

Serverless

Pulumi has first-class support for serverless workloads, allowing your application and infrastructure code to live side by side (and both be written in Go if you want!). Easily deploy a lambda running a Go handler on AWS:

args := &lambda.FunctionArgs{
 Handler: pulumi.String("handler"),
 Role: role.Arn,
 Runtime: pulumi.String("go1.x"),
 Code: pulumi.NewFileArchive("./handler/handler.zip"),
 }

// Create the lambda using the args.
function, err := lambda.NewFunction(
 ctx,
 "basicLambda",
 args,
 pulumi.DependsOn([]pulumi.Resource{logPolicy}),
 )
if err != nil {
 return err
 }

// Export the lambda ARN.
ctx.Export("lambda", function.Arn)

And with Pulumi’s multi-cloud approach you can do the equivalent on Google Cloud:

// Set arguments for creating the function resource.
args := &cloudfunctions.FunctionArgs{
 SourceArchiveBucket: bucket.Name,
 Runtime: pulumi.String("go111"),
 SourceArchiveObject: bucketObject.Name,
 EntryPoint: pulumi.String("Handler"),
 TriggerHttp: pulumi.Bool(true),
 AvailableMemoryMb: pulumi.Int(128),
}

// Create the function using the args.
function, err := cloudfunctions.NewFunction(ctx, "basicFunction", args)
if err != nil {
 return err
}

// Export the trigger URL.
ctx.Export("function", function.HttpsTriggerUrl)

Containers

Pulumi makes it easy to work with containers. To start with, you can build and publish your docker images as part of a Pulumi deployment. This example uses the Azure Container Registry:

// Create a registry.
registryArgs := containerservice.RegistryArgs{
 ResourceGroupName: resourceGroup.Name,
 AdminEnabled: pulumi.Bool(true),
 Sku: pulumi.String("Premium"),
}
registry, err := containerservice.NewRegistry(ctx, "registry", &registryArgs)
if err != nil {
 return err
}

// Create the docker image.
imageArgs := docker.ImageArgs{
 ImageName: pulumi.Sprintf("%s/mynodeapp:v1.0.0", registry.LoginServer),
 Build: &docker.DockerBuildArgs{
 Context: pulumi.String("./app"),
 },
 Registry: &docker.ImageRegistryArgs{
 Server: registry.LoginServer,
 Username: registry.AdminUsername,
 Password: registry.AdminPassword,
 },
}
image, err := docker.NewImage(ctx, "node-app", &imageArgs)
if err != nil {
 return err
}

You can even deploy a container into your cloud environment based on that image from within the same Pulumi program ensuring that application and infrastructure deployments are always in sync:

portArgs := containerservice.GroupContainerPortArgs{
 Port: pulumi.Int(80),
 Protocol: pulumi.String("TCP"),
}
containerArgs := containerservice.GroupContainerArgs{
 Cpu: pulumi.Float64(0.5),
 Image: image.ImageName,
 Memory: pulumi.Float64(1.5),
 Name: pulumi.String("hello-world"),
 Ports: containerservice.GroupContainerPortArray{portArgs},
}
groupArgs := containerservice.GroupArgs{
 ResourceGroupName: resourceGroup.Name,
 ImageRegistryCredentials: containerservice.GroupImageRegistryCredentialArray{credentialArgs},
 OsType: pulumi.String("Linux"),
 Containers: containerservice.GroupContainerArray{containerArgs},
 IpAddressType: pulumi.String("public"),
 DnsNameLabel: pulumi.String("acigo"),
}
group, err := containerservice.NewGroup(ctx, "aci", &groupArgs)
if err != nil {
 return err
}

ctx.Export("endpoint", group.Fqdn)

Kubernetes

Pulumi provides access not just to your cloud (AWS, Azure, GCP and more), but also to the whole Kubernetes API. The Pulumi Kubernetes provider is based on the OpenAPI specification, so is always up to date with the full functionality of the Kubernetes API. Using Pulumi to provision your Kubernetes resources gives you the declarative benefits of YAML without any of the drawbacks. Using a familiar, general-purpose language to orchestrate resources brings the benefits of strong typing, safety, and opens up options for encapsulation. Here’s a snippet from the guestbook example:

// Redis leader Service
_, err = corev1.NewService(ctx, "redis-leader", &corev1.ServiceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("redis-master"),
 Labels: redisLeaderLabels,
 },
 Spec: &corev1.ServiceSpecArgs{
 Ports: corev1.ServicePortArray{
 corev1.ServicePortArgs{
 Port: pulumi.Int(6379),
 TargetPort: pulumi.Int(6379),
 },
 },
 Selector: redisLeaderLabels,
 },
})
if err != nil {
 return err
}

With the expressiveness of go, we have the power to encapsulate services and deployments behind higher-order components:

type ServiceDeployment struct {
 pulumi.ResourceState

 FrontendIP pulumi.StringPtrOutput
 Deployment *appsv1.Deployment
 Service *corev1.Service
}

type ServiceDeploymentArgs struct {
 AllocateIPAddress pulumi.Bool
 Image pulumi.StringInput
 IsMinikube pulumi.Bool
 Ports pulumi.IntArrayInput
 Replicas pulumi.IntPtrInput
}

func NewServiceDeployment(...){...}

This allows us to simplify boilerplate and design strongly-typed APIs for our Kubernetes infrastructure (in this case, combining a Service and a Deployment into a single, simple, API with nice defaults for our use case):

_, err := NewServiceDeployment(ctx, "redis-leader", &ServiceDeploymentArgs{
 Image: pulumi.String("k8s.gcr.io/redis:e2e"),
 Ports: pulumi.IntArray{pulumi.Int(6379)},
})

Kubernetes + Infra

The power doesn’t stop with managing Kubernetes resources. You can provision cloud resources right alongside. Here we use Go to provision an EKS cluster:

// Create EKS Cluster
eksCluster, err := eks.NewCluster(ctx, "eks-cluster", &eks.ClusterArgs{
 RoleArn: pulumi.StringInput(eksRole.Arn),
 VpcConfig: &eks.ClusterVpcConfigArgs{
 PublicAccessCidrs: pulumi.StringArray{
 pulumi.String("0.0.0.0/0"),
 },
 SecurityGroupIds: pulumi.StringArray{
 clusterSg.ID().ToStringOutput(),
 },
 SubnetIds: toPulumiStringArray(subnet.Ids),
 },
})

Try Go + Pulumi Today

The magic doesn’t stop here. There’s a wealth of examples, and we’re excited to see where the community takes it. If you’d like to try out Pulumi today, you can get started with your favorite cloud here:

We’d love to hear what you think. We look forward to hanging out on Pulumi Slack or hearing from your thoughts in a GitHub issue.

Deploy Kubernetes and Applications with Go

Levi Blackstone — Tue, 14 Apr 2020 00:00:00 +0000

We’re excited that Go is now a first-class language in Pulumi and that you can build your infrastructure with Go on AWS, Azure, GCP, and many other clouds. Users often ask, “Can I use Pulumi to manage Kubernetes infrastructure in Go today?” With the release of Pulumi 2.0., the answer is “Yes!”

Building your Kubernetes infrastructure with Infrastructure as Code offers several benefits:

Strongly-typed inputs and outputs for resources and invokes
First-class language support in editors/IDEs like vim, VS Code, GoLand, and atom
Reusable components and classes: Abstract standard functionality into classes and libraries for code reuse, and clean infrastructure design, instead of copy/pasting pages of YAML.
Embed Pulumi programs within other Go-based tools

These benefits provide a productive experience for Go developers using Kubernetes.

Tour of Kubernetes with Go

Let’s start with a basic example, e.g. deploying an Nginx pod into a Kubernetes cluster.

pod, err := corev1.NewPod(ctx, "pod", &corev1.PodArgs{
 Spec: corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx"),
 }
 },
 },
})
if err != nil {
 return err
}

We can deploy this with pulumi up.

$ pulumi up
Previewing update (nginx):
Type Name Plan
+ pulumi:pulumi:Stack kubernetes-go-nginx create
+ └─ kubernetes:core:Pod pod create
Resources:
+ 2 to create
Do you want to perform this update? yes
Updating (nginx):
Type Name Status
+ pulumi:pulumi:Stack kubernetes-go-nginx created
+ └─ kubernetes:core:Pod pod created
Resources:
+ 2 created
Duration: 16s

Although programming languages are imperative, Pulumi describes the desired state of the infrastructure. If we change our program, Pulumi will compute the minimum delta and apply it to our Kubernetes cluster, which will transition to the new desired state. For example, we can modify our Kubernetes resources in place inside the cluster by adding a label to our Pod:

pod, err := corev1.NewPod(ctx, "pod", &corev1.PodArgs{
+ Metadata: &metav1.ObjectMetaArgs{
+ Labels: pulumi.StringMap{"app": pulumi.String("nginx")},
+ },
 Spec: corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx"),
 }},
 },
})
if err != nil {
 return err
}

When we update the deployment with ‘pulumi up`, the preview shows that the pod is updated in place with the new label.

$ pulumi up
Previewing update (nginx):
Type Name Plan Info
pulumi:pulumi:Stack kubernetes-go-nginx
~ └─ kubernetes:core:Pod pod update [diff: ~metadata]
Resources:
~ 1 to update
1 unchanged
Do you want to perform this update? details
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:nginx::kubernetes-go::pulumi:pulumi:Stack::kubernetes-go-nginx]
~ kubernetes:core/v1:Pod: (update)
[id=default/pod-yxmgq7q2]
[urn=urn:pulumi:nginx::kubernetes-go::kubernetes:core/v1:Pod::pod]
~ metadata: {
~ labels: {
+ app: "nginx"
}
}
Do you want to perform this update? yes
Updating (nginx):
Type Name Status Info
pulumi:pulumi:Stack kubernetes-go-nginx
~ └─ kubernetes:core:Pod pod updated [diff: ~metadata]
Resources:
~ 1 updated
1 unchanged
Duration: 13s

The benefits of Go start to shine when we extract common code into a reusable component. For example, we can create a new ServiceDeployment component that combines both a Kubernetes Service and Deployment with opinionated defaults. Our ServiceDeployment component can describe entire Kubernetes applications (100s of lines of YAML), in a short snippet of Go:

// Initialize config
conf := config.New(ctx, "")

// Minikube does not implement services of type `LoadBalancer` so
// require the user to specify if we're running on minikube. If so,
// create only services of type ClusterIP.
isMinikube := conf.GetBool("isMinikube")

// Redis leader Deployment + Service
_, err := NewServiceDeployment(ctx, "redis-master", &ServiceDeploymentArgs{
 Image: pulumi.String("k8s.gcr.io/redis:e2e"),
 Ports: pulumi.IntArray{pulumi.Int(6379)},
})
if err != nil {
 return err
}

// Redis follower Deployment + Service
_, err = NewServiceDeployment(ctx, "redis-slave", &ServiceDeploymentArgs{
 Image: pulumi.String("gcr.io/google_samples/gb-redisslave:v3"),
 Ports: pulumi.IntArray{pulumi.Int(6379)},
})
if err != nil {
 return err
}

// Frontend Deployment + Service
frontend, err := NewServiceDeployment(ctx, "frontend", &ServiceDeploymentArgs{
 AllocateIPAddress: true,
 Image: pulumi.String("gcr.io/google-samples/gb-frontend:v4"),
 IsMinikube: pulumi.Bool(isMinikube),
 Ports: pulumi.IntArray{pulumi.Int(80)},
 Replicas: pulumi.Int(3),
})
if err != nil {
 return err
}

if isMinikube {
 ctx.Export("frontendIP", frontend.Service.Spec.ApplyT(
 func(spec *corev1.ServiceSpec) *string { return spec.ClusterIP }))
} else {
 ctx.Export("frontendIP", frontend.FrontendIP)
}

The complete implementation of the ServiceDeployment component is detailed in the Guestbook example in the Pulumi Examples repo on GitHub.

Building Docker Images for Kubernetes with Go

In the previous examples, we specified the Docker image to deploy as part of our Kubernetes Deployments by referring to an image in a registry. If we wanted to push our custom Docker image that uses our application’s source code, and deploy that in our Kubernetes Pod or Deployment, we can do it with the pulumi-docker package. For example, we can deploy our custom Nginx Docker image with the following:

image, err := docker.NewImage(ctx, "node-app", &docker.ImageArgs{
 ImageName: pulumi.String("my-username/my-nginx"),
 Build: &docker.DockerBuildArgs{
 Context: pulumi.String("./app"),
 },
})
if err != nil {
 return err
}

pod, err := corev1.NewPod(ctx, "pod", &corev1.PodArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{"app": pulumi.String("nginx")},
 },
 Spec: corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: image.ImageName,
 }},
 },
})
if err != nil {
 return err
}

We can use our Dockerfile from the app folder in our Pulumi project, instead of using the default Nginx image from DockerHub. For example, adding the following in our Dockerfile deploys customized files in the app/content folder to our Nginx server:

FROM nginx
COPY content /usr/share/nginx/html

When we deploy this with Pulumi, the Dockerfile is built locally, pushed to a registry, then made available by specifying the image name in the registry referenced from the Pod in Kubernetes. Our changes happen automatically, seamlessly deploying and versioning both the application code and infrastructure together with a simple pulumi up.

We can also push to another Docker container registry (like ACR, GCR, ECR, or others) by using additional parameters on the pulumi.docker.ImageArgs class.

Cloud + Kubernetes with Go

Pulumi works with both Kubernetes and cloud providers (AWS, Azure, GCP, and more.) Within the same Pulumi program, you can build a Kubernetes cluster and then deploy applications and services into that cluster.

For example, we can deploy a managed GKE cluster, and then deploy a Pod into it:

masterVersion := "1.14.10-gke.27"
cluster, err := container.NewCluster(ctx, "demo-cluster", &container.ClusterArgs{
 InitialNodeCount: pulumi.Int(2),
 MinMasterVersion: pulumi.String(masterVersion),
 NodeVersion: pulumi.String(masterVersion),
 NodeConfig: &container.ClusterNodeConfigArgs{
 MachineType: pulumi.String("n1-standard-1"),
 OauthScopes: pulumi.StringArray{
 pulumi.String("https://www.googleapis.com/auth/compute"),
 pulumi.String("https://www.googleapis.com/auth/devstorage.read_only"),
 pulumi.String("https://www.googleapis.com/auth/logging.write"),
 pulumi.String("https://www.googleapis.com/auth/monitoring"),
 },
 },
})
if err != nil {
 return err
}

ctx.Export("kubeconfig", generateKubeconfig(
 cluster.Endpoint, cluster.Name, cluster.MasterAuth))

k8sProvider, err := providers.NewProvider(
 ctx, "k8sprovider", &providers.ProviderArgs{
 Kubeconfig: generateKubeconfig(
 cluster.Endpoint, cluster.Name, cluster.MasterAuth),
}, pulumi.DependsOn([]pulumi.Resource{cluster}))
if err != nil {
 return err
}

namespace, err := corev1.NewNamespace(ctx, "app-ns", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("demo-ns"),
 },
}, pulumi.Provider(k8sProvider))
if err != nil {
 return err
}

appLabels := pulumi.StringMap{
 "app": pulumi.String("demo-app"),
}
_, err = appsv1.NewDeployment(ctx, "app-dep", &appsv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: namespace.Metadata.Elem().Name(),
 },
 Spec: appsv1.DeploymentSpecArgs{
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: appLabels,
 },
 Replicas: pulumi.Int(3),
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: appLabels,
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 corev1.ContainerArgs{
 Name: pulumi.String("demo-app"),
 Image: pulumi.String("jocatalin/kubernetes-bootcamp:v2"),
 }},
 },
 },
 },
}, pulumi.Provider(k8sProvider))
if err != nil {
 return err
}

This example first deploys a GKE Kubernetes Cluster into GCP, then deploys Kubernetes resources into that GKE Cluster. This example combines infrastructure and application deployment in a few dozen lines of declarative and strongly typed Go code.

Check out the full GKE in Go example on GitHub.

Conclusion

Kubernetes support is one of several significant new additions made available to Pulumi’s Go community and more improvements are on the way. Get started with Kubernetes and Go today, and let us know what you think!

Modern Cloud Infrastructure in Go - The Road to 2.0

Evan Boyle — Thu, 27 Feb 2020 00:00:00 +0000

Here at Pulumi, everyone on our engineering team is a Gopher. Go has quickly become the “language of the cloud,” and so when we chose to build our open-source pulumi/pulumi engine and SaaS backend, we chose Go. As such, we are very excited to welcome Go to the family of supported infrastructure as code languages as part of Pulumi 2.0.

What is Pulumi?

Go has become the lingua franca of cloud-native infrastructure development. From projects like Docker, to Kubernetes, the community has embraced Go. The result is an ecosystem rich in tooling and packages designed for systems and cloud engineering.

Pulumi lets you use familiar languages to express your application’s infrastructure needs, using a powerful technique called “infrastructure as code.” Using infrastructure as code, you declare desired infrastructure, and an engine provisions it for you so that it’s automated, easy to replicate, and robust enough for demanding production requirements. Pulumi improves this process by leveraging general-purpose languages and making modern cloud infrastructure patterns, such as containers and serverless programs, first-class and easy.

Automatically create, update, or delete cloud resources using Pulumi’s infrastructure as code engine, removing manual point-and-clicking in the cloud provider consoles, UIs, and ad-hoc scripts.
Use your favorite IDEs and tools, including Visual Studio Code, taking advantage of features like auto-completion, refactoring, and interactive documentation.
Catch mistakes early on with standard compiler errors, and an infrastructure-specific policy engine for enforcing security, compliance, and best practices.
Reuse any existing go packages, or distribute your own, whether that’s for infrastructure best practices, productivity, or just general programming patterns.
Build scalable cloud applications using classic infrastructure cloud-native technologies. including serverless functions, and highly scalable databases such as AWS Aurora, into your core development experience, bringing them closer to your application code.

Pulumi’s free open source SDK, which includes a CLI and assortment of libraries, enables these capabilities.

Creating an AWS S3 Bucket is a familiar and simple example that demonstrates these advantages:

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an AWS resource (S3 Bucket)
 bucket, err := s3.NewBucket(ctx, "my-bucket", nil)
 if err != nil {
 return err
 }

 // Export the name of the bucket
 ctx.Export("bucketName", bucket.ID())
 return nil
 })
}

Although this code looks remarkably similar to using your cloud’s Go SDK to imperatively change infrastructure, there is a major and important difference: this is declarative infrastructure as code. This code declares to Pulumi a goal state for our infrastructure which Pulumi can then carry out. That means we can preview changes with full diffs before they are made, that we always have a history of changes we’ve made to our infrastructure, and that it works both for the initial provisioning of a new environment as well as revisions to an existing one.

An Expressive and Idiomatic Type System

Making Go a first-class citizen is a goal for the Pulumi 2.0 release. We especially wanted Go to have more idiomatic and stronger typing than interface{}.

Pulumi’s programming model is inherently asynchronous. Consider creating a new AWS SecurityGroup, and using it to provision an EC2 instance. The EC2 instance must wait for the Group to finish provisioning, and the Pulumi engine must track this dependency. While two resources, modern cloud deployments commonly manage hundreds of resources. Pulumi models these resources using promise-like wrappers referred to as Inputs and Outputs allowing us to build a dependency graph, and parallelize cloud resource provisioning where possible.

Inputs, Outputs, and functions that operate over them (All, Apply, Sprintf) must be compatible with raw type primitives (string, integer, list, map, boolean), and user-defined structs. In other words, we must support polymorphism in a language that doesn’t have generics. We hide the gritty details behind a rich, strongly-typed API that we were able to iterate on easily thanks to the utilization of code generation.

Try Pulumi + Go Today

Although Pulumi for Go is in “preview” status until 2.0 is released, it supports all of the essential Pulumi programming model features (and the rest is on its way). Our goal is to gather feedback and over the next few weeks, and work hard to improve the Go experience across the board, including more examples and better documentation. To get started check out:

GoDoc
Getting started with Go and AWS, Azure, or Google Cloud.
Author and deploy serverless functions in Go on AWS Lambda and GCP Cloud Functions.
Build a serverless container-based application on AWS ECS Fargate.
Deploy a web server on a virtual machine with AWS or Azure.
Deploy a Kubernetes cluster using Azure Kubernetes Service.

To keep up with our progress, follow Pulumi on Github, and feel free to open a new issue with ideas, feedback, and bugs. Join the community in Slack to discuss your scenarios, and to get any needed assistance from the team and other end users.

Coming Soon

Some exciting things to keep an eye out for: