Pulumi Blog: Packages

Introducing envVarMappings for Provider Credentials

Guinevere Saenger — Thu, 12 Feb 2026 00:00:00 +0000

Running multiple providers with different credentials in the same Pulumi program has always been tricky. Providers expect fixed environment variable names like AWS_ACCESS_KEY_ID or ARM_CLIENT_SECRET, so if you need two AWS providers targeting different accounts, you couldn’t configure them both via environment variables.

Pulumi v3.220.0 introduces envVarMappings, a new resource option that solves this problem by letting you remap provider environment variables to custom keys.

When configuring a Pulumi provider to authenticate against a cloud provider, there are two main options available. You can set authentication values as secrets in your Pulumi.yaml config:

$ pulumi config set azure-native:clientSecret <clientSecret> --secret

Alternately, you can also use the terminal environment of where you’re running your Pulumi commands:

$ export ARM_CLIENT_SECRET=1234567

Using environment variables in this manner is especially useful in CI environments, or when you’d rather not write that auth token to state, even encrypted. But there’s currently several use cases where this breaks down, due to the hard-coded nature of the environment variables that a given provider expects.

Multiple providers or provider instances that expect different authentication values but have the same variable key

For example, if you are using multiple explicit providers targeting different Azure accounts, you were not able to set these separate configurations via environment variable.

Instead, users would have to set these values in the provider config, which may not be desirable for all use cases. Not only does the provider config write secrets to state (albeit of course encrypted), but it can also result in a noisy diff on an otherwise no-op upgrade when token rotation is used.

Remapping environment variables

For this and similar scenarios, we have a new solution for you: setting mappings of environment variable keys on your provider. The concept is as follows:

“For any environment variable that my Pulumi provider expects, I want to be able to tell the provider to use the value of a custom-defined environment variable instead.”

Example

Let’s say your provider expects ARM_CLIENT_SECRET, but you want it to use a different value than the one set in your shell. First, define a custom environment variable with your desired value:

$ export CUSTOM_ARM_CLIENT_SECRET=7654321

Then, use envVarMappings to tell the provider: “When you look for ARM_CLIENT_SECRET, read from CUSTOM_ARM_CLIENT_SECRET instead.” The mapping format is { "SOURCE_VAR": "TARGET_VAR" }:

const provider = new command.Provider("command-provider", {}, {
 envVarMappings: {
 // If CUSTOM_ARM_CLIENT_SECRET exists, provider sees the value of ARM_CLIENT_SECRET
 "CUSTOM_ARM_CLIENT_SECRET": "ARM_CLIENT_SECRET",
 },
});

provider = command.Provider("command-provider",
 opts=pulumi.ResourceOptions(
 env_var_mappings={
 # If CUSTOM_ARM_CLIENT_SECRET exists, provider sees the value of ARM_CLIENT_SECRET
 "CUSTOM_ARM_CLIENT_SECRET": "ARM_CLIENT_SECRET",
 }
 )
)

provider, err := command.NewProvider(
 ctx,
 "command-provider",
 &command.ProviderArgs{},
 pulumi.EnvVarMappings(map[string]string{
 // If CUSTOM_ARM_CLIENT_SECRET exists, provider sees the value of ARM_CLIENT_SECRET
 "CUSTOM_ARM_CLIENT_SECRET": "ARM_CLIENT_SECRET",
 }),
)

var provider = new Command.Provider("command-provider", new Command.ProviderArgs(), new CustomResourceOptions
{
 EnvVarMappings = new Dictionary<string, string>
 {
 // If CUSTOM_ARM_CLIENT_SECRET exists, provider sees the value of ARM_CLIENT_SECRET
 { "CUSTOM_ARM_CLIENT_SECRET", "ARM_CLIENT_SECRET" }
 }
});

var provider = new Provider("command-provider", ProviderArgs.Empty, CustomResourceOptions.builder()
 .envVarMappings(Map.of(
 // If CUSTOM_ARM_CLIENT_SECRET exists, provider sees the value of ARM_CLIENT_SECRET
 "CUSTOM_ARM_CLIENT_SECRET", "ARM_CLIENT_SECRET"
 ))
 .build());

resources:
 command-provider:
 type: pulumi:providers:command
 options:
 envVarMappings:
 # If CUSTOM_ARM_CLIENT_SECRET exists, provider sees the value of ARM_CLIENT_SECRET
 CUSTOM_ARM_CLIENT_SECRET: ARM_CLIENT_SECRET

You can now customize each environment variable value your provider sees by defining a new environment variable, and then mapping your provider’s defined variable to yours. Try it out with Pulumi v3.220.0 today!

For full details, see the envVarMappings documentation.

Happy coding!

Leveling up Pulumi AI with the Pulumi Registry

Aaron Friel — Mon, 22 May 2023 00:00:00 +0000

Pulumi AI harnesses a form of generative AI, known as large language models, to help you discover, learn, and use new cloud infrastructure APIs with ease. Think of Pulumi AI as a sophisticated compass, guiding you through the ever-changing landscape of cloud infrastructure and pointing you in the direction of the most suitable solutions for your unique requirements.

In this blog post, we’ll explore our recent enhancements to Pulumi AI, focusing on how we’ve integrated Pulumi Package schema data to generate more accurate and relevant Pulumi programs. We’ll also share examples showcasing the improvements and how they can benefit your infrastructure as code journey.

Overcoming the Challenges in Pulumi AI

As powerful as Pulumi AI may seem, it was initially limited by the training cutoff of the model. Pulumi AI wasn’t aware of new providers or changes to resources after late 2021. Additionally, the limitations in generative AI made it difficult for the model to reproduce all the input properties and relevant details of the resources it was trained on from “memory.”

These factors led to a phenomenon researchers call “hallucination.” Pulumi AI would sometimes generate an API for a non-existent provider when asked to produce a Pulumi program. To overcome these challenges, we turned to the rich Pulumi Package schema data and CrossCode technology that powers the Pulumi Registry docs, our code conversion tools, and other Pulumi enhancements.

Integrating Pulumi Package Schema Data

Think of the Pulumi Package schema as a treasure trove of information about our providers. It’s a well-organized, comprehensive library that holds the keys to unlocking the full potential of Pulumi AI. However, we faced a challenge: these schemas were much larger than any current generative AI system could handle.

What puts the “large” in large language model is the volume of data required to train and the number of parameters in the model, typically in the trillions of tokens and billions of parameters, respectively. Unfortunately, these models aren’t capable of handling that much information in their context window. The context window is the thread of conversation or the “short-term memory” of the system. It’s how one can enhance a model trained a year ago with today’s data.

To better understand the limitations, let’s talk about tokens. Instead of operating on Unicode text, these models operate on token arrays, mapping many sequences of characters (words or parts of words) to 16-bit integers. For example, “Script” is one token, and “JavaScript” is typically two tokens. We found that despite this compression of words to tokens, the latest schemas for all of our providers would use on the order of one billion tokens.

The most powerful systems to date support at most 32,000 tokens in their prompt – enough to fit a novella! That should be enough, right?

We did the math: one billion minus 32,000… And we’re still about a billion tokens over the limit.

The right information, at the right time

To work around this limitation, we need to carefully select and inject the right schema information into the context window, ensuring that the model could understand, process, and generate the best possible solutions for users.

We cracked open our college textbooks on information storage & retrieval, evaluated the latest vector databases and embedding systems, and perused the literature. One internal demo we did used a technique similar to Hypothetical Document Embeddings (HyDE). While this did improve accuracy and reduce hallucinations, that approach also requires pre-processing the user’s query. That additional step results in a much higher time-to-first-token (TTFT) latency, negatively impacting user experience.

After iterating on these approaches in the weeks after launch, we’ve implemented a method that yields comparable results to HyDE without slowing the system down. Our current approach allowed us to maintain an engaging and responsive interface while still enhancing Pulumi AI’s accuracy.

We continue to follow the latest research in this space to improve Pulumi AI, and we hope we can contribute our learnings back in this exciting field of research!

Improved User Experience and Examples

With the integration of Pulumi Package schema data, Pulumi AI now generates more accurate and relevant Pulumi programs, favoring our high-level components and the generally available providers we recommend, such as the awsx, eks, and azure-native packages.

Powerful components

For EKS clusters, Pulumi AI now typically recommends our eks provider’s high level component when asked for “an eks cluster”:

To achieve your goal of creating an EKS cluster, I recommend using the eks.Cluster Pulumi resource, which is a higher-level component encapsulating the necessary AWS and Kubernetes resources. This resource makes it easy to provision, scale, and manage EKS clusters in a Pulumi Python program.

To proceed, you’ll need to have the pulumi-eks package installed:
pip install pulumi-eks
Here’s a simple Pulumi Python program that creates an EKS cluster:
import pulumi
from pulumi_eks import Cluster

# Create an EKS cluster.
my_cluster = Cluster("my-cluster")

# Export the cluster kubeconfig.
pulumi.export("kubeconfig", my_cluster.kubeconfig)

Simple building blocks

Pulumi AI now offers a more succinct solution for common queries such as “a static website on Azure”:

To accomplish your goal, I will generate a Pulumi TypeScript program using the azure-static-website.Website resource from the Pulumi Registry. This resource is designed specifically for hosting static websites on Azure.

Here’s a Pulumi TypeScript program to deploy a static website on Azure using the azure-static-website.Website resource.
import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";
import * as staticWebsites from "@pulumi/azure-static-website";

const website = new staticWebsites.Website("website", {
 sitePath: websitePath,
 resourceGroupName: resourceGroup.name,
 withCDN: true,
});

New products

When asked for modern resources such as Google AlloyDB, launched six weeks ago, Pulumi AI now references Registry documentation to answer “google alloydb with an automatic backup policy”:

Based on the Pulumi Registry results and trying to accomplish your goal of creating an AlloyDB cluster with an automatic backup policy, we can create a Google Cloud Platform (GCP) AlloyDB cluster using the gcp.alloydb.Cluster resource. You can set the automatedBackupPolicy property with the necessary configuration to enable automatic backups.

Here’s a Pulumi program in TypeScript that creates an AlloyDB Cluster with an automatic backup policy:

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";

const projectId = gcp.config.project;
const location = "us-central1";

const network = new gcp.compute.Network("alloydb-network", {
 autoCreateSubnetworks: false,
});

const alloydbCluster = new gcp.alloydb.Cluster("alloydbCluster", {
 clusterId: "my-alloydb-cluster",
 location,
 project: projectId,
 network: network.selfLink,
 automatedBackupPolicy: {
 enabled: true,
 labels: {
 "backup-type": "automated",
 },
 location: "us-central1",
 quantityBasedRetention: {
 count: 7,
 },
 timeBasedRetention: {
 retentionPeriod: "7d",
 },
 weeklySchedule: {
 daysOfWeeks: ["MONDAY", "WEDNESDAY", "FRIDAY"],
 startTimes: [
 {startTime: "04:00:00"},
 ],
 },
 },
});

export const alloyDBClusterName = alloydbCluster.name;

Getting Started with the Enhanced Pulumi AI

Our enhancements to Pulumi AI have made it an even more useful tool for generating accurate and relevant cloud infrastructure solutions. We’re excited about the future of Pulumi AI as we continue to integrate it into Pulumi Cloud and empower everyone to become a cloud engineer.

We’ve also added new ways to reach Pulumi AI. In the Pulumi Registry, “Explore with Pulumi AI” links provide interactive documentation. And after you’ve had a conversation with Pulumi AI, you can share it via a “Share conversation” link in the bottom right corner of the message box:

Try out Pulumi AI today, and happy coding!

The Easier Way to Create Pulumi Providers in Go

Kyle Dixler — Thu, 19 Jan 2023 00:00:00 +0000

We are excited to announce that we’ve updated the Pulumi Provider Boilerplate to make custom provider implementation easier!

This major update brings a wealth of usability improvements to the Pulumi Provider Boilerplate by incorporating our brand new Pulumi Go Provider SDK.

Note: This update will not have any impact on existing providers that are utilizing an older version of the repository.

Background

The Pulumi Provider Boilerplate (view on GitHub) is a template repository that contains a minimal example of a working provider and has served as a starting point for writing your own provider.

For those who may not be familiar with provider authoring, it involves three distinct parts:

Package schema definition
Provider implementation
SDK generation

Providers are implemented as gRPC servers that handle requests to perform CRUD (create, read, update, delete) operations on resources on behalf of the Pulumi engine. SDK generation, on the other hand, involves generating source code libraries for a Pulumi program to utilize the provided resources.

In the past, implementing a provider using the old provider boilerplate could be difficult because you had to implement:

the provider’s schema
the CRUD operations on a provider’s resources
the gRPC server
SDK code generation to consume these resources in a Pulumi program

Now, using our code-first approach, you can implement your provider in Go by defining the provider’s metadata, resources, and functions, and then compiling the provider binary. The library takes care of the provider schema and gRPC server for you and the Pulumi CLI takes care of the SDK code generation with the following command:

pulumi package gen-sdk <path/to/provider-binary>

You can leave the heavy lifting to us and focus on the implementation details that matter to your organization.

Boilerplate code

Note: The completed boilerplate is attached at the bottom of this document and in the pulumi-provider-boilerplate repository (view on GitHub).

Provider Entrypoint

The code below uses the new pulumi-go-provider v1 APIs. Make sure you are using the latest version of github.com/pulumi/pulumi-go-provider.

The pulumi provider boilerplate program is currently quite short and the main entrypoint is as follows.

package main

import (
 "context"
 "log"

 "github.com/pulumi/pulumi-go-provider/infer"
)

func main() {
 prov, err := infer.NewProviderBuilder().
 WithNamespace("my-team-name").
 WithResources(
 // We tell the provider what resources it needs to support.
 // In this case, a single custom resource.
 infer.Resource(Random{}),
 ).
 Build()
 if err != nil {
 log.Fatal(err.Error())
 }

 _ = prov.Run(context.Background(), "xyz", "v0.0.1")
}

Resource Implementation

Resources consist of arguments, state, and CRUD operations to control them.

// Each resource has in input struct, defining what arguments it accepts.
type RandomArgs struct {
 // Fields projected into Pulumi must be public and hava a `pulumi:"..."` tag.
 // The pulumi tag doesn't need to match the field name, but its generally a
 // good idea.
 Length int `pulumi:"length"`
}

// Each resource has a state, describing the fields that exist on the created resource.
type RandomState struct {
 // It is generally a good idea to embed args in outputs, but it isn't strictly necessary.
 RandomArgs
 // Here we define a required output called result.
 Result string `pulumi:"result"`
}

// Each resource has a controlling struct.
type Random struct{}

Resource behavior is determined by implementing methods on the controlling struct.

// All resources must implement Create at a minumum.
func (Random) Create(ctx p.Context, name string, input RandomArgs, preview bool) (string, RandomState, error) {
 state := RandomState{RandomArgs: input}
 if preview {
 return name, state, nil
 }
 state.Result = makeRandom(input.Length)
 return name, state, nil
}

The Create method is mandatory, but other methods are optional.

Check: Remap inputs before they are typed.
Diff: Change how instances of a resource are compared.
Update: Mutate a resource in place.
Read: Get the state of a resource from the backing provider.
Delete: Custom logic when the resource is deleted.
Annotate: Describe fields and set defaults for a resource.
WireDependencies: Control how outputs and secrets flows through values.

Wrapping up

We encourage everyone to try the new authoring experience, including devs who may have previously found it challenging in comparison to the ease of program authorship.

Additional Example: Command provider

If you want a more involved example, the command provider (view on GitHub) has been rewritten to use the Pulumi Go Provider library entirely and is heavily commented. You can actually compare the project before and after using it to see how it has streamlined provider implementation.

Completed example

The complete boilerplate provider looks as follows:

package main

import (
 "context"
 "log"

 "github.com/pulumi/pulumi-go-provider/infer"
)

// Version is initialized by the Go linker to contain the semver of this build.
var Version string

func main() {
 prov, err := infer.NewProviderBuilder().
 WithNamespace("my-team-name").
 WithResources(
 // We tell the provider what resources it needs to support.
 // In this case, a single custom resource.
 infer.Resource(Random{}),
 ).
 Build()
 if err != nil {
 log.Fatal(err.Error())
 }

 _ = prov.Run(context.Background(), "xyz", Version)
}

// Each resource has a controlling struct.
// Resource behavior is determined by implementing methods on the controlling struct.
// The `Create` method is mandatory, but other methods are optional.
// - Check: Remap inputs before they are typed.
// - Diff: Change how instances of a resource are compared.
// - Update: Mutate a resource in place.
// - Read: Get the state of a resource from the backing provider.
// - Delete: Custom logic when the resource is deleted.
// - Annotate: Describe fields and set defaults for a resource.
// - WireDependencies: Control how outputs and secrets flows through values.
type Random struct{}

// Each resource has in input struct, defining what arguments it accepts.
type RandomArgs struct {
 // Fields projected into Pulumi must be public and hava a `pulumi:"..."` tag.
 // The pulumi tag doesn't need to match the field name, but its generally a
 // good idea.
 Length int `pulumi:"length"`
}

// Each resource has a state, describing the fields that exist on the created resource.
type RandomState struct {
 // It is generally a good idea to embed args in outputs, but it isn't strictly necessary.
 RandomArgs
 // Here we define a required output called result.
 Result string `pulumi:"result"`
}

// All resources must implement Create at a minumum.
func (Random) Create(ctx p.Context, name string, input RandomArgs, preview bool) (string, RandomState, error) {
 state := RandomState{RandomArgs: input}
 if preview {
 return name, state, nil
 }
 state.Result = makeRandom(input.Length)
 return name, state, nil
}

func makeRandom(length int) string {
 seededRand := rand.New(rand.NewSource(time.Now().UnixNano()))
 charset := []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789")

 result := make([]rune, length)
 for i := range result {
 result[i] = charset[seededRand.Intn(len(charset))]
 }
 return string(result)
}

Preview of .NET resource providers

Fraser Waters — Wed, 18 Jan 2023 00:00:00 +0000

Today we are pleased to announce the Preview of .NET support for custom resource providers. This means you can build custom providers using your favorite .NET language, including C#, F#, and VB.NET.

This is a first look preview release of the provider interface for .NET. It has minimal documentation but is feature complete. We hope to gather early feedback to refine the API and quickly bring this to a production quality release.

You will need to use version 3.52.0 to access the new APIs. You can install this in your project with:

dotnet add package Pulumi --version 3.52.0

You can see an example of using the new provider interface in the dotnet integration tests directory. This has replaced the old test provider that was written in Go.

Please view the provider interface and raise issues and discussions on GitHub or our Community Slack in the #dotnet channel — we can’t wait to hear what you think!

Multicloud with Kubernetes and Pulumi

Guinevere Saenger — Thu, 09 Jun 2022 11:40:47 -0700

In this article we’ll show you how to use Pulumi Components and the Pulumi Automation API to make golden path decisions which will both support your customers on multiple different clouds, and enable infrastructure teams and frontend service teams to more easily own their respective parts of your codebase.

Let’s say you provide a hosted service for your customers. You would like to offer customizable Kubernetes cluster provisioning, on multiple different clouds, as well as your product service itself.

There’s a lot of configuration fine-tuning and manual repetition that is required when bringing up Kubernetes clusters on different clouds.

This article is a written summary of our KubeCrash live demo, see it below. Please find the repository with the full version of the code here.

Part One: Create a component

A Component Resource is an abstraction on top of other Pulumi packages, often combining a few different providers into something that fits your infrastructure needs precisely.

Prerequisites

In our example, we are using three separate clouds, one of these being a local KinD cluster.

Nodejs
Pulumi CLI
Have the Pulumi Registry documentation handy for some Pulumi providers:
- Pulumi Civo Provider
- Pulumi Linode Provider
- Pulumi Kubernetes Provider
- Pulumi KinD Provider (Kubernetes-in-Docker) Note: As this is a community package, follow installation instructions from the README. This provider only supports Go and Nodejs.

Gather access tokens for Linode and Civo, if using, and set them in the environment or as secrets.

Start by writing the code for each cloud

The below code, when run against pulumi up, will create a single cloud stack with three separate Kubernetes clusters. The Stack output will be the cluster’s name and kubeconfig.

import * as pulumi from "@pulumi/pulumi";
import * as civo from '@pulumi/civo';
import * as linode from "@pulumi/linode";
import * as kind from "@pulumi/kind";


//Civo

// Create a firewall
const fw = new civo.Firewall("guin-fw", {
 region: "nyc1",
 createDefaultRules: true,
});

const civoCluster = new civo.KubernetesCluster("guin-civo-demo", {
 name: "guin-demo",
 region: "nyc1",
 firewallId: fw.id,
 pools: {
 nodeCount: 3,
 size: "g4s.kube.xsmall",
 }
});

export const clusterNameCivo = civoCluster.name
export const kcCivo = civoCluster.kubeconfig

//Linode

const linodeCluster = new linode.LkeCluster("guin-linode-demo", {
 k8sVersion: "1.22",
 label: "guin-demo",
 pools: [{
 count: 2,
 type: "g6-standard-2",
 }],
 region: "us-central",
 tags: ["guin-demo"],
});

export const clusterLabelLinode = linodeCluster.label
export const kcLinode = linodeCluster.kubeconfig

// Kind

const kindCluster = new kind.cluster.Cluster(
 'guin-kind',
 {
 name: pulumi.interpolate`guin-kind`,
 nodes: [
 {
 role: kind.node.RoleType.ControlPlane,
 extraPortMappings: [
 {
 containerPort: args.nodePort,
 hostPort: args.nodePort,
 },
 ],
 },
 {
 role: kind.node.RoleType.Worker,
 },
 ],
 },
)

export const clusterNameKind = kindCluster.name
export const kcKind = kindCluster.kubeconfig

We could stop here, export the kubeconfigs to a file, and start interacting with our clusters via kubectl. But that would require our cluster operators to be deeply familiar with each cloud provider’s Kubernetes implementation. Let’s see if we can do better.

Create a custom component for the Kubernetes cluster

Our goal today is to bring up a “generic” Kubernetes cluster that fits a single use case for multiple clouds. To do so, we create a default implementation as a wrapper around each cloud provider. We implement the following features:

Choice of cloud
Cluster Naming
Kubeconfig Output for later use

The Pulumi SDK allows us to wrap resources into a ComponentResource, which in our Typescript example means we create a new class that extends pulumi.ComponentResource.

export class ShinyCluster extends pulumi.ComponentResource {
 constructor(
 name: string,
 args: {},
 opts: {},
 ) {
 # ...
 }
}

Because we like all things shiny, we’ll call it a ShinyCluster, and pass pkg:index:ShinyCluster as the unique resource name.

We’ll also register the name and kubeconfig for each ShinyCluster as Outputs:

import { Output } from '@pulumi/pulumi'; // add this to dependencies

#...

export class ShinyCluster extends pulumi.ComponentResource {
 constructor(
 name: string,
 args: {},
 opts: {},
 ) {
 super('pkg:index:ShinyCluster', name, {}, opts);
 }
 name: Output<string>;
 kubeConfig: Output<string>;
}

Write the implementation for each cloud provider:

export class ShinyCluster extends pulumi.ComponentResource {
 constructor(
 name: string,
 args: {
 cloud: 'linode' | 'civo' | 'kind';
 nodePort?: number; // this is a KinD-specific add-on so we can connect to our workload
 },
 opts: {},
 ) {
 super('pkg:index:ShinyCluster', name, {}, opts);
 switch (args.cloud) {
 case 'linode':
 const linodeCluster = new linode.LkeCluster(
 'my-cluster',
 {
 label: 'guin-linode',
 k8sVersion: '1.22',
 pools: [
 {
 count: 2,
 type: 'g6-standard-2',
 },
 ],
 region: 'us-central',
 tags: ['guin'],
 },
 { parent: this },
 );
 this.name = linodeCluster.label;
 this.kubeConfig = linodeCluster.kubeconfig.apply((x) =>
 Buffer.from(x, 'base64').toString(),
 );
 break;
 case 'civo':
 const fw = new civo.Firewall(
 'guin-fw',
 {
 name: 'guin-civo',
 region: 'nyc1',
 createDefaultRules: true,
 },
 { parent: this },
 );

 const civoCluster = new civo.KubernetesCluster(
 'guin-civo',
 {
 region: 'nyc1',
 name: 'guin-civo',
 firewallId: fw.id,
 pools: {
 nodeCount: 3,
 size: 'g4s.kube.xsmall',
 },
 },
 { parent: this },
 );
 this.name = civoCluster.name;
 this.kubeConfig = civoCluster.kubeconfig;
 break;
 case 'kind':
 const kindCluster = new kind.cluster.Cluster(
 'guin-kind',
 {
 name: 'guin-kind',
 nodes: [
 {
 role: kind.node.RoleType.ControlPlane,
 extraPortMappings: [
 {
 containerPort: args.nodePort,
 hostPort: args.nodePort,
 },
 ],
 },
 {
 role: kind.node.RoleType.Worker,
 },
 ],
 },
 { parent: this },
 );
 this.name = kindCluster.name;
 this.kubeConfig = kindCluster.kubeconfig;
 }
 }
 name: Output<string>;
 kubeConfig: Output<string>;
}

We are now ready to use our component.

Read in the configuration from the config file and create a cluster:

const config = new pulumi.Config();

const shinyCluster = new ShinyCluster(
 'guin',
 {
 cloud: config.require('cloud'),
 nodePort: config.getNumber('nodePort'),
 },
 {},
);

export const shinyName = shinyCluster.name;
export const shinyConfig = pulumi.secret(shinyCluster.kubeConfig);

In our terminal, we can now create a stack called dev, set the necessary inputs in the Pulumi config, and run pulumi up:

$ pulumi stack init dev
Created stack 'dev'
$ pulumi config set shinycluster:cloud civo
$ pulumi up
Previewing update (dev)

View Live: https://app.pulumi.com/guinevere/shinycluster/dev/previews/cf968a89-dadc-461f-badd-b062601c18b4

 Type Name Plan
 + pulumi:pulumi:Stack shinycluster-dev create

At this point, we have created a Pulumi Component Resource that we can use to scaffold Kubernetes infrastructure.

Side Quest: Ship an app!

The purpose of this demo is not focused on the details of deploying a web app on Kubernetes with Pulumi, so we won’t go into too much detail here. We created a shinyapp folder to hold a Pulumi program (written in yaml!) We could now follow the Kubernetes provider configuration steps to set up and deploy our shinyapp to each of our clusters.

But if we have multiple Kubernetes backends, and would like to deploy our shinyapp to each of them, this would get tedious soon. Here is where we automate deploying our frontend as well.

Part 2: Leverage the Automation API

The Pulumi Automation API allows us to run Pulumi commands without using the CLI manually.

Let’s review our basic project file structure first, for context. We place both our shinycluster backend and shinyapp frontend Pulumi projects in their own folders, to separate concerns, and for visual clarity.

.
├── shinyapp
│ ├── Main.yaml # our web app, declared in yaml
│ └── Pulumi.yaml # Pulumi project config for `shinyapp`, with yaml runtime
├── shinycluster
│ ├── index.ts # our showcased Pulumi Component Resource for a Kubernetes cluster
│ └── Pulumi.yaml # Pulumi project config for `shinycluster`, with nodejs (Typescript) runtime

The neat thing here is that both our Kubernetes clusters and our application service are in their own contained folder, and in fact each can be deployed independently using Pulumi. Let’s go ahead and implement the functionality that allows us to automate it all by adding the following files to the root of the project:

├── Pulumi.yaml # Pulumi project config for the entire demo
├── index.ts # Input and output logic for our frontend and backend stacks
├── outputFormatter.ts # Prettify output
├── runPulumiProject.ts # Our specific automation implementation, using Automation API
└── stackModule.ts # Wrapper logic to show which types are expected by each part of our project

Not shown: dependency and package files.

The Stack Module

First, we define a generic Pulumi stack, with variable working directories and outputs, as expected by the automation API.

// stackModule.ts
import * as pulumi from '@pulumi/pulumi';

export type Unwrapped<T> = NonNullable<pulumi.Unwrap<Awaited<T>>>;

export type OutputMap<T> = {
 [K in keyof T]: {
 value: T[K];
 secret: boolean;
 };
};

export type StackModule<T> = {
 workDir: string;
 projectName: string;
 stack(): Promise<T>;
};

type StackOutputs<T> = T extends StackModule<infer O> ? O : never;

/**
 * The type of the outputs, as returned by the automation API as an "OutputMap".
 */
export type StackOutputMap<T> = OutputMap<Unwrapped<StackOutputs<T>>>;

/**
 * The type of the outputs, as if executed and run in a Pulumi program:
 */
export type StackOutputValues<T> = Unwrapped<StackOutputs<T>>;

/**
 * The type of a function that gets values by name from the outputs of a stack.
 *
 * See `stackOutputConfig`.
 */
export type StackOutputGetter<T> = <
 K extends keyof StackOutputValues<T> & string,
>(
 key: K,
) => StackOutputValues<T>[K];

Wrap the Automation API

We have a few functionalities that we can expand on and implement in just the way we like, so we put them in a file called runPulumiProject.ts.

// runPulumiProject.ts
import {
 ConfigMap,
 LocalProgramArgs,
 LocalWorkspace,
 OpMap,
 PreviewResult,
 Stack,
 UpResult
} from '@pulumi/pulumi/automation';

For the purposes of this demo repo, we added a formatter as well, but it’s not necessary for the functionality:

import chalk from 'chalk';
import { Formatter } from './outputFormatter';

And we import the output mappings we abstracted away in the stack module, above.

import { OutputMap, Unwrapped } from './stackModule';

We can now decide how we want to implement our automation options.

Let’s define the options we’d like this program to be able to run with. We need:

Directory name
Project name
Stack name
Pulumi operation to run
Extra configuration (so we can pass in that kubeconfig)
The formatter
Stack outputs

// runPulumiProject.ts
interface PulumiRunOptions {
 dir: string;
 project: string;
 stackName: string;
 operation: 'up' | 'preview' | 'destroy';
 additionalConfig: ConfigMap;
 formatter: Formatter;
}
interface PulumiRunResult<T> {
 stack: Stack;
 projectName: string;
 outputs: OutputMap<Unwrapped<T>>;
}

In this example, the pulumi commands we automate are:

Preview
Up
Destroy

export async function runPulumiProject<T extends object>({
 dir, project, stackName, operation, additionalConfig, formatter,
}: PulumiRunOptions): Promise<PulumiRunResult<T> | undefined> {
 const localProgramArgs: LocalProgramArgs = {
 stackName,
 workDir: dir,
 };
 const stack = await LocalWorkspace.createOrSelectStack(localProgramArgs, {});

 formatter(`Spinning up stack ${project}/${stackName}`);
 await stack.setAllConfig({
 ...additionalConfig,
 });

 formatter('Refreshing');
 await stack.refresh();

 let result: UpResult | PreviewResult;
 let status: string = 'succeeded';
 let outputs: OutputMap<Unwrapped<T>>;
 let summaryMessage: string | undefined;
 let operations: OpMap;
 switch (operation) {
 case 'preview':
 formatter('Previewing');

 const previewResult = await stack.preview({ onOutput: formatter });
 operations = previewResult.changeSummary;
 outputs = (await stack.outputs()) as OutputMap<Unwrapped<T>>;
 break;
 case 'up':
 formatter('Deploying');

 const upResult = await stack.up({ onOutput: formatter });
 operations = upResult.summary.resourceChanges;
 status = upResult.summary.result;
 summaryMessage = upResult.summary.message;
 outputs = upResult.outputs as OutputMap<Unwrapped<T>>;
 break;
 case 'destroy':
 formatter('Destroying');

 const destroyResult = await stack.destroy({ onOutput: formatter });
 operations = destroyResult.summary.resourceChanges;
 status = destroyResult.summary.result;
 summaryMessage = destroyResult.summary.message;
 break;
 }

 if (status !== 'succeeded') {
 formatter(result.stderr);
 formatter(summaryMessage);
 throw new Error();
 }

 if (operations) {
 formatter('Succeeded! Resource summary:');
 const fmtNum = (num?: number) => `${num}`.padStart(3);
 if (operations?.create) {
 formatter(`${fmtNum(operations?.create)} ${chalk.green('created')}`);
 }
 if (operations?.replace) {
 formatter(`${fmtNum(operations?.replace)} ${chalk.magenta('replaced')}`);
 }
 if (operations?.update) {
 formatter(`${fmtNum(operations?.update)} ${chalk.yellow('updated')}`);
 }
 if (operations?.same) {
 formatter(`${fmtNum(operations?.same)} ${chalk.bold('unchanged')}`);
 }
 }

 return {
 stack,
 projectName: project,
 outputs: outputs as OutputMap<Unwrapped<T>>,
 };
}

In addition, this code gives us the ability to take a stack’s Output and apply it to another stack, which is what we will do next, in our top-level index.js file.

Putting it all together

// index.js
import { chooseColor, outputFormatter } from './outputFormatter';
import { runPulumiProject } from './runPulumiProject';

interface ClusterOutputs {
 shinyName: string;
 shinyConfig: string;
}

ClusterOutputs defines the information we need to pass on the kubeconfig to our Kubernetes web app, and the cluster name to associate each frontend stack with its backend Kubernetes cluster.

Here, we’re deploying our shinyapp to a local KinD cluster, but we have the ability to add any cluster to this list:

const clusters = [
 {name: 'kind-local', cloud: 'kind', nodePort: 32001},
];

First, we wait for our clusters to be created (note the operation: 'up' being passed):

await Promise.allSettled(
 clusters.map(async (clusterDefinition) => {
 let theme = chooseColor();

 const serviceType = clusterDefinition.nodePort ? 'NodePort' : 'LoadBalancer';
 const serviceNodePort = `${clusterDefinition.nodePort ?? 0}`;
 const cluster = await runPulumiProject<ClusterOutputs>({
 dir: './shinycluster',
 project: 'shinycluster',
 stackName: clusterDefinition.name,
 operation: 'up',
 additionalConfig: {
 cloud: { value: clusterDefinition.cloud },
 nodePort: { value: serviceNodePort },
 },
 formatter: outputFormatter(`cluster ${clusterDefinition.name}`, theme)
 });
 // here we will call runPulumiProject, this time on the web app
 }),
 );

And then deploy our web app:

runPulumiProject({
 dir: './shinyapp',
 project: 'shinyapp',
 stackName: clusterDefinition.name,
 operation: 'up',
 additionalConfig: {
 kubeconfig: cluster.outputs.shinyConfig,
 serviceType: { value: serviceType },
 serviceNodePort: { value: serviceNodePort },
 },
 formatter: outputFormatter(`app ${clusterDefinition.name}`, theme)
});

Putting it all together:

// index.js
import { chooseColor, outputFormatter } from './outputFormatter';
import { runPulumiProject } from './runPulumiProject';

interface ClusterOutputs {
 shinyName: string;
 shinyConfig: string;
}

async function main() {
 const clusters = [
 { name: 'kind-local', cloud: 'kind', nodePort: 32001 },
 ];

 await Promise.allSettled(
 clusters.map(async (clusterDefinition) => {
 let theme = chooseColor();

 const serviceType = clusterDefinition.nodePort ? 'NodePort' : 'LoadBalancer';
 const serviceNodePort = `${clusterDefinition.nodePort ?? 0}`;
 const cluster = await runPulumiProject<ClusterOutputs>({
 dir: './shinycluster',
 project: 'shinycluster',
 stackName: clusterDefinition.name,
 operation: 'up',
 additionalConfig: {
 cloud: { value: clusterDefinition.cloud },
 nodePort: { value: serviceNodePort },
 },
 formatter: outputFormatter(`cluster ${clusterDefinition.name}`, theme)
 });

 runPulumiProject({
 dir: './shinyapp',
 project: 'shinyapp',
 stackName: clusterDefinition.name,
 operation: 'up',
 additionalConfig: {
 kubeconfig: cluster.outputs.shinyConfig,
 serviceType: { value: serviceType },
 serviceNodePort: { value: serviceNodePort },
 },
 formatter: outputFormatter(`app ${clusterDefinition.name}`, theme)
 });
 }),
 );
}

main();

Run the program

All that is left to do is run our top-level program:

$ ts-node ./index.js

Once that’s done running, go visit your webapp appear on localhost with the specified port:

$ curl localhost:32002
<!DOCTYPE html>
<html>

<head>
 <meta http-equiv="Content-Type" content="text/html" charset="utf-8">
</head>

<body>
 <h1>Made with love by Pulumi!</h1>
</body>

</html>

Summary

We chose some cloud providers for bringing up Kubernetes clusters
We created a ShinyCluster custom resource that abstracts away any cloud provider implementation details, giving us a small set of defaults we care about
We programmatically deployed infrastructure using the Automation API
We deployed an app to this infrastructure, also using Automation API.

We hope you have gotten a little bit of inspiration from our whirlwind tour through these Pulumi features. Happy coding!

How we manage GitHub at Pulumi with Pulumi

Guinevere Saenger — Tue, 22 Mar 2022 11:29:33 -0700

We recently updated this article to reflect that parentTeamId of the Team resource accepts a string instead of an int since v5.9.1 of the Pulumi Github package.

Ah, GitHub. The home of all developers. The place where we share code. The world’s most awkward social media site. The secret LinkedIn for techies. The tool we use for company org structure, work planning, code ownership, and permissions…

Wait.

That’s quite a lot.

GitHub is good at many things, but a full-on organization management tool it is not.

Have you ever needed your manager to manually enable admin permission on a repo for you? Or have you needed to page the CEO to add you to a team, because your manager was out that day? Have you ever wondered who is on what team? Or which team owns a repo? What if you change teams, or a team changes names? A reorg happens, and the “platform-integrations” team is no more, but we still need to call it that on GitHub because it is the team with all the repository accesses?

When I joined Pulumi in 2021, all of the above happened to me within my first few weeks.

We at Pulumi wanted to reduce this kind of management friction, and we decided to solve it the Pulumi way: with declarative infrastructure using the Pulumi GitHub provider.

The Goal

First, we set a few ground rules. We decided that administrative privileges over Pulumi’s GitHub repositories should flow as follows:

An employee is part of one or more GitHub teams.
Teams have appropriately scoped permissions over each repository they own.
Any Pulumi employee should be able to request code repository access in an official way.

Additionally, any current team membership and repository access should not be disrupted during development of this tool.

Getting Started

Create a new Pulumi program:

$ pulumi new go
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (team-management)
project description: (A minimal Go Pulumi program)
Created project 'team-management'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev) pulumi/prod
Created stack 'prod'

Installing dependencies...

go: downloading github.com/pulumi/pulumi/sdk/v3 v3.26.1
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run 'pulumi up'

$ ls
Pulumi.yaml go.mod go.sum main.go

We now have a pulumi project YAML configuration file and the beginnings of a small Go program all set up.

According to the provider configuration documentation, we need to add a properly scoped token, as well as set “pulumi” as our GitHub organization owner.

$ export GITHUB_TOKEN=YYYYYYYYYYYYYY
$ export GITHUB_OWNER=pulumi

Capturing Current State of Resources

Now, Pulumi is great at creating new infrastructure from scratch via code. But this wasn’t a from-scratch situation. We had to migrate existing resources—GitHub teams—to Pulumi, without disrupting anyone’s access.

Enter Pulumi Import.

What Pulumi Import does, in a nutshell, is find existing infrastructure by unique ID (in the GitHub provider’s case, the team ID), and add them to a Pulumi Stack. You can find the specific import instructions on the registry documentation for each resource. In this case we want the GitHub import instructions.

$ pulumi import github:index/team:Team animals 1234567
Previewing import (pulumi/prod)

 Type Name Plan
 pulumi:pulumi:Stack team-mgmt-prod
 = └─ github:index:Team animals import

Resources:
 = 1 to import
 12 unchanged

Do you want to perform this import? yes
Importing (pulumi/prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 = └─ github:index:Team animals imported

Resources:
 = 1 imported
 1 unchanged

Now that we have imported the Team resource, it is part of our Pulumi Stack. But it is not part of our main.go program yet. Helpfully, Pulumi Import replies with sample code on how to add the imported resource to your Pulumi program, in the language you selected for your project:

Please copy the following code into your Pulumi application. Not doing so will cause Pulumi to report that an update will happen on the next update command.

Please note that the imported resources are marked as protected. To destroy them you will need to remove the `protect` option and run `pulumi update` *before* the destroy will take effect.


package main

import (
 "github.com/pulumi/pulumi-github/sdk/v4/go/github"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := github.NewTeam(ctx, "animals", &github.TeamArgs{
 CreateDefaultMaintainer: pulumi.Bool(false),
 Description: pulumi.String("should get imported"),
 Name: pulumi.String("animals"),
 Privacy: pulumi.String("closed"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

After adding the team to main.go as suggested, this is the output of pulumi preview:

Previewing update (pulumi/prod)

 Type Name Plan Info
 pulumi:pulumi:Stack team-mgmt-prod

Resources:
 2 unchanged

Since pulumi preview shows no changes, we now know that our code reflects the existing infrastructure. It’s a bit funny to think about your program working well when it does nothing, but this was a huge first step in preserving existing infrastructure and ensuring all of our coworkers could continue their daily work uninterrupted!

To finish up, we unprotect the resource:

$ pulumi state unprotect 'urn:pulumi:prod::team-mgmt::github:index/team:Team::animals'

Creating Configuration

Next, we import the remaining teams and refactor the code to handle multiple teams at once. This is also the time to write our organization structure into a YAML configuration file.

---
org: "pulumi-demo--org"
teams:
 - name: "animals"
 slug: "animals"
 description: "A team in charge of animals"
 - name: "newteam"
 slug: "newteam"
 description: "I am a new team"

We then read this information into a Go struct with the necessary properties to pass into the Pulumi provider.

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 Slug string `yaml:"slug"`
}
type Organization struct {
 Org string `yaml:"org"`
 Teams []Team `yaml:"teams"`
}

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

//import accurate team info from yaml
orgFilePath, err := filepath.Abs("./org.yaml")
if err != nil {
 return err
}
yamlFile, err := ioutil.ReadFile(orgFilePath)
if err != nil {
 return err
}
var org Organization
err = yaml.Unmarshal(yamlFile, &org)
if err != nil {
 return err
}

for _, team := range org.Teams {
 ghTeam, err := github.NewTeam(ctx, team.Slug, &github.TeamArgs{
 Description: pulumi.String(team.Description),
 Name: pulumi.String(team.Name),
 Privacy: pulumi.String("closed"),
}, pulumi.Protect(false))
if err != nil {
fmt.Println("encountered error creating new Pulumi GitHub team: ", team.Name)
 return err
}
 return nil
 })
}

Notice how the GitHub provider allows us to use team names to create teams. In contrast, team IDs are only needed for importing to the Stack, not for maintaining state.

Relationships Are Hard

The next step involved a lot of thinking about team memberships and team nesting. GitHub allows teams to be nested. This groups teams by area of responsibility but also allows for certain properties, such as permissions, to be inherited by subteams.

Parent Team Parent
 |
 -------------------------
 | |
Child Teams Child1 Child2
(inherit all permissions)

To establish this relationship, we set a ParentTeamId on child teams. First off, we add a new field to our Teams struct:

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 Slug string `yaml:"slug"`
 ParentTeamId int
 Teams []Team `yaml:"teams"`
}

Here is where things get a little tricky. Any GitHub Team can have subteams, but not every Team has a parent Team. We also did not want to have to hardcode GitHub Team IDs into our configuration files. We needed to meet the following requirements:

Create a parent team, using a team name.
Obtain its team ID.
Create any child teams, also using a team name.
Write the parent team’s ID into the ParentTeamId field of each child team.
Do all of the above in a single pulumi up.

This is where maintaining infrastructure with Pulumi truly shines. In our code, we can use Pulumi Apply to hold on to the promise of a parent team ID, and pass this promise into the appropriate field:

func setupTeams(ctx *pulumi.Context, parentTeam *Team) error {

// set up parent team, i.e. engineering
 ghParentTeam, err := github.NewTeam(ctx, parentTeam.Slug, &github.TeamArgs{
 Description: pulumi.String(parentTeam.Description),
 Name: pulumi.String(parentTeam.Name),
 Privacy: pulumi.String("closed"),
 }, pulumi.Protect(false))
 if err != nil {
 fmt.Println("encountered error creating new Pulumi github team: ", parentTeam.Name)
 return err
 }

//set up nested teams
 for _, childTeam := range parentTeam.Teams {
 // set each child team's parent team ID to the current team ID
 ghChildTeam, err := github.NewTeam(ctx, childTeam.Slug, &github.TeamArgs{
 Description: pulumi.String(childTeam.Description),
 Name: pulumi.String(childTeam.Name),
 Privacy: pulumi.String("closed"),
 ParentTeamId: ghParentTeam.ID(),
 }, pulumi.Protect(false))
 if err != nil {
 fmt.Println("encountered error creating new Pulumi github team: ", childTeam.Name)
 return err
 }
 }
 return nil
}

Running this as part of main.go will result in beautifully nested teams on the GitHub UI. But with Pulumi, we can do even better. We can set pulumi.Parent() on the child teams:

for _, childTeam := range parentTeam.Teams {
 ghChildTeam, err := github.NewTeam(ctx, childTeam.Slug, &github.TeamArgs{#...},
 pulumi.Protect(false),
 pulumi.Parent(ghParentTeam))
 #...
}

With this, we can see the relationship between the resources on pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + └─ github:index:Team newteam created
 + └─ github:index:Team newchildteam created
Resources:
 + 2 created
2 unchanged

Which will be reflected on the Pulumi Stack’s Graph View in the Pulumi Service:

Adding this relationship to Pulumi is mostly a visual nicety in our case; however, certain Pulumi properties get inherited in a helpful way. Note that on GitHub, it is possible to create multiple layers of teams, whereas the example code in this blog post only shows a single layer of subteam structure. We are small enough at Pulumi to not need more, but you may want to expand on the solution if you do.

Adding People

Next, teams should have members! Let’s add them to the YAML config:

org: "pulumi-demo-org"
teams:
 - name: "animals"
 slug: "animals"
 description: "A team in charge of animals"
 members:
 - username: "platypus"
 - username: "owlcat"

A TeamMembership in GitHub is a cross reference between a Team and a User. Fortunately these do not need to be explicitly imported, as they are merely establishing relationships between GitHub Users and Teams. We can add TeamMemberships to the Stack with a Members struct, a new Members[] field on the Team struct, and an extra function. Again, Pulumi lets us use the promised output of the team ID to set the TeamId field in the TeamMembership:

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 ParentTeamId int
 Teams []Team `yaml:"teams"`
 Slug string `yaml:"slug"`
 Members []Member `yaml:"members"`
}

type Member struct {
 UserName string `yaml:"username"`
 Role string `yaml:"role"`
}

#...

func addMembers(ctx *pulumi.Context, members []Member, team *github.Team, teamName string) error {
 for _, member := range members {
 // unique name for TeamMembership
 utmName := teamName + "-" + member.UserName
 _, err := github.NewTeamMembership(ctx, utmName, &github.TeamMembershipArgs{
 TeamId: team.ID(),
 Username: pulumi.String(member.UserName),
 })
 if err != nil {
 return err
 }
 }
 return nil
}

Running pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + ├─ github:index:TeamMembership animals-owlcat created
 + └─ github:index:TeamMembership animals-platypus created

Resources:
 + 2 created
 4 unchanged

Alright! So now we have

Imported existing teams to a Pulumi Stack to track and organize all GitHub teams from a central repository, and
Added team members to teams.

That’s pretty great so far! While there are many org chart tools, what makes this one useful to us is that we can declare the desired org state in a config file, and then let Pulumi figure out what changes should be applied. Let’s automate that!

Add CI

In order for this tool to be used by everyone, we keep code and configuration in a GitHub repository. We can use Pulumi’s GitHub Action to run a pulumi preview on a pull request, and a pulumi up on merge to main.

Here’s what that looks like on the pull request:

name: Pull Request preview
on:
 pull_request:
 workflow_dispatch:

env:
 AWS_REGION: us-west-2
 GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_OWNER: pulumi

jobs:
 pull-request:
 runs-on: ubuntu-latest
 name: Preview
 steps:
 - uses: actions/checkout@v2
 - name: Install Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.17
 - name: Build Binary
 run: |
 go mod tidy
 go build main.go
 - name: Pulumi Preview
 uses: pulumi/actions@v3
 with:
 command: preview
 refresh: true
 stack-name: pulumi/prod
 comment-on-pr: true

And the pulumi up on merge to main:

name: Update
on:
 push:
 branches:
 - "main"
 workflow_dispatch:

env:
 AWS_REGION: us-west-2
 GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_OWNER: pulumi

jobs:
 main:
 runs-on: ubuntu-latest
 name: Update
 steps:
 - uses: actions/checkout@v2
 - name: Install Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.17
 - name: Build Binary
 run: |
 go mod tidy
 go build main.go
 - name: Pulumi update
 uses: pulumi/actions@v3
 with:
 command: up
 refresh: true
 stack-name: pulumi/prod

Note that we are calling refresh: true in both Workflows, which uses Pulumi Refresh to make sure that the existing GitHub resources are aligned with the resource state in our Stack.

Now, anyone with access to the GitHub management repo can:

Create, re-parent, delete, or rename teams and re-create any memberships via pull request to the config file.
Add and remove team members via pull request.
Audit and explicitly maintain org structure via git history and review processes.

But of course… there’s more!

Managing Permissions

The next step involved a lot of thinking about repository permissions and permission access.

Shortly after I joined Pulumi, my team was combined with another team. They brought along all of their repositories—none of which I had access to. Moreover, all of our team names were outdated. I could rename the teams using pulumi-github… but I could not transfer all of the repository access we all needed to our new team. Yet.

On GitHub, parent teams pass permissions down to child teams. But what if we wanted permissions to be more granular?

Here’s where TeamRepositories come in. A TeamRepository has three fields: a team ID, a repository name, and a permission. Just like TeamMemberships, this resource is a link between a Team and a Repository and does not need to be imported.

We gave each team their own config file with permission levels listing repository names:

team: "animals"
permissions:
 - role: "admin"
 repos:
 - "capybaras"
 - "owls"
 - role: "maintain"
 repos:
 - role: "push"
 repos:
 - role: "read"
 repos:

This can be read into our Pulumi program as follows:

// TeamPermissions describes a github team and the levels of permissions available (i.e. admin, maintain)
type TeamPermissions struct {
 TeamName string `yaml:"team-name"`
 Permissions []Permission `yaml:"permissions"`
}

// Permission is a single permission and a list of repos with that permission.
type Permission struct {
 Role string `yaml:"role"`
 Repos []string `yaml:"repos"`
}

#...

func setupTeamRepos(ctx *pulumi.Context, team *github.Team, teamName string) error {

 repoFilePath, err := filepath.Abs("./team-repos/" + teamName + ".yaml")
 if err != nil {
 fmt.Println(err.Error())
 return err
 }
 yamlFile, err := ioutil.ReadFile(repoFilePath)
 if err != nil {
 return err
 }

 var teamPerms TeamPermissions
 err = yaml.Unmarshal(yamlFile, &teamPerms)
 if err != nil {
 return err
 }
 for _, permission := range teamPerms.Permissions {
 for _, repo := range permission.Repos {
 _, err := github.NewTeamRepository(ctx, fmt.Sprintf("%s-%s-%s", repo, teamName, permission.Role), &github.TeamRepositoryArgs{
 Permission: pulumi.String(permission.Role),
 Repository: pulumi.String(repo),
 TeamId: team.ID(),
 })

 if err != nil {
 return err
 }
 }
 }
 return nil
}

And the output of running pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + ├─ github:index:TeamRepository owls-animals-admin created
 + ├─ github:index:TeamRepository capybaras-animals-admin created


Resources:
 + 2 created
 2 changes. 4 unchanged

With this addition, we can now combine teams with zero access disruptions as follows:

Create the new team in the org config.
Create a TeamRepository config for the new team with desired permission levels.
Open a pull request and let CI and Pulumi do the rest.

Now we can develop standards around repository permissions based on teams’ roles and areas of responsibility, and we have tooling in place that can maintain these standards for everyone in the org to see. In fact, we can standardize the meaning of “code ownership” via access levels in this way—one centralized management location rather than asking your grandboss to dig through the UI for you.

Future challenges

We want to automate some of these steps: adding new teammates to the org should also automatically add them to this repository so they can self-serve their onboarding and see Pulumi work in action!

To prevent state drift, we will run a regular reconciliation job in CI.

Of course, we will add other Pulumi providers on top of this org structure so we can sync our other tooling.

Introducing Resource Methods for Pulumi Packages

Justin Van Patten — Mon, 11 Oct 2021 08:00:00 -0700

It’s now possible to provide resource methods from Pulumi Packages. Resource methods are similar to functions, but instead of being exposed as top-level functions in a module, methods are exposed as methods on a resource class. This allows for a more object-oriented approach to exposing functionality—operations performed by a resource (that potentially use the resource’s state) can now be exposed as methods on the resource. Resource methods can be implemented once, in your language of choice, and made available to users in all Pulumi languages.

When authoring component resources, it’s often useful to provide additional functionality through methods on the component. For example, the Cluster component in the eks package has a getKubeconfig method that can be used to generate a kubeconfig for authentication with the cluster that does not use the default AWS credential provider chain, but is instead scoped based on the passed-in arguments. Until now, that method has only been available from JavaScript/TypeScript (the language the Cluster component was written in). With the new support for resource methods for Pulumi Packages, we can make this method available to all the Pulumi languages, which is exactly what we’ve done in pulumi-eks v0.34.0.

const cluster = new eks.Cluster("mycluster");
const kubeconfig = cluster.getKubeconfig(...);

cluster = eks.Cluster("mycluster")
kubeconfig = cluster.get_kubeconfig(...)

var cluster = new Cluster("mycluster");
var kubeconfig = cluster.GetKubeconfig(...);

cluster, err := eks.NewCluster(ctx, "mycluster", nil)
if err != nil {
 return err
}
kubeconfig, err := cluster.GetKubeconfig(ctx, &eks.ClusterGetKubeconfigArgs{...})
if err != nil {
 return err
}

This post will show how you can provide resource methods from your component packages.

Authoring Methods

It’s always been possible to provide module-level functions from Pulumi Packages. For example, the aws package provides many module-level functions, such as the getAmi function, which can be used to get the ID of an existing Amazon Machine Image (AMI). Functions are declared in the package’s schema, and their functionality is implemented in the provider through the Invoke remote procedure call (RPC).

Methods are authored in a similar manner to functions. Methods are declared in the schema and implemented in the provider’s Call RPC (similar to Invoke).

Let’s walk through an example. We’ll author a Message component that accepts a message as input. The component then provides a getMessage method that accepts a recipient name and returns the message customized for the recipient. To get started authoring a component package, refer to the package documentation.

Schema

We’ll start with declaring the method and component in the Pulumi schema. First, define the function representing the method:

 "functions": {
 "example:index:Message/getMessage": {
 "inputs": {
 "properties": {
 "__self__": {
 "$ref": "#/resources/example:index:Message"
 },
 "recipient": {
 "type": "string"
 }
 },
 "required": ["__self__", "recipient"]
 },
 "outputs": {
 "properties": {
 "result": {
 "type": "string"
 }
 },
 "required": ["result"]
 }
 }
 },

Our method has two required arguments: __self__ and recipient. __self__ is a special input required by all methods that represents the component resource and is typed as such. The method has one result named result.

Next, define our Message component:

 "resources": {
 "example:index:Message": {
 "isComponent": true,
 "inputProperties": {
 "message": {
 "type": "string"
 },
 },
 "requiredInputs": ["message"],
 "properties": {
 "message": {
 "type": "string"
 },
 },
 "required": ["message"],
 "methods": {
 "getMessage": "example:index:Message/getMessage"
 }
 }
 },

Our component has a single required input/output property: message. The method is specified in the methods property, which references the method’s function definition: "getMessage": "example:index:Message/getMessage".

Here’s our schema all together:

{
 "version": "0.0.1",
 "name": "example",
 "functions": {
 "example:index:Message/getMessage": {
 "inputs": {
 "properties": {
 "__self__": {
 "$ref": "#/resources/example:index:Message"
 },
 "recipient": {
 "type": "string"
 }
 },
 "required": ["__self__", "recipient"]
 },
 "outputs": {
 "properties": {
 "result": {
 "type": "string"
 }
 },
 "required": ["result"]
 }
 }
 },
 "resources": {
 "example:index:Message": {
 "isComponent": true,
 "inputProperties": {
 "message": {
 "type": "string"
 },
 },
 "requiredInputs": ["message"],
 "properties": {
 "message": {
 "type": "string"
 },
 },
 "required": ["message"],
 "methods": {
 "getMessage": "example:index:Message/getMessage"
 }
 }
 },
 "language": {
 "csharp": {
 "packageReferences": {
 "Pulumi": "3.12"
 },
 "liftSingleValueMethodReturns": true
 },
 "go": {
 "liftSingleValueMethodReturns": true
 },
 "nodejs": {
 "devDependencies": {
 "@types/node": "latest"
 },
 "liftSingleValueMethodReturns": true
 },
 "python": {
 "liftSingleValueMethodReturns": true
 }
 }
}

To make it easier for users of our method, we also set "liftSingleValueMethodReturns": true for each language, which makes single-value methods return the single value directly, rather than wrapping the results in a Result type.

Component Implementation

We can implement the component and make it available to any Pulumi language. We’re going to show some implementation examples in TypeScript/JavaScript, Python, and Go.

Here’s the implementation of the Message component:

import * as pulumi from "@pulumi/pulumi";

interface MessageArgs {
 message: pulumi.Input<string>;
}

class Message extends pulumi.ComponentResource {
 public readonly message!: pulumi.Output<string>;

 constructor(name: string, args: MessageArgs, opts?: pulumi.ComponentResourceOptions) {
 const props = { message: args?.message }
 super("example:index:Message", name, props, opts);

 if (opts?.urn) {
 // Skip further initialization when being constructed from a resource reference.
 return;
 }

 this.registerOutputs(props);
 }

 getMessage(recipient: pulumi.Input<string>): pulumi.Output<string> {
 return pulumi.iterpolate `${recipient}, ${this.message}!`;
 }
}

from typing import Optional

import pulumi

class Message(pulumi.ComponentResource):
 @property
 @pulumi.getter
 def message(self) -> pulumi.Output[str]:
 return pulumi.get(self, "message")

 def __init__(self,
 resource_name: str,
 opts: Optional[pulumi.ResourceOptions] = None,
 message: Optional[pulumi.Input[str]] = None) -> None:

 args = {"message": message}
 super().__init__("example:index:Message", resource_name, args, opts)

 if opts and opts.urn:
 # Skip further initialization when being constructed from a resource reference.
 return

 self.register_outputs(args)

 def get_message(self, recipient: pulumi.Input[str]) -> pulumi.Output[str]:
 return pulumi.Output.concat(recipient, ", ", self.message, "!")

Authoring multi-language components in .NET is not yet supported.

import (
 "errors"

 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

type Message struct {
 pulumi.ResourceState
 Message pulumi.StringOutput `pulumi:"message"`
}

type MessageArgs struct {
 Message pulumi.StringInput `pulumi:"message"`
}

func NewMessage(ctx *pulumi.Context, name string, args *MessageArgs, opts ...pulumi.ResourceOption) (*Message, error) {
 if args == nil {
 return nil, errors.New("args is required")
 }

 message := &Message{}
 err := ctx.RegisterComponentResource("example:index:Message", name, message, opts...)
 if err != nil {
 return nil, err
 }

 message.Message = args.Message.ToStringOutput()
 if err := ctx.RegisterResourceOutputs(message, pulumi.Map{
 "message": args.Message,
 }); err != nil {
 return nil, err
 }

 return message, nil
}

type GetMessageArgs struct {
 Recipient pulumi.StringInput `pulumi:"recipient"`
}

func (c *Message) GetMessage(args *GetMessageArgs) StringOutput {
 return pulumi.Sprintf("%s, %s!", args.Recipient, c.Message)
}

Provider RPCs

Next, wire up the provider RPCs with the component implementation:

import * as pulumi from "@pulumi/pulumi";
import * as provider from "@pulumi/pulumi/provider";

class Provider implements provider.Provider {
 public readonly version = "0.0.1";

 constructor() {
 // Register any resources that can come back as resource references that need to be rehydrated.
 pulumi.runtime.registerResourceModule("example", "index", {
 version: this.version,
 construct: (name, type, urn) => {
 switch (type) {
 case "example:index:Message":
 return new Component(name, <any>undefined, { urn });
 default:
 throw new Error(`unknown resource type ${type}`);
 }
 },
 });
 }

 async construct(name: string, type: string, inputs: pulumi.Inputs,
 options: pulumi.ComponentResourceOptions): Promise<provider.ConstructResult> {
 if (type != "example:index:Message") {
 throw new Error(`unknown resource type ${type}`);
 }

 const message = new Message(name, <MessageArgs>inputs, options);
 return {
 urn: message.urn,
 state: inputs,
 };
 }

 async call(token: string, inputs: pulumi.Inputs): Promise<provider.InvokeResult> {
 switch (token) {
 case "example:index:Message/getMessage":
 const self: Component = inputs.__self__;
 return {
 outputs: {
 result: self.getMessage(inputs.recipient),
 },
 };

 default:
 throw new Error(`unknown method ${token}`);
 }
 }
}

export function main(args: string[]) {
 return provider.main(new Provider(), args);
}

main(process.argv.slice(2));

from typing import Optional
import sys

import pulumi
import pulumi.provider as provider


class Provider(provider.Provider):
 VERSION = "0.0.1"

 class Module(pulumi.runtime.ResourceModule):
 def version(self):
 return Provider.VERSION

 def construct(self, name: str, typ: str, urn: str) -> pulumi.Resource:
 if typ == "example:index:Message":
 return Component(name, pulumi.ResourceOptions(urn=urn))
 else:
 raise Exception(f"unknown resource type {typ}")

 def __init__(self):
 super().__init__(Provider.VERSION)
 pulumi.runtime.register_resource_module("example", "index", Provider.Module())

 def construct(self, name: str, resource_type: str, inputs: pulumi.Inputs,
 options: Optional[pulumi.ResourceOptions] = None) -> provider.ConstructResult:

 if resource_type != "example:index:Message":
 raise Exception(f"unknown resource type {resource_type}")

 message = Message(name, opts=options, message=inputs["message"])

 return provider.ConstructResult(
 urn=component.urn,
 state=inputs)

 def call(self, token: str, args: pulumi.Inputs) -> provider.CallResult:
 if token != "example:index:Message/getMessage":
 raise Exception(f'unknown method {token}')

 message: Message = args["__self__"]
 outputs = {
 "result": message.get_message(args["recipient"])
 }
 return provider.CallResult(outputs=outputs)


if __name__ == "__main__":
 provider.main(Provider(), sys.argv[1:])

Authoring multi-language components in .NET is not yet supported.

import (
 "errors"
 "fmt"

 "github.com/blang/semver"

 "github.com/pulumi/pulumi/pkg/v3/resource/provider"
 "github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 pulumiprovider "github.com/pulumi/pulumi/sdk/v3/go/pulumi/provider"
)

const providerName = "example"
const version = "0.0.1"

type module struct {
 version semver.Version
}

func (m *module) Version() semver.Version {
 return m.version
}

func (m *module) Construct(ctx *pulumi.Context, name, typ, urn string) (r pulumi.Resource, err error) {
 if typ != "example:index:Message" {
 return nil, fmt.Errorf("unknown resource type: %s", typ)
 }
 r = &Component{}
 err = ctx.RegisterResource(typ, name, nil, r, pulumi.URN_(urn))
 return
}

type GetMessageResult struct {
 Result pulumi.StringOutput `pulumi:"result"`
}

func main() {
 pulumi.RegisterResourceModule("example", "index", &module{semver.MustParse(version)})

 if err := provider.MainWithOptions(provider.Options{
 Name: providerName,
 Version: version,
 Construct: func(ctx *pulumi.Context, typ, name string, inputs pulumiprovider.ConstructInputs,
 options pulumi.ResourceOption) (*pulumiprovider.ConstructResult, error) {

 if typ != "example:index:Message" {
 return nil, fmt.Errorf("unknown resource type %s", typ)
 }

 args := &MessageArgs{}
 if err := inputs.CopyTo(args); err != nil {
 return nil, fmt.Errorf("setting args: %w", err)
 }

 message, err := NewMessage(ctx, name, args, options)
 if err != nil {
 return nil, fmt.Errorf("creating component: %w", err)
 }

 return pulumiprovider.NewConstructResult(message)
 },
 Call: func(ctx *pulumi.Context, tok string, args pulumiprovider.CallArgs) (*pulumiprovider.CallResult, error) {
 if tok != "example:index:Message/getMessage" {
 return nil, fmt.Errorf("unknown method %s", tok)
 }

 methodArgs := &GetMessageArgs{}
 res, err := args.CopyTo(methodArgs)
 if err != nil {
 return nil, fmt.Errorf("setting args: %w", err)
 }
 message := res.(*Message)

 result, message.GetMessage(methodArgs)
 return pulumiprovider.NewCallResult(&GetMessageResult{
 Result: result
 })
 },
 }); err != nil {
 cmdutil.ExitError(err.Error())
 }
}

The Construct RPC is called when creating an instance of the component. The Call RPC is called when a method is called.

Using the Component

Now we can build and try out using the component and its method from any Pulumi language:

const component = new example.Message("mycomponent", {
 message: "hello world",
});
export const message = component.getMessage({ recipient: "Alice" }); // Exports "Alice, hello world!"

component = example.Message("mycomponent", message="hello world")
message = component.get_message(recipient="Alice")
pulumi.export("message", message) # Exports "Alice, hello world!"

var component = new Message("mycomponent", new MessageArgs
{
 Message = "hello world",
});
// Exports "Alice, hello world!"
this.Message = component.GetMessage(new MessageGetMessageArgs
{
 Recipient = "Alice",
});

component, err := example.NewMessage(ctx, "mycomponent", &example.MessageArgs{
 Message: "hello world",
})
if err != nil {
 return err
}
message, err := component.GetMessage(ctx, &example.MessageGetMessageArgs{Recipient: "Alice"})
if err != nil {
 return err
}
pulumi.Export("message", message) // Exports "Alice, hello world!"

Wrapping Up

With support for resource methods for Pulumi Packages, you can now create component resources with methods and make them available to use from any Pulumi language. We look forward to seeing the components you create!

👉 Author your first Pulumi Package

Announcing Pulumi 3.0

Luke Hoban — Tue, 20 Apr 2021 00:00:06 -0700

Today we’re excited to announce the availability of Pulumi 3.0, the next major version of the Pulumi open source project, and the foundation for Pulumi’s Cloud Engineering Platform.

Pulumi offers the most complete infrastructure as code platform for building, deploying and managing modern cloud infrastructure and applications. Pulumi enables cloud engineers to use familiar languages to describe their cloud infrastructure - bringing core software engineering tools and practices to bear on managing and getting the maximum value from their cloud platforms of choice - across dozens of cloud and SaaS providers.

Pulumi 3.0 includes dozens of significant new features and hundreds of improvements that build on this foundation. This release includes more than 200 contributions from over 150 members of the Pulumi community, and builds on feedback from working with thousands of Pulumi users and customers over the last year.

Some of the most exciting new features in this release include:

Pulumi Automation API: All the power of Pulumi, now available to embed within your own software and applications in Node.js, Python, Go and .NET - opening up a brand new wave of modern cloud infrastructure use cases, including self-service infrastructure portals, custom CLIs, and richer CI/CD workflows.
Native Pulumi Providers for Azure (GA) and Google Cloud (Preview): New providers with 100% platform coverage, same-day support for new features, and the exact same resource models defined by the cloud providers - the most complete and consistent way to manage cloud infrastructure in Azure and Google Cloud.
Pulumi Packages and Multi-Language Components: Take any Pulumi components written in your language of choice, and make them available as a Pulumi Package to users in all other Pulumi languages - bringing together each of the language ecosystems Pulumi supports into a single Pulumi component ecosystem.
Improved Go and Python SDKs: Updates to the core Go and Python SDKs make APIs simpler and more consistent, unlock more tooling support, and improve compile times - making it even easier to use all the benefits of your favorite programming language when building and managing cloud infrastructure.
New Integrations and Features in the Pulumi Service: Alongside Pulumi 3.0, users of the Pulumi Service also now have access to new dashboards for organization-level cloud infrastructure insights, a CI/CD Assistant for easy version control and CI/CD integration with just a few clicks, and new identity, access control, and auditing capabilities.

👉 Get started with Pulumi 3.0 today or check out the Pulumi 3.0 migration guide to update your existing Pulumi projects to Pulumi 3.0.

Pulumi Automation API

Pulumi’s Automation API was released in preview last year and is now GA as part of Pulumi 3.0. It provides an API for embedding Pulumi within your own applications, and supports all Pulumi languages including TypeScript/JavaScript, Python, Go, and C#. Instead of working with the pulumi CLI directly, or writing scripts to execute the CLI within your CI/CD environment, you can now build your own software solutions that build on top of Pulumi’s robust, desired state infrastructure as code deployment engine. It’s simple to embed Pulumi in your software-defined workflows with just a few lines of code.

This opens up so many exciting new opportunities for how your team can manage cloud infrastructure; we’ve seen amazing creativity as Pulumi’s customers and communities have created solutions on top of Automation API. In addition, Pulumi Automation API powers several other new features like the Pulumi Kubernetes Operator, the Pulumi GitHub Action v3, and Concourse CI support for Pulumi.

👉 Read more in the Pulumi Automation API blog post.

Native Providers for Azure and Google Cloud

Pulumi offers packages for managing cloud infrastructure in more than 50 cloud and SaaS applications, with more than a dozen new providers added since Pulumi 2.0 including partnerships with Civo, Venafi, Okta, Auth0, Equinix Metal, Spot Ocean for Azure, and Splunk.

For the largest and most widely used cloud providers, we’ve created something new: Pulumi native providers, which give you the most complete and consistent interface for the modern cloud. These native providers are automatically generated from cloud provider APIs and resource models that are maintained directly by cloud provider service teams. This enables native providers to offer 100% API coverage, same-day support for new features, the exact same API designed by the provider service team, and fewer bugs and reliability issues. This makes native providers the best way to manage infrastructure in the supported cloud platforms, combining the benefits of Pulumi Infrastructure as Code with the resource models and pace of delivery of the cloud provider themselves.

Along with Pulumi 3.0, the new Azure Native provider, originally released in preview last year, is now generally available as version 1.0. The Azure Native provider is built on top of the Azure Resource Manager REST API specifications developed and maintained by Azure service teams. It supports more than 2x the number of resources, and 3x the number of resource properties as the classic Azure provider. Azure Native also supports every version of each resource API for maximum compatibility with your existing infrastructure. New versions of the Azure Native provider with support for the latest features of Azure are published every night, ensuring immediate availability of new features and improvements from the Azure service teams.

Today, we’re also excited to be releasing a preview of the new Google Cloud Native provider. This native provider offers 100% coverage of the Google Cloud platform, with 70% more resource properties supported compared to the existing provider. It also supports every version of each supported resource, including alpha, beta, and GA Google Cloud offerings.

👉 Read more in the Pulumi Native Providers blog post and the Pulumi Google Native Provider blog post. And of course, we haven’t forgotten AWS - an AWS Native provider is expected later this year.

Pulumi Packages and Multi-Language Components

Pulumi Packages are the core technology that enables cloud infrastructure resource provisioning to be defined once, and made available to users in all languages supported by Pulumi. If you’ve ever used a cloud provider in Pulumi, you’ve used a Pulumi Package. With Pulumi 3.0, we are extending Pulumi Packages to also support building and packaging Pulumi Components: higher-level abstractions over cloud infrastructure patterns described as normal Pulumi programs. This makes it easy to take a Pulumi program that defines a collection of resources and turn it into something reusable: a Pulumi Package that can be published and used by all Pulumi users, across all languages supported by Pulumi.

One example of building and packaging Pulumi Components as a Pulumi Package is the Pulumi EKS package. We originally released a component for easily building and managing production-ready EKS clusters nearly two years ago, but it was only available for Node.js users of Pulumi as an npm package. This component has been incredibly popular, as it abstracts away a lot of complexity in all the moving pieces that need to be managed to stand up an EKS cluster across both AWS resources and in-cluster Kubernetes resources. Recently, we turned this component into a Pulumi Package, making it available to users in Python, Go, and .NET as well, with just a single implementation. Pulumi EKS is just the first example of applying this technology; there’s a lot more in the works here in the coming months!

import pulumi
import pulumi_eks as eks

cluster = eks.Cluster("my-cluster")

pulumi.export("kubeconfig", cluster.kubeconfig)

package main

import (
 "github.com/pulumi/pulumi-eks/sdk/go/eks"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 cluster, err := eks.NewCluster(ctx, "my-cluster", nil)
 if err != nil {
 return err
 }
 ctx.Export("kubeconfig", cluster.Kubeconfig)
 return nil
 })
}

import * as eks from "@pulumi/eks";

const cluster = new eks.Cluster("my-cluster");

export const kubeconfig = cluster.kubeconfig;

using Pulumi;
using Pulumi.Eks;

class MyStack : Stack
{
 [Output("kubeconfig")]
 public Output<object> Kubeconfig { get; set; }

 public MyStack()
 {
 var cluster = new Cluster("my-cluster");

 Kubeconfig = cluster.Kubeconfig;
 }
}

All Pulumi packages define a common Pulumi schema, which describes the resource model the package exposes and can be used to generate language-specific SDKs and documentation. There are several different types of Pulumi Package:

Native Pulumi Provider Package: Use the full features of the Pulumi resource model to create a provider for a new cloud platform. Examples: the kubernetes, azure-native, and google-native packages.
Bridged Provider Package: Take an existing resource provider from another supported ecosystem (like a Terraform provider), and bridge it to be exposed as a Pulumi Package. Examples:the aws, tls, and cloudflare packages.
Component Package: Write a Pulumi Component in your language of choice and expose it to users in all languages supported by Pulumi. Example: the eks package.

Together, these options provide a combination of breadth and depth for the kinds of resources and components that can be exposed to Pulumi developers via Pulumi Packages.

👉 Read more in the Pulumi Packages blog post.

Improved Go and Python SDKs

Pulumi’s language SDKs bring the power of cloud infrastructure management into your language of choice in a natural and productive way, enabling you to use all of your existing software engineering knowledge and tools for your cloud infrastructure projects as well. In Pulumi 3.0, we’ve made improvements to the Go and Python SDKs to improve developer productivity and enable additional tooling.

For Go users, Pulumi 3.0 reduces binary sizes by 4x and improves build times by up to 3x, leading to a much faster infrastructure development loop. Along with this, the Go APIs for working with Pulumi Output values have been simplified to provide a single way of transforming outputs instead of three different options. This makes the Pulumi Go SDK more consistent and easier to get started with. You can learn more about these changes in the Pulumi 3.0 migration guide.

For Python users, Pulumi 3.0 brings a new, more strongly typed way to provide inputs to cloud resources, ensuring that users get completion lists, type checking, documentation, and more within their IDE of choice. Along with this, some inconsistencies in input property naming have also been removed; now, all property names follow a consistent and predictable naming convention. You can learn more about these changes in the Pulumi 3.0 migration guide.

New Integrations and Features in the Pulumi Service

In addition to the Pulumi 3.0 open source release, we’re also making available new features for users of the Pulumi Service: the simple and easy to use default backend for all Pulumi users, with additional features for teams and enterprises.

Integrating Pulumi into CI/CD systems is a key step in realizing the full benefits of modern infrastructure as code, and we’ve worked to make this easier than ever. Since 2.0, we’ve added partner integrations with Spinnaker, GitHub Actions and CodeFresh as well as a new Pulumi Kubernetes Operator. In addition, we’ve introduced a new CI/CD Assistant to the service to help configure version control and CI/CD integration for your Pulumi projects with just a few clicks.

For enterprise users, we’ve added several enhancements to SAML SSO and SCIM support. Pulumi’s SSO support now works seamlessly with Okta (as well as Google Identity, Azure Active Directory and more), users and groups can now be synced via SCIM support in your SAML provider, and new Team Admin roles make team management simpler for large organizations.

Lastly, the Pulumi Service now offers a dashboard landing page which brings insights about your infrastructure front-and-center whether you are an individual user, or part of a large organization.

👉 Read more in the Pulumi CI/CD Assistant blog post and the SAML SSO and SCIM blog post.

Keep exploring

Pulumi 3.0 is a significant new milestone for the Pulumi project and lays the foundations for the Pulumi Cloud Engineering Platform with support for the very best of the cloud via new Pulumi Native Providers, support for a unified ecosystem of reusable infrastructure components with Pulumi Packages, and support for a new era of cloud infrastructure management scenarios enabled by Pulumi Automation API. Each of these is just a start: there’s so much more to come on top of these foundations and we’re excited to work with our customers and community on the next wave of innovation in Cloud Engineering!

To learn more about each of the new areas - check out these blog posts with more details:

You can get started with Pulumi 3.0 today, or check out the Pulumi 3.0 migration guide to update your existing Pulumi projects to Pulumi 3.0.

We can’t wait to see what you build with Pulumi 3.0!

Introducing Pulumi Packages and multi-language Components

Alex Mullans — Tue, 20 Apr 2021 00:00:03 -0700

Pulumi Packages are the core technology that enables cloud infrastructure resource provisioning to be defined once, in your language of choice, and made available to users in all Pulumi languages. If you’ve used a Pulumi cloud provider, including one of our new Pulumi native providers, you’ve used a Pulumi Package. But until today, Pulumi Packages only worked with Pulumi Resources: direct, low-level representations of individual cloud services like object storage. Many of us, however, enjoy creating Pulumi Components, which combine low-level resources into higher-level, more opinionated building blocks like the production-grade Kubernetes cluster component in Pulumi EKS. Unfortunately, those components, though powerful and unique to Pulumi’s IaC approach, were previously confined to a single language: so if your infrastructure team built a component in Python, your developers who might want to use TypeScript could not use it.

In Pulumi 3.0 Pulumi Components can now be authored in a fundamentally multi-language way, enabling you to write them once in your preferred language and make them available in all the other languages supported by Pulumi—all thanks to Pulumi Packages.

👉 Read on to learn more about Pulumi Packages, or jump right in and author your own with the user guide

Using Pulumi Packages

Pulumi Packages bring the full power of the modern cloud and the amazing creativity of the Pulumi Community to you. Using a Pulumi Package is easy: find one, install it into your project, and get started.

Choose from any of Pulumi’s cloud provider packages in the Registry, or search the npm registry for other Pulumi Packages
Install into your project with npm i [package-name]
Reference a namespace in the newly-installed package and start writing code

Choose from any of Pulumi’s cloud provider packages in the Registry, or search PyPI for other Pulumi Packages
Install into your project with pip install [package-name]
Reference a namespace in the newly-installed package and start writing code

Choose from any of Pulumi’s cloud provider packages in the Registry, or search the NuGet Gallery for other Pulumi Packages
Install into your project with dotnet add package [package-name]
Reference a namespace in the newly-installed package and start writing code

Choose from any of Pulumi’s cloud provider packages in the Registry
Install into your project with go get [package-url]
Reference a namespace in the newly-installed package and start writing code

Notably, you don’t need to know how a Pulumi Package was authored to use it. Using any Pulumi Package is the same regardless of whether that package contains native cloud resources, custom components, resources that came from a bridged Terraform provider, or anything of the other many possibilities. No matter what, the same idiomatic Pulumi IaC programming model is used to interact with those resource types and objects.

Creating Pulumi Packages

You can use Pulumi successfully without ever creating a Pulumi Package. But, we’d love to have you join the community of Pulumi Package authors! Here’s a quick overview of how Pulumi Packages work and how to create one.

A Pulumi Package is comprised of two inputs: a Pulumi schema, which describes the resource model the package exposes, and the Resources or Components described in that schema. From those inputs, Pulumi Packages can generate language-specific SDKs and documentation and then package those into SDK packages (npm, NuGet, and Python packages) that can be published to a public or private package registry. Along the way, a resource provider plugin is also created. The plugin is used by the Pulumi engine to instantiate, configure, and destroy the Resources or Components defined by the package.

There are several different ways to author a Pulumi Package depending on the kind of cloud infrastructure you want to define:

Native Pulumi Provider Package: Use the full features of the Pulumi resource model to create a provider for a new cloud platform. Examples: the kubernetes, azure-native, and google-native packages.
Bridged Provider Package: Take an existing resource provider from another supported ecosystem (like a Terraform provider), and bridge it to be exposed as a Pulumi Package. Examples: the aws, tls, and cloudflare packages.
Component Package: Write a Pulumi Component in your language of choice and expose it to users in all Pulumi languages. Example: the eks package.

Together, these options provide a combination of breadth and depth for the kinds of resources and components that can be exposed to Pulumi developers via Pulumi Packages. In the weeks ahead, we have many improvements underway to make the authoring and publishing of packages much easier. We also look forward to your feedback on the overall experience!

👉 Author your first Pulumi Package

Keep exploring

Pulumi Packages are just one part of Pulumi’s Cloud Engineering Platform. Watch the PulumiUP! event for more.

👉 PulumiUP!

Creating and Reusing Cloud Components using Package Managers

Chris Smith — Thu, 09 Aug 2018 00:00:00 +0000

Hello! A few weeks back I wrote a post on serving static websites on AWS with Pulumi detailing how to host a static website on AWS. Pulumi allowed me to wire four different AWS products together in only 200 lines of code. It would be a shame, however if I needed to copy and paste that code every time I wanted to to stand up a new website. Instead, we can package up, share, and reuse our code just like any other Node.js library. It just so happens that this one can be used to create cloud infrastructure.

Creating a Package

To create a reusable package for the static website example, I simply put the code into a new GitHub repo and publish it to NPM. We’ll walk through exactly how to do that below, but afterwards anyone who wants to stand up a static website on AWS can just reuse that package.

The static-website-aws package exports a StaticWebsite class, which is a logical, reusable cloud “component” resource. This class offers a simple constructor that internally does all of the hard work of creating the assortment of AWS cloud resources in the prior blog post – an S3 Bucket and objects per piece of static content, CloudFront Distribution, and Route53 DNS A-Record – and exposes them afterwards as readonly properties:

/**
 * Static website using Amazon S3, CloudFront, and Route53.
 */
export declare class StaticWebsite extends pulumi.ComponentResource  {
 readonly contentBucket: aws.s3.Bucket;
 readonly logsBucket: aws.s3.Bucket;
 readonly cdn: aws.cloudfront.Distribution;
 readonly aRecord?: aws.route53.Record;

 /**
    * Creates a new static website hosted on AWS.
    * @param name  The _unique_ name of the resource.
    * @param contentArgs  The arguments to configure the content being served.
    * @param domainArgs  The arguments to configure the domain and DNS settings.
    * @param opts  A bag of options that control this resource's behavior.
    */

 constructor(name: string , contentArgs: ContentArgs,
 domainArgs?: DomainArgs, opts?: pulumi.ResourceOptions);
}

This class is placed into an NPM package as usual, including a package.json file that gives it a name and a version. After doing this, we can publish it to NPM simply by running

$ npm publish

All the complexities of creating, configuring, and wiring up the necessary AWS resources – admittedly only 200 lines of code, but still intricate – has now been hidden beneath an ultra-simple component resource named StaticWebsite.

That’s how you create a new package. The great thing about packages, however, is that you define them once and then use them a lot. So now let’s turn to see see how standing up a new CDN-backed website can now become just a few lines of very simple, understandable code.

Reusing Infrastructure as Code

To use this new package, head on over to a Pulumi program, or create a new one (for instance, with pulumi new aws-typescript). Then just add a reference like any other dependency:

$ npm install static-website-aws

At that point, we can import our StaticWebsite class from the static-website-aws package like usual in Node.js programs, either using require

let swa = require("static-website-aws");

or using the new ES6 module import syntax, supported by TypeScript and modern JavaScript

import { StaticWebsite } from "static-website-aws";

Afterwards, we’ll just new up a StaticWebsite object, which internally creates everything for us. Of course, to create a useful website, we’ll need to supply some interesting parameters . In this case, let’s just pass a path to the content on disk (which will automatically get uploaded into S3 objects during a pulumi up) in addition to a path to a custom 404 HTML page:

import  { StaticWebsite } from "static-website-aws";

const website  = new StaticWebsite ("browserhack", {
 pathToContent:"./browserhack",
 custom404Path:"/404.html",
});

export const cloudfrontDomain = website .cdn .domainName ;

As soon as we have our program, we can stand it up with a single pulumi up command:

$ pulumi up

This will show us the entire resource graph so that we can see what it’s creating internally and once it’s done the resulting domain name will be printed out:

Let’s then curl it to see whether it worked:

$ curl $(pulumi stack output cloudfrontDomain)`[<!doctype html>
]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}[<!--
vim: set sw=2 ts=2 et : -->
]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}[<html>
]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}[
 <head>
]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}[...
etc, etc, etc
...]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}

Huzzah! (Note that new CloudFront Domains can take a while to spin up; if you get an error “Could not resolve host: xxx.cloudfront.net”, just wait a little bit and try again.)

The full source code for this example is available at chrsmith/browserhack-demo. This program deploys an instance of BrowserHack (a web-based port of the seminal console game NetHack). The repo also includes an example of assigning a custom domain name and ACM-managed SSL certificate, useful capabilities supported by StaticWebsite but omitted here for brevity .

Resource Components

We glossed over what components are earlier on. This is an advanced concept, so feel free to skip this section; but if you want to understand how components work under the hood, read on.

The StaticWebsite type extends pulumi.ComponentResource. A ComponentResource is a Pulumi-managed resource that aggregates other resources into a higher-level abstraction.

Pulumi will manage creating and updating the underlying resources like normal, and you will see updates to them in the resource tree. The difference is that by using the ComponentResource abstraction we can present those resources as logically grouped into a new, static website type.

This parent/child relationship between resources is declared using the optional pulumi.ResourceOptions type passed to Pulumi resources.

Here’s a code snippet from the code, where we create the S3 bucket and specify its logical parent resource. Every child resource created is passed defaultResourceOptions, which sets the parent property to this (the instance of StaticWebsite ).

// Default resource options for this component's child resources.
const defaultResourceOptions: pulumi.ResourceOptions = { parent:this };

...

// Create the logs bucket to store CloudFront request logs.

this.logsBucket = new aws.s3.Bucket(`${name }-logs`,
 { acl:"private" },
 defaultResourceOptions,
);

Components are a powerful construct in Pulumi, and let you build larger first class resources out of smaller ones. There are other benefits, including seeing resource attribution in the CLI tree view, and the richer visualization in the Pulumi Service:

If you want to learn more about components, see the documentation.

Package Everything!

In this blog, you’ve seen how Pulumi’s code-centric approach to infrastructure can make you more productive programming the cloud. “Code” has a lot of advantages over static configuration languages, for example rich IDE support. Another big advantage of using code, however, is that it allows you to create useful abstractions that can be reused in other programs

With Pulumi being able to take advantage of existing package managers, it’s super easy to create high-quality reusable components. We demonstrated NPM here for Node.js programs, but Pulumi also supports PyPI for Python, and traditional library techniques for Go.

But don’t just take our words for it. Recently Mikhail Shilkov blogged about creating a Pulumi component to keep AWS Lambda functions warm. And James Nugent open sourced a component that creates a properly subnetted AWS VPC. This is a great way to encode and share best practices broadly. Over time, we look forward to seeing the many other battle-tested components the community creates for common cloud applications and infrastructure needs.

If you’ve created a Pulumi component that is useful or want some design advice, come and join us in the Pulumi Community Slack – we’d love to hear from you!