Pulumi Blog: Myles Haynes

Stack READMEs in the Pulumi Service

Evan Boyle — Thu, 19 May 2022 09:35:01 -0700

Starting today, users can create Stack READMEs in the Pulumi Service that dynamically update based on Stack Outputs.

Each Pulumi Stack you deploy manages a key set of cloud infrastructure for your organization. The Pulumi Console includes a variety of features for exposing key information about your stack for other users within your organization - configuration, outputs, resources under management, links to cloud providers, and a graph of all resources. However, it’s often useful to allow the author of a Pulumi Stack to describe in their own words the key elements of a stack, so future viewers can quickly understand the components and cloud resources that are managed.

A README is a text file that introduces and explains a project. A Pulumi Service Stack README can contain documentation and common links for the Stack. The key difference between a source control README, such as a GitHub README, and a Pulumi Service Stack README, is that it is dynamically populated with details from your stack outputs. It does this by interpolating output variables on the stack (${outputs.instances[0].ARN}) so that each stack can construct links to dashboards, shell commands, and other pieces of documentation. All of this content stays up to date as you stand up new stacks, rename resources, and refactor your infrastructure.

There are a lot of operational activities that happen around Pulumi Stacks, but critical information is spread across many tools. We have heard from our users that it can be time consuming switching from different tools and web apps to manage their deployments- from their cloud Console, to the Pulumi Service User Interface (UI), to the CLI, and so on. By collecting resource outputs in the Pulumi Service UI, we are reducing this friction and keeping relevant Stack information in one place so that is kept up to date automatically.

README templates can reference Stack outputs and resource properties, for example ${outputs.vpc.ARN}. To walk you through a tangible example, a user can deploy an RDS instance and then use a variable in their README template to link to their CloudWatch dashboard to keep an eye on operational metrics. This CloudWatch dashboard link will dynamically update when deployments happen. You can use the same README template across dev, testing and production and have the correct dashboard links for each Stack. Learn more in the Stack READMEs documentation page.

The new experience lives in the Stack page, which can be navigated to through Projects and clicking on the specific Stack you want to view the README for. Let’s take a look at this new feature!

The Pulumi.README.md file we added to the Stack ‘Production’ above is a template README for our Pulumi Service. In it we have links to our AWS authentication tool, our CloudWatch metrics, to our production deployment documentation, and so on. The common places that we navigate to and from when managing this Stack. Here is some of the Markdown for this README:

# Pulumi Service README

[Sign in to AWS to view stack resources!](https://top-secret-url.com)

## On Call Operations

### Monitor

**Cloudwatch Metrics**
Monitor holistic metrics tracking overall service health
[Link](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#dashboards:name=${outputs.dashboardName})

**RDS Performance Metrics**
Monitor RDS performance (wait times, top queries)
[Link](https://us-west-2.console.aws.amazon.com/rds/home?region=us-west-2#performance-insights-v20206:/resourceId/${outputs.databaseClusterId}/resourceName/${outputs.rdsClusterWriterInstance})

**Cloudwatch Logs**
Search across service logs
[Link](https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logStream:group=${outputs.cloudwatchLogGroup})

How to add a Stack README to your Stack

In order to add a README to your Pulumi Stack, you will need to do the following:

Step 1

Export a Stack output named readme that contains your templated Stack README markdown, commonly by reading a file, i.e. Pulumi.README.md.

import { readFileSync } from "fs";
export const strVar = "foo";
export const arrVar = ["fizz", "buzz"];
// add readme to stack outputs. must be named "readme".
export const readme = readFileSync("./Pulumi.README.md").toString();

import pulumi
pulumi.export('strVar', 'foo')
pulumi.export('arrVar', ['fizz', 'buzz'])
# open template readme and read contents into stack output
with open('./Pulumi.README.md') as f:
 pulumi.export('readme', f.read())

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 strVar := "foo"
 arrVar := []string{"fizz", "buzz"}
 readmeBytes, err := ioutil.ReadFile("./Pulumi.README.md")
 if err != nil {
 return fmt.Errorf("failed to read readme: %w", err)
 }
 ctx.Export("strVar", pulumi.String(strVar))
 ctx.Export("arrVar", pulumi.ToStringArray(arrVar))
 ctx.Export("readme", pulumi.String(string(readmeBytes)))
 return nil
 })
}

using Pulumi;
class MyStack : Stack
{
 public MyStack()
 {
 this.StrVar = "foo";
 this.ArrVar = new string[] { "fizz", "buzz" };
 this.Readme = System.IO.File.ReadAllText("./Pulumi.README.md");
 }
 [Output]
 public Output<string> StrVar { get; set; }
 [Output]
 public Output<string[]> ArrVar { get; set; }
 [Output]
 public Output<string> Readme { get; set; }
}

package stackreadme;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
public class App {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 var strVar = "foo";
 var arrVar = new String[]{ "fizz", "buzz" };
 try {
 var readme = Files.readString(Paths.get("./Pulumi.README.md"));
 ctx.export("strVar", Output.of(strVar));
 ctx.export("arrVar", Output.of(arrVar));
 ctx.export("readme", Output.of(readme));
 } catch (IOException e) {
 throw new RuntimeException(e);
 }
 });
 }
}

name: stack-readme-yaml
runtime: yaml
description: A minimal Pulumi YAML program demonstrating stack readme feature
variables:
 readme:
 Fn::ReadFile: ./Pulumi.README.md
outputs:
 strVar: foo
 arrVar:
 - fizz
 - buzz
 readme: ${readme}

Step 2

Create a README template for the Stack. In this example, we will create a Pulumi.README.md file that looks as follows:

# Stack README

Full markdown support! Substitute stack outputs dynamically so that links can depend on your infrastructure! Link to dashboards, logs, metrics, and more.

1. Reference a string stack output: ${outputs.strVar}
2. Reference an array stack output: ${outputs.arrVar[1]}

Step 3

Run pulumi up on that Stack

Step 4

Open the Pulumi Service UI, navigate to Projects and then the Stack you have updated. Once on the Stack page you will see the README tab with your README file.

Ta da! 🎉

We now have a README on the Stack.

Refer to the Stack README documentation page for more details on how to use this feature. As always, please feel free to submit feature requests and bug reports to the Pulumi Service GitHub Repo. We love hearing feedback from users about ways we can improve your productivity when using Pulumi. We look forward to seeing how you make Stack READMEs fit your needs!

Announcing the Pulumi Service Provider

Myles Haynes — Wed, 04 May 2022 06:55:00 -0700

One of the advantages of having a large and vocal community like we have, is the quantity and quality of product feedback we receive. This was highlighted by a GitHub issue submitted by a community member for a Pulumi Service Provider:

It’s a bit funny that a service that is all about configuration as code can’t be configured with code.

The rest of the community agreed too, as this is one of our top customer product requests. Luckily Piers Karsenbarg, one of the Solutions Engineers at Pulumi, saw this and decided to build a Pulumi Provider for the Pulumi Service during a recent company hackathon. Once we saw the demo we knew we had to get in the hands of our customers! Over the last couple months, we’ve added a number of additional features, and polished some of the APIs for managing the Pulumi Service via Pulumi Infrastructure as Code. As a result, starting today, Pulumi’s users can manage all of their cloud infrastructure using Pulumi, including managing the state of the Pulumi Service itself!

The Pulumi Service Provider builds on top of the Pulumi Service REST API which is another feature available to our customers to programmatically configuring the Pulumi Service. The Pulumi Service REST API is includes functionality to interact with and manipulate any kind of metadata managed by Pulumi. That includes Projects and Stacks, Previews and Updates, Organizations and Audit Logs. We have already seen Cloud Engineering teams using the Pulumi REST API to build all sorts of custom functionality. These new capabilities are especially powerful when used in combination with the Automation API.

The Pulumi Service Provider can be used in any Pulumi language, TypeScript, Python, Go, .NET, Java and YAML. It supports configuration of Teams, Access Tokens, Stack Tags and Webhooks using infrastructure as code.

What resources are available?

Based on feedback from customers, we’ve picked the following resources to support at launch:

How do I get started?

If you aren’t already familiar with Pulumi, you’ll want to head over to Pulumi’s getting started guide first.

The Pulumi Service Provider requires an access token to authenticate with the Pulumi Service. You can set it in a few ways:

Via the PULUMI_ACCESS_TOKEN environment variable.
Via the pulumiservice:accessToken configuration value.
Directly via the provider’s constructor

Once your access token is set up, you are ready to use the provider.

The provider will only have access to resources that your token has access to.

Here’s a sample of what you can do:

name: pulumi-service-teams-example-yaml
runtime: yaml
description: An example of using yaml to create a team
resources:
 team:
 type: pulumiservice:index:Team
 properties:
 name: pulumi-up-team
 description: This was created with Pulumi
 displayName: PulumiUP Team
 organizationName: pulumi-up
 teamType: pulumi
 members:
 - piers
 - myles
 - meagan
outputs:
 members: ${team.members}

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const team = new service.Team("team", {
 name: "pulumi-up-team",
 displayName: "PulumiUP Team",
 description: "This was created with Pulumi",
 organizationName: "pulumi-up",
 teamType: "pulumi",
 members: [
 "piers",
 "myles",
 "meagan"
 ],
});

export const members = team.members;

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 team, err := pulumiservice.NewTeam(ctx, "team", &pulumiservice.TeamArgs{
 Name: pulumi.String("pulumi-up-team"),
 Description: pulumi.String("This was created with Pulumi"),
 DisplayName: pulumi.String("PulumiUP Team"),
 OrganizationName: pulumi.String("pulumi-up"),
 TeamType: pulumi.String("pulumi"),
 Members: pulumi.ToStringArray([]string{"piers", "myles", "meagan"}),
 })
 if err != nil {
 return err
 }
 ctx.Export("members", team.Members)
 return nil
 })
}

import pulumi
from pulumi_pulumiservice import Team

team = Team("team",
 name="pulumi-up-team",
 description="This was created with Pulumi",
 display_name="PulumiUP Team",
 organization_name="pulumi-up",
 team_type="pulumi",
 members=["piers", "myles", "meagan"])

pulumi.export("members", team.members)

using Pulumi;
using Pulumi.PulumiService;
class ExampleTeam: Stack {
 public ExampleTeam()
 {
 var team = new Team("team", new TeamArgs
 {
 Name = "pulumi-up-team",
 Description = "This was created with Pulumi",
 DisplayName = "PulumiUP Team",
 OrganizationName = "pulumi-up",
 TeamType = "pulumi",
 Members = {"piers", "myles", "meagan"}
 });

 this.Members = team.Members;
 }

 [Output] public Output<string> Members { get; set; }
}

Once you’ve created your team, you can navigate to the Pulumi Service in your browser and view your team:

Can Self Hosted customers use the Pulumi Service Provider?

Yes! You will be able to use the Pulumi Service Provider. You can configure the URL via the following methods:

Via the PULUMI_BACKEND_URL environment variable. You’ll want to set it to the API URL of the service.
Via the pulumiservice:apiUrl configuration value.

The default URL for the provider is: https://api.pulumi.com.

What’s next?

We’re excited to be able to offer Pulumi Infrastructure as Code users the ability to manage the Pulumi Service with IaC. Over the coming months, we have plans to add more resources, and to build on the Pulumi Service Provider functionality as we see what our customers use it for. As always, please feel free to submit feature requests and bug reports to Pulumi Service Provider GitHub Repo. We look forward to seeing what you build with the Pulumi Service Provider!