Pulumi Blog: Github actions

How We Eliminated Long-Lived CI Secrets Across 70+ Repos

Boris Schlosser — Tue, 31 Mar 2026 00:00:00 +0000

Supply chain attacks on CI/CD pipelines are accelerating. A growing pattern involves attackers compromising popular GitHub Actions through tag poisoning — rewriting trusted version tags to point to malicious code that harvests environment variables, cloud credentials, and API tokens from runner environments. The stolen credentials are then exfiltrated to attacker-controlled infrastructure, often before anyone notices.

For every engineering organization, the question is no longer if your CI pipeline will encounter a compromised dependency, but what is exposed when it does.

At Pulumi, we asked ourselves that question and decided the answer should be “nothing useful.” Here’s how we got there.

The problem with static CI secrets

Most organizations store long-lived cloud credentials, API tokens, and service account keys as GitHub repository or organization secrets. But this approach has several well-known problems:

Broad availability. Every workflow run on a repository can access every secret stored in that repo. A compromised action in any workflow can read them all.
No expiration. Secrets persist until someone manually rotates them. If exfiltrated, they give attackers persistent access for weeks or months.
No granular audit trail. GitHub tells you a secret was used, but not which workflow, which step, or what it was used for.
Secret sprawl. Across dozens or hundreds of repos, the same credentials are often duplicated, making rotation a coordinated, error-prone effort.

In a supply chain attack scenario, this is exactly what attackers count on: a single compromised action that can dump a trove of long-lived credentials.

Our approach: zero static secrets

We replaced every static GitHub Secret across our CI pipelines with short-lived, dynamically fetched credentials using Pulumi ESC and OpenID Connect (OIDC). The credential flow works in layers, each scoped and ephemeral:

GitHub generates a short-lived OIDC token scoped to the specific workflow run, repository, and branch. This token is cryptographically signed by GitHub’s OIDC provider.
The token is exchanged with Pulumi Cloud for a short-lived Pulumi access token. Pulumi Cloud validates the OIDC claims (organization, repository, branch) against a configured trust policy before issuing the token.
The Pulumi access token opens an ESC environment to retrieve the credentials the workflow needs — cloud provider keys, API tokens, or other secrets.
Cloud credentials themselves are dynamic. ESC environments use OIDC login providers to fetch short-lived credentials directly from AWS, Azure, or GCP. No static keys or cloud credentials are stored anywhere.

The pulumi/esc-action GitHub Action handles this entire flow in a single workflow step.

sequenceDiagram
participant Runner as GitHub Actions Runner
participant GH as GitHub OIDC Provider
participant PC as Pulumi Cloud
participant ESC as Pulumi ESC
participant Cloud as Cloud Provider (AWS/Azure/GCP)
Runner->>GH: Request OIDC token (scoped to workflow run)
GH-->>Runner: Short-lived JWT
Runner->>PC: Exchange JWT for Pulumi access token
PC-->>Runner: Short-lived access token
Runner->>ESC: Open environment with access token
ESC->>Cloud: OIDC login (assume role / federated identity)
Cloud-->>ESC: Short-lived cloud credentials
ESC-->>Runner: Cloud credentials + secrets
Note over Runner,Cloud: Nothing is stored. Everything expires.

What the change looks like

Before this migration, our workflows referenced static secrets stored in GitHub:

env:
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

After the migration, an ESC environment handles credential fetching via OIDC. Here is what the environment definition looks like:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-role
 sessionName: esc-${context.pulumi.user.login}
 # Optional: scope down the session beyond what the role allows
 policyArns:
 - arn:aws:iam::123456789012:policy/ci-build-minimal
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}

The roleArn and optional policyArns make least-privilege straightforward: each login provider assumes a specific role, and policyArns can scope the session down further. You can use multiple login providers in one environment or separate environments per workflow to match permissions to each job’s needs.

The workflow itself becomes minimal — a single step that authenticates via OIDC and injects the credentials:

permissions:
 contents: read
 id-token: write  # Required for OIDC

steps:
 - name: Fetch secrets from ESC
 uses: pulumi/esc-action@v3
 with:
 environment: '<your-organization>/<your-esc-env>'

The static secrets.* references are gone entirely. Every credential is fetched at runtime through ESC.

Scale: 70+ repos, zero static secrets

We didn’t do this for one or two flagship repos; we rolled it out across every Pulumi provider repository: AWS, Azure, GCP, Kubernetes, and over 60 more. The migration was managed centrally through our ci-mgmt tooling, which generates consistent workflow configurations across all provider repos.

The pattern is the same everywhere:

Each repo has a corresponding ESC environment under a github-secrets/ project.
All workflow-level ${{ secrets.* }} references have been removed.
The pulumi/esc-action step with OIDC auth is the single entry point for all credentials.

When every repo follows the same pattern like this, security posture is much more easily verifiable and auditable.

Auditability and centralized control

Beyond eliminating static secrets, this migration gave us centralized visibility and control that GitHub Secrets cannot provide:

Audit logging. ESC records which credentials were accessed, when, and by which workflow. This is a meaningful improvement over GitHub’s binary “secret was used” signal.
Centralized access policies. Access rules are defined once in ESC rather than scattered across individual repository settings pages.
Single-point rotation. Because ESC environments can import other environments, shared credentials live in a common base that all 70+ repo environments are composed of. Update it once, and every repo picks up the change on its next run.
Dynamic credentials by default. For cloud providers like AWS, Azure, and GCP, ESC fetches credentials via OIDC at open time. There is nothing to rotate because nothing is stored.

What happens if a GitHub Action is compromised

With this architecture in place, here is what an attacker gets if a compromised GitHub Action runs in our CI:

No GitHub Secrets to dump. The repository settings page has no stored secrets for a malicious action to exfiltrate.
OIDC tokens are scoped and short-lived. The GitHub-issued JWT is valid only for the specific workflow run and expires within minutes.
Cloud credentials are ephemeral. Any AWS, Azure, or GCP credentials fetched through ESC are short-lived and scoped to the role assumed during that run.
No persistent access. There are no long-lived tokens to reuse hours or days later.

Compare this to the traditional model, where a single compromised action could exfiltrate AWS access keys that remain valid until someone manually rotates them — which could be weeks or months.

The goal is not to prevent every possible attack. It is to make the blast radius as small as possible when something goes wrong.

Get started

If you want to adopt the same pattern in your own CI pipelines:

Tutorial: Using ESC with GitHub Actions — Step-by-step setup guide.
Announcing the Pulumi ESC GitHub Action — Full feature overview and capabilities.
Configuring OIDC for ESC — Set up OIDC trust between ESC and your cloud providers.
Pulumi ESC documentation — Full documentation for environments, secrets, and configuration.

Your CI secrets do not have to be a liability. With OIDC and Pulumi ESC, they do not have to exist at all.

How Elkjøp Nordic enables self-service infrastructure for developers

George Huang — Fri, 01 Jul 2022 00:00:00 +0000

At PulumiUP 2022, Tomas Jansson, software architect at Elkjøp Nordic, gave a presentation on how to enable developers to self-service infrastructure by using Pulumi’s Automation API.

Elkjøp Nordic is the leading consumer electronics retailer in the Nordics. The company sells consumer electronics, mobile phones, computers, white goods, domestic appliances, and services linked to these products both directly to consumers and to businesses. It is an omnichannel retailer and serves customers both online and through more than 400 stores. Elkjøp has retail outlets in Norway, Sweden, Finland and Denmark, and franchise operations in Greenland and Faroe Islands.

To support the company’s modernization strategy, Tomas developed a self-service infrastructure platform that would enable any developer to easily provision cloud infrastructure that uses approved resources and security practices. The goal was to empower teams at Elkjøp to increase their development velocity by using the cloud, while maintaining guardrails that uphold security and compliance.

Watch the full presentation below or read on to learn the highlights of his presentation:

The Problem: Balancing Agility and Governance

To achieve his goals, Tomas knew he had to solve the problem of balancing freedom with control. On the one hand, he wanted to foster the creativity of his developers and encourage innovation. He wanted his team to feel they were trusted and had the room to create the best solutions they could. On the other hand, he knew he had to maintain some control. Elkjøp needed to keep its applications secure. It also needed good governance so that teams can keep track of who is making changes. Tomas also wanted to limit the impact of deploying untested code. Code that was still being developed needed to be confined to the developer’s own environment. Finding the right balance between independence and control was paramount. Tomas believed this would result in great software and in developers who would enjoy their work.

The Solution: A Self-Service Infrastructure Platform

Tomas’ solution was to build a self-service infrastructure platform. Self-service infrastructure makes it possible for developers to easily access the infrastructure they need while limiting access to resources they don’t need. In effect, self-service infrastructure is a walled garden. Within the garden, developers have all the freedom they need. However, nothing they do affects what’s outside the wall. Code that causes unintended changes only affects them and no one else.

Tomas built an application that provides a “walled garden” of self-service infrastructure. Developers can create the environments they need, including an Azure resource group and a Kubernetes namespace, all through a graphical user interface. Developers can define whether the environment is for development, test or production. They can also assign values, such as which team members should have admin privileges to the associated GitHub repositories or who can contribute. From there, the platform handles provisioning and configuring all of these resources.

Here is an example that shows how a developer can create a repository simply by selecting a few parameters from a drop-down list.

Tomas determined that the best way to implement self-service infrastructure was with Pulumi and its Automation API. By using Pulumi, he could use standard programming languages to define infrastructure while using familiar software tools like IDEs and test frameworks. Pulumi also natively supports Azure and Kubernetes with 100% coverage of each provider’s APIs and same-day access to new features and updates. Finally, the Pulumi Automation API would enable him to program infrastructure automation directly into his application.

The Pulumi Automation API is a programmatic interface for running Pulumi programs without the Pulumi CLI. Automation API encapsulates the functionality of the CLI (pulumi up, pulumi preview, pulumi destroy, pulumi stack init, etc.) but with more flexibility since infrastructure operations can be controlled programmatically.

How the self-service infrastructure platform was built

Here is the workflow for the Elkjøp self-service infrastructure platform.

The self-service platform is a web application that runs the Pulumi Automation API on the server side. The application uses Pulumi to provision and configure every environment resource such as Azure resource groups, Kubernetes namespaces, service principals, and Kubernetes service accounts. Each environment comes with a Git repository that’s used by developers to manage their deployments. The Git repositories are the locus of each environment because they store everything needed to deploy changes such as infrastructure code and environment credentials.

Using the Pulumi Automation API, the application creates an Azure resource group, a team in both Azure AD and GitHub, and a new GitHub repository. The application then grants administrative access to the GitHub repo to members of the AD group. In addition to creating a Git repository for each environment, the application uses stack references for each Azure environment to inject the associated credentials into the GitHub Secret Store. These credentials are scoped to each GitHub environment so that developers can only work in their own environment. If a team has multiple environments, each environment has its own credentials so developers can’t, for example, deploy from a test environment to a production environment. Secrets are managed through the Pulumi Service.

Every new repository comes with Pulumi programs that developers can use to deploy their application. GitHub Actions initiate the deployment. When an action triggers, Pulumi reads the credentials that were generated by the self-service application, allowing Pulumi to deploy new Kubernetes or Azure resources to the environment.

Watch the demo

To illustrate how Elkjøp’s self-service platform works, Tomas presented a simplified demo application that’s based on Elkjøp’s production platform. The code for the application is available at https://github.com/mastoj/pulumi-automation-demo.

The demo is a TypeScript application that includes an API that provisions Azure and GitHub resources. Pulumi Automation API code is used to deploy new stacks by calling handler code, which is a Pulumi program that defines resources and outputs. The handler code can create Azure resource groups or GitHub repositories, and stack references are used so that outputs generated by the Azure resources can be stored in GitHub (e.g., resource group credentials).

Ready to dive into how this works in detail? Watch the demo in the video below starting around 10:20.

Next steps

After watching the demo of a self-service infrastructure platform, you can try it out for yourself by checking out the demo code. You can also learn more about Pulumi Automation API in our documentation.

If you’re new to Pulumi, then follow our getting started guide.

How we manage GitHub at Pulumi with Pulumi

Guinevere Saenger — Tue, 22 Mar 2022 11:29:33 -0700

We recently updated this article to reflect that parentTeamId of the Team resource accepts a string instead of an int since v5.9.1 of the Pulumi Github package.

Ah, GitHub. The home of all developers. The place where we share code. The world’s most awkward social media site. The secret LinkedIn for techies. The tool we use for company org structure, work planning, code ownership, and permissions…

Wait.

That’s quite a lot.

GitHub is good at many things, but a full-on organization management tool it is not.

Have you ever needed your manager to manually enable admin permission on a repo for you? Or have you needed to page the CEO to add you to a team, because your manager was out that day? Have you ever wondered who is on what team? Or which team owns a repo? What if you change teams, or a team changes names? A reorg happens, and the “platform-integrations” team is no more, but we still need to call it that on GitHub because it is the team with all the repository accesses?

When I joined Pulumi in 2021, all of the above happened to me within my first few weeks.

We at Pulumi wanted to reduce this kind of management friction, and we decided to solve it the Pulumi way: with declarative infrastructure using the Pulumi GitHub provider.

The Goal

First, we set a few ground rules. We decided that administrative privileges over Pulumi’s GitHub repositories should flow as follows:

An employee is part of one or more GitHub teams.
Teams have appropriately scoped permissions over each repository they own.
Any Pulumi employee should be able to request code repository access in an official way.

Additionally, any current team membership and repository access should not be disrupted during development of this tool.

Getting Started

Create a new Pulumi program:

$ pulumi new go
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (team-management)
project description: (A minimal Go Pulumi program)
Created project 'team-management'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev) pulumi/prod
Created stack 'prod'

Installing dependencies...

go: downloading github.com/pulumi/pulumi/sdk/v3 v3.26.1
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run 'pulumi up'

$ ls
Pulumi.yaml go.mod go.sum main.go

We now have a pulumi project YAML configuration file and the beginnings of a small Go program all set up.

According to the provider configuration documentation, we need to add a properly scoped token, as well as set “pulumi” as our GitHub organization owner.

$ export GITHUB_TOKEN=YYYYYYYYYYYYYY
$ export GITHUB_OWNER=pulumi

Capturing Current State of Resources

Now, Pulumi is great at creating new infrastructure from scratch via code. But this wasn’t a from-scratch situation. We had to migrate existing resources—GitHub teams—to Pulumi, without disrupting anyone’s access.

Enter Pulumi Import.

What Pulumi Import does, in a nutshell, is find existing infrastructure by unique ID (in the GitHub provider’s case, the team ID), and add them to a Pulumi Stack. You can find the specific import instructions on the registry documentation for each resource. In this case we want the GitHub import instructions.

$ pulumi import github:index/team:Team animals 1234567
Previewing import (pulumi/prod)

 Type Name Plan
 pulumi:pulumi:Stack team-mgmt-prod
 = └─ github:index:Team animals import

Resources:
 = 1 to import
 12 unchanged

Do you want to perform this import? yes
Importing (pulumi/prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 = └─ github:index:Team animals imported

Resources:
 = 1 imported
 1 unchanged

Now that we have imported the Team resource, it is part of our Pulumi Stack. But it is not part of our main.go program yet. Helpfully, Pulumi Import replies with sample code on how to add the imported resource to your Pulumi program, in the language you selected for your project:

Please copy the following code into your Pulumi application. Not doing so will cause Pulumi to report that an update will happen on the next update command.

Please note that the imported resources are marked as protected. To destroy them you will need to remove the `protect` option and run `pulumi update` *before* the destroy will take effect.


package main

import (
 "github.com/pulumi/pulumi-github/sdk/v4/go/github"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := github.NewTeam(ctx, "animals", &github.TeamArgs{
 CreateDefaultMaintainer: pulumi.Bool(false),
 Description: pulumi.String("should get imported"),
 Name: pulumi.String("animals"),
 Privacy: pulumi.String("closed"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

After adding the team to main.go as suggested, this is the output of pulumi preview:

Previewing update (pulumi/prod)

 Type Name Plan Info
 pulumi:pulumi:Stack team-mgmt-prod

Resources:
 2 unchanged

Since pulumi preview shows no changes, we now know that our code reflects the existing infrastructure. It’s a bit funny to think about your program working well when it does nothing, but this was a huge first step in preserving existing infrastructure and ensuring all of our coworkers could continue their daily work uninterrupted!

To finish up, we unprotect the resource:

$ pulumi state unprotect 'urn:pulumi:prod::team-mgmt::github:index/team:Team::animals'

Creating Configuration

Next, we import the remaining teams and refactor the code to handle multiple teams at once. This is also the time to write our organization structure into a YAML configuration file.

---
org: "pulumi-demo--org"
teams:
 - name: "animals"
 slug: "animals"
 description: "A team in charge of animals"
 - name: "newteam"
 slug: "newteam"
 description: "I am a new team"

We then read this information into a Go struct with the necessary properties to pass into the Pulumi provider.

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 Slug string `yaml:"slug"`
}
type Organization struct {
 Org string `yaml:"org"`
 Teams []Team `yaml:"teams"`
}

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

//import accurate team info from yaml
orgFilePath, err := filepath.Abs("./org.yaml")
if err != nil {
 return err
}
yamlFile, err := ioutil.ReadFile(orgFilePath)
if err != nil {
 return err
}
var org Organization
err = yaml.Unmarshal(yamlFile, &org)
if err != nil {
 return err
}

for _, team := range org.Teams {
 ghTeam, err := github.NewTeam(ctx, team.Slug, &github.TeamArgs{
 Description: pulumi.String(team.Description),
 Name: pulumi.String(team.Name),
 Privacy: pulumi.String("closed"),
}, pulumi.Protect(false))
if err != nil {
fmt.Println("encountered error creating new Pulumi GitHub team: ", team.Name)
 return err
}
 return nil
 })
}

Notice how the GitHub provider allows us to use team names to create teams. In contrast, team IDs are only needed for importing to the Stack, not for maintaining state.

Relationships Are Hard

The next step involved a lot of thinking about team memberships and team nesting. GitHub allows teams to be nested. This groups teams by area of responsibility but also allows for certain properties, such as permissions, to be inherited by subteams.

Parent Team Parent
 |
 -------------------------
 | |
Child Teams Child1 Child2
(inherit all permissions)

To establish this relationship, we set a ParentTeamId on child teams. First off, we add a new field to our Teams struct:

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 Slug string `yaml:"slug"`
 ParentTeamId int
 Teams []Team `yaml:"teams"`
}

Here is where things get a little tricky. Any GitHub Team can have subteams, but not every Team has a parent Team. We also did not want to have to hardcode GitHub Team IDs into our configuration files. We needed to meet the following requirements:

Create a parent team, using a team name.
Obtain its team ID.
Create any child teams, also using a team name.
Write the parent team’s ID into the ParentTeamId field of each child team.
Do all of the above in a single pulumi up.

This is where maintaining infrastructure with Pulumi truly shines. In our code, we can use Pulumi Apply to hold on to the promise of a parent team ID, and pass this promise into the appropriate field:

func setupTeams(ctx *pulumi.Context, parentTeam *Team) error {

// set up parent team, i.e. engineering
 ghParentTeam, err := github.NewTeam(ctx, parentTeam.Slug, &github.TeamArgs{
 Description: pulumi.String(parentTeam.Description),
 Name: pulumi.String(parentTeam.Name),
 Privacy: pulumi.String("closed"),
 }, pulumi.Protect(false))
 if err != nil {
 fmt.Println("encountered error creating new Pulumi github team: ", parentTeam.Name)
 return err
 }

//set up nested teams
 for _, childTeam := range parentTeam.Teams {
 // set each child team's parent team ID to the current team ID
 ghChildTeam, err := github.NewTeam(ctx, childTeam.Slug, &github.TeamArgs{
 Description: pulumi.String(childTeam.Description),
 Name: pulumi.String(childTeam.Name),
 Privacy: pulumi.String("closed"),
 ParentTeamId: ghParentTeam.ID(),
 }, pulumi.Protect(false))
 if err != nil {
 fmt.Println("encountered error creating new Pulumi github team: ", childTeam.Name)
 return err
 }
 }
 return nil
}

Running this as part of main.go will result in beautifully nested teams on the GitHub UI. But with Pulumi, we can do even better. We can set pulumi.Parent() on the child teams:

for _, childTeam := range parentTeam.Teams {
 ghChildTeam, err := github.NewTeam(ctx, childTeam.Slug, &github.TeamArgs{#...},
 pulumi.Protect(false),
 pulumi.Parent(ghParentTeam))
 #...
}

With this, we can see the relationship between the resources on pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + └─ github:index:Team newteam created
 + └─ github:index:Team newchildteam created
Resources:
 + 2 created
2 unchanged

Which will be reflected on the Pulumi Stack’s Graph View in the Pulumi Service:

Adding this relationship to Pulumi is mostly a visual nicety in our case; however, certain Pulumi properties get inherited in a helpful way. Note that on GitHub, it is possible to create multiple layers of teams, whereas the example code in this blog post only shows a single layer of subteam structure. We are small enough at Pulumi to not need more, but you may want to expand on the solution if you do.

Adding People

Next, teams should have members! Let’s add them to the YAML config:

org: "pulumi-demo-org"
teams:
 - name: "animals"
 slug: "animals"
 description: "A team in charge of animals"
 members:
 - username: "platypus"
 - username: "owlcat"

A TeamMembership in GitHub is a cross reference between a Team and a User. Fortunately these do not need to be explicitly imported, as they are merely establishing relationships between GitHub Users and Teams. We can add TeamMemberships to the Stack with a Members struct, a new Members[] field on the Team struct, and an extra function. Again, Pulumi lets us use the promised output of the team ID to set the TeamId field in the TeamMembership:

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 ParentTeamId int
 Teams []Team `yaml:"teams"`
 Slug string `yaml:"slug"`
 Members []Member `yaml:"members"`
}

type Member struct {
 UserName string `yaml:"username"`
 Role string `yaml:"role"`
}

#...

func addMembers(ctx *pulumi.Context, members []Member, team *github.Team, teamName string) error {
 for _, member := range members {
 // unique name for TeamMembership
 utmName := teamName + "-" + member.UserName
 _, err := github.NewTeamMembership(ctx, utmName, &github.TeamMembershipArgs{
 TeamId: team.ID(),
 Username: pulumi.String(member.UserName),
 })
 if err != nil {
 return err
 }
 }
 return nil
}

Running pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + ├─ github:index:TeamMembership animals-owlcat created
 + └─ github:index:TeamMembership animals-platypus created

Resources:
 + 2 created
 4 unchanged

Alright! So now we have

Imported existing teams to a Pulumi Stack to track and organize all GitHub teams from a central repository, and
Added team members to teams.

That’s pretty great so far! While there are many org chart tools, what makes this one useful to us is that we can declare the desired org state in a config file, and then let Pulumi figure out what changes should be applied. Let’s automate that!

Add CI

In order for this tool to be used by everyone, we keep code and configuration in a GitHub repository. We can use Pulumi’s GitHub Action to run a pulumi preview on a pull request, and a pulumi up on merge to main.

Here’s what that looks like on the pull request:

name: Pull Request preview
on:
 pull_request:
 workflow_dispatch:

env:
 AWS_REGION: us-west-2
 GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_OWNER: pulumi

jobs:
 pull-request:
 runs-on: ubuntu-latest
 name: Preview
 steps:
 - uses: actions/checkout@v2
 - name: Install Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.17
 - name: Build Binary
 run: |
 go mod tidy
 go build main.go
 - name: Pulumi Preview
 uses: pulumi/actions@v3
 with:
 command: preview
 refresh: true
 stack-name: pulumi/prod
 comment-on-pr: true

And the pulumi up on merge to main:

name: Update
on:
 push:
 branches:
 - "main"
 workflow_dispatch:

env:
 AWS_REGION: us-west-2
 GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_OWNER: pulumi

jobs:
 main:
 runs-on: ubuntu-latest
 name: Update
 steps:
 - uses: actions/checkout@v2
 - name: Install Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.17
 - name: Build Binary
 run: |
 go mod tidy
 go build main.go
 - name: Pulumi update
 uses: pulumi/actions@v3
 with:
 command: up
 refresh: true
 stack-name: pulumi/prod

Note that we are calling refresh: true in both Workflows, which uses Pulumi Refresh to make sure that the existing GitHub resources are aligned with the resource state in our Stack.

Now, anyone with access to the GitHub management repo can:

Create, re-parent, delete, or rename teams and re-create any memberships via pull request to the config file.
Add and remove team members via pull request.
Audit and explicitly maintain org structure via git history and review processes.

But of course… there’s more!

Managing Permissions

The next step involved a lot of thinking about repository permissions and permission access.

Shortly after I joined Pulumi, my team was combined with another team. They brought along all of their repositories—none of which I had access to. Moreover, all of our team names were outdated. I could rename the teams using pulumi-github… but I could not transfer all of the repository access we all needed to our new team. Yet.

On GitHub, parent teams pass permissions down to child teams. But what if we wanted permissions to be more granular?

Here’s where TeamRepositories come in. A TeamRepository has three fields: a team ID, a repository name, and a permission. Just like TeamMemberships, this resource is a link between a Team and a Repository and does not need to be imported.

We gave each team their own config file with permission levels listing repository names:

team: "animals"
permissions:
 - role: "admin"
 repos:
 - "capybaras"
 - "owls"
 - role: "maintain"
 repos:
 - role: "push"
 repos:
 - role: "read"
 repos:

This can be read into our Pulumi program as follows:

// TeamPermissions describes a github team and the levels of permissions available (i.e. admin, maintain)
type TeamPermissions struct {
 TeamName string `yaml:"team-name"`
 Permissions []Permission `yaml:"permissions"`
}

// Permission is a single permission and a list of repos with that permission.
type Permission struct {
 Role string `yaml:"role"`
 Repos []string `yaml:"repos"`
}

#...

func setupTeamRepos(ctx *pulumi.Context, team *github.Team, teamName string) error {

 repoFilePath, err := filepath.Abs("./team-repos/" + teamName + ".yaml")
 if err != nil {
 fmt.Println(err.Error())
 return err
 }
 yamlFile, err := ioutil.ReadFile(repoFilePath)
 if err != nil {
 return err
 }

 var teamPerms TeamPermissions
 err = yaml.Unmarshal(yamlFile, &teamPerms)
 if err != nil {
 return err
 }
 for _, permission := range teamPerms.Permissions {
 for _, repo := range permission.Repos {
 _, err := github.NewTeamRepository(ctx, fmt.Sprintf("%s-%s-%s", repo, teamName, permission.Role), &github.TeamRepositoryArgs{
 Permission: pulumi.String(permission.Role),
 Repository: pulumi.String(repo),
 TeamId: team.ID(),
 })

 if err != nil {
 return err
 }
 }
 }
 return nil
}

And the output of running pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + ├─ github:index:TeamRepository owls-animals-admin created
 + ├─ github:index:TeamRepository capybaras-animals-admin created


Resources:
 + 2 created
 2 changes. 4 unchanged

With this addition, we can now combine teams with zero access disruptions as follows:

Create the new team in the org config.
Create a TeamRepository config for the new team with desired permission levels.
Open a pull request and let CI and Pulumi do the rest.

Now we can develop standards around repository permissions based on teams’ roles and areas of responsibility, and we have tooling in place that can maintain these standards for everyone in the org to see. In fact, we can standardize the meaning of “code ownership” via access levels in this way—one centralized management location rather than asking your grandboss to dig through the UI for you.

Future challenges

We want to automate some of these steps: adding new teammates to the org should also automatically add them to this repository so they can self-serve their onboarding and see Pulumi work in action!

To prevent state drift, we will run a regular reconciliation job in CI.

Of course, we will add other Pulumi providers on top of this org structure so we can sync our other tooling.

Supercharging our GitHub Action with the Automation API

Paul Stack — Mon, 01 Mar 2021 00:00:00 +0000

Guest Article: Simen A. W. Olsen is a Software Architect and Manager at Bjerk, a software development agency based in Oslo, Norway. He joins Paul Stack to talk about the new GitHub Action powered by the Pulumi Automation API.

In this article, we’ll take a look at how our GitHub Action has been re-architected to be built on top of our Automation API.

As a GitHub Actions launch partner in 2018, we released our original GitHub Action, built as a Docker container action. This Action packaged all the prerequisites with the Pulumi CLI to let users run Pulumi commands. As the Pulumi ecosystem has grown, so too did the container run in GitHub Actions.

Since then, GitHub released an alternative way to create a GitHub Action by writing JavaScript. At Pulumi, and in our community, we love expressing everything we possibly can in code, so the new JavaScript support got us quite excited about the “better together” possibilities! Enter Pulumi’s new Automation API, a new feature we launched in public preview in October 2020. This new capability lets you build infrastructure as code—in pure code—without even needing to run a CLI. This sounded like a perfect match for a JavaScript Action. Creating a JavaScript Action allows us to programmatically invoke the right Pulumi commands in response to actions, without needing to awkwardly script a CLI. Instead, we were able to leverage the Automation API to fundamentally rethink our GitHub Action.

Under the Hood

Let’s take a look at the new JavaScript-based GitHub Action and see how it uses Automation API to execute Pulumi:

const main = async () => {
 const config = await makeConfig();

 if (environmentVariables.PULUMI_ACCESS_TOKEN !== '') {
 await pulumiCli.run('login');
 } else if (config.cloudUrl) {
 await pulumiCli.run('login', config.cloudUrl);
 }

 const workDir = resolve(
 environmentVariables.GITHUB_WORKSPACE,
 config.workDir,
 );

 const stack = await LocalWorkspace.selectStack({
 stackName: config.stackName,
 workDir: workDir,
 });

 const onOutput = (msg: string) => {
 .....
 };

 const actions: Record<Commands, () => Promise<string>> = {
 up: () => stack.up({ onOutput }).then((r) => r.stdout),
 refresh: () => stack.refresh({ onOutput }).then((r) => r.stdout),
 destroy: () => stack.destroy({ onOutput }).then((r) => r.stdout),
 preview: async () => {
 const { stdout, stderr } = await stack.preview();
 onOutput(stdout);
 onOutput(stderr);
 return stdout;
 },
 };

 const output = await actions[config.command]();

 const outputs = await stack.outputs();

 for (const [outKey, outExport] of Object.entries(outputs)) {
 core.setOutput(outKey, outExport.value);
 if (outExport.secret) {
 core.setSecret(outExport.value);
 }
 }
};

The Action code is clean and concise because the power lies in the Automation API itself. The Action uses the Automation API to drive Pulumi as part of your GitHub Actions workflows. This means that your workflows can take advantage of changes to Pulumi and the Automation API as soon as they’re released! And, of course, you can always write your own custom logic using the Automation API like our Action does.

Customizing Your Environment

The new Action puts you in control of the environment that it runs on. By comparison, the previous Docker-based Action bundled specific tool versions, which left your CI pipeline at the mercy of our opinionated container. Now, if you want to pin to specific versions of NodeJS / Python / Go / .NET or even Pulumi, you have that power!

You can even include a matrix strategy to test your Pulumi code against different language runtime versions and different operating systems:

name: 'Sample Pulumi NodeJS Application'
on:
 pull_request:

jobs:
 test-workflow:
 runs-on: ${{ matrix.os }}
 name: Testing Pulumi Preview on ${{ matrix.os }} with NodeJS ${{ matrix.node_version }}
 strategy:
 matrix:
 os: [ubuntu-latest, macos-latest, windows-latest]
 node_version: [10.x, 12.x, 14.x]
 fail-fast: false
 steps:
 - uses: actions/checkout@v2
 - uses: actions/setup-node@v1
 with:
 node-version: ${{ matrix.node_version }}
 - name: Install pulumi
 uses: pulumi/action-install-pulumi-cli@v1.0.1
 - run: npm install
 - uses: pulumi/actions@v2
 env:
 PULUMI_CONFIG_PASSPHRASE: not-a-secret
 with:
 command: preview
 cloud-url: file://~
 stack-name: dev

Cloud Interaction

The previous Docker-based Action would look for the GOOGLE_CREDENTIALS environment variable; if your workflow set that variable, the Action would run the following commands on your behalf:

gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud --quiet auth configure-docker $GOOGLE_DOCKER_HOSTNAME_LIST

With the new Action, you’re fully in control of authentication to cloud providers. To authenticate with AWS / Azure / Google Cloud, you can now use their official Actions:

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1
 with:
 aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
 aws-region: ${{ secrets.AWS_REGION }}
 aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

 - uses: azure/login@v1
 with:
 creds: ${{ secrets.AZURE_CREDENTIALS }}

 - uses: google-github-actions/setup-gcloud@v0
 with:
 service_account_key: ${{ secrets.GCP_KEY }}
 project_id: ${{ env.PROJECT_ID }}
 export_default_credentials: true

We recommend that you store all sensitive environment variables, like cloud credentials, in the Actions secret storage and consume them using the secrets attribute on your workflow’s action.

Faster

The new GitHub Action is much smaller and no longer needs to download a (very!) large Docker container. In the Docker-based Action, we frequently saw workflows take up to 3 minutes to download the container. The new Action is much faster; let’s take a look:

Old Docker-based Action (v1)

New JavaScript-based Action, powered by Automation API (v2)

The new Pulumi GitHub Action is ready for you to use. Here’s how to get started:

Already use the v1 Pulumi GitHub Action? Migrate to the new one with our migration guide.
New to Pulumi and GitHub Actions? Use our getting started guide to set up your first Pulumi workflow.

We’re eager to hear your feedback on this new Action. If you haven’t already signed up for our Community Slack, it’s quick and easy! You can join in on conversations you like and get help from other community members, as well as the Pulumi Team. That’s it for now!

Infrastructure CI/CD With Github Actions and Pulumi

Paul Stack — Mon, 14 Dec 2020 00:00:00 +0000

GitHub Actions help automate tasks within your software development life cycle. At Pulumi, we use GitHub Actions internally as part of every build/release cycle, and we run these tasks many, many times per day. This helps us to automate our CI/CD process and eliminate manual steps. Pulumi and many of our customers also deliver infrastructure resources as part of a CI/CD process: delivering infrastructure and applications in a single, integrated pipeline.

Better Automation with GitHub Actions and Pulumi

Today, we’re pleased to announce that Pulumi is available as a verified GitHub Action in the GitHub Marketplace. It’s now easier than ever to start automating your infrastructure deployments with Pulumi and GitHub Actions. Our new guide shows you how to get started with GitHub Actions and Pulumi. Here are just a few great examples of how to use the Pulumi GitHub Action that will simplify your cloud engineering tasks:

Master Builds

name: Pulumi
on:
 push:
 branches:
 - master
jobs:
 up:
 name: Update
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v1
 - uses: pulumi/actions@v1
 with:
 command: up
 env:
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

Pull Request Builds

name: Pulumi
on:
 - pull_request
jobs:
 preview:
 name: Preview
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v2
 - uses: pulumi/actions@v1
 with:
 command: preview
 env:
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 IS_PR_WORKFLOW: true
 COMMENT_ON_PR: 1

Announcing the New Pulumi CI/CD Integration Assistant

We often hear from developers who want to setup CI/CD automation for their stacks but don’t have the time to figure out how to configure a working pipeline. Today, we are happy to announce the launch of the CI/CD Integration Assistant to make setting up CI/CD automation more straightforward. You can easily get started using popular CI/CD services without the hassle of the initial setup.

The integration assistant guides you with instructions based on your preferred CI/CD service and provides the customized pipeline configuration for CI/CD services that you can add to your Version Control System (VCS) repository.

We are super excited to offer GitHub Actions support as part of the CI/CD Integration Assistant and other popular tools like Azure Pipelines and more. If you’d like to request support for more CI/CD services, please use this convenient template on the Pulumi GitHub repo.

Managing Your Infrastructure with a VCS

You may already be familiar with Git-based services such as GitHub, Bitbucket, and GitLab. Adding your code to a central VCS allows you to secure your codebase against failures and boost team collaboration. When you add an automated CI/CD pipeline, your team will deliver software safely and efficiently by ensuring that automated tests are passed prior to every deployment. This process is valuable for application code and infrastructure code.

The Assistant also goes beyond helping you get started with an automated pipeline. It works throughout the Pulumi Service providing hints for configuring a VCS for projects that are not already configured. It can also help identify updates to stacks that were not run in an automated pipeline (in the activity tab). These additions will help you quickly identify the stacks where best practices are not followed and take corrective actions. You can access the Assistant in the Integrations option of your stack’s settings page to help you through the whole process of configuring a VCS to setting up a working automated pipeline.

Learn more about the CI/CD Integration Assistant in our user guide.

We are eager to hear your feedback on how we can improve your experience with this feature. If you haven’t already signed up for our Community Slack, it’s quick and easy! You can join in on conversations you like and get help from other community members, as well as the Pulumi Team. That’s it for now! 👋

Deploying an OAuth Server for Netlify's CMS

Zephyr Zhou — Wed, 09 Sep 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

In our previous post, we deployed our CMS app on AWS instead of Netlify. We couldn’t use Netlify’s Identity Service, which manages GitHub access to Netlify CMS, because we deployed on AWS. As a result, we needed to implement an external OAuth Server.

We used Netlify’s Go example to deploy on ECS Fargate and configure the domain and certificate. To deploy the application on Fargate, we used a Typescript Pulumi project. This is a polyglot application where the OAuth server is implemented in Go and the infrastructure is deployed with Typescript. We’ll show how we accomplished the deployment.

Backend is a package that supports communications between Netlify CMS and repositories like GitHub, GitLab, and Bitbucket. The Pulumi example code uses backend to authenticate to the CMS. The OAuth Server also enables authorization for GitLab and Bitbucket by changing the callback URL to https://{{YOUR_OAUTH_SERVER_URL}}/callback/{{YOUR_BACKEND_NAME}}. To learn more about configuring OAuth Server, refer to the Environment Variable and Pulumi Stack Configuration section.

Building the OAuth Server

Netlify’s CMS website provides External OAuth Client examples for various languages and platforms. We used the Go example as a template for our server.

In Netlify’s example, the ./dotenv/dotenv.go retrieves environment variables from a file. The main.go file uses goth to instantiate the OAuth provider. The ./randstr/randstr.go file generates a random string for the SESSION_SECRET environment variable, which is used for authentication. However, Pulumi can implement both functions, replacing the code in goth by using Pulumi’s random package.

// Create a random string and also mark its `result` property as a secret,
// so it is not stored in plaintext in the stack's state.
const sessionSecretRandomString = new random.RandomPassword("random", {
 length: 32,
}, { additionalSecretOutputs: ["result"] });

Because we replaced those functions, we can remove the randstr and dotenv folders in the OAuth client example. Later, we’ll show how to pass the environment variables when we create the Fargate service.

Setting GitHub Scope

The default GitHub scope in Netlify’s example is read-only. When we call the github.New function, read and write access must be set to edit a target repository.

We set GITHUB_SCOPE to public_repo, granting read and write access to a public repository. The GITHUB_SCOPE value is read and parsed when the Fargate Service is created, which is set in the Github.New function in the OAuth server. We can also set GITHUB_SCOPE to a list of scope values.

githubScope := os.Getenv("GITHUB_SCOPE")
 if githubScope == "" {
 goth.UseProviders(
 github.New(
 os.Getenv("GITHUB_KEY"), os.Getenv("GITHUB_SECRET"),
 // concatenate with the host name
 fmt.Sprintf("%s/callback/github", host),
 "public_repo",
 ),
 bitbucket.New(
 os.Getenv("BITBUCKET_KEY"), os.Getenv("BITBUCKET_SECRET"),
 fmt.Sprintf("%s/callback//bitbucket", host),
 ),
 gitlabProvider,
 )
 } else {
 scopeArray := strings.Split(githubScope, ",")
 goth.UseProviders(
 github.New(
 os.Getenv("GITHUB_KEY"), os.Getenv("GITHUB_SECRET"),
 // concatenate with the host name
 fmt.Sprintf("%s/callback/github", host),
 scopeArray...,
 ),
 bitbucket.New(
 os.Getenv("BITBUCKET_KEY"), os.Getenv("BITBUCKET_SECRET"),
 fmt.Sprintf("%s/callback//bitbucket", host),
 ),
 gitlabProvider,
 )
 }

Creating the OAuth Server Image

To build our OAuth server container, replace the working directory in the Dockerfile with the path to your repository. Setting the working directory copies the application when Docker builds the image.

WORKDIR /go/src/github.com/pulumi/aws-ts-netlify-cms-and-oauth/cms-oauth

Infrastructure

Deploying to Fargate

We’ll use the Hello Fargate Example as a template to deploy the OAuth server to AWS Fargate. This example, written in Typescript, creates an ECS cluster, an Application Load Balancer (alb) with a listener, and a Fargate Service. We’ll change the script to deploy our OAuth server Docker image.

We set the alb port to 443 to use HTTPS, which is more secure when passing the CMS access token.

// Define an ec2 application load balancer alb to distribute incomming application traffic across multiple targets, such as EC2 instances, in multiple Availability Zones.
const alb = new awsx.elasticloadbalancingv2.ApplicationLoadBalancer(
 "net-lb", { external: true, securityGroups: cluster.securityGroups });

// alb need a listener to listen to 443 the standard port for the HTTPS traffic, certificate is using the certificate we created above
const web = alb.createListener("web", {
 port: 443,
 external: true,
 protocol: "HTTPS",
 certificateArn: certificate.arn
});

We create a target group for the load balancers to distribute traffic. The target group port defaults port 80 if it’s not set.

let inputTargetGroupPort: pulumi.Input<number> = cmsStackConfig.targetGroupPort!;

if (inputTargetGroupPort === undefined) {
 inputTargetGroupPort = 80;
}

// when Listener Rule is satisfied then traffic is route to this target group.
const tg = alb.createTargetGroup("oauth-tg", {
 port: inputTargetGroupPort,
 loadBalancer: alb
});

We can also set the target group port as an environment variable with pulumi config set cms-OAuth:targetGroupPort. The load balancer implements a listener and a rule to forward all requests to the target group.

// when the request are forwarded then every requests are send to the target group we created
new awsx.lb.ListenerRule("oauth-listener-rule", web, {
 actions: [{
 type: "forward",
 targetGroupArn: tg.targetGroup.arn,
 }],
 conditions: [{
 field: "path-pattern",
 values: "/*", //wildcard says every request would be send
 }],
});

Environment Variables and Pulumi Stack Configuration

The Fargate service requires several environment variables to start. Here are the variables we need to configure:

HOST is the URL for the OAuth Server, and we set it by running pulumi config set pulumi-website-cms:targetDomain <OAuth server URL>
SESSION_SECRET is generated by using Pulumi’s random package
GITHUB_SECRET and GITHUB_TOKEN are provided by GitHub when the GitHub OAuth application is registered. Netlify provides instructions for obtaining these credentials
The targetDomain is the URL of our CMS
GITHUB_SCOPE sets the type of access the CMS has for the target repository

We can set the github key and secret in the stack configuration. The –secret flag encrypts your GitHub secret. It is good practice to use the –secret flag, because appending the secret in the command line stores it in the command history. The –secret flag lets you enter the secret at the prompt, which hides the value.

$ pulumi config set netlify-cms-oauth-provider-infrastructure:githubKey {{YOUR_GITHUB_KEY}}
$ pulumi config set --secret netlify-cms-oauth-provider-infrastructure:githubSecret
$ {{YOUR_GITHUB_SECRET}}

When the Fargate service is created, it uses the environment variables set in the Pulumi project file and generates the session secret.

// Create a Fargate service task that can scale out.
const appService = new awsx.ecs.FargateService("app-svc", {
 cluster,
 taskDefinitionArgs: {
 container: {
 image: img,
 memory: 128 /*MB*/,
 portMappings: [ tg ],
 environment: [
 {
 name: "HOST",
 // The target domain which would concatenate with callbacks in main.go
 value: pulumi.interpolate `https://${cmsStackConfig.targetDomain}`
 },
 {
 name: "SESSION_SECRET",
 value: sessionSecretRandomString.result
 },
 {
 name: "GITHUB_KEY",
 value: cmsStackConfig.githubKey
 },
 {
 name: "GITHUB_SECRET",
 value: cmsStackConfig.githubSecret
 },
 {
 name: "TARGET_PORT",
 value: pulumi.interpolate `${inputTargetGroupPort}`
 },
 {
 name: "GITHUB_SCOPE",
 value: inputGithubScope
 },
 ]
 },
 },
 desiredCount: 1,
});

The environment variables are passed to the OAuth server in main.go. For example, the callback URL is the targetDomain with callback and the repository provider appended, e.g., https://<targetDomain>/callback/github/. You can also specify a different repository provider, such as Bitbucket or GitLab.

 github.New(
 os.Getenv("GITHUB_KEY"), os.Getenv("GITHUB_SECRET"),
 // concatenate with the host name
 fmt.Sprintf("%s/callback/github", host),
 "public_repo",
 )

We also pass TARGET_PORT to targetGroupPort, which is the port used to serve the application. We can check if this variable is set in main.go if we need to test locally without Pulumi.

targetGroupPort := os.Getenv("TARGET_PORT")
 if targetGroupPort == "" {
 targetGroupPort = "80"
 }
 listenPort := ":" + targetGroupPort
 fmt.Print("Started running on", listenPort, "\n")
 // listen on port 80 where we created the target group
 fmt.Println(http.ListenAndServe(listenPort, nil))

Create a Certificate for the OAuth Server

We use the same process for creating a certificate that’s documented in our previous post, but we use only the code for creating the certificate and discard the rest. The certificate used with the load balancer must be set to the same region as the fully qualified domain name (FQDN). The domain name in the certificate must be validated.

// Get a east provider
const eastRegion = new aws.Provider("east", {
 profile: aws.config.profile,
 region: "us-east-1", // Per AWS, ACM certificate must be in the us-east-1 region.
});

// Creating a Certificate for the given domain.
const certificate = new aws.acm.Certificate("certificate", {
 domainName: cmsStackConfig.targetDomain,
 validationMethod: "DNS",
}, { provider: eastRegion });

// Split given domain in the configuration to domain and subdomain
const domainParts = getDomainAndSubdomain(cmsStackConfig.targetDomain);

// Get the zone of the given domain
const hostedZoneId = aws.route53.getZone({ name: domainParts.parentDomain }, { async: true }).then(zone => zone.zoneId);

// The temporation record for the validation domain has 10 to be live
const tenMinutes = 60 * 10;

/**
 * Create a DNS record to prove that we _own_ the domain we're requesting a certificate for.
 * See https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-validate-dns.html for more info.
 */
const certificateValidationDomain = new aws.route53.Record(`${cmsStackConfig.targetDomain}-validation`, {
 name: certificate.domainValidationOptions[0].resourceRecordName,
 zoneId: hostedZoneId,
 type: certificate.domainValidationOptions[0].resourceRecordType,
 records: [certificate.domainValidationOptions[0].resourceRecordValue],
 ttl: tenMinutes,
});

We configure the Alias Record with the application load balancer settings.

// Creates a new Route53 DNS record pointing the domain to the CloudFront distribution.
function createAliasRecord(
 targetDomain: string, lb: awsx.elasticloadbalancingv2.ApplicationLoadBalancer): aws.route53.Record {
 const domainParts = getDomainAndSubdomain(targetDomain);
 const hostedZoneId = aws.route53.getZone({ name: domainParts.parentDomain }, { async: true }).then(zone => zone.zoneId);
 return new aws.route53.Record(
 targetDomain,
 {
 name: domainParts.subdomain,
 zoneId: hostedZoneId,
 type: "A",
 aliases: [
 {
 name: lb.loadBalancer.dnsName,
 zoneId: lb.loadBalancer.zoneId,
 evaluateTargetHealth: true,
 },
 ],
 });
}
// Create the aliasRecord with targetdomain and application load balancer
const aRecord = createAliasRecord(cmsStackConfig.targetDomain, alb);

GitHub Workflow (Optional)

We can automate our build with GitHub Workflow. The build-and-deploy.yml file in cms-oauth/.github/workflows/ is used by GitHub Actions to build our application. Note that the workflow uses the repository secret set in GitHub.

Finishing Up

Our last task is to update the CMS configuration file. We combine the CMS and CMS OAuth Server deployment by setting site_domain and URL in ./public/config.yml of the CMS folder.

backend:
 name: github
 # Replace this with the GitHub repo you want to make change
 repo: github-username/target-github-repo
 # This site_domain and base_url are placeholders, replace with the correct one.
 site_domain: https://some-cms-domain.com # CMS's domain
 base_url: https://some-oauth-domain.com # OAuth Server's domain

The site_domain is the CMS URL, and the base_url is the OAuth Server URL. They are the same as the targetDomain variable we set in the stack configuration.

Up and Running

Congratulations on building and deploying both CMS web application and the OAuth client-server! If everything works perfectly, you should see the Login with GitHub button and click it to redirect to the GitHub login page.

People with access can now use the CMS.

You can find the project code for the CMS and the OAuth Client Server on GitHub.

Special Thanks to:

Tony Alves @talves for providing the template to separate CMS as a stand-alone React App
Igor Kuznetsov @igk1972 for providing go code example for providing the OAuth Client-Server source code
Paul Stack @stack72 for developing Pulumi example of deploying Dockerized App using ECS Fargate and Pulumi example of deploying the static website on AWS
Everyone at Pulumi who helped me out with this project.

Deploying Netlify CMS on AWS with Pulumi

Zephyr Zhou — Tue, 01 Sep 2020 00:00:00 +0000

Netlify CMS is an open-source content management system that provides UI for editing content and adopting Git workflow. Initially, we want to take advantage of it to increase efficiency to edit Pulumi’s website. However, during development, we found few examples are deploying the CMS application on AWS instead of Netlify, its home platform. Therefore, in this blog post, we would like to share how to organize Netlify’s file structure and use Pulumi to store the content on S3 buckets, connect to CloudFront, and configure certificate in Certificate Manager.

Because we are deploying CMS on AWS, we can not use Netlify’s Identity Service, which manages the access token sent by Github API. We will introduce how to write the External OAuth Client-Server and deploy it on AWS in the next post.

For this project, we use Netlify CMS’s Github backend. Our CMS app changes website content stored in another Github repository under the same account. The website uses Hugo as the static site generator.

Extracting CMS As a Stand-alone React App

The starter template provided by Netlify implements the CMS application inside the target repository. We extract the CMS application from the target repository and copy it in another repository under the same Github account to improve the modularity.

We can use the templates from @talves to extract the CMS as a stand-alone React application.

CMS Configuration File

The config.yml file configures Netlify CMS to look for content segments that can be edited and which backend instructions it needs to follow.

The template, cms/public/index.html contains the entry point for React to insert the generated content under <div id=root>. We can change website’s title, but more importantly, we linked the configuration file cms/public/config.yml of the Netlify CMS by following the instructions.

The template uses config.json in ./src/components/NetlifyCMS/config.json, and sets CMS Manual Init in file ./src/components/NetlifyCMS/setup.js and file ./src/components/NetlifyCMS/index.js.

Alternatively, we can add the path to the config.yml file in index.html and put config.yml in the public folder. This reduces the steps to set the CMS init step, which can be confusing to beginners.

Point the CMS to Another Repository

The template separates the CMS as a stand-alone React app, while the CMS remains in the target repository. We can change the value field repo: in the backend: field of the config.yml file to point to the specific repository that the CMS can change.

File Structure

Simplifying the file and deleting unneeded parameters produces this:

The ./src/index.js file is the entry point for React to create the App class inside ./public/index.html’s <div id=root>. App (defined in ./src/App.js) is a React Component that returns the NetlifyCMS React component defined inside ./src/components/NetlifyCMS/index.js. We can also specify the custom preview and custom widget for the CMS app in this file.

To preview the CMS app content, run yarn start.

Now that we have preprocessed the file structure. Let’s deploy the CMS onto the AWS!

Implement Infrastructure

First, we create an infrastructure folder to store the Pulumi project. To create a new Pulumi project, run pulumi new typescript to generate a typescript template.

We can use Pulumi’s static website example as a model for deploying our application.

Step 1: Build the CMS App

We need to build the website with yarn build. This creates a build folder under the root directory that holds the compiled version of the website contents to be uploaded to an S3 bucket. Once uploaded, the S3 bucket contains a working version of our CMS app, which reduces the number of files to upload.

Step 2: Fetch Configuration

The first step is to create a stack configuration with our stack name.

// Replace "pulumi-website-cms" with your stack name
const stackConfig = new pulumi.Config("pulumi-website-cms");
const config = {
 pathToWebsiteContents: stackConfig.require("pathToWebsiteContents"),
 targetDomain: stackConfig.require("targetDomain"),
 certificateArn: stackConfig.get("certificateArn"),
};

The configuration takes three parameters set with the Pulumi CLI

$ pulumi config set pulumi-website-cms:pathToWebsiteContent ../build

The pathToWebsiteContent parameter specifies the path to the directory with the website content. This directory is synched to the S3 bucket by using the crawlDirectory method. We pass in the build folder, which is the directory that stores our built CMS website.

Similarly, we can set the CMS’s target domain name and the optional certificate arn parameter with stackConfig.get instead of stackConfig.require to get the value. For the target domain, we can use any subdomain name of an existing parent domain, and the code creates the subdomain under the parent domain automatically. For the certificate arn, if we already have a certificate registered for CMS, then we can pass it in as the certificate arn. Otherwise, the code generates a new certificate.

# replace the targetDomain with the domain name for your CMS
$ pulumi config set pulumi-website-cms:targetDomain https://some-cms-domain.com
# replace the value of certificateArn with the correct one
$ pulumi config set pulumi-website-cms:certificateArn arnarnarnxxx

Note that the ACM certificate has to be in the same as the project AWS region.

$ pulumi config set aws:region us-east-1

Step 3: Bucket Creation

We create the S3 bucket.

// contentBucket is the S3 bucket that the website's contents will be stored in.
const contentBucket = new aws.s3.Bucket("contentBucket",
 {
 bucket: config.targetDomain,
 acl: "public-read",
 // Configure S3 to serve bucket contents as a website. This way S3 will automatically convert
 // requests for "foo/" to "foo/index.html".
 website: {
 indexDocument: "index.html",
 },
 });

We crawl the content directory and convert them to bucket objects that are the content for the site.

function crawlDirectory(dir: string, f: (_: string) => void) {
 const files = fs.readdirSync(dir);
 for (const file of files) {
 const filePath = `${dir}/${file}`;
 const stat = fs.statSync(filePath);
 if (stat.isDirectory()) {
 crawlDirectory(filePath, f);
 }
 if (stat.isFile()) {
 f(filePath);
 }
 }
}

// Sync the contents of the source directory with the S3 bucket, which will in-turn show up on the CDN.
const webContentsRootPath = path.join(process.cwd(), config.pathToWebsiteContents);
console.log("Syncing contents from local disk at", webContentsRootPath);
crawlDirectory(
 webContentsRootPath,
 (filePath: string) => {
 const relativeFilePath = filePath.replace(webContentsRootPath + "/", "");
 const contentFile = new aws.s3.BucketObject(
 relativeFilePath,
 {
 key: relativeFilePath,

 acl: "public-read",
 bucket: contentBucket,
 contentType: mime.getType(filePath) || undefined,
 source: new pulumi.asset.FileAsset(filePath),
 },
 {
 parent: contentBucket,
 });
 });

We also create a private CDN request log bucket for storing logs that we can use for debugging the application.

// logsBucket is an S3 bucket that will contain the CDN's request logs.
const logsBucket = new aws.s3.Bucket("requestLogs",
 {
 bucket: `${config.targetDomain}-logs`,
 acl: "private",
 });

Step 4: Certificate Creation and Validation

If the certificateArn is not provided as a configuration, the code will create a new certificate. We set the AWS region to match the Project region, which ACM certificate requires.

Then we create a new certificate.

const certificate = new aws.acm.Certificate("certificate", {
 domainName: config.targetDomain,
 validationMethod: "DNS",
 }, { provider: eastRegion });

Next, we create a Route53 DNS Record which is needed when requesting the ACM certificate.

const domainParts = getDomainAndSubdomain(config.targetDomain);
const hostedZoneId = aws.route53.getZone({ name: domainParts.parentDomain }, { async: true }).then(zone => zone.zoneId);

/**
 * Create a DNS record to prove that we _own_ the domain we're requesting a certificate.
 * See https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-validate-dns.html for more info.
 */
const certificateValidationDomain = new aws.route53.Record(`${config.targetDomain}-validation`, {
 name: certificate.domainValidationOptions[0].resourceRecordName,
 zoneId: hostedZoneId,
 type: certificate.domainValidationOptions[0].resourceRecordType,
 records: [certificate.domainValidationOptions[0].resourceRecordValue],
 ttl: tenMinutes,
});

The getDomainAndSubdomain method for separates the parent domain and child domain.

We validate the certificate by setting the CertificateValidation resource and set the certificateArn.

/**
 * This is a _special_ resource that waits for ACM to complete validation via the DNS record
 * checking for a status of "ISSUED" on the certificate itself. No actual resources are
 * created (or updated or deleted).
 *
 * See https://www.terraform.io/docs/providers/aws/r/acm_certificate_validation.html for slightly more detail
 * and https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_acm_certificate_validation.go
 * for the actual implementation.
 */
 const certificateValidation = new aws.acm.CertificateValidation("certificateValidation", {
 certificateArn: certificate.arn,
 validationRecordFqdns: [certificateValidationDomain.fqdn],
 }, { provider: eastRegion });

 certificateArn = certificateValidation.certificateArn;

Configure the CloudFront distribution

We set the configuration parameters for CloudFront in distributionArgs :

// distributionArgs configures the CloudFront distribution. Relevant documentation:
// https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html
// https://www.terraform.io/docs/providers/aws/r/cloudfront_distribution.html
const distributionArgs: aws.cloudfront.DistributionArgs = {
 enabled: true,
 // Alternate aliases the CloudFront distribution can be reached at, in addition to https://xxxx.cloudfront.net.
 // Required if you want to access the distribution via config.targetDomain as well.
 aliases: [ config.targetDomain ],

 // We only specify one origin for this distribution, the S3 content bucket.
 origins: [
 {
 originId: contentBucket.arn,
 domainName: contentBucket.websiteEndpoint,
 customOriginConfig: {
 // Amazon S3 doesn't support HTTPS connections when using an S3 bucket configured as a website endpoint.
 // https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesOriginProtocolPolicy
 originProtocolPolicy: "http-only",
 httpPort: 80,
 httpsPort: 443,
 originSslProtocols: ["TLSv1.2"],
 },
 },
 ],

 defaultRootObject: "index.html",

 // A CloudFront distribution can configure different cache behaviors based on the request path.
 // Here we just specify a single, default cache behavior which is just read-only requests to S3.
 defaultCacheBehavior: {
 targetOriginId: contentBucket.arn,

 viewerProtocolPolicy: "redirect-to-https",
 allowedMethods: ["GET", "HEAD", "OPTIONS"],
 cachedMethods: ["GET", "HEAD", "OPTIONS"],

 forwardedValues: {
 cookies: { forward: "none" },
 queryString: false,
 },

 minTtl: 0,
 defaultTtl: tenMinutes,
 maxTtl: tenMinutes,
 },

 // "All" is the most broad distribution, and also the most expensive.
 // "100" is the least broad, and also the least expensive.
 priceClass: "PriceClass_100",

 // You can customize error responses. When CloudFront receives an error from the origin (e.g. S3 or some other
 // web service) it can return a different error code, and return the response for a different resource.
 // customErrorResponses: [
 // { errorCode: 404, responseCode: 404, responsePagePath: "/404.html" },
 // ],

 restrictions: {
 geoRestriction: {
 restrictionType: "none",
 },
 },

 viewerCertificate: {
 acmCertificateArn: certificateArn, // Per AWS, ACM certificate must be in the us-east-1 region.
 sslSupportMethod: "sni-only",
 },

 loggingConfig: {
 bucket: logsBucket.bucketDomainName,
 includeCookies: false,
 prefix: `${config.targetDomain}/`,
 },
};

const cdn = new aws.cloudfront.Distribution("cdn", distributionArgs);

Step 5: Alias Record

Next, we create an alias record that points to the CloudFront distribution.

function createAliasRecord(
 targetDomain: string, distribution: aws.cloudfront.Distribution): aws.route53.Record {
 const domainParts = getDomainAndSubdomain(targetDomain);
 const hostedZoneId = aws.route53.getZone({ name: domainParts.parentDomain }, { async: true }).then(zone => zone.zoneId);
 return new aws.route53.Record(
 targetDomain,
 {
 name: domainParts.subdomain,
 zoneId: hostedZoneId,
 type: "A",
 aliases: [
 {
 name: distribution.domainName,
 zoneId: distribution.hostedZoneId,
 evaluateTargetHealth: true,
 },
 ],
 });
}

const aRecord = createAliasRecord(config.targetDomain, cdn);

We’ve finished implementing our application infrastructure, and we’re ready to deploy. Run pulumi up to execute the Pulumi code to deploy to AWS. However, there is a more convenient way to do so with Github.

Github Workflow (Optional)

Github Actions adopts the Continuous Delivery/Integration concept and specifies a series of steps to run and deploy the program. We can take advantage of it with our CMS.

In the workflow, Github secrets are accessed from ${{ secrets.ACCESS_TOKEN }} where ACCESS_TOKEN is the Github credential. We can specify the secret in the project setting of Github.

PULUMI_STACK is the Pulumi stack name that we are using. PULUMI_ACCESS_TOKEN is required for the automatic process and can be generated by selecting “New Access Token” in the Pulumi Service settings.

Summary and Next Step

We have deployed the stand-alone React CMS application on AWS. The complete code is available in Pulumi’s Example Repository.

The next article will show how to deploy on AWS instead of Netlify. We describe how to substitute the Netlify Identity Service by writing an External OAuth Client-Server and deploying it on AWS. Stay tuned for our next blog post.