Pulumi Blog: Matt Ellis

How to Manage Secrets with Pulumi

Matt Ellis — Fri, 17 May 2019 00:00:00 +0000

We recommend Pulumi ESC for managing secrets with Pulumi. Since this blog post, we have launched Pulumi ESC which offers first class support for secrets, getting dynamic short-term credentials from cloud providers, simplifying your configurations with hierarchical environments, and much more.

We’ve had a 1st class concept of encrypted secrets configuration ever since first releasing Pulumi. Customers have told us they love having such a simple and easy way to ensure safe management of tokens, database passwords, and more. Since launching, however, we’ve also heard that you’d like more control over encryption and to see this protection expanded to cover not just configuration, but all of the secret data within their Pulumi deployments.

To support this, we’ve added two new features to Pulumi in our latest 0.17.12 release:

Automatic tracking of secret values throughout a Pulumi program to ensure that all such values are always encrypted in the resulting state, no matter how they are used.
A new option to use custom client-side encryption, instead of the default of using the Pulumi backend for encryption, to have full control over the secrets encryption and decryption.

Together, these features provide you with complete control over how secrets are managed within Pulumi deployments. We have worked with customers with advanced security and compliance needs while developing this feature, enabling them to use our online hosted SaaS with even greater confidence.

Why Secrets Management Matters in Cloud Infrastructure

Like many infrastructure as code systems, Pulumi uses a state file to describe the current state of your infrastructure. When you run pulumi up, Pulumi takes your existing state file, runs your program to compute a new desired state and compares the two states. It makes updates to the current state so it matches the desired state and updates its state file as it does so. As part of this, Pulumi needs to retain all the input values passed to resources in the state file, so it can detect if they have changed from run to run.

While Pulumi has allowed you to pass --secret to force configuration values to be encrypted before being stored in a stack’s configuration file, if you used these configuration values as inputs to resources, they would be stored in plain text in the state file. While the state file itself is stored securely (we encrypt all state files in transit and at rest), anyone with access to the state file itself would be able to see the plain text for all of these secrets.

By adding first class support for secrets with Pulumi, we are now able to automatically track secrets across your program’s execution and ensure that secret values are encrypted in the state file. This means you can use secrets confidently without worrying about accidentally leaking plain text values. Let’s take a look at how it works!

How Pulumi Encrypts & Manages Secrets Automatically

To start, let’s talk a bit about Output, one of the centerpieces of the Pulumi programming model. Output<T> ties together a value (which may not be ready yet, since it could depend on some data from a cloud resource that is still being created) and resources that the output depends on. When you create a resource with Pulumi, the properties of that resource are Output’s that you can pass to other resources. Pulumi uses the information tracked by Output to understand dependencies between different resources. For example when an Output<string> is used to construct a resource, Pulumi knows this resource depends on any resources that were used to generate that output. The underlying value that the Output wraps is what we store in the state file as an input for this new resource.

With 0.17.11 of Pulumi, we now have Output<T> track if it contains secret data. If it does, we ensure that the data is encrypted before we store it in the state file. There are few ways to create secret Outputs today:

By fetching values from the Config object in the JavaScript and Python SDKs, using the newly added getSecret or requireSecret (JavaScript) and get_secret or require_secret (Python), as well as some type specific overloads. These methods fetch the requested value from the configuration bag and then wrap it up in an Output which is marked as a secret. By using pulumi.secret (JavaScript) or pulumi.Output.secret (Python) to take an existing value and wrap it up in an Output which is marked as a secret. These behave the same way as pulumi.output (JavaScript) and pulumi.Output.from_input (Python) except they also mark the returned output as a secret. By retrieving an output that is marked as a secret from a resource.

When constructing a resource that has one or more secret inputs for a property, the entire corresponding output property of the resource is marked as a secret as well. In addition, as you combine outputs together, via all or apply, the resulting output is marked as a secret if any of the inputs values where themselves secrets. This means that just like dependency information, the “secret-ness” of an output flow naturally as you combine it with other data.

Let’s take a look at a small program which creates an AWS Systems Manager parameter, based on a secret configuration value.

Here’s the program we’ll be using:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const cfg = new pulumi.Config();
const secretMessage = cfg.requireSecret("secretMessage");

const param = new aws.ssm.Parameter("secretParameter", {
 type: "SecureString",
 value: secretMessage.apply(s => s.toUpperCase())
});

export const paramId = param.id;
export const paramValue = param.value;

In the above code sample, we’re using the new requireSecret method to pull out a configuration value as a secret. In addition, we use an apply to transform the string to all uppercase before creating our SSM Parameter. Because secretMessage was marked as a secret, this new value is also marked as a secret. It’s important to note the function that runs during the apply has access to the unencrypted value, so you need to be sure that your code inside the apply does not cause the secret to leak (for example, don’t write it to a text file!)

For a demo, let’s create a new stack, target us-west-2 in AWS and set a secret message:

$ pulumi stack init dev
$ pulumi config set aws:region us-west-2
$ pulumi config set --secret secretMessage "it's a secret to everybody"

Now, when we run pulumi up after creating a new stack, we’ll see the following preview:

Previewing update (dev):
Type Name Plan
+ pulumi:pulumi:Stack secrets-blog-dev create
+ └─ aws:ssm:Parameter secretParameter create
Resources:
+ 2 to create

If we look at the details for this deployment (before actually running the update), we can see that the value of this resource has been marked as a secret:

+ pulumi:pulumi:Stack: (create)
[urn=urn:pulumi:dev::secrets-blog::pulumi:pulumi:Stack::secrets-blog-dev]
+ aws:ssm/parameter:Parameter: (create)
[urn=urn:pulumi:dev::secrets-blog::aws:ssm/parameter:Parameter::secretParameter]
name : "secretParameter-1d79dca"
type : "SecureString"
value : "[secret]"

Once we’ve deployed our program, we can use pulumi stack export to look at the state file for our deployment, we see that the value is encrypted there as well (I’ve removed some uninteresting fields here, for clarity):

{
"urn": "urn:pulumi:dev::secrets-blog::aws:ssm/parameter:Parameter::secretParameter",
...snip...
"inputs": {
...snip...
"name": "secretParameter-56f0ffb",
"type": "SecureString",
"value": {
"4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
"ciphertext": "AAABAMo1ZLFpKzoHxUkGPXsUMjLBANri5fkPiveYUrjuMzsqONi2U1LnZSPxsN1vvFTs50skEru+Ff6N"
}
},
"outputs": {
...snip...
"name": "secretParameter-56f0ffb",
"type": "SecureString",
"value": {
"4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
"ciphertext": "AAABAOjpCFOLHMzP4G9OXc4r+mQs6/4DJv2aWO+vX0LyYHLjfawAHWFlRmv3dErda6Ip48pRB19bBL9t"
}
}
...snip...
}

As you can see, the value is encrypted in the state file for this resource! Also note that because the value was marked as a secret input, the corresponding copy in the output section of the state file was also marked as a secret. Pulumi ensures that any outputs with the same names as inputs which had secret data are also considered secrets. If there are additional outputs you want to set as secrets, you can pass the additionalSecretOutputs (JavaScript) or additional_secret_outputs (Python) resource option when constructing a resource to provide a list of other property names you want treated as secrets including computed output properties of a resource which might be sensitive, like generated passwords or access credentials.

Manage secrets with Pulumi ESC

Store encrypted secrets, pull short-lived cloud credentials, and share configuration across your stacks and environments from a single source with Pulumi ESC.

Get started

Configuring Your Secrets Provider

You might be wondering how these values are actually encrypted. We use the same encryption that we have always used for our configuration system. This means when storing state with Pulumi Cloud, we use a key managed by the app.pulumi.com service, specific to your stack, to encrypt everything. Some users have asked for more control over what key is used (and the ability to use a key not managed by Pulumi at all!)

When creating a new stack (via pulumi stack init or pulumi new), you may now pass --secrets-provider passphrase to specify that both configuration secrets and secrets stored in the state file should be encrypted using a key derived from a passphrase (if you’ve used Pulumi’s local state storage mode, this will be familiar to you). When you use a passphrase, we use PBKDF2 to derive a 32 byte encryption key, which we then use with the AES-256-GCM encryption algorithm to encrypt your value (using a random 12 byte nonce per value encrypted). Let’s run through deploying the same code but using the passphrase secret provider:

First, I create a new stack, setting the secrets provider to passphrase:

$ pulumi stack init dev --secrets-provider passphrase
Enter your passphrase to protect config/secrets:
Re-enter your passphrase to confirm:
Created stack 'dev'

As part of creating the stack, I had to enter a passphrase, which I’ll have to use during future updates. This passphrase is used to derive the key used for both configuration and state management. I can now configure my stack as I please:

$ pulumi config set aws:region us-west-2
$ pulumi config set secretMessage --secret "it's a secret to everybody"
Enter your passphrase to unlock config/secrets
(set PULUMI_CONFIG_PASSPHRASE to remember):

Note that to set the secret value, I had to provide my passphrase (since it is needed to generate the key that is used to encrypt the value).

Finally, I can run pulumi up, here I’m prompted to enter my passphrase again. I could also set PULUMI_CONFIG_PASSPHRASE in my environment. You might do this locally as part of your local development loop (so you don’t have to type your passphrase over and over) or in your CI system (where you’d be unable to type your passphrase in interactively).

$ pulumi up
Enter your passphrase to unlock config/secrets
(set PULUMI_CONFIG_PASSPHRASE to remember):

If we use pulumi stack export again to examine the state file, we can see that the structure of the ciphertext has changed:

"value": {
"4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
"ciphertext": "v1:qdfpSdF8vCWRJIDa:4gPQAMRSXi+5ap0koiZBsSVRnqzbp79cSEyWnLYkD9M5S/oI8qhgy521IBA="
}

The change is because we are no longer using the Pulumi service to encrypt or decrypt this data, instead the encryption and decryption happens locally, the data never leaves your machine. So while I get to continue to use <app.pulumi.com> to store state for my stack, I don’t have to worry about my secrets being encrypted with a key managed by a third party.

Support for changing the secrets provider for an existing stack is on its way. To track progress on this feature, please see GitHub issue pulumi/pulumi#481.

What’s Next for Secrets Management

In addition to passphrase based encryption, we plan to add support for encrypting using AWS KMS, Azure KeyVault and GCP KMS in the coming weeks.

The whole team is super excited about this feature and we love how nicely we were able to integrate it into our overall programming model. With these two new features, Pulumi users gain full control over how their secrets are managed, but without sacrificing usability and productivity. We’re excited for you all to start playing around with it! Pulumi is open source, free to use, and works today with variety of clouds. Try it today!

Managing GitHub Webhooks with Pulumi

Matt Ellis — Fri, 13 Jul 2018 00:00:00 +0000

At Pulumi, we do all of our development on GitHub, with a workflow built around topic branches. When a developer wants to make a change, they push a branch to GitHub, open a pull request and (in theory) once it’s merged, delete the branch. In practice, we’ll often forget to delete the topic branch (I’m probably the worst offender), which means we end up having topic branches linger on our main repository until they are explicitly cleaned up. While it’s a lot of fun to go a click through the GitHub UI from time to time, deleting merged branches, it’s even more fun to build automation to do this for us. Since GitHub has a rich set of webhooks and Pulumi makes it easy to write serverless functions, it felt like it would be natural to use Pulumi to write a hook that would clean up branches after a pull request got merged. In addition, Pulumi lets us leverage real programming languages to build abstractions, which means we can build a simple framework that hides much of the ceremony behind defining a hook and lets us focus on the core logic of our hook, without worrying about how it is deployed and managed.

A simple hook

When you register a webhook with GitHub, you provide an HTTP endpoint and a set of events you’d like to listen for. When one of the events happens on GitHub, it makes a POST request to your HTTP endpoint with a payload that describes the event. We’ll start by creating a simple endpoint that just logs when an event happens. We’ll start by using the aws-serverless package to define an API that just calls console.log to write the X-GitHub-Event header that is sent with the request:

import * as serverless from "@pulumi/aws-serverless";

const api = new serverless.apigateway.API("hook", {
 routes: [{
 path: "/",
 method: "POST",
 handler: async (req, ctx) => {
 console.log(`Got event ${req.headers['X-GitHub-Event']}`);
 return {
 statusCode: 200,
 body: ""
 }
 }
 },],
});

export const url = api.url;

After running pulumi update, we’ll have a HTTP endpoint we can give to GitHub:

Updating stack 'github-hook-blog-dev'
Performing changes:
Type Name Status Info
+ pulumi:pulumi:Stack github-hook-blog-github-hook-blog-dev created
+ ├─ aws-serverless:apigateway:API hook created
+ │ ├─ aws:apigateway:RestApi hook created
+ │ ├─ aws:apigateway:Deployment hook created
+ │ ├─ aws:lambda:Permission hook-980655da created
+ │ └─ aws:apigateway:Stage hook created
+ └─ aws:serverless:Function hook980655da created
+ ├─ aws:iam:Role hook980655da created
+ ├─ aws:iam:RolePolicyAttachment hook980655da-32be53a2 created
+ └─ aws:lambda:Function hook980655da created
---outputs:---
url: "https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/"

Now that we have our Webhook deployed, we’ll add it to GitHub. In this case, I have a little throwaway GitHub repository I use for testing this sort of stuff. In that repository, I go to Settings -> Webhooks -> Add webhook and fill in my information:

Payload URL: The value of the url output of my Pulumi program (in this case it is https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/).
Content Type: application-json. I know we’ll be inspecting this content as we develop the hook and since we’re writing the implementation in TypeScript, it will be easier to interact with JSON encoded data.
Secret: Generate a random string and put it here. I used random.org to do so, but any random string will suffice. We’ll use this string to validate that the request came from GitHub.
Events: I only care about pull_request events, so I picked “Let me select individual events” and then checked “Pull requests” and unchecked the other event types.
Active: Since we want GitHub to deliver events, we’ll keep this checked.

Once that’s done, we can test our hook by opening a pull request, waiting a few moments and then use pulumi logs to ensure our hook was triggered:

$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T09:06:51.000-07:00.
2018-07-09T10:06:00.610-07:00[ hook980655da-d0462d4] START RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Version: $LATEST
2018-07-09T10:06:00.612-07:00[ hook980655da-d0462d4] 2018-07-09T17:06:00.611Z 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Got event ping
2018-07-09T10:06:00.630-07:00[ hook980655da-d0462d4] END RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9
2018-07-09T10:06:00.630-07:00[ hook980655da-d0462d4] REPORT RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Duration: 10.99 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 20 MB
2018-07-09T10:06:23.241-07:00[ hook980655da-d0462d4] START RequestId: 68445127-839a-11e8-80a6-759a29eb6005 Version: $LATEST
2018-07-09T10:06:23.243-07:00[ hook980655da-d0462d4] 2018-07-09T17:06:23.243Z 68445127-839a-11e8-80a6-759a29eb6005 Got event pull_request
2018-07-09T10:06:23.250-07:00[ hook980655da-d0462d4] END RequestId: 68445127-839a-11e8-80a6-759a29eb6005
2018-07-09T10:06:23.250-07:00[ hook980655da-d0462d4] REPORT RequestId: 68445127-839a-11e8-80a6-759a29eb6005 Duration: 1.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 20 MB

Here we can see our hook got two events, the first is the ping event, which GitHub sends to verify the hook. The second event is from when I opened a pull request in the repository.

Before we go any further, let’s start using the secret we set to validate the Webhook payload. When GitHub sends an event to a Webhook, it sets a special header X-Hub-Signature which is an HMAC digest of the request body. The value we placed in the Secret field when configuring the Webhook is the key. So I’ll take that secret value, set it as a configuration value in my project:

$ pulumi config set --secret hookSecret [READACTED]

Once that’s done, I can use this value to validate the signature. We’ll use some functions from node’s crypto library at runtime to handle this. If the signature we computed doesn’t match what was provided with the request, we’ll return error code 400. Our program now looks like this:

import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";

const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");

const api = new serverless.apigateway.API("hook", {
 routes: [{
 path: "/",
 method: "POST",
 handler: async (req, ctx) => {
 // Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm

 // First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
 // should decode it.
 let body = req.body;
 if (req.isBase64Encoded) {
 body = Buffer.from(body, 'base64').toString();
 }

 // Now compute the HMAC.
 const crypto = await import("crypto");
 const hmac = crypto.createHmac("sha1", hookSecret);
 hmac.update(body);

 const computedSignature = `sha1=${hmac.digest("hex")}`;

 // Compare the signature that came in with the request to what we computed.
 if (req.headers['X-Hub-Signature'] === undefined ||
 !crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
 {
 console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
 return {
 statusCode: 400,
 body: "bad signature"
 }
 }

 console.log(`Got event ${req.headers['X-GitHub-Event']} and signature ${computedSignature} matched`);
 return {
 statusCode: 200,
 body: ""
 }
 }
 },],
});

export const url = api.url;

After deploying, we can trigger another event by closing an existing pull request or opening a new one, and then use pulumi logs to confirm that our validation is working as intended:

$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T12:43:05.000-07:00.
2018-07-09T13:43:00.630-07:00[ hook980655da-d0462d4] START RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Version: $LATEST
2018-07-09T13:43:00.693-07:00[ hook980655da-d0462d4] 2018-07-09T20:43:00.693Z ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Got event pull_request and signature sha1=b83320205edc7cb9aeea1a845b85de0c83092e1d matched
2018-07-09T13:43:00.731-07:00[ hook980655da-d0462d4] END RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9
2018-07-09T13:43:00.731-07:00[ hook980655da-d0462d4] REPORT RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Duration: 128.75 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 20 MB

Building an abstraction

We now have a nice little skeleton that we can use when writing a webhook on GitHub. Let’s start to leverage some other Pulumi features to make things simpler and provide a nicer interface for writing hooks. Let’s start by abstracting away the code that handles validating the request and returning a response behind a helper method. We’ll pass a function to this helper which will be called after the validation has completed. We’ll also parse the body into a JSON object that our handler can use.

import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";

const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");

function createWebhook(name: string, handler: ((req: serverless.apigateway.Request, body: any) => Promise<void>)) {
 return new serverless.apigateway.API(name, {
 routes: [{
 path: "/",
 method: "POST",
 handler: async (req, ctx) => {
 // Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm

 // First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
 // should decode it.
 let body = req.body;
 if (req.isBase64Encoded) {
 body = Buffer.from(body, 'base64').toString();
 }

 // Now compute the HMAC.
 const crypto = await import("crypto");
 const hmac = crypto.createHmac("sha1", hookSecret);
 hmac.update(body);

 const computedSignature = `sha1=${hmac.digest("hex")}`;

 // Compare the signature that came in with the request to what we computed.
 if (req.headers['X-Hub-Signature'] === undefined ||
 !crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
 {
 console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
 return {
 statusCode: 400,
 body: "bad signature"
 }
 }

 // Call the handler after parsing the body as JSON
 await handler(req, JSON.parse(body));

 return {
 statusCode: 200,
 body: ""
 }
 }
 },],
 });
}

const api = createWebhook("hook", async (req, body) => {
 console.log(`Got event ${req.headers['X-GitHub-Event']} with action ${body.action}`);
});

export const url = api.url;

The shape of the JSON object for this event is documented on GitHub. Here we’re pulling out the action field, which tells us something about the state of the pull request. For our bot, we only care about PRs where the action is closed and the merged key is set to true. When this holds, we’ll want to delete the branch the pull request came from, if it was a topic branch (instead of a PR opened from a fork). We’ll leverage the existing @octokit/rest NPM package to do this. We can generate a personal access token, and give it repo level scope, so it can act as us and delete branches. We’ll then add this key to our configuration:

$ pulumi config set --secret githubToken [REDACTED]

And we can now update our handler to use this API:

import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";

const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");

function createWebhook(name: string, handler: ((req: serverless.apigateway.Request, body: any) => Promise<void>)) {
 /* Same as the previous version */
}

const githubToken = cfg.require("githubToken");

const api = createWebhook("hook", async (req, body) => {
 if (body.action === "closed" && body.pull_request.merged &&
 body.pull_request.base.user.login === body.pull_request.head.user.login &&
 body.pull_request.base.repo.name === body.pull_request.head.repo.name)
 {
 const octokit = require('@octokit/rest')()
 octokit.authenticate({
 type: 'token',
 token: githubToken
 });

 console.log(`Deleting reference heads/${body.pull_request.head.ref}`);

 await octokit.gitdata.deleteReference({
 owner: body.pull_request.head.user.login,
 repo: body.pull_request.head.repo.name,
 ref: `heads/${body.pull_request.head.ref}`
 });
 }
});

export const url = api.url;

After pulumi updating, try merging a pull request! Shortly after you click the merge button, the source branch should be deleted automatically. You can also see our log message in the output of pulumi logs:

$ pulumi logs
[matell@matell throwaway]$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T13:30:26.000-07:00.
2018-07-09T14:28:52.254-07:00[ hook980655da-d0462d4] START RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325 Version: $LATEST
2018-07-09T14:28:52.431-07:00[ hook980655da-d0462d4] 2018-07-09T21:28:52.371Z 1368afec-83bf-11e8-bf0d-cf623578c325 Deleting reference heads/test-branch
2018-07-09T14:28:53.478-07:00[ hook980655da-d0462d4] END RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325
2018-07-09T14:28:53.478-07:00[ hook980655da-d0462d4] REPORT RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325 Duration: 1221.97 ms Billed Duration: 1300 ms Memory Size: 128 MB Max Memory Used: 33 MB

So, in about 75 lines of code, we’ve defined a small abstraction for authoring a GitHub Webhook and we’ve used it to write a little bot that deletes topic branches after pull requests from them have been merged. In itself, that’s pretty cool, but we can do some more cool stuff with Pulumi. As is, while we can create the Webhook using Pulumi, we have to register it using the GitHub console, which is a little tedious. What if we could manage the registration of the hook with Pulumi itself? We’d really like to model the hook’s association with GitHub as a resource that can be created, updated and deleted. Is there a way we can do that? Yes, there is!

Managing all the things with Pulumi

Pulumi uses gRPC to define a contract between resource providers and the rest of Pulumi. So, we could go implement that contract in a language like go, like we do in our Kubernetes provider. Another option would be to take the existing Terraform GitHub Provider and use Pulumi’s terraform bridge wrap it into a Pulumi resource provider, like we do with our pulumi-aws provider. There’s one final option, which is to use the “dynamic provider”, which allows us to implement a resource provider in JavaScript itself. We wrote this provider to help us with testing, so we could mock out resources and control their lifecycle, but we can also use it to create a resource that manages registration of a webhook with GitHub. To do so, we need to implement dynamic.ResourceProvider interface. Once we have the provider, we can also write a resource that uses it.

class GithubWebhookProvider implements dynamic.ResourceProvider {

 // Check ensures that all required properties are set. In this case we have three required parameters.
 check = async (olds: any, news: any) => {
 const failedChecks = [];

 for (const prop of ["url", "owner", "repo"]) {
 if (news[prop] === undefined) {
 failedChecks.push({property: prop, reason: `required property '${prop}' missing`});
 }
 }

 return { inputs: news, failedChecks: failedChecks };
 }

 // Today the engine does the diff between properties to detect if there is a change but this method does
 // tell the engine if the changes between the old and new values require the resource to be "replaced"
 // (that is a new one is created and the old one is deleted) vs being edited in place. For us, if the owner
 // or repo the hook is installed on changes, we'll trigger a replacement.
 diff = async (id: pulumi.ID, olds: any, news: any) => {
 const replaces = [];

 for (const prop of ["owner", "repo"]) {
 if (olds[prop] !== news[prop]) {
 replaces.push(prop);
 }
 }

 return { replaces: replaces };
 }

 // Create actually creates the hook. We use octokit under the hood and return the ID of the hook that was created.
 // Pulumi retains this ID and gives it to us when we need to update or delete the hook.
 create = async (inputs: any) => {
 const octokit = require("@octokit/rest")();
 octokit.authenticate({
 type: "token",
 token: githubToken,
 });

 const res = await octokit.repos.createHook({
 owner: inputs["owner"],
 repo: inputs["repo"],
 name: "web",
 events: ["pull_request"],
 config: {
 content_type: "json",
 url: inputs["url"],
 secret: hookSecret,
 },
 });

 if (res.status !== 201) {
 throw new Error(`bad response: ${JSON.stringify(res)}`);
 }

 // The engine expects that the ID property is a string.
 return {
 id: `${res.data.id}`,
 };
 }

 update = async (id: pulumi.ID, olds: any, news: any) => {
 const octokit = require("@octokit/rest")();
 octokit.authenticate({
 type: "token",
 token: githubToken,
 });

 const res = await octokit.repos.editHook({
 hook_id: id,
 owner: news["owner"],
 repo: news["repo"],
 events: ["pull_request"],
 config: {
 content_type: "json",
 url: news["url"],
 },
 });

 if (res.status !== 200) {
 throw new Error(`bad response: ${JSON.stringify(res)}`);
 }

 return {}
 }

 delete = async (id: pulumi.ID, props: any) => {
 const octokit = require("@octokit/rest")();

 octokit.authenticate({
 type: "token",
 token: githubToken,
 });

 const res = await octokit.repos.deleteHook({
 hook_id: id,
 owner: props["owner"],
 repo: props["repo"],
 });

 if (res.status !== 204) {
 throw new Error(`bad response: ${JSON.stringify(res)}`);
 }
 }
}

interface GitHubWebhookResourceArgs {
 url: pulumi.Input<string>;
 owner: pulumi.Input<string>;
 repo: pulumi.Input<string>;
}

class GitHubWebhookResource extends dynamic.Resource {
 constructor(name: string, args: GitHubWebhookResourceArgs, opts?: pulumi.ResourceOptions) {
 super(new GithubWebhookProvider(), name, args, opts);
 }
}

With this new dynamic resource, registering the hook itself becomes easy:

new GitHubWebhookResource("hook-registration", {
 url: api.url,
 owner: "ellismg",
 repo: "testing",
});

Before running pulumi update go and manually delete the hook registration on GitHub. Now, when you pulumi update you’ll see the new resource created:

Updating stack 'github-hook-blog-dev'
Performing changes:
Type Name Status Info
* pulumi:pulumi:Stack github-hook-blog-github-hook-blog-dev done
+ └─ pulumi-nodejs:dynamic:Resource hook-registration created
---outputs:---
url: "https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/"
info: 1 change performed:
+ 1 resource created
10 resources unchanged
Update duration: 4.649249758s

And if you look on GitHub, the hook should once again be registered! Now, if the URL for our handler ends up changing or we destroy our stack, the hook registration will be updated accordingly. Since our little program is getting large (~200 lines at this point), let’s do a little more work on our abstraction.

Building a Component

Pulumi has the concept of a ComponentResource which is a resource that aggregates other resources. Many resources that we interact with day to day in Pulumi are actually ComponentResources (In fact, the API resource we’re using here is itself a component). In another file, we can create the component resource and move much of our logic into there:

The interesting new code looks like this:

export interface GitHubWebhookArgs {
 owner: pulumi.Input<string>;
 repo: pulumi.Input<string>;
 handler: (req: serverless.apigateway.Request, body: any) => Promise<void>;
}

export class GitHubWebhook extends pulumi.ComponentResource {
 public readonly url: pulumi.Output<string>;

 constructor(name: string, args: GitHubWebhookArgs, opts?: pulumi.ResourceOptions) {
 super("github:webhooks:Hook", name, {}, opts);

 const api = new serverless.apigateway.API("hook", {
 routes: [
 {
 path: "/",
 method: "POST",
 handler: async (req, ctx) => {
 // Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm

 // First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
 // should decode it.
 let body = req.body;
 if (req.isBase64Encoded) {
 body = Buffer.from(body, 'base64').toString();
 }

 // Now compute the HMAC.
 const crypto = await import("crypto");
 const hmac = crypto.createHmac("sha1", hookSecret);
 hmac.update(body);

 const computedSignature = `sha1=${hmac.digest("hex")}`;

 // Compare the signature that came in with the request to what we computed.
 if (req.headers['X-Hub-Signature'] === undefined ||
 !crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
 {
 console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
 return {
 statusCode: 400,
 body: "bad signature"
 }
 }

 // Call the handler after parsing the body as JSON:
 await args.handler(req, JSON.parse(body));

 return {
 statusCode: 200,
 body: ""
 }
 },
 },
 ],
 }, {
 parent: this,
 });

 new GitHubWebhookResource(`${name}-registration-${args.owner}-${args.repo}`, {
 owner: args.owner,
 repo: args.repo,
 url: api.url,
 }, {
 parent: this,
 });

 this.url = api.url;
 }
}

Here, we’ve transformed our createWebhook method into an actual GitHubWebhook component that manages both the API of the hook as the hook’s registration with GitHub. With this abstraction (and all of this complexity hidden off in a separate github.ts file), the code we focus on when actually writing our bot is quite small:

import * as pulumi from "@pulumi/pulumi";
import { GitHubWebhook } from "./github";

const cfg = new pulumi.Config(pulumi.getProject());
const githubToken = cfg.require("githubToken");

const hook = new GitHubWebhook("hook", {
 owner: "ellismg",
 repo: "testing",
 handler: async (req, body) => {
 if (body.action === "closed" && body.pull_request.merged &&
 body.pull_request.base.user.login === body.pull_request.head.user.login &&
 body.pull_request.base.repo.name === body.pull_request.head.repo.name)
 {
 const octokit = require('@octokit/rest')()
 octokit.authenticate({
 type: 'token',
 token: githubToken
 });

 console.log(`Deleting reference heads/${body.pull_request.head.ref}`);

 await octokit.gitdata.deleteReference({
 owner: body.pull_request.head.user.login,
 repo: body.pull_request.head.repo.name,
 ref: `heads/${body.pull_request.head.ref}`
 });
 }
 },
});

export const url = hook.url;

Going Further

From here, there’s a lot of ways we could extend this code. For example, while we require the configuration value hookSecret to be set before using this abstraction, we could manage this on behalf of the user. To do so, we could use the dynamic provider to create a special “random” resource, this resource would generate a new random string when created, but update calls would not impact the resource. Since Pulumi stores the state of every resource across invocations in the checkpoint, our random resource would not need to be backed by any cloud infrastructure. We could change the shape of the GitHubWebhook component to allow multiple owner/repository pairs to be provided and it would generate a single AWS API and then multiple GitHubWebhookResource’s to register the same hook across multiple repositories. We could even extend this to allow registration of both repository and organization level webhooks. Or instead of hard-coding in the pull_request event as the only event we care about take it as another input property to GitHubWebhook.

One of my favorite things about Pulumi is while it is easy to build these abstractions, it’s also easy to share them. For example, I’ve implemented a few of these ideas to build a framework for our use at Pulumi. You can find the source on GitHub and I’ve published it as an NPM package at @ellismg/pulumi-github-webhooks.