Pulumi Blog: Piers Karsenbarg

Pulumi + Bun: Supercharge Your Infrastructure Dependencies

Piers Karsenbarg — Mon, 28 Jul 2025 09:50:26 +0100

Today, we are announcing a new addition that allows Pulumi users to use Bun as their package manager, offering an alternative to npm, Yarn, or pnpm. Bun, a recently released JavaScript runtime, bundler, transpiler, and package manager, provides a faster way to manage Pulumi project dependencies.

What is this Bun thing you’re talking about?

If you’ve spent time around the JavaScript ecosystem, then you’ve probably heard of Bun. If you haven’t, then be aware that it presents itself as a JavaScript toolkit, a direct drop-in replacement for NodeJS. However, the part that we like at Pulumi is its use as a package manager. According to the Bun install documentation, switching from npm install to bun install can make your installations up to 25 times faster.

Pulumi uses Bun solely as a package manager. You cannot use Bun to write Automation API-based applications, nor do we have a language plugin for Bun.
This requires Bun version 1.2 or greater due to the text-based lockfile introduced in this version.

Getting started with Bun and Pulumi

If you’re already using NodeJS with Pulumi, the easiest way to install Bun is to run the following:

npm install -g bun

The Bun installation documentation has more information on other methods available.

How to use in a new project

If you want to try this out on a new project, you can use

mkdir new-bun-project && cd new-bun-project
pulumi new typescript

You’ll get the usual prompts to set the project name, the description, and the stack name:

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

Project name (new-bun-project):
Project description (A minimal TypeScript Pulumi program):
Created project 'new-bun-project'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
Stack name (dev):

Then you’ll be shown a list of the package managers to choose from (obviously we’ll use the arrow keys to select the bun option):

The package manager to use for installing dependencies [Use arrows to move, type to filter]
 npm
 pnpm
 yarn
> bun

Hit the ENTER key to make your selection, and you’ll see the bun install output:

The package manager to use for installing dependencies bun
Installing dependencies...

bun install v1.2.18 (0d4089ea)

+ @types/node@18.19.120 (v24.1.0 available)
+ typescript@5.8.3
+ @pulumi/pulumi@3.186.0

272 packages installed [1240.00ms]

Blocked 1 postinstall. Run `bun pm untrusted` for details.
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

The files in the folder look a little different:

drwxr-xr-x@ - piers 28 Jul 18:13 node_modules
.rw-r--r--@ 21 piers 28 Jul 18:11 .gitignore
.rw-r--r--@ 61k piers 28 Jul 18:13 bun.lock
.rw-r--r--@ 42 piers 28 Jul 18:11 index.ts
.rw-r--r--@ 217 piers 28 Jul 18:11 package.json
.rw-r--r--@ 198 piers 28 Jul 18:13 Pulumi.yaml
.rw-r--r--@ 438 piers 28 Jul 18:11 tsconfig.json

(note the bun.lock instead of the package-lock.json)

and in the Pulumi.yaml file, the packagemanager option is set to bun:

name: new-bun-project
description: A minimal TypeScript Pulumi program
runtime:
name: nodejs
options:
packagemanager: bun
config:
pulumi:tags:
value:
pulumi:template: typescript

We will use the Random provider to generate a new password. Let’s add the @pulumi/random package:

❯ bun add @pulumi/random
bun add v1.2.18 (0d4089ea)

installed @pulumi/random@4.18.3

[345.00ms] done

Since we’re still using the Pulumi NodeJS runtime, that’s all we need Bun for right now.

The rest is business as usual, so the index.ts file looks like:

import * as random from "@pulumi/random";

const password = new random.RandomPassword("password", {
 length: 20
});

export const pw = password.result;

Then it’s just a case of running:

pulumi up

and creating the resources.

Migrating an existing project

If you already have a project set up and are using npm as your package manager, there are a few steps you’ll need to follow.

For this example, I’m going to use the code from the Bun example above but set it up using npm instead:

❯ pulumi new typescript
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

Project name (migrate-project):
Project description (A minimal TypeScript Pulumi program):
Created project 'migrate-project'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
Stack name (dev):

The package manager to use for installing dependencies npm
Installing dependencies...


added 293 packages, and audited 294 packages in 9s

43 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

Then we’ll install the @pulumi/random package:

npm install @pulumi/random

and checking the folder listing:

drwxr-xr-x@ - piers 28 Jul 18:32 node_modules
.rw-r--r--@ 21 piers 28 Jul 18:32 .gitignore
.rw-r--r--@ 42 piers 28 Jul 18:32 index.ts
.rw-r--r--@ 149k piers 28 Jul 18:32 package-lock.json
.rw-r--r--@ 217 piers 28 Jul 18:32 package.json
.rw-r--r--@ 198 piers 28 Jul 18:32 Pulumi.yaml
.rw-r--r--@ 438 piers 28 Jul 18:32 tsconfig.json

we can see that we’ve got the usual package-lock.json file that we expect, and the Pulumi.yaml file looks like this:

name: migrate-project
description: A minimal TypeScript Pulumi program
runtime:
name: nodejs
options:
packagemanager: npm
config:
pulumi:tags:
value:
pulumi:template: typescript

(Note the packagemanager option).

To migrate this project to use Bun as the package manager instead, we need to update the Pulumi.yaml to set the packagemanager to be bun:

name: migrate-project
description: A minimal TypeScript Pulumi program
runtime:
name: nodejs
options:
packagemanager: bun
config:
pulumi:tags:
value:
pulumi:template: typescript

We can use the bun CLI to migrate the package-lock.json file to the bun.lock file, running the following command:

bun pm migrate

which gives us the file structure:

drwxr-xr-x@ - piers 28 Jul 18:32 node_modules
.rw-r--r--@ 21 piers 28 Jul 18:32 .gitignore
.rw-r--r--@ 64k piers 28 Jul 18:43 bun.lock
.rw-r--r--@ 42 piers 28 Jul 18:32 index.ts
.rw-r--r--@ 149k piers 28 Jul 18:32 package-lock.json
.rw-r--r--@ 217 piers 28 Jul 18:32 package.json
.rw-r--r--@ 198 piers 28 Jul 18:32 Pulumi.yaml
.rw-r--r--@ 438 piers 28 Jul 18:32 tsconfig.json

We can then delete the package-lock.json if we want to.

Then we can run pulumi install to install the packages to Bun’s global cache:

❯ pulumi install
Installing dependencies...

bun install v1.2.18 (0d4089ea)

15 packages installed [60.00ms]
Finished installing dependencies

and continue with

pulumi up

What else?

We’ve baked Bun into the pulumi/pulumi container, allowing you to make use of the faster package installer in your CI/CD pipeline, including in Pulumi Deployments. If you’d like to see Bun become more integrated into Pulumi, there’s an open issue that you can vote on, or even implement if you’re interested.

Final thoughts

Bun is in the process of establishing itself as the new player in the JavaScript ecosystem. Its focus on speed makes it a great alternative to npm, Yarn, and pnpm when managing Pulumi project dependencies. Give it a try on your next project, and you may wonder how you ever lived without it.

Accessing Secrets Safely in Lambda Functions

Piers Karsenbarg — Wed, 12 Oct 2022 17:19:53 +0100

The subject of how to make use of secrets in Lambda Functions comes up a fair bit, and although there seems to be a lot of discussion on where you should store them, the one thing that comes up is that you should never store the plain text values of secrets in the Lambda Function’s environment variables. One such discussion I was having with a customer made me think about how it should be possible to take the secrets that you’ve got on your stack config file and then use them to configure your Lambda Function, with the plain text values going into the Function’s environment variables and the encrypted secret values going into AWS’ Secrets Manager.

Getting started

I’m going to assume you’ve already got Pulumi and AWS set up, you’re logged into your backend, and you’ve already created a project and stack. I’m also using the Pulumi Service as my secrets provider.

Although Lambda Functions are free to deploy, and you get a generous allowance as part of AWS’ free tier, storing and accessing secrets is not. You can view the cost on the AWS Secrets Manager Pricing page.

Structured configuration

We’re going to start by adding settings, both secret and plaintext, to our stack configuration, in a structured way. We’re going to add these settings as a dictionary so we can use the names as the variable names in our Lambda Function.

First we add two settings in plaintext that are going to be used as environment variables in our lambda:

pulumi config set --path 'lambdawithsecrets.envvars["envvar1"]' envvar1value
pulumi config set --path 'lambdawithsecrets.envvars["envvar2"]' envvar2value

and then we add the secret settings:

pulumi config set --path 'lambdawithsecrets.secrets["secret1"]' secretvalue1 --secret
pulumi config set --path 'lambdawithsecrets.secrets["secret2"]' secretvalue2 --secret

If we take a look at the stack config (in my stack named dev it’s the Pulumi.dev.yaml file):

config:
 lambda-secrets:lambdawithsecrets:
 envvars:
 envvar1: envvar1value
 envvar2: envvar2value
 secrets:
 secret1:
 secure: AAABALZIAzDaE45MbdO08w/QMHK6CE4hssidMK70pyYt6hIFHpKeq0mBNic=
 secret2:
 secure: AAABAI+y9HpdgQ1u1rSPKGZeBkAIIbZfki84zmRrcWVltIMI5SGWZbYOyfQ=

there are several things I want to point out:

The entire config item is named lambdawithsecrets (this is how we’ll retrieve the configuration later)
There are two sections, envvars and secrets. These are how I’ll refer to the dictionaries later on in the code, but you can name these in any way you want.
We’ll use the dictionary keys (envvar1, envvar2, secret1, secret2) to refer to the variables in the Lambda Function so we know what they’re named and can easily access them. Again, you can name these to be whatever values you want.

Writing our Pulumi program

Next we’re going to start updating our index.ts file. Everything from now on can be copied and pasted in order into the file, and I’ll post the entire contents at the end.

As we’re only going to use the Pulumi and the AWS NPM packages you can replace everything in the file with the following:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

A warning about accessing the configuration

Before we continue, it’s worth pointing out the following (from the Pulumi documentation on secrets ):

On pulumi up, secret values are decrypted and made available in plaintext at runtime. These may be read through any of the standard pulumi.Config getters. While it is possible to read a secret using ordinary non-secret getters, this is almost certainly not what you want. Use the secret variants of the configuration APIs instead since this ensures that all transitive uses of that secret are themselves also marked as secrets.

Now that’s out the way, let’s continue.

Accessing the configuration

Since we’re using structured configuration to store the values, we can access them in a strongly-typed way. We’re going to create an interface to hold the structure:

interface LambdaConfig {
 envvars: string[];
 secrets: pulumi.Output<string>[];
}

You’ll notice that the property names match the dictionary names in the stack config and that the secrets property is of type pulumi.Output<string>[] instead of string[]. This is because they’re Pulumi secrets, and we want to handle them in the correct way.

Next we’ll access the config and store the values in a variable:

const config = new pulumi.Config();
const lambdaconfig = config.requireObject<LambdaConfig>("lambdawithsecrets");

Storing the secret values

As I said in the beginning, we’re going to store the secrets from the stack config in AWS Secrets Manager, and then the ARN of each of the secrets will be stored in the Lambda Function’s environment variables.

We’re going to create a couple of arrays to store (non-sensitive) information about the Secret resources that we’re going to create to use later on. One to hold the Secret resources and one to hold the ARNs of the secrets. Due to the fact that it’s the SecretVersion where we store the secret value, it’s pretty safe to store the Secret.

Next we’ll loop through the config’s secrets dictionary, grab the values and for each one create a Secret and a SecretValue resource, and then populate the two arrays with the relevant values.

const secretsArnArray: pulumi.Output<string>[] = new Array();
const secretArray: aws.secretsmanager.Secret[] = new Array();
for (const key in lambdaconfig.secrets) {
 if (lambdaconfig.secrets.hasOwnProperty(key)) {
 const secret = new aws.secretsmanager.Secret(`${key}`);
 const secretVersion = new aws.secretsmanager.SecretVersion(`secretversion-${key}`, {
 secretId: secret.id,
 secretString: lambdaconfig.secrets[key],
 });
 secretArray.push(secret);
 secretsArnArray[key.toLocaleUpperCase()] = secret.id;
 }
}

We’re not actually doing anything with the SecretVersion resources, so that’s why we’re not assigning them to any variables.

Identity and Access Management

There are a few resources we’re going to need to create and assign to the Lambda Function to give it access to the Secret resources:

A role that we can assign to the Lambda Function
A Policy that is going to give that role read access to the values of the secrets we’ve created
A couple of RolePolicyAttachment resources which is how we attach policies to the above role

Firstly, the role:

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(aws.iam.Principals.LambdaPrincipal),
});

We’re using the assumeRolePolicyForPrincipal function because doesn’t that look better than this:

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [{
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Sid: "",
 Principal: {
 Service: "lambda.amazonaws.com",
 },
 }],
 }),
});

(either of the above Role resources can be used, but I’m using the top one because I think it’s neater).

Next we create a policy document and a policy. We’re going to use the secretArray variable that we created earlier to allow us to only allow the Lambda Function access to the secrets that we’ve created:

const secretManagerPolicyDoc = aws.iam.getPolicyDocumentOutput({
 statements: [
 {
 effect: "Allow",
 actions: [
 "secretsmanager:GetSecretValue",
 ],
 resources: secretArray.map(x => pulumi.interpolate`${x.arn}`),
 },
 ],
});

const secretManagerPolicy = new aws.iam.Policy("secretsPolicy", {
 policy: secretManagerPolicyDoc.apply(doc => doc.json),
});

Finally we’re going to attach the above policy and the AWS Managed AWSLambdaBasicExecutionRole policy to the IAM role we created:

const rpaBasic = new aws.iam.RolePolicyAttachment("rpa-basic", {
 role: role,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
});

const rpaSecrets = new aws.iam.RolePolicyAttachment("rpa-secrets", {
 role: role,
 policyArn: secretManagerPolicy.arn,
});

The aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole is one of our enums that points at the relevant ARN for this policy.

Finally, the Function

There are two parts to this: the Lambda Function resource and the code we’re going to deploy as part of the function.

The Resource

The function resource itself is pretty simple. The interesting bit is how we take the array of the ARNs we created earlier, combine it with the array of environment variable values we have in the config and assign them to the environment input:

const lambdaEnvVars: pulumi.Input<{
 [key: string]: pulumi.Input<string>;
}> = {};

for (const key in secretsArnArray) {
 if (secretsArnArray.hasOwnProperty(key)) {
 lambdaEnvVars[key] = secretsArnArray[key];
 }
}

for (const key in lambdaconfig.envvars) {
 if (lambdaconfig.envvars.hasOwnProperty(key)) {
 lambdaEnvVars[key.toLocaleUpperCase()] = lambdaconfig.envvars[key];
 }
}

The reason that we’re using the .toLocaleUpperCase() function is because as a rule, the usual way of defining environment variable names is using upper case letters.

Once we’ve done this we can put together the function resource:

const lambda = new aws.lambda.Function("lambdaWithSecrets", {
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./app"),
 }),
 role: role.arn,
 handler: "index.handler",
 runtime: aws.lambda.Runtime.NodeJS16dX,
 environment: {
 variables: lambdaEnvVars,
 },
 timeout: 15,
});

export const lambdaName = lambda.name;

There’s nothing complicated about the function. We’re using the role from earlier, setting the environment variables to be the object we created just above and pointed the code input at a local folder called app, which we’ll look at next. The lambda name being exported at the end is so that we can use the Pulumi CLI to get this value and invoke the function using the AWS CLI.

The Code

First we need somewhere to put the code:

mkdir app && cd app

Then we create a new NPM application and install the AWS client-secrets-manager package that we’ll use to access the secrets in the code:

npm init -y && npm install @aws-sdk/client-secrets-manager

We’re also going to update the type setting in the package.json file to be module so that we can have a top level async function. Here’s what our file looks like:

{
 "name": "app",
 "version": "1.0.0",
 "description": "",
 "main": "index.js",
 "scripts": {
 "test": "echo \"Error: no test specified\" && exit 1"
 },
 "keywords": [],
 "author": "",
 "license": "ISC",
 "dependencies": {
 "@aws-sdk/client-secrets-manager": "^3.183.0"
 },
 "type": "module"
}

The index.js file for the lambda isn’t exciting. We’re going to make the function more efficient by making the calls to the secrets manager API outside of the main handler code, so it doesn’t get run for all invocations (just the ones during a cold start):

import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager"; // ES Modules import
const client = new SecretsManagerClient();

const getSecretForLambda = async (secretId) => {
 const input = { "SecretId": secretId };
 const command = new GetSecretValueCommand(input);
 console.log('Retrieving secret during top-level await');
 return await client.send(command);
}

const secret1 = getSecretForLambda(process.env.SECRET1);
const secret2 = getSecretForLambda(process.env.SECRET2);

export async function handler() {
 console.log('Using secret in handler');
 return {
 secret1: (await secret1).SecretString,
 secret2: (await secret2).SecretString,
 };
};

The above JavaScript code will return the secret values from the function and is purely for demo purposes. In the Real World^TM you definitely shouldn’t do this.

Deployment time

Now that we’ve got everything together we can deploy this:

pulumi up

and you should see something like the following:

Previewing update (dev)

View Live: https://app.pulumi.com/acmecorp/lambda-secrets/dev/previews/209f07fb-190d-4596-af9f-d2d72e9c5cc9

 Type Name Plan
 + pulumi:pulumi:Stack lambda-secrets-dev create
 + ├─ aws:secretsmanager:Secret secret1 create
 + ├─ aws:secretsmanager:Secret secret2 create
 + ├─ aws:iam:Role roleLambdaWithSecrets create
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret2 create
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret1 create
 + ├─ aws:iam:Policy secretsPolicy create
 + ├─ aws:iam:RolePolicyAttachment rpa-basic create
 + ├─ aws:iam:RolePolicyAttachment rpa-secrets create
 + └─ aws:lambda:Function lambdaWithSecrets create

Outputs:
 lambdaName: "lambdaWithSecrets-b8ac5e9"

Resources:
 + 10 to create

Do you want to perform this update? yes
Updating (dev)

View Live: https://app.pulumi.com/acmecorp/lambda-secrets/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack lambda-secrets-dev created
 + ├─ aws:secretsmanager:Secret secret1 created
 + ├─ aws:secretsmanager:Secret secret2 created
 + ├─ aws:iam:Role roleLambdaWithSecrets created
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret2 created
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret1 created
 + ├─ aws:iam:Policy secretsPolicy created
 + ├─ aws:iam:RolePolicyAttachment rpa-basic created
 + ├─ aws:lambda:Function lambdaWithSecrets created
 + └─ aws:iam:RolePolicyAttachment rpa-secrets created

Outputs:
 lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 + 10 created

Duration: 24s

Let’s see what this has done. If we go to look at the Lambda Function in the console, click on the Configuration tab and select Environment variables you’ll see the two plaintext environment variables and their values along with the two secret environment variables containing the ARNs to the relative secret in the Secrets Manager. Since we know the environment variable key (the uppercase version of what’s in the stack config), we can reliably use that same name in the function code (the two lines where we call the getSecretForLambda function).

For me, it looks something like this:

Finally, let’s run the function via the AWS CLI and see what we get:

aws lambda invoke --function-name $(pulumi stack output lambdaName) /dev/stdout

You should see something like the following (depending on what values you added to the stack config at the beginning):

{"secret1":"secret1value","secret2":"secret2value"}

You can also use the test function in the AWS Console and see the results there.

Tidying up

As stated earlier, it costs to store and access secrets so we need to make sure that we remove the infrastructure once we’ve completed this demo. As ever, you can do this by running pulumi destroy:

Previewing destroy (dev)

View Live: https://app.pulumi.com/pierskarsenbarg/lambda-secrets/dev/previews/f111f92e-1dbb-4615-9871-30de417bf8f1

 Type Name Plan
 - pulumi:pulumi:Stack lambda-secrets-dev delete
 - ├─ aws:iam:RolePolicyAttachment rpa-secrets delete
 - ├─ aws:iam:RolePolicyAttachment rpa-basic delete
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret1 delete
 - ├─ aws:iam:Policy secretsPolicy delete
 - ├─ aws:lambda:Function lambdaWithSecrets delete
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret2 delete
 - ├─ aws:secretsmanager:Secret secret2 delete
 - ├─ aws:iam:Role roleLambdaWithSecrets delete
 - └─ aws:secretsmanager:Secret secret1 delete

Outputs:
 - lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 - 10 to delete

Do you want to perform this destroy? [Use arrows to move, enter to select, type
Do you want to perform this destroy? [Use arrows to move, enter to select, type
Do you want to perform this destroy? yes
Destroying (dev)

View Live: https://app.pulumi.com/pierskarsenbarg/lambda-secrets/dev/updates/64

 Type Name Status
 - pulumi:pulumi:Stack lambda-secrets-dev deleted
 - ├─ aws:iam:RolePolicyAttachment rpa-secrets deleted
 - ├─ aws:iam:Policy secretsPolicy deleted
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret1 deleted
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret2 deleted
 - ├─ aws:lambda:Function lambdaWithSecrets deleted
 - ├─ aws:iam:RolePolicyAttachment rpa-basic deleted
 - ├─ aws:secretsmanager:Secret secret2 deleted
 - ├─ aws:secretsmanager:Secret secret1 deleted
 - └─ aws:iam:Role roleLambdaWithSecrets deleted

Outputs:
 - lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 - 10 deleted

Duration: 9s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
If you want to remove the stack completely, run `pulumi stack rm dev`.

And to complete the tidying up, let’s delete the stack and all the relevant history as well:

pulumi stack rm dev

Complete code

If you just want to copy and paste the entire index.ts file without worrying which section goes where, I got you covered:

// Copyright 2016-2022, Pulumi Corporation. All rights reserved.

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

interface LambdaConfig {
 envvars: string[];
 secrets: pulumi.Output<string>[];
}

const config = new pulumi.Config();
const lambdaconfig = config.requireObject<LambdaConfig>("lambdawithsecrets");

const secretsArnArray: pulumi.Output<string>[] = new Array();
const secretArray: aws.secretsmanager.Secret[] = new Array();
for (const key in lambdaconfig.secrets) {
 if (lambdaconfig.secrets.hasOwnProperty(key)) {
 const secret = new aws.secretsmanager.Secret(`${key}`);
 const secretVersion = new aws.secretsmanager.SecretVersion(`secretversion-${key}`, {
 secretId: secret.id,
 secretString: lambdaconfig.secrets[key],
 });
 secretArray.push(secret);
 secretsArnArray[key.toLocaleUpperCase()] = secret.id;
 }
}

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(aws.iam.Principals.LambdaPrincipal),
});

const rpaBasic = new aws.iam.RolePolicyAttachment("rpa-basic", {
 role: role,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
});



const secretManagerPolicyDoc = aws.iam.getPolicyDocumentOutput({
 statements: [
 {
 effect: "Allow",
 actions: [
 "secretsmanager:GetSecretValue",
 ],
 resources: secretArray.map(x => pulumi.interpolate`${x.arn}`),
 },
 ],
});

const secretManagerPolicy = new aws.iam.Policy("secretsPolicy", {
 policy: secretManagerPolicyDoc.apply(doc => doc.json),
});

const rpaSecrets = new aws.iam.RolePolicyAttachment("rpa-secrets", {
 role: role,
 policyArn: secretManagerPolicy.arn,
});

const lambdaEnvVars: pulumi.Input<{
 [key: string]: pulumi.Input<string>;
}> = {};

for (const key in secretsArnArray) {
 if (secretsArnArray.hasOwnProperty(key)) {
 lambdaEnvVars[key] = secretsArnArray[key];
 }
}

for (const key in lambdaconfig.envvars) {
 if (lambdaconfig.envvars.hasOwnProperty(key)) {
 lambdaEnvVars[key.toLocaleUpperCase()] = lambdaconfig.envvars[key];
 }
}

const lambda = new aws.lambda.Function("lambdaWithSecrets", {
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./app"),
 }),
 role: role.arn,
 handler: "index.handler",
 runtime: aws.lambda.Runtime.NodeJS16dX,
 environment: {
 variables: lambdaEnvVars,
 },
 timeout: 15,
});

export const lambdaName = lambda.name;

Summary

This post describes one of the ways that you can securely access secrets in your Lambda Functions by adding secrets to your stack config, uploading them to AWS Secrets Manager and then accessing them in your functions, all without exposing the plaintext values. There are other ways, but I think this is one of the neatest.

Of course you can use other tools to manage your secrets, such as Azure Key Vault, Google Cloud Secret Manager or Hashicorp Vault, getting the path to the key back from the resource and using it in the same way as we used the ARN from the Secret.

If you’d like more information on how you can use Pulumi to securely and easily deploy your cloud infrastructure using modern programming languages, get in touch and ask us how.

How to Use ECS Anywhere with Pulumi [Step-by-Step Guide]

Piers Karsenbarg — Thu, 27 May 2021 00:00:00 +0000

This post is outdated and contains references to a pre-release version of Pulumi Crosswalk (@pulumi/awsx). For updated AWSx documentation and examples, see the AWS Guides.

When Amazon’s Elastic Container Service (ECS) first launched in 2014, it enabled an easy and convenient way of deploying and scheduling containers in the AWS ecosystem. Back then, you would run a set of EC2 instances, and ECS would deploy containers to instances based on the size, resources, and placement requirements you specified.

In 2018, AWS released Fargate, bringing a serverless feel to the platform. You could have ECS run your container without having to manage any infrastructure.

The next iteration is ECS Anywhere, which allows you to deploy your containers to almost any server. You can deploy to an EC2 instance inside AWS, a virtual machine (VM) in another cloud provider, or even a bare metal server in your private data center. Best of all, you can control the deployments in the same way you would with EC2 or Fargate.

Why would I want to use this?

As we’ve seen over the past few years, the ECS system makes it a convenient service to use. There are several reasons why you might want to use ECS as a control plane for your container-based applications but not run the actual applications within your AWS infrastructure. These include:

Latency requirements: You have systems on another network, and you need your containers to be running on that same network to give you the low latency required.
Regulatory requirements: There may be legal, security, or privacy needs that can only be satisfied by running your application in your own data center.
Operational requirements: you don’t want to manage a complex container cluster yourself on another network and are already running workloads on ECS.

Behind the scenes

The crucial part of understanding how ECS Anywhere works is that you’re using ECS to manage the deployment of the containers to your VMs (which we’ll call nodes going forward). The actual interaction between your application and whatever network traffic or instructions it receives is separate. You’ll see this shown in the example below where we deploy two VMs and a load balancer. Once the application is up and running, ECS takes a back seat and waits for the instructions to deploy a new version or stop the application from running altogether.

There are three parts to getting this to work:

1. Setting up the AWS infrastructure

AWS’ Systems Manager has a feature called “Hybrid Activations.” We’ll use it to link the external nodes to the ECS cluster using an activation code and an activation id. We’ll also create various IAM roles necessary to activate the nodes and assign them to the ECS cluster. In our example, we’ll also set up IAM roles so that our application can send logs to CloudWatch as well.

2. Deploying the nodes

During the node setup process, we’re going to download and run a custom script from AWS that performs several tasks:

It installs the docker engine.
It installs the SSM agent and registers it with the Systems Manager’s Fleet manager (which keeps track of the nodes and assigns an IAM role)
It installs the ECS agent, which connects to the ECS cluster and listens for instructions.

3. Deploying the application

There are very few differences between deploying an application to ECS internally on AWS infrastructure here. We set up a Task Definition and a Service within the ECS infrastructure, and the only difference is that we’ll use the new launch type of EXTERNAL instead of EC2 or FARGATE, which tells the cluster to push the instructions for running the application to our nodes.

Deploying a Node.js application using ECS Anywhere

In this example, we will be deploying a basic Node.js application within a container to a pair of virtual machines (VMs) behind a load balancer. You’ll need multiple VMs because ECS won’t deploy the multiple task definitions with the same open network port to the same node (this also happens within the EC2 flavor of ECS).

Let’s be constant

Before setting up our cloud resources, we begin by setting some variables that we’ll use later on throughout our Pulumi program.

We’ll first set up our configuration settings in our command line:

pulumi config set aws:region us-east-1
pulumi config set numberNodes 2

The first tells Pulumi which region to deploy the resources to (and we’ll also use it when setting up our nodes and CloudWatch logging). The second tells us how many nodes to create, and we’ll need at least two to allow ECS to perform a rolling deployment.

Since we’re using a third-party compute provider to prove that ECS can deploy the containers to non-AWS infrastructure, we need to provide an API token. In this example, we’ll add it to the config, but you should read our Setup Guide for more information on how to deploy.

To add your token to the config (as a secret), you should run the following command:

pulumi config set digitalocean:token XXXXXXXXXXXX --secret

Exchange XXXXXXXXXXXX for a personal access token located in the Control Panel.

We’ll use these to set variables in our code:

import * as pulumi from "@pulumi/pulumi";

const awsConfig = new pulumi.Config("aws");
const awsRegion = awsConfig.get("region");

const projectConfig = new pulumi.Config();
const numberNodes = projectConfig.getNumber("numberNodes") || 2;

(notice that for the number of nodes, we’re setting a default value in case this configuration setting does not exist)

Setting up IAM roles

We’re going to need three IAM roles:

The first is for the AWS Systems manager. We’re going to attach two managed policies to this: one to enable its core functionality (AmazonSSMManagedInstanceCore) and the default EC2 role for ECS (AmazonEC2ContainerServiceforEC2Role).
The second is the ECS execution role, and to that, we’re going to attach the managed policy to allow other AWS services to run ECS tasks (AmazonECSTaskExecutionRolePolicy)
The final one is the Task role, and we’re going to add a custom policy to that role to allow us to push logs from the nodes to CloudWatch.

const ssmRole = new aws.iam.Role("ssmRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.SsmPrincipal
 ),
});

new aws.iam.RolePolicyAttachment("rpa-ssmrole-ssminstancecore", {
 policyArn: aws.iam.ManagedPolicy.AmazonSSMManagedInstanceCore,
 role: ssmRole,
});

new aws.iam.RolePolicyAttachment("rpa-ssmrole-ec2containerservice", {
 policyArn: aws.iam.ManagedPolicy.AmazonEC2ContainerServiceforEC2Role,
 role: ssmRole,
});

const executionRole = new aws.iam.Role("taskExecutionRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.EcsTasksPrincipal
 ),
});

new aws.iam.RolePolicyAttachment("rpa-ecsanywhere-ecstaskexecution", {
 role: executionRole,
 policyArn: aws.iam.ManagedPolicy.AmazonECSTaskExecutionRolePolicy,
});

const taskRole = new aws.iam.Role("taskRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.EcsTasksPrincipal
 ),
});

new aws.iam.RolePolicy("taskRolePolicy", {
 role: taskRole.id,
 policy: {
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Action: [
 "ssmmessages:CreateControlChannel",
 "ssmmessages:CreateDataChannel",
 "ssmmessages:OpenControlChannel",
 "ssmmessages:OpenDataChannel",
 ],
 Resource: "*",
 },
 {
 Effect: "Allow",
 Action: ["logs:DescribeLogGroups"],
 Resource: "*",
 },
 {
 Effect: "Allow",
 Action: [
 "logs:CreateLogStream",
 "logs:CreateLogGroup",
 "logs:DescribeLogStreams",
 "logs:PutLogEvents",
 ],
 Resource: "*",
 },
 ],
 }
});

Deploying the ECS cluster and node infrastructure

Next, we’re going to set up the System Manager activation, which provides an Id and a code that lets our nodes connect to the cluster. We’ll set them up at the same time as the CloudWatch log group that we’ll push application logs to later on:

const ssmActivation = new aws.ssm.Activation("ecsanywhere-ssmactivation", {
 iamRole: ssmRole.name,
 registrationLimit: numberNodes,
});

const cluster = new aws.ecs.Cluster("cluster");

export const clusterName = cluster.name;

const logGroup = new aws.cloudwatch.LogGroup("logGroup");

We’re using the ssmRole we created earlier to allow the Systems Manager to do this. You’ll also notice we’re exporting the clusterName here. We will use this to tidy up later on when we tear down our infrastructure.

When we create the nodes, we will run an AWS bash script provided as part of the setup and pass it into the resource as userData. In this script, we pass in the cluster name, the activation id, the activation code, and the AWS region where the cluster will be deployed.

const userData = pulumi
 .all([ssmActivation.activationCode, ssmActivation.id, cluster.name])
 .apply(
 ([activationCode, activationId, clusterName]) => `#!/bin/bash
# Download the ecs-anywhere install Script
curl -o "ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" && sudo chmod +x ecs-anywhere-install.sh

# (Optional) Check integrity of the shell script
curl -o "ecs-anywhere-install.sh.sha256" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh.sha256" && sha256sum -c ecs-anywhere-install.sh.sha256

# Run the install script
sudo ./ecs-anywhere-install.sh \
 --cluster ${clusterName} \
 --activation-id ${activationId} \
 --activation-code ${activationCode} \
 --region ${awsRegion}
`
 );

This next section of code deploys the VMs.

We’re going to create a tag that attaches the nodes to a load balancer. Next, we’ll create the VMs and assign the tag. Finally, we’ll deploy the load balancer and assign the same tag to route traffic to the nodes. We’ll also take the IP address of the load balancer and mark it as an output so we have something we can either use in a browser or call with curl to ensure that our deployment has worked.

In the end, we’re going to output the load balancer IP address so we have something we can either use in a browser or call with curl to ensure that our deployment has worked.

const loadBalancerTag = new digitalocean.Tag("lb");

for(let i = 1; i <= numberNodes; i++) {
 new digitalocean.Droplet(`droplet-${i}`, {
 region: digitalocean.Regions.NYC1,
 size: "s-1vcpu-2gb",
 image: "ubuntu-20-04-x64",
 userData: userData,
 tags: [loadBalancerTag.id]
 });
}

const lb = new digitalocean.LoadBalancer("lb", {
 region: digitalocean.Regions.NYC1,
 forwardingRules: [{
 entryPort: 80,
 entryProtocol: digitalocean.Protocols.HTTP,
 targetPort: 80,
 targetProtocol: digitalocean.Protocols.HTTP
 }],
 healthcheck: {
 port: 80,
 protocol: digitalocean.Protocols.HTTP,
 path: "/"
 },
 dropletTag: loadBalancerTag.name
});

export const ip = lb.ip;

The last part of the example will be familiar to anyone using ECS as a container scheduler: setting up the container repository, describing the task definition, and creating the service. The difference is that we use new compatibility in the task definition and a new launch type in the service - EXTERNAL. This tells ECS to push the container and start them on the external nodes.

const repo = new awsx.ecr.Repository("app");

const image = repo.buildAndPushImage("./app");

const taskDefinition = pulumi.all([image, logGroup.name, logGroup.namePrefix]).apply(
 ([img, logGroupName, nameprefix]) =>
 new aws.ecs.TaskDefinition("taskdefinition", {
 family: "ecs-anywhere",
 requiresCompatibilities: ["EXTERNAL"],
 taskRoleArn: taskRole.arn,
 executionRoleArn: executionRole.arn,
 containerDefinitions: JSON.stringify([
 {
 name: "app",
 image: img,
 cpu: 256,
 memory: 256,
 essential: true,
 portMappings: [
 {
 containerPort: 80,
 hostPort: 80
 },
 ],
 logConfiguration: {
 logDriver: "awslogs",
 options: {
 "awslogs-group": logGroupName,
 "awslogs-region": awsRegion,
 "awslogs-stream-prefixs": nameprefix
 }
 }
 },
 ]),
 })
);

const service = new aws.ecs.Service("service", {
 launchType: "EXTERNAL",
 taskDefinition: taskDefinition.arn,
 cluster: cluster.id,
 desiredCount: numberNodes - 1,
 }
);

Now you can run pulumi up, and once it’s all deployed, run curl ${pulumi stack output ip}.

If you want to update the Dockerfile or the Node.js application, run pulumi up again. You’ll see the new container deployed. Then you can run curl ${pulumi stack output ip} again to see the new response.

Finally, when you want to destroy the infrastructure, there is an additional step before running the usual pulumi destroy. This is because the nodes are registered to AWS Systems Manager and the ECS cluster as part of the node setup and happen outside of the Pulumi stack.

Run the following in your command line (you’ll need to install jq for this to work):

aws ssm describe-instance-information | jq ".InstanceInformationList | .[] | .InstanceId" | grep "mi-" | xargs -L 1 aws ssm deregister-managed-instance --instance-id

# Cleanup ECS resources
aws ecs list-container-instances --cluster ${pulumi stack output clusterName} | jq ".containerInstanceArns | .[]" | xargs -L 1 aws ecs deregister-container-instance --cluster ${pulumi stack output clusterName} --force --container-instance

# Refresh Pulumi stack
pulumi refresh -y

And then you can tear down all your infrastructure with the usual pulumi destroy.

You can see this code in its entirety in our examples repository on GitHub.

Summary

ECS Anywhere is an excellent addition to Amazon Web Services’ suite of container orchestration services. I can see many companies using it to run containers in their private infrastructure but don’t necessarily have the means, know-how, or drive to set up and run their Kubernetes cluster or another such tool.

We’ve shown you how to set up your ECS cluster to use external nodes, build and deploy the cluster and application and see for yourself how easy it is to deploy subsequent changes. Pulumi makes it simple to create container images and provision and manage cloud infrastructure on any cloud using familiar programming languages, including TypeScript, Python, Go, and .NET. Cloud engineers can manage container images, ECR registries, and multi-cloud deployments within the same infrastructure definition.