Pulumi Blog: Stack references

Dependent Stack Updates with Pulumi Deployments

Evan Boyle — Tue, 20 Jun 2023 00:00:00 +0000

As infrastructure projects grow in size and complexity, you need to decompose infrastructure into smaller stacks to limit the blast radius of errors, extract and reference common layers like networking, and limit access to sensitive components. This comes with a coordination cost as you now need to figure out how to detect and propagate changes to downstream stacks in your dependency graph. Today we’re announcing two features that can help you manage this complexity by automatically updating dependent stacks:

Deployment Webhook Destinations for triggering downstream stack updates automatically
The Pulumi Auto Deploy Package for easily expressing dependencies and configuring downstream updates via Deployment Webhooks.

The following example shows automatic deployment of stacks with the following dependency graph
a
├── b
│ ├── d
│ ├── e
│ └── f
└── c
Whenever a node in the graph is updated, all downstream nodes will be automatically updated
via a webhook triggering Pulumi Deployments.

Automatically updating dependent stacks and providing a way to write Pulumi programs that can express nested stack updates are two of the most requested features across the Pulumi project, and we’re excited to enable both of these via Pulumi Cloud and Pulumi Deployments.

Deployment Webhook Destinations

Pulumi Webhooks allow you to pick one or more event types on a stack (i.e. update succeeded, or refresh failed) and automatically deliver an event to a destination such a slack channel or a webserver you’ve deployed with custom code. Now you can choose Pulumi Deployments as a webhook destination. When an update completes on the parent stack, a deployment will automatically be triggered on the destination stack. This makes it easy to keep dependent stacks up to date. For instance, you can use Deployment Webhook Destinations to configure every successful update on your networking stack to trigger downstream updates on your database and compute stacks.

import * as pulumi from "@pulumi/pulumi";
import * as pulumiservice from "@pulumi/pulumiservice";

const databaseWebhook = new pulumiservice.Webhook("databaseWebhook", {
 organizationName: "org",
 projectName: "network",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "database/prod",
 active: true,
 displayName: "deploy-database",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});
const computeWebhook = new pulumiservice.Webhook("computeWebhook", {
 organizationName: "org",
 projectName: "database",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "compute/prod",
 active: true,
 displayName: "deploy-compute",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using PulumiService = Pulumi.PulumiService;

return await Deployment.RunAsync(() =>
{
 var databaseWebhook = new PulumiService.Webhook("databaseWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "network",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "database/prod",
 Active = true,
 DisplayName = "deploy-database",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

 var computeWebhook = new PulumiService.Webhook("computeWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "database",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "compute/prod",
 Active = true,
 DisplayName = "deploy-compute",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

});

import pulumi
import pulumi_pulumiservice as pulumiservice

database_webhook = pulumiservice.Webhook("databaseWebhook",
 organization_name="org",
 project_name="network",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="database/prod",
 active=True,
 display_name="deploy-database",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])
compute_webhook = pulumiservice.Webhook("computeWebhook",
 organization_name="org",
 project_name="database",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="compute/prod",
 active=True,
 display_name="deploy-compute",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := pulumiservice.NewWebhook(ctx, "databaseWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("network"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("database/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-database"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 _, err = pulumiservice.NewWebhook(ctx, "computeWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("database"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("compute/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-compute"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
resources:
 databaseWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: network
 stackName: prod
 format: pulumi_deployments
 payloadUrl: database/prod
 active: true
 displayName: deploy-database
 filters:
 - update_succeeded
 computeWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: database
 stackName: prod
 format: pulumi_deployments
 payloadUrl: compute/prod
 active: true
 displayName: deploy-compute
 filters:
 - update_succeeded

Expressing Stack Dependencies with Pulumi Auto Deploy

We’ve also shipped a new Pulumi package called Pulumi Auto Deploy. It lets you simply express dependencies between stacks, and takes care of creating and updating the necessary Deployment Webhooks under the hood.

import * as autodeploy from "@pulumi/auto-deploy";
import * as pulumi from "@pulumi/pulumi";

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

const organization = pulumi.getOrganization();
const project = "dependency-example"

export const f = new autodeploy.AutoDeployer("auto-deployer-f", {
 organization,
 project,
 stack: "f",
 downstreamRefs: [],
});

export const e = new autodeploy.AutoDeployer("auto-deployer-e", {
 organization,
 project,
 stack: "e",
 downstreamRefs: [],
});

export const d = new autodeploy.AutoDeployer("auto-deployer-d", {
 organization,
 project,
 stack: "d",
 downstreamRefs: [],
});

export const c = new autodeploy.AutoDeployer("auto-deployer-c", {
 organization,
 project,
 stack: "c",
 downstreamRefs: [],
});

export const b = new autodeploy.AutoDeployer("auto-deployer-b", {
 organization,
 project,
 stack: "b",
 downstreamRefs: [d.ref, e.ref, f.ref],

});

export const a = new autodeploy.AutoDeployer("auto-deployer-a", {
 organization,
 project,
 stack: "a",
 downstreamRefs: [b.ref, c.ref],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AutoDeploy = Pulumi.AutoDeploy;

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

return await Deployment.RunAsync(() =>
{
 var projectVar = "dependency-example";

 var organization = "pulumi";

 var f = new AutoDeploy.AutoDeployer("f", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "f",
 DownstreamRefs = new[] {},
 });

 var e = new AutoDeploy.AutoDeployer("e", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "e",
 DownstreamRefs = new[] {},
 });

 var d = new AutoDeploy.AutoDeployer("d", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "d",
 DownstreamRefs = new[] {},
 });

 var c = new AutoDeploy.AutoDeployer("c", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "c",
 DownstreamRefs = new[] {},
 });

 var b = new AutoDeploy.AutoDeployer("b", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "b",
 DownstreamRefs = new[]
 {
 d.Ref,
 e.Ref,
 f.Ref,
 },
 });

 var a = new AutoDeploy.AutoDeployer("a", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "a",
 DownstreamRefs = new[]
 {
 b.Ref,
 c.Ref,
 },
 });

});

import pulumi
import pulumi_auto_deploy as auto_deploy

'''

The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
Whenever a node in the graph is updated,
all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
'''

project_var = "dependency-example"
organization = pulumi.get_organization()
f = auto_deploy.AutoDeployer("f",
 organization=organization,
 project=project_var,
 stack="f",
 downstream_refs=[])
e = auto_deploy.AutoDeployer("e",
 organization=organization,
 project=project_var,
 stack="e",
 downstream_refs=[])
d = auto_deploy.AutoDeployer("d",
 organization=organization,
 project=project_var,
 stack="d",
 downstream_refs=[])
c = auto_deploy.AutoDeployer("c",
 organization=organization,
 project=project_var,
 stack="c",
 downstream_refs=[])
b = auto_deploy.AutoDeployer("b",
 organization=organization,
 project=project_var,
 stack="b",
 downstream_refs=[
 d.ref,
 e.ref,
 f.ref,
 ])
a = auto_deploy.AutoDeployer("a",
 organization=organization,
 project=project_var,
 stack="a",
 downstream_refs=[
 b.ref,
 c.ref,
 ])

package main

import (
 "github.com/pulumi/pulumi-auto-deploy/sdk/go/autodeploy"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 /**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

 pulumi.Run(func(ctx *pulumi.Context) error {
 projectVar := "dependency-example"
 organization := "pulumi"
 f, err := autodeploy.NewAutoDeployer(ctx, "f", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("f"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 e, err := autodeploy.NewAutoDeployer(ctx, "e", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("e"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 d, err := autodeploy.NewAutoDeployer(ctx, "d", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("d"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 c, err := autodeploy.NewAutoDeployer(ctx, "c", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("c"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 b, err := autodeploy.NewAutoDeployer(ctx, "b", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("b"),
 DownstreamRefs: pulumi.StringArray{
 d.Ref,
 e.Ref,
 f.Ref,
 },
 })
 if err != nil {
 return err
 }
 _, err = autodeploy.NewAutoDeployer(ctx, "a", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("a"),
 DownstreamRefs: pulumi.StringArray{
 b.Ref,
 c.Ref,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
variables:
 project: dependency-example
 # TODO: update once https://github.com/pulumi/pulumi-yaml/issues/461 is fixed
 organization: pulumi
resources:
 f:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: f
 downstreamRefs: []
 e:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: e
 downstreamRefs: []
 d:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: d
 downstreamRefs: []
 c:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: c
 downstreamRefs: []
 b:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: b
 downstreamRefs:
 - ${d.ref}
 - ${e.ref}
 - ${f.ref}
 a:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: a
 downstreamRefs:
 - ${b.ref}
 - ${c.ref}

Just declare an AutoDeployer resource for each stack, and any downstream stacks will automatically be updated whenever a stack in its dependency chain is updated.

Scale Infrastructure with Software

At Pulumi, we build tools to help you scale your infrastructure footprint with software instead of just humans. Deployment Webhook Destinations and pulumi-auto-deploy give you the tools to keep all of your dependent infrastructure up to date with automation instead of manual runbooks and human-driven operations.

Check out these resources to get started today:

Check out the Pulumi Deployments documentation and learn about configuring Deployment Settings for your stacks.
Get started with the Pulumi Auto Deploy Package.
Learn more about Pulumi Cloud Webhooks.
Join us in the #pulumi-deployments community slack channel.

🚀🚀🚀 Happy building! 🚀🚀🚀

Streamline access to stack outputs with OutputDetails

Abhinav Gupta — Fri, 24 Mar 2023 00:00:00 +0000

You can use the new StackReference.getOutputDetails method to get output values from other stacks directly—without calling Output.apply.

What are Stack References?

If you already understand stack references, skip on to Introducing OutputDetails.

Stack references are a useful feature of Pulumi that you can use to access output values from one stack in another.

For instance, let’s say you have a ’network’ stack that provisions an AWS VPC, and an ‘app’ stack that deploys your application on an EC2 instance in that VPC. The app stack can reference the output of the network stack to obtain the VPC ID, which it needs to configure the EC2 instance.

const network = new pulumi.StackReference("acmecorp/network/other");
const vpcId = network.getOutput("vpcId");

The getOutput method above returns an Output value that holds the result. You can call Output.apply to perform transformations on the output value. As an example, if you need to extract a subnet ID from a JSON string in the output, you can use the following code:

const subnetConfig = network.getOutput("subnetConfig");
const subnetId = subnetConfig.apply(output => JSON.parse(output).subnetId);

This approach works but is limited because the output value obtained from the apply transformation is also an Output value. You cannot access or manipulate this value directly, and you must continue to use apply repeatedly to consume the value.

As an aid in resolving this problem, we have introduced OutputDetails. These provide direct access to the output of one stack from another stack.

Introducing `OutputDetails`

We’ve added a new method, StackReference.getOutputDetails, to allow direct access to output values from stack references. This method can be used as follows:

const vpcIdDetails = await network.getOutputDetails("vpcId");
const vpcId = vpcIdDetails.value;

Unlike the getOutput method, the resulting value is a plain value, not an Output value. As a result, there is no need to call apply repeatedly to transform or consume the value.

Revisiting the previous example of extracting a value from a JSON string, getOutputDetails can be used as shown below:

const subnetConfig = (await network.getOutputDetails("subnetConfig")).value;
const subnetId = JSON.parse(subnetConfig).subnetId;
console.log(`Using subnet ID ${subnetId}`);

What about secrets?

getOutputDetails can also be used to access [secret outputs generated by the referenced stack. For example, if the referenced stack sets up a database and produces the password as a stack output, you can access it as follows:

const passwordDetails = await dbStack.getOutputDetails("databasePassword");
const password = passwordDetails.secretValue;

Can we do this for all `Output` values?

Unfortunately, it is not feasible to do this to every Output value. Stack references are unique in that the referenced stack must already have deployed for the reference to be resolved. Therefore, when the stack reference is created, the outputs of the referenced stack are available immediately. This is not the case for other Output values. Their values only become available as pulumi update runs, so we cannot access them in the same way.

Availability

This feature is available in all stable SDKs for Pulumi, starting from v3.55.0. Give it a spin and share your thoughts with us on Slack!

Architect AWS Application Infrastructure

Paul Stack — Thu, 17 Oct 2019 00:00:00 +0000

In this post, we will talk about the best way to architect your Pulumi applications. We are going to build out the following infrastructure in AWS:

AWS Fargate service that does not serve traffic directly
AWS ALB as the entry point to the Fargate Service
AWS RDS Instance that is stored in a separate network from the Application and does not service traffic directly from the internet

To do this, we are going to split the infrastructure into two AWS VPCs. One VPC will be used for the application based resources (e.g Fargate Service, ALB) and the other VPC will be for the database resources. The VPCs will be peered to allow communication between them so that the application can communicate with the database securely.

Layer it up

When we think of infrastructure, we tend to think of layers. A layer is a logical set of components / resources that can be grouped together. In our case, we can think of the networking as a layer upon which our database and application can be built. For the purposes of this post, an infrastructure layer will be represented as a project in Pulumi. So let’s break our application into separate Pulumi projects:

$ tree -d -L 1
.
├── application
├── database
└── networking

This means all of the work required for the networking components can be in the network project. We can use Stack References in Pulumi to expose any required information from the networking layer, to the database and application layers.

Let’s create the networking project.

Building our Networking Project

$ cd networking && pulumi new aws-typescript
$ pulumi config set aws:region us-east-2

This gives us the shell of our project we need to create our resources. In thinking of how we build our networking resources, we will create two copies of each resource due to separate VPCs for application and database. Of course, we can use a loop to do this, but the more elegant way would be to create a class that we could pass parameters into to make life easy. We want an API as follows:

const availabilityZones = aws.getAvailabilityZones({
 state: "available",
});

const myVpc = new Vpc("my-vpc", {
 description,
 baseTags,

 baseCidr,
 availabilityZoneNames,
 enableFlowLogs,
});

We can see the power of Pulumi when we build our infrastructure with TypeScript. We create a TypeScript class that extends Pulumi ComponentResource. ComponentResource is a resource that aggregates one or more other child resources into a higher level abstraction. The component resource itself is a resource, but does not require custom create, read, update and delete operations for provisioning.

The outline of the class will look as follows:

export class Vpc extends ComponentResource {
 // ...
}

We want to pass a name and some custom parameters to the class, so we create an interface for the parameters:

export interface VpcArgs {
 // ...
}

We update our class with a constructor that allows us to create a new instance of the VPC class:

export class Vpc extends ComponentResource {
 constructor(name: string, args: VpcArgs, opts?: ComponentResourceOptions) {
 super("vpc", name, {}, opts);

 // ... resources here ...
 }
}

After we implement all of the resources needed for our VPC, our Pulumi application looks like the following:

async function main() {
 const config = new Config();
 const azCount = config.getNumber("azCount") || 2;

 const availabilityZones = aws.getAvailabilityZones({
 state: "available",
 });

 const appVpc = new Vpc("app-vpc", {
 baseCidr: "172.28.0.0/16",
 availabilityZoneNames: availabilityZones.names.slice(0, azCount),
 enableFlowLogs: true,
 });

 const dataVpc = new Vpc("data-vpc", {
 baseCidr: "172.18.0.0/16",
 availabilityZoneNames: availabilityZones.names.slice(0, azCount),
 enableFlowLogs: true,
 });

 return {
 appVpcId: appVpc.vpcId(),
 dataVpcId: dataVpc.vpcId(),
 }
}

module.exports = main();

This application will create two instances of the VPC class containing all of the resources needed for a VPC. This reusable VPC class` is a more efficient way of building out infrastructure than to simply loop over the resources. We can take advantage of relating the resources to the correct parent to build out graph.

We take it one step further, and be able to peer the VPCs. TypeScript, again, gives us the ability to do this in a concise manner by using a member function. The member function of the VPC class looks as follows:

public configurePeering(args: PeerToArgs) {
 // ...
}

This gives us the ability to peer the VPCs:

appVpc.configurePeering({
 peerVpc: dataVpc,
 nameTag: `${baseTags.ManagedBy} Peer App to Data`,
 routeSubnets: "private",
});

After creating a similar member function for the peering security group, we instruct Pulumi to create the resources by running the command pulumi up. This displays a preview of changes first and the list of AWS resources it will create:

$ pulumi up

After being prompted and selecting “yes”, your deployment will begin. After completion, you will see:

...
 + │ ├─ aws:ec2:VpcEndpoint data-vpc-s3-endpoint created
 + │ └─ aws:ec2:VpcEndpoint data-vpc-dynamodb-endpoint created
 + └─ aws:ec2:SecurityGroup data-vpc-app-vpc-sg created

Outputs:
 appVpcId : "vpc-07420f4b22cce694a"
 appVpcPrivateSubnetIds : [
 [0]: "subnet-07ebb7bb1894f76ea"
 [1]: "subnet-0cc6c64ffce7fa74a"
 ]
 appVpcPublicSubnetIds : [
 [0]: "subnet-07eec9d474411d47c"
 [1]: "subnet-093c9e201b5bb3975"
 ]
 dataVpcId : "vpc-04d28ded952454899"
 dataVpcPrivateSubnetIds: [
 [0]: "subnet-048e68b5d12d6189d"
 [1]: "subnet-088fbb5acf3d0a58c"
 ]
 dataVpcPublicSubnetIds : [
 [0]: "subnet-0711e414ccc7458a3"
 [1]: "subnet-0b8d54dfec34b286b"
 ]
 peeredSecurityGroupId : "sg-08b58260a840a971b"

Resources:
 + 61 created

Duration: 3m8s

Building our Database Project

$ cd database && pulumi new aws-typescript
$ pulumi config set aws:region us-east-2

Our database project follows the same principals as the networking project - it has a RDS class that extends the Pulumi ComponentResource.

export interface RdsArgs {
 // ...
}
export class RdsInstance extends ComponentResource {
 constructor(name: string, args: RdsArgs, opts?: ComponentResourceOptions) {
 super("db", name, {}, opts);

 // ... resources here ...
 }
}

This project needs other configuration options set:

pulumi config set database-layer:dbUsername MyDBUser
pulumi config set database-layer:dbName initialDb
pulumi config set database-layer:dbPassword MyDBPassword1234! --secret

The database project needs to access the details of the resources creating in the networking layer. To do this, we use a Pulumi StackReference. We pass the correct stack name that the database project needs to talk to, so we can pass that as a configuration option. A stack reference in Pulumi will follow the structure org/project/stack. In our case it will be stack72/networking-layer/dev and we can pass that as a configuration option to the database project:

$ pulumi config set networkingStack stack72/networking-layer/dev

Inside our database project, we can access any of the exported values from the networking project. The first thing we need access to is the database private subnet Ids. To access those subnetIds, we would do the following:

import {Config, StackReference} from "@pulumi/pulumi";

const config = new Config();
const networkingStack = new StackReference(config.require("networkingStack"))

const subnetIds = networkingStack.getOutput("dataVpcPrivateSubnetIds")

Please Note: We need to pass the exact name of the output as specified in the networking stack!

Our database project looks like this:

const rds = new RdsInstance("db-instance", {
 description: `${baseTags.Project} DB Instance`,
 baseTags: baseTags,

 subnetIds: networkingStack.getOutput("dataVpcPrivateSubnetIds"),

 username: dbUsername,
 password: dbPassword,
 initalDbName: dbName,

 allocatedStorage: 40,
 engineVersion: "11.4",
 instanceClass: aws.rds.InstanceTypes.R3_Large,
 storageType: "gp2",

 finalSnapshotIdentifier: finalSnapshotIdentifier,

 sendEnhancedLogsToCloudwatch: true,
 monitoringInterval: 10,

 securityGroupIds: [networkingStack.getOutput("peeredSecurityGroupId")],
});

We can then instruct Pulumi to create our resources with a pulumi up command, as before. This will show you a preview of changes first and the list of AWS resources it will create:

$ pulumi up

After being prompted and selecting “yes”, your deployment will begin. After completion, you will see:

Updating (dev):

 Type Name Status
 + pulumi:pulumi:Stack database-layer-dev created
 + ├─ db db-instance created
 + │ ├─ aws:rds:SubnetGroup db-instance-subnet-group created
 + │ ├─ aws:iam:Role db-instance-enhanced-monitoring-role created
 + │ ├─ aws:iam:RolePolicyAttachment enhanced-policy-attachment created
 + │ └─ aws:rds:Instance db-instance-rds created
 + └─ random:index:RandomString my-random-string created

Outputs:
 dbAddress : "db-instance-rdsd86e208.cep6renxwgtk.us-west-2.rds.amazonaws.com"
 dbEndpoint: "db-instance-rdsd86e208.cep6renxwgtk.us-west-2.rds.amazonaws.com:5432"
 dbName : "initialDb"
 dbPassword: "[secret]"
 dbPort : "5432"
 dbUsername: "MyDBUser"

Resources:
 + 7 created

Duration: 7m49s

Building the Application Project

$ cd application && pulumi new aws-typescript
$ pulumi config set aws:region us-east-2

It will come as no surprise to you that we will build our application project in the same manner as that of networking and database. So we will have an Application class that extends Pulumi ComponentResource.

export interface ApplicationArgs {
 // ....
}
export class Application extends ComponentResource {
 constructor(name: string, args: ApplicationArgs, opts?: ComponentResourceOptions) {
 super("application", name, {}, opts);

 // ... resources here ...
 }
}

The application project differs from the database project in that it needs stack references to the networking and database projects. Let’s specify those via configuration as we did in the database project:

$ pulumi config set networkingStack stack72/networking-layer/dev
$ pulumi config set databaseStack stack72/database-layer/dev

We can then access details like, database endpoint, database user, application Vpc Id within our application project:

import * as awsx from "@pulumi/awsx";
import { Config, getStack, StackReference } from "@pulumi/pulumi";
import {Application} from "./application";

const config = new Config();

const networkingStack = new StackReference(config.require("networkingStack"));
const databaseStack = new StackReference(config.require("databaseStack"));

const app = new Application("app", {

 vpcId: networkingStack.getOutput("appVpcId"),

 // ALB in public subnets
 albSubnetIds: networkingStack.getOutput("appVpcPublicSubnetIds"),

 // App resources in private subnets
 appSubnetIds: networkingStack.getOutput("appVpcPrivateSubnetIds"),

 appImage: awsx.ecs.Image.fromPath("app", "./src/backend"),
 appPort: 80,

 dbName: databaseStack.getOutput("dbName"),
 dbUsername: databaseStack.getOutput("dbUsername"),
 dbPassword: databaseStack.getOutput("dbPassword"),
 dbPort: databaseStack.getOutput("dbPort"),
 dbHost: databaseStack.getOutput("dbAddress"),
});

export const albAddress = app.albAddress();

Notice that we are taking advantage of the Pulumi AWS Extensions. Pulumi AWSX is a set of component wrappers around many AWS operations. In our case, we are taking advantage of being able to create an AWS ECS container image from a known path to a dockerfile. This is a concise API that removes us from the need to run docker build as a separate process. It will happen as part of our Pulumi application.

We can, again, instruct Pulumi to create our resources for us by issuing the pulumi up command. In the interactive output, you will see how Pulumi interacts with docker and also see how it reads both of the stack references we told it to read information from. It will also show you a preview of the changes and the list of AWS resources it will create:

$ pulumi up

Testing our application

The application in this repository has a number of endpoints that we can use to test that the AWS ECS Fargate service is working as expected. We can test them either from a browser or from the command line. We can access the service using the address of the AWS ALB that was created to serve traffic to the application. Let’s test the endpoints from the CLI:

$ curl $(pulumi stack output albAddress)
{
 "hash": "59846138362875439341244157139878939483",
 "message": "hello world",
 "postgres": "please use the /postgres endpoint"
}

$ curl $(pulumi stack output albAddress)/postgres
{
 "hash": "59846138362875439341244157139878939483",
 "postgres": [
 [
 "PostgreSQL 11.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit"
 ]
 ]
}

$ curl $(pulumi stack output albAddress)/internet
I CAN connect to google.com%

$ curl $(pulumi stack output albAddress)/public-ip
<p>my external IP is: 52.24.188.63 (source: jsonip.com) </p>%

Our application is up and running and we can connect to it!

Cleaning up

To cleanup the resources we created, we need to go into each project directory and run the command

$ pulumi destroy --yes && pulumi stack rm

Once the resources have been deleted, then we can follow the interactive prompt to delete the stack.

Wrapping Up

In this post, we have discussed how we would break our infrastructure into logically grouped projects and how we share information between these projects. You can find a runable version of this infrastructure in our examples repository.

This demonstrates the suggested way of architecting Pulumi applications both for operability and collaboration.

You can try this code today get started with Pulumi today at https://www.pulumi.com and following the Getting Started for managing AWS resources with Pulumi.