Pulumi Blog: Javascript

Observable Infrastructure as Code

Kay Plößer — Thu, 20 Jan 2022 08:36:11 -0600

Pulumi community member Kay Plößer spent some time digging into setting up observability of a Pulumi deployment using Honeycomb. Read more to find out all the details on configuring Honeycomb and Pulumi together, with a side dish of Automation API!

Last few weeks, I procrastinated doing real product-related things with setting up monitoring for my infrastructure deployment.

I’m using @pulumidev, a cloud-provider independent IaC framework that allows me to define my infrastructure in TypeScript and use whatever provider I need.

I read something about Honeycomb’s observability approach to monitoring, and it got me thinking about my deployment code. Later, it will probably run with something like GitHub Actions, so it’s not on my machine when it executes. This means I need some insights in case things go wrong. But it would also be good to see how the deployment behaves over time. How do new resources affect deployment durations etc.?

This is what a trace could look like in Honeycomb:

Why not setting things up before I move deployment from my development machine to the cloud?

This article will show you how to make your Pulumi deployment observable with @honeycomb_io

Prerequisites

The Pulumi Automation-API

This API is a programmatical interface to the Pulumi CLI. Think Puppeteer for Pulumi instead of Chrome.

Since it controls the CLI, it can do everything the CLI can and more.

For our use case, we need information about which resource is deployed to tell Honeycomb about it, and it will create nice trace graphs for us.

To run a pulumi up command with the automation API, we need the following code:

const { LocalWorkspace } = require("@pulumi/pulumi/automation")

const main = async () => {
 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up()
}

main()

The @pulumi/pulumi package already includes the automation API, so it’s the only one we need.

We use the LocalWorkspace object to load our stack. This leaves the wholes stack intact; we can still deploy/destroy it with the CLI.

Then we call stack.up() and wait. The outputs generated from the API aren’t as pretty as the ones from the CLI, but they contain the same information.

Also, the up command of the API allows us to catch all events that are happening during the deployment. This is the data we’re after.

The Deployment Events

The up method of the stack object takes a config object. We can use the onEvent attribute to listen to all the changes.

stack.up({
 onEvent: (event) => {
 console.log(event)
 },
})

Here we log out every event. If we look at the output, we notice there are different types of events.

-The preludeEvent fires at the start with information about the stack config. -The resourcePreEvent fires before a resource is deployed, and the resOutputsEvent fires after it has been deployed. -The diagnosticEvent contains eventual errors that could happen. -The summaryEvent includes info about what has been done and if e everything was successful.

The event structure is a bit strange. The events don’t have a type attribute we could switch over, but an attribute called like the type of the event.

The following code shows how to get the right event, and where the data hides in that event:

onEvent: (event) => {
 if (event["preludeEvent"] !== undefined)
 return console.log(event.preludeEvent.config)

 if (event["resourcePreEvent"] !== undefined)
 return console.log(event.resourcePreEvent.metadata)

 if (event["resOutputsEvent"] !== undefined)
 return console.log(event.resOutputsEvent.metadata)

 if (event["diagnosticEvent"] !== undefined)
 return console.log(event.diagnosticEvent)

 if (event["summaryEvent"] !== undefined)
 return console.log(event.summaryEvent)
},

If we run things like this, we get all the information we need to send to Honeycomb. The next step is to tell Honeycomb what’s up.

Sending Data to Honeycomb

Honeycomb provides us with two libraries. The Node.js Beeline and Libhoney.

Libhoney is a low-level Honeycomb client that allows sending raw events to their service API. The Beeline is an abstraction on top of Libhoney to help with instrumentation and traces.

Usually, it would be best if you used the Beeline because it’s easy to set up and works with Node.js’s internal HTTP module and the Express framework out of the box.

In this case, we aren’t building an HTTP server, so the Beelines instrumentation doesn’t help much.

So for our deployment example, I will stick to Libhoney.

Honeycomb Events & Traces

Honeycomb uses events to figure out what’s happening in your system. Suppose the events have special attributes, like trace.trace_id and trace.span_id, Honeycomb can correlate them. This way, it can tell you things like, “The trace for your request took 300ms and consisted of 100ms authentication and 200ms database access.”

So you can pump all your information to Honeycomb via events. Sometimes it doesn’t need to be correlated; you only want to tell Honeycomb that your server uses 20% of memory. But in our case, we want to tie up all events related to one deployment into a trace, so we can figure out what we deploy, how long it takes, and if something goes wrong, what resource was responsible.

Linking up Pulumi & Honeycomb

We need to initialize Libhoney and send the correct “Honeycomb event” for every “Pulumi event.”

But let’s start with getting one event into Honeycomb: the pulumi up event.

const Libhoney = require("libhoney")
const { LocalWorkspace } = require("@pulumi/pulumi/automation")

const hny = new Libhoney({
 writeKey: "<HONEYCOMB_API_KEY>",
 dataset: "example-iac",
})

const id = (name) => `${name}-${Date.now()}`

const traceId = id`trace`
const rootSpanId = id`trace-root-span`
const startTimestamp = Date.now()

const main = async () => {
 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up({
 onEvent: (event) => {},
 })
}

main().then(() => {
 hny.sendNow({
 name: "up",
 service_name: "Pulumi",
 "trace.trace_id": traceId,
 "trace.span_id": rootSpanId,
 duration_ms: Date.now() - startTimestamp,
 })
})

We define a traceId, a rootSpanId, and a startTimestamp at the beginning, and then we send it off when everything is done.

An event with trace.trace_id will be grouped up with all other events with the same trace ID. In this case, we only have one event.

The name is usually the action you perform and want to measure, and the service_name is, well, the service that performs this action. In this case, we run pulumi up.

In the end, we also send a duration_ms attribute to track how long things take.

The result will look like this:

Not very interesting, but at least we know it run without crashing and how long it took.

Getting More Details

The next step is to get the details. What were the config parameters? How long did different parts of the deployment take?

For this, we need to look into two events, the preludeEvent and the summaryEvent.

Until now, we only send one Honeycomb event for the Pulumi up command.

Now, we will send three events for the stack.

The init event will contain the time it took to get to the actual deployment of resources and the config parameters of the stack
The run event will contain the duration all resources took to do their thing.
The finish event will contain the duration of shutting down the automation API and the summary data.

Let’s look at the updated example code:

...

const traceId = id`trace`
const rootSpanId = id`trace`
const startTimestamp = Date.now()
let stackFinishStartTimestamp

const main = async () => {
 const initStartTimestamp = Date.now()
 const runStackSpanId = id`stack`
 let stackRunStartTimestamp

 const stack = await LocalWorkspace.createOrSelectStack({
 stackName: "development",
 workDir: ".",
 })
 await stack.up({
 onEvent: (event) => {
 if (event["preludeEvent"] !== undefined) {
 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(initStartTimestamp)
 hnyEvent.add({
 name: "init",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": id`stack`,
 duration_ms: Date.now() - initStartTimestamp,
 ...event.preludeEvent.config,
 })
 hnyEvent.send()
 stackRunStartTimestamp = Date.now()
 return
 }

 if (event["summaryEvent"] !== undefined) {
 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(stackRunStartTimestamp)
 hnyEvent.add({
 name: "run",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": runStackSpanId,
 duration_ms: Date.now() - stackRunStartTimestamp,
 })
 hnyEvent.send()
 stackFinishStartTimestamp = Date.now()
 return
 }
 },
 })
}

main().then(() => {
 let hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(stackFinishStartTimestamp)
 hnyEvent.add({
 name: "finish",
 service_name: "Stack",
 "trace.trace_id": traceId,
 "trace.parent_id": rootSpanId,
 "trace.span_id": id`stack`,
 duration_ms: Date.now() - stackFinishStartTimestamp,
 })
 hnyEvent.send()

 hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(startTimestamp)
 hnyEvent.add({
 name: "up",
 service_name: "Pulumi",
 "trace.trace_id": traceId,
 "trace.span_id": rootSpanId,
 duration_ms: Date.now() - startTimestamp,
 })
 hnyEvent.send()
})

First, we need to sprinkle variables on the code that capture the start times of our three events.

The init event starts before the call to the up command and takes until the preludeEvent is fired by the automation API.

The run event starts right after the init event finishes and takes until the summaryEvent fires.

The finish event starts right after run finishes and stops right before the parent event for Pulumi up is finished.

These events all get the trace.parent_id of the Pulumi up event, so they get nested in the Honeycomb view later.

Also, the init event gets the event.preludeEvent.config object, so we can see how the stack was configured inside Honeycomb.

If we run that example, we should see something like this:

Getting Resource Deployment Durations

The resources have additional details for us. As a name for the event, we will use the operation (create, update, delete) and as service_name we use the name of the resource in the Pulumi definition.

To get the parent of a resource, we have to save the URNs of every resource for later. But since we have to keep the start times anyway, we can do this in one step when the resourcePreEvent is fired.

Let’s look at the example code. This time I removed some boilerplate code and the other events to make things a bit more readable.

const main = async () => {
 // ...

 const resourceSpans = new Map()

 // ...

 await stack.up({
 onEvent: (event) => {
 // ...

 if (event["resourcePreEvent"] !== undefined) {
 const { metadata } = event.resourcePreEvent
 resourceSpans.set(metadata.urn, {
 spanId: id`span`,
 startTimestamp: Date.now(),
 })
 return
 }

 if (event["resOutputsEvent"] !== undefined) {
 const { metadata } = event.resOutputsEvent

 const serviceName = metadata.urn.split("::")[3]
 const { spanId, startTimestamp } = resourceSpans.get(metadata.urn)
 const parentUrn = metadata.new?.parent ?? metadata.old?.parent
 const parentId = resourceSpans.get(parentUrn)?.spanId ?? runStackSpanId

 const hnyEvent = hny.newEvent()
 hnyEvent.timestamp = new Date(startTimestamp)
 hnyEvent.add({
 name: metadata.op,
 service_name: serviceName,
 "trace.trace_id": traceId,
 "trace.parent_id": parentId,
 "trace.span_id": spanId,
 duration_ms: Date.now() - startTimestamp,
 })

 if (metadata.op === "update")
 hnyEvent.addField("diffs", metadata.diffs.join(", "))

 hnyEvent.send()
 return
 }

 // ...
 },
 })
}

So, what’s going on here?

First, the resources are deployed by their hierarchy, but siblings can be deployed in a different order. That’s why we need to save their spanId and startTimestamp when we get a resourcePreEvent.

I used a Map for this and keyed it with the URN of the resource because URNs are used for parent-child relationships and are part of all events we care about here.

Later, when the resOutputsEvent fires, it’s time to wrap it all up and send it to Honeycomb.

The event has the URN to find the related spanId and startTimestamp. We can also use the URN in the parent attribute of our resource to fetch the spanId of our resource’s parent. This way, our spans are correctly nested in Honeycomb later.

If the operation for the resource is an update, we also add the diffs to that event. This way, we can see in Honeycomb what inputs have changed since the last deployment.

The final trace should look a bit like this:

Getting Errors

The last step, getting deployment errors, I leave as an exercise for the reader.

But I’ll give you a few hints!

First, the interesting event is the diagnosticEvent, which comes with a severity and message attribute. If you only need errors, you have to filter out debug and info severity.

Also, the event includes an urn that can be used to correlate the error to a Honeycomb span later (resourceSpans etc.). If the errors isn’t related to an URN, you can simply use the run stack event as target.

If you add an error field to a Honeycomb event, it will automatically counted to your errors in Honeycomb.

Summary

IaC observability becomes rather simple when using Pulumi and Honeycomb.

Pulumi’s automation API allows us to control the CLI with JavaScript, and Honeycomb’s Libhoney is low-level enough to instrument every part of our deployment process.

This way we can keep track which resources are slowing our deployment down, or which was responsible for a failed deployment.

This article was originally posted at https://dev.to/fllstck/observable-infrastructure-as-code-52ha. Reposted here with permission.

Mapbox IOT-as-code with Pulumi Crosswalk for AWS

Chris Toomey — Wed, 12 Jun 2019 00:00:00 +0000

Guest Author: Chris Toomey, Solution Architect Lead @ Mapbox

With 8 billion+ connected IoT devices and 2 billion GPS-equipped smartphones already online, logistics businesses are tracking assets at every step in the supply chain. At this scale and complexity, it is imperative to have a flexible way to ingest, process, and act upon this data, without sacrificing security or best practices.

To meet this need, Mapbox has created an Asset Tracking Solution that uses Pulumi’s open source JavaScript libraries (AWS, AWSX) available with multi-language support with Pulumi Crosswalk for AWS. Pulumi Crosswalk for AWS is an open source framework that streamlines creation, deployment and management of AWS services with built-in AWS Best Practices and minimal lines of code in common programming languages.

In this blog, we will show snippets of the Javascript code that embraces the power of Pulumi to program AWS service APIs to create the Mapbox solution. To see the full architecture in action with a live bike race across America, please refer to this webinar recorded on June 13th 2019 and the Mapbox asset tracking solution. Also refer to this blog of the Race across America showcased live during the webinar tomorrow.

Prerequisites: Install Pulumi; Install Node.js and Setup AWS

The diagram represents how Mapbox’s solution design on AWS services is built with Pulumi AWS and AWSX libraries in Javascript. Data is ingested in the Asset tracking IOT solution with a REST API. Mapbox uses its stream processor to perform enrichments such as geofencing, traffic-aware ETA calculations, and high-prevision elevation. This data, exposed by the API is backed by a high performance database (DynamoDB) to enable visualization in realtime on the map client.

The fun part – Pulumi Service console helps you map your cloud architecture shown above as a connected set of DAG resources so you need not remember what you deployed as your cloud environments scale:

Lets now work through the solution flow with some sample snippets of code. To get access to the complete piece of code, please connect with Mapbox solution team here and Pulumi team here. We will be happy to help you sample test this solution on AWS!

STEP 1: Create a Pulumi project using an AWS JavaScript template

$ brew install pulumi/tap/pulumi
$ mkdir asset-tracking && cd asset-tracking
$ pulumi new aws-javascript
$ yarn install
$ ls
Pulumi.dev.yaml Pulumi.yaml index.js node_modules package-lock.json package.json

STEP 2: Create the ingestion REST API with AWS Services

To visualize data the system has to accept large volumes of data from multiple sources into the rest of the system. Mapbox uses Pulumi’s AWS & AWSX libraries to build an IOT ingestion rule and forwards it to Kinesis streams. A sample index.js looks like this:

"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");
const awsx = require("@pulumi/awsx");

// Create Kinesis stream for ingestion
const ingestStream = new aws.kinesis.Stream("ingestAssets", {
 shardCount: 1,
 retentionPeriod: 72
});

// Create IoT Rule to push into Kinesis stream
const iotRole = new aws.iam.Role("iotRole", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: {
 Service: "iot.amazonaws.com"
 },
 Action: "sts:AssumeRole"
 }
 ]
 })
});

const iotRolePolicy = new aws.iam.RolePolicy("iotRolePolicy", {
 policy: pulumi.interpolate`{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "kinesis:*"
 ],
 "Resource": "${ingestStream.arn}"
 }
 ]
 }`,
 role: iotRole.id
});

const iotRule = new aws.iot.TopicRule("iotTrigger", {
 description: "Pass from IoT Core to Asset Tracking",
 name: "iotAssetIngest",
 enabled: true,
 kinesis: {
 partitionKey: "id",
 roleArn: iotRole.arn,
 streamName: ingestStream.name
 },
 sql: "SELECT * FROM 'topic'",
 sqlVersion: "2015-10-08"
});

Kinesis streams are used to ingest data into multiple AWS services defined as IOT rules, Firehose and Lambda functions. Each ingest stream has the right IAM role and policy to allow Kinesis to send the data into these AWS services.

The code above does not shown the Javascript code for Lambda functions used by Mapbox streamprocessor and Firehose all created with Pulumi AWS and AWSX libraries.

To get this piece of code, please connect with Mapbox solution team here and Pulumi team here.

STEP 3: Create a consumption API for the map client to consume from a data source

Once, data is flowing through the system we build another API to query DynamoDB, transform that data, and provide it to our mapping client in geojson. With Pulumi AWSX, we create a APIGateway endpoint API that can query this data out of DynamoDB. The sample code in index.js will look like this:

To get this piece of code, please connect with Mapbox solution team here and Pulumi team here.

// Create DynamoDB Table
const assetTable = new aws.dynamodb.Table("assetTable", {
 attributes: [
 {
 name: "id",
 type: "S"
 },
 {
 name: "ts",
 type: "N"
 }
 ],
 hashKey: "id",
 rangeKey: "ts",
 ttl: {
 attributeName: "expiration",
 enabled: true
 },
 billingMode: "PAY_PER_REQUEST"
 });

// Create API to read DynamoDB
const endpoint = new awsx.apigateway.API("assetQuery", {
 routes: [
 {
 path: "/",
 method: "GET",
 eventHandler: (request, ctx, cb) => {
 const AWS = require("aws-sdk");
 const ddb = new AWS.DynamoDB.DocumentClient({
 apiVersion: "2012-10-08"
 });
 const tableName = assetTable.name.value;
 const params = {
 TableName: tableName
 };
 ddb.scan(params, (err, data) => {
 const features = data.Items.map(item => {
 const point = turf.point([item.longitude, item.latitude], {
 id: item.id,
 speed: item.speed
 });
 return point;
 });
 const featureCollection = turf.featureCollection(features);
 cb(undefined, {
 statusCode: 200,
 body: Buffer.from(
 JSON.stringify(featureCollection),
 "utf8"
 ).toString("base64"),
 isBase64Encoded: true,
 headers: { "content-type": "application/json" }
 });
 });
 }
 }
 ],
 stageName: "dev"
});

Once we’ve created the API, we start defining our routes and then in-line we can define a Lambda function. It can use NPM modules and Pulumi will handle bundling all the dependencies.

This endpoint assumes we are querying Dynamo - and so we are going to use the DynamoDB Document Client. Using that straightforward scan syntax, we query the table and then start to parse our results. Since every data point has a latitude and longitude as GeoJSON, the data is transformed using Turf.js before sending to the client. Even more beautiful is how Pulumi handles all the heavy lifting of packaging, deploying, and giving you the endpoint to query immediately.

STEP 4: Hook up a map!

Now we have data coming and going from our Solution, we just have to hook it up to a Mapbox map. This is actually quite simple now that Pulumi has built us a REST API.

Following this example, we paste in the endpoint from Pulumi and pick our interval.

Additional Resources

For more information from Mapbox check out the Mapbox Blog. For more solutions from Pulumi please visit our Blog or head over to the Pulumi Examples repository. For AWS solutions visit the AWS partner blog site.

Get Started with Docker on AWS Fargate using Pulumi

Joe Duffy — Tue, 30 Apr 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Update: Check out the Learning Machine Case Study where provisioning went from 3 weeks to 1 hour with Pulumi and AWS.

“The impact of serverless capabilities was also transformative for the Learning Machine business. Pulumi enabled a rapid shift to Amazon ECS, AWS Fargate and AWS Lambda — the net effect of which was a 67% reduction in AWS charges. This enabled the team to spend less time focused on maintaining existing infrastructure and more time deploying new applications on AWS and adding new customers.

Pulumi is the foundational technology that allowed us to transform our organization,” said Hughes. The entire DevOps process was streamlined and in addition to realizing better productivity and higher quality, the team has new insight into their SaaS offering that they never thought possible.”

The Docker Getting Started tutorial shows how to develop, build, and run a modern containerized application, from a single custom Docker container published to the Docker Hub, to a scaled out service with load balancing. But there are challenges: it requires you to program in YAML, run (or script) many CLI commands, and manage your own Swarm or Kubernetes cluster. There is an easier way. By using Pulumi’s infrastructure as code, we can build a custom Docker image, publish it to a private AWS container registry, and spin up an AWS Fargate load balanced service running that container, all in 28 lines of TypeScript code and a single pulumi up command. The result leverages the best of what AWS has to offer, with the entire platform at your fingertips, with a single approach. In this article, we’ll see how.

Defining Everything in Code

The Docker Getting Started tutorial goes from a simple developer desktop scenario to a fully scaled out service in production. There is a lot of ceremony, however: YAML, many CLI commands, running your own Swarm cluster by hand, etc. This is complicated and doesn’t lend itself to repeatable deployments across many environments.

What if we could just write 28 lines of TypeScript instead, and accomplish the same end result, while leveraging the latest innovations in the AWS platform, natively?

In fact, we can. This article will show you how Pulumi’s infrastructure as code solution enables exactly this.

*Note: Everything we’re about to define will be in a single TypeScript file, index.ts. Other languages are available, including using plain Node.js JavaScript. Pulumi also supports other clouds, including Azure and GCP, however to fully exploit the power of the underlying AWS platform, these examples will leverage native capabilities like VPCs, ALBs, and Fargate.

To cut to the chase, here is the entire program. We’ll spend the rest of the post walking through it line by line:

import * as awsx from "@pulumi/awsx";

// Step 1: Create an ECS Fargate cluster.
const cluster = new awsx.ecs.Cluster("cluster");

// Step 2: Define the Networking for our service.
const alb = new awsx.lb.ApplicationLoadBalancer(
    "net-lb", { external: true, securityGroup: cluster.securityGroup });
const web = alb.createListener("web", { port: 80, external: true });

// Step 3: Build and publish a Docker image to a private ECR registry.
const img = awsx.ecs.Image.fromPath("app-img", "./app");

// Step 4: Create a Fargate service task that can scale out.
const appService = new awsx.ecs.FargateService("app-svc", {
    cluster,
    taskDefinitionArgs: {
        container: {
            image: img,
            cpu: 102 /*10% of 1024*/,
            memory: 50 /*MB*/,
            portMappings: [ web ],
        },
    },
    desiredCount: 5,
});

// Step 5: Export the Internet address for the service.
export const url = web.endpoint.hostname;

Step 1. Create a Cluster

The opening stanza imports Pulumi’s open source AWSX NPM package, @pulumi/awsx. It contains high level AWS best practices and patterns, and leverages real languages to eliminate boilerplate YAML templating:

import * as awsx from "@pulumi/awsx";

To provision an entire Elastic Container Service (ECS) cluster, AWS’s native container orchestration technology, we just new one up in a single line of code:

const cluster = new awsx.ecs.Cluster("cluster");

This will use the default Virtual Private Cloud (VPC) in our account. If we want to customize this aspect along with the auto-scaling policies, we can easily override the defaults by passing the appropriate arguments.

Step 2. Define the Networking

Next we need to think about the networking for our desired application. First, we will need a load balancer, as we’re going to create a scaled-out application and want a fixed address. Second, we will want to expose port 80 to the Internet, for incoming traffic. This leads to these two lines:

const alb = new awsx.lb.ApplicationLoadBalancer(
    "net-lb", { external: true, securityGroups: cluster.securityGroups });
const web = lb.createListener("web", { port: 80, external: true });

An Elastic Application Load Balancer (ALB) in AWS is a fully featured Layer 7 load balancer, with advanced features around SSL termination, content based routing, and HTTP/2. The AWSX package enables you to whip together simple routing when appropriate, while still having the ability to dig deeper into the advanced capabilities as you scale your workload to handle more sophisticated scenarios. Simple things simple, hard things possible!

Step 3. Build and Publish the Container Image

The next line declares a Docker image:

const img = awsx.ecs.Image.fromPath("app-img", "./app");

This single line is quite the work horse, and does a lot. Let’s unpack it a bit.

First and foremost, this allocates a private Elastic Container Registry (ECR) repository for your Docker image. This is AWS’s hosted Docker registry solution, privately encapsulated inside your AWS account.

But there is more: this also builds the Dockerfile underneath ./app and publishes the result to said registry, each time you run pulumi up! Any Dockerfile will work. This means you get a single command line to both provision and update your infrastructure, in addition to publishing and rolling out your images.

Step 4. Create a Scale-Out Service

Next up, let’s deploy our published image as a containerized service, behind our load balanced network on port 80, using our ECS Fargate cluster. Fargate is a managed container orchestrator that lets us skip the messy details of installing and managing Swarm on our own. This effectively replaces the docker-compose.yml from the Docker Getting Started tutorial, with a similarly simple sequence of code, and which gives us full access to the AWS platform:

const appService = new awsx.ecs.FargateService("app-svc", {
    cluster,
    taskDefinitionArgs: {
        container: {
            image: img,
            cpu: 128 /*~10%*/,
            memory: 50 /*MB*/,
            portMappings: [ web ],
        },
    },
    desiredCount: 5,
});

Notice that img just refers to the Docker image built in Step 3, and the portMappings structure references our load balancer’s listener defined in Step 2. Pulumi tracks all the dependencies so that it knows how to perform deployment activities in the right order. The task definition structure here exposes all of the underlying AWS ECS capabilities.

Step 5. Deploy the App and Kick the Tires!

On to the final step! Let’s deploy our infrastructure and application, and kick the tires.

The last line of code exposes the load balancer address so it’s easy to access:

export const url = web.endpoint.hostname;

The net result is that, with this project defined with the 28 lines of code above, we can simply run pulumi up; this command will show us a preview of the changes it will make and then, once confirmed, deploy everything to our AWS account, and hand back our service’s hostname:

After deploying everything, you’ve got a fully functioning application. To give it a try, let’s grab the URL exported by our program from the stack output command, and curl it:

$ curl http://$(pulumi stack output url)
<h3>Hello World!</h3>
<b>Hostname:</b> ip-172-31-39-18.ec2.internal<br/>
<b>Visits:</b> <i>cannot connect to Redis, counter disabled</i>

Or open it in our web browser:

$ open http://$(pulumi stack output url)

As an added bonus, the entire time we are doing these deployments, the Pulumi Service is keeping track of deployment state, to ensure everything works reliably and in a team setting, similar to how Git works with GitHub. For example, I can see a history of my deployments:

Congratulations!

In this article, you’ve seen how to take 28 lines of code, and turn them, plus a Dockerfile, into a fully functioning AWS microservice, including the infrastructure and application resources. No YAML required, no complex sequence of CLI commands, and no manual cluster installation.

Pulumi’s “everything is code” approach means deploying everything can be done with a CLI command, unlocking the power of the entire Docker platform with a great inner development loop, that works from the desktop all the way to production. This entire flow can be easily integrated into your favorite CI/CD pipeline, including GitOps workflows.

If we want to augment our service with other AWS resources – like S3 buckets, Aurora or RDS databases, Lambda functions, CloudWatch dashboards, etc – they can be provisioned and managed the same was created our clusters and load balancers: just new them up!

Remember that all of this not only works for native AWS ECS containers, but also Azure and GCP cloud providers, in addition to Kubernetes!

Want to go deeper?

Testing Your Infrastructure as Code with Pulumi

Joe Duffy — Wed, 17 Apr 2019 00:00:00 +0000

Some parts of this blog post are out-of-date. Please refer to our Testing Guide for the updated overview and tutorials.

Using Pulumi and general purpose languages for infrastructure as code comes with many benefits: leveraging existing skills and knowledge, eliminating boilerplate through abstraction, and using the same ecosystem of tools like IDEs and linters that your team already knows and loves. In general, these are all attributes of software engineering, which not only make us more productive, but also improve the quality of our code. It’s only natural, therefore, that using general purpose languages unlocks another important software engineering practice: testing.

In this article, we will see the many ways in which Pulumi lets us test our infrastructure as code.

Why Test Your Infrastructure?

Before jumping into the details, it’s worth asking: Why test your infrastructure, anyway? There are many reasons to do this, however here are a few examples:

Unit testing individual functions or pieces of logic in your program
Ensuring that the program’s desired infrastructure state meets certain constraints
Catching common mistakes, like forgetting to encrypt your storage buckets, or exposing virtual machines unprotected to the open Internet
Ensuring that the provisioned infrastructure has certain desired attributes
Doing runtime testing of the application logic that is running inside of your compute infrastructure, to ensure that it is healthy after your infrastructure has been provisioned

As we see, there’s a broad spectrum of infrastructure testing options available. Pulumi has mechanisms available for testing at each point along this spectrum. Let’s dig in and see how.

Unit Testing

Pulumi programs are authored in a general purpose language like JavaScript, Python, TypeScript, or Go. The full power of each language is available, including access to tools and libraries for that runtime, including testing frameworks. Pulumi is also multi-cloud, meaning the full underlying platform for each cloud is available too, and therefore available for testing.

(Despite being multi-language and multi-cloud, for the purposes of this article let’s assume we are using JavaScript, with Mocha for testing, and targeting AWS. Python unittest, Go’s test framework, or any of your favorite unit test frameworks in your language of choice would do, and of course Pulumi works great with Azure, Google Cloud, and Kubernetes, among others.)

As we’ve seen, there are several reasons you may want to test your infrastructure code. One of them is basic unit testing. Because your code may have functions – say, to compute CIDR blocks, dynamically compute names and tags, etc. – you probably want to test them. Doing this is the same as writing ordinary unit tests against any program in your language as choice.

Going one step further, you may want to test the way your program allocates resources. To illustrate, let’s imagine we want to create a simple EC2-based webserver, but want to ensure:

Instances have a Name tag.
Instances must not use an inline userData script – we must use an AMI (image).
Instances must not have SSH open to the Internet.

Our running example is loosely based on our aws-js-webserver example:

index.js:

let aws = require("@pulumi/aws");

let group = new aws.ec2.SecurityGroup("web-secgrp", {
    ingress: [
 { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
    ],
});

let userData =
`#!/bin/bash echo "Hello, World!" > index.html nohup python -m SimpleHTTPServer 80 &`;

let server = new aws.ec2.Instance("web-server-www", {
    instanceType: "t2.micro",
    securityGroups: [ group.name ], // reference the group object above
    ami: "ami-c55673a0" // AMI for us-east-2 (Ohio),
    userData: userData // start a simple web server
});

exports.group = group;
exports.server = server;
exports.publicIp = server.publicIp;
exports.publicHostName = server.publicDns;

This is a basic Pulumi program: it simply allocates an EC2 security group and instance. Notice, however, that we are violating all three of the rules stated above – let’s write some tests!

Writing the Tests

The overall structure and scaffolding of our tests will look like any ordinary Mocha testing:

ec2tests.js

let assert = require("assert");
let mocha = require("mocha");
let pulumi = require("@pulumi/pulumi");
let infra = require("./index");

describe("Infrastructure", function() {
    let server = infra.server;
    describe("#server", function() {
 // TODO(check 1): Instances have a Name tag.
 // TODO(check 2): Instances must not use an inline userData script.
    });
    let group = infra.group;
    describe("#group", function() {
 // TODO(check 3): Instances must not have SSH open to the Internet.
    });
});

Now let’s implement our first test: ensuring that instances have a Name tag. To verify this we simply need to grab hold of the EC2 instance object, and check the relevant tags property:

// check 1: Instances have a Name tag.
it("must have a name tag", function(done) {
 pulumi.all([server.urn, server.tags]).apply(([urn, tags]) => {
     if (!tags || !tags["Name"]) {
     done(new Error(`Missing a name tag on server ${urn}`));
     } else {
      done();
     }
    });
});

This looks like a normal test, with a few noteworthy pieces:

Since we’re querying resource state before we’ve done a deployment, our tests are always effectively running in “plan” (or “preview”) mode. Thus, there are many properties whose values will simply not resolve yet or will be undefined. This includes any output properties computed by your cloud provider. That’s fine for these tests – we’re checking for valid inputs anyway. We’ll revisit this later when it comes to integration testing.
Because all Pulumi resource properties are “outputs” – since many of them are computed asynchronously – we need to use the apply method to get access to the values. This is a lot like promises and the then function.
Because we are using multiple properties, mostly just so we can show the resource URN in the error message, we need to use the pulumi.all function to join all of them.
Finally, since these outputs are resolved asynchronously, we need to use Mocha’s built-in asynchronous test capability, with the done callback, or by returning a promise.

After we’ve gotten through that setup, we get access to the raw inputs as plain JavaScript values. The tags property is a map, so we just make sure it is (1) not falsey, and (2) not missing an entry for the Name key in the map. This is very basic, but we can check anything!

Now let’s write our second check. It’s even easier:

// check 2: Instances must not use an inline userData script.
 it("must not use userData (use an AMI instead)", function(done) {
 pulumi.all([server.urn, server.userData]).apply(([urn, userData]) => {
 if (userData) {
 done(new Error(`Illegal use of userData on server ${urn}`));
 } else {
 done();
 }
    });
});

And finally, let’s write our third check. It’s a bit more complex because we’re searching for ingress rules associated with a security group – of which there may be many – and CIDR blocks within those ingress rules – of which there may also be many. But it’s still easy:

// check 3: Instances must not have SSH open to the Internet.
it("must not open port 22 (SSH) to the Internet", function(done) {
 pulumi.all([ group.urn, group.ingress ]).apply(([ urn, ingress ]) => {
 if (ingress.find(rule =>
 rule.fromPort == 22 && rule.cidrBlocks.find(block => block === "0.0.0.0/0"))) {
     done(new Error(`Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0) on group ${urn}`));
 } else {
     done();
 }
    });
});

That’s it – now let’s actually run the tests!

Running the Tests

For the most part, we can run the tests in the usual way in your test framework of choice. There is one thing specific to Pulumi we need to cover first, however.

Usually you run your Pulumi programs with the pulumi CLI. That configures the language runtime, monitors execution from the engine so that resource operations can be recorded and turned into a plan, and so on. There is one problem, however: When running under the watchful eye of your test framework, this connection to the CLI, and Pulumi’s engine, will not exist.

To work around this problem, we simply have to mock three pieces of information:

Your project name, which can be supplied with the PULUMI_NODEJS_PROJECT environment variable (or more generally PULUMI_<LANGUAGE>_PROJECT in other languages).
Your stack name, which can be supplied with the PULUMI_NODEJS_STACK environment variable (or more generally PULUMI_<LANGUAGE>_STACK in other languages).
Your stack configuration variables. These can be supplied using the PULUMI_CONFIG environment variable, and its format is a JSON map of configuration key/value pairs.

The program will emit some warning messages indicating the connection to the CLI/engine is not available at runtime. This is important to know, because your program won’t actually be deploying anything – which can come as a surprise if that’s not what you meant to do! To tell Pulumi that’s what you intended, you can set PULUMI_TEST_MODE to true.

Imagine we want a project name of my-ws, stack name of dev, and AWS region of us-west-2. The command line to run your Mocha tests would therefore be:

$ PULUMI_TEST_MODE=true \
 PULUMI_NODEJS_STACK="my-ws" \
 PULUMI_NODEJS_PROJECT="dev" \
 PULUMI_CONFIG='{ "aws:region": "us-west-2" }' \
 mocha tests.js

Running this will tell us that we have three failing tests, as we had hoped!

 Infrastructure
    #server
1) must have a name tag      2) must not use userData (use an AMI instead)
    #group
3) must not open port 22 (SSH) to the Internet
  0 passing (17ms)
  3 failing
 1) Infrastructure
       #server
 must have a name tag:
Error: Missing a name tag on server
urn:pulumi:my-ws::my-dev::aws:ec2/instance:Instance::web-server-www
 2) Infrastructure
       #server
 must not use userData (use an AMI instead):
Error: Illegal use of userData on server
urn:pulumi:my-ws::my-dev::aws:ec2/instance:Instance::web-server-www
 3) Infrastructure
       #group
 must not open port 22 (SSH) to the Internet:
Error: Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0) on group

Let’s fix our program to comply:

"use strict";

let aws = require("@pulumi/aws");

let group = new aws.ec2.SecurityGroup("web-secgrp", {
    ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
    ],
});

let server = new aws.ec2.Instance("web-server-www", {
    tags: { "Name": "web-server-www" },
    instanceType: "t2.micro",
    securityGroups: [ group.name ], // reference the group object above
    ami: "ami-c55673a0" // AMI for us-east-2 (Ohio),
});

exports.group = group;
exports.server = server;
exports.publicIp = server.publicIp;
exports.publicHostName = server.publicDns;

And then rerun our tests:

 Infrastructure
    #server
      ✓ must have a name tag
      ✓ must not use userData (use an AMI instead)
    #group
      ✓ must not open port 22 (SSH) to the Internet
 3 passing (16ms)

They all passed – huzzah! ✓✓✓

Deployment Testing

This style of testing is powerful and lets us do white box testing of the innards of how our infrastructure code works. This style of testing is somewhat limited in what we can check, however. The tests run on an in-memory representation of Pulumi’s deployment plan – before actually performing the deployment – and so cannot test anything the deployment itself. For that, Pulumi has an integration test framework. The two approaches work great together!

Pulumi’s integration test framework is written in Go and is how we test much of our internal code. If the earlier unit test approach took more of a white box testing approach, the integration test framework is very much black box. (There are also options for thorough internals-oriented testing.) This framework has been built to take a directory containing a full Pulumi program and drive various lifecycle operations against it – such as deploying a new stack from scratch, updating it with variations, and tearing it down afterwards, potentially multiple times. We run these in checkin suites, on a recurring basis (such as nightly), and as stress tests.

(We have an open work item which will bring similar integration testing capabilities to each native language SDK. You can use the Go integration test framework no matter the language your Pulumi program is written in, however.)

By running a program through this integration test framework, you can ensure:

Your project’s code is syntactically well formed and runs without errors.
Your stack’s configuration and secrets settings work and are interpreted correctly.
Your project can be successfully deployed to your cloud provider of choice.
Your project can be successfully updated from its starting state to N other states.
Your project can be successfully destroyed and removed from your cloud provider.

As we will see soon, you can also leverage this framework to perform runtime validation.

A Basic Integration Test

To see this in action, we’ll look from the pulumi/examples repo, since our team and community here at Pulumi use that to validate our own examples in pull requests, checkins, and nightlies.

The following is a simplified test of our example that provisions an S3 bucket and some objects in Pulumi:

example_test.go:

package test

import (
    "os"
    "path"
    "testing"

    "github.com/pulumi/pulumi/pkg/testing/integration"
)

func TestExamples(t *testing.T) {
    awsRegion := os.Getenv("AWS_REGION")
    if awsRegion == "" {
 awsRegion = "us-west-1"
    }
    cwd, _ := os.Getwd()
    integration.ProgramTest(t, &integration.ProgramTestOptions{
 Quick: true,
 SkipRefresh: true,
 Dir: path.Join(cwd, "..", "..", "aws-js-s3-folder"),
 Config: map[string]string{
     "aws:region": awsRegion,
 },
    })
}

This test runs through a basic lifecycle of stack creation, updating, and destruction, for the aws-js-s3-folder example. It takes about a minute or so to report success:

$ go test .
PASS
ok ... 43.993s

There are many, many options to control the behavior of these tests. For a full set of options, see the ProgramTestOptions data structure. For instance, you can configure a Jaeger endpoint for tracing (Tracing), tell the harness to expect failure for negative testing (ExpectFailure), apply a series of “edits” to the program for a sequence of update state transitions (EditDirs), and more. Let’s see how to use these to do validate that we deployed what we expected.

Validating the Shape of Resources

All of the above integration ensures that our program “works” – as in, it does not crash. But what if we want to validate properties of the resulting stack? For example, that certain kinds of resources did (or did not) get provisioned, and that they have certain attributes.

The ExtraRuntimeValidation option for ProgramTestOptions allows us to look at the post-deployment state that Pulumi has captured, so that we can do extra validation on it. This includes a full snapshot of the resulting stack’s state, including configuration, exported outputs, all resources and their property values, and all inter-resource dependencies.

To see a basic example of this, let’s check that our program creates a single S3 Bucket:

integration.ProgramTest(t, &integration.ProgramTestOptions{
 // as before...
 ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
     var foundBuckets int
     for _, res := range stack.Deployment.Resources {
 if res.Type == "aws:s3/bucket:Bucket" {
      foundBuckets++
 }
     }
     assert.Equal(t, 1, foundBuckets, "Expected to find a single AWS S3 Bucket")
 },
})

Now when we run go test, it will not only run through the battery of lifecycle tests but will also, after successfully standing up the stack, perform the extra validation against the resulting state.

Runtime Testing

All of the testing thus far has been exclusively about deployment behavior and the Pulumi resource model. What if you want to test that your provisioned infrastructure is actually working, however? For example, that our VM is alive, our S3 bucket contains what we expect, and so on.

You may have already guessed how to accomplish some of this: the ExtraRuntimeValidation option for ProgramTestOptions is the perfect opportunity to do this. At this point, you are running an arbitrary Go test, with access to the full state of your program’s resources. This state includes information such as VM IP addresses, URLs, and everything you need to actually interact with your resulting cloud applications and infrastructure to perform verification.

For instance, our test program above exports the bucket’s websiteEndpoint property, as the stack output named websiteUrl, which is a full URL that we can use to fetch the bucket’s configured index document. Although we could just as well have dug through the state file to find the bucket and read this property directly, many times our stacks will export useful properties like this that we can use conveniently for verification:

    integration.ProgramTest(t, &integration.ProgramTestOptions{
     // as before ...
 ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
     url := "http://" + stack.Outputs["websiteUrl"].(string)
     resp, err := http.Get(url)
     if !assert.NoError(t, err) {
 return
     }
     if !assert.Equal(t, 200, resp.StatusCode) {
 return
     }
     defer resp.Body.Close()
     body, err := ioutil.ReadAll(resp.Body)
     if !assert.NoError(t, err) {
 return
     }
     assert.Contains(t, string(body), "Hello, Pulumi!")
 },
})

Like our previous runtime validation checks, this will run in the harness right after the stack is stood up, all in response to a simple go test invocation. This just scratches the surface of what’s possible – anything you can write in code using Go’s test capabilities is within reach!

Continuous Infrastructure Integration

It’s wonderful to be able to easily run these on demand, on a laptop, when you’re iterating on infrastructure changes and want to verify them before submitting for code review. But we and many of our customers run infrastructure testing at various points in the development lifecycle:

In each pull request opened, to validate before allowing it to be merged.
In response to each commit, to double check it was merged correctly.
On a recurring basis, such as nighty or even weekly for extended testing.
As part of more exotic and thorough test programs, such as performance or stress testing, which typically run over a longer period of time, execute many things in parallel, and/or deploy the same program multiple times.

For each of these, Pulumi supports integration with your favorite continuous integration system. This gives you similar continuous integration testing coverage for your infrastructure as you might already enjoy for your application software or in other programming domains.

Pulumi supports your existing CI systems. Here are a few of those supported:

Please refer to the Continuous Delivery documentation for a more comprehensive guide.

Ephemeral Environments

A very powerful capability this unlocks is the ability to spin up ephemeral environments solely for purposes of acceptance testing. Pulumi’s concept of projects and stacks is designed to make it very easy to stand up entirely isolated and independent environments, and to tear them down, all in either a few easy CLI gestures, or by using the integration testing framework.

If you are using GitHub, Pulumi offers a GitHub App that helps to glue together your Pull Request workflow with this sort of acceptance testing run inside of your CI pipelines. Simply install the App into your GitHub repos, and Pulumi in your CI, and your Pull Requests will light up with infrastructure previews, updates, and test results seamlessly:

By leveraging Pulumi for your core acceptance test workflow, you’ll unlock new automation capabilities that improve your team’s productivity and confidence in the quality of changes.

In Summary

In this post, we saw that by using general purpose programming languages, we unlock many software engineering practices we’ve benefited from in our application development. This includes unit testing, integration testing, and even the intersection of both to do extensive runtime testing. It’s easy to run these on demand and harness them in your CI system of choice.

Pulumi is open source, free to use, and works with your favorite languages and clouds – give it a try today!

Easy Serverless Apps and Infrastructure

Joe Duffy — Mon, 11 Mar 2019 00:00:00 +0000

With Pulumi, you can create, deploy, and manage any cloud resource using your favorite language. This includes application and infrastructure related resources, often in the same program.

One area this gets really fun is serverless computing. Because we’re using general purpose languages, we can create resources, and then wire up event handlers, just like normal event-driven programming. This is the way serverless architecture should be!

In this article, we’ll see how. There’s a broad range of options depending on what you want to do, and how your team likes to operate. We’ll be using Amazon Web Services (AWS) and TypeScript, but other clouds and languages are available.

A Simple Serverless App

In this article, we’ll use a very simple serverless app. We have two S3 buckets: one for TPS reports, and the other to store archived zipfiles of those reports. Anytime a new report comes in, our Lambda will zip and archive it.

Simple enough. This can be visualized as follows:

All source code for this example is available on GitHub.

Event Sources and Event Handlers

Serverless app models today make you think of the event sources – the S3 buckets – and event handlers – the Lambdas and associated code – as very different things – “infrastructure” versus “app code” – managed with distinct tools and workflows. Pulumi, in contrast, gives you a single CLI, pulumi, to manage everything consistently.

You can create event sources and handlers in the same program, but Pulumi’s approach is flexible, allowing you to ease into serverless without boiling the ocean. You can choose to wire functions up to existing resources (say, if your team already created the buckets), or even use functions your team has already made.

Also, because Pulumi provisions and manages resources, updating your functions after creating them is easy. Just edit your code, run pulumi up, and Pulumi will diff and compute the minimal set of changes it can make to upgrade your code – no downtime required. We’ll see this in action shortly. But first let’s create our initial app.

Creating an Event Handler

There are multiple approaches to creating a function. Let’s start with the easiest.

Approach 1: Magic Functions

Our favorite way to create functions is to write them inline. This blurs the line between infrastructure and application logic in a way that makes Pulumi programs feel like real, first class distributed apps.

First, we have our TPS reports and zips buckets (for now we’ll new them up, more on that later):

import * as aws from "@pulumi/aws";
const tpsReports = new aws.s3.Bucket("tpsReports");
const tpsZips = new aws.s3.Bucket("tpsZips");

The bucket objects have many properties, including obvious ones like the ARN, domain name, CORS rules, and so on. But if we look closely, we’ll see there are some onX methods. These register event handlers:

Let’s use onObjectCreated to create a Lambda that will zip up any new reports:

tpsReports.onObjectCreated("zipTpsReports", (e) => {
   const AdmZip = require("adm-zip");
   const s3 = new aws.sdk.S3();
   for (const rec of e.Records || []) {
      const zip = new AdmZip();
      const [ buck, key ] = [ rec.s3.bucket.name, rec.s3.object.key ];
      console.log(`Zipping ${buck}/${key} into ${tpsZips.bucket.get()}/${key}.zip`);
      const data = await s3.getObject({ Bucket: buck, Key: key }).promise);
      zip.addFile(key, data.Body);
      await s3.putObject({
         Bucket: tpsZips.bucket.get(),
         Key: `${key}.zip`,
         Body: zip.toBuffer(),
      }).promise();
   }
});

Most of the code here is the application logic – precisely what we want to be focusing on! In fact, this code looks just like any ordinary JavaScript event-driven program.

A neat thing that Pulumi can do is capture references to other resources. So, notice here, we actually capture a reference to tpsZips from within the event handler, and Pulumi figures out how to serialize the resulting closure. For the most part, any code you write there will work, unlocking all sorts of powerful composition scenarios: multiple functions can be composed together, you can use modules as usual, and so on. Just program like you normally would, without the awkward environment variables and configuration. That’s why we call these magic functions.

Now, let’s deploy the resources to AWS:

After confirming, yes, everything is up and running after just a few seconds:

Note the resource graph. Pulumi uses familiar languages, which allows encapsulation and hiding of unnecessary complexity. So underneath that simple-looking onObjectCreated are many AWS resources, like IAM attachments. This is a lot like how we can ignore all the messy OS API calls when using Node.js abstractions.

Pulumi has many convenience functions built-in, and one of them is pulumi logs, which will let us tail the logs associated with all compute in our stack (including serverless functions):

$ pulumi logs -f Collecting logs for stack dev since 2019-03-10T10:09:56.000-07:00…

Now that we’re tailing the logs, let’s copy a file over using the AWS CLI:

$ aws s3 cp ./tps001.txt s3://$(pulumi stack output tpsReportsBucket) upload: ../tps001.txt to s3://tpsreports-96458ef/tps001.txt

We have used the pulumi stack output command to conveniently fetch the bucket name. After the upload completes, we’ll see that our function comes alive in the pulumi logs command:

2019-03-10T11:10:48.617-07:00[zipTpsReports] Zipping tpsreports-96458ef/tps001.txt into tpszips-edfde11/tps001.txt.zip

And, sure enough, if we list the contents of the zip bucket, we will see our newly added file:

$ aws s3 ls s3://$(pulumi stack output tpsZipsBucket) 2019-03-10 11:10:50 6206 tps001.txt.zip

Voila! A fully functioning serverless application.

Note also that the CallbackFunction class – the powerful abstraction behind all of this – is exported, and offers some knobs, in case you want to do things like reuse existing IAM roles rather than creating new ones. See the pulumi/aws/lambda documentation for details. For instance, say we want to increase the RAM available to our function from 128MB to 256MB:

tpsReports.onObjectCreated(
   "zipTpsReports",
   new aws.lambda.CallbackFunction("zipTpsReportsFunc", {
      memorySize: 256 /*MB*/,
      callback: (e) => { /* same code as before */ },
   },
);

Soon we’ll see how to manage and update our functions, but first let’s see some alternative coding styles.

Approach 2: Manual Function Resources

We just saw that we can write our serverless application logic inside of our Pulumi program, alongside our infrastructure resource definitions. This can feel magical and super productive. But not everybody wants to mix these together. Sometimes you want to stick to the classic way of doing this: keeping your function’s application logic in one place, and your function’s infrastructure definition in another. This aligns more closely with many team workflows.

There is another practical reason to want this: if your functions are written in one language (say, C# or Java) while your Pulumi infrastructure is authored in a different one (say, JavaScript or Python).

Pulumi supports this, by simply defining your Lambda function like any other infrastructure resource:

// First, create some IAM machinery:
const zipFuncRole = new aws.iam.Role("zipTpsReportsFuncRole", {
   assumeRolePolicy: {
      Version: "2012-10-17",
      Statement: [{
         Action: "sts:AssumeRole",
         Principal: {
            Service: "lambda.amazonaws.com",
         },
         Effect: "Allow",
         Sid: "",
      }],
   },
});
new aws.iam.RolePolicyAttachment("zipTpsReportsFuncRoleAttach", {
   role: zipFuncRole,
   policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

// Next, create the Lambda function itself:
const zipFunc = new aws.lambda.Function("zipTpsReportsFunc", {
   environment: {
      variables: {
         "TPS_ZIP_BUCKET": tpsZips.bucket,
      },
   },
   code: new pulumi.asset.AssetArchive({
      ".": new pulumi.asset.FileArchive("./app"),
   }),
   runtime: "nodejs12.x",
   role: zipFuncRole.arn,
});

// Finally, register the Lambda to fire when a new TPS report arrives:
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

Because we’re taking matters into our own hands, we have to create the IAM machinery ourselves. We also need a way to communicate the zip bucket name, so we use an environment variable. This highlights both the benefits and drawbacks to programming at this level – we need to know how to configure all of these ancillary resources, but as a result, the entire power of Lambda is at our fingertips.

Notice that we’ve pointed to our application logic inside of ./app. Pulumi will create the zipfile for you. If we instead wanted to use a zipfile we’ve already packaged, just change code as follows:

// ...
   code: new pulumi.asset.FileArchive("./app.zip"),
// ...

Using Pulumi’s Asset and Archive classes, we can fetch code from anywhere – even the network.

Although managing your functions manually isn’t quite as magical, it is practically useful, and still delivers all the usual infrastructure as code benefits, enabling robust infrastructure management and versioning.

Approach 3: Wire Up an Existing Function

This post is mainly about using serverless functions with Pulumi. But what if you’re using Pulumi mainly for infrastructure, and another solution for the functions themselves? No problem – Pulumi can still glue everything together, even if it’s not provisioning all of the resources.

Every resource type in Pulumi has a static get that looks up an existing resource. These resources aren’t managed by Pulumi, but you can still access their properties and use them. In this case, we’ll just look up an existing Lambda, zipTpsReportsFunc, and register it as an event handler:

const zipFunc = aws.lambda.Function.get("zipTpsReportsFunc", "zipTpsReports-19d51dc");
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

We’ve given the function’s ID, zipTpsReports-19d51dc, which allows Pulumi to locate it in your account and reuse it. This can make it easy to incrementally adopt Pulumi one piece at a time, collaborate between teams, or stitch together resources managed by different stacks.

More About Functions

Now that we’ve seen the basics, let’s look at a few associated function management topics.

Updating a Function

Updating the function is as easy as can be. Imagine we’re running into a problem and want to do some good old fashioned printf debugging inside of our lambda. Maybe we want to log the size of the report and its resulting zipfile. Just add the relevant line:

// ...
zip.addFile(key, data.Body);
console.log(`Report: ${data.ContentLength}, Zip: ${zip.toBuffer().length}`);
s3.putObject(
// ...

And run pulumi up – it will show us what changed:

We can view the full diff, by choosing details, including a Git-like diff of the code changes themselves!

After selecting yes, the function is updated in just a couple of seconds. Similarly, we can update any of the resource definitions, and Pulumi will figure out how to incrementally update them.

Splitting Up your Codebase

A nice middle ground between magic and manual functions is to use your language’s module system to structure code differently. This is similar to how you might structure a typical application: route definitions over here, business logic over there, markup over here, etc. Pulumi can figure out the diffs regardless of how you’ve structured your code, so updates are always based only on what’s changed.

For example, maybe we’ve defined our callback function in ./app:

export async function zipReport(e: aws.s3.BucketEvent): Promise<void> {
   // app code, as shown above, goes here
}

And now we can go back to our infrastructure code, and eliminate the application logic entirely:

import { zipReport } from "./app";
...
tpsReports.onObjectCreated("zipTpsReports", zipReport);

We can take this further and use dynamic package management to split up the code, possibly even spreading pieces of infrastructure and application code across multiple repos and/or packages. This works well for larger teams with independent components versioning at their own pace.

Lastly, it’s possible to use Pulumi stacks to actually break apart your cloud resources and functions into independently deployable pieces. This allows teams to leverage features like RBAC. For instance, it’s common for the DevOps team to manage the physical cloud resources like queues, topics, and buckets, while the development team authors and manages the serverless functions attached to them. Read more about this in the organizaing projects & stacks documentation.

More About Event Sources

We glossed over one of the more interesting points: what resources have event handlers associated with them, and how to get our hands on them.

Provision a New Resource

The simplest answer here is to create a new resource in your Pulumi program using new, as we saw above. Because Pulumi is an infrastructure as code platform, any resources in any cloud are available – AWS, Azure, GCP, Kubernetes, etc. When you new one up, Pulumi understands how to provision and manage it.

We saw simple examples of this earlier:

const tpsReports = new aws.s3.Bucket("tpsReports");

Note that the full set of configuration options are available on each resource. Let’s say we want to turn on server-side encryption of our TPS archive bucket:

const tpsReports = new aws.s3.Bucket("tpsReports", {
   serverSideEncryptionConfiguration: {
      sseAlgorithm: "AES256",
   },
});

Just as with functions, subsequent updates will be diffed and updated in the minimally impactful way.

Connect to an Existing Resource

Although it’s nice we can provision the event sources and targets in a single program, sometimes different members of the team manage different parts of the infrastructure. For example, maybe your DevOps engineers provision resources like buckets, topics, and so on, and your developers wire up the functions.

Not to worry, you can import an existing resource and program against it in the same way:

const tpsReports = aws.s3.Bucket.get("tpsReports", "arn:aws:s3:::tpsReports-4f64efc");

This time, when you run your Pulumi program, you’ll still see this resource but notice it says read instead of create or update. This just means that the resource is read from your account as-is. No matter what you do, the bucket itself will not be modified by Pulumi, other than to subscribe an event.

This, of course, can be combined with the earlier similar functionality for functions, to glue together a bucket and a Lambda, where neither was actually provisioned by Pulumi!

const tpsReports = aws.s3.Bucket.get("tpsReports", "arn:aws:s3:::tpsReports-4f64efc");
const zipFunc = aws.lambda.Function.get("zipTpsReportsFunc", "zipTpsReports-19d51dc");
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

Other Notable Event Sources

Finally, it’s worth noting there are many event handler functions exposed in the AWS package:

apigateway.x.API: create serverless APIs using an Express.js style
cloudwatch.onSchedule: fire a CloudWatch event on a particular schedule, e.g. a cron expression
cloudwatch.Event.onEvent: fire an event when a particular CloudWatch event occurs
cloudwatch.LogGroup.onEvent: fire an event when a CloudWatch logs event occurs
dynamodb.Table.onEvent: fire events for DynamoDB insert, modify, or remove operations
kinesis.Stream.onEvent: fire Kinesis Stream events at particular times or batch sizes
s3.Bucket.onObjectCreated: trigger a function anytime an object is created in an S3 Bucket
s3.Bucket.onObjectRemoved: trigger a function anytime an object is removed from an S3 Bucket
s3.Bucket.onEvent: trigger a function for a wide range of S3 Bucket events
sns.Topic.onEvent: fire SNS Topic events when new messages arrive
sqs.Queue.onEvent: fire SQS Queue events when new messages are enqueued (or on DLQ events, etc)

All of these handlers can be programmed using the spectrum of techniques outlined above.

In Conclusion

In this post, we saw many different options for serverless programming with Pulumi, from magic functions that make serverless apps feel like true event-driven programming, to provisioning and managing infrastructure as code.

We also saw that all of these options can be combined in any way, depending on you and your team’s needs. This is the magic of programming languages in action!

If you’d like to try it out, check out the code for this post on GitHub. Happy serverless hacking!

Serverless on AWS with Pulumi: Simple, Event-based Functions

Cyrus Najmabadi — Mon, 14 Jan 2019 00:00:00 +0000

One of Pulumi’s goals is to provide the simplest way possible to do serverless programming on AWS by enabling you to create cloud infrastructure with familiar programming languages that you are already using today. We believe that the existing constructs already present in these languages, like flow control, inheritance, composition, and so on, provide the right abstractions to effectively build up infrastructure in a simple and familiar way.

In a previous post we focused on how Pulumi could allow you to simply create an AWS Lambda out of your own JavaScript function. While this was much easier than having to manually create a Lambda Deployment Package yourself, it could still be overly complex to integrate these Lambdas into complete serverless application.

To get a sense of that complexity, let’s look at how one would normally have to work with AWS’s resource system to create a simple Serverless application:

// Simplified for brevity
import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Create a lambda that will post a message to slack when the bucket changes.
// We can pass a simple JavaScript/TypeScript lambda here thanks to the magic of "Lambdas as Lambdas"
// See: www.pulumi.com/blog/lambdas-as-lambdas-the-magic-of-simple-serverless-functions/
const lambda = new aws.lambda.CallbackFunction("postToSlack", {
 callback: async (e) => {
 const client = new slack.WebClient(...);
 for (const rec of e.Records) {
 await client.chat.postMessage({ ... });
 }
 },
 ...
});

// Give the bucket permission to invoke the lambda.
const permission = new aws.lambda.Permission("invokelambda", {
 function: lambda,
 action: "lambda:InvokeFunction",
 principal: "s3.amazonaws.com", sourceArn: bucket.id.apply(bucketName => `arn:aws:s3:::${bucketName}`), }));

// Now hookup a notification that will trigger the lambda when any object is created in the bucket.
const notification = new aws.s3.BucketNotification("onAnyObjectCreated", {
 bucket: bucket.id,
 lambdaFunctions: [{
 events: ["s3:ObjectCreated:*"],
 lambdaFunctionArn: lambda.arn,
 }],
})

Phew… that’s a lot of code :-/ So what happened above? Well, in the AWS resource-oriented view of the world, most things are nouns (i.e objects). Your bucket is an object. The lambda is an object. The permission is an explicit object. And even the connection between the bucket and lambda (the BucketNotification) is itself an object. That’s one way to decompose the world, and it makes sense in some domains where everything is just ‘data’. But that isn’t a natural fit for a programming language, and it just doesn’t feel very idiomatic or appropriate for how people would expect things to work in their programming language of choice. Programming languages have lots of other facilities at their disposal to work with data. You can have methods/functions/messages you can send to an object. You can have events and callbacks on objects. You can encapsulate objects behind other objects. You can ‘run code’ to do things based on those objects, etc. etc.

Now, while data should be the start of how you interact with system, it shouldn’t be the end. Toward that goal, Pulumi makes it possible to simply compose and connect these resources in a more natural fashion. By just adding our own components, and leveraging what programming languages can already do, we can make it possible to simplify the above down to:

import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Create a lambda that will post a message to slack when the bucket changes.
bucket.onObjectCreated("postToSlack", async (e) => {
 const client = new slack.WebClient(...);
 for (const rec of e.Records) {
 await client.chat.postMessage({ ... });
 }
});

This now feels far more like how one might expect to express this concept with a normal application. A simple conceptual idea now maps to a simple code pattern. Instead of manually creating objects to represent the connection between the Bucket and the Lambda, we simply directly ask the Bucket to call the Lambda when the ObjectCreated event fires. Because this is all ‘code’, we can take care of all the boring cruft (like creating permissions) on your behalf. Of course, if you need to tweak this, that’s still possible. We believe you should always be in control of what’s going on. The actual low level details of how that is done are interesting as well, but a followup post will explain how we did it (and how you can do it too!). If you want to create your own composable modules for making cloud deployments easier then definitely look for that post!

Now, while it’s how the above examples worked, it’s also worth noting that the use of a JavaScript function for the AWS Lambda is not required. You can hook up a serverless event to call an AWS Lambda you create just by using new aws.lambda.Function. Or, you can get a reference to an existing AWS Lambda created outside of Pulumi and have that be the receiver of your serverless event. Here’s how that would look:

import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Retrieve an existing Lambda Function already created in your AWS infrastructure.
const func = aws.lambda.Function.get("postToSlack", "... existing lambda arn ...");

// Call that Lambda Function when an object is created in the bucket.
bucket.onObjectCreated("postToSlack", func);

We’ve tried to make it this simple to hook up many interesting AWS serverless events. For example, you can register to hear about events on S3 Buckets, SNS Topics, SQS Queues, Kinesis Streams, DynamoDB Tables, Cloudwatch Events, and more. We’ve tried to provide these prebuilt components for the most common and interesting cases. If there’s some serverless event we haven’t added support for that you want to use, definitely let us know!

Finally, because we think it’s great to be able to use a JavaScript or TypeScript function as the code for your Lambda, we’ve introduced a simple quality-of-life improvement for writing those functions. Specifically, because it’s so common that one will use the real AWS SDK for Node.js within a Lambda, we’ve provided a quick and simple way to get to a fully initialized instance of that API without needing any additional requires or imports in your code. Here’s how that would look:

// The only module you need to import.
import * as aws from "@pulumi/aws";

// ...

bucket.onObjectCreated("postToSlack", async (e) => {
 // direct access to the aws-sdk through `aws.sdk`.
 const sqs = new aws.sdk.SQS();
 sqs.sendMessage(/*...*/);
});

This allows you to use @pulumi/aws as both the deployment-time API to define your AWS infrastructure, as well as being the run-time API that you can use when your Lambda executes to access the full set of AWS functionality.

To get a sense of a more complete example of how this all ties together you can see many of our examples. One example that helps demonstrate many of the above concept is Twitter+Athena. In it, we use serverless events to define a schedule, a callback to invoke when the even t fires, use of a third-party twitter API to retrieve a set of relevant Tweets, use of the aws-sdk to store the Tweet information into an S3 Bucket, and finally the creation of an Athena Query to extract information from the information stored into the bucket. You could augment this example with any number of resources and events for all sorts of interesting use cases.

As we’ve seen, with a tiny amount of code, you can easily define the cloud resources you want to create (or reference your existing cloud resources). It’s then simple to add code to respond to events and call new or existing Lambdas. And, if you use a JavaScript function to create your Lambda, you can reference those resources along with the AWS SDK easily, all within the same application code!

From Terraform to Infrastructure as Software

Pat Gavlin — Fri, 02 Nov 2018 00:00:00 +0000

Here at Pulumi, we love programming the cloud using infrastructure as code. From the project’s outset, we’ve been inspired by technologies like Terraform, AWS CloudFormation, and Helm, and in fact leverage the Terraform Providers ecosystem, to support a broad range of clouds, including AWS, Azure, and Google Cloud.

Just recently, we extended this with first class support for Kubernetes. Pulumi delivers the same infrastructure as code workflows only using general purpose languages like JavaScript, TypeScript, Python, and Go, extending robust infrastructure provisioning with abstraction and reuse, highly productive tooling, and access to all the other things we already know and love about programming languages.

In this article, we will convert existing Terraform configuration to Pulumi TypeScript. By doing so, we’ll see how using general purpose programming languages can help you create simpler, more flexible infrastructure as code, with greater productivity and less repetition. The infrastructure we’ll be working with describes a load-balanced web server hosted by an AWS EC2 instance per availability zone with an option to allow SSH access. Of course, these same benefits would also accrue were we to target Azure, Google Cloud, or Kubernetes instead.

Starting from Terraform Configuration

The Terraform configuration we’ll be starting with is below. This configuration accepts a list of AZs and an optional public key as inputs and produces a (possibly empty) list of the web servers’ public IP addresses and the URL of the load-balanced web site as outputs. When a public key is provided, this configuration will enable SSH access to the virtual machines that serve the web site.

Let’s walk the example one step at a time – feel free to skip ahead if you’re a Terraform expert and just want to see how Pulumi is different.

First, we declare that availability_zones and public_key are parameterizable inputs:

variable "availability_zones" {}
variable "public_key" {
default = ""
}

And then we look up the latest AWS AMI for Ubuntu Trusty in our current region. This is what our web server will run when we get around to provisioning it later:

data "aws_ami" "ubuntu" {
 most_recent = true
 filter {
   name   = "name"
   values = ["ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"]
 }
 filter {
   name   = "virtualization-type"
   values = ["hvm"]
 }
 owners = ["099720109477"] # Canonical
}

Next, we’ll create security groups that allow HTTP ingress on port 80 and, if a public key is available, SSH over port 22, in addition to allowing outbound Internet access. Notice that we are using the count property for the SSH rule to enable it conditionally:

resource "aws_security_group" "default" {
name_prefix = "example_sg"
}
resource "aws_security_group_rule" "allow_http" {
type = "ingress"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
security_group_id = "${aws_security_group.default.id}"
}
resource "aws_security_group_rule" "allow_all_outbound" {
type = "egress"
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
security_group_id = "${aws_security_group.default.id}"
}
resource "aws_security_group_rule" "allow_ssh" {
 count = "${var.public_key == "" ? 0 : 1}"
 type = "ingress"
 from_port = 22
 to_port = 22
 protocol = "tcp"
 cidr_blocks = ["0.0.0.0/0"]
 security_group_id = "${aws_security_group.default.id}"
}

Now let’s actually create the web servers. This is where things get a little awkward in a DSL, because we want to conditionally attach the key_name, based on whether SSH access is enabled. This requires that we duplicate some configuration for the key and no-key code paths, in addition to splitting, transforming, and indexing into our list of AZs:

 resource "aws_key_pair" "default" {
 count = "${var.public_key == "" ? 0 : 1}"
 key_name_prefix = "default"
 public_key = "${var.public_key}"
}
resource "aws_instance" "web-server-key" {
 count = "${var.public_key == "" ? 0 : length(split(",", var.availability_zones))}"
 ami = "${data.aws_ami.ubuntu.id}"
 instance_type = "t2.micro"
 availability_zone = "${element(split(",", var.availability_zones), count.index)}"
 user_data = "${file("userdata.sh")}"
 security_groups = ["${aws_security_group.default.name}"]
 key_name = "${aws_key_pair.default.0.key_name}"
}
resource "aws_instance" "web-server-nokey" {
 count = "${var.public_key == "" ? length(split(",", var.availability_zones)) : 0}"
 ami = "${data.aws_ami.ubuntu.id}"
 instance_type = "t2.micro"
 availability_zone = "${element(split(",", var.availability_zones), count.index)}"
 user_data = "${file("userdata.sh")}"
 security_groups = ["${aws_security_group.default.name}"]
}

After creating the web servers, we will create and attach an ELB, so that traffic is spread across the AZs. This also requires the same conditional duplication for key and no-key configurations:

resource "aws_elb" "elb" {
name_prefix = "webelb"
availability_zones = ["${split(",", var.availability_zones)}"]
listener {
instance_port = 80
instance_protocol = "http"
lb_port = 80
lb_protocol = "http"
}
health_check {
healthy_threshold = 2
unhealthy_threshold = 2
timeout = 3
target = "HTTP:80/"
interval = 30
}
}
resource "aws_elb_attachment" "web-server-key" {
count = "${var.public_key == "" ? 0 : length(split(",", var.availability_zones))}"
elb = "${aws_elb.elb.id}"
instance = "${element(aws_instance.web-server-key.*.id, count.index)}"
}
resource "aws_elb_attachment" "web-server-nokey" {
count = "${var.public_key == "" ? length(split(",", var.availability_zones)) : 0}"
elb = "${aws_elb.elb.id}"
instance = "${element(aws_instance.web-server-nokey.*.id, count.index)}"
}

Finally, we’ll export the public instance IPs for SSH access (if enabled) and the ELB URL:

output "public_ips" {
 value = "${aws_instance.web-server-key.*.public_ip}"
}
output "url" {
 value = "http://${aws_elb.elb.dns_name}"
}

And that’s it. This program provisions an EC2 VM per AZ in your region of choice, behind a load balancer, with conditional SSH enablement.The full code is available here if you’d like to see it in one place.

Migrating to a General Purpose Language

Now let’s see how to migrate this program to using a general purpose language. There are several opportunities to simplify the program – most immediately apparent, using familiar if statements instead of the the awkward conditionals that use count variables.

We could just go ahead and convert the program by hand, however the tf2pulumi tool was designed to take a Terraform HCL program as input and perform the entire translation automatically for you. This is a great place to get started – and from there we can then refactor the resulting program gradually to leverage languages better.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

The first step is to create a Pulumi project. Download Pulumi here, and then run:

$ pulumi new aws-typescript
-n webservers -d "Pulumi Web Servers" -g -f

Run this from wherever your Terraform program resides. This will initialize and scaffold a minimal Pulumi TypeScript program targeting AWS in the pulumi subdirectory.

We’ll then install tf2pulumi, and run it, to generate the Pulumi TypeScript program in index.ts:

$ tf2pulumi

The full program created by this command can be found here. Notice that it looks structurally very similar to the Terraform HCL program above, except that it’s written in TypeScript instead. Even this rote translation has benefits – for example, open it in your favorite editor, and you’ll notice you get syntax highlighting, statement completion, and integrated documentation. But we’ll soon see that once we have this code in our favorite language, we can refactor it to leverage familiar programming languages more idiomatically, reducing a lot of boilerplate.

After the conversion is complete, we can create, configure, and update a Pulumi stack to create the resources described by the generated Pulumi program:

$ cd pulumi
$ npm install
$ pulumi stack init webservers-prod
$ pulumi config set aws:region us-west-2
$ pulumi config set availabilityZones "us-west-2a,us-west-2b,us-west-2c"
$ pulumi up

A Pulumi stack is an environment that can be configured and deployed to independently of all other stacks. Here we’ve created webservers-prod and set it to deploy into the us-west-2a, us-west-2b, and us-west-2c availability zones in AWS’s us-west-2 region. pulumi up will then show you a plan and then let you confirm it to deploy your resources:

After selecting “yes”, the update will proceed with interactive status updates:

Leveraging Programming Languages

This is great – we now have a working Pulumi program. Let’s now go about simplifying the generated code so that it is easier to read, understand, and maintain. We’ll only scratch the surface in this post, as many of the real benefits shine when you go beyond a single program and use reusable constructs like functions, classes, and packages, to cut down on copy and paste. But even some of the syntactic transformations we’ll make can provide a real boost to productivity.

First, notice that our package dependencies are just NPM packages. The full ecosystem of our language’s native package manager is available to us:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from "fs";

The first order of business is to place the list of configured availability zones into a local variable:

const availabilityZones = config.require("availabilityZones").split(",");

Because Pulumi programs are ordinary programs, we can use variables in the usual ways. Notice that we’ve not only defined a variable, but also split the string, so that we don’t need to do it on every usage within the file, using the standard JavaScript split function.

Terraform data sources are represented as function calls in our program. So, to fetch the Ubuntu AMI, we simply call the getAmi function, passing in the filters as parameters to it:

const ubuntuAmi = aws.getAmi({
 filters: [
 { name: "name", values: ["ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"] },
 { name: "virtualization-type", values: ["hvm"] },
 ],
 mostRecent: true,
 owners: ["099720109477"], // Canonical
});

We already saw how the use of variables can save us from repeating ourselves over and over again, and again, we can use this to avoid repeating the 0.0.0.0/0 CIDR block everywhere:

const anyCidr = ["0.0.0.0/0"];
const webServerSg = new aws.ec2.SecurityGroup("default", {
 egress: [{ cidrBlocks: anyCidr, fromPort: 0, protocol: "-1", toPort: 0 }],
 ingress: [{ cidrBlocks: anyCidr, fromPort: 80, protocol: "tcp", toPort: 80 }],
});

Now we can change how we conditionally create the web server instances. This is where we’ll start to see the big wins, because we can use any of the familiar programming language control constructs, including for loops, case statements, if blocks, function calls, and even higher order list functions like map and filter.

In order to conditionally create a resource in Terraform, we had to set its count to either 0 or the desired AZ count. However, this affects any resources that depend on the conditional resource: the dependent resources either become conditional themselves or require complex interpolation expressions in order to ensure that the configuration does not reference any undefined properties. In our original Terraform configuration, the conditional creation of an AWS Key Pair forces us to declare our web servers twice: once with the key pair and once without.

resource "aws_key_pair" "default" {
count = "${var.public_key == "" ? 0 : 1}"
…
resource "aws_instance" "web-server-key" {
count = "${var.public_key == "" ? 0 : length(split(",", var.availability_zones))}"
…
resource "aws_elb_attachment" "web-server-key" {
count = "${var.public_key == "" ? 0 : length(split(",", var.availability_zones))}"
…
resource "aws_instance" "web-server-nokey" {
count = "${var.public_key == "" ? length(split(",", var.availability_zones)) : 0}"
…
resource "aws_elb_attachment" "web-server-nokey" {
count = "${var.public_key == "" ? length(split(",", var.availability_zones)) : 0}"
…

This is rather cumbersome. The syntactic transformation performed by tf2pulumi preserved this same structure to ensure the semantics of translation are identical, but we can now go back and clean it up considerably.

First, to conditionally allocate an EC2 KeyPair and to open port 22, we use an if statement:

let keyName: pulumi.Output<string> | undefined;
if (publicKey) {
 keyName = new aws.ec2.KeyPair("default", { publicKey }).id;
 new aws.ec2.SecurityGroupRule("allow_ssh", {
 type: "ingress",
 cidrBlocks: anyCidr,
 fromPort: 22,
 toPort: 22,
 protocol: "tcp",
 securityGroupId: webServerSg.id,
 });
}

The use of conditionals just works and interacts with planning just as you might expect, so that we always see the result of the conditional evaluation before we’ve decided to deploy anything.

Another subtle advantage is that we can just pass the webServerSg.id property – Pulumi understands the resource DAG, and the use of this property automatically forms an edge.

Now let’s create our web servers. We might use a for loop for this, and append each one to an array, however it’s just as easy to use the JavaScript map function to do this more declaratively:

const webServers: aws.ec2.Instance[] = availabilityZones.map(az =>
 new aws.ec2.Instance(`web-server-key-${az}`, {
 ami: ubuntuAmi.then(ami => ami.id),
 associatePublicIpAddress: publicKey !== "",
 availabilityZone: az,
 instanceType: "t2.micro",
 securityGroups: [ webServerSg.name ],
 userData: fs.readFileSync("userdata.sh", "utf-8"),
 keyName: keyName,
 }),
);

Interestingly, we’ve passed keyName to the constructor here, even though it might not have been defined (the JavaScript default value of undefined). Pulumi understands that this means “the absence of a value”, which is how we were able to simplify the web server provisioning so greatly. We also read in the userdata.sh file using standard JavaScript I/O functionality.

Now that we’ve simplified our configuration such that the web servers are declared once, we can simplify their interaction with the load balancer. We can just create the ELB last and pass it a list of all of our instances. We can also improve the reliability of configuration by ensuring that the ELB is associated with each availability zone in which we’ve created a web server:

const webServerElb = new aws.elasticloadbalancing.LoadBalancer("elb", {
 availabilityZones: webServers.map(instance => instance.availabilityZone),
 instances: webServers.map(instance => instance.id),
 healthCheck: {
 healthyThreshold: 2,
 interval: 30,
 target: "HTTP:80/",
 timeout: 3,
 unhealthyThreshold: 2,
 },
 listeners: [{
 instancePort: 80,
 instanceProtocol: "http",
 lbPort: 80,
 lbProtocol: "http",
 }],
});

Finally, the way we express outputs is by using standard JavaScript module exports:

export const publicIps = webServers.map(v => v.publicIp);
export const url = webServerElb.dnsName.apply(hostname => `http://${hostname}`);

And that’s it. After doing this refactoring, you can simply run pulumi up, and it will show you any diffs compared to your previous deployment, which you can choose to accept and deploy. The full code is available here.

Next Steps

In this post, we’ve applied general purpose programming languages to infrastructure as code to achieve more expressive and reusable programs, simplify our infrastructure’s readability and maintainability, and deliver a heightened level of productivity. Pulumi is open source and free to use – give it a try today!

Provisioning and Managing Cloud Infrastructure with Pulumi

Donna Malayeri — Fri, 20 Jul 2018 00:00:00 +0000

If you’ve been following the blog, you know that Pulumi is great for building serverless applications, container-based applications, and a combination of the two. But, did you know that you can manage any cloud resource in AWS, Azure, or Google Cloud Platform?

You can use the AWS, Azure, or Google Cloud libraries to manage cloud resources. Using these libraries, you can directly manage the properties of any cloud resource.

For example, in just a few lines of code, you can provision a security group and an EC2 instance:

let group = new aws.ec2.SecurityGroup("web-secgrp", {
ingress: [
{ protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
],
});
let server = new aws.ec2.Instance("web-server-www", {
instanceType: "t2.micro",
securityGroups: [ group.name ], // reference the group object above
ami: "ami-7172b611",
});

Next, let’s add a a CloudWatch metric alarm that is triggered when when the CPU utilization is over 80%:

const metricAlarm = new aws.cloudwatch.MetricAlarm("mymetricalarm", {
comparisonOperator: "GreaterThanOrEqualToThreshold",
evaluationPeriods: 2,
metricName: "CPUUtilization",
namespace: "AWS/EC2",
period: 120,
statistic: "Average",
threshold: 80,
alarmDescription: "This metric monitors ec2 cpu utilization"
});

In addition to alerting, you may want to provision a CloudWatch dashboard to have a single monitoring view for all your application resources. You can define this dashboard directly in code using Pulumi. The following code defines a dashboard and specifies the widgets to include:

const dashboard = new aws.cloudwatch.Dashboard("mydashboard", {
dashboardName: "my-dashboard",
dashboardBody: JSON.stringify({
widgets: [
{
type: "metric",
x: 0,
y: 0,
width: 12,
height: 6,
properties: {
metrics: [
[
"AWS/EC2",
"CPUUtilization",
"InstanceId",
"i-012345"
]
],
period: 300,
stat: "Average",
region: "us-east-1",
title: "EC2 Instance CPU"
}
}
]
})
});

Perhaps you want to post to an SNS topic whenever a user signs in to the AWS console for your production account. This requires provisioning three resources: the SNS topic, a CloudWatch event rule, and a CloudWatch event target. With Pulumi, this can all be specified with just a few lines of JavaScript:

const loginsTopic = new aws.sns.Topic("myloginstopic");
const eventRule = new aws.cloudwatch.EventRule("myeventrule", {
eventPattern: JSON.stringify({
"detail-type": [
"AWS Console Sign In via CloudTrail"
]
})
});
const eventTarget = new aws.cloudwatch.EventTarget("myeventtarget", {
rule: eventRule.name,
targetId: "SendToSNS",
arn: loginsTopic.arn
})

These are just a few examples of the AWS resources you can manage in Pulumi. You can provision Athena databases, DynamoDB tables, IAM users, roles, groups, and role policies, Kinesis streams, and more.

To learn more, take a look at the AWS API documentation and the sample code that provisions a variety of infrastructure resources.

AWS SQS to Lambda with Pulumi: Serverless Queue Example

Cyrus Najmabadi — Tue, 10 Jul 2018 00:00:00 +0000

Two weeks ago Amazon added Simple Queue Service (SQS) as a supported event source for Lambda. SQS is one of AWS’s oldest services, providing access to a powerful message queue that can do things like guarantee messages will be delivered at least once, or messages that will be processed in the same order they were received in. Adding SQS as a supported event source for Lambda means that now it’s possible to use SQS in a serverless computing infrastructure, where Lambdas are triggered in response to messages added to your SQS queue. Now, instead of needing some sort of Service dedicated to polling your SQS queue, or creating Simple Notification Service (SNS) notifications from your messages, you can instead just directly trigger whatever Lambda you want.

Example: Using AWS SQS and Lambda to post messages to Slack

Here’s a simple example of using SQS as an event source with Pulumi: upon receipt of a message via SQS we post a new message into Slack. The full project is available on Github in our examples repo, and includes the instructions to get this up and running.

let aws = require("@pulumi/aws");
let serverless = require("@pulumi/aws-serverless");
let config = require("./config");

let queue = new aws.sqs.Queue("mySlackQueue", { visibilityTimeoutSeconds: 180 });

serverless.queue.subscribe("mySlackPoster", queue, async (e) => {
 let slack = require("@slack/client");
 let client = new slack.WebClient(config.slackToken);
 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*: ${rec.body}`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to Slack channel ${config.slackChannel}`);
 }
}, { batchSize: 1 });

module.exports = {
 queueURL: queue.id,
};

Implementing SQS in Pulumi

When we heard about this new functionality, we immediately knew we wanted to let people access this new power from their Pulumi applications. Based on Amazon’s documentation on this new capability we felt it would likely be very easy to provide. Amazon introduced this functionality by reusing existing systems and APIs. Specifically, to create a Lambda trigger off an SQS message you only need to create a new Event Source Mapping mapping between the two. This seemed like it would be very simple, and we set off to expose the right Pulumi API to make this possible.

@pulumi/aws-serverless is Pulumi’s convenience library for creating Serverless computing components. And we added the necessary code to expose the functionality in our simple Pulumi programming model here. With this new API you would now be able to write code in your Pulumi application like so:

import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

// Create the queue. Provide whatever specific configuration you need here.
const sqsQueue = new aws.sqs.Queue("queue", {
 visibilityTimeoutSeconds: 300,
});

// Set up a subscription that will fire whenever the queue receives a message. Here we ask
// for 'batchSize = 1' so we will only process a single message at a time.
serverless.queue.subscribe("subscription", sqsQueue, async (event) => {
 // Add whatever code you want here to run in the AWS lambda. 'event' will contain the
 // all the necessary data about the message added to the queue.
}, { batchSize: 1 });

This example also demonstrates several of the great capabilities of a Pulumi application. First, the ability to define your infrastructure directly in code (i.e. the AWS Queue, Lambda, and Event Subscriptions), instead of needing to do that separately. Second, the ability to avoid needing to specify every single AWS resource necessary to get up and running. For example, I did not have to supply explicit Roles, Role-Policy-Attachments, or Permissions here as Pulumi took care of that for me by default. Third, the ability to create an AWS Lambda directly from a JavaScript/TypeScript arrow-function. That async (event) => ... is written in code as a normal async-arrow-function, but it will be converted by Pulumi into all the machinery necessary to have a true AWS Lambda running in your infrastructure.

After adding the API and writing up a simple app to test things, we then ran a pulumi update to deploy our application to AWS. Unfortunately, we immediately ran into a problem:

Plan apply failed:
Error creating Lambda event source mapping: ValidationException: 1 validation error detected: Value '' at
'startingPosition' failed to satisfy constraint: Member must satisfy enum value set:
[LATEST, AT_TIMESTAMP, TRIM_HORIZON]

So what happened? Well, as it turns out, Pulumi leverages and integrates with many other amazing Open Source projects under the covers. In this case, it was our integration with Terraform that was causing this small snag. It turns out that while Amazon relaxed their API to no longer have that constraint on ‘startingPosition’, Terraform was still referencing an older awssdk API that still contained that constraint.

At Pulumi we think that Terraform is fantastic, and we love using it. So we thought the best thing we could do in this position was to help out that project to support this new functionality as well. We did this around two weeks ago by contributing this PR to their project. We worked with Terraform over a couple over a few days to get the PR up to snuff, and eventually got it in. With this, now both Terraform and Pulumi customers will be able to benefit from these changes in the upcoming releases for both projects!

After getting the change in we went back and tested our example and saw that now everything worked as expected. We also expanded things out in an example to show how you might use this in practice. In this example, we receive the event, and then just write the data of it into an S3 Bucket. Running this example we now successfully see:

pulumi update
Updating stack 'cysqstest'
Performing changes:
Type Name Status
+ pulumi:pulumi:Stack queue-cysqstest created
+ ├─ aws:serverless:Function subscription-queue-subscription created
+ │ ├─ aws:iam:Role subscription-queue-subscription created
+ │ ├─ aws:iam:RolePolicyAttachment subscription-queue-subscription-32be53a2 created
+ │ ├─ aws:iam:RolePolicyAttachment subscription-queue-subscription-7cd09230 created
+ │ └─ aws:lambda:Function subscription-queue-subscription created
+ ├─ aws:sqs:Queue queue created
+ ├─ aws:s3:Bucket testbucket created
+ └─ aws-serverless:queue:QueueEventSubscription subscription created
+ ├─ aws:lambda:Permission subscription created
+ └─ aws:lambda:EventSourceMapping subscription created
info: 11 changes performed:
+ 11 resources created
Update duration: 28s

Once created, we could then see this Stack at app.pulumi.com, and we could easily use the functionality there to even navigate to those resources over at our AWS Console. Over there were were able to use the console to both send a message to the Queue, verify our Lambda got triggered, and even check our Bucket to see the data written into it. In only around a dozen lines of code, we were able to provision 11 AWS resources, and create a real end-to-end Serverless message-receiving and processing pipeline. Now, we can expand on this functionality with more resources or more functionality, run pulumi update and just simply and safely update our cloud infrastructure!

You’ll see this functionality lighting up in our next release of @pulumi/pulumi and @pulumi/aws-serverless. We hope it helps you out and makes your cloud life that much easier. Happy coding!

Code, Deploy, and Manage a Serverless REST API on AWS

Donna Malayeri — Fri, 22 Jun 2018 00:00:00 +0000

Pulumi makes it easy to build serverless applications and connect to other cloud resources. In this blog post, we’ll create a simple REST API that counts the number of times a route has been hit, using JavaScript to define both the infrastructure and application code.

In Pulumi, you define your application infrastructure in regular code, using JavaScript, Python or Go, and you can target AWS, Azure, Google Cloud, or Kubernetes. The Pulumi command line tool transforms your into a declarative plan, following the best practices of immutable infrastructure. You can write your app code in any language supported by your serverless platform.

Example app: serverless route counter

In this tutorial, we’ll build a simple REST API that counts the number of times a route has been hit. For example:

$ curl $URL/wow
{"route":"wow","count":1}
$ curl $URL/pulumi
{"route":"pulumi","count":1}
$ curl $URL/wow
{"route":"wow","count":2}

We’ll implement this using API Gateway, Lambda, and Dynamo DB:

Setup

If this is your first time using Pulumi, go to app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Create the App

First we’ll create a Pulumi project and add code for both the infrastructure definitions and application code.

To create a new Pulumi project, run the following commands:
```
mkdir hello-http && cd hello-http
pulumi new aws-javascript
```
This creates a new project in the directory hello-http.

Replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");
/* Create a mapping from 'route' to a count */
let counterTable = new cloud.Table("counterTable", "route");
/* Create an REST API endpoint */
let endpoint = new cloud.API("hello-world");
endpoint.get("/{route+}", (req, res) => {
let route = req.params["route"];
console.log(`Getting count for '${route}'`);
/* get previous value and increment */
/* reference outer counterTable object */
counterTable.get({ route }).then(value => {
let count = (value && value.count) || 0;
counterTable.insert({ route, count: ++count }).then(() => {
res.status(200).json({ route, count });
console.log(`Got count ${count} for '${route}'`);
});
});
});
exports.endpoint = endpoint.publish().url;

The definition for counterTable stores a counter for each route, using cloud.Table. On AWS, this provisions a DynamoDB instance. To create a new API Gateway instance, we create an instance of cloud.API. New routes can be added to this endpoint using functions such as get, post, put etc.

The function passed to get is the interesting part: this becomes the body of a new AWS Lambda function that is called on a GET request to the API Gateway. The body of this function can use variables defined in the main program, such as counterTable. This is translated to a lookup on the provisioned DynamoDB instance; there is no need to store the table name in an environment variable.

Finally, add @pulumi/cloud-aws NPM package:

$ npm install --save @pulumi/cloud @pulumi/cloud-aws

Deploy the App

To deploy both the infrastructure and app code, we’ll run pulumi update. This command first shows a preview of all the resources that will be created and prompts for confirmation.

Choose the “yes” option to deploy to AWS. At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to this link and click the Resources tab. You’ll see all the resources you’ve created, including links to the AWS Console.

Test the App

Now that the app is deployed, let’s try it out! With the Pulumi CLI, you can easily view output properties for your stack. This line in the JavaScript code creates a stack output.

exports.endpoint = endpoint.publish().url;

You can view stack outputs in the Pulumi Service, or via pulumi stack output:

$ pulumi stack output
Current stack outputs (1):
OUTPUT VALUE
endpoint https://5e8xrktey3.execute-api.us-west-2.amazonaws.com/stage/

Now, let’s curl some routes!

$ curl $(pulumi stack output endpoint)/wow
{"route":"wow","count":1}
$ curl $(pulumi stack output endpoint)/pulumi
{"route":"pulumi","count":1}
$ curl $(pulumi stack output endpoint)/wow
{"route":"wow","count":2}

You can also view logs for the Lambda via pulumi logs. To get a log tail, use the --follow or -f flag:

$ pulumi logs
Collecting logs for stack hello-http-dev since 2018-06-15T12:53:35.000-07:00.
2018-06-15T13:52:30.123-07:00[ hello-world4fcc7b60] Getting count for 'wow'
2018-06-15T13:52:34.361-07:00[ hello-world4fcc7b60] Got count 1 for 'wow'
2018-06-15T13:52:39.621-07:00[ hello-world4fcc7b60] Getting count for 'pulumi'
2018-06-15T13:52:39.757-07:00[ hello-world4fcc7b60] Got count 1 for 'pulumi'
2018-06-15T13:52:42.189-07:00[ hello-world4fcc7b60] Getting count for 'wow'
2018-06-15T13:52:42.325-07:00[ hello-world4fcc7b60] Got count 2 for 'wow'

Clean up

To clean up the resources, run pulumi destroy.

Next steps

The sample code for this application is available in the Pulumi examples repo on GitHub. For an end-to-end TypeScript application with a frontend, see the URL shortener sample.

Build a Video Thumbnailer on AWS

Donna Malayeri — Thu, 21 Jun 2018 00:00:00 +0000

Pulumi makes it easy to build cloud applications that use a combination of containers, lambdas, and connected data services and infrastructure: Colada apps.

An example of a Colada app is extracting a thumbnail from a video. A serverless function can only run for 5 minutes, so we’ll run a container in AWS Fargate to do the video processing.

In this app, a Lambda function is triggered whenever a new video is uploaded to S3. This function launches a task in Fargate that uses FFmpeg to extract a video thumbnail. A second Lambda function is triggered when a new thumbnail has been created.

This post was inspired by a blog post from Serverless, Inc that shows how to combine AWS Fargate and Lambda for a long-running process. Let’s see how a similar app would be implemented in Pulumi entirely in code.

Setup

If this is your first time using Pulumi, go to https://app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Finally, make sure Docker is installed and running.

Create the App

We’ll create a Pulumi project, define the infrastructure and app code in JavaScript, and create a Dockerfile for the Fargate task.

First, to create a new Pulumi project, run the following commands:

mkdir hello-colada && cd hello-colada
pulumi new aws-javascript

This creates a new project in the hello-colada directory.

Second, replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");

// A bucket to store videos and thumbnails.
const bucket = new cloud.Bucket("bucket");
const bucketName = bucket.bucket.id;

// A task which runs a containerized FFMPEG job to extract a thumbnail image.
const ffmpegThumbnailTask = new cloud.Task("ffmpegThumbTask", {
 build: "./", // folder containing the Dockerfile
 memoryReservation: 128,
});

// When a new video is uploaded, run the FFMPEG task on the video file.
// Use the time index specified in the filename (e.g. cat_00-01.mp4 uses timestamp 00:01)
bucket.onPut("onNewVideo", async (bucketArgs) => {
 console.log(`*** New video: file ${bucketArgs.key} was uploaded at ${bucketArgs.eventTime}.`);
 const file = bucketArgs.key;

 const thumbnailFile = file.substring(0, file.indexOf('_')) + '.jpg';
 const framePos = file.substring(file.indexOf('_')+1, file.indexOf('.')).replace('-',':');

 await ffmpegThumbnailTask.run({
 environment: {
 "S3_BUCKET": bucketName.get(),
 "INPUT_VIDEO": file,
 "TIME_OFFSET": framePos,
 "OUTPUT_FILE": thumbnailFile,
 },
 });
 console.log(`Running thumbnailer task.`);
}, { keySuffix: ".mp4" });

// When a new thumbnail is created, log a message.
bucket.onPut("onNewThumbnail", async (bucketArgs) => {
 console.log(`*** New thumbnail: file ${bucketArgs.key} was saved at ${bucketArgs.eventTime}.`);
}, { keySuffix: ".jpg" });

// Export the bucket name.
exports.bucketName = bucketName;

This code uses cloud.Task, a high-level, convenient component for working with containers. The component automatically provisions a container registry instance in ECR, runs a Docker build, and saves the Docker image to the provisioned ECR instance. It also defines an ECS task and configures it to use the built image. All this in just 4 lines of code!

Next, in the same directory, create a Dockerfile with the following contents. We’ll use an existing FFmpeg container and install the AWS CLI. When the container is started, it copies the video file from S3, runs ffmpeg, and copies the output back to S3.

 FROM jrottenberg/ffmpeg
RUN apt-get update &&
apt-get install python-dev python-pip -y &&
apt-get clean
RUN pip install awscli
WORKDIR /tmp/workdir
ENTRYPOINT
echo "Starting ffmpeg task..." &&
echo "Copying video from S3" &&
aws s3 cp s3://${S3_BUCKET}/${INPUT_VIDEO} ./${INPUT_VIDEO} &&
ffmpeg -v error -i ./${INPUT_VIDEO} -ss ${TIME_OFFSET} -vframes 1 -f image2 -an -y ${OUTPUT_FILE} &&
echo "Copying thumbnail to S3" &&
aws s3 cp ./${OUTPUT_FILE} s3://${S3_BUCKET}/${OUTPUT_FILE}

Install the @pulumi/cloud-aws NPM package:

$ npm install --save @pulumi/cloud-aws @pulumi/cloud

Finally, configure Pulumi to use AWS Fargate. Note that, currently, Fargate is available only in us-east-1, us-east-2, us-west-2, and eu-west-1.

$ pulumi config set cloud-aws:useFargate true

That’s it! Now, let’s deploy the app.

Deploy the App

To deploy both the infrastructure and app code, we’ll run pulumi update. This command first shows a preview of all the resources that will be created and prompts for confirmation. During the preview phase, Pulumi invokes docker build.

Choose the “yes” option to deploy to AWS. This will take about 5 minutes. Pulumi automatically builds and provisions an AWS container repository in ECR, builds the Docker container, and places the image in the repository. This all happens automatically and does not require manual configuration on your part.

At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to the Resources tab and filter to S3 resources. Then, on the S3 bucket, click the link to go to the AWS Console.

Upload a Video

Upload an .mp4 video to your S3 bucket, making sure to encode the desired time index in the filename. For instance, use myvideo_00-02.mp4 to extract the frame at 0 minutes and 2 seconds.

You can use this video of my cat, which you should save as cat_00-01.mp4.

Upload this video in the S3 console. Or, use the AWS CLI along with the pulumi stack output command:

$ aws s3 cp cat_00-01.mp4 s3://$(pulumi stack output bucketName)
upload: cat_00-01.mp4 to s3://bucket-3a4f226/cat_00-01.mp4

To see what the app is doing, use the pulumi logs command, with the --follow or -f parameter. This command aggregates all the logs for your compute, so the logs for the Lambda function and the Fargate task are all in one place!

Once the thumbnail has been generated, either view it in the S3 console, or download it with the AWS CLI:

$ aws s3 cp s3://$(pulumi stack output bucketName)/cat.jpg .
download: s3://bucket-0c91106/cat.jpg to ./cat.jpg

Clean Up

To clean up the resources we’ve provisioned, run pulumi destroy.

Next Steps

In this post, we saw how easy it is to use containers and serverless functions in one application. With Pulumi, you get the best of both worlds, and don’t have to choose one or the other.

The sample code for this post is available in the Pulumi examples repo on GitHub.

For a version of this sample that includes AWS Rekognition, see the Video Thumbnailer with Machine Learning JavaScript example.

Deploying production-ready containers with Pulumi

Donna Malayeri — Wed, 20 Jun 2018 00:00:00 +0000

Containers are a great way to deploy applications to the cloud, especially with new execution models like AWS Fargate. Pulumi makes it easy to deploy production Docker containers, handling details such as creating a container registry instance in ECR, creating task definitions in ECS, and configuring a load balancer. With Pulumi, deploying a container to production is almost as easy as running it locally!

In this blog post, we’ll deploy a simple Docker container running NGINX.

Setup

If this is your first time using Pulumi, go to https://app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Finally, make sure Docker is installed and running.

Create the app

We’ll create a Pulumi project, define the infrastructure in JavaScript, and create a Dockerfile.

To create a new Pulumi project, run the following commands:

mkdir hello-containers && cd hello-containers
pulumi new aws-javascript

This creates a new project in the directory hello-containers. Next, replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");
let service = new cloud.Service("pulumi-nginx", {
 containers: {
 nginx: {
 build: "./app",
 memory: 128,
 ports: [{ port: 80 }],
 },
 },
 replicas: 2,
});

/* export just the hostname property of the container frontend */
exports.url = service.defaultEndpoint.apply(e => `http://${e.hostname}`);

These 15 lines of code are everything you need to deploy a custom container! We’re using cloud.Service, which is a high-level, convenient interface for building containers and provisioning an AWS container service. Using the build property, we point to a folder containing a Dockerfile, which is app in this case.

Now, let’s create a Dockerfile and a static page. Create a subfolder app with the following files.

Add the following as Dockerfile:

FROM nginx
COPY index.html /usr/share/nginx/html

Add the following as index.html:

<html>
 <head>
 <title>Hello World</title>
 <meta charset="UTF-8">
 </head>
 <body>
 <p>Hello containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

You should have the following directory structure:

Install the @pulumi/cloud-aws NPM package:

npm install –save @pulumi/cloud-aws @pulumi/cloud
Configure Pulumi to use AWS Fargate. Note that, currently, Fargate is available only in us-east-1, us-east-2, us-west-2, and eu-west-1.

pulumi config set cloud-aws:useFargate true

Deploy the app

Choose the yes option to deploy to AWS. This will take about 5 minutes. Pulumi automatically builds and provisions an AWS container repository in ECR, builds the Docker container, and places the image in the repository. This all happens automatically and does not require manual configuration on your part.

At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to this link and click the Resources tab. You’ll see all the resources you’ve created. Notice that Pulumi has created an ECR repository, a load balancer, an ECS service and task definition, and IAM roles.

Now, go back to the stack details page, which shows the stack configuration, as well as the stack output property. The following line creates the stack output url:

exports.url = service.defaultEndpoint.apply(e => `http://${e.hostname}`);

If you navigate to the link for url you’ll see the following page:

View logs

You can view container frontend and compute logs via the pulumi logs command. You can see here that most of the traffic is hitting routes that don’t exist, aside from the last log line where I navigated to the site using Chrome. Since this is an HTTP server that’s open to the internet, it’s not uncommon for attackers to try to find a security vulnerability on a site. Fortunately, our container is serving only static content. The logs command makes it easy to get details on your site traffic.

$ pulumi logs
Collecting logs for stack hello-containers-dev since 2018-06-15T15:01:18.000-07:00.
2018-06-15T15:27:47.468-07:00[ pulumi-nginx] 2018/06/15 22:27:47 [error] 5#5: *4 open() "/usr/share/nginx/html/proxychecker.axd" failed (2: No such file or directory), client: 62.210.157.152, server: localhost, request: "GET http://proxy.dazhou.net/proxychecker.axd?e=457758511%40qq.com&p=http%3A%2F%2F34.201.27.186%3A80&s=AS HTTP/1.1", host: "proxy.dazhou.net"
2018-06-15T15:27:47.468-07:00[ pulumi-nginx] 62.210.157.152 - - [15/Jun/2018:22:27:47 +0000] "GET http://proxy.dazhou.net/proxychecker.axd?e=457758511%40qq.com&p=http%3A%2F%2F34.201.27.186%3A80&s=AS HTTP/1.1" 404 170 "-" "Go-http-client/1.1" "-"
2018-06-15T15:32:02.128-07:00[ pulumi-nginx] 2018/06/15 22:32:02 [error] 5#5: *382 "/usr/share/nginx/html/phpmyadmin/index.html" is not found (2: No such file or directory), client: 178.239.177.212, server: localhost, request: "HEAD http://34.201.27.186:80/phpmyadmin/ HTTP/1.1", host: "34.201.27.186"
2018-06-15T16:07:05.874-07:00[ pulumi-nginx] 172.31.44.144 - - [15/Jun/2018:23:07:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36" "-"

Clean up

To clean up the resources we’ve provisioned, run pulumi destroy.

Next steps

The sample code for this post is available in the Pulumi examples repo on GitHub. For an example application that connects two containers, see the Voting App TypeScript sample.