Pulumi Blog: Donna Malayeri

Provisioning and Managing Cloud Infrastructure with Pulumi

Donna Malayeri — Fri, 20 Jul 2018 00:00:00 +0000

If you’ve been following the blog, you know that Pulumi is great for building serverless applications, container-based applications, and a combination of the two. But, did you know that you can manage any cloud resource in AWS, Azure, or Google Cloud Platform?

You can use the AWS, Azure, or Google Cloud libraries to manage cloud resources. Using these libraries, you can directly manage the properties of any cloud resource.

For example, in just a few lines of code, you can provision a security group and an EC2 instance:

let group = new aws.ec2.SecurityGroup("web-secgrp", {
ingress: [
{ protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
],
});
let server = new aws.ec2.Instance("web-server-www", {
instanceType: "t2.micro",
securityGroups: [ group.name ], // reference the group object above
ami: "ami-7172b611",
});

Next, let’s add a a CloudWatch metric alarm that is triggered when when the CPU utilization is over 80%:

const metricAlarm = new aws.cloudwatch.MetricAlarm("mymetricalarm", {
comparisonOperator: "GreaterThanOrEqualToThreshold",
evaluationPeriods: 2,
metricName: "CPUUtilization",
namespace: "AWS/EC2",
period: 120,
statistic: "Average",
threshold: 80,
alarmDescription: "This metric monitors ec2 cpu utilization"
});

In addition to alerting, you may want to provision a CloudWatch dashboard to have a single monitoring view for all your application resources. You can define this dashboard directly in code using Pulumi. The following code defines a dashboard and specifies the widgets to include:

const dashboard = new aws.cloudwatch.Dashboard("mydashboard", {
dashboardName: "my-dashboard",
dashboardBody: JSON.stringify({
widgets: [
{
type: "metric",
x: 0,
y: 0,
width: 12,
height: 6,
properties: {
metrics: [
[
"AWS/EC2",
"CPUUtilization",
"InstanceId",
"i-012345"
]
],
period: 300,
stat: "Average",
region: "us-east-1",
title: "EC2 Instance CPU"
}
}
]
})
});

Perhaps you want to post to an SNS topic whenever a user signs in to the AWS console for your production account. This requires provisioning three resources: the SNS topic, a CloudWatch event rule, and a CloudWatch event target. With Pulumi, this can all be specified with just a few lines of JavaScript:

const loginsTopic = new aws.sns.Topic("myloginstopic");
const eventRule = new aws.cloudwatch.EventRule("myeventrule", {
eventPattern: JSON.stringify({
"detail-type": [
"AWS Console Sign In via CloudTrail"
]
})
});
const eventTarget = new aws.cloudwatch.EventTarget("myeventtarget", {
rule: eventRule.name,
targetId: "SendToSNS",
arn: loginsTopic.arn
})

These are just a few examples of the AWS resources you can manage in Pulumi. You can provision Athena databases, DynamoDB tables, IAM users, roles, groups, and role policies, Kinesis streams, and more.

To learn more, take a look at the AWS API documentation and the sample code that provisions a variety of infrastructure resources.

Code, Deploy, and Manage a Serverless REST API on AWS

Donna Malayeri — Fri, 22 Jun 2018 00:00:00 +0000

Pulumi makes it easy to build serverless applications and connect to other cloud resources. In this blog post, we’ll create a simple REST API that counts the number of times a route has been hit, using JavaScript to define both the infrastructure and application code.

In Pulumi, you define your application infrastructure in regular code, using JavaScript, Python or Go, and you can target AWS, Azure, Google Cloud, or Kubernetes. The Pulumi command line tool transforms your into a declarative plan, following the best practices of immutable infrastructure. You can write your app code in any language supported by your serverless platform.

Example app: serverless route counter

In this tutorial, we’ll build a simple REST API that counts the number of times a route has been hit. For example:

$ curl $URL/wow
{"route":"wow","count":1}
$ curl $URL/pulumi
{"route":"pulumi","count":1}
$ curl $URL/wow
{"route":"wow","count":2}

We’ll implement this using API Gateway, Lambda, and Dynamo DB:

Setup

If this is your first time using Pulumi, go to app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Create the App

First we’ll create a Pulumi project and add code for both the infrastructure definitions and application code.

To create a new Pulumi project, run the following commands:
```
mkdir hello-http && cd hello-http
pulumi new aws-javascript
```
This creates a new project in the directory hello-http.

Replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");
/* Create a mapping from 'route' to a count */
let counterTable = new cloud.Table("counterTable", "route");
/* Create an REST API endpoint */
let endpoint = new cloud.API("hello-world");
endpoint.get("/{route+}", (req, res) => {
let route = req.params["route"];
console.log(`Getting count for '${route}'`);
/* get previous value and increment */
/* reference outer counterTable object */
counterTable.get({ route }).then(value => {
let count = (value && value.count) || 0;
counterTable.insert({ route, count: ++count }).then(() => {
res.status(200).json({ route, count });
console.log(`Got count ${count} for '${route}'`);
});
});
});
exports.endpoint = endpoint.publish().url;

The definition for counterTable stores a counter for each route, using cloud.Table. On AWS, this provisions a DynamoDB instance. To create a new API Gateway instance, we create an instance of cloud.API. New routes can be added to this endpoint using functions such as get, post, put etc.

The function passed to get is the interesting part: this becomes the body of a new AWS Lambda function that is called on a GET request to the API Gateway. The body of this function can use variables defined in the main program, such as counterTable. This is translated to a lookup on the provisioned DynamoDB instance; there is no need to store the table name in an environment variable.

Finally, add @pulumi/cloud-aws NPM package:

$ npm install --save @pulumi/cloud @pulumi/cloud-aws

Deploy the App

To deploy both the infrastructure and app code, we’ll run pulumi update. This command first shows a preview of all the resources that will be created and prompts for confirmation.

Choose the “yes” option to deploy to AWS. At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to this link and click the Resources tab. You’ll see all the resources you’ve created, including links to the AWS Console.

Test the App

Now that the app is deployed, let’s try it out! With the Pulumi CLI, you can easily view output properties for your stack. This line in the JavaScript code creates a stack output.

exports.endpoint = endpoint.publish().url;

You can view stack outputs in the Pulumi Service, or via pulumi stack output:

$ pulumi stack output
Current stack outputs (1):
OUTPUT VALUE
endpoint https://5e8xrktey3.execute-api.us-west-2.amazonaws.com/stage/

Now, let’s curl some routes!

$ curl $(pulumi stack output endpoint)/wow
{"route":"wow","count":1}
$ curl $(pulumi stack output endpoint)/pulumi
{"route":"pulumi","count":1}
$ curl $(pulumi stack output endpoint)/wow
{"route":"wow","count":2}

You can also view logs for the Lambda via pulumi logs. To get a log tail, use the --follow or -f flag:

$ pulumi logs
Collecting logs for stack hello-http-dev since 2018-06-15T12:53:35.000-07:00.
2018-06-15T13:52:30.123-07:00[ hello-world4fcc7b60] Getting count for 'wow'
2018-06-15T13:52:34.361-07:00[ hello-world4fcc7b60] Got count 1 for 'wow'
2018-06-15T13:52:39.621-07:00[ hello-world4fcc7b60] Getting count for 'pulumi'
2018-06-15T13:52:39.757-07:00[ hello-world4fcc7b60] Got count 1 for 'pulumi'
2018-06-15T13:52:42.189-07:00[ hello-world4fcc7b60] Getting count for 'wow'
2018-06-15T13:52:42.325-07:00[ hello-world4fcc7b60] Got count 2 for 'wow'

Clean up

To clean up the resources, run pulumi destroy.

Next steps

The sample code for this application is available in the Pulumi examples repo on GitHub. For an end-to-end TypeScript application with a frontend, see the URL shortener sample.

Build a Video Thumbnailer on AWS

Donna Malayeri — Thu, 21 Jun 2018 00:00:00 +0000

Pulumi makes it easy to build cloud applications that use a combination of containers, lambdas, and connected data services and infrastructure: Colada apps.

An example of a Colada app is extracting a thumbnail from a video. A serverless function can only run for 5 minutes, so we’ll run a container in AWS Fargate to do the video processing.

In this app, a Lambda function is triggered whenever a new video is uploaded to S3. This function launches a task in Fargate that uses FFmpeg to extract a video thumbnail. A second Lambda function is triggered when a new thumbnail has been created.

This post was inspired by a blog post from Serverless, Inc that shows how to combine AWS Fargate and Lambda for a long-running process. Let’s see how a similar app would be implemented in Pulumi entirely in code.

Setup

If this is your first time using Pulumi, go to https://app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Finally, make sure Docker is installed and running.

Create the App

We’ll create a Pulumi project, define the infrastructure and app code in JavaScript, and create a Dockerfile for the Fargate task.

First, to create a new Pulumi project, run the following commands:

mkdir hello-colada && cd hello-colada
pulumi new aws-javascript

This creates a new project in the hello-colada directory.

Second, replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");

// A bucket to store videos and thumbnails.
const bucket = new cloud.Bucket("bucket");
const bucketName = bucket.bucket.id;

// A task which runs a containerized FFMPEG job to extract a thumbnail image.
const ffmpegThumbnailTask = new cloud.Task("ffmpegThumbTask", {
 build: "./", // folder containing the Dockerfile
 memoryReservation: 128,
});

// When a new video is uploaded, run the FFMPEG task on the video file.
// Use the time index specified in the filename (e.g. cat_00-01.mp4 uses timestamp 00:01)
bucket.onPut("onNewVideo", async (bucketArgs) => {
 console.log(`*** New video: file ${bucketArgs.key} was uploaded at ${bucketArgs.eventTime}.`);
 const file = bucketArgs.key;

 const thumbnailFile = file.substring(0, file.indexOf('_')) + '.jpg';
 const framePos = file.substring(file.indexOf('_')+1, file.indexOf('.')).replace('-',':');

 await ffmpegThumbnailTask.run({
 environment: {
 "S3_BUCKET": bucketName.get(),
 "INPUT_VIDEO": file,
 "TIME_OFFSET": framePos,
 "OUTPUT_FILE": thumbnailFile,
 },
 });
 console.log(`Running thumbnailer task.`);
}, { keySuffix: ".mp4" });

// When a new thumbnail is created, log a message.
bucket.onPut("onNewThumbnail", async (bucketArgs) => {
 console.log(`*** New thumbnail: file ${bucketArgs.key} was saved at ${bucketArgs.eventTime}.`);
}, { keySuffix: ".jpg" });

// Export the bucket name.
exports.bucketName = bucketName;

This code uses cloud.Task, a high-level, convenient component for working with containers. The component automatically provisions a container registry instance in ECR, runs a Docker build, and saves the Docker image to the provisioned ECR instance. It also defines an ECS task and configures it to use the built image. All this in just 4 lines of code!

Next, in the same directory, create a Dockerfile with the following contents. We’ll use an existing FFmpeg container and install the AWS CLI. When the container is started, it copies the video file from S3, runs ffmpeg, and copies the output back to S3.

 FROM jrottenberg/ffmpeg
RUN apt-get update &&
apt-get install python-dev python-pip -y &&
apt-get clean
RUN pip install awscli
WORKDIR /tmp/workdir
ENTRYPOINT
echo "Starting ffmpeg task..." &&
echo "Copying video from S3" &&
aws s3 cp s3://${S3_BUCKET}/${INPUT_VIDEO} ./${INPUT_VIDEO} &&
ffmpeg -v error -i ./${INPUT_VIDEO} -ss ${TIME_OFFSET} -vframes 1 -f image2 -an -y ${OUTPUT_FILE} &&
echo "Copying thumbnail to S3" &&
aws s3 cp ./${OUTPUT_FILE} s3://${S3_BUCKET}/${OUTPUT_FILE}

Install the @pulumi/cloud-aws NPM package:

$ npm install --save @pulumi/cloud-aws @pulumi/cloud

Finally, configure Pulumi to use AWS Fargate. Note that, currently, Fargate is available only in us-east-1, us-east-2, us-west-2, and eu-west-1.

$ pulumi config set cloud-aws:useFargate true

That’s it! Now, let’s deploy the app.

Deploy the App

To deploy both the infrastructure and app code, we’ll run pulumi update. This command first shows a preview of all the resources that will be created and prompts for confirmation. During the preview phase, Pulumi invokes docker build.

Choose the “yes” option to deploy to AWS. This will take about 5 minutes. Pulumi automatically builds and provisions an AWS container repository in ECR, builds the Docker container, and places the image in the repository. This all happens automatically and does not require manual configuration on your part.

At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to the Resources tab and filter to S3 resources. Then, on the S3 bucket, click the link to go to the AWS Console.

Upload a Video

Upload an .mp4 video to your S3 bucket, making sure to encode the desired time index in the filename. For instance, use myvideo_00-02.mp4 to extract the frame at 0 minutes and 2 seconds.

You can use this video of my cat, which you should save as cat_00-01.mp4.

Upload this video in the S3 console. Or, use the AWS CLI along with the pulumi stack output command:

$ aws s3 cp cat_00-01.mp4 s3://$(pulumi stack output bucketName)
upload: cat_00-01.mp4 to s3://bucket-3a4f226/cat_00-01.mp4

To see what the app is doing, use the pulumi logs command, with the --follow or -f parameter. This command aggregates all the logs for your compute, so the logs for the Lambda function and the Fargate task are all in one place!

Once the thumbnail has been generated, either view it in the S3 console, or download it with the AWS CLI:

$ aws s3 cp s3://$(pulumi stack output bucketName)/cat.jpg .
download: s3://bucket-0c91106/cat.jpg to ./cat.jpg

Clean Up

To clean up the resources we’ve provisioned, run pulumi destroy.

Next Steps

In this post, we saw how easy it is to use containers and serverless functions in one application. With Pulumi, you get the best of both worlds, and don’t have to choose one or the other.

The sample code for this post is available in the Pulumi examples repo on GitHub.

For a version of this sample that includes AWS Rekognition, see the Video Thumbnailer with Machine Learning JavaScript example.

Deploying production-ready containers with Pulumi

Donna Malayeri — Wed, 20 Jun 2018 00:00:00 +0000

Containers are a great way to deploy applications to the cloud, especially with new execution models like AWS Fargate. Pulumi makes it easy to deploy production Docker containers, handling details such as creating a container registry instance in ECR, creating task definitions in ECS, and configuring a load balancer. With Pulumi, deploying a container to production is almost as easy as running it locally!

In this blog post, we’ll deploy a simple Docker container running NGINX.

Setup

If this is your first time using Pulumi, go to https://app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Finally, make sure Docker is installed and running.

Create the app

We’ll create a Pulumi project, define the infrastructure in JavaScript, and create a Dockerfile.

To create a new Pulumi project, run the following commands:

mkdir hello-containers && cd hello-containers
pulumi new aws-javascript

This creates a new project in the directory hello-containers. Next, replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");
let service = new cloud.Service("pulumi-nginx", {
 containers: {
 nginx: {
 build: "./app",
 memory: 128,
 ports: [{ port: 80 }],
 },
 },
 replicas: 2,
});

/* export just the hostname property of the container frontend */
exports.url = service.defaultEndpoint.apply(e => `http://${e.hostname}`);

These 15 lines of code are everything you need to deploy a custom container! We’re using cloud.Service, which is a high-level, convenient interface for building containers and provisioning an AWS container service. Using the build property, we point to a folder containing a Dockerfile, which is app in this case.

Now, let’s create a Dockerfile and a static page. Create a subfolder app with the following files.

Add the following as Dockerfile:

FROM nginx
COPY index.html /usr/share/nginx/html

Add the following as index.html:

<html>
 <head>
 <title>Hello World</title>
 <meta charset="UTF-8">
 </head>
 <body>
 <p>Hello containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

You should have the following directory structure:

Install the @pulumi/cloud-aws NPM package:

npm install –save @pulumi/cloud-aws @pulumi/cloud
Configure Pulumi to use AWS Fargate. Note that, currently, Fargate is available only in us-east-1, us-east-2, us-west-2, and eu-west-1.

pulumi config set cloud-aws:useFargate true

Deploy the app

Choose the yes option to deploy to AWS. This will take about 5 minutes. Pulumi automatically builds and provisions an AWS container repository in ECR, builds the Docker container, and places the image in the repository. This all happens automatically and does not require manual configuration on your part.

At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to this link and click the Resources tab. You’ll see all the resources you’ve created. Notice that Pulumi has created an ECR repository, a load balancer, an ECS service and task definition, and IAM roles.

Now, go back to the stack details page, which shows the stack configuration, as well as the stack output property. The following line creates the stack output url:

exports.url = service.defaultEndpoint.apply(e => `http://${e.hostname}`);

If you navigate to the link for url you’ll see the following page:

View logs

You can view container frontend and compute logs via the pulumi logs command. You can see here that most of the traffic is hitting routes that don’t exist, aside from the last log line where I navigated to the site using Chrome. Since this is an HTTP server that’s open to the internet, it’s not uncommon for attackers to try to find a security vulnerability on a site. Fortunately, our container is serving only static content. The logs command makes it easy to get details on your site traffic.

$ pulumi logs
Collecting logs for stack hello-containers-dev since 2018-06-15T15:01:18.000-07:00.
2018-06-15T15:27:47.468-07:00[ pulumi-nginx] 2018/06/15 22:27:47 [error] 5#5: *4 open() "/usr/share/nginx/html/proxychecker.axd" failed (2: No such file or directory), client: 62.210.157.152, server: localhost, request: "GET http://proxy.dazhou.net/proxychecker.axd?e=457758511%40qq.com&p=http%3A%2F%2F34.201.27.186%3A80&s=AS HTTP/1.1", host: "proxy.dazhou.net"
2018-06-15T15:27:47.468-07:00[ pulumi-nginx] 62.210.157.152 - - [15/Jun/2018:22:27:47 +0000] "GET http://proxy.dazhou.net/proxychecker.axd?e=457758511%40qq.com&p=http%3A%2F%2F34.201.27.186%3A80&s=AS HTTP/1.1" 404 170 "-" "Go-http-client/1.1" "-"
2018-06-15T15:32:02.128-07:00[ pulumi-nginx] 2018/06/15 22:32:02 [error] 5#5: *382 "/usr/share/nginx/html/phpmyadmin/index.html" is not found (2: No such file or directory), client: 178.239.177.212, server: localhost, request: "HEAD http://34.201.27.186:80/phpmyadmin/ HTTP/1.1", host: "34.201.27.186"
2018-06-15T16:07:05.874-07:00[ pulumi-nginx] 172.31.44.144 - - [15/Jun/2018:23:07:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36" "-"

Clean up

To clean up the resources we’ve provisioned, run pulumi destroy.

Next steps

The sample code for this post is available in the Pulumi examples repo on GitHub. For an example application that connects two containers, see the Voting App TypeScript sample.