Pulumi Blog: Cyrus Najmabadi

Scheduling Serverless

Cyrus Najmabadi — Tue, 25 Feb 2020 00:00:00 +0000

Scheduling events has long been an essential part of automation; many tasks need to run at specific times or intervals. You could be checking StackOverflow for new questions every 20 minutes or compiling a report that is emailed every other Friday at 4:00 pm. Today, many of these tasks can be efficiently accomplished in the cloud. While each cloud has its flavor of scheduled functions, this post steps you through an example using AWS CloudWatch with the help of Pulumi.

Taking out the trash

Let’s assume we have an S3 Bucket similar to the Recycle Bin on your desktop. It contains discarded items that we might not be ready to delete yet, but because we are paying for storage, we want to empty this bucket every Friday at 6:00 pm EST.

import * as aws from "@pulumi/aws";
import { ObjectIdentifier } from "aws-sdk/clients/s3";

// Create an AWS resource (S3 Bucket)
const trashBucket = new aws.s3.Bucket("trash");

We’re now ready to create our Lambda function and schedule it. Pulumi simplifies creating Lambda functions by allowing us to provide a handler function to an event such as cloudwatch.onSchedule. To learn more, check out our Lambdas as Lambdas post. Let’s create a cloudwatch.EventRuleEventHandler async function that will get all the items from the trash bucket and delete them.

// A handler function that will list objects in the bucket and bulk delete them
const emptyTrash: aws.cloudwatch.EventRuleEventHandler = async (
 event: aws.cloudwatch.EventRuleEvent
) => {
 const s3Client = new aws.sdk.S3(); //creates interface to service
 const bucket = trashBucket.id.get();

 const { Contents = [] } = await s3Client //get list of objects in bucket
 .listObjects({ Bucket: bucket })
 .promise();
 const objects: ObjectIdentifier[] = Contents.map(object => {
 return { Key: object.Key! };
 });

 await s3Client //delete objects
 .deleteObjects({
 Bucket: bucket,
 Delete: { Objects: objects, Quiet: false }
 })
 .promise()
 .catch(error => console.log(error));
 console.log(
 `Deleted ${Contents.length} item${
 Contents.length === 1 ? "" : "s"
 } from ${bucket}.`
 );
};

Now that we have our handler function, we can create a CloudWatch event that fires based on the specified schedule, which is every Friday at 6:00 pm EST. We’ll need to represent that as a Schedule Expression. CloudWatch events support cron and rate expressions. If we wanted our function to run on a set interval (i.e., every 20 minutes), we would choose a rate expression. Since we want fine-grained schedule control, we’ll be using a cron expression. You can learn more at the Schedule Expressions documentation. Because time for cron jobs uses UTC, we’ll need to schedule our event to fire on Friday at 11:00 pm UTC.

// Schedule the function to run every Friday at 11:00pm UTC (6:00pm EST)
// More info on Schedule Expressions at
// https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
const emptyTrashSchedule: aws.cloudwatch.EventRuleEventSubscription = aws.cloudwatch.onSchedule(
 "emptyTrash",
 "cron(0 23 ? * FRI *)",
 emptyTrash);

// Export the name of the bucket
export const bucketName = trashBucket.id;

Now, run pulumi up to deploy your new scheduled function.

$ pulumi up
Previewing update (dev):
…

Updating (dev):

 Type Name Status
 pulumi:pulumi:Stack aws-ts-scheduled-function-dev
 + └─ aws:cloudwatch:EventRuleEventSubscription emptyTrash created
 + ├─ aws:cloudwatch:EventRule emptyTrash created
 + ├─ aws:iam:Role emptyTrash created
 + ├─ aws:iam:RolePolicyAttachment emptyTrash-32be53a2 created
 + ├─ aws:lambda:Function emptyTrash created
 + ├─ aws:lambda:Permission emptyTrash created
 + └─ aws:cloudwatch:EventTarget emptyTrash created

Outputs:
 bucketName: "trash-1cde441"

Resources:
 + 7 created
 2 unchanged

Duration: 14s

Test it out by adding some items to your bucket and see if they’re deleted. You can use pulumi logs to view the console.log from the handler function. If you can’t wait until Friday, experiment with changing the cron expression and rerun pulumi up. You can find the full code in our GitHub repository.

Want to know more?

This simple example demonstrates what you can do with serverless functions, such as AWS Lambda, without having to set up your infrastructure. Check out more serverless examples in the Pulumi Examples to see what you can build.

Building and Publishing Docker Images to an ECR Repository

Cyrus Najmabadi — Tue, 18 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Amazon Elastic Container Registry (ECR) is a fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. ECR is integrated with Amazon Elastic Container Service (ECS), including for Kubernetes (EKS), simplifying your development to production workflow, securing access through IAM, and eliminating the need to operate your own container repositories or worry about scaling the underlying infrastructure. ECR hosts your images in a highly available and scalable architecture, allowing you to reliably deploy containers for your applications. In this article, we’ll see how Pulumi Crosswalk for AWS lets you use infrastructure as code to easily build, publish, and pull from private ECR repositories.

A Simple ECS Fargate Service

Let’s see how using Pulumi you can provision an ECR repository, and build and publish to it, in just a few lines of code. Additional features such as lifecycle management make it easy to declare policies specifying how and when stale images should be dropped. Built images can then be referenced from your ECS services (whether EC2, Fargate, or EKS), allowing you to easily version both your images and your infrastructure with one simple, auditable, system.

Let’s take a look at this in action. First, we’ll just start with a simple Pulumi program that creates a load balanced Fargate Service that accessible to the internet, but uses the public nginx container image from the Docker Hub:

// A simple NGINX service, scaled out over two containers.
const nginx = new awsx.ecs.FargateService("nginx", {
 cluster,
 desiredCount: 2,
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: "nginx",
 memory: 128,
 portMappings: [new awsx.elasticloadbalancingv2.ApplicationListener("nginx", { port: 80 })],
 },
 },
 },
});

export const nginxEndpoint = nginxListener.endpoint;

Running this give us:

Updating (teststack):
Type Name Status
+ pulumi:pulumi:Stack teststack created
+ ├─ awsx:x:elasticloadbalancingv2:ApplicationLoadBalancer nginx created
+ │ ├─ awsx:x:elasticloadbalancingv2:ApplicationTargetGroup nginx created
+ │ │ └─ aws:elasticloadbalancingv2:TargetGroup nginx created
+ │ ├─ awsx:x:elasticloadbalancingv2:ApplicationListener nginx created
... more output trimmed ***
Outputs:
nginxEndpoint: { hostname: "********", port: 80 }
Resources:
+ 36 created
Duration: 3m19s

We have trimmed the output, since this simple program provisioned 36 AWS resources, connected them accordingly, and stood up a load balanced ECS Fargate service that automatically uses cloud deployment best practices.

Here, through the use of image: "nginx" we are letting ECS know that we want to use this publicly available Docker Hub image. ECS supports pulling these images for you by default and making them available to your services. If you’re using a public image, this works great and no further work is necessary on your part.

Using a Private ECR Repository

There are many times, however, when you or your organization may not want to use public images, such as private applications that aren’t meant to be shared with the Internet. The Docker Hub supports private images, however, if you’re already building on AWS, Amazon ECR is a valuable service that allows you to host those images in your AWS account, leveraging IAM for secure authentication, and ensuring easy, fast and secure access from your containers.

Let’s see what that looks like with Pulumi. First, we’ll create a simple Dockerfile that starts with the base nginx image and slightly modifies it to contain our own custom index.html file:

The Dockerfile:

FROM nginx
COPY content/index.html /usr/share/nginx/html

The index.html file:

<h1> Hi from Pulumi! </h1>

Now, let’s see how we’d update our Pulumi app to build this custom Docker image, push it to ECR, get the resulting image name, and reference it from our ECS service (it works the same in both ECS and EKS):

// common code from before trimmed out
const repository = new awsx.ecr.Repository("repo", {
 forceDelete: true,
});

// Invoke 'docker' to actually build the DockerFile that is in the 'app' folder relative to
// this program. Once built, push that image up to our personal ECR repo.
const image = new awsx.ecr.Image("image", {
 repositoryUrl: repository.url,
 path: `./app`,
});

const service = new awsx.ecs.FargateService("service", {
 // ... common code from before trimmed out
 taskDefinitionArgs: {
 containers: {
 service: {
 image: image,
 ...

So let’s see what happens when we actually try to run this:

As you can see Pulumi is actually launching the real docker executable locally to use the Dockerfile to build the image. As docker runs, the output is captured and automatically shown in the real-time Pulumi update display. When the image is finished building, it is pushed by docker itself to the ECR repo. Pulumi safely passes temporary repo credentials to the docker executable so it can login and push the image up. Finally, once available in ECR, the task-definition and service are appropriately updated to now reference this new image. ECS will then ensure that the old services are spun down and the new services are spun up.

TL;DR, just run pulumi up, and it takes care of all the heavy lifting. In the end we see:

In less than 30 seconds, Pulumi and Docker built the private image, made it available on ECR, and properly moved the Service over to using it. This was all done with a single command, with Pulumi smartly figuring out at the end of the day exactly what changes needed to be made. From the above we can see just the creation of the Repository components, and the updates of the Service to now use it. A nice minimal change that exactly matches our intuition around what would happen. If necessary, the awsx.ecr.Image resource can also take many more options to control what’s happening with docker. Options around tagging and caching can be configured, and the docker command line can also just be augmented if necessary to handle advanced scenarios.

Managing ECR Lifecycle Policies

In practice, an organization may be producing and uploading many images to their private ECR repositories. This can add up in costs as time goes on, especially when old images are stored that will never be used again. To aid with this, Pulumi makes it easy to set up ECR Lifecycle Policies to control the lifetime of your images and to easily purge unneeded images based on flexible criteria to meet your needs.

As a simple example, here’s a way to just remove any untagged images that are older than one week old:

// common code from before trimmed out
const repository = new awsx.ecr.Repository("repo", {
 lifeCyclePolicyArgs: {
 rules: [{
 selection: "untagged",
 maximumAgeLimit: 7,
 }],
 },
});

Now, you can keep your last two weeks of images around if you want them with all tagged images (like 'latest') being preserved. You can continue pushing rapidly to your repo without having to worry about manually going and cleaning up the stale garbage in the future.

Next Steps

We’ve shown how Pulumi Crosswalk for AWS can create a tight developer inner loop for building, publishing, and consuming Docker images, using private ECR repositories, while keeping all your ECS or EKS services and tasks updated properly and securely. This can be done with flexible policies to ensure that your repos contain the images you care about and don’t keep holding onto images you no longer need, and is all possible from a single Pulumi app with simple commands to keep everything in sync.

Pulumi is open source and free to use. For more information on Getting Started, check out:

We think there’s no easier way to do containers in a tight inner development loop, and we hope you agree!

AWS CloudWatch made easy with Pulumi Infrastructure-as-Code

Cyrus Najmabadi — Fri, 14 Jun 2019 00:00:00 +0000

Pulumi Crosswalk for AWS modules can be used to get first class insights and visualizations directly inside your Pulumi application.

As cloud applications tend to be long-lived, we think it’s vital that it be possible to get regular insights on the performance of the application at all times. Using Crosswalk for AWS Pulumi applications allow you to easily define and visualize the appropriate metrics that show the health of your services, create alarms to let you know when something is wrong, and easily create dashboards to get live visualization of what is happening in the cloud. Because this is vital to the health of the application, we think this should be something built in from the start, and not something added after the fact as an out of band artifact.

Let’s see what that means in practice! I’m going to use a recent hackathon project I created where I built a Slack bot to notify me about all the @mentions I’d received. Using this, I’d have a place I could go back to to look at, to find messages I knew had come in, but maybe I wanted to take a second glance at. The bot just sends me messages in a private channel we share like so:

I intend to add more functionality to this bot over time to help me out. If you’d like to create something similar, the code and instructions are here. The bot works by hearing about incoming events (pushed to it from Slack’s servers), looking for mentions of my name, then pushing requests back to Slack’s server to update our private channel.

Now the bot I wrote has to do something interesting due to restrictions in the Slack messaging system. Slack will only give you a few seconds to deal with a message they send you. If you don’t complete the request in a timely fashion then they’ll assume something has gone wrong, and they’ll resend. This will happen a few times before they just give up. This means that we don’t really want to try to process a message, and then also make an outgoing network call to Slack all in the same request. That outgoing call could take far too long, causing Slack to think there was a problem. So, instead, we simply receive the request, and push it right away to an SNS topic. We can then process that topic message later with another Lambda.

The skeleton of the code basically looks like this:

// Location to store the incoming slack messages to.
const messageTopic = new aws.sns.Topic("messages");
// Create an API endpoint that slack will use to push events to us with.
const endpoint = new awsx.apigateway.API("mentionbot", {
routes: [{
path: "/events",
method: "POST",
eventHandler: async (event) => {
// process the different types of messages. Push normal messages to 'messageTopic'
// ...
// Return success quickly so that Slack doesn't just immediately resend this message to us.
return { statusCode: 200, body: "" };
},
}],
});
// Hook up a lambda that will then process the topic when possible.
const subscription = messageTopic.onEvent("processTopicMessage", async ev => {
// process the messages in here. Look for mentions, then notify slack to post a message in
// right channel if we find one.
});

Overall, this is a very simple app. But how do we know if the app is actually doing well? What if something starts to go wrong and the endpoint is no longer responding in a timely fashion. How would we find out? It turns out to be pretty simple to add this functionality here! Let’s see how that would look.

First, we want to be profiling the Lambda Functions that were created by those two resources. That’s very simple to do:

const endpointFunc = endpoint.getFunction("/events");
const topicFunc = subscription.func;

Then, we want to get information about how long these functions are taking. With Pulumi’s Crosswalk APIs it’s simple to get at this information. Inside all the main “Crosswalk for AWS” modules are exposed Metrics for almost anything you might ever need. In this case, we want to know how long those Functions are taking so we can get that information as follows:

const endpointFuncDuration = awsx.lambda.metrics.duration({ function: endpointFunc });
const topicProcessDuration = awsx.lambda.metrics.duration({ function: topicFunc });

Metrics are very customizable. But for now we’ll just use the defaults returned for those functions by AWS. Now that we have the metrics, it’s simple to create an CloudWatch alarm that will alert us when things take too long:

// Notify us if takes us longer than 5 seconds to process a Slack message. But only notify if this
// happened at least once every 5 minutes in a 15 minute period. We're ok with the occasional
// random network hiccup causing delays.
const alarm = endpointFuncDuration.withUnit("Seconds")
.withPeriod(300)
.createAlarm("TooLong", {
threshold: 5,
evaluationPeriods: 3,
});

Finally, we can use both those metrics and the alarm we just created to make a great looking dashboard that we can use to get an overall view of the health of this app:

const dashboard = new awsx.cloudwatch.Dashboard("mentionbot", {
widgets: [
new awsx.cloudwatch.LineGraphMetricWidget({
width: 12,
title: "Receiving duration",
metrics: [ endpointFuncDuration.with({ extendedStatistic: 99, label: "Duration p99" }) ],
annotations: new awsx.cloudwatch.HorizontalAnnotation(alarm),
}),
new awsx.cloudwatch.LineGraphMetricWidget({
width: 12,
title: "Processing duration",
metrics: [ topicProcessDuration.with({ extendedStatistic: 99, label: "Duration p99" }) ],
}),
],
});

The Crosswalk for AWS API for dashboards gives a lot of options for building up a dashboard programmatically. There are a wealth of Widgets and Annotations to expose the full power of Cloudwatch’s dashboards to your app in a simple to use manner.

With the above Dashboard code we get this useful chart:

We can see that things look healthy, and the chart shows us both the information about our p99 latencies, as well as the threshold point that would cause our alarm to trigger.

With these facilities I now have an app that does what I need, but also directly exposes mechanisms to track and get insights to its health. What’s great is that if I now change my app (for example, in a way that causes new Lambda Functions to be created), I don’t need to do anything to keep these alarms and dashboards working. Because they’re directly referencing the real Cloud resources they will be kept up to date automatically as my app grows and changes over time. That’s really powerful, and much simpler than having to manage all these different parts independently!

We hope these new Crosswalk for AWS APIs will be just as useful for you! For more information on Pulumi Crosswalk for AWS checkout other related content:

Simple Serverless programming with Google Cloud Functions

Cyrus Najmabadi — Wed, 10 Apr 2019 00:00:00 +0000

Pulumi lets you create, deploy, and manage Google Cloud applications and infrastructure in familiar languages like JavaScript, TypeScript, and Python, and without needing to learn new DSLs or YAML templating solutions. This means great productivity and getting to use tools you already know and love. Since serverless is all about focusing more on your application code, and less on infrastructure and configuration toil, we absolutely love Google Functions.

The Simplest Way to Serverless

In fact, serverless has never been simpler than it is when you combine Pulumi with Google Cloud Functions. Want to serve a simple HTTP API with no fixed costs? It’s just a few lines of code – and no, we’re not hiding any YAML here:

import * as gcp from "@pulumi/gcp";

let greeting = new gcp.cloudfunctions.HttpCallbackFunction("greeting", (req, res) => {
    // Change this code to fit your needs!
    res.send(`Greetings from ${req.body.name || 'Google Cloud Functions'}!`);
});

export const url = greeting.httpsTriggerUrl;

Thanks to Pulumi’s language heritage, we know how to take that function, serialize it, and publish it to a Google Bucket and Object, eliminating those tedious manual steps behind a single pulumi up CLI invocation.

Or perhaps a pubsub topic that runs some custom code on every message received. This sounds like a perfect use case for an event-driven style of programming, so why don’t we write it that way!

// Create a PubSub Topic
let requests = new gcp.pubsub.Topic("requests");
requests.onMessagePublished("newMessage", (data) => {
    // Print out a log message for every message on the Topic.
    // Change this code to fit your needs!
    console.log(Buffer.from(data.data, "base64").toString());
});

Because Pulumi knows how to manage infrastructure as code, it can provision and manage not only the Google Cloud Function, but also the pubsub topic resource itself, for each creation and management.

Or perhaps, we’d like a function that respond to any uploads of new objects to your storage bucket:

// Create a Storage Bucket
let bucket = new gcp.storage.Bucket("data");
bucket.onObjectFinalized("newobject", (data) => {
    // Print out a log message for every upload to the Bucket.
    // Change this code to fit your needs!
    console.log(`New file uploaded: ${data.name}`);
});

A Complete Google Functions Slack Bot

For an idea of how you might fit these together to make a real-world cloud application, let’s look a simple skeleton structure for a Slack Bot using Pulumi and Google Cloud Functions:

// secure config tokens to use to validate incoming messages as well as authenticate ourself to slack
const config = new pulumi.Config("mentionbot");
const slackToken = config.get("slackToken");
const verificationToken = config.get("verificationToken");

// A topic that we can enqueue slack events to so they can be processed in batch later on
const messageTopic = new gcp.pubsub.Topic("messages");

// Create an http endpoint that slack will use to push events to us.
const endpoint = new gcp.cloudfunctions.HttpCallbackFunction("bot", {
    callbackFactory: () => {
        const app = express();
        app.use(bodyParser.json());
        app.post("/events", (req, res) => {
            // Importantly: This is the code that will run in your serverless GCP cloud function!

            const body = req.body;

            // Process the body as appropriate. If it's something we need to respond to immediately
            // (like a verification request), then do so. Otherwise, add the message to our pubsub
            // topic to be processed later:
            const pubSub = new PubSub();
            const topic = pubSub.topic(messageTopic.name.get());
            topic.publish(Buffer.from(JSON.stringify(body)));

            // Quickly respond with success so that slack doesn't retry.
            res.status(200).end();
        });

        return app;
    }
});

messageTopic.onMessagePublished("processTopicMessage", async (data) => {
    // Actually handle the 'data' in the pubsub message.
    // Importantly: This is the code that will run in your serverless GCP cloud function!
});

// Give this url to slack to let them know where to post their events to.
export const url = endpoint.httpsTriggerUrl;

This is the real code for a complete SlackBot application running on GCP, from the cloud resources to the serverless code, all within a unified Pulumi application! Customizing this for your own use case is as simple as changing the code in the two JavaScript arrow-functions.

To see the complete project, take a look at our @mentionbot example. That example will listen for mentions of your name and will notify you of them in a channel of your choosing, giving you you a persistent timeline you can go back to look at to make sure you can find all these messages.

Although it’s a simple example, there are a lot of moving parts this takes care of that you would normally be responsible for. This includes:

Figuring out the shape (the input/output-types) for your Cloud Functions, and then creating an appropriate program that exports the right entrypoint that matches. In this case, because we’ve exposed the right abstractions (like HttpCallbackFunction and Topic.onMessagePublished), the arrow-functions you pass in will all have the right types, and your program will be typechecked by TypeScript.
Creating separate Cloud Functions for each serverless callback. One for listening and responding to the initial Slack events and the second for processing the messages in the Topic. Here, you can write a single Pulumi App where all the code can be placed how you like it (in this case in a single file).
Packaging each callback up in the appropriate structure Cloud Functions expects, including how to get all your dependencies in place.
Creating a Storage Bucket for all of your packaged programs to live in.
Uploading each packaged program to a Bucket Object in that bucket.
Configuring the appropriate triggers on your Cloud Functions stating how they should be triggered (for example, in response to an HTTP trigger or a Pub/Sub trigger).
Including the right information in the function so you can interact with your other cloud resources in the Pulumi App. Without this, you would need to find a way to include that data in each Cloud Function’s environment variables (or just hardcode them in ‘1’) so that your program can access the rest of your cloud infrastructure. In the above example, you can see how you can just reference your resources directly (like the PubSub Topic) directly from your Cloud Function callback. Pulumi makes sure this all works, and that the data you use is available in that Cloud Function when it finally is triggered.
Figuring out a safe and secure way to encode and access secrets for your Cloud Function. Here, we can use Pulumi’s Config Secrets to safely encrypt and manage secrets for your Cloud Function code.

Not to mention that by doing all of that, you can achieve continuous deployment Pulumi and Google Cloud Build.

Updating Your Google Functions Code

This is a lot to figure out to manage cloud applications and infrastructure. If you want to tweak things even slightly you might need to go make many manual changes and updates to ensure everything is properly updated. With Pulumi, all this complexity is handled with a single update! Just run pulumi up , and it figures out the rest.

Let’s just take a look at what Pulumi does when you tweak a few things in the program:

$ pulumi up
Updating (slackbot):
     Type Name Status Info
     pulumi:pulumi:Stack slackbot
 +- ├─ gcp:pubsub:Topic messages replaced [diff: +labels]
     │ └─ gcp:cloudfunctions:CallbackFunction processTopicMessage
 +- │ ├─ gcp:storage:BucketObject processTopicMessage replaced [diff: ~name,source]
 ~ │ └─ gcp:cloudfunctions:Function processTopicMessage updated [diff: ~eventTrigger,sourceArchiveObject]
     └─ gcp:cloudfunctions:CallbackFunction mentionbot
 +- ├─ gcp:storage:BucketObject mentionbot replaced [diff: ~name,source]
 ~ └─ gcp:cloudfunctions:Function mentionbot updated [diff: ~sourceArchiveObject]
Outputs:
    url: "https://***.cloudfunctions.net/mentionbot"
Resources:
    ~ 2 updated
    +-3 replaced
    2 changes. 5 unchanged
Duration: 52s

One command later, and your entire stack is updated properly!

Winding Down

The cloud provides tremendous potential, and we want to make it easy for developers to tap into that potential. Using Pulumi, it’s easy to blur the line between business logic and serverless resources, bringing the focus back to the problems you care about. One codebase that’s easy to create, update, and maintain.

To check things out, get started today:

PS: If you’re interested in how Pulumi manages to take a JavaScript/TypeScript => function and somehow analyze and transform it into a form that Cloud Functions can use, please see our deep dive on this topic in: Lambdas as Lambdas: The magic of simple serverless Functions. We’re leveraging the same great programming language and analysis framework to power our GCP solution here.

Simplified Outputs in Pulumi 0.17

Cyrus Najmabadi — Tue, 19 Mar 2019 00:00:00 +0000

This blog post refers to an outdated version of Pulumi (0.17). For current information about Outputs and the latest features, please refer to the Inputs and Outputs documentation.

Pulumi allows cloud developers to use programming languages like JavaScript, TypeScript and Python to define and deploy cloud infrastructure and applications. To do this, Pulumi exposes a notion of Outputs that track how the outputs of one cloud resource are used and transformed as part of creating another cloud resource.

These Output types are heavily used in many Pulumi application. They are the way that Resources expose their values and are commonly used to pass values from one Resource to another. Outputs are also a key part of how Pulumi tracks dependencies between resources. In fact, Outputs are similar to promises/futures that you may be familiar with from other programming models but also carry along dependency information.

We’ve heard a lot of feedback about how Outputs can lead otherwise-simple code to become hard to write and reason about. So we’ve looked into what we can do to significantly simplify the experience working with Outputs - making the user experience simpler while maintaining the rich dependency tracking and type checking that Pulumi has always provided for cloud infrastructure.

Simplifying Outputs

In 0.17.0 of the @pulumi/pulumi package, we’ve made most common patterns for working with Outputs much simpler.

Prior to this release of @pulumi/pulumi package, it was fairly common to have to write code like the following:

const cert = new aws.acm.Certificate("cert", {
 domainName: "example.com",
 validationMethod: "DNS",
});

const certValidation = new aws.route53.Record("cert_validation", {
 records: [cert.domainValidationOptions.apply(domainValidationOptions => domainValidationOptions[0].resourceRecordValue)],
 ttl: 60,
 type: cert.domainValidationOptions.apply(domainValidationOptions => domainValidationOptions[0].resourceRecordType),
});

In particular, creating the aws.route53.Record involves a fair amount of complexity with those arrow functions. i.e.:

 records: [cert.domainValidationOptions.apply(domainValidationOptions => domainValidationOptions[0].resourceRecordValue)],
 ttl: 60,
 type: cert.domainValidationOptions.apply(domainValidationOptions => domainValidationOptions[0].resourceRecordType),
 zoneId: zone.apply(zone => zone.id),

Yikes! This is so verbose, it doesn’t even fit on the width of the page!

The idea of the .apply function is similar to Promise.then. It allows one to pass a piece of code that will be applied to the underlying value once it becomes available (after the corresponding cloud resource is created or updated) and will return an Output that then points to the transformed underlying value. Importantly, the new Output will still track dependency information properly. In the above example the aws.route53.Record will know that it depends on cert, even though the Certificate resource is not itself passed directly do the constructor.

Unfortunately, while .apply gives a lot of power and flexibility, it is also somewhat verbose and clunky for describing such a simple concept. Fortunately, we found a way to improve the situation greatly. This realization came about from great work done in our Python package. First, before diving into the low level details, let’s first see what the above code would now look like in 0.17.0:

const certValidation = new aws.route53.Record("cert_validation", {
 records: [cert.domainValidationOptions[0].resourceRecordValue],
 ttl: 60,
 type: cert.domainValidationOptions[0].resourceRecordType,
 zoneId: zone.id,
});

That’s a lot nicer than before! The following improvements happened:

There’s no more callbacks!
There’s no need for a repetitive lambda parameter (which might conflict with some other name in scope).
The code is just pure simple idiomatic JavaScript/TypeScript that clearly conveys its intent.

Importantly, no information has been lost here. The exact same dependency information flows along here like it did before. And, thanks to TypeScript’s flexible type system, the above is totally typesafe and will let you know the right types of things and will still error if you happen to make mistakes.

So, how was this done? Well, the core part of the change is thanks to a little-known feature of JavaScript: Proxies.

JavaScript proxies allow libraries to return objects that intercept and override many facets of how JavaScript works at runtime. There are many things a Proxy lets you control, however for our needs the most important bit was that it allows you to override how member-lookup works. In other words, when the code contains someProxy.someMember the proxy actually gets a chance to determine what should happen and what someMember should actually return! With that flexible interception point available, we actually took our core Output type and made it into a Proxy. We then added the right interception code so that if the code contains someOutput.someMember that that gets translated exactly into someOutput.apply(o => o.someMember).

This also works just fine for array-accesses (which are just property-lookups from JavaScript’s perspective). In other words, at runtime, the member-lookup form and the .apply form will be equivalent (except that the former is so much nicer to write!). This is why, for example, certCertificate.domainValidationOptions[0].resourceRecordValue will have the correct value (with all the right dependency information). At runtime it really will be equivalent to the original certCertificate.apply(certCertificate => certCertificate.domainValidationOptions[0].resourceRecordValue) form.

Now, while this was fairly easy to get working at runtime from a JavaScript perspective, it was a little more challenging to figure out how to make this work in TypeScript’s typing system. For example, if you had a value like so:

const cert: Output<{ domainValidationOptions: pulumi.Output<{ domainName: string, resourceRecordName: string, resourceRecordType: string, resourceRecordValue: string }[]> }>;
const firstOption: Output<{ domainName: string, resourceRecordName: string, resourceRecordType: string, resourceRecordValue: string }> = cert[0];
const domainName = firstOption.domainName;

Then how does TypeScript know that cert should have a property on it called domainValidationOptions? And how can it know that domainValidationOptions can be indexed into? And how would it know that once indexed, that Output would have a domainName property? Clearly, these are all Outputs. Yet, each Output<...> has a different set of properties exposed off of it!

To do this required taking advantage of some very interesting and advanced parts of TypeScript’s type system. If this is a part of TypeScript that interests you, or you just want to see how we did this, you can dive in deep into the source here! And, if you want to see how we did the actual runtime Proxy work, that code is self-contained here. Thanks to TypeScript’s advanced type-system, these types can be properly expressed and the entire tooling ecosystem understands them. For example, in VSCode, if you try to write code like the above, you’ll see all the expected properties with the expected types:

Conclusion

We definitely hope these changes to @pulumi/pulumi in 0.17.0 will make the programming experience simpler and smoother for many common use cases. And, if you’ve ever wanted to do some fancy tricks like what we’re doing here, these updates can show you how you too can approach some of these advanced techniques for both JavaScript and TypeScript!

Serverless on AWS with Pulumi: Simple, Event-based Functions

Cyrus Najmabadi — Mon, 14 Jan 2019 00:00:00 +0000

One of Pulumi’s goals is to provide the simplest way possible to do serverless programming on AWS by enabling you to create cloud infrastructure with familiar programming languages that you are already using today. We believe that the existing constructs already present in these languages, like flow control, inheritance, composition, and so on, provide the right abstractions to effectively build up infrastructure in a simple and familiar way.

In a previous post we focused on how Pulumi could allow you to simply create an AWS Lambda out of your own JavaScript function. While this was much easier than having to manually create a Lambda Deployment Package yourself, it could still be overly complex to integrate these Lambdas into complete serverless application.

To get a sense of that complexity, let’s look at how one would normally have to work with AWS’s resource system to create a simple Serverless application:

// Simplified for brevity
import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Create a lambda that will post a message to slack when the bucket changes.
// We can pass a simple JavaScript/TypeScript lambda here thanks to the magic of "Lambdas as Lambdas"
// See: www.pulumi.com/blog/lambdas-as-lambdas-the-magic-of-simple-serverless-functions/
const lambda = new aws.lambda.CallbackFunction("postToSlack", {
 callback: async (e) => {
 const client = new slack.WebClient(...);
 for (const rec of e.Records) {
 await client.chat.postMessage({ ... });
 }
 },
 ...
});

// Give the bucket permission to invoke the lambda.
const permission = new aws.lambda.Permission("invokelambda", {
 function: lambda,
 action: "lambda:InvokeFunction",
 principal: "s3.amazonaws.com", sourceArn: bucket.id.apply(bucketName => `arn:aws:s3:::${bucketName}`), }));

// Now hookup a notification that will trigger the lambda when any object is created in the bucket.
const notification = new aws.s3.BucketNotification("onAnyObjectCreated", {
 bucket: bucket.id,
 lambdaFunctions: [{
 events: ["s3:ObjectCreated:*"],
 lambdaFunctionArn: lambda.arn,
 }],
})

Phew… that’s a lot of code :-/ So what happened above? Well, in the AWS resource-oriented view of the world, most things are nouns (i.e objects). Your bucket is an object. The lambda is an object. The permission is an explicit object. And even the connection between the bucket and lambda (the BucketNotification) is itself an object. That’s one way to decompose the world, and it makes sense in some domains where everything is just ‘data’. But that isn’t a natural fit for a programming language, and it just doesn’t feel very idiomatic or appropriate for how people would expect things to work in their programming language of choice. Programming languages have lots of other facilities at their disposal to work with data. You can have methods/functions/messages you can send to an object. You can have events and callbacks on objects. You can encapsulate objects behind other objects. You can ‘run code’ to do things based on those objects, etc. etc.

Now, while data should be the start of how you interact with system, it shouldn’t be the end. Toward that goal, Pulumi makes it possible to simply compose and connect these resources in a more natural fashion. By just adding our own components, and leveraging what programming languages can already do, we can make it possible to simplify the above down to:

import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Create a lambda that will post a message to slack when the bucket changes.
bucket.onObjectCreated("postToSlack", async (e) => {
 const client = new slack.WebClient(...);
 for (const rec of e.Records) {
 await client.chat.postMessage({ ... });
 }
});

This now feels far more like how one might expect to express this concept with a normal application. A simple conceptual idea now maps to a simple code pattern. Instead of manually creating objects to represent the connection between the Bucket and the Lambda, we simply directly ask the Bucket to call the Lambda when the ObjectCreated event fires. Because this is all ‘code’, we can take care of all the boring cruft (like creating permissions) on your behalf. Of course, if you need to tweak this, that’s still possible. We believe you should always be in control of what’s going on. The actual low level details of how that is done are interesting as well, but a followup post will explain how we did it (and how you can do it too!). If you want to create your own composable modules for making cloud deployments easier then definitely look for that post!

Now, while it’s how the above examples worked, it’s also worth noting that the use of a JavaScript function for the AWS Lambda is not required. You can hook up a serverless event to call an AWS Lambda you create just by using new aws.lambda.Function. Or, you can get a reference to an existing AWS Lambda created outside of Pulumi and have that be the receiver of your serverless event. Here’s how that would look:

import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Retrieve an existing Lambda Function already created in your AWS infrastructure.
const func = aws.lambda.Function.get("postToSlack", "... existing lambda arn ...");

// Call that Lambda Function when an object is created in the bucket.
bucket.onObjectCreated("postToSlack", func);

We’ve tried to make it this simple to hook up many interesting AWS serverless events. For example, you can register to hear about events on S3 Buckets, SNS Topics, SQS Queues, Kinesis Streams, DynamoDB Tables, Cloudwatch Events, and more. We’ve tried to provide these prebuilt components for the most common and interesting cases. If there’s some serverless event we haven’t added support for that you want to use, definitely let us know!

Finally, because we think it’s great to be able to use a JavaScript or TypeScript function as the code for your Lambda, we’ve introduced a simple quality-of-life improvement for writing those functions. Specifically, because it’s so common that one will use the real AWS SDK for Node.js within a Lambda, we’ve provided a quick and simple way to get to a fully initialized instance of that API without needing any additional requires or imports in your code. Here’s how that would look:

// The only module you need to import.
import * as aws from "@pulumi/aws";

// ...

bucket.onObjectCreated("postToSlack", async (e) => {
 // direct access to the aws-sdk through `aws.sdk`.
 const sqs = new aws.sdk.SQS();
 sqs.sendMessage(/*...*/);
});

This allows you to use @pulumi/aws as both the deployment-time API to define your AWS infrastructure, as well as being the run-time API that you can use when your Lambda executes to access the full set of AWS functionality.

To get a sense of a more complete example of how this all ties together you can see many of our examples. One example that helps demonstrate many of the above concept is Twitter+Athena. In it, we use serverless events to define a schedule, a callback to invoke when the even t fires, use of a third-party twitter API to retrieve a set of relevant Tweets, use of the aws-sdk to store the Tweet information into an S3 Bucket, and finally the creation of an Athena Query to extract information from the information stored into the bucket. You could augment this example with any number of resources and events for all sorts of interesting use cases.

As we’ve seen, with a tiny amount of code, you can easily define the cloud resources you want to create (or reference your existing cloud resources). It’s then simple to add code to respond to events and call new or existing Lambdas. And, if you use a JavaScript function to create your Lambda, you can reference those resources along with the AWS SDK easily, all within the same application code!

Lambdas as Lambdas: The magic of simple serverless Functions

Cyrus Najmabadi — Wed, 10 Oct 2018 00:00:00 +0000

Pulumi’s approach to infrastructure as code uses familiar languages instead of YAML or DSLs. One major advantage of this approach is that AWS Lambdas, Azure Functions, Google Cloud Functions, et al. can just be real lambdas in your favorite language, offering a flexible and simple path to serverless. Such functions behave as normal functions, allowing you to treat serverless code as part of your application instead of separate “infrastructure” that needs to be configured, managed, and versioned manually. In this post, we’ll examine this capability in JavaScript, which is already very function- and callback-oriented, making serverless feel like a natural extension of the language we already know and love.

While Functions as a Service (FaaS) systems have become more popular, getting up and running can still feel overly complex compared to normal application development. FaaS offerings today divide the development experience between “infrastructure” – doing all the work to configure the Lambda runtime itself (i.e. how much memory to use, what environment variables should be present, etc.) – and writing and maintaining the code that will execute in the function itself when triggered. Most developers just want to focus on the latter, write some code, and have it work.

For AWS Lambda, for example, the code itself needs to written and packaged up with all its dependencies into a single zip-file, stored as a blob that is then placed in AWS S3. This blob is then referenced by Lambda so that it can be loaded on-demand when necessary. Updating just a single character of this code, like fixing a bug, then requires fully repackaging and re-uploading everything. Compounding all of that, this code is necessary independent of any other pieces of your application. If the code for the Lambda needs to read or write to an S3 bucket, publish to an AWS SNS topic, etc., it will need to be manually configured to have that bucket’s information passed to it, often using out of band in environment variables. These same issues also apply when creating FunctionApps using Node.js on Azure.

At Pulumi, we thought there could be a better way to do this. Why have to jump out of your application code just to create another bit of configuration (in YAML or JSON) just to run your application? Put simply, we wanted developers to write event-driven code, in their favorite language and in their favorite IDE, and simply hit “deploy” when done.

Magic Functions

From a desire to do better here, the “closure serialized function” was born. However, “closure serialized function” is a geeky mouthful, so I like just calling them “magic functions.” Our team is comprised of cloud, language, and runtime geeks alike, so we figured out a way to hook into the language runtimes to just make lambdas as lambdas work.

Before diving more deeply into how magic functions work, let’s take a look at how they look in action:

// Example simplified to not get too bogged down :)
const sharp = require("sharp");
const videoBucket = new aws.s3.Bucket("videos");
const videoTopic = new cloud.Topic("videoTopic");
videoBucket.onObjectCreated("onNewVideo", async (event) => {
for (const record of event.Records) {
const dimensions = computeDimensions(record.s3.object.key);
videoTopic.publish({ s3ObjectKey: record.s3.object.key, time: record.eventTime });
}
});
function computeDimensions(objectKey: string): { height: number, width: number } {
/* simplified */
const metadata = await sharp(...);
return { height: metadata.height, width: metadata.width };
}
videoTopic.subscribe("compressVideo", async (event) => {
const { s3ObjectKey, time } = event;
// actually compress video.
});
videoTopic.subscribe("createThumbnail", async (event) => {
const { s3ObjectKey, time } = event;
// actually create thumbnail.
});

Let’s still walk through it to see what’s happening. It can be a little subtle because we are mixing deployment-time code – code that declares infrastructure as code resources – and *runtime *code – application code that runs as a function.

First, we’re just defining two simple resources: 1) an s3.Bucket, where we expect new video files to be uploaded to, and 2) a cloud.Topic that will inform interested parties when new videos are uploaded. (The full surface area of AWS is available in the aws package, and the cloud package offers multi-cloud abstractions that work at a higher level of abstraction.) Right after defining the resources, we start creating our first FaaS resources. videoBucket has an onObjectCreated event subscription function that will fire when new objects appear, and we invoke it with a real JavaScript (or TypeScript) arrow-function defining the code that will execute in an AWS Lambda at runtime.

Not So Fast…

This code is pretty magical in a few ways. To see how it’s magic, let’s consider a fairly naive way to create a Lambda for this onObjectCreated event handler function – something we need to do by hand with FaaS solutions today. Imagine we took the raw source text of this arrow-function by toStringing it and using that as the text of the Lambda code:

// index.js
module.exports = async (event) => {
foreach (var record in event.Records) {
const dimensions = computeDimensions(record.s3.object.key);
videoTopic.publish({ s3ObjectKey: record.s3.object.key, time: record.eventTime });
}
};

Now, the above translation could have worked for an extremely basic function, but totally breaks down here for several reasons. First, computeDimensions isn’t included, which will make this immediately fail if ever called. Second, if computeDimensions was included, it wouldn’t work unless the sharp module was properly imported for it. Third, even if the module was imported in the code, the actual NPM package would have to be available along with the code in the cloud. Fourth, if that module was properly required, and the package was properly included, the code is both referencing videoTopic (a resource defined when the application originally ran), and it is invoking a helper method on it. Remember, this code is going to be executing at cloud runtime, not at the time when Pulumi is actually executing the application on your developer machine, and actually doing things like running the const videoTopic = new cloud.Topic("videoTopic") code.

To summarize, we have quickly realized, we can’t just analyze everything locally to figure out what has to be included, instead the entire transitive closure must be considered – a task usually left up to FaaS developers to perform manually! This coupled with the manual configuration using YAML can lead to plenty a headache.

Compilers to the Rescue

Despite all those issues, Pulumi is able to make that original application code work and it is able to magically effectively translate the above into a working Lambda. The real secret sauce here is that, by leveraging familiar languages, we can stand on the shoulders of all the great compiler work underpinning them.

So let’s dig in – how does this all really work?

(Note that, if you want some of the more nitty gritty details, we have a doc going in depth. For now, we’ll try to stick to providing a high-level overview of what’s going on, so that we can grok the basic approach.)

Pulumi executes your application as a normal Node.js application. As such, when it hits the const videoBucket = new aws.s3.Bucket("videos") lines, it actually goes and creates those resources. When it then hits the videoBucket.onObjectCreated("onNewVideo", async (event) => { line this executes a normal line of JavaScript. Namely, onObjectCreated is a normal JavaScript function that will be passed the name parameter as the first arg, and the JavaScript function as the second arg. onObjectCreated eventually bottoms out with a call to a call to pulumi.runtime.serializeFunction. This function is the root of where our deep analysis of your JavaScript function analysis and translation will happen.

It’s worth pausing here. We can use all the standard approaches to function composition to build bigger functions out of smaller functions, even before we hit that serialization routine. The closures are built up in the V8 runtime as usual, and are eagerly awaiting the transformation into magic functions as soon as they are needed. Pretty cool!

Once we get to that point, using the awesome analysis APIs provided by TypeScript, we start introspecting the function instance, seeing what code it contains, and what external objects it references. The legal objects that can be included in the captured function’s environment include (but aren’t limited) to things like simple JavaScript objects, other functions (like computeDimensions), modules (like sharp), other Pulumi resources (like videoTopic), and even complex beasts like Promises. Basically, anything that Pulumi knows how to serialize and rehydrate inside the Lambda.

When Pulumi determines that these objects are referenced, it automatically includes enough information about them into the final translated Lambda code file so that they are available and can be effectively used at runtime. Thanks to JavaScript’s deep, dynamic introspection capabilities, as well as its rich APIs for creating nearly anything at runtime, it’s very possible to effectively hydrate out all the information present when Pulumi runs, and put it into a form that can be cheaply and accurately rehydrated into runtime objects again when the Lambda runs.

This analysis also understands the NPM package structure of an application and its dependencies, ensuring that necessary imported packages are properly included with the final Lambda. This analysis is also very smart about not including things that aren’t necessary at all. For example, it might drop information from the serialized form if it can prove that would never be observable, in addition to relying on built-in runtime functions being there without needing any sort of serialization. There are definite subtleties and interesting cases that can arise here, and for those very curious, reading the documentation linked above may be very enlightening.

Pulumi also understands diffs and dependencies, so updating is a cinch. Any changes to the code in the JavaScript function (including anything it depends on) is properly tracked and understood by Pulumi, allowing an easy development model where FaaS code can be easily modified and republished without jumping through extra hoops.

The Result? Simple serverless Functions

With this approach, serverless programming becomes dramatically simpler, whether it’s an AWS Lambda, Azure Function, or GCP Cloud Function. (Stay tuned, we’re hoping to bring this to Kubernetes FaaS offerings soon.)

In short, you just write event-driven JavaScript code that’s familiar and easy, you run pulumi up, and everything is taken care of from there. No YAML configuration files, no point and clicking in a cloud console UI, no fuss.

On top of all of that, because this is just code itself, and because these are normal APIs, other tooling (like your editor) knows exactly what is going on and can guide you through this. For example:

While there’s a lot of really cool tech going on under the covers here, we really hope that this ends up just feeling super-awesome to use for you. Now that we have this functionality, we wouldn’t want to ever write an AWS Lambda any other way!

Get started building serverless functions on Pulumi today!

Running a Serverless Node.js HTTP Server on AWS and Azure

Cyrus Najmabadi — Tue, 02 Oct 2018 00:00:00 +0000

The newly introduced cloud.HttpServer in Pulumi makes it easy to serve a standard Node.js HTTP server as a serverless API on any cloud platform. This new API brings together the flexibility and rich ecosystem of Node.js HTTP servers, the cost and operational simplicity of serverless APIs, and the multi-cloud authoring and deployment of Pulumi. In this post, we walk through some of the background on why we introduced this new API and how it fits into the Node.js HTTP ecosystem.

Almost 10 years ago, Node.js was introduced, helping to usher in an era of server-side JavaScript development. From the beginning, Node.js made it simple and easy to stand up an HTTP server and to start listening and responding to requests in an environment that felt natural to people already comfortable working with JavaScript. From the earliest commits this was a core focus, and from early on Node.js exposed the initial JavaScript API for creating an HTTP Server. That API evolved over time to become the well known HTTP module that is the backbone of so many projects.

The approach taken by Node.js was to provide an API, http.createServer, to easily stand up a reliable and performant HTTP server, but it left the processing and control of those actual HTTP conversations up to a simple callback function that a user of the API would provide. This callback function itself was very simple, with the entire API having the form:

http.createServer((req, res) => ...);

With this a developer could now easily plug in a function that would be called with the information about an incoming request in the callback’s first parameter, and which could supply the appropriate response through the callback’s second parameter. This approach allowed Node.js to focus on being simple and easy to get the server stood up, while being relatively unopinionated giving the developer a lot of flexibility in terms of how they would actually process requests and responses.

Thanks to this split, the Node.js ecosystem then came forward to provide more structured design patterns to handle those requests and responses to make things more manageable for developers. These communities provided ‘middleware’ APIs that could bridge between this extremely basic entry-point into one more structured and opinionated, but simpler to understand and work with. Middleware libraries like Express.js allowed one to write simple route-handlers like:

const app = express();
// GET method route
app.get('/', function (req, res) {
res.send('GET request to the homepage')
})
// POST method route
app.post('/', function (req, res) {
res.send('POST request to the homepage')
})
http.createServer(app).listen();

These route-handlers also have extensible middleware points to plug in other libraries (like Body Parser, Passport, and many others). This large effort across the entire ecosystem has made the task of serving websites using JavaScript dramatically simpler, and has helped propel a huge amount of popularity in this space.

Interestingly enough though, the Cloud space has gone a slightly different route here (pun intended). While cloud providers like AWS and Azure provide ways to both create HTTP servers and to use Node.js, they offer a very different shape altogether for processing HTTP messages. For example, in AWS, one needs to first set up an API gateway. That gateway will then have ‘methods’ defined on it which point at ‘Lambda Proxies’ to finally process the request. The connected AWS Lambda then will manage the request and response in a decidedly AWS-specific manner. For example, instead of working with IncomingMessage and ServerResponse like one normally would with Node.js, AWS specific messages like so are the required. Responding to messages with your own data uses a very different form as well. Code has to be written like so:

exports.handler = function(event, context, callback) {
 // process AWS 'event' using information from AWS 'context'
 // ...
 // build response
 var response = {
 statusCode: responseCode,
 headers: {
 "x-custom-header" : "my custom header value"
 },
 body: JSON.stringify(responseBody)
 };

 callback(null, response);
}

Azure follows its own distinct style as well for processing HTTP messages. These approaches end up working, but often feel decidedly un-Node.js-like. They’re full of cloud-provider-specific values and behaviors, and they end up making it more difficult to work with the existing ecosystem of middleware components out there. (Note that Google Cloud HTTP Functions actually do natively support the Node.js request/response pattern!).

We thought this was a place we could definitely help Pulumi applications with a simpler approach for these cloud ecosystems. To further this goal, we’ve created a new API called cloud.HttpServer. HttpServer is a Pulumi Resource, but is designed to work well with the existing large middleware ecosystem out there. And critically, the same HttpServer API can be implemented consistently on AWS, Azure and GCP - so you can write once and deploy to any cloud. The core API shape that accomplishes this is:

// Factory function for creating a requestListener function. The returned function is the same
// callback function that would be passed to http.createServer. See:
// https://nodejs.org/api/http.html#http_http_createserver_options_requestlistener for more details.
export type RequestListenerFactory = () => (req: http.IncomingMessage, res: http.ServerResponse) => void;

export interface HttpServerConstructor {
 /** * @param createRequestListener Function that, when called, will produce the [[requestListener]] * function that will be called for each http request to the server. The function will be * called once when the module is loaded. As such, it is a suitable place for expensive * computation (like setting up a set of routes). The function returned can then utilize the * results of that computation. */
 new (name: string, createRequestListener: RequestListenerFactory, opts?: pulumi.ResourceOptions): HttpServer;
}

The idea here is that a Pulumi app can now simply take the previous Express.js example and rewrite it as:

// The `() => { ... }` callback will actually become the code inside an AWS Lambda!**
const server = new cloud.HttpServer("myserver", () => {
 const app = express();

 // GET method route
 app.get('/', function (req, res) {
 res.send('GET request to the homepage')
 })

 // POST method route
 app.post('/', function (req, res) {
 res.send('POST request to the homepage')
 })

 return app;
});

The majority of this code is identical to the original Node.js form. The only small difference is the creation of the cloud.HttpServer resource (which is needed for Pulumi to track work and dependencies) and the returning of the ‘app’ variable instead of directly calling http.createServer.

Pulumi will take that code and convert the JavaScript callback^**^ into all the cloud-provider-specific resources necessary to make this work. For example, on AWS, an API Gateway and Lambda. The API Gateway will be properly setup to communicate with the Lambda function, and the Lambda function will properly handle AWS’s specific input/output forms and will map them along so they can properly work with Node.js’ http library expectations.

Effectively, the above callback will get translated into the following code that is uploaded to AWS Lambda.

// index.js
module.exports = WrapAwsEventAndContextAndCallBackAndForwardTo((() => {
 const app = express();

 // GET method route
 app.get('/', function (req, res) {
 res.send('GET request to the homepage')
 })

 // POST method route
 app.post('/', function (req, res) {
 res.send('POST request to the homepage')
 })

 return app;
})())

In other words, that factory function will get called, returning the expected requestListener function. Pulumi will then inject the call to the helper that will wrap that function with the right translation code to map from the AWS specific inputs to the expected Node.js form, and from the Node.js outputs to the AWS expected forms.

Right now, this functionality has been enabled for AWS and Azure in our @pulumi/cloud-aws and @pulumi/cloud-azure packages (support for GCP is on the roadmap!). This means that the above code will run properly on either platform without consumers having to write platform specific code or figure out how they would need to translate between the two. And, because of the translation from AWS/Azure specific APIs to the common Node.js API, you can now use virtually any middleware components effectively on either platform. Currently, this only works if you are writing a Pulumi application. However, we are tracking the work to extract the specialized code we’ve written into its own library. Once that work is done, you’ll then be able to easily do something like create an Azure FunctionApp manually, setup the appropriate bindings, and then upload your code which will setup the appropriate Azure endpoint, but then call into this library to translate to the Node.js http form that will allow you to then code things up as simply as if this was just an Express.js app!

^**^ One thing glossed over was how Pulumi knows how to convert a JavaScript callback into an AWS Lambda or an Azure FunctionApp. Stay tuned for more on this subject! It’s an incredibly cool part of Pulumi, and we can’t wait to dive deeper into that functionality and how it can make it simple and clear to create an entire Cloud App in one single package, even including the code in-line that will end up executing in the cloud at runtime! That magic, along with powerful components like the new HttpServer API can help make cloud applications dramatically simpler to write and maintain. Happy coding!

You can dig in to serverless coding with Pulumi here, and join us on Wednesday 3rd October at 11am PDT to hear more about serverless programming with Pulumi on our YouTube live stream.

Advanced TypeScript type FTW!

Cyrus Najmabadi — Wed, 19 Sep 2018 00:00:00 +0000

We at Pulumi love TypeScript for cloud apps and infrastructure, because of its rich type system and great ahead-of-time typechecking – making for a more productive inner loop and helping to find errors sooner. The typesystem magic behind how this works for infrastructure as code can be fascinating!

A core part of the Pulumi programming model is that we allow people to express complex dependency data that may eventually be available. Traditional JavaScript programming might expose that as a Promise, but we’ve taken that one step further by introducing a type we call:

type Input<T> = T | Promise<T> | Output<T>;
// Like a Promise<T>, but also keeps track of dependency information so we can figure
// out what the upstream/downstream impact is whenever any values change in the system
interface Output<T> { /* ... */ }

This complex type can often show up as inputs to all Pulumi objects like so:

interface VpcInput {
ipAddresses: Input<Input<string>[]>;
}

Breaking that down, it’s basically saying “I’ll take an array of strings. But those actual string values can individually be strings, promises-of-strings, or strings-carrying-dependency-information.” It even goes one step beyond that, because the entire array itself could be just an array-of-those-things, or a promise-of-array-of-those-things, or an array-of-those-things-with-dependency-information. Whew…!

This approach is great for people producing the data. As a producer I can do:

new Vpc({ ipAddresses: ["an-address"] }); // or even
new Vpc({ ipAddresses: ["an-address", computeAddressAsynchronously() ]}); // or even
new Vpc({ ipAddresses: ["an-address", computeAddressAsynchronously(), someDependency.address ]}); // or even
new Vpc({ ipAddresses: computeAnArrayAnyOfTheAboveAsynchrousnously() }); // etc. etc.

It’s super flexible, and let’s people naturally produce the values in whatever way they want without having to work too hard to make it conform to some very narrow type.

However, while easy to produce, this can sometimes be difficult to consume. For example, if you wanted to take in that Input<Input<string>[]> and then check if there was a very specific value in it, you might have to go through and do a bunch of work. You’d have to first deal with potentially getting a Input<string>[] or a Promise<Input<string>[]> or even an Output<Input<string>[]>. Then, once you even got to the underlying array, you’d have to deal with each potential element in it being a string… or a Promise<string>… or a… well… you get the idea. It’s not fun 😃.

Up until now, we’d supplied some simple helpers to make this more manageable. Our helpers would effectively ‘unwrap’ one layer of this sort of structure allowing you take the external value and deal with an internal value of a known shape. So it effectively worked like this:

function unwrap<T>(input: Input<T>): T { /* ... */ }
// so, you could then write:
const ipAddresses: Input<string>[] = unwrap(vpc.ipAddresses);

This helped things somewhat. But it still meant that as you processed that array you had to then unwrap each and every element. As our data model has grown, and we’ve tackled more and more complex domains, we started seeing code start to form the programming Pyramid of Doom, where you just kept on going deeper and deeper, unwrapping all the way. We started seeing this happen so often that we wanted to see if we there was anything we could do to make life much nicer for people using our APIs and using these helpers.

Our core goal was to change our ‘unwrap’ function from just peeling away one layer of this onion, to peeling away all layers. So, for example, if you started with an Input<Input<string>[]> you would just get back a plain old string[] that you could use completely naturally in all the ways you’d expect. We were initially uncertain if we could make this work, but thanks to the power of JavaScript and the expressivity of TypeScript, we think we found a great solution here.

We knew that providing the implementation such a function itself was not going to the true hard part. After all, we already had the code to peel back one layer. So peeling back all the layers was more a matter of just having the function call itself recursively when encountering things like arrays (or other objects that themselves contained Inputs).

The real hard part came in when were trying figure out how would we even express this in a type-safe manner TypeScript. After all, what’s the type signature when you effectively want to say:

function unwrap(input: Input<...MaybeHasInputsOrPromisesOrOutputsArbitrarilyDeep...>): ThatThingWithAllTheDeepInputsPromisesAndOutputsErased;

This was a big head scratcher for a while. But after exploring some of the powerful new features that TypeScript has been bringing about in recent releases, we finally found a great way to manage it. In the end, it ends up looking something like this:

type primitive = string | number | boolean | undefined | null;
type Unwrap<T> =
T extends Promise<infer U> ? UnwrapSimple<U> :
T extends Output<infer U> ? UnwrapSimple<U> :
UnwrapSimple<T>;
type UnwrapSimple<T> =
T extends primitive ? T :
T extends Array<infer U> ? UnwrappedArray<U> :
T extends object ? UnwrappedObject<T> :
never;
interface UnwrappedArray<T> extends Array<Unwrap<T>> {}
type UnwrappedObject<T> = {
[P in keyof T]: Unwrap<T[P]>;
};
function unwrap<T>(val: T): Unwrap<T>;

It’s a bit of a doozy containing a bunch of type definitions and the final function declaration that uses them. It makes use of many advanced TypeScript typing features. Including ‘Union Types’ (the ‘primitive’ type), ‘Mapped Types’ (which ‘UnwrappedObject’ is an example of), ‘Conditional Types’ (the types with ? and : in them) and ‘Inferred Types’ (the use of infer U in several places). You can read more about these in the Advanced Types documentation.

Put together, the Unwrap<T> itself is effectively a recursively defined structural pattern matching type. You can see it reference itself once you walk into UnwrappedArray<T> and UnwrappedObject<T> types. As it recurses, it matches the type it has and then ‘peels away’ the wrappings that have been caused by Promise<T> and Output<T>. As it hits Arrays and Objects, it dives deeper, unwrapping along the way. This is all manageable in TypeScript because it does this sort of matching and unwrapping in a lazy fashion, allowing us to express what is effectively an infinitely large pattern-matching type, without it having any trouble understanding it or applying it!

It amazes us how well this works and how expressive TypeScript is here. As you can see here, TypeScript completely understands what’s going on and will give you all the great type-checking support and tooling that we know and love:

Since adding support for this, we’ve been able to start using this new function across our own codebase and we’ve seen things get dramatically simpler. The pyramid-of-doom goes away and code becomes much cleaner and easier to read and maintain. JavaScript helped us write this system easily, but thanks to TypeScript we can express these complex intents and have the safety and understanding to do this sort of thing with confidence.

A special thanks to @DanielRosenwasser over from the TypeScript team for putting up with my constant pestering on how to work through this and deal with some especially thorny areas!

AWS SQS to Lambda with Pulumi: Serverless Queue Example

Cyrus Najmabadi — Tue, 10 Jul 2018 00:00:00 +0000

Two weeks ago Amazon added Simple Queue Service (SQS) as a supported event source for Lambda. SQS is one of AWS’s oldest services, providing access to a powerful message queue that can do things like guarantee messages will be delivered at least once, or messages that will be processed in the same order they were received in. Adding SQS as a supported event source for Lambda means that now it’s possible to use SQS in a serverless computing infrastructure, where Lambdas are triggered in response to messages added to your SQS queue. Now, instead of needing some sort of Service dedicated to polling your SQS queue, or creating Simple Notification Service (SNS) notifications from your messages, you can instead just directly trigger whatever Lambda you want.

Example: Using AWS SQS and Lambda to post messages to Slack

Here’s a simple example of using SQS as an event source with Pulumi: upon receipt of a message via SQS we post a new message into Slack. The full project is available on Github in our examples repo, and includes the instructions to get this up and running.

let aws = require("@pulumi/aws");
let serverless = require("@pulumi/aws-serverless");
let config = require("./config");

let queue = new aws.sqs.Queue("mySlackQueue", { visibilityTimeoutSeconds: 180 });

serverless.queue.subscribe("mySlackPoster", queue, async (e) => {
 let slack = require("@slack/client");
 let client = new slack.WebClient(config.slackToken);
 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*: ${rec.body}`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to Slack channel ${config.slackChannel}`);
 }
}, { batchSize: 1 });

module.exports = {
 queueURL: queue.id,
};

Implementing SQS in Pulumi

When we heard about this new functionality, we immediately knew we wanted to let people access this new power from their Pulumi applications. Based on Amazon’s documentation on this new capability we felt it would likely be very easy to provide. Amazon introduced this functionality by reusing existing systems and APIs. Specifically, to create a Lambda trigger off an SQS message you only need to create a new Event Source Mapping mapping between the two. This seemed like it would be very simple, and we set off to expose the right Pulumi API to make this possible.

@pulumi/aws-serverless is Pulumi’s convenience library for creating Serverless computing components. And we added the necessary code to expose the functionality in our simple Pulumi programming model here. With this new API you would now be able to write code in your Pulumi application like so:

import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

// Create the queue. Provide whatever specific configuration you need here.
const sqsQueue = new aws.sqs.Queue("queue", {
 visibilityTimeoutSeconds: 300,
});

// Set up a subscription that will fire whenever the queue receives a message. Here we ask
// for 'batchSize = 1' so we will only process a single message at a time.
serverless.queue.subscribe("subscription", sqsQueue, async (event) => {
 // Add whatever code you want here to run in the AWS lambda. 'event' will contain the
 // all the necessary data about the message added to the queue.
}, { batchSize: 1 });

This example also demonstrates several of the great capabilities of a Pulumi application. First, the ability to define your infrastructure directly in code (i.e. the AWS Queue, Lambda, and Event Subscriptions), instead of needing to do that separately. Second, the ability to avoid needing to specify every single AWS resource necessary to get up and running. For example, I did not have to supply explicit Roles, Role-Policy-Attachments, or Permissions here as Pulumi took care of that for me by default. Third, the ability to create an AWS Lambda directly from a JavaScript/TypeScript arrow-function. That async (event) => ... is written in code as a normal async-arrow-function, but it will be converted by Pulumi into all the machinery necessary to have a true AWS Lambda running in your infrastructure.

After adding the API and writing up a simple app to test things, we then ran a pulumi update to deploy our application to AWS. Unfortunately, we immediately ran into a problem:

Plan apply failed:
Error creating Lambda event source mapping: ValidationException: 1 validation error detected: Value '' at
'startingPosition' failed to satisfy constraint: Member must satisfy enum value set:
[LATEST, AT_TIMESTAMP, TRIM_HORIZON]

So what happened? Well, as it turns out, Pulumi leverages and integrates with many other amazing Open Source projects under the covers. In this case, it was our integration with Terraform that was causing this small snag. It turns out that while Amazon relaxed their API to no longer have that constraint on ‘startingPosition’, Terraform was still referencing an older awssdk API that still contained that constraint.

At Pulumi we think that Terraform is fantastic, and we love using it. So we thought the best thing we could do in this position was to help out that project to support this new functionality as well. We did this around two weeks ago by contributing this PR to their project. We worked with Terraform over a couple over a few days to get the PR up to snuff, and eventually got it in. With this, now both Terraform and Pulumi customers will be able to benefit from these changes in the upcoming releases for both projects!

After getting the change in we went back and tested our example and saw that now everything worked as expected. We also expanded things out in an example to show how you might use this in practice. In this example, we receive the event, and then just write the data of it into an S3 Bucket. Running this example we now successfully see:

pulumi update
Updating stack 'cysqstest'
Performing changes:
Type Name Status
+ pulumi:pulumi:Stack queue-cysqstest created
+ ├─ aws:serverless:Function subscription-queue-subscription created
+ │ ├─ aws:iam:Role subscription-queue-subscription created
+ │ ├─ aws:iam:RolePolicyAttachment subscription-queue-subscription-32be53a2 created
+ │ ├─ aws:iam:RolePolicyAttachment subscription-queue-subscription-7cd09230 created
+ │ └─ aws:lambda:Function subscription-queue-subscription created
+ ├─ aws:sqs:Queue queue created
+ ├─ aws:s3:Bucket testbucket created
+ └─ aws-serverless:queue:QueueEventSubscription subscription created
+ ├─ aws:lambda:Permission subscription created
+ └─ aws:lambda:EventSourceMapping subscription created
info: 11 changes performed:
+ 11 resources created
Update duration: 28s

Once created, we could then see this Stack at app.pulumi.com, and we could easily use the functionality there to even navigate to those resources over at our AWS Console. Over there were were able to use the console to both send a message to the Queue, verify our Lambda got triggered, and even check our Bucket to see the data written into it. In only around a dozen lines of code, we were able to provision 11 AWS resources, and create a real end-to-end Serverless message-receiving and processing pipeline. Now, we can expand on this functionality with more resources or more functionality, run pulumi update and just simply and safely update our cloud infrastructure!

You’ll see this functionality lighting up in our next release of @pulumi/pulumi and @pulumi/aws-serverless. We hope it helps you out and makes your cloud life that much easier. Happy coding!