Pulumi Blog: Marc Holmes

Delivering Cloud Native Infrastructure as Code

Marc Holmes — Wed, 05 Dec 2018 00:00:00 +0000

Enterprise software has undergone a slow shift from containerless servers to serverless containers.

The evolution of the cloud, combined with the shift to increasingly ephemeral infrastructure, and the connection of application code and infrastructure code, demands a different view of cloud development and devops.

To a first approximation, all developers are cloud developers, all applications are cloud native, and all operations are cloud-first. Yet, there is a lack of a consistent approach to delivering cloud native applications and infrastructure. The tools and processes differ by technology generation, and even by cloud vendor, and so deny the full potential of cloud native application delivery.

In our latest white paper, Delivering Cloud Native Infrastructure as Code, we make the case for a consistent programming model for the cloud and examine:

How the cloud has already evolved three times as it increasingly moves toward stateless compute to deliver on the opportunities afforded by unprecedented economies of scope and scale.
How stateless compute has shifted infrastructure management concerns from ‘at rest’ to ‘in motion’, and moved these concerns up the stack to development.
How the growth of DSL-based tools has lead to complexity for DevOps teams, failed to deliver on the promised collaboration between development and operations functions, and does not satisfy the need for increasing delivery speed in the cloud.

You can read the paper in full here.

Let us know what you think!

Meet the Pulumi team at AWS re:Invent

Marc Holmes — Thu, 15 Nov 2018 00:00:00 +0000

Heading to AWS re:Invent? Concerned about how you’ll manage to get that much YAML into your carry on bag? Or maybe you just like purple.

Whatever the reason, the Pulumi team will be there all week at **Booth 316, Startup Central, Aria Quad, **and we’d love to chat with you about AWS and Pulumi.

Catch up with us on serverless functions, containers and Kubernetes, managed services and any other cloud native infrastructure as code, and see how you can more productively manage your AWS cloud resources with general purpose programming languages. We can even help you migrate your CloudFormation to Pulumi.

If you want to grab a specific time to talk through your needs, then use this link, otherwise we’ll just see you at the booth!

Data science on demand: spinning up a Wallaroo cluster

Marc Holmes — Fri, 02 Nov 2018 00:00:00 +0000

This guest post is from Simon Zelazny of Wallaroo Labs. Find out how Wallaroo powered their cluster provisioning with Pulumi, for data science on demand.

Last month, we took a long-running pandas classifier and made it run faster by leveraging Wallaroo’s parallelization capabilities. This time around, we’d like to kick it up a notch and see if we can keep scaling out to meet higher demand. We’d also like to be as economical as possible: provision infrastructure as needed and de-provision it when we’re done processing.

If you don’t feel like reading the post linked above, here’s a short summary of the situation: there’s a batch job that you’re running every hour, on the hour. This job receives a CSV file and classifies each row of the file, using a Pandas-based algorithm. The run-time of the job is starting to near the one-hour mark, and there’s concern that the pipeline will break down once the input data grows past a particular point.

In the blog post, we show how to split up the input data into smaller dataframes, and distribute them among workers in an ad-hoc Wallaroo cluster, running on one physical machine. Parallelizing the work in this manner buys us a lot of time, and the batch job can continue processing increasing amounts of data.

Sure, we can handle a million rows in reasonable time, but what if the data set grows by orders of magnitude? By running our classifier on a local Wallaroo cluster, we were able to cut the processing time of a million rows to ~16 minutes, thus fitting within our allotted time slot of one hour. But if the input data is, say, 10x more, we’re going to have a hard time processing it all locally.

Let’s see how we can keep up with the data growth by launching a cloud-based Wallaroo cluster on-demand, running the job, collecting the data, and shutting down the cluster, all in a fully automated fashion.

Tools of the trade

Wallaroo’s big idea is that your application doesn’t have to know whether it’s running on one process, several local processes, or a distributed system comprising many physical machines. In this sense, there’s no extra work involved in ‘migrating’ our classifier application from the previous blog post.

We will need some tools to help us set up and manage our cluster in the cloud. Wallaroo can work with a lot of different tools. Our friends at Pulumi provide an excellent tool that removes the headaches involved in provisioning infrastructure. We’ll use Pulumi to define, set up, and finally tear down our processing cluster in this example.

We’ll also need Ansible to start, stop, and inspect the state of our cluster, and, last but not least, we’ll need an AWS account where our machines will live.

Let’s jump into it!

A sample run

First of all, if you’d like to follow along (and spend some money provisioning EC2 servers), please download and set up Pulumi.

Next, clone the wallaroo blog examples repo and navigate to provisioned-classifier. If you followed along with the previous Pandas blog post, you’ll find our old application nested away here, underclassifier. What’s more interesting are the two new directories: pulumi and ansible.

Without delving into details, let’s see how to run our application on a freshly-provisioned cluster in the EC2 cloud:

make up run-cluster get-results down INPUT_LINES=1000000 CLUSTER_SIZE=3

Let’s break that down and see what’s really going on here.

make up CLUSTER_SIZE=3 configures the cluster to consist of 3 machines, and delegates to pulumi up the actual business of spinning up the infrastructure. Our physical cluster will contain 3 nodes for processing, and one extra metrics_host node for hosting our Metrics UI, and collecting results.
Once provisioning is complete, the next make task: run-cluster INPUT_LINES=1000000 uses our Ansible playbooks to upload application code from classifier/* to all 3 machines provisioned above, and then start up a Wallaroo cluster with 7 worker processes per machine.
Next, Ansible starts sending 1 million lines of our synthetic CSV data, and waits for 1 million lines to arrive at the data_receiver process. When those lines arrive, they are compressed, and the cluster is shut down.
make get-results pulls the compressed result file to output/results.tgz,
And finally, make down destroys the cloud infrastructure that was used to power our computation.

The Pulumi cluster definition

Let’s take a look at how our infrastructure is defined. This is the core of the definition:

 function instance(name) {
 return new aws.ec2.Instance(
 name,
 {associatePublicIpAddress: true,
 instanceType: instanceType,
 securityGroups: [secGrp.name],
 ami: ami,
 tags: {"Name": name},
 keyName: keyPair.keyName})
 }

 let metrics_host = instance("classifier-metrics_host");
 let initializer = instance("classifier-initializer");
 let workers = [];
 for(var i=0; i<clusterSize-1; i++){
 workers.push(instance("classifier-"+(i+1).toString()));
 }

As you can see from the above, our little instance() function encapsulates the common settings for every machine that we want to provision.

The metrics_host and initializer are ec2.Instance objects with descriptive names, while the workers are ec2.Instances that are distinguished solely by their ordinal number. Pulumi lets us define – in code – things like Security Groups, SSH keypairs, and practically every other aspect of cloud infrastructure.

An example of this capability is the keyPair object that’s used to access the instances via SSH. Our Makefile ensures that an ssh key is generated on-the-fly for our cluster, and Pulumi knows how to use it to set up SSH access for newly-provisioned nodes:

let pubKey = fs.readFileSync("../ssh_pubkey_in_ec2_format.pub").toString();
let keyPair = new aws.ec2.KeyPair("ClassifierKey", {publicKey: pubKey});

With the relevant bits of our computing infrastructure thus defined, we can tell Pulumi to take action in the real world and make it conform to our definition:

When we run make up CLUSTER_SIZE=3, we’ll see Pulumi output something like the following:

Performing changes:
+ pulumi:pulumi:Stack classifier-classifier-demo creating
+ aws:ec2:KeyPair ClassifierKey creating
(...)
+ aws:ec2:Instance classifier-2 created
---outputs:---
metrics_host: [
[0]: {
name : "classifier-metrics_host"
private_ip: "172.31.47.236"
public_dns: "ec2-54-245-53-87.us-west-2.compute.amazonaws.com"
}
(...)
]
info: 7 changes performed:
+ 7 resources created
Update duration: 1m59.524017637s

We can use Pulumi’s output to stitch together an Ansible inventory, which will let us interact programmatically with our provisioned instances.

If we want to modify our cluster, we can edit pulumi/index.js, and then rerun make up CLUSTER_SIZE=N. If the changes don’t require restarting the instances (for example if they only concern the Security Group), Pulumi will do the right thing and not disturb the rest of the infrastructure.

If you examine the Pulumi definition file in depth, you’ll see that it relies on a mystery AMI: ami-058d2ca16567a23f7. This is an experimental ubuntu-based image, with Wallaroo binaries (machida, cluster_shutdown,cluster_shrinker, and data_receiver) pre-loaded. For now, it only exists in the us-west-2 AWS region, but we hope to make Wallaroo AMIs available for experimentation in all regions starting with the next Wallaroo release.

Running the computation

Now that we know what magic powers conjured up our AWS infrastructure, let’s take a look at how we use it to run our task. Fundamentally, the components of our cluster can be described as follows:

The data source – In our case it’s the file send.py, which can generate and transmit randomly-generated CSV data for our computation to consume.
The machida processes: one Initializer and a bunch of Workers. The distinction between the two is only relevant at cluster startup.
The data receiver – A process that listens on a TCP port for the output of our computation. This is the data_receiver, provided as part of a Wallaroo installation.
The metrics UI – Our Elixir-powered realtime dashboard.

Our Ansible playbook takes care of coordinating the launch of the various components and making sure that their input, output and control ports match up. In particular, that the cluster initializer starts up knowing the total number of workers in the cluster, and every other worker connects to the initializer’s internal IP and control port.

This is what ends up running on the servers when we launch our Ansible playbooks:

Once the cluster is up and running, and the initializer node’s tcp source is listening for connections, we start up the sender and instruct it to send a stream of data to the TCP Source. In a realistic batch scenario, this sender could be implemented as a Connector that reads a particular file from a remote filesystem or S3. For our purposes, we’ll simulate this by generating a set number of CSV lines on-demand, and then shutting down.

While the work is being performed, we can take a look at the metrics URL printed out on the screen to find out how the work is being distributed.

To see the cluster’s real-time metrics, please visit http://ec2-54-200-198-6.us-west-2.compute.amazonaws.com:4000

In the screenshot above, you can see that the Initializer and B03a909b23nodes are processing about 4k messages per second each, and all the other workers have the classification work split evenly among them. Don’t be surprised that two of the workers are processing orders of magnitude more messages! Remember our pandas application pipeline?

 ab.new_pipeline("Classifier",
wallaroo.TCPSourceConfig(in_host, in_port, decode))
ab.to_stateful(batch_rows, RowBuffer, "CSV rows + global header state")
ab.to_parallel(classify)
ab.to_sink(wallaroo.TCPSinkConfig(out_host, out_port, encode))
return ab.build()

The CSV rows come in one-by-one, but we batch them and convert them to dataframes of a hundred, so that our classification algorithm can tackle more than one row at a time. Worker B03a909b23 just happens to be the worker where our state named “CSV rows + global header state” lives. Let’s take a look at its metrics:

Indeed, we can see that this worker is processing about 4k/sec messages in the “Batch Rows Of Csv, Emit Dataframes” step. Every other worker is busy classifying! Let’s see the breakdown for a different, random worker:

Looking good. All that’s left for us to do is wait until the job completes and we receive our zipped data onto our disk.

The numbers!

As a reminder, let’s take a look at the numbers we obtained by running our classifier on a single c5.4xlarge instance in AWS:

# SINGLE-MACHINE RUNNING TIMES (NO PROVISIONING)
CSV rows 1 worker 4 workers 8 workers
----------- ---------- ----------- -----------
10 000 39s 20s 11s
100 000 6m28s 3m16s 1m41s
1 000 000 1h03m46s 32m12s 16m33s

Now, let’s see how much speedup we can achieve from scaling out with our provisioned-on-demand infrastructure.

# MULTI-MACHINE RUNNING TIMES (PROVISIONING + COMPUTATION)
----------------------------------------------------------
CSV rows 4 machines/ 8 machines/ 16 machines/
28 workers 56 workers 112 workers
------------ -------------- -------------- ---------------
10 000 2m13s 2m36s 2m43s
100 000 3m38s 3m42s 3m48s
1 000 000 7m38s 6m41s 5m56s
10 000 000 40m56s 33m10s 23m24s
30 000 000 > 2h 1h45m 1h12m
----------------------------------------------------------

Even though there is some constant overhead involved in spinning up the required infrastructure (clearly too much overhead to justify spinning up clusters for less than 1 million rows), we’re now able to classify a hefty 10 million rows of CSV data in under half an hour, and 30 million in a little over an hour. This gives us some perspective on when our application will need extra resources, or perhaps some performance optimizations.

As long as the data fits in the 1 million : 10 million range, it seems that a cluster of 4 machines represents a sweet-spot between price and performance – we can process the data and still fit in the hour-long window allotted for our batch process, but not have to incur unnecessary infrastructure costs if we don’t need the extra speed.

Conclusion

The above figures illustrate how Wallaroo can be used as an ad-hoc compute cloud, using Pulumi and Ansible to provision, run workloads remotely, and shut down the infrastructure once the results are in.

In the case or our batch job, we can leverage this pattern to scale horizontally on-demand, even when the incoming workloads exceed the capacity of one physical machine – all while running the exact same Wallaroo application that we run locally as part of our regular development. Wallaroo handles the scale-aware layer of our program, so we can focus on the business logic and flow of our data.

If you’re hitting limits when running your hourly, daily or nightly batch jobs and are looking into scaling out horizontally, don’t hesitate to reach out or drop in to our IRC channel. We’d love to chat!

Building new Pulumi projects and stacks from templates

Marc Holmes — Mon, 01 Oct 2018 00:00:00 +0000

When you’re able to build an app for any cloud using familiar languages, the obvious question is “Where to start?”. We hear you, and so we’ve built some new features to help you scaffold your app and program the cloud even faster than before.

In this post, we’ll look at how to use pulumi new and our selection of templates to build your Pulumi app.

There is template support for:

AWS, Microsoft Azure, Google Cloud, Kubernetes, and OpenStack in each of…
TypeScript, JavaScript, Python, and Go (K8S is currently just JS/TS)
You can see all these options at https://github.com/pulumi/templates where we’ll gladly accept updates and new templates as PRs

Creating a project from the Pulumi dashboard

Head over to app.pulumi.com and - supposing you’re logged in - you’ll be presented with the usual homepage. But now we’ve added a shiny new ‘Add New Project’ button. Here’s how that works:

Clicking the ‘Add New Project’ button takes you to a series of available templates - essentially a matrix of cloud provider and language: from Go on Google Cloud, through TypeScript on AWS, to Python on OpenStack, and everything in-between.
Choosing an option will present a screen with some standard configuration options that you can change or leave defaulted.
Click again, and Pulumi creates a project which is ready for you to initialize your first stack. The project page shows the command line instruction to run to get started with your chosen stack.

We can use that code by creating a folder, and then running the command in the folder to get build our boilerplate stack. All of the stack templates create a storage bucket (e.g. an AWS S3 bucket).

mkdir MyStack && cd MyStack
pulumi new https://github.com/pulumi/templates/aws-javascript -s mehzer/my-stack-dev

The output of this command will create the new stack from the template, display a preview of the stack, and ask if you’d like to create it for real. Assuming you have setup the relevant cloud provider, that’s all there is to it.

Back in the dashboard, you can now see the state of your stack, and you’re ready to flesh it out with the detail you need.

Creating a project from the Pulumi CLI

Alternatively, you can do the same thing from the CLI. Create a folder, then type pulumi new in there to retrieve a list of available templates. Templates ahoy!

After you’ve selected the template you’d like, you can again pulumi preview or pulumi up to get your stack running.

Wait, wasn’t that just a link to a repo? Does that mean..?

Yes it was, and yes you can run pulumi up against an arbitrary repo supposing that there is a Pulumi.yaml and package metadata (e.g. package.json or Gopkg.toml or requirements.txt files) alongside the stack code meaning you can build your own templates and share them.

We hope you like these helpers. If you’re keen to get stuck in:

Join the Slack conversation - it’s heating up in there.
Try out the many examples we have, and dive into the docs.
Submit new templates as PRs - contribution == t-shirts at least.

Build your first serverless app using only JavaScript

Marc Holmes — Thu, 05 Jul 2018 00:00:00 +0000

This post is outdated. The hello-aws-javascript template and @pulumi/cloud-aws package used in this post are no longer maintained. We recommend using TypeScript for new Pulumi projects. See the Get Started guide to build your first serverless app with TypeScript.

In this tutorial, we’ll use Pulumi to build a complete serverless application using only JavaScript. When we say ‘using only JavaScript’, we’re not kidding:

write code just like an Express app… but end up with a fully deployable serverless app
lambdas are… just lambdas
no YAML required… freedom from indentation
all the features of the V8 runtime… async await ahoy
all the behaviors of immutable infrastructure as code tools… but we really mean ‘as code’

Pulumi also supports containers (including Kubernetes), managed services, infrastructure and everything else in between that you might need for building cloud applications. Better than that, you can even combine them all in the same program.

Our first serverless app in 5 lines of JavaScript

After installing the Pulumi CLI, a small index.js file (the main Pulumi code) represents a simple but complete serverless application:

// Add the required package
const cloud = require("@pulumi/cloud-aws");

// Declare an HTTP endpoint
const endpoint = new cloud.API("hello");

// Serve up some static content on that endpoint
endpoint.static("/", "www");

// Create a simple serverless function responding to GET
endpoint.get("/source", (req, res) => res.json({name: "AWS"}));

// Publish the URL so we can easily access our app
exports.url = endpoint.publish().url;

So what’s happening here? This code creates a cloud.API which exposes an HTTP endpoint to the internet. It serves static content at the root of the API from what’s in the www folder on your local machine. And it serves a REST API on GET /source which runs the JavaScript callback to return the JSON object {name: “AWS”} (our serverless function). Finally, it exports the URL where the HTTP API is exposed.

Now just run pulumi update to deploy the application. Before the app deploys, you’ll get a preview of what will be deployed and you can choose to go ahead with the full deployment. Pulumi figures out all of the cloud resources needed to run the code above and prepares that for deployment.

Once deployed, your application will run in the cloud. Use pulumi stack output url to get the URL where your app is running, and open it up in your browser:

With Pulumi, we’ve built a serverless JavaScript app in just a few lines of code. This approach avoided significant amounts of configuration (YAML, or point-and-click). Pulumi also supports containers (including Kubernetes), managed services, infrastructure and everything else in between that you might need for building cloud applications. Get started with Pulumi.

Why JavaScript for serverless programming?

There are many great reasons to use JavaScript for serverless programming (and cloud programming generally). Pulumi also supports TypeScript, Python, and Go with more languages on the way.

Pulumi uses NPM, Express, and supports modern JavaScript features like async await, module etc. and so you can bring your JavaScript skills directly to cloud programming.

Code is the best Config

Serverless platforms such as AWS Lambda offer an easy and cost-effective way to run your code in the cloud. But using these serverless platforms often requires learning lots of finicky details of the cloud platform, and using esoteric YAML configuration files to tell the cloud platform how to run your app. Often, the work needed to configure the app can outweigh the work needed to code the app.

Pulumi changes that, and lets you create serverless cloud applications using only JavaScript (or TypeScript). These applications can use all the features of cloud platforms like Amazon Web Services (AWS), Azure or Google Cloud Platform (GCP) - but exposed as simple JavaScript APIs on NPM.

This means that you can write code that contains logic alongside code that defines the services and infrastructure needed to run it and Pulumi can then deployed the app in seconds and run as a managed, scalable and - in the case of serverless apps such as this one - extremely low cost service on your chosen cloud.

Code is more productive, expressive, and… fun

One of the biggest advantages of a pure code approach is the automatic gains from better tooling support. In particular:

Code completion. There are a lot of services in each cloud. Wouldn’t it be great to be able to autocomplete?
Error checking. Less worry about correctly indented YAML, more real error checking.
Versioning and Packaging. Much improved versioning and packaging scenarios through Github and NPM workflows.
Reusable components. Build on top of base classes, and share for easy reuse, standard configurations and so on.

Want more?

To learn more take a look at more tutorials and example code:

Our origin story: Hello, Pulumi!
Tutorial: Deploying Containers with Pulumi
Tutorial: Build a video thumbnailer using AWS Lambda, Fargate, and S3 in JavaScript
Pulumi Quickstart
Pulumi Community Slack
Pulumi Examples on GitHub
Node.js and Pulumi
Node.js Examples for AWS and Azure