Pulumi Blog: Logging

Pulumi Watch: Fast Inner Loop Development for Infrastructure

Luke Hoban — Mon, 02 Dec 2019 00:00:00 +0000

A big part of our vision with Pulumi is to bring application developers and infrastructure teams closer together in the cloud. That includes both providing infrastructure teams with better software engineering tools, as well as providing developers with easier access to cloud infrastructure. We are often inspired by looking at great software engineering experiences in other development stacks and applying them to the cloud infrastructure space. Whether it be general-purpose languages and rich IDEs, testing and package management, or components and rich APIs, at Pulumi, we’ve repeatedly applied successful development tools and practices to the challenges of building and scaling modern cloud infrastructure.

As we look across other areas of successful application development, one of the most critical drivers of creativity and productivity is the speed of iteration. When developers can make a change and see or experience the results nearly instantaneously, they can explore solutions quicker, can take more creative risks (and thus learn more), and can ultimately iterate more quickly toward their desired result. From fast build times as a driving force in the design of the Go language, to the DevTools in Chrome, to Edit-and-Continue in Visual Studio, to the watch modes on tools like TypeScript, this idea of enabling quick or near-instantaneous feedback as code changes are made is a driving force behind many of the most successful developer experiences across a wide variety of software development environments.

Introducing Pulumi Watch

In Pulumi 1.5, we released a preview of a new pulumi watch command. This command provides a mode for developing cloud infrastructure that speeds up the rate of iteration and allows cloud developers to focus on their code and infrastructure, instead of on the mechanics of their deployments. With pulumi watch, cloud infrastructure feels like it’s truly at your fingertips.

The video above shows an example of working with some S3 resources. Every time I hit “save” in my editor, pulumi watch automatically deploys any changes to my infrastructure. If I add a new BucketObject resources and hit save, the object appears in S3. If I add a loop around it and hit save, 10 new objects appear and the original object is removed. And if I add an event handler, the 5 Lambda and IAM resources needed are deployed in seconds and are hooked up to the bucket. You don’t have to switch back and forth between an editor and a terminal, and you don’t have to go through heavyweight infrastructure deployment tools like Terraform or wait on slow external services like CloudFormation or CI/CD. Instead, changes appear directly in the cloud within seconds of hitting enter. You can try things out, experiment, learn, and rapidly iterate on your cloud infrastructure.

This new watch mode isn’t a replacement for a standard pulumi preview/pulumi update. For production infrastructure deployments, it is critical to understand precisely what changes will happen before applying them, to integrate with CI/CD systems, to be able to do targeted updates, and to apply policy to guard the production environment against misconfiguration. But that same level of care isn’t necessarily the right way to approach building new features in a development stack. And it’s in these development environments that pulumi watch opens up a new experience for infrastructure development. Although they are shorter periods relative to the full life of an infrastructure deployment, they are the majority of the time that creativity and creation is applied to cloud infrastructure, and thus represent a key opportunity to empower developers.

Not all cloud resources can be deployed or updated in seconds - but many of the most popular modern cloud primitives are built on platforms that enable and encourage quick iteration - and in particular both serverless and Kubernetes. Watch mode can be used during the development of any infrastructure, but can really shine in these areas where the cloud platforms natively support fast iteration.

Kubernetes Infrastructure + Applications

For Kubernetes, pulumi watch allows us to deploy changes to our cluster automatically as we author our Kubernetes resources. Change the number of replicas for our Deployment and hit save and the new pods scale-out. Add a Service to expose the deployment and public endpoint is available immediately. This pairs beautifully with the recently released Pulumi Kubernetes Extensions (kx) library, which allows you to build Kubernetes applications with just a few lines of code - avoiding a lot of the boilerplate that slows down typical Kubernetes development when working with YAML. We can see this combination in action in the video below:

Both of the previous videos also highlight another interesting element of modern cloud infrastructure development. In both the S3 example and the Kubernetes example, the Pulumi program deployed not just the infrastructure, but also the application code and assets. By authoring and delivering both infrastructure and application code together, we remove another aspect of the silos that separate developers from infrastructure, and in doing so, unlock even faster iteration. This is a pattern that Pulumi enables in a wide variety of unique ways, from Lambdas as lambdas, to build-and-push of Docker images, to deploying assets for static sites.

From Zero to Cloud Application in 90 Seconds

We can bring these ideas together, and show another example of what’s possible with watch mode using a serverless application example. As with so many things these days, a tweet inspired this example:

I want a website that does one thing: displays the number of times it’s been viewed. What is the shortest number of steps you can think of to get it running in the cloud?

— Slava Akhmechet (@spakhm)

Indeed, such simple tasks are often surprisingly complex and cumbersome with today’s infrastructure tools. But with pulumi watch and the Pulumi Cloud Framework, we can go from nothing to having this application deployed in AWS (or Azure!) in under 90 seconds.

Conclusion

When using pulumi watch, working with cloud infrastructure feels so different - it feels malleable, it feels accessible, and it feels like it’s right at your fingertips. By building on the same robust incremental deployment infrastructure as the rest of Pulumi, you get the same access to any cloud, and to a variety of languages and all the other benefits of Pulumi’s modern infrastructure as code tooling. Pulumi watch triggers a deploy on any save - including both of files in the Pulumi program itself, as well as other assets that the Pulumi program may reference like application source code or static assets. Pulumi watch also displays runtime logs for supported platforms (Lambda, ECS, and Cloudwatch currently, more coming soon), and can be paired with another new feature, Pulumi Query, to get customized status about the runtime environment during rapid iteration.

The pulumi watch command is available in preview now when the PULUMI_EXPERIMENTAL environment variable is set to true. We are just getting started - and see many more opportunities to tighten the iteration cycle further, to support more logging capabilities, to integrate more deeply with query, and to be applied to more modern infrastructure scenarios.

Give it a try today and drop us a note about where you’d like to see this go for Pulumi 2.0!

AWS CloudWatch made easy with Pulumi Infrastructure-as-Code

Cyrus Najmabadi — Fri, 14 Jun 2019 00:00:00 +0000

Pulumi Crosswalk for AWS modules can be used to get first class insights and visualizations directly inside your Pulumi application.

As cloud applications tend to be long-lived, we think it’s vital that it be possible to get regular insights on the performance of the application at all times. Using Crosswalk for AWS Pulumi applications allow you to easily define and visualize the appropriate metrics that show the health of your services, create alarms to let you know when something is wrong, and easily create dashboards to get live visualization of what is happening in the cloud. Because this is vital to the health of the application, we think this should be something built in from the start, and not something added after the fact as an out of band artifact.

Let’s see what that means in practice! I’m going to use a recent hackathon project I created where I built a Slack bot to notify me about all the @mentions I’d received. Using this, I’d have a place I could go back to to look at, to find messages I knew had come in, but maybe I wanted to take a second glance at. The bot just sends me messages in a private channel we share like so:

I intend to add more functionality to this bot over time to help me out. If you’d like to create something similar, the code and instructions are here. The bot works by hearing about incoming events (pushed to it from Slack’s servers), looking for mentions of my name, then pushing requests back to Slack’s server to update our private channel.

Now the bot I wrote has to do something interesting due to restrictions in the Slack messaging system. Slack will only give you a few seconds to deal with a message they send you. If you don’t complete the request in a timely fashion then they’ll assume something has gone wrong, and they’ll resend. This will happen a few times before they just give up. This means that we don’t really want to try to process a message, and then also make an outgoing network call to Slack all in the same request. That outgoing call could take far too long, causing Slack to think there was a problem. So, instead, we simply receive the request, and push it right away to an SNS topic. We can then process that topic message later with another Lambda.

The skeleton of the code basically looks like this:

// Location to store the incoming slack messages to.
const messageTopic = new aws.sns.Topic("messages");
// Create an API endpoint that slack will use to push events to us with.
const endpoint = new awsx.apigateway.API("mentionbot", {
routes: [{
path: "/events",
method: "POST",
eventHandler: async (event) => {
// process the different types of messages. Push normal messages to 'messageTopic'
// ...
// Return success quickly so that Slack doesn't just immediately resend this message to us.
return { statusCode: 200, body: "" };
},
}],
});
// Hook up a lambda that will then process the topic when possible.
const subscription = messageTopic.onEvent("processTopicMessage", async ev => {
// process the messages in here. Look for mentions, then notify slack to post a message in
// right channel if we find one.
});

Overall, this is a very simple app. But how do we know if the app is actually doing well? What if something starts to go wrong and the endpoint is no longer responding in a timely fashion. How would we find out? It turns out to be pretty simple to add this functionality here! Let’s see how that would look.

First, we want to be profiling the Lambda Functions that were created by those two resources. That’s very simple to do:

const endpointFunc = endpoint.getFunction("/events");
const topicFunc = subscription.func;

Then, we want to get information about how long these functions are taking. With Pulumi’s Crosswalk APIs it’s simple to get at this information. Inside all the main “Crosswalk for AWS” modules are exposed Metrics for almost anything you might ever need. In this case, we want to know how long those Functions are taking so we can get that information as follows:

const endpointFuncDuration = awsx.lambda.metrics.duration({ function: endpointFunc });
const topicProcessDuration = awsx.lambda.metrics.duration({ function: topicFunc });

Metrics are very customizable. But for now we’ll just use the defaults returned for those functions by AWS. Now that we have the metrics, it’s simple to create an CloudWatch alarm that will alert us when things take too long:

// Notify us if takes us longer than 5 seconds to process a Slack message. But only notify if this
// happened at least once every 5 minutes in a 15 minute period. We're ok with the occasional
// random network hiccup causing delays.
const alarm = endpointFuncDuration.withUnit("Seconds")
.withPeriod(300)
.createAlarm("TooLong", {
threshold: 5,
evaluationPeriods: 3,
});

Finally, we can use both those metrics and the alarm we just created to make a great looking dashboard that we can use to get an overall view of the health of this app:

const dashboard = new awsx.cloudwatch.Dashboard("mentionbot", {
widgets: [
new awsx.cloudwatch.LineGraphMetricWidget({
width: 12,
title: "Receiving duration",
metrics: [ endpointFuncDuration.with({ extendedStatistic: 99, label: "Duration p99" }) ],
annotations: new awsx.cloudwatch.HorizontalAnnotation(alarm),
}),
new awsx.cloudwatch.LineGraphMetricWidget({
width: 12,
title: "Processing duration",
metrics: [ topicProcessDuration.with({ extendedStatistic: 99, label: "Duration p99" }) ],
}),
],
});

The Crosswalk for AWS API for dashboards gives a lot of options for building up a dashboard programmatically. There are a wealth of Widgets and Annotations to expose the full power of Cloudwatch’s dashboards to your app in a simple to use manner.

With the above Dashboard code we get this useful chart:

We can see that things look healthy, and the chart shows us both the information about our p99 latencies, as well as the threshold point that would cause our alarm to trigger.

With these facilities I now have an app that does what I need, but also directly exposes mechanisms to track and get insights to its health. What’s great is that if I now change my app (for example, in a way that causes new Lambda Functions to be created), I don’t need to do anything to keep these alarms and dashboards working. Because they’re directly referencing the real Cloud resources they will be kept up to date automatically as my app grows and changes over time. That’s really powerful, and much simpler than having to manage all these different parts independently!

We hope these new Crosswalk for AWS APIs will be just as useful for you! For more information on Pulumi Crosswalk for AWS checkout other related content:

Unified Logs with pulumi logs

Luke Hoban — Tue, 02 Apr 2019 00:00:00 +0000

Pulumi makes developing and deploying rich serverless and container-based applications a breeze. But how do you monitor and observe those applications while they are being developed and once they are deployed? There are many great answers: from the built-in capabilities of the underlying cloud services (Lambda, ECS, Kubernetes, and more), to great 3rd party solutions like IOpipe and Epsagon which we highlighted recently on this blog.

The Pulumi CLI provides another way to do logging, without requiring the additional setup of these existing solutions and seamlessly integrated into your Pulumi development workflow. The pulumi logs command provides a great first place to start for understanding your Pulumi application’s behavior. Especially during development, this command provides direct insight into the behavior of your application, bringing together logs across all of the different forms of compute you are using - from code running in serverless functions to containers to VMs.

Let’s take a quick look at pulumi logs and some of the ways it can be used as part of the inner loop of your Pulumi development.

Note: Currently pulumi logs supports only logs for applications running in AWS. In the near future it will be expanded to Kubernetes, Azure, GCP and additional cloud platforms.

Logging for Inner-Loop Development

When developing locally, it is common to use logging as an interactive part of the development process. With pulumi logs, we can bring some of these same approaches into the inner loop of the development process even for applications being actively developed/deployed in the cloud.

Let’s take a simple example:

import * as aws from '@pulumi/aws'
const bucket = new aws.s3.Bucket("mybucket");
bucket.onObjectCreated("newObject", async (ev) => {
    console.log(JSON.stringify(ev));
});
export const bucketName = bucket.id;

We can deploy this app, copy an object into the newly provisioned bucket, and then see the logs with the following:

$ pulumi up --skip-preview
 
$ aws s3 cp somefile.txt s3://$(pulumi stack output bucketName)
 
$ pulumi logs -f
Collecting logs for stack dev since
 
 2019-03-28T21:10:05.667-07:00[ newObject] {"Records":[{"eventVersion":"2.1","eventSource":"aws:s3","awsRegion":"us-west-2","eventTime":"2019-03-29T04:10:05.318Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"AWS:AIDAIB2I7NPKMQ2QAKXZ6"},"requestParameters":{"sourceIPAddress":"24.35.94.249"},"responseElements":{"x-amz-request-id":"F2D7DF26693AEFEE","x-amz-id-2":"o2leqNNwCVXTI8jfvUGk5KE7w6CiKsJYP/Q4nT0fHqZPrqwaVlSP0lNhspv5NiLAAwutCzJwlLc="},"s3":{"s3SchemaVersion":"1.0","configurationId":"tf-s3-lambda-20190328222809603300000002","bucket":{"name":"mybucket-97f10e0","ownerIdentity":{"principalId":"A3FZDZNSWOKQHV"},"arn":"arn:aws:s3:::mybucket-97f10e0"},"object":{"key":"out.txt","size":36000,"eTag":"818ac0cf0273f7e40e673d9892405e31","sequencer":"005C9D9A9D311D5C11"}}}]}

I can use this to quickly see what kind of values I get back on an S3 Bucket Object event, using live data running in the cloud. But I can do this whole thing in just a couple of seconds - from deployment to logs. This lets me iterate on my application live in the cloud with ease.

If there are errors in my code, I find out about them immediately:

bucket.onObjectCreated("newObject", async (ev, ctx) => {
    throw new Error("oops");
});

Results in:

$ pulumi logs --since 1m
Collecting logs for stack dev since 2019-03-28T20:17:28.000-07:00.
 
 2019-03-28T21:17:01.269-07:00[ newObject] {"errorMessage":"oops","errorType":"Error","stackTrace":["/var/task/__index.js:26:11","Generator.next (<anonymous>)","/var/task/__index.js:12:71","new Promise (<anonymous>)","Object.<anonymous> (/var/task/__index.js:8:12)","Object.__f1 [as __awaiter] (/var/task/__index.js:17:34)","/var/task/__index.js:24:21","__f0 (/var/task/__index.js:30:34)"]}

Aggregating Logs

As applications grow to span multiple forms of compute – such as micro services, lambdas, containers, and VMs – it becomes important to be able to correlate logs across services.

One great example of this is the video thumbnailer example which implements this simple architecture involving two buckets, a lambda event, and a long-running Fargate container task.

The Pulumi program deploys several components, using different types of cloud resources. But using Pulumi logs we can see all logs in a single, unified view.

From the image above, you can see the onNewVideo Lambda function, then triggering the ffmpegThumbTask Fargate container, and then copying a file back into S3 triggering the onNewThumbtail Lambda function. So you don’t need to jump around between different systems for viewing logs, you can get a consistent view of what’s going on in your cloud application!

Underneath the hood, these logs live in different and uncorrelated CloudWatch LogGroups, spread across an AWS account. But with pulumi logs we can get the application-centric view we need to understand the overall behavior of our application architecture.

Controlling Log Rendering

The pulumi logs command supports a few options for controlling what gets logged.

The --since flag retrieves logs only since a given relative or absolute time. For example, pulumi logs --since 1m will get logs just since 1 minute ago - useful for limiting logs to only what’s happened very recently - especially during inner-loop development. And to get logs since a specific point in time (such as the timestamp of a known event from a previous execution of pulumi logs) you can use pulumi logs --since 2019-03-28T21:17:01.269-07:00.

A couple of the examples above used the --follow flag to follow the log output until cancelled. This is useful when waiting for log output to come in during debugging.

And for applications with many resources, it is often important to scope results down to one or more specific resources, which can be done with the --resource flag. For example, logs for just the Fargate task above could be retrieved with pulumi logs --resource ffmpegThumbTask.

Working Seamlessly with Your Cloud Provider

Like many other Pulumi features, pulumi logs works by building on top of the great building blocks provided by the underlying cloud provider services. Instead of adding new runtime logging infrastructure to your application, pulumi logs lets you easily extract out logs directly from your existing infrastructure. But because Pulumi understands the natural application scope for what resources are part of a single application, it can bring together the logs from all of those resources into one place, instead of having to think about individual resources.

Behind the scenes, pulumi logs makes the same calls to CloudWatch Logs or other backend log providers that you would make yourself via the aws CLI or AWS Console. But you don’t have to think about the physical identities of the cloud resources themselves - Pulumi takes care of finding the resources with supported logs, collecting those logs based on the requested filters, and then presenting them in unified logical view.

What’s Next?

We’ve seen Pulumi users working with Lambda, ECS and other AWS compute offerings become heavy users of pulumi logs during development. But supporting a broad set of cloud providers is key to our mission at Pulumi - so we really want to bring these benefits to all Pulumi users. We’ll be expanding pulumi logs to Kubernetes, Azure and GCP in the near future - with easy support for plugging in other cloud providers and cloud resources as sources for log data.

Even better, because Pulumi allows you to work with multiple cloud providers at once - like AWS and Kubernetes, or GCP and Azure, or even two different AWS regions - pulumi logs will be able to aggregate logs across multiple cloud providers and offer a seamless application-centric view of your logs.

Users of the Pulumi Service at app.pulumi.com also would love to have access to these logs at the click of a button - and this is something we’re also excited to enable in the near future.

You can take pulumi for a spin today by checking out the Getting Started Guide guide or some of the Pulumi Examples on GitHub. Then join us in the Pulumi Community Slack to chat about where you want to see us go (and where you’d like to help out!) with pulumi logs and other features of the Pulumi open source project in the coming weeks and months.