Pulumi Blog: Chris Smith

Six Things You Might Not Know About the Pulumi Service

Chris Smith — Mon, 24 Jan 2022 00:00:00 +0000

As a reader of this blog, you’ve probably heard of the Pulumi Service, the default state-management backend of the Pulumi CLI. If that’s the case, there’s also a good chance you’ve heard of a number of the Service’s key features, like helping you organize your projects and stacks, collaborate with others with the help of organizations, or handle sensitive data securely with built-in support for encrypted secrets.

What you might not know, though, is that we’re adding new features to the Pulumi Service all the time, and that some of these features can be fairly easy to miss. So in this post, we’ll highlight a handful of the features you might not be aware of, and that we think make it even easier to manage your infrastructure with Pulumi.

Let’s get started!

Feature 1: Stack Tags

The first thing we’ll cover is the Pulumi stack tags feature. Stack tags are key/value pairs that are associated with your Pulumi stack. You can use them for any number of purposes, from keeping things organized to even flagging whether or not it’s okay for a script to reclaim the stack’s resources.

While you can create, edit, and remove stack tags using the Pulumi CLI (see pulumi stack tag), you can also do so in the Pulumi Console:

Once set, you can quickly filter stacks on your organization’s Projects tab as well.

Feature 2: Link Updates to your CI/CD pipeline

Another helpful thing the Pulumi Service can do is link the Pulumi stack update to the CI/CD job or run that was used to perform it, as well as to the specific source commit:

When Pulumi updates a stack, it will store some information about the local machine state, such the current git SHA if the stack resides in a git repository. It is with this data that the Pulumi Service links to relevant services where possible.

If you’re curious to know all of the information Pulumi has for a stack update, you can navigate to the Environment tab for that update:

Feature 3: Custom Update Messages

One of the most useful pieces of additional data stored with each update is the message property. By default, Pulumi will use whatever the latest git commit message is. But did you know you can customize that message from the command line?

By passing the --message (or -m) flag to pulumi up, you can provide a custom update message, which can help explain the context for why you made a particular infrastructure change:

$ pulumi up --message "Release the hounds\!"

Feature 4: Viewing Resource Changes

Another thing you might have missed is that the Pulumi Service supports multiple views for a stack update’s logs.

The standard view mimics the Pulumi CLI’s output. However, you can switch to the Diff or Diagnostic views, too.

The Diff view displays all of the properties of modified resources, along with their previous/updated values. This can be helpful when trying to understand exactly what happened for a stack update.

The Diagnostic view just outputs so-called “diagnostic” messages from the Pulumi update. Most commonly, this is where you will see the output of things like calls to JavaScript’s console.log(...) or Python’s print(...) functions. It’ll also include the output of dependent tools, like the output when building Docker containers.

Feature 5: Stack Resource Visualization

Another cool feature of the Pulumi Service is being able to visualize a stack’s resources and their relationships to one another. This is a really important feature!

One of the advantages of using an infrastructure as code tool like Pulumi is that you aren’t just creating a flat list of resources. Instead, by the virtue of how the code and resource graph is structured, you are building a graph of resources.

This means that Pulumi has a semantic understanding of not only what resources are in your stack but also how they relate to one another.

If you navigate to a stack’s Resources tab, by default you see them listed in a table. You can filter this list by resource type, or by providing a custom filter:

However, in order to see the expanded resource graph, click the Graph View button. This will switch to rendering the resources as an interactive graph.

Notice here that the resources that are part of a component are grouped together. Also, you can double click on those nodes in the graph to expand or collapse them!

Feature 6: Drilling into Resource Properties

But you don’t need to just settle for a “top-level view” of your resources. You can click on any specific resource on the Resources tab, and see all of the properties of that resource.

This allows you to see all of the properties that Pulumi is tracking for that resource. It will also show you any relationships to other resources in the same stack.

The Pulumi Service also links you to the resource’s cloud provider’s console where possible, so you can see even more details.

Is your resource missing a link to the cloud provider console? With new services being launched from the myriad providers Pulumi supports every day, it’s hard to keep up. So if you would like us to add cloud provider links, please file an issue over in the pulumi/console-requests GitHub repository so we can update our database.

There are plenty of other features to check out in the Pulumi Service, and we have exciting improvements on the way! As always, feel free to stop by the Pulumi Community Slack to learn more, ask questions, or share anything cool you’re up to!

Managing AWS Credentials on CI/CD: Part 3

Chris Smith — Thu, 14 May 2020 00:00:00 +0000

This article is the third part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we cover the last leg of the continuous delivery process to update your AWS resources and how to store sensitive data using Pulumi securely.

Recap

In part 1, we created a dedicated IAM User with limited privileges. Then in part 2, we set up a simple, serverless Pulumi program that periodically rotated the User’s access keys and updated the CI/CD system.

In this final post, we will use those AWS credentials to update cloud resources as part of a CI/CD workflow using Pulumi.

We will cover AWS’s IAM Roles and how they can be used to safely manage the access Pulumi has to your AWS account. We will also cover how sensitive data in your stack is stored within the Pulumi Service.

Assuming IAM Roles for Performing Updates

In the previous post, we automatically create and rotate the AWS access keys for an IAM User. So it seems natural to think that we should just use that same AWS access key to gain access to AWS resources. However, using the access keys for an IAM User has a few major drawbacks.

First, those access keys are long-lived. If we didn’t create that Pulumi program to rotate them automatically, the key would last forever. (Meaning that a bad actor, perhaps elsewhere, or far in the future, could take advantage of a compromised key.) Second, that access key could be used to do anything that the IAM User could. So you cannot differentiate the set of capabilities based on the task at hand. (For example, if updating the Pulumi stack website/staging, we might need a different set of permissions than if we were updating the stack website/production.)

The preferred alternative to using IAM User access keys is to “assume” an IAM Role. An IAM Role is similar to an IAM User in that it grants a set of policies and permissions. However, unlike an IAM User, an IAM Role does not have a password, nor can it be added to a Group and inherit any policies. Also, the credentials for using an IAM Role are always short-lived. (Lasting no more than 12 hours, and likely much, much less.)

Instead, an IAM Role defines a set of capabilities that can be performed by any principal capable of assuming it. So in other words, you can exchange your limited access IAM User’s credentials for a specialized IAM Role. The IAM Role then can be specifically tailored to a specific— wait for it— role to perform.

Don’t worry if that doesn’t all make sense at first; let’s look at a concrete example.

Desired End State

In our situation, we want to create an IAM Role WebsiteStackUpdaterRole that grants permissions necessary for Pulumi to update the website/production stack. We will custom tailor this role to only grant access to the set of resources and operations we know the stack needs.

We also want to restrict who can assume the WebsiteStackUpdaterRole, so that only specific users/accounts can access it. e.g., the IAM User cicd-bot but not IAM User dwight-over-in-sales.

And just to show a more real-world example, we’ll do this using multiple AWS accounts, where the website/production stack’s resources are housed in a “production” AWS account, and the cicd-bot IAM User is defined in a bastion AWS account.

Creating the IAM Role

The following snippet creates an IAM Role resource using Pulumi. The most important input property of which is the assumeRolePolicy document, which defines who can assume this role.

This is a situation where Pulumi’s ability to declare the policy document in code is super-useful, as it makes it much easier to reference other resources (e.g., another AWS account’s ID) as well as supporting inline comments.

// IAM Role used by cicd-bot to update the website stack.
export const websiteStackUpdaterRole = new aws.iam.Role("WebsiteStackUpdaterRole", {
 name: "WebsiteStackUpdaterRole",
 description: "Broad access for updating Pulumi stacks in the 'production' AWS account.",
 maxSessionDuration: 30 * 60, // 30 minutes
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: {
 // We grant the "bastion account", which defines IAM Users the ability to assume this role.
 // The IAM Policies in that other AWS account will then determine which users, groups, etc.
 // will be able to use the `sts:AssumeRole` action on this IAM Role.
 "AWS": `arn:aws:iam::${awsAccountIds.bastionAccount}:root`,
 },
 // But as an additional precaution against misconfiguration in the bastion account, we add
 // some additional checks to determine who can access this policy.
 Condition: {
 "ForAllValues:StringEquals": {
 "aws:PrincipalTag/access-to-production": "true",
 "sts:ExternalId": [
 "contoso/website/production",
 "contoso/website/staging",
 ],
 },
 },
 Action: "sts:AssumeRole",
 },
 ],
 },
});

There’s a lot packed into that IAM Role’s configuration.

First, the “principal” (who can assume the role) references a specific AWS account. This is an example of how you can associate one AWS account with another. (And is best practice for isolating data between different departments, etc.) We’ll cover how to configure the AWS bastion account side to use this role later.

The next thing to call out is the use of the Condition element of the policy document. To assume the role, we require two checks to be performed.

That the principal assuming the role (e.g., IAM User) has the tag "access-to-production" with value "true." This way, we will reject any IAM User in the bastion account who isn’t tagged with "access-to-production" from assuming this role.
We also verify that the “external ID” (an optional parameter passed when assuming an IAM Role) matches a well-known Pulumi stack name. This way, we can double-check that the entity using the IAM Role is doing so to use a specific stack. (This is less of a security measure and more of a general hygiene type requirement.)

IAM Policy Document

Now we have an IAM Role; we need to attach a policy document that defines what permissions are granted.

This is the most important step since it defines precisely what the CI/CD bot can able to do.

If we define the policy too broadly, then it is a general risk. For example, what if a compromised NPM package in your Pulumi program did something nefarious. If the policy is defined too narrowly, then your pulumi up might fail because the necessary permissions weren’t available. When in doubt, err on the side of having a more restrictive policy.

It can be advantageous to have an identical “staging” and “production” instance of the same Pulumi stack, so you can detect any permission issues before you update the user-facing production stack.

It can take some time to winnow down the specific set of permissions necessary since that is ultimately specific to the Pulumi program. For example, if the stack governs the networking configuration, then you can simply not deny to the s3:* actions.

Unfortunately, Pulumi doesn’t currently support generating a reasonable IAM Policy document for your stack. For a good introduction to some of the “worst-case scenarios” that could arise from a misconfigured IAM Policy, see this blog post from BishopFox.

The following is a hypothetical IAM policy that restricts the operations performed to just those for the AWS CloudFront and S3 products. (Which might be sufficient for updating a CDN-based website hosted on AWS, but likely not much else.)

// Policy document defining the permissions granted to the WebsiteStackUpdaterRole.
export const websiteUpdaterPolicyDocument: aws.iam.PolicyDocument = {
 Version: "2012-10-17",
 Statement: [
 // Grant access to all AWS S3 actions, but scoped to just the specific S3 buckets
 // used for the given Pulumi stack.
 {
 Sid: "GrantS3Access",
 Effect: "Allow",
 Action: "s3:*",
 Resource: [
 // Content bucket.
 "arn:aws:s3:::website-bucket",
 "arn:aws:s3:::website-bucket/*",
 // S3 bucket which stores the access logs.
 "arn:aws:s3:::website-logs-bucket",
 "arn:aws:s3:::website-logs-bucket/*",
 ],
 },
 // Grant access to update a CloudFront distribution. For more information, see:
 // https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/access-control-managing-permissions.html
 {
 Sid: "GrantCloudFrontAccess",
 Effect: "Allow",
 Action: [
 "cloudfront:CreateDistribution",
 "cloudfront:DeleteDistribution",
 "cloudfront:GetDistribution",
 "cloudfront:GetDistributionConfig",
 "cloudfront:ListDistributions",
 "cloudfront:UpdateDistribution",
 "cloudfront:ListCloudFrontOriginAccessIdentities",
 ],
 // The specific CloudFront distribution used in the stack.
 Resource: "arn:aws:cloudfront::0123456789:distribution/A123B1CABWZZ",
 },
 ],
};

const websiteUpdaterPolicy = new aws.iam.Policy("WebsiteUpdaterRolePolicy", {
 name: "WebsiteUpdaterRolePolicy",
 description: "Policy granting the permissions needed for updating the website/production Pulumi stack",
 policy: websiteUpdaterPolicyDocument,
}, resourceOptions);

To actually associate the IAM Policy document with the new IAM Role, we need to create a PolicyAttachment associating the two.

const rolePolicyAttachment = new aws.iam.RolePolicyAttachment("WebsiteStackUpdaterRolePolicyAttachment", {
 role: websiteStackUpdaterRole,
 policyArn: websiteUpdaterPolicy.arn,
});

Reminder, we defined the IAM Role in our production AWS account, which allows any principal in the AWS bastion account (with the right tag, external ID) to assume that role. We then need to update the bastion account to grant the sts:AssumeRole action on that IAM Role so it can be used! That isn’t covered in this post.

Assuming the IAM Role

Finally! With everyone in place, we can now use the IAM Role.

The following is a set of commands you can run on your CI/CD worker to exchange the low-privilege IAM User credentials for the specific IAM Role we just created in the production AWS account. But there are many alternatives for programmatically assuming an IAM Role, such as using the AWS_PROFILE environment variable) or the assume-role tool.

# Use the AWS credentials passed to the CI system and available as environment variables.
# These are the low-privilege credentials that automatically get rotated.
aws configure set aws_access_key_id ${AWS_ACCESS_KEY_ID}
aws configure set aws_secret_access_key ${AWS_SECRET_ACCESS_KEY}
aws configure set region "${AWS_REGION:-us-west-2}"

# Assume the desired IAM Role. Note how we pass the stack name in the --external-id parameter,
# as required to assume the role.
readonly CREDS_JSON=$(aws sts assume-role \
 --role-arn "${ARN_OF_IAM_ROLE_TO_ASSUME}" \
 --role-session-name "<useful description of the context, such as CI/CD job ID>" \
 --external-id "${TARGET_STACK}")

# The `aws sts assume-role` command just returns a JSON blob. So we now tease out the actual
# access key and set the corresponding environment variables.
export AWS_ACCESS_KEY_ID=$(echo "${CREDS_JSON}" | jq ".Credentials.AccessKeyId" --raw-output)
export AWS_SECRET_ACCESS_KEY=$(echo "${CREDS_JSON}" | jq ".Credentials.SecretAccessKey" --raw-output)
export AWS_SESSION_TOKEN=$(echo "${CREDS_JSON}" | jq ".Credentials.SessionToken" --raw-output)

# If everything worked correctly, then running the get-caller-identity command will refer to
# the IAM Role and not our IAM User.
echo "Assumed AWS identity:"
aws sts get-caller-identity

Storing Sensitive Data using Pulumi

As a developer, the most critical security threat is access to your cloud account since that’s ultimately where your or your customer’s data is stored. But beyond that, there are places where Pulumi stores sensitive data.

Pulumi goes to great lengths to safeguard your data. Here we review where Pulumi interacts with your sensitive data and what you can do to safeguard it better.

Secrets in Configuration

A Pulumi stack can have configuration data associated with it, which allows you to parameterize cloud resources. For example, a machine-size configuration key might determine which class of virtual machine to use. A dev stack might only need a "t2.small" but your production instance might need something more powerful.

Configuration data is stored in a file like Pulumi.website-production.yaml should be checked in along with the source code for a stack, so that builds and stack updates are reproducible.

However, what if those configuration settings contain secrets? Like you need to store an API key to use a 3rd party API.

Pulumi supports encrypting sensitive configuration data. You just need to add the --secret flag.

# Securely add an API key to the stack's configuration.
pulumi config set api-key "hunter2" --secret

For stacks hosted on the Pulumi Service, the default is that your configuration data is encrypted using a key specific to your stack. (So the ciphertext stored in the Pulumi.yaml file is safe to check into your source tree, since it cannot be copied/decrypted for another stack.)

Secrets in Checkpoint Files

Pulumi keeps track of your cloud resources in a something called a checkpoint file, and that too might contain sensitive information. For example, a Pulumi resource might have a "password" output property.

Pulumi has support to mark that resource output as “secret” and make sure that it is encrypted within the checkpoint file. (So if you were to look at the checkpoint file contents via pulumi stack export, you would not be able to recover the data.

Just like for secret configuration values, the default for stacks hosted on the Pulumi Service is to encrypt this data using a key that Pulumi manages and is specific to your stack.

Custom Secrets Providers

We at Pulumi take great care in safeguarding your data, especially configuration or checkpoint data that is marked as sensitive. However, you might feel more comfortable if your stack’s data were encrypted using a key that you controlled. (So even if the data stored in the Pulumi Service were available, it would be useless without your specific key.)

If that’s the case, then you can use a configurable secrets provider, and swap out the default “Pulumi Service managed” encryption scheme for your own. (And we won’t take it personally, promise.)

When you create a new stack using pulumi stack init, you can optionally specify a --secrets-provider flag. That will determine where and how secrets get managed on your stack.

For example, to use your own KMS key for encrypting data, you can pass the secrets provider "awskms://alias/ExampleAlias?region=us-east-1". The Pulumi command-line will work and behave the same. Except that whenever it needs to encrypt or decrypt some data, it will refer to the custom provider.

As you would expect, if you do use a custom secrets provider and lose access to your encryption key, then there is no way to recover the encrypted data stored on your stack.

For more information on custom secret providers, see Peace of Mind with Cloud Secret Providers or Managing Secrets with Pulumi.

Wrapping Up

Securely managing any sensitive data is challenging to get right, especially if there are many moving parts, such as when coordinating access between your AWS account, CI/CD provider, and your Pulumi stack.

In this blog series, we walked through the best practices in creating, securing, and managing access to an AWS account using IAM Users and Roles. We then reviewed the specifics of where and how Pulumi stores sensitive data, in case you want to take more control over your data.

If you have any other questions, corrections, or other ideas you’d like to discuss about Pulumi, Continuous Delivery, or security, please join the conversation over on the Pulumi Community Slack.

Managing AWS Credentials on CI/CD: Part 2

Chris Smith — Thu, 26 Mar 2020 00:00:00 +0000

This article is the second part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we go in-depth on providing AWS credentials securely to a 3rd party and introduce a Pulumi program to automate rotating access keys.

NOTE: These recommendations do not apply if you are running a CI/CD system within your AWS account, e.g., running a Jenkins server on EC2 or using AWS CodeDeploy. In those cases, please refer to AWS’s documentation for how to assume IAM Roles when running on an EC2 instance instead.

Provide IAM credentials to your CI/CD system

In the first post in our series, we created a dedicated IAM User to perform updates to AWS resources within your CI/CD system. The next step is to pass the AWS access keys for that user to your CI/CD system.

We need to take great caution. AWS’s documentation states, “Do not provide your access keys to a third party”.

Providing AWS access keys to any system is dangerous because of the risk that a bad actor could obtain those keys and do something nefarious. Even if your CI/CD service takes great care to protect your secrets, those AWS credentials could be inadvertently exposed in debugging output, system logs, or in some other way.

This is one of the main advantages of performing continuous deployment from within your AWS account. For example, running Jenkins on an EC2 instance you manage, or using the AWS CodeDeploy service. Because when performing a deployment from within an AWS compute environment, you can use built-in mechanisms to obtain credentials securely.

However, there are many reasons to use a hosted service for performing your CI/CD, such as developer productivity, ease of use, performance, or simply because you don’t want to recreate your existing deployment workflows.

Ultimately, it is essential to understand the risks involved and how to mitigate them.

Properly Storing Credentials

If you choose to provide your CI/CD with credentials, the most important thing is to mark them as secret. Your CI/CD provider typically has built-in support for handling “secure variables” or other sensitive information. As opposed to general, configuration data or environment variables.

User-supplied configuration values are typically write-only and only accessible by the CI/CD worker jobs at runtime. For example, Travis CI has encrypted environment variables or secure environment variables in CircleCI.

Please consult your CI/CD provider’s documentation for how to pass and store sensitive information appropriately.

Why not use a “secrets manager” service?

A common question when discussing is how to pass access keys to a CI/CD system securely:

Rather than giving your CI/CD provider AWS credentials, why not have your CI/CD system obtain credentials from a specialized “secrets manager” service?

In other words, if you choose not to trust your CI/CD system with this data, can you instead trust some other system dedicated for securely storing and retrieving sensitive information?

It sounds like a good idea on the surface but doesn’t seem to make your data any more secure.

Abstractly, the difference here is that the credentials are provided on-demand, rather than being available to the CI/CD job when it starts (and stored via the CI/CD provider). Instead, your CI/CD job would obtain credentials from the “secrets manager” only when needed.

There are some advantages to this approach, such as providing a clear audit trail for access and more control over the distribution of sensitive information.

However, the secrets provider system needs to be presented with some form of credentials. And those credentials need to be available to your CI/CD environment. So using a secrets manager leaves you in the same place you started, i.e., needing to provide sensitive data to your CI/CD provider.

Also, by adding a dependency on a secrets manager, you introduce additional risks. Not only do you need to be even more security-conscious about that secrets manager, but it also needs to be highly available. Any outage for that service would mean that you would be unable to perform deployments!

So it does not seem that using a secrets manager to dole out AWS credentials to your CI/CD system is a good practice to follow. Or, at the very least, makes some tradeoffs without fundamentally making your approach to CI/CD any more or less secure.

Instead, it is better to focus on other ways to reduce the risk if those AWS credentials do get exposed.

Automate Rotating and Revoking AWS Credentials

When providing credentials to a 3rd party, rather than hoping it will be 100% secure forever (which is impossible), we can make those credentials volatile. If we regularly invalidate and rotate the credentials supplied to the CI/CD system, we can dramatically reduce the impact of any accidental disclosure. Even if the IAM User’s credentials were leaked, by the time they were discovered and used, they would no longer be valid.

The AWS security blog describes how to rotate access keys for IAM Users, by using the following steps:

Create a second access key in addition to the one in use.
Update all your applications to use the new access key.
Change the state of the previous access key to inactive.
Verify everything is still working as expected.
Delete the inactive access key.

That all sounds simple enough, but it’s certainly tedious. And if you need to repeat the process across dozens, if not hundreds of different CI/CD pipelines, you need a better solution. Fortunately, Pulumi provides an excellent and extensible way for writing a serverless program to automate rotating credentials.

The next few sections describe a simple infrastructure application for automating AWS IAM credential rotation. You can see the full application on GitHub at chrsmith/pulumi-aws-travis-cicd-demo.

AWS Credential Rotator 9000

The affectionally titled AWS Credential Rotator 9000 is a simple, serverless Pulumi application for rotating AWS access keys.

It creates an AWS lambda function that’s triggered on a regular schedule, e.g., every 30 minutes, and performs the next step in the sequence for rotating access keys as outlined above.

First, it creates a new access key and pushes the new value out. On the next iteration, it marks the older access key as “inactive.” On the following iteration, the inactive key is deleted. The process repeats, generating a new key and removing the inactive key.

Periodically Invoking an AWS Lambda

The heart of the application is triggering it to execute on a fixed interval. Thankfully this is super-easy to do using Pulumi since it allows you to seamlessly blend your “cloud infrastructure” with “code” in a natural way. The user guide for Pulumi Crosswalk for AWS has more information on serverless eventing if you would like to learn more.

The following snippet is the core part of the credential rotator app. We define a function to handle the logic of key rotation in rotateIAMUserKeys. Then we create an AWS Lambda resource named lambda. Finally, the triggerSchedule resource invokes our lambda on a fixed schedule, thereby ensuring that the key rotation process goes on indefinitely. You can see the full code here.

async function rotateIAMUserKeys(iamUser: aws.iam.User, onNewCredentials: (key: string, secret: string) => Promise<void>) {
 ...
}

const lambda = new aws.lambda.CallbackFunction<aws.cloudwatch.EventRuleEvent, void>(
 "keyRotatorLambda",
 {
 callback: async (e) => {
 await rotateIAMUserKeys(userToRotate, async (newKey: string, newSecret: string) => {
 console.log("onNewCredentials callback called!");
 console.log(`A new AWS access key "${newKey}" has been created.`);
 });
 },

 role: lambdaRole,
 runtime: "nodejs10.x",
 });

const triggerSchedule = aws.cloudwatch.onSchedule(
 "keyRotatorScheduler", "rate(1 hour)", lambdaCallbackFn);

When writing reusable infrastructure components in Pulumi however, it is helpful to organize things into a custom resource.

For example, we can bundle together the AWS Lambda, CloudWatcn schedule, and the associated IAM policies into a single conceptual resource AccessKeyRotator. Bundling resources allows for the code reuse.

const rotator = new AccessKeyRotator("rotator", {
 interval: config.require("rate"),
 user: user,
 credentialPusher: demoTravisCIPusher,
})

Pushing Credentials

With the mechanics of updating an AWS access key out of the way, the next step is to notify dependent systems what the new access key should be. The key rotator app has a CredentialPusher abstraction to provide a pluggable way for you to send credentials to where they need to be. The example on GitHub only supports updating a Travis CI project (See credential-pusher-travis.ts.), but it could be easily extended to support other CI/CD systems.

The following snippet shows how the CredentialPusher is configured in the key rotator app. We load a Travis CI access token from the Pulumi stack’s configuration, and hard-code the specific set of projects and encrypted environment variables to store the new access key.

// A "credential pusher" is the component that pushes new AWS IAM credentials out to 3rd parties
// as the older ones get rotated. For demonstration purposes, this updates the Travis CI settings
// for the chrsmith/pulumi-aws-travis-cicd-demo repo. But you can imagine another implementation
// that pushes the new IAM credentials to GitLab CI, or updating multiple CI/CD pipelines.

const demoTravisCIPusher = new CredentialPusher(
 new TravisCIPusher(),
 {
 accessKey: config.require("travis-ci-token"),
 projects: [
 {
 project: "chrsmith/pulumi-aws-travis-cicd-demo",
 // In the Travis CI configuration for that GitHub repo, there are two
 // environment variables for storing the AWS credentials. So whenever the
 // AWS credentials get rotated, the job's configuration settings will be
 // updated to reflect the new values.
 accessKeyIDLocation: "AWS_ACCESS_KEY_ID",
 secretAccessKeyLocation: "AWS_SECRET_ACCESS_KEY",
 }
 ],
 });

Demo

To demonstrate the access keys rotation, we can examine the log files generated from AWS Lambda. These can be accessed from the command-line using the pulumi logs command.

Here’s a summary of the output for clarity:

 START RequestId: 9406913e-a9d0-46a4-b861-2efdee508b2b Version: $LATEST
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO IAM User has 2 keys:
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO - AKIASHIVKXX3SA2K3ME7 [Inactive] Tue Mar 24 2020 09:37:03 GMT+0000 (Coordinated Universal Time)
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO Deleting older, inactive access key AKIASHIVKXX3SA2K3ME7 [Inactive]
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO Key rotation step complete.
 END RequestId: 9406913e-a9d0-46a4-b861-2efdee508b2b

 START RequestId: 05644e4a-81a4-4197-8ad7-fc3517998270 Version: $LATEST
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO IAM User has 1 keys:
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Created new key key AKIASHIVKXX3YX4IXRVH
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Pushing out the new key to 3rd party services...
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Pushing new credentials to Travis CI project 'chrsmith/pulumi-aws-travis-cicd-demo'
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updating env var 'AWS_ACCESS_KEY_ID' (b246ab3b-c4cb-e76f-a6fc-50f4cbf451c0)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updated AWS access key ID. Got response code (200)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updating env var 'AWS_SECRET_ACCESS_KEY' (ba9ac4be-fbb3-4da9-c5b8-35a1afe9e02b)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updated AWS secret access key. Got response code (200)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Key rotation step complete.
 END RequestId: 05644e4a-81a4-4197-8ad7-fc3517998270

 START RequestId: ca204959-32a7-44a8-ba20-738d139542ba Version: $LATEST
 ca204959-32a7-44a8-ba20-738d139542ba INFO IAM User has 2 keys:
 ca204959-32a7-44a8-ba20-738d139542ba INFO - AKIASHIVKXX3YX4IXRVH [Active] Tue Mar 24 2020 15:37:03 GMT+0000 (Coordinated Universal Time)
 ca204959-32a7-44a8-ba20-738d139542ba INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 ca204959-32a7-44a8-ba20-738d139542ba INFO Invalidating older access key AKIASHIVKXX3QEQNZNX5
 ca204959-32a7-44a8-ba20-738d139542ba INFO Key rotation step complete.
 END RequestId: ca204959-32a7-44a8-ba20-738d139542ba

 START RequestId: de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 Version: $LATEST
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO IAM User has 2 keys:
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO - AKIASHIVKXX3YX4IXRVH [Active] Tue Mar 24 2020 15:37:03 GMT+0000 (Coordinated Universal Time)
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO - AKIASHIVKXX3QEQNZNX5 [Inactive] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO Deleting older, inactive access key AKIASHIVKXX3QEQNZNX5 [Inactive]
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO Key rotation step complete.
 END RequestId: de753fb4-0720-4f4f-a7aa-ad6b9d4c8832

As you can see, the AWS credentials for the IAM User are automatically updated every few hours. And whenever a new access key is created, the value is pushed out to the impacted Travis CI projects automatically.

Wrapping Up

In this post, we covered some of the things to consider when providing AWS credentials to your CI/CD system. (In short, be cautious and follow best practices.)

We then showed how relatively easy it is to stand up a Pulumi infrastructure application for automatically rotating AWS access keys and updating your CI/CD system. By rotating credentials, you can limit the impact if the value is inadvertently disclosed.

At this point, we’ve now given access keys to a CI/CD system for a low-privilege AWS IAM User account. But we still can’t do anything with it. The IAM User whose credentials we have given to the CI/CD system doesn’t have access to your production AWS account.

In the next post, we’ll go into the details about IAM Roles, and they can be used to securely and temporarily gain access to additional resources. This is how our low-privilege IAM User can access needed to update production data for your CI/CD pipelines.

Managing AWS Credentials on CI/CD: Part 1

Chris Smith — Thu, 12 Mar 2020 00:00:00 +0000

Continuous delivery requires providing highly sensitive credentials to your deployment pipeline. Understanding the risks, mitigations, and best practices for handling those credentials can be difficult. In this guide, we describe the best practices for providing AWS credentials to a CI/CD system and to securely automate updating your cloud infrastructure using Pulumi.

You’ve heard about Infrastructure as Code and decided to try your hand at using Pulumi to manage cloud infrastructure using familiar programming languages. Naturally, you’re curious and eager to start updating your cloud infrastructure in your CI/CD pipeline – but then panic hits… Is updating your cloud infrastructure secure? If you start doing cloud deployments in your CI/CD system, doesn’t that mean you need to give it your production credentials? Aren’t those the most sensitive things your company has?!?

Take a deep breath. It’s going to be OK. You can securely provide AWS credentials to your CI/CD system and practice continuous delivery. This is the first post in a series going in-depth on how to do just that.

The goal of this article series is to give you a clear understanding of AWS credential management and how that relates to using Pulumi within a CI/CD environment. Once you have the AWS credentials in-place, you can then follow our Continuous Delivery guide for configuring your specific CI/CD service, whether you want to use CircleCI, GitLab CI, or Travis CI.

NOTE: These recommendations do not apply if you are running your own CI/CD system within your AWS account, e.g., running a Jenkins server on EC2 or using AWS CodeDeploy. In those cases, please refer to AWS’s documentation for how to assume IAM Roles when running on an EC2 instance instead.

Overview

The recommendations in this series describe a general “one-size fits most” approach for credential management, which requires a minimal amount of work to configure and maintain. Depending on your specific environment, needs, and constraints, there may be a better alternative for your use case.

Create a dedicated IAM User for your CI/CD

The first step for securely automating CI/CD is to create a dedicated IAM User for use in your CI/CD pipelines. (Sometimes referred to as a “robot account.”)

The following code snippet shows how to create a new AWS IAM User using Pulumi. (The code is in TypeScript, but you could another language like Python, C#, or Go as well.)

const user = new aws.iam.User("cicdUser", {
 name: "cicd-bot",
 tags: {
 "purpose": "Account used to perform Pulumi stack updates on CI/CD.",
 }
});

By default, that IAM User doesn’t have permissions to do anything. The recommended best practice is to use groups to assign permissions to IAM Users. So next, we will create a “Pulumi Stack updaters” IAM Group, and add our robot account as a member.

const group = new aws.iam.Group("pulumiStackUpdaters", {
 name: "PulumiStackUpdaters",
});

const groupMembership = new aws.iam.GroupMembership("cicdUserMembership", {
 group: group.name,
 users: [ user.name ],
});

Group Permissions

The next part is where things start to get tricky. We now need to grant permissions to this IAM Group’s members so they can access various APIs and resources. (For example, if your Pulumi program will need to create or update EC2 instances, then you will need to provide AWS credentials that contain the ec2:StopInstances or ec2:StartInstances action.)

But rather than grant the newly created cicd-bot IAM User those permissions, we will instead only grant them access to the sts:AssumeRole action. This is an AWS API that allows an IAM User to gain temporary credentials that have a different set of permissions associated with them. (That is, the permissions associated with an IAM Role.)

We’ll go into more detail about the exact difference between an IAM User and an IAM Role in the next post in the series since it isn’t entirely obvious why one would be more secure than the other.

const currentAwsIdentity = aws.getCallerIdentity();

const groupPolicy = new aws.iam.GroupPolicy("pulumiStackUpdatersPolicy", {
 group: group.name,
 policy: {
 Version: "2012-10-17",
 Statement: [{
 Action: [
 // Allow anybody (i.e., members of the group) to call the sts:AssumeRole API.
 // This allows them to "assume the role" of a more permissive IAM Role
 // when they go to update a stack later.
 "sts:AssumeRole",
 ],
 Effect: "Allow",
 // This is the set of resources that the "sts:AssumeRole" operation could be
 // performed on, which is to say any IAM role in the current AWS account.
 Resource: pulumi.interpolate `arn:aws:iam::${currentAwsIdentity.accountId}:role/*`,
 Sid: ""
 }],
 },
});

That’s it for the first post in our series. We’ve taken the first step and created an IAM User specifically for performing updates to AWS resources using Pulumi.

In the next post, we will review over the security concerns when passing AWS credentials to your CI/CD provider, as well as show a Pulumi program that you can use to rotate the AWS credentials for a user automatically.

Pulumi Service Improvements, February 2020

Chris Smith — Mon, 02 Mar 2020 00:00:00 +0000

We’ve been hard at work making it easier to manage stacks, permissions, and organizations in the Pulumi Service. Adding new features like first-class support for stack tags, deep links into CI/CD providers, and downloadable checkpoint files.

In this post, we showcase what’s new!

First-class Support for Tags

Pulumi has had support for stack tags for a while, enabling you to add attributes to your stacks with custom data such as the account-id or environment. But previously the data was only available on the command-line, via the pulumi stack tag command (documentation.

We’ve added first-class support for stack tags in the Pulumi Service as well. You can create, update, and delete tags from within the console.

Making it easier to create and edit tag data is just the beginning. We’ve started to add new search/filtering capabilities based on stack tags. For example, you can now use filters to rapidly add stacks to Pulumi teams.

We will continue wiring stack tags throughout other parts of the Pulumi Service, so stay tuned! Also, this new feature was added by Pulumi’s first (and extremely awesome) intern, Tasia (👋)!

Deep Linking into CI/CD Systems

Many people practice continuous delivery using Pulumi, automatically updating stacks in response to code check-ins or merges. We’ve now added support for deep links from the Pulumi Service back into your CI/CD tool of choice. So, for example, you’ll now see links to the Circle CI job or Travis CI build from the Pulumi Service.

Pulumi supports a variety of CI/CD providers, but if yours isn’t listed in our CI/CD guide let us know or contribute it on your own.

Pretty Printing JSON / Multi Configuration

We now pretty print JSON configuration values and have much better support for viewing multi-line data.

Download Earlier Checkpoints

The most important job of the Pulumi Service is to maintain a durable, accurate snapshot of your cloud resource data. While in most cases your stack’s checkpoint data is a low-level detail you don’t need to worry about; in some advanced scenarios, you may need to inspect or edit it manually.

You can now download a stack’s checkpoint file directly from the Pulumi Service. You can get the same data from the command-line, using pulumi stack export (documentation, which now supports a --version flag to export older checkpoint files too.

Reverse Stack Permissions View

The Enterprise Edition supports role-based access control using teams. But a common problem we’ve heard from people in large organizations is that it can be difficult to review exactly what access someone has to a stack and why.

Fortunately, the Pulumi Service now has the ability for organization administrators to see all stacks a particular member has access to.

They can even go one step further, review the specific access granted from various sources.

Log Rendering Performance

We’ve made some changes that dramatically improve the Pulumi Service’s performance when rendering large update logs.

Audit Logs

Pulumi Enterprise customers typically have many stacks and teams. Multiple teams and stacks can make it difficult to review what’s currently in-motion and to understand a sequence of events a few days or few weeks in the past.

As we announced earlier we launched the ability to view and download audit logs with all the changes within an organization.

Conclusion

Phew! As I said, we’ve been busy this month working on the Pulumi Service. And there is plenty more awesomeness still in the pipe! If you want to get a peek at other things in-development, check out the Pulumi 2.0 Roadmap.

As always, we love to hear if you have any feedback, suggestions, or ideas for other improvements we could make to the Pulumi Service. Just let us know on the Pulumi Community Slack, Twitter @PulumiCorp, or on GitHub.

Getting to ChatOps with Pulumi Webhooks

Chris Smith — Wed, 23 Jan 2019 00:00:00 +0000

Today we are delighted to announce the availability of Webhooks on Pulumi. Webhooks are a very common mechanism to enable teams to be notified or react to events. In Pulumi’s case, this means: notifications of infrastructure changes (be it on Kubernetes, AWS, or any other cloud); responding to those changes as part of ‘ChatOps’; or other build pipelines, to improve the delivery of cloud native infrastructure.

Pulumi Webhooks are available for the Team and Enterprise editions of Pulumi. If you’re keen to try them out, start a trial of Team Edition here.

ChatOps — the idea of conversation-driven collaboration — is an increasingly common pattern, and as such we’ve gotten a lot of requests to integrate Pulumi with cloud services such as Slack, GitHub, and Twillio to facilitate that. These requests are rooted in users wanting to have more awareness of changes being made to their cloud infrastructure or wanting to build extensions to automate continuous delivery pipelines. Pulumi Webhooks offer the flexibility to design these notification and response systems as required for your team.

Using Pulumi Webhooks

You can attach webhooks directly to stacks or organizations. Stack webhooks will send notifications whenever changes to a stack are made (pulumi up) or simply previewed (pulumi preview). Webhooks attached to an organization will be sent the events for every stack within the organization.

When events occur, we’ll send a HTTP POST request to any registered listeners. Webhooks can then be used to send notifications to Slack or SMS, start running automated tests, or maybe even update another stack!

We’ve been using Pulumi webhooks internally for raising awareness when our test and production environments get updated. A common sight after submitting some code is:

Building a Pulumi Webhook handler using… Pulumi!

Pulumi is perfect for setting up cloud infrastructure, and so what better tool to use for handling notifications from Pulumi than Pulumi!

In this example, we’ll create a simple Pulumi program that stands up a webhook handler that will receive webhook requests and echo them to Slack. You can find the full code in the examples repo, under aws-ts-pulumi-webhooks.

The code is very straight forward, thanks to the @pulumi/cloud package.

import * as pulumi from "@pulumi/pulumi";
import * as cloud from "@pulumi/cloud";

import * as slack from "@slack/client";

const config = new pulumi.Config();

const stackConfig = {
 slackToken: config.require("slackToken"),
 slackChannel: config.require("slackChannel"),
};

const webhookHandler = new cloud.HttpEndpoint("pulumi-webhook-handler");

webhookHandler.get("/", async (_, res) => {
 res.status(200).end("🍹 Pulumi Webhook Responder🍹");
});

webhookHandler.post("/", async (req, res) => {
 const webhookID = req.headers["pulumi-webhook-id"];
 const webhookKind = req.headers["pulumi-webhook-kind"];
 console.log(`Received webhook from Pulumi ${webhookID} [${webhookKind}]`);

 const payload = <string>req.body.toString();
 const prettyPrintedPayload = JSON.stringify(JSON.parse(payload), null, 2);
 const client = new slack.WebClient(stackConfig.slackToken);
 await client.chat.postMessage({
 channel: stackConfig.slackChannel,
 text: `Pulumi Service Webhook (`${webhookKind}`)` + "```" + prettyPrintedPayload + "```",
 as_user: true,
 });

 res.status(200).end(`posted to Slack channel ${stackConfig.slackChannel}`);
});

export const url = webhookHandler.publish().url;

But to stand up the cloud application, we’ll need to create the stack and set some configuration. And finally deploy it using pulumi up.

$ pulumi stack init pulumi-webhook-handler
Created stack 'pulumi-webhook-handler'
$ pulumi config set cloud:provider aws
$ pulumi config set aws:region us-west-2
$ pulumi config set slackChannel #ops-notificatinos
$ pulumi config set slackToken ****** --secret
$ pulumi up --yes
Updating (robot-co/pulumi-webhook-handler):
... snip ...
Outputs:
url: "https://6tfwcc3vg3.execute-api.us-west-2.amazonaws.com/stage/"
Resources:
~ 1 updated
15 unchanged
Duration: 4s
Permalink: https://app.pulumi.com/robot-co/deploytron-9000/updates/6

The stack emits the URL to the webhook handler in the stack’s outputs. That URL is what will receive and process Pulumi webhooks.

With the infrastructure in-place, we just need to register the webhook in the Pulumi Service. Organization administrators can do so under the organization’s SETTINGS tab.

That’s it! As stacks are created, updated, and so on, the webhook handler will be notified and you can start building any custom actions you can think of.

There are three kinds of webhook events.

Event Type	Trigger
`stack`	Fired whenever a stack is created or deleted within an organization.
`stack_update`	Fired when a stack is updated. (Be it from `pulumi up`, `pulumi refresh`, or `pulumi destroy`.)
`stack_preview`	Fired whenever changes to a stack are previewed.

Tell us what you think

Get started with Pulumi Webhooks on the Pulumi app, and let us know how you’re using them so we can continue to extend their capabilities. We’d love to hear your ideas, as well as any feedback you have on the Pulumi Community Slack.

Reusable CI/CD components with CircleCI Orbs for Pulumi

Chris Smith — Wed, 07 Nov 2018 00:00:00 +0000

This morning CircleCI announced the launch of CircleCI Orbs which enable you to create reusable components for CircleCI workflows. Orbs enable you to simplify your CI/CD configuration by reusing existing orb jobs or commands, in much the same way Pulumi enables you to simplify the delivery of your cloud native infrastructure by sharing and reusing existing components.

Pulumi is proud to be a CircleCI technology partner, and we were excited to get a head start on seeing how orbs could make it easier to take Pulumi into production within CircleCI. The Pulumi Orbs for CircleCI are available today for you to start using.

Pulumi Orbs

You can see the Pulumi orbs in-action in the following CircleCI config.yml. It’s a simple job to build and update a JavaScript-based stack.

version: 2.1
Orbs:
pulumi: pulumi/pulumi@1.0.0
jobs:
build:
docker:
- image: circleci/node:7.10
working_directory: ~/repo
steps:
- checkout
- pulumi/login
- run:
command: |
npm install
npm run build
- pulumi/update:
stack: website-prod

First, the Pulumi orbs (and fixed version) are referenced by the name pulumi/pulumi@1.0.0. Then, the two Pulumi orbs pulumi/login and pulumi/update are invoked.

CircleCI orbs can be parameterized, so you can set the version of the Pulumi client to download, whether the update should skip its preview, and so on. (Full orb reference)

With other CI/CD systems, you’d typically need to write one-off Bash scripts to download the client from https://get.pulumi.com, add it to the current $PATH and so on. Orbs allow for simpler, more declarative CI/CD workflows.

The source code for the Pulumi orbs is available on GitHub, if you have any suggestions or feedback.

Other Integrations

Of course Pulumi’s integration with CircleCI doesn’t end with orbs.

Newer versions of the Pulumi command-line client will start associating CircleCI build metadata with stack updates, so you you will soon see links from app.pulumi.com to any stack updates or previews that happened during a CircleCI workflow.

Also, if you are using CircleCI with GitHub, you can consider installing the Pulumi GitHub application. The Pulumi GitHub app will surface the results of any previews or updates from your CI/CD on the source GitHub pull request. It’s always good to know if a pull request is going to lead to changes to your cloud infrastructure!

We are continuing to make Pulumi the best tool for the “Continuous Deployment” part of CI/CD workflows, and with the release of CircleCIs Orbs, it’s just that much easier.

Having trouble? Questions? Join our community Slack or drop us a line.

Links:

Get Started with Circle CI Orbs

Creating and Reusing Cloud Components using Package Managers

Chris Smith — Thu, 09 Aug 2018 00:00:00 +0000

Hello! A few weeks back I wrote a post on serving static websites on AWS with Pulumi detailing how to host a static website on AWS. Pulumi allowed me to wire four different AWS products together in only 200 lines of code. It would be a shame, however if I needed to copy and paste that code every time I wanted to to stand up a new website. Instead, we can package up, share, and reuse our code just like any other Node.js library. It just so happens that this one can be used to create cloud infrastructure.

Creating a Package

To create a reusable package for the static website example, I simply put the code into a new GitHub repo and publish it to NPM. We’ll walk through exactly how to do that below, but afterwards anyone who wants to stand up a static website on AWS can just reuse that package.

The static-website-aws package exports a StaticWebsite class, which is a logical, reusable cloud “component” resource. This class offers a simple constructor that internally does all of the hard work of creating the assortment of AWS cloud resources in the prior blog post – an S3 Bucket and objects per piece of static content, CloudFront Distribution, and Route53 DNS A-Record – and exposes them afterwards as readonly properties:

/**
 * Static website using Amazon S3, CloudFront, and Route53.
 */
export declare class StaticWebsite extends pulumi.ComponentResource  {
 readonly contentBucket: aws.s3.Bucket;
 readonly logsBucket: aws.s3.Bucket;
 readonly cdn: aws.cloudfront.Distribution;
 readonly aRecord?: aws.route53.Record;

 /**
    * Creates a new static website hosted on AWS.
    * @param name  The _unique_ name of the resource.
    * @param contentArgs  The arguments to configure the content being served.
    * @param domainArgs  The arguments to configure the domain and DNS settings.
    * @param opts  A bag of options that control this resource's behavior.
    */

 constructor(name: string , contentArgs: ContentArgs,
 domainArgs?: DomainArgs, opts?: pulumi.ResourceOptions);
}

This class is placed into an NPM package as usual, including a package.json file that gives it a name and a version. After doing this, we can publish it to NPM simply by running

$ npm publish

All the complexities of creating, configuring, and wiring up the necessary AWS resources – admittedly only 200 lines of code, but still intricate – has now been hidden beneath an ultra-simple component resource named StaticWebsite.

That’s how you create a new package. The great thing about packages, however, is that you define them once and then use them a lot. So now let’s turn to see see how standing up a new CDN-backed website can now become just a few lines of very simple, understandable code.

Reusing Infrastructure as Code

To use this new package, head on over to a Pulumi program, or create a new one (for instance, with pulumi new aws-typescript). Then just add a reference like any other dependency:

$ npm install static-website-aws

At that point, we can import our StaticWebsite class from the static-website-aws package like usual in Node.js programs, either using require

let swa = require("static-website-aws");

or using the new ES6 module import syntax, supported by TypeScript and modern JavaScript

import { StaticWebsite } from "static-website-aws";

Afterwards, we’ll just new up a StaticWebsite object, which internally creates everything for us. Of course, to create a useful website, we’ll need to supply some interesting parameters . In this case, let’s just pass a path to the content on disk (which will automatically get uploaded into S3 objects during a pulumi up) in addition to a path to a custom 404 HTML page:

import  { StaticWebsite } from "static-website-aws";

const website  = new StaticWebsite ("browserhack", {
 pathToContent:"./browserhack",
 custom404Path:"/404.html",
});

export const cloudfrontDomain = website .cdn .domainName ;

As soon as we have our program, we can stand it up with a single pulumi up command:

$ pulumi up

This will show us the entire resource graph so that we can see what it’s creating internally and once it’s done the resulting domain name will be printed out:

Let’s then curl it to see whether it worked:

$ curl $(pulumi stack output cloudfrontDomain)`[<!doctype html>
]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}[<!--
vim: set sw=2 ts=2 et : -->
]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}[<html>
]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}[
 <head>
]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}[...
etc, etc, etc
...]{style="color: red; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 13px;"}

Huzzah! (Note that new CloudFront Domains can take a while to spin up; if you get an error “Could not resolve host: xxx.cloudfront.net”, just wait a little bit and try again.)

The full source code for this example is available at chrsmith/browserhack-demo. This program deploys an instance of BrowserHack (a web-based port of the seminal console game NetHack). The repo also includes an example of assigning a custom domain name and ACM-managed SSL certificate, useful capabilities supported by StaticWebsite but omitted here for brevity .

Resource Components

We glossed over what components are earlier on. This is an advanced concept, so feel free to skip this section; but if you want to understand how components work under the hood, read on.

The StaticWebsite type extends pulumi.ComponentResource. A ComponentResource is a Pulumi-managed resource that aggregates other resources into a higher-level abstraction.

Pulumi will manage creating and updating the underlying resources like normal, and you will see updates to them in the resource tree. The difference is that by using the ComponentResource abstraction we can present those resources as logically grouped into a new, static website type.

This parent/child relationship between resources is declared using the optional pulumi.ResourceOptions type passed to Pulumi resources.

Here’s a code snippet from the code, where we create the S3 bucket and specify its logical parent resource. Every child resource created is passed defaultResourceOptions, which sets the parent property to this (the instance of StaticWebsite ).

// Default resource options for this component's child resources.
const defaultResourceOptions: pulumi.ResourceOptions = { parent:this };

...

// Create the logs bucket to store CloudFront request logs.

this.logsBucket = new aws.s3.Bucket(`${name }-logs`,
 { acl:"private" },
 defaultResourceOptions,
);

Components are a powerful construct in Pulumi, and let you build larger first class resources out of smaller ones. There are other benefits, including seeing resource attribution in the CLI tree view, and the richer visualization in the Pulumi Service:

If you want to learn more about components, see the documentation.

Package Everything!

In this blog, you’ve seen how Pulumi’s code-centric approach to infrastructure can make you more productive programming the cloud. “Code” has a lot of advantages over static configuration languages, for example rich IDE support. Another big advantage of using code, however, is that it allows you to create useful abstractions that can be reused in other programs

With Pulumi being able to take advantage of existing package managers, it’s super easy to create high-quality reusable components. We demonstrated NPM here for Node.js programs, but Pulumi also supports PyPI for Python, and traditional library techniques for Go.

But don’t just take our words for it. Recently Mikhail Shilkov blogged about creating a Pulumi component to keep AWS Lambda functions warm. And James Nugent open sourced a component that creates a properly subnetted AWS VPC. This is a great way to encode and share best practices broadly. Over time, we look forward to seeing the many other battle-tested components the community creates for common cloud applications and infrastructure needs.

If you’ve created a Pulumi component that is useful or want some design advice, come and join us in the Pulumi Community Slack – we’d love to hear from you!

Serving a Static Website on AWS with Pulumi

Chris Smith — Tue, 17 Jul 2018 00:00:00 +0000

Hello! This post covers using Pulumi to create the infrastructure for serving a static website on AWS. The full source code for this example is available on GitHub.

Setting up the infrastructure to serve a static website doesn’t sound like it would be all that difficult, but when you consider HTTPS certificates, content distribution networks, and attaching it to a custom domain, integrating all the components can be quite daunting.

Fortunately this is a task where Pulumi really shines. Pulumi’s code-centric approach not only makes configuring cloud resources easier to do and maintain, but it also eliminates the pain of integrating multiple products together.

This isn’t a hypothetical benefit of using the Pulumi programming model. We use a setup similar to the one described in this post for powering our own static websites, like www.pulumi.com and get.pulumi.com.

Overview

The architecture we will use for the website is to follow AWS “Web Application Hosting” reference architecture (pdf). This integrates several AWS products:

Amazon S3, used to store the website’s contents
Amazon CloudFront, a CDN improving site performance
Amazon Route53, for managing DNS
Amazon Certificate Manager, for serving content over HTTPS

How it Works

If you are already familiar with static hosting on AWS, feel free to go directly to our examples repository on GitHub, in the aws-ts-static-website folder.

If you are new to AWS, we’ll break down how to configure each component using Pulumi.

Uploading static content to S3

Amazon S3 is a service for storing and retrieving files. This is where we will store the website’s contents, and it is done by just creating an S3 bucket resources and then crawling the www-directory when the Pulumi program executes and creating an S3 bucket object resource for each file.

The main thing to point out here is that we set the contentType property for each bucket object. This way, the right HTTP headers are returned so that files are interpreted the right way. (e.g. images are rendered as images.) The process for inferring the MIME type based on file extension is done by the mime NPM package. No need to write the code on your own, rather we can just reuse an existing library - written in a different programming language, no less! - from our TypeScript application.

const contentBucket = new aws.s3.Bucket("contentBucket",
{
...
});
...
const webContentsRootPath = path.join(process.cwd(), config.pathToWebsiteContents);
crawlDirectory(
webContentsRootPath,
(filePath: string) => {
const relativeFilePath = filePath.replace(webContentsRootPath + "/", "");
const contentFile = new aws.s3.BucketObject(
relativeFilePath,
{
key: relativeFilePath,
acl: "public-read",
bucket: contentBucket,
contentType: mime.getType(filePath) || undefined,
source: new pulumi.asset.FileAsset(filePath),
},
{
parent: contentBucket,
});
});

With the S3 bucket populated with the website’s contents, we need to be able to serve it.

S3 supports the ability to serve a website directly, but S3 when used alone doesn’t support serving the content over HTTPS. Also, by serving the content via a content distribution network, like Amazon CloudFront, content can be served much faster by caching resources across the world. Rather than requiring every web request to go directly to the source S3 bucket.

Setting up a CloudFront CDN backed by S3

There are a lot of details to configuring a CloudFront distribution, from caching policies to rendering custom error pages. While some of these details cannot be avoided, Pulumi programs being just code, allows for using using constants like tenMinutes rather than the number 600.

const distributionArgs: aws.cloudfront.DistributionArgs = {
enabled: true,
aliases: [ config.targetDomain ],
...
// A CloudFront distribution can configure different cache behaviors based on the request path.
// Here we just specify a single, default cache behavior which is just read-only requests to S3.
defaultCacheBehavior: {
targetOriginId: contentBucket.arn,
viewerProtocolPolicy: "redirect-to-https",
allowedMethods: ["GET", "HEAD", "OPTIONS"],
cachedMethods: ["GET", "HEAD", "OPTIONS"],
forwardedValues: {
cookies: { forward: "none" },
queryString: false,
},
minTtl: 0,
defaultTtl: tenMinutes,
maxTtl: tenMinutes,
},
...
};

Attaching it to a domain via Route53

Finally, we hook up our domain to the CloudFront distribution. We just create an alias (A) record that aliases our own domain (e.g. www.pulumi.com) to the CloudFront distribution (e.g. dhy4niicdm7ba.cloudfront.net).

There is a little extra processing we need to do to get the Amazon Route53 Hosted Zone ID for the domain. But we can do that directly in the Pulumi program, and taking advantage of TypeScript’s async/await support.

// Creates a new Route53 DNS record pointing the domain to the CloudFront distribution.
async function createAliasRecord(
targetDomain: string, distribution: aws.cloudfront.Distribution): Promise {
const domainParts = getDomainAndSubdomain(targetDomain);
const hostedZone = await aws.route53.getZone({ name: domainParts.parentDomain });
return new aws.route53.Record(
targetDomain,
{
name: domainParts.subdomain,
zoneId: hostedZone.zoneId,
type: "A",
aliases: [
{
name: distribution.domainName,
zoneId: distribution.hostedZoneId,
evaluateTargetHealth: true,
},
],
});
}
const aRecord = createAliasRecord(config.targetDomain, cdn);

When the Pulumi program is run, the DNS record is created and after a few minutes to allow for worldwide propagation, the website is live. No need to manually log into the AWS console, enter various DNS records, etc. You can create and populate the S3 bucket, setup the CloudFront CDN, and attach it to Route53 all within the same Pulumi program.

Wrapping Up

With around 200 lines of code we were able to integrate four different AWS products using Pulumi to host a static website, served over HTTPS and from a world-wide CDN. Of course there are other ways to host static websites too, and some products or services can do all of that without needing any code at all.

But the benefit of using Pulumi for this is that you are in control of your infrastructure. If later you need to add more functionality, e.g. require authentication or serving some routes dynamically, you can just write a little more code to configure CloudFront. If you wanted to setup a testing or staging environment on a different domain, that would just be a matter of running the same Pulumi program in a different stack.

Pulumi opens up a lot of possibilities and we are excited to see what sorts of things people build using it. If you are interested in using Pulumi for more sophisticated website hosting, or just have questions about serving static files like described here, feel free to ask away on our Pulumi Community Slack.