Pulumi Blog: Managing AWS Credentials on CI/CD Series

Managing AWS Credentials on CI/CD: Part 3

Chris Smith — Thu, 14 May 2020 00:00:00 +0000

This article is the third part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we cover the last leg of the continuous delivery process to update your AWS resources and how to store sensitive data using Pulumi securely.

Recap

In part 1, we created a dedicated IAM User with limited privileges. Then in part 2, we set up a simple, serverless Pulumi program that periodically rotated the User’s access keys and updated the CI/CD system.

In this final post, we will use those AWS credentials to update cloud resources as part of a CI/CD workflow using Pulumi.

We will cover AWS’s IAM Roles and how they can be used to safely manage the access Pulumi has to your AWS account. We will also cover how sensitive data in your stack is stored within the Pulumi Service.

Assuming IAM Roles for Performing Updates

In the previous post, we automatically create and rotate the AWS access keys for an IAM User. So it seems natural to think that we should just use that same AWS access key to gain access to AWS resources. However, using the access keys for an IAM User has a few major drawbacks.

First, those access keys are long-lived. If we didn’t create that Pulumi program to rotate them automatically, the key would last forever. (Meaning that a bad actor, perhaps elsewhere, or far in the future, could take advantage of a compromised key.) Second, that access key could be used to do anything that the IAM User could. So you cannot differentiate the set of capabilities based on the task at hand. (For example, if updating the Pulumi stack website/staging, we might need a different set of permissions than if we were updating the stack website/production.)

The preferred alternative to using IAM User access keys is to “assume” an IAM Role. An IAM Role is similar to an IAM User in that it grants a set of policies and permissions. However, unlike an IAM User, an IAM Role does not have a password, nor can it be added to a Group and inherit any policies. Also, the credentials for using an IAM Role are always short-lived. (Lasting no more than 12 hours, and likely much, much less.)

Instead, an IAM Role defines a set of capabilities that can be performed by any principal capable of assuming it. So in other words, you can exchange your limited access IAM User’s credentials for a specialized IAM Role. The IAM Role then can be specifically tailored to a specific— wait for it— role to perform.

Don’t worry if that doesn’t all make sense at first; let’s look at a concrete example.

Desired End State

In our situation, we want to create an IAM Role WebsiteStackUpdaterRole that grants permissions necessary for Pulumi to update the website/production stack. We will custom tailor this role to only grant access to the set of resources and operations we know the stack needs.

We also want to restrict who can assume the WebsiteStackUpdaterRole, so that only specific users/accounts can access it. e.g., the IAM User cicd-bot but not IAM User dwight-over-in-sales.

And just to show a more real-world example, we’ll do this using multiple AWS accounts, where the website/production stack’s resources are housed in a “production” AWS account, and the cicd-bot IAM User is defined in a bastion AWS account.

Creating the IAM Role

The following snippet creates an IAM Role resource using Pulumi. The most important input property of which is the assumeRolePolicy document, which defines who can assume this role.

This is a situation where Pulumi’s ability to declare the policy document in code is super-useful, as it makes it much easier to reference other resources (e.g., another AWS account’s ID) as well as supporting inline comments.

// IAM Role used by cicd-bot to update the website stack.
export const websiteStackUpdaterRole = new aws.iam.Role("WebsiteStackUpdaterRole", {
 name: "WebsiteStackUpdaterRole",
 description: "Broad access for updating Pulumi stacks in the 'production' AWS account.",
 maxSessionDuration: 30 * 60, // 30 minutes
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: {
 // We grant the "bastion account", which defines IAM Users the ability to assume this role.
 // The IAM Policies in that other AWS account will then determine which users, groups, etc.
 // will be able to use the `sts:AssumeRole` action on this IAM Role.
 "AWS": `arn:aws:iam::${awsAccountIds.bastionAccount}:root`,
 },
 // But as an additional precaution against misconfiguration in the bastion account, we add
 // some additional checks to determine who can access this policy.
 Condition: {
 "ForAllValues:StringEquals": {
 "aws:PrincipalTag/access-to-production": "true",
 "sts:ExternalId": [
 "contoso/website/production",
 "contoso/website/staging",
 ],
 },
 },
 Action: "sts:AssumeRole",
 },
 ],
 },
});

There’s a lot packed into that IAM Role’s configuration.

First, the “principal” (who can assume the role) references a specific AWS account. This is an example of how you can associate one AWS account with another. (And is best practice for isolating data between different departments, etc.) We’ll cover how to configure the AWS bastion account side to use this role later.

The next thing to call out is the use of the Condition element of the policy document. To assume the role, we require two checks to be performed.

That the principal assuming the role (e.g., IAM User) has the tag "access-to-production" with value "true." This way, we will reject any IAM User in the bastion account who isn’t tagged with "access-to-production" from assuming this role.
We also verify that the “external ID” (an optional parameter passed when assuming an IAM Role) matches a well-known Pulumi stack name. This way, we can double-check that the entity using the IAM Role is doing so to use a specific stack. (This is less of a security measure and more of a general hygiene type requirement.)

IAM Policy Document

Now we have an IAM Role; we need to attach a policy document that defines what permissions are granted.

This is the most important step since it defines precisely what the CI/CD bot can able to do.

If we define the policy too broadly, then it is a general risk. For example, what if a compromised NPM package in your Pulumi program did something nefarious. If the policy is defined too narrowly, then your pulumi up might fail because the necessary permissions weren’t available. When in doubt, err on the side of having a more restrictive policy.

It can be advantageous to have an identical “staging” and “production” instance of the same Pulumi stack, so you can detect any permission issues before you update the user-facing production stack.

It can take some time to winnow down the specific set of permissions necessary since that is ultimately specific to the Pulumi program. For example, if the stack governs the networking configuration, then you can simply not deny to the s3:* actions.

Unfortunately, Pulumi doesn’t currently support generating a reasonable IAM Policy document for your stack. For a good introduction to some of the “worst-case scenarios” that could arise from a misconfigured IAM Policy, see this blog post from BishopFox.

The following is a hypothetical IAM policy that restricts the operations performed to just those for the AWS CloudFront and S3 products. (Which might be sufficient for updating a CDN-based website hosted on AWS, but likely not much else.)

// Policy document defining the permissions granted to the WebsiteStackUpdaterRole.
export const websiteUpdaterPolicyDocument: aws.iam.PolicyDocument = {
 Version: "2012-10-17",
 Statement: [
 // Grant access to all AWS S3 actions, but scoped to just the specific S3 buckets
 // used for the given Pulumi stack.
 {
 Sid: "GrantS3Access",
 Effect: "Allow",
 Action: "s3:*",
 Resource: [
 // Content bucket.
 "arn:aws:s3:::website-bucket",
 "arn:aws:s3:::website-bucket/*",
 // S3 bucket which stores the access logs.
 "arn:aws:s3:::website-logs-bucket",
 "arn:aws:s3:::website-logs-bucket/*",
 ],
 },
 // Grant access to update a CloudFront distribution. For more information, see:
 // https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/access-control-managing-permissions.html
 {
 Sid: "GrantCloudFrontAccess",
 Effect: "Allow",
 Action: [
 "cloudfront:CreateDistribution",
 "cloudfront:DeleteDistribution",
 "cloudfront:GetDistribution",
 "cloudfront:GetDistributionConfig",
 "cloudfront:ListDistributions",
 "cloudfront:UpdateDistribution",
 "cloudfront:ListCloudFrontOriginAccessIdentities",
 ],
 // The specific CloudFront distribution used in the stack.
 Resource: "arn:aws:cloudfront::0123456789:distribution/A123B1CABWZZ",
 },
 ],
};

const websiteUpdaterPolicy = new aws.iam.Policy("WebsiteUpdaterRolePolicy", {
 name: "WebsiteUpdaterRolePolicy",
 description: "Policy granting the permissions needed for updating the website/production Pulumi stack",
 policy: websiteUpdaterPolicyDocument,
}, resourceOptions);

To actually associate the IAM Policy document with the new IAM Role, we need to create a PolicyAttachment associating the two.

const rolePolicyAttachment = new aws.iam.RolePolicyAttachment("WebsiteStackUpdaterRolePolicyAttachment", {
 role: websiteStackUpdaterRole,
 policyArn: websiteUpdaterPolicy.arn,
});

Reminder, we defined the IAM Role in our production AWS account, which allows any principal in the AWS bastion account (with the right tag, external ID) to assume that role. We then need to update the bastion account to grant the sts:AssumeRole action on that IAM Role so it can be used! That isn’t covered in this post.

Assuming the IAM Role

Finally! With everyone in place, we can now use the IAM Role.

The following is a set of commands you can run on your CI/CD worker to exchange the low-privilege IAM User credentials for the specific IAM Role we just created in the production AWS account. But there are many alternatives for programmatically assuming an IAM Role, such as using the AWS_PROFILE environment variable) or the assume-role tool.

# Use the AWS credentials passed to the CI system and available as environment variables.
# These are the low-privilege credentials that automatically get rotated.
aws configure set aws_access_key_id ${AWS_ACCESS_KEY_ID}
aws configure set aws_secret_access_key ${AWS_SECRET_ACCESS_KEY}
aws configure set region "${AWS_REGION:-us-west-2}"

# Assume the desired IAM Role. Note how we pass the stack name in the --external-id parameter,
# as required to assume the role.
readonly CREDS_JSON=$(aws sts assume-role \
 --role-arn "${ARN_OF_IAM_ROLE_TO_ASSUME}" \
 --role-session-name "<useful description of the context, such as CI/CD job ID>" \
 --external-id "${TARGET_STACK}")

# The `aws sts assume-role` command just returns a JSON blob. So we now tease out the actual
# access key and set the corresponding environment variables.
export AWS_ACCESS_KEY_ID=$(echo "${CREDS_JSON}" | jq ".Credentials.AccessKeyId" --raw-output)
export AWS_SECRET_ACCESS_KEY=$(echo "${CREDS_JSON}" | jq ".Credentials.SecretAccessKey" --raw-output)
export AWS_SESSION_TOKEN=$(echo "${CREDS_JSON}" | jq ".Credentials.SessionToken" --raw-output)

# If everything worked correctly, then running the get-caller-identity command will refer to
# the IAM Role and not our IAM User.
echo "Assumed AWS identity:"
aws sts get-caller-identity

Storing Sensitive Data using Pulumi

As a developer, the most critical security threat is access to your cloud account since that’s ultimately where your or your customer’s data is stored. But beyond that, there are places where Pulumi stores sensitive data.

Pulumi goes to great lengths to safeguard your data. Here we review where Pulumi interacts with your sensitive data and what you can do to safeguard it better.

Secrets in Configuration

A Pulumi stack can have configuration data associated with it, which allows you to parameterize cloud resources. For example, a machine-size configuration key might determine which class of virtual machine to use. A dev stack might only need a "t2.small" but your production instance might need something more powerful.

Configuration data is stored in a file like Pulumi.website-production.yaml should be checked in along with the source code for a stack, so that builds and stack updates are reproducible.

However, what if those configuration settings contain secrets? Like you need to store an API key to use a 3rd party API.

Pulumi supports encrypting sensitive configuration data. You just need to add the --secret flag.

# Securely add an API key to the stack's configuration.
pulumi config set api-key "hunter2" --secret

For stacks hosted on the Pulumi Service, the default is that your configuration data is encrypted using a key specific to your stack. (So the ciphertext stored in the Pulumi.yaml file is safe to check into your source tree, since it cannot be copied/decrypted for another stack.)

Secrets in Checkpoint Files

Pulumi keeps track of your cloud resources in a something called a checkpoint file, and that too might contain sensitive information. For example, a Pulumi resource might have a "password" output property.

Pulumi has support to mark that resource output as “secret” and make sure that it is encrypted within the checkpoint file. (So if you were to look at the checkpoint file contents via pulumi stack export, you would not be able to recover the data.

Just like for secret configuration values, the default for stacks hosted on the Pulumi Service is to encrypt this data using a key that Pulumi manages and is specific to your stack.

Custom Secrets Providers

We at Pulumi take great care in safeguarding your data, especially configuration or checkpoint data that is marked as sensitive. However, you might feel more comfortable if your stack’s data were encrypted using a key that you controlled. (So even if the data stored in the Pulumi Service were available, it would be useless without your specific key.)

If that’s the case, then you can use a configurable secrets provider, and swap out the default “Pulumi Service managed” encryption scheme for your own. (And we won’t take it personally, promise.)

When you create a new stack using pulumi stack init, you can optionally specify a --secrets-provider flag. That will determine where and how secrets get managed on your stack.

For example, to use your own KMS key for encrypting data, you can pass the secrets provider "awskms://alias/ExampleAlias?region=us-east-1". The Pulumi command-line will work and behave the same. Except that whenever it needs to encrypt or decrypt some data, it will refer to the custom provider.

As you would expect, if you do use a custom secrets provider and lose access to your encryption key, then there is no way to recover the encrypted data stored on your stack.

For more information on custom secret providers, see Peace of Mind with Cloud Secret Providers or Managing Secrets with Pulumi.

Wrapping Up

Securely managing any sensitive data is challenging to get right, especially if there are many moving parts, such as when coordinating access between your AWS account, CI/CD provider, and your Pulumi stack.

In this blog series, we walked through the best practices in creating, securing, and managing access to an AWS account using IAM Users and Roles. We then reviewed the specifics of where and how Pulumi stores sensitive data, in case you want to take more control over your data.

If you have any other questions, corrections, or other ideas you’d like to discuss about Pulumi, Continuous Delivery, or security, please join the conversation over on the Pulumi Community Slack.

Managing AWS Credentials on CI/CD: Part 2

Chris Smith — Thu, 26 Mar 2020 00:00:00 +0000

This article is the second part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we go in-depth on providing AWS credentials securely to a 3rd party and introduce a Pulumi program to automate rotating access keys.

NOTE: These recommendations do not apply if you are running a CI/CD system within your AWS account, e.g., running a Jenkins server on EC2 or using AWS CodeDeploy. In those cases, please refer to AWS’s documentation for how to assume IAM Roles when running on an EC2 instance instead.

Provide IAM credentials to your CI/CD system

In the first post in our series, we created a dedicated IAM User to perform updates to AWS resources within your CI/CD system. The next step is to pass the AWS access keys for that user to your CI/CD system.

We need to take great caution. AWS’s documentation states, “Do not provide your access keys to a third party”.

Providing AWS access keys to any system is dangerous because of the risk that a bad actor could obtain those keys and do something nefarious. Even if your CI/CD service takes great care to protect your secrets, those AWS credentials could be inadvertently exposed in debugging output, system logs, or in some other way.

This is one of the main advantages of performing continuous deployment from within your AWS account. For example, running Jenkins on an EC2 instance you manage, or using the AWS CodeDeploy service. Because when performing a deployment from within an AWS compute environment, you can use built-in mechanisms to obtain credentials securely.

However, there are many reasons to use a hosted service for performing your CI/CD, such as developer productivity, ease of use, performance, or simply because you don’t want to recreate your existing deployment workflows.

Ultimately, it is essential to understand the risks involved and how to mitigate them.

Properly Storing Credentials

If you choose to provide your CI/CD with credentials, the most important thing is to mark them as secret. Your CI/CD provider typically has built-in support for handling “secure variables” or other sensitive information. As opposed to general, configuration data or environment variables.

User-supplied configuration values are typically write-only and only accessible by the CI/CD worker jobs at runtime. For example, Travis CI has encrypted environment variables or secure environment variables in CircleCI.

Please consult your CI/CD provider’s documentation for how to pass and store sensitive information appropriately.

Why not use a “secrets manager” service?

A common question when discussing is how to pass access keys to a CI/CD system securely:

Rather than giving your CI/CD provider AWS credentials, why not have your CI/CD system obtain credentials from a specialized “secrets manager” service?

In other words, if you choose not to trust your CI/CD system with this data, can you instead trust some other system dedicated for securely storing and retrieving sensitive information?

It sounds like a good idea on the surface but doesn’t seem to make your data any more secure.

Abstractly, the difference here is that the credentials are provided on-demand, rather than being available to the CI/CD job when it starts (and stored via the CI/CD provider). Instead, your CI/CD job would obtain credentials from the “secrets manager” only when needed.

There are some advantages to this approach, such as providing a clear audit trail for access and more control over the distribution of sensitive information.

However, the secrets provider system needs to be presented with some form of credentials. And those credentials need to be available to your CI/CD environment. So using a secrets manager leaves you in the same place you started, i.e., needing to provide sensitive data to your CI/CD provider.

Also, by adding a dependency on a secrets manager, you introduce additional risks. Not only do you need to be even more security-conscious about that secrets manager, but it also needs to be highly available. Any outage for that service would mean that you would be unable to perform deployments!

So it does not seem that using a secrets manager to dole out AWS credentials to your CI/CD system is a good practice to follow. Or, at the very least, makes some tradeoffs without fundamentally making your approach to CI/CD any more or less secure.

Instead, it is better to focus on other ways to reduce the risk if those AWS credentials do get exposed.

Automate Rotating and Revoking AWS Credentials

When providing credentials to a 3rd party, rather than hoping it will be 100% secure forever (which is impossible), we can make those credentials volatile. If we regularly invalidate and rotate the credentials supplied to the CI/CD system, we can dramatically reduce the impact of any accidental disclosure. Even if the IAM User’s credentials were leaked, by the time they were discovered and used, they would no longer be valid.

The AWS security blog describes how to rotate access keys for IAM Users, by using the following steps:

Create a second access key in addition to the one in use.
Update all your applications to use the new access key.
Change the state of the previous access key to inactive.
Verify everything is still working as expected.
Delete the inactive access key.

That all sounds simple enough, but it’s certainly tedious. And if you need to repeat the process across dozens, if not hundreds of different CI/CD pipelines, you need a better solution. Fortunately, Pulumi provides an excellent and extensible way for writing a serverless program to automate rotating credentials.

The next few sections describe a simple infrastructure application for automating AWS IAM credential rotation. You can see the full application on GitHub at chrsmith/pulumi-aws-travis-cicd-demo.

AWS Credential Rotator 9000

The affectionally titled AWS Credential Rotator 9000 is a simple, serverless Pulumi application for rotating AWS access keys.

It creates an AWS lambda function that’s triggered on a regular schedule, e.g., every 30 minutes, and performs the next step in the sequence for rotating access keys as outlined above.

First, it creates a new access key and pushes the new value out. On the next iteration, it marks the older access key as “inactive.” On the following iteration, the inactive key is deleted. The process repeats, generating a new key and removing the inactive key.

Periodically Invoking an AWS Lambda

The heart of the application is triggering it to execute on a fixed interval. Thankfully this is super-easy to do using Pulumi since it allows you to seamlessly blend your “cloud infrastructure” with “code” in a natural way. The user guide for Pulumi Crosswalk for AWS has more information on serverless eventing if you would like to learn more.

The following snippet is the core part of the credential rotator app. We define a function to handle the logic of key rotation in rotateIAMUserKeys. Then we create an AWS Lambda resource named lambda. Finally, the triggerSchedule resource invokes our lambda on a fixed schedule, thereby ensuring that the key rotation process goes on indefinitely. You can see the full code here.

async function rotateIAMUserKeys(iamUser: aws.iam.User, onNewCredentials: (key: string, secret: string) => Promise<void>) {
 ...
}

const lambda = new aws.lambda.CallbackFunction<aws.cloudwatch.EventRuleEvent, void>(
 "keyRotatorLambda",
 {
 callback: async (e) => {
 await rotateIAMUserKeys(userToRotate, async (newKey: string, newSecret: string) => {
 console.log("onNewCredentials callback called!");
 console.log(`A new AWS access key "${newKey}" has been created.`);
 });
 },

 role: lambdaRole,
 runtime: "nodejs10.x",
 });

const triggerSchedule = aws.cloudwatch.onSchedule(
 "keyRotatorScheduler", "rate(1 hour)", lambdaCallbackFn);

When writing reusable infrastructure components in Pulumi however, it is helpful to organize things into a custom resource.

For example, we can bundle together the AWS Lambda, CloudWatcn schedule, and the associated IAM policies into a single conceptual resource AccessKeyRotator. Bundling resources allows for the code reuse.

const rotator = new AccessKeyRotator("rotator", {
 interval: config.require("rate"),
 user: user,
 credentialPusher: demoTravisCIPusher,
})

Pushing Credentials

With the mechanics of updating an AWS access key out of the way, the next step is to notify dependent systems what the new access key should be. The key rotator app has a CredentialPusher abstraction to provide a pluggable way for you to send credentials to where they need to be. The example on GitHub only supports updating a Travis CI project (See credential-pusher-travis.ts.), but it could be easily extended to support other CI/CD systems.

The following snippet shows how the CredentialPusher is configured in the key rotator app. We load a Travis CI access token from the Pulumi stack’s configuration, and hard-code the specific set of projects and encrypted environment variables to store the new access key.

// A "credential pusher" is the component that pushes new AWS IAM credentials out to 3rd parties
// as the older ones get rotated. For demonstration purposes, this updates the Travis CI settings
// for the chrsmith/pulumi-aws-travis-cicd-demo repo. But you can imagine another implementation
// that pushes the new IAM credentials to GitLab CI, or updating multiple CI/CD pipelines.

const demoTravisCIPusher = new CredentialPusher(
 new TravisCIPusher(),
 {
 accessKey: config.require("travis-ci-token"),
 projects: [
 {
 project: "chrsmith/pulumi-aws-travis-cicd-demo",
 // In the Travis CI configuration for that GitHub repo, there are two
 // environment variables for storing the AWS credentials. So whenever the
 // AWS credentials get rotated, the job's configuration settings will be
 // updated to reflect the new values.
 accessKeyIDLocation: "AWS_ACCESS_KEY_ID",
 secretAccessKeyLocation: "AWS_SECRET_ACCESS_KEY",
 }
 ],
 });

Demo

To demonstrate the access keys rotation, we can examine the log files generated from AWS Lambda. These can be accessed from the command-line using the pulumi logs command.

Here’s a summary of the output for clarity:

 START RequestId: 9406913e-a9d0-46a4-b861-2efdee508b2b Version: $LATEST
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO IAM User has 2 keys:
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO - AKIASHIVKXX3SA2K3ME7 [Inactive] Tue Mar 24 2020 09:37:03 GMT+0000 (Coordinated Universal Time)
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO Deleting older, inactive access key AKIASHIVKXX3SA2K3ME7 [Inactive]
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO Key rotation step complete.
 END RequestId: 9406913e-a9d0-46a4-b861-2efdee508b2b

 START RequestId: 05644e4a-81a4-4197-8ad7-fc3517998270 Version: $LATEST
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO IAM User has 1 keys:
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Created new key key AKIASHIVKXX3YX4IXRVH
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Pushing out the new key to 3rd party services...
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Pushing new credentials to Travis CI project 'chrsmith/pulumi-aws-travis-cicd-demo'
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updating env var 'AWS_ACCESS_KEY_ID' (b246ab3b-c4cb-e76f-a6fc-50f4cbf451c0)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updated AWS access key ID. Got response code (200)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updating env var 'AWS_SECRET_ACCESS_KEY' (ba9ac4be-fbb3-4da9-c5b8-35a1afe9e02b)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updated AWS secret access key. Got response code (200)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Key rotation step complete.
 END RequestId: 05644e4a-81a4-4197-8ad7-fc3517998270

 START RequestId: ca204959-32a7-44a8-ba20-738d139542ba Version: $LATEST
 ca204959-32a7-44a8-ba20-738d139542ba INFO IAM User has 2 keys:
 ca204959-32a7-44a8-ba20-738d139542ba INFO - AKIASHIVKXX3YX4IXRVH [Active] Tue Mar 24 2020 15:37:03 GMT+0000 (Coordinated Universal Time)
 ca204959-32a7-44a8-ba20-738d139542ba INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 ca204959-32a7-44a8-ba20-738d139542ba INFO Invalidating older access key AKIASHIVKXX3QEQNZNX5
 ca204959-32a7-44a8-ba20-738d139542ba INFO Key rotation step complete.
 END RequestId: ca204959-32a7-44a8-ba20-738d139542ba

 START RequestId: de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 Version: $LATEST
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO IAM User has 2 keys:
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO - AKIASHIVKXX3YX4IXRVH [Active] Tue Mar 24 2020 15:37:03 GMT+0000 (Coordinated Universal Time)
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO - AKIASHIVKXX3QEQNZNX5 [Inactive] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO Deleting older, inactive access key AKIASHIVKXX3QEQNZNX5 [Inactive]
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO Key rotation step complete.
 END RequestId: de753fb4-0720-4f4f-a7aa-ad6b9d4c8832

As you can see, the AWS credentials for the IAM User are automatically updated every few hours. And whenever a new access key is created, the value is pushed out to the impacted Travis CI projects automatically.

Wrapping Up

In this post, we covered some of the things to consider when providing AWS credentials to your CI/CD system. (In short, be cautious and follow best practices.)

We then showed how relatively easy it is to stand up a Pulumi infrastructure application for automatically rotating AWS access keys and updating your CI/CD system. By rotating credentials, you can limit the impact if the value is inadvertently disclosed.

At this point, we’ve now given access keys to a CI/CD system for a low-privilege AWS IAM User account. But we still can’t do anything with it. The IAM User whose credentials we have given to the CI/CD system doesn’t have access to your production AWS account.

In the next post, we’ll go into the details about IAM Roles, and they can be used to securely and temporarily gain access to additional resources. This is how our low-privilege IAM User can access needed to update production data for your CI/CD pipelines.

Managing AWS Credentials on CI/CD: Part 1

Chris Smith — Thu, 12 Mar 2020 00:00:00 +0000

Continuous delivery requires providing highly sensitive credentials to your deployment pipeline. Understanding the risks, mitigations, and best practices for handling those credentials can be difficult. In this guide, we describe the best practices for providing AWS credentials to a CI/CD system and to securely automate updating your cloud infrastructure using Pulumi.

You’ve heard about Infrastructure as Code and decided to try your hand at using Pulumi to manage cloud infrastructure using familiar programming languages. Naturally, you’re curious and eager to start updating your cloud infrastructure in your CI/CD pipeline – but then panic hits… Is updating your cloud infrastructure secure? If you start doing cloud deployments in your CI/CD system, doesn’t that mean you need to give it your production credentials? Aren’t those the most sensitive things your company has?!?

Take a deep breath. It’s going to be OK. You can securely provide AWS credentials to your CI/CD system and practice continuous delivery. This is the first post in a series going in-depth on how to do just that.

The goal of this article series is to give you a clear understanding of AWS credential management and how that relates to using Pulumi within a CI/CD environment. Once you have the AWS credentials in-place, you can then follow our Continuous Delivery guide for configuring your specific CI/CD service, whether you want to use CircleCI, GitLab CI, or Travis CI.

NOTE: These recommendations do not apply if you are running your own CI/CD system within your AWS account, e.g., running a Jenkins server on EC2 or using AWS CodeDeploy. In those cases, please refer to AWS’s documentation for how to assume IAM Roles when running on an EC2 instance instead.

Overview

The recommendations in this series describe a general “one-size fits most” approach for credential management, which requires a minimal amount of work to configure and maintain. Depending on your specific environment, needs, and constraints, there may be a better alternative for your use case.

Create a dedicated IAM User for your CI/CD

The first step for securely automating CI/CD is to create a dedicated IAM User for use in your CI/CD pipelines. (Sometimes referred to as a “robot account.”)

The following code snippet shows how to create a new AWS IAM User using Pulumi. (The code is in TypeScript, but you could another language like Python, C#, or Go as well.)

const user = new aws.iam.User("cicdUser", {
 name: "cicd-bot",
 tags: {
 "purpose": "Account used to perform Pulumi stack updates on CI/CD.",
 }
});

By default, that IAM User doesn’t have permissions to do anything. The recommended best practice is to use groups to assign permissions to IAM Users. So next, we will create a “Pulumi Stack updaters” IAM Group, and add our robot account as a member.

const group = new aws.iam.Group("pulumiStackUpdaters", {
 name: "PulumiStackUpdaters",
});

const groupMembership = new aws.iam.GroupMembership("cicdUserMembership", {
 group: group.name,
 users: [ user.name ],
});

Group Permissions

The next part is where things start to get tricky. We now need to grant permissions to this IAM Group’s members so they can access various APIs and resources. (For example, if your Pulumi program will need to create or update EC2 instances, then you will need to provide AWS credentials that contain the ec2:StopInstances or ec2:StartInstances action.)

But rather than grant the newly created cicd-bot IAM User those permissions, we will instead only grant them access to the sts:AssumeRole action. This is an AWS API that allows an IAM User to gain temporary credentials that have a different set of permissions associated with them. (That is, the permissions associated with an IAM Role.)

We’ll go into more detail about the exact difference between an IAM User and an IAM Role in the next post in the series since it isn’t entirely obvious why one would be more secure than the other.

const currentAwsIdentity = aws.getCallerIdentity();

const groupPolicy = new aws.iam.GroupPolicy("pulumiStackUpdatersPolicy", {
 group: group.name,
 policy: {
 Version: "2012-10-17",
 Statement: [{
 Action: [
 // Allow anybody (i.e., members of the group) to call the sts:AssumeRole API.
 // This allows them to "assume the role" of a more permissive IAM Role
 // when they go to update a stack later.
 "sts:AssumeRole",
 ],
 Effect: "Allow",
 // This is the set of resources that the "sts:AssumeRole" operation could be
 // performed on, which is to say any IAM role in the current AWS account.
 Resource: pulumi.interpolate `arn:aws:iam::${currentAwsIdentity.accountId}:role/*`,
 Sid: ""
 }],
 },
});

That’s it for the first post in our series. We’ve taken the first step and created an IAM User specifically for performing updates to AWS resources using Pulumi.

In the next post, we will review over the security concerns when passing AWS credentials to your CI/CD provider, as well as show a Pulumi program that you can use to rotate the AWS credentials for a user automatically.